How to decode IEX DEEP pcap fileHow to merge two dictionaries in a single expression?How do I check if a list is empty?How do I check whether a file exists without exceptions?How do I copy a file in Python?How can I safely create a nested directory?How do I sort a dictionary by value?How do I list all files of a directory?How to read a file line-by-line into a list?unable to decode Python web requestReading binary from pickle file
Prime parity peregrination
Put my student loan in parents’ second mortgage - help?
Who voices the character "Finger" in The Fifth Element?
Details of video memory access arbitration in Space Invaders
Graph problems as integer programs
Why do changes to /etc/hosts take effect immediately?
What game is this character in the Pixels movie from?
What exactly did Ant-Man see that made him say that their plan worked?
Donkey as Democratic Party symbolic animal
Is there a canon reason why Klingon and Romulan vessels are so similar in shape?
Why won't the ground take my seed?
How to get a character's limb regrown at 3rd level?
Do launching rockets produce a sonic boom?
How is this practical and very old scene shot?
Procedurally generate regions on island
Was it really unprofessional of me to leave without asking for a raise first?
Should fiction mention song names and iPods?
What's the safest way to inform a new user of their password on an invite-only website?
How can a valley surrounded by mountains be fertile and rainy?
Should I report a leak of confidential HR information?
Balanced parentheses using STL C++
Using a concentration spell on top of another spell from another spell list?
Just graduated with a master’s degree, but I internalised nothing
Step into the Octagram
How to decode IEX DEEP pcap file
How to merge two dictionaries in a single expression?How do I check if a list is empty?How do I check whether a file exists without exceptions?How do I copy a file in Python?How can I safely create a nested directory?How do I sort a dictionary by value?How do I list all files of a directory?How to read a file line-by-line into a list?unable to decode Python web requestReading binary from pickle file
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am trying to decode DEEP data from IEX(the Investors Exchange), which is in pcap form. I am currently working in Python.
I am currently using Scapy to parse pcap file. The problem is that I do not know how to decode the raw load(binary form) into human-readable text.
Here is an example.
from scapy.all import *
packets = rdpcap(r'20171104_IEXTP1_DEEP1.0.pcap')
packets[0].show()
The result is like this
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 16005
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0x49e2
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x6c15
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00BDx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00xce/xec:x98xdexf3x14'
Is there any way I can know the content of the raw load there? It is supposed to be information about the order book data. I do not see any existing tool to decode this.
If interested, you can access those files from the official website of IEX(https://iextrading.com/trading/market-data/#deep). Those pcap files are usually pretty large, so I am trying on a small one 20171104_IEXTP1_DEEP1.0.pcap.
python scapy pcap
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
add a comment |
I am trying to decode DEEP data from IEX(the Investors Exchange), which is in pcap form. I am currently working in Python.
I am currently using Scapy to parse pcap file. The problem is that I do not know how to decode the raw load(binary form) into human-readable text.
Here is an example.
from scapy.all import *
packets = rdpcap(r'20171104_IEXTP1_DEEP1.0.pcap')
packets[0].show()
The result is like this
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 16005
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0x49e2
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x6c15
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00BDx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00xce/xec:x98xdexf3x14'
Is there any way I can know the content of the raw load there? It is supposed to be information about the order book data. I do not see any existing tool to decode this.
If interested, you can access those files from the official website of IEX(https://iextrading.com/trading/market-data/#deep). Those pcap files are usually pretty large, so I am trying on a small one 20171104_IEXTP1_DEEP1.0.pcap.
python scapy pcap
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
Can you share sample PCAP file?
– Alderven
Mar 25 at 13:23
@Alderven Sure. I do not know how to attach files, so I edit the post to add the website to download those files. Hope it helps.
– Li Zou
Mar 25 at 13:34
Have you read specification?
– Alderven
Mar 25 at 14:44
@Alderven Yes, but I have no experience dealing with decoding problems. Even with specification, I have no clue how to get started.
– Li Zou
Mar 25 at 15:19
You might need to implement the protocol in Scapy, so that you can access each field
– Cukic0d
Mar 25 at 19:20
add a comment |
I am trying to decode DEEP data from IEX(the Investors Exchange), which is in pcap form. I am currently working in Python.
I am currently using Scapy to parse pcap file. The problem is that I do not know how to decode the raw load(binary form) into human-readable text.
Here is an example.
from scapy.all import *
packets = rdpcap(r'20171104_IEXTP1_DEEP1.0.pcap')
packets[0].show()
The result is like this
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 16005
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0x49e2
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x6c15
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00BDx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00xce/xec:x98xdexf3x14'
Is there any way I can know the content of the raw load there? It is supposed to be information about the order book data. I do not see any existing tool to decode this.
If interested, you can access those files from the official website of IEX(https://iextrading.com/trading/market-data/#deep). Those pcap files are usually pretty large, so I am trying on a small one 20171104_IEXTP1_DEEP1.0.pcap.
python scapy pcap
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
I am trying to decode DEEP data from IEX(the Investors Exchange), which is in pcap form. I am currently working in Python.
I am currently using Scapy to parse pcap file. The problem is that I do not know how to decode the raw load(binary form) into human-readable text.
Here is an example.
from scapy.all import *
packets = rdpcap(r'20171104_IEXTP1_DEEP1.0.pcap')
packets[0].show()
The result is like this
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 16005
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0x49e2
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x6c15
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00BDx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00xce/xec:x98xdexf3x14'
Is there any way I can know the content of the raw load there? It is supposed to be information about the order book data. I do not see any existing tool to decode this.
If interested, you can access those files from the official website of IEX(https://iextrading.com/trading/market-data/#deep). Those pcap files are usually pretty large, so I am trying on a small one 20171104_IEXTP1_DEEP1.0.pcap.
python scapy pcap
python scapy pcap
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
edited Mar 25 at 20:18
jfleach
3061 gold badge3 silver badges17 bronze badges
3061 gold badge3 silver badges17 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
asked Mar 25 at 13:17
Li ZouLi Zou
62 bronze badges
62 bronze badges
Can you share sample PCAP file?
– Alderven
Mar 25 at 13:23
@Alderven Sure. I do not know how to attach files, so I edit the post to add the website to download those files. Hope it helps.
– Li Zou
Mar 25 at 13:34
Have you read specification?
– Alderven
Mar 25 at 14:44
@Alderven Yes, but I have no experience dealing with decoding problems. Even with specification, I have no clue how to get started.
– Li Zou
Mar 25 at 15:19
You might need to implement the protocol in Scapy, so that you can access each field
– Cukic0d
Mar 25 at 19:20
add a comment |
Can you share sample PCAP file?
– Alderven
Mar 25 at 13:23
@Alderven Sure. I do not know how to attach files, so I edit the post to add the website to download those files. Hope it helps.
– Li Zou
Mar 25 at 13:34
Have you read specification?
– Alderven
Mar 25 at 14:44
@Alderven Yes, but I have no experience dealing with decoding problems. Even with specification, I have no clue how to get started.
– Li Zou
Mar 25 at 15:19
You might need to implement the protocol in Scapy, so that you can access each field
– Cukic0d
Mar 25 at 19:20
Can you share sample PCAP file?
– Alderven
Mar 25 at 13:23
Can you share sample PCAP file?
– Alderven
Mar 25 at 13:23
@Alderven Sure. I do not know how to attach files, so I edit the post to add the website to download those files. Hope it helps.
– Li Zou
Mar 25 at 13:34
@Alderven Sure. I do not know how to attach files, so I edit the post to add the website to download those files. Hope it helps.
– Li Zou
Mar 25 at 13:34
Have you read specification?
– Alderven
Mar 25 at 14:44
Have you read specification?
– Alderven
Mar 25 at 14:44
@Alderven Yes, but I have no experience dealing with decoding problems. Even with specification, I have no clue how to get started.
– Li Zou
Mar 25 at 15:19
@Alderven Yes, but I have no experience dealing with decoding problems. Even with specification, I have no clue how to get started.
– Li Zou
Mar 25 at 15:19
You might need to implement the protocol in Scapy, so that you can access each field
– Cukic0d
Mar 25 at 19:20
You might need to implement the protocol in Scapy, so that you can access each field
– Cukic0d
Mar 25 at 19:20
add a comment |
1 Answer
1
active
oldest
votes
If you are looking for a more readable dump of the load, try using the Scapy hexdump method:
from scapy.all import *
from scapy.utils import hexdump
packets = rdpcap(r'20180127_IEXTP1_DEEP1.0.pcap')
packets[0].show()
raw = packets[0].lastlayer()
hexdump(raw)
Which will output something like this:
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 53693
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0xb6a9
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x955b
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00x96Dx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00x91x08xf0M:xacrx15'
0000 01000480010000000000964400000000 ...........D....
0010 00000000000000000100000000000000 ................
0020 9108F04D3AAC0D15 ...M:...
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55338707%2fhow-to-decode-iex-deep-pcap-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you are looking for a more readable dump of the load, try using the Scapy hexdump method:
from scapy.all import *
from scapy.utils import hexdump
packets = rdpcap(r'20180127_IEXTP1_DEEP1.0.pcap')
packets[0].show()
raw = packets[0].lastlayer()
hexdump(raw)
Which will output something like this:
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 53693
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0xb6a9
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x955b
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00x96Dx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00x91x08xf0M:xacrx15'
0000 01000480010000000000964400000000 ...........D....
0010 00000000000000000100000000000000 ................
0020 9108F04D3AAC0D15 ...M:...
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
add a comment |
If you are looking for a more readable dump of the load, try using the Scapy hexdump method:
from scapy.all import *
from scapy.utils import hexdump
packets = rdpcap(r'20180127_IEXTP1_DEEP1.0.pcap')
packets[0].show()
raw = packets[0].lastlayer()
hexdump(raw)
Which will output something like this:
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 53693
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0xb6a9
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x955b
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00x96Dx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00x91x08xf0M:xacrx15'
0000 01000480010000000000964400000000 ...........D....
0010 00000000000000000100000000000000 ................
0020 9108F04D3AAC0D15 ...M:...
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
add a comment |
If you are looking for a more readable dump of the load, try using the Scapy hexdump method:
from scapy.all import *
from scapy.utils import hexdump
packets = rdpcap(r'20180127_IEXTP1_DEEP1.0.pcap')
packets[0].show()
raw = packets[0].lastlayer()
hexdump(raw)
Which will output something like this:
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 53693
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0xb6a9
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x955b
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00x96Dx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00x91x08xf0M:xacrx15'
0000 01000480010000000000964400000000 ...........D....
0010 00000000000000000100000000000000 ................
0020 9108F04D3AAC0D15 ...M:...
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
If you are looking for a more readable dump of the load, try using the Scapy hexdump method:
from scapy.all import *
from scapy.utils import hexdump
packets = rdpcap(r'20180127_IEXTP1_DEEP1.0.pcap')
packets[0].show()
raw = packets[0].lastlayer()
hexdump(raw)
Which will output something like this:
###[ Ethernet ]###
dst = 01:00:5e:57:15:04
src = 00:1e:67:f2:62:24
type = 0x800
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 68
id = 53693
flags = DF
frag = 0
ttl = 64
proto = udp
chksum = 0xb6a9
src = 23.226.155.132
dst = 233.215.21.4
options
###[ UDP ]###
sport = 10378
dport = 10378
len = 48
chksum = 0x955b
###[ Raw ]###
load = 'x01x00x04x80x01x00x00x00x00x00x96Dx00x00x00x00x00x00x00x00x00x00x00x00x01x00x00x00x00x00x00x00x91x08xf0M:xacrx15'
0000 01000480010000000000964400000000 ...........D....
0010 00000000000000000100000000000000 ................
0020 9108F04D3AAC0D15 ...M:...
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
answered Mar 25 at 18:43
jfleachjfleach
3061 gold badge3 silver badges17 bronze badges
3061 gold badge3 silver badges17 bronze badges
add a comment |
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55338707%2fhow-to-decode-iex-deep-pcap-file%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you share sample PCAP file?
– Alderven
Mar 25 at 13:23
@Alderven Sure. I do not know how to attach files, so I edit the post to add the website to download those files. Hope it helps.
– Li Zou
Mar 25 at 13:34
Have you read specification?
– Alderven
Mar 25 at 14:44
@Alderven Yes, but I have no experience dealing with decoding problems. Even with specification, I have no clue how to get started.
– Li Zou
Mar 25 at 15:19
You might need to implement the protocol in Scapy, so that you can access each field
– Cukic0d
Mar 25 at 19:20