I was searching for a long time to find a framework can support fast & custom network traffic analysis.
some specific features of traffic data from monitor, such as interval of SYN and SYN-ACK, should be extracted and grouped by host.
i find Bro is so widely used, which seems can fulfill the requirement.
Can i disable other functions embedded in Bro, and add a plugin myself?
What is the point to archieve this, modify the core .cpp source file or add a .bro file?
i find Bro is so widely used, which seems can fulfill the requirement.
Good to hear it works for you and welcome aboard!
Can i disable other functions embedded in Bro, and add a plugin myself?
With the upcoming release of 2.0, Bro enables policy-neutral protocol
analysis by default, meaning it gives you a neutral picture of what's
going on in your network. For additional analyses and detectors, you
need to load the corresponding scripts in the policy directory;
local.bro is a good starting point. That said, you only pay for basic
protocol decoding by default.
What is the point to archieve this, modify the core .cpp source file or add
a .bro file?
This depends on the functionality you would like to add. Would you
mind elaborating a bit so that we can give you more helpful advice?
Changing the format of the log output or modifying analyzer behavior
generally works at the scripting layer. Bro features a Turing-complete
scripting language. You can write your own new functions and events.
If you would like to haul C/C++ functionality up to the scripting
layer, you might want to consider writing your own built-in function
(BiF). See src/bro.bif for examples. If you would like to add a new
protocol analyzer, then BinPAC is the right tool for you.
Matthias
i find Bro is so widely used, which seems can fulfill the requirement.
Good to hear it works for you and welcome aboard!
It is a greate platform, thank you for your works.
Can i disable other functions embedded in Bro, and add a plugin myself?
With the upcoming release of 2.0, Bro enables policy-neutral protocol
analysis by default, meaning it gives you a neutral picture of what's
going on in your network. For additional analyses and detectors, you
need to load the corresponding scripts in the policy directory;
local.bro is a good starting point. That said, you only pay for basic
protocol decoding by default.What is the point to archieve this, modify the core .cpp source file or add
a .bro file?This depends on the functionality you would like to add. Would you
mind elaborating a bit so that we can give you more helpful advice?
Changing the format of the log output or modifying analyzer behavior
generally works at the scripting layer. Bro features a Turing-complete
scripting language. You can write your own new functions and events.
If you would like to haul C/C++ functionality up to the scripting
layer, you might want to consider writing your own built-in function
(BiF). See src/bro.bif for examples. If you would like to add a new
protocol analyzer, then BinPAC is the right tool for you.
I want to match tcp handshake pairs and record the intervals between
each SYN and SYN-ACK pairs with their arrival time. At the same time,
roughly packet loss rate (vs different timescales) should be calculated
by tcp retransmission rate. It is a statistical analysis on network traffic
that would be processed by .bro files i think. some of them are similar
with functions already existed. Would you please give me some notes
on which files i should start with?
btw: I read the document and find that all C/C++ code is designed
for decoding packets. bro files take charge in statistal or general processing.
Is it right? Any general pictures were provided in bro?
I want to match tcp handshake pairs and record the intervals between
each SYN and SYN-ACK pairs with their arrival time. At the same time,
roughly packet loss rate (vs different timescales) should be calculated
by tcp retransmission rate. It is a statistical analysis on network
traffic that would be processed by .bro files i think. some of them
are similar with functions already existed. Would you please give me
some notes on which files i should start with?
Yes, you're right. This sort of analysis can be entirely done at the
scripting layer, i.e., it only involves *.bro scripts.
Bro reassembles the full TCP byte stream. When you deal with connection
data, duplicates/retransmission are already removed. If you want to
compute round-trip times at the packet level, you can write a handler
for the events new_connection (which is generated for every new
SYN packet) and connection_established (which is generated for a
successful TCP handshake after a SYN-ACK).
For retransmissions, have a look at the events: rexmit_inconsistency,
content_gap, and gap_report. Unfortunately I cannot provide more
detailed information other than pointing you to our ongoing
documentation effort in the git repository:
git clone git://git.bro-ids.org/bro.git
git checkout topic/script-reference
less src/event.bif
Maybe others can chime in and give you further guidance.
(Also, to measure system/NIC capture loss, there is
policy/misc/capture-loss.bro.)
btw: I read the document and find that all C/C++ code is designed
for decoding packets. bro files take charge in statistal or general
processing. Is it right? Any general pictures were provided in bro?
That's correct. Packet "decoding" is done at the Bro core. Bro
reassembles the TCP byte stream and presents it as a connection to the
user. You may find our workshop materials helpful to better understand
the architecture of Bro: http://www.bro-ids.org/bro-workshop-2011
Matthias
At the same time, roughly packet loss rate (vs different timescales) should be calculated
by tcp retransmission rate.
Are you interested in the packet loss rate for the three-way handshake only, or for all retransmissions in each connection?
What level of detail are you looking for? (IE. Just that the segment was retransmitted or perhaps more specific data about how it was retransmitted such as in response to a retransmission timeout or other recovery method?)
At the same time, roughly packet loss rate (vs different timescales)
should be calculated
by tcp retransmission rate.Are you interested in the packet loss rate for the three-way handshake
only, or for all retransmissions in each connection?
all retransmissions should be collected.
What level of detail are you looking for? (IE. Just that the segment was
retransmitted or perhaps more specific data about how it was retransmitted
such as in response to a retransmission timeout or other recovery method?)
it would be better that retrans mode were detected.
I want to match tcp handshake pairs and record the intervals between
each SYN and SYN-ACK pairs with their arrival time. At the same time,
roughly packet loss rate (vs different timescales) should be calculated
by tcp retransmission rate. It is a statistical analysis on network
traffic that would be processed by .bro files i think. some of them
are similar with functions already existed. Would you please give me
some notes on which files i should start with?Yes, you're right. This sort of analysis can be entirely done at the
scripting layer, i.e., it only involves *.bro scripts.Bro reassembles the full TCP byte stream. When you deal with connection
data, duplicates/retransmission are already removed. If you want to
compute round-trip times at the packet level, you can write a handler
for the events new_connection (which is generated for every new
SYN packet) and connection_established (which is generated for a
successful TCP handshake after a SYN-ACK).For retransmissions, have a look at the events: rexmit_inconsistency,
content_gap, and gap_report. Unfortunately I cannot provide more
detailed information other than pointing you to our ongoing
documentation effort in the git repository:git clone git://git.bro-ids.org/bro.git
git checkout topic/script-reference
less src/event.bifMaybe others can chime in and give you further guidance.
(Also, to measure system/NIC capture loss, there is
policy/misc/capture-loss.bro.)btw: I read the document and find that all C/C++ code is designed
for decoding packets. bro files take charge in statistal or general
processing. Is it right? Any general pictures were provided in bro?That's correct. Packet "decoding" is done at the Bro core. Bro
reassembles the TCP byte stream and presents it as a connection to the
user. You may find our workshop materials helpful to better understand
the architecture of Bro: http://www.bro-ids.org/bro-workshop-2011Matthias
Thank you very much. It is very useful!
Correct me if I am wrong Matthias.
Bro can do most of what you are looking for out of the box. Sampling the round-trip time of the three-way handshake is doable at scripting level. You can get bro to output the retransmission data through the tcp_rexmit event. It does not give detailed information about how the data was retransmitted but will tell you how many bytes were retransmitted and how much data was outstanding when the retransmission occurred.
That should be sufficient for what you are looking for.
If you need more detailed information, I am currently working on an analyzer for Bro that attempts to give more detailed information about the retransmission behavior of a TCP connection as part of on-going research. However, It is not in a state that is ready for release.
If you need more detailed information, I am currently working on an
analyzer for Bro that attempts to give more detailed information about the
retransmission behavior of a TCP connection as part of on-going research.
You should for sure contact Katrina LaCurts <katrina@csail.mit.edu>,
who did an internship with us working on integrating this sort of analysis
into Bro.
Vern
I’ve actually seen quite a bit of her work that she emailed to me last year. It was a phenomenal base for what I’ve tried to expand upon. I’d be eager to see if there had been further developments with it aside from what I’ve seen in the development branch.
Katrina could give a more accurate description of the stats analyzer than perhaps I could.
Thanks James 
Readon, my code can give you the RTT based on the 3-way handshake, as well as some additional RTT estimates as a connection continues, without any scripting work on your part. Email me and I'll be happy to point you towards the development branch and explain the events to you.
Katrina
I want to match tcp handshake pairs and record the intervals between
each SYN and SYN-ACK pairs with their arrival time. At the same time,
roughly packet loss rate (vs different timescales) should be calculated
by tcp retransmission rate. It is a statistical analysis on network
traffic that would be processed by .bro files i think. some of them
are similar with functions already existed. Would you please give me
some notes on which files i should start with?Yes, you're right. This sort of analysis can be entirely done at the
scripting layer, i.e., it only involves *.bro scripts.Bro reassembles the full TCP byte stream. When you deal with connection
data, duplicates/retransmission are already removed. If you want to
compute round-trip times at the packet level, you can write a handler
for the events new_connection (which is generated for every new
SYN packet) and connection_established (which is generated for a
successful TCP handshake after a SYN-ACK).
I have wrtie a script called local.bro which was applied to connect event connection_established & connection_first_ACK
but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website.
the script is followed as attachment.
local.bro (666 Bytes)
I want to match tcp handshake pairs and record the intervals between
each SYN and SYN-ACK pairs with their arrival time. At the same time,
roughly packet loss rate (vs different timescales) should be calculated
by tcp retransmission rate. It is a statistical analysis on network
traffic that would be processed by .bro files i think. some of them
are similar with functions already existed. Would you please give me
some notes on which files i should start with?Yes, you're right. This sort of analysis can be entirely done at the
scripting layer, i.e., it only involves *.bro scripts.Bro reassembles the full TCP byte stream. When you deal with connection
data, duplicates/retransmission are already removed. If you want to
compute round-trip times at the packet level, you can write a handler
for the events new_connection (which is generated for every new
SYN packet) and connection_established (which is generated for a
successful TCP handshake after a SYN-ACK).
I have wrtie a script called local.bro which was applied to connect event connection_established & connection_first_ACK
but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website.
There are some points need to be noticed.
1.Bro 2.0 beta was installed correctly. I have tested some example.
2.bro_script_loaded event (in the script) works as well as log.
The script file is attached. Thank you.
local.bro (666 Bytes)
I have wrtie a script called local.bro which was applied to connect event connection_established & connection_first_ACK
but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website.
If you check reporter.log, there's some hints indicating that your c$loc optional field value is missing at the times when you try to write to the log (meaning the event handlers are actually invoked, but don't do anything because of the error). To fix it you should first check that c$loc is initialized in the handlers and also fill in any of its fields that you can. Have a look at the alterations I made in the attached file to see if it makes sense for what you were trying to do.
+Jon
local.bro (899 Bytes)
I have wrtie a script called local.bro which was applied to connect event connection_established & connection_first_ACK
but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website.If you check reporter.log, there's some hints indicating that your c$loc optional field value is missing at the times when you try to write to the log (meaning the event handlers are actually invoked, but don't do anything because of the error). To fix it you should first check that c$loc is initialized in the handlers and also fill in any of its fields that you can. Have a look at the alterations I made in the attached file to see if it makes sense for what you were trying to do.
That is the point.
It works now, but i didn't find reporter.log in the directory. Is there some thing important i have missing in command?
Correct me if I am wrong Matthias.
Bro can do most of what you are looking for out of the box. Sampling the
round-trip time of the three-way handshake is doable at scripting level.
You can get bro to output the retransmission data through the tcp_rexmit
event. It does not give detailed information about how the data was
retransmitted but will tell you how many bytes were retransmitted and how
much data was outstanding when the retransmission occurred.That should be sufficient for what you are looking for.
I checked the code of tcp_rexmit event in TCP.cc.
It seems that the event was processed with max_top_seq.
There are two issues should be considered.
1. how can i distinguish tcp_retrasmission caused by packet loss & out of order?
2. if the retransmission occurs when handshaking, would it be correctly triggered?
I checked the code of tcp_rexmit event in TCP.cc.
It seems that the event was processed with max_top_seq.
There are two issues should be considered.
- how can i distinguish tcp_retrasmission caused by packet loss & out of order?
It would be entirely possible to make certain assumptions during post-processing of your logs. It is possible to determine which retransmissions are legitimately out-of-order and not actual retransmissions, if you have some sense of the round trip time of the connection or other methods. Perhaps Katrina or someone else could chime in and explain this in more detail. I am curious to know as well.
- if the retransmission occurs when handshaking, would it be correctly triggered?
The sequence number to be acknowledged remains the same during handshaking. The event should be correctly triggered.
int seq_delta = top_seq - max_top_seq;
if ( seq_delta <= 0 )
Consider that top_seq is always set to the sequence number in the TCP header plus the length of the packet. Given that the syn/syn-ack packets never carry data, you will always trigger the retransmit event upon the second transmission of a syn/syn-ack segment with a sequence number less than or equal to the maximum sequence number observed. It should always trigger.
– James Swaro
It is possible to determine which retransmissions are legitimately out-of-order and not actual retransmissions, if you have some sense of the round trip time of the connection or other methods. Perhaps Katrina or someone else could chime in and explain this in more detail. I am curious to know as well.
That's the general idea. You can check retransmissions vs. out-of-order (vs. replay packets) by examining the obvious things such as IP IDs and sequence numbers, and then checking the inter-arrival time between the packet in question and the previous packet. If that IAT is less than the minimum RTT you've observed on the connection, then you're likely dealing with either a replay packet or an out-of-order packet (and that distinction can be resolved with sequence numbers and IP IDs).
It is a bit of a pain (one has to keep track of RTTs, what sequence numbers we've seen, etc.), but that's how my analyzer handles it.
Katrina