# using Bro as traffic analyzer. **URL:** \ **Category:** Zeek\ **Created:** [December 4, 2011, 7:09am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144 "2011-12-04T07:09:54Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 4, 2011, 7:09am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/1 "2011-12-04T07:09:54Z")
I was searching for a long time to find a framework can support fast & custom network traffic analysis. some specific features of traffic data from monitor, such as interval of SYN and SYN-ACK, should be extracted and grouped by host. i find Bro is so widely used, which seems can fulfill the requirement. Can i disable other functions embedded in Bro, and add a plugin myself? What is the point to archieve this, modify the core .cpp source file or add a .bro file? ---
**Author:** ![Matthias\_Vallentin1](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/matthias_vallentin1/32/596_2.png) [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)\ **Post date:** [December 4, 2011, 7:44am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/2 "2011-12-04T07:44:45Z")
> i find Bro is so widely used, which seems can fulfill the requirement. Good to hear it works for you and welcome aboard! > Can i disable other functions embedded in Bro, and add a plugin myself? With the upcoming release of 2.0, Bro enables policy-neutral protocol analysis by default, meaning it gives you a neutral picture of what's going on in your network. For additional analyses and detectors, you need to load the corresponding scripts in the policy directory; local.bro is a good starting point. That said, you only pay for basic protocol decoding by default. > What is the point to archieve this, modify the core .cpp source file or add > a .bro file? This depends on the functionality you would like to add. Would you mind elaborating a bit so that we can give you more helpful advice? Changing the format of the log output or modifying analyzer behavior generally works at the scripting layer. Bro features a Turing-complete scripting language. You can write your own new functions and events. If you would like to haul C/C++ functionality up to the scripting layer, you might want to consider writing your own built-in function (BiF). See src/bro.bif for examples. If you would like to add a new protocol analyzer, then BinPAC is the right tool for you.     Matthias ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 5, 2011, 2:47am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/3 "2011-12-05T02:47:48Z")
> > i find Bro is so widely used, which seems can fulfill the requirement. > > Good to hear it works for you and welcome aboard! It is a greate platform, thank you for your works. > > Can i disable other functions embedded in Bro, and add a plugin myself? > > With the upcoming release of 2.0, Bro enables policy-neutral protocol > analysis by default, meaning it gives you a neutral picture of what's > going on in your network. For additional analyses and detectors, you > need to load the corresponding scripts in the policy directory; > local.bro is a good starting point. That said, you only pay for basic > protocol decoding by default. > > > What is the point to archieve this, modify the core .cpp source file or add > > a .bro file? > > This depends on the functionality you would like to add. Would you > mind elaborating a bit so that we can give you more helpful advice? > Changing the format of the log output or modifying analyzer behavior > generally works at the scripting layer. Bro features a Turing-complete > scripting language. You can write your own new functions and events. > If you would like to haul C/C++ functionality up to the scripting > layer, you might want to consider writing your own built-in function > (BiF). See src/bro.bif for examples. If you would like to add a new > protocol analyzer, then BinPAC is the right tool for you. I want to match tcp handshake pairs and record the intervals between each SYN and SYN-ACK pairs with their arrival time. At the same time, roughly packet loss rate (vs different timescales) should be calculated by tcp retransmission rate. It is a statistical analysis on network traffic that would be processed by .bro files i think. some of them are similar with functions already existed. Would you please give me some notes on which files i should start with? btw: I read the document and find that all C/C++ code is designed for decoding packets. bro files take charge in statistal or general processing. Is it right? Any general pictures were provided in bro? ---
**Author:** ![Matthias\_Vallentin1](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/matthias_vallentin1/32/596_2.png) [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)\ **Post date:** [December 5, 2011, 3:19am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/4 "2011-12-05T03:19:58Z")
> I want to match tcp handshake pairs and record the intervals between > each SYN and SYN-ACK pairs with their arrival time. At the same time, > roughly packet loss rate (vs different timescales) should be calculated > by tcp retransmission rate. It is a statistical analysis on network > traffic that would be processed by .bro files i think. some of them > are similar with functions already existed. Would you please give me > some notes on which files i should start with? Yes, you're right. This sort of analysis can be entirely done at the scripting layer, i.e., it only involves \*.bro scripts. Bro reassembles the full TCP byte stream. When you deal with connection data, duplicates/retransmission are already removed. If you want to compute round-trip times at the packet level, you can write a handler for the events new\_connection (which is generated for every new SYN packet) and connection\_established (which is generated for a successful TCP handshake after a SYN-ACK). For retransmissions, have a look at the events: rexmit\_inconsistency, content\_gap, and gap\_report. Unfortunately I cannot provide more detailed information other than pointing you to our ongoing documentation effort in the git repository:     git clone git://git.bro-ids.org/bro.git     git checkout topic/script-reference     less src/event.bif Maybe others can chime in and give you further guidance. (Also, to measure system/NIC capture loss, there is policy/misc/capture-loss.bro.) > btw: I read the document and find that all C/C++ code is designed > for decoding packets. bro files take charge in statistal or general > processing. Is it right? Any general pictures were provided in bro? That's correct. Packet "decoding" is done at the Bro core. Bro reassembles the TCP byte stream and presents it as a connection to the user. You may find our workshop materials helpful to better understand the architecture of Bro: [http://www.bro-ids.org/bro-workshop-2011](http://www.bro-ids.org/bro-workshop-2011)     Matthias ---
**Author:** ![James\_Swaro](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@James\_Swaro](https://community.zeek.org/u/James_Swaro)\ **Post date:** [December 5, 2011, 6:54am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/5 "2011-12-05T06:54:06Z")
> At the same time, roughly packet loss rate (vs different timescales) should be calculated > by tcp retransmission rate. Are you interested in the packet loss rate for the three-way handshake only, or for all retransmissions in each connection? What level of detail are you looking for? (IE. Just that the segment was retransmitted or perhaps more specific data about how it was retransmitted such as in response to a retransmission timeout or other recovery method?) ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 5, 2011, 11:03am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/6 "2011-12-05T11:03:13Z")
> > At the same time, roughly packet loss rate (vs different timescales) > > should be calculated > > by tcp retransmission rate. > > Are you interested in the packet loss rate for the three-way handshake > only, or for all retransmissions in each connection? all retransmissions should be collected. > What level of detail are you looking for? (IE. Just that the segment was > retransmitted or perhaps more specific data about how it was retransmitted > such as in response to a retransmission timeout or other recovery method?) it would be better that retrans mode were detected. ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 5, 2011, 11:33am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/7 "2011-12-05T11:33:17Z")
> > I want to match tcp handshake pairs and record the intervals between > > each SYN and SYN-ACK pairs with their arrival time. At the same time, > > roughly packet loss rate (vs different timescales) should be calculated > > by tcp retransmission rate. It is a statistical analysis on network > > traffic that would be processed by .bro files i think. some of them > > are similar with functions already existed. Would you please give me > > some notes on which files i should start with? > > Yes, you're right. This sort of analysis can be entirely done at the > scripting layer, i.e., it only involves \*.bro scripts. > > Bro reassembles the full TCP byte stream. When you deal with connection > data, duplicates/retransmission are already removed. If you want to > compute round-trip times at the packet level, you can write a handler > for the events new\_connection (which is generated for every new > SYN packet) and connection\_established (which is generated for a > successful TCP handshake after a SYN-ACK). > > For retransmissions, have a look at the events: rexmit\_inconsistency, > content\_gap, and gap\_report. Unfortunately I cannot provide more > detailed information other than pointing you to our ongoing > documentation effort in the git repository: > >    git clone git://git.bro-ids.org/bro.git >    git checkout topic/script-reference >    less src/event.bif > > Maybe others can chime in and give you further guidance. > > (Also, to measure system/NIC capture loss, there is > policy/misc/capture-loss.bro.) > > > btw: I read the document and find that all C/C++ code is designed > > for decoding packets. bro files take charge in statistal or general > > processing. Is it right? Any general pictures were provided in bro? > > That's correct. Packet "decoding" is done at the Bro core. Bro > reassembles the TCP byte stream and presents it as a connection to the > user. You may find our workshop materials helpful to better understand > the architecture of Bro: [http://www.bro-ids.org/bro-workshop-2011](http://www.bro-ids.org/bro-workshop-2011) > >    Matthias Thank you very much. It is very useful! ---
**Author:** ![James\_Swaro](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@James\_Swaro](https://community.zeek.org/u/James_Swaro)\ **Post date:** [December 5, 2011, 7:31pm UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/8 "2011-12-05T19:31:00Z")
Correct me if I am wrong Matthias. Bro can do most of what you are looking for out of the box. Sampling the round-trip time of the three-way handshake is doable at scripting level. You can get bro to output the retransmission data through the tcp\_rexmit event. It does not give detailed information about how the data was retransmitted but will tell you how many bytes were retransmitted and how much data was outstanding when the retransmission occurred. That should be sufficient for what you are looking for. If you need more detailed information, I am currently working on an analyzer for Bro that attempts to give more detailed information about the retransmission behavior of a TCP connection as part of on-going research. However, It is not in a state that is ready for release. ---
**Author:** ![Vern](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/vern/32/630_2.png) [@Vern](https://community.zeek.org/u/Vern)\ **Post date:** [December 6, 2011, 3:51am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/9 "2011-12-06T03:51:15Z")
> If you need more detailed information, I am currently working on an > analyzer for Bro that attempts to give more detailed information about the > retransmission behavior of a TCP connection as part of on-going research. You should for sure contact Katrina LaCurts \, who did an internship with us working on integrating this sort of analysis into Bro.     Vern ---
**Author:** ![James\_Swaro](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@James\_Swaro](https://community.zeek.org/u/James_Swaro)\ **Post date:** [December 6, 2011, 6:37am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/10 "2011-12-06T06:37:58Z")
I’ve actually seen quite a bit of her work that she emailed to me last year. It was a phenomenal base for what I’ve tried to expand upon. I’d be eager to see if there had been further developments with it aside from what I’ve seen in the development branch. Katrina could give a more accurate description of the stats analyzer than perhaps I could. ---
**Author:** ![Katrina\_LaCurts](https://avatars.discourse-cdn.com/v4/letter/k/f475e1/32.png) [@Katrina\_LaCurts](https://community.zeek.org/u/Katrina_LaCurts)\ **Post date:** [December 6, 2011, 5:53pm UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/11 "2011-12-06T17:53:55Z")
Thanks James 🙂 Readon, my code can give you the RTT based on the 3-way handshake, as well as some additional RTT estimates as a connection continues, without any scripting work on your part. Email me and I'll be happy to point you towards the development branch and explain the events to you. Katrina ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 9, 2011, 10:36am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/12 "2011-12-09T10:36:21Z")
> > I want to match tcp handshake pairs and record the intervals between > > each SYN and SYN-ACK pairs with their arrival time. At the same time, > > roughly packet loss rate (vs different timescales) should be calculated > > by tcp retransmission rate. It is a statistical analysis on network > > traffic that would be processed by .bro files i think. some of them > > are similar with functions already existed. Would you please give me > > some notes on which files i should start with? > > Yes, you're right. This sort of analysis can be entirely done at the > scripting layer, i.e., it only involves \*.bro scripts. > > Bro reassembles the full TCP byte stream. When you deal with connection > data, duplicates/retransmission are already removed. If you want to > compute round-trip times at the packet level, you can write a handler > for the events new\_connection (which is generated for every new > SYN packet) and connection\_established (which is generated for a > successful TCP handshake after a SYN-ACK). I have wrtie a script called local.bro which was applied to connect event connection\_established & connection\_first\_ACK but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website. the script is followed as attachment. [local.bro](https://community.zeek.org/uploads/short-url/cFgwB2iNDvrPUeCbgU7cGQjGbp4.bro) (666 Bytes) ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 9, 2011, 10:49am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/13 "2011-12-09T10:49:01Z")
> > I want to match tcp handshake pairs and record the intervals between > > each SYN and SYN-ACK pairs with their arrival time. At the same time, > > roughly packet loss rate (vs different timescales) should be calculated > > by tcp retransmission rate. It is a statistical analysis on network > > traffic that would be processed by .bro files i think. some of them > > are similar with functions already existed. Would you please give me > > some notes on which files i should start with? > > Yes, you're right. This sort of analysis can be entirely done at the > scripting layer, i.e., it only involves \*.bro scripts. > > Bro reassembles the full TCP byte stream. When you deal with connection > data, duplicates/retransmission are already removed. If you want to > compute round-trip times at the packet level, you can write a handler > for the events new\_connection (which is generated for every new > SYN packet) and connection\_established (which is generated for a > successful TCP handshake after a SYN-ACK). I have wrtie a script called local.bro which was applied to connect event connection\_established & connection\_first\_ACK but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website. There are some points need to be noticed. 1.Bro 2.0 beta was installed correctly. I have tested some example. 2.bro\_script\_loaded event (in the script) works as well as log. The script file is attached. Thank you. [local.bro](https://community.zeek.org/uploads/short-url/cFgwB2iNDvrPUeCbgU7cGQjGbp4.bro) (666 Bytes) ---
**Author:** ![Siwek\_Jon](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)\ **Post date:** [December 9, 2011, 4:16pm UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/14 "2011-12-09T16:16:23Z")
> I have wrtie a script called local.bro which was applied to connect event connection\_established & connection\_first\_ACK > but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website. If you check reporter.log, there's some hints indicating that your c$loc optional field value is missing at the times when you try to write to the log (meaning the event handlers are actually invoked, but don't do anything because of the error). To fix it you should first check that c$loc is initialized in the handlers and also fill in any of its fields that you can. Have a look at the alterations I made in the attached file to see if it makes sense for what you were trying to do. +Jon [local.bro](https://community.zeek.org/uploads/short-url/pvqqMOccHR8iyk6NVIwfsYgaVak.bro) (899 Bytes) ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 10, 2011, 2:10am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/15 "2011-12-10T02:10:56Z")
> > I have wrtie a script called local.bro which was applied to connect event connection\_established & connection\_first\_ACK > > but it seems that the event have not triggered. I tested the script with network trace "http.pcap" provided in Bro website. > > If you check reporter.log, there's some hints indicating that your c$loc optional field value is missing at the times when you try to write to the log (meaning the event handlers are actually invoked, but don't do anything because of the error). To fix it you should first check that c$loc is initialized in the handlers and also fill in any of its fields that you can. Have a look at the alterations I made in the attached file to see if it makes sense for what you were trying to do. That is the point. It works now, but i didn't find reporter.log in the directory. Is there some thing important i have missing in command? ---
**Author:** ![Readon\_Shaw](https://avatars.discourse-cdn.com/v4/letter/r/59ef9b/32.png) [@Readon\_Shaw](https://community.zeek.org/u/Readon_Shaw)\ **Post date:** [December 10, 2011, 7:29am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/16 "2011-12-10T07:29:03Z")
> Correct me if I am wrong Matthias. > > Bro can do most of what you are looking for out of the box. Sampling the > round-trip time of the three-way handshake is doable at scripting level. > You can get bro to output the retransmission data through the tcp\_rexmit > event. It does not give detailed information about how the data was > retransmitted but will tell you how many bytes were retransmitted and how > much data was outstanding when the retransmission occurred. > > That should be sufficient for what you are looking for. I checked the code of tcp\_rexmit event in TCP.cc. It seems that the event was processed with max\_top\_seq. There are two issues should be considered. 1. how can i distinguish tcp\_retrasmission caused by packet loss & out of order? 2. if the retransmission occurs when handshaking, would it be correctly triggered? ---
**Author:** ![James\_Swaro](https://avatars.discourse-cdn.com/v4/letter/j/13edae/32.png) [@James\_Swaro](https://community.zeek.org/u/James_Swaro)\ **Post date:** [December 10, 2011, 8:28am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/17 "2011-12-10T08:28:23Z")
> I checked the code of tcp\_rexmit event in TCP.cc. > It seems that the event was processed with max\_top\_seq. > There are two issues should be considered. > > 1. how can i distinguish tcp\_retrasmission caused by packet loss & out of order? It would be entirely possible to make certain assumptions during post-processing of your logs. It is possible to determine which retransmissions are legitimately out-of-order and not actual retransmissions, if you have some sense of the round trip time of the connection or other methods. Perhaps Katrina or someone else could chime in and explain this in more detail. I am curious to know as well. > 1. if the retransmission occurs when handshaking, would it be correctly triggered? The sequence number to be acknowledged remains the same during handshaking. The event should be correctly triggered. > int seq\_delta = top\_seq - max\_top\_seq; > if ( seq\_delta \<= 0 ) Consider that top\_seq is always set to the sequence number in the TCP header plus the length of the packet. Given that the syn/syn-ack packets never carry data, you will always trigger the retransmit event upon the second transmission of a syn/syn-ack segment with a sequence number less than or equal to the maximum sequence number observed. It should always trigger. – James Swaro ---
**Author:** ![Katrina\_LaCurts](https://avatars.discourse-cdn.com/v4/letter/k/f475e1/32.png) [@Katrina\_LaCurts](https://community.zeek.org/u/Katrina_LaCurts)\ **Post date:** [December 11, 2011, 3:26am UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/18 "2011-12-11T03:26:09Z")
> It is possible to determine which retransmissions are legitimately out-of-order and not actual retransmissions, if you have some sense of the round trip time of the connection or other methods. Perhaps Katrina or someone else could chime in and explain this in more detail. I am curious to know as well. That's the general idea. You can check retransmissions vs. out-of-order (vs. replay packets) by examining the obvious things such as IP IDs and sequence numbers, and then checking the inter-arrival time between the packet in question and the previous packet. If that IAT is less than the minimum RTT you've observed on the connection, then you're likely dealing with either a replay packet or an out-of-order packet (and that distinction can be resolved with sequence numbers and IP IDs). It is a bit of a pain (one has to keep track of RTTs, what sequence numbers we've seen, etc.), but that's how my analyzer handles it. Katrina ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:40pm UTC](https://community.zeek.org/t/using-bro-as-traffic-analyzer/2144/19 "2022-05-06T15:40:02Z")