# Fragmentation and TCP overlapping Issues **URL:** \ **Category:** Zeek\ **Created:** [November 24, 2010, 3:25pm UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720 "2010-11-24T15:25:36Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Veronica\_Estrada](https://avatars.discourse-cdn.com/v4/letter/v/ba9def/32.png) [@Veronica\_Estrada](https://community.zeek.org/u/Veronica_Estrada)\ **Post date:** [November 24, 2010, 3:25pm UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/1 "2010-11-24T15:25:36Z")
Hello everyone, First of all, tomorrow is thanksgiving and I would like to thank all of you for all the feedback I’ve always received to my posts. I continue with my research on anomalies, now focus on evasion techniques, and I need to ask you some help to understand how BRO deals with fragmentation and TCP overlapping issues. For reference, I am using Bro 1.5.1 in offline analysis. 1. Although I am loading “frag”, I am not receiving any event related with fragmentation. What could be wrong? libpcap library? my BRO version? 1. What are the possible events triggered by weird analyzer related with tcp overlapping? (because I am not getting any of them although I think I should see them on my trace) 2. TCP overlapping problems may generate “partial\_ftp\_request”, “partial\_RPC\_request” or other partial events? and also confuse BRO on how the connection should be flagged? For example a connection with flag “S0”, no reply seen could be related with TCP overlapping problems? 3. How does BRO perform TCP reassembly? I mean, is the traffic on ALL ports reassembled? Is there any way to apply a default policy for doing TCP reassembly? Like Policy First or Last or Unix… 4. There is an “active mapping” function to improve TCP reassembly. Can we define the host profile database without this active function? 5. Can we configure the size of the reassembly buffer? I read in historical msg (from 2006) there wasn’t such config and BRO presented a vulnerability against an adversary trying to exhaust memory, is this a current possibility? 6. By doing offline analysis, I understood that BRO will analyze all the packets without loss even if the CPU is running at 100%. Still, I need information about dropping packets for other reasons. For example, if BRO encounters TCP overlapping, Does it drop all the packets? Choose some of them? Are these actions log somewhere? The same with fragmentations issues. Where can I check the portion of fragments that where reassembled? how many frames discarded, etc? 7. I am not seeing any difference in bro logs when I analyze 2 pcap files. One file contains some malformed packet at the end and wireshark says “the packet is bigger than 65535”, the other pcap file is the same file but truncated using editcap to avoid this “malformed packet” (if I check the hex using hd, the part truncated represents 850MB ). All the logs of BRO when input is one file or the other are identical. Is this the expected result? Veronica Estrada Nakao Lab. Network System Research Group University of Tokyo ---
**Author:** ![Veronica\_Estrada](https://avatars.discourse-cdn.com/v4/letter/v/ba9def/32.png) [@Veronica\_Estrada](https://community.zeek.org/u/Veronica_Estrada)\ **Post date:** [December 5, 2010, 10:06am UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/2 "2010-12-05T10:06:52Z")
Hello, Sorry for sending again these questions to the list but I haven’t received any answer to my old post in bro list. Besides, I want to add more details. We try to understand the behavior of BRO regarding fragmentation and other related issues. I am validating results with another IDS (Snort) and results are surprisingly different, but to understand the differences we need answers to the questions in e-mail below. For example, check question 7: in our traces, there is a file that contains corrupted headers and several packets. Snort output is different whether we use the original file or a truncated version: Original file (contains corrupted headers): DISCARD: 87.9% TCP: 11.751% OTHER: 1379 Total fragments: 36 TCP Overlaps: 1494 ALERTS: 1836 Processed packets: 3,771,244 (discard: 3311766) Same file but truncated since the point where wireshark triggers “corrupted header” and cannot read anymore DISCARD: 0 % TCP: 94% OTHER: 22819 Total fragments: 1072 (fragments reassembled 522) TCP Overlaps: 34580 ALERTS: 116877 Processed packets: 3,795,710 Original file 4Gb, Truncated file 3.9 Gb BRO output is identical for both files. Moreover, I cannot check how many discard packets or how BRO deals with the corrupted header packets. Regarding question 1 below, I am using libpcap 1.0.0. Thank you in advance, Veronica Estrada ---
**Author:** ![Vern](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/vern/32/630_2.png) [@Vern](https://community.zeek.org/u/Vern)\ **Post date:** [December 6, 2010, 5:38am UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/3 "2010-12-06T05:38:54Z")
> 1. Although I am loading "frag", I am not receiving any event related with > fragmentation. > > What could be wrong? libpcap library? my BRO version? As usual, it helps a great deal if you include a trace and the command line you're using. One possibility is that you're trying to analyze UDP fragments, since the filter included by frag.bro only analyzes TCP fragments (to avoid heavy load from NFS traffic; this is vestigial, and makes sense to remove). > 2. What are the possible events triggered by weird analyzer related with tc= > p > overlapping? The analyzer doesn't look for overlap per se (which is not a TCP spec violation, and occurs for benign reasons; see our IEEE S&P 2008 paper). It looks for inconsistent TCP byte streams. For those, it doesn't use the "weird" interface but generates a distinct event, rexmit\_inconsistency. > (because I am not getting any of them although I think I should > see them on my trace) Again, including a trace and the command line you're using to analyze it is far and away the best way to get help for problems like this. > 3. TCP overlapping problems may generate "partial\_ftp\_request", No. In this case, "partial" means that the connection was torn down in the middle of a request. It doesn't have anything to do with TCP overlap. > 4. How does BRO perform TCP reassembly? I mean, is the traffic on ALL ports > reassembled? Traffic to ports for which there's an analyzer that uses the byte stream. You can also contro lthis using tcp\_reassembler\_ports\_orig and tcp\_reassembler\_ports\_resp. > Is there any way to apply a default policy for doing TCP > reassembly? Like Policy First or Last or Unix No, this is not supported. > 5. There is an "active mapping" function to improve TCP reassembly. Can we > define the host profile database without this active function? No, that's the only way; and it is no longer supported. > 6. Can we configure the size of the reassembly buffer? I read in historical > msg (from 2006) there wasn't such config and BRO presented a vulnerability > against an adversary trying to exhaust memory, is this a current > possibility? See bro.init for the comments discussing tcp\_max\_above\_hole\_without\_any\_acks and tcp\_excessive\_data\_without\_further\_acks. > 7. By doing offline analysis, I understood that BRO will analyze all the > packets without loss even if the CPU is running at 100%. Still, I need > information about dropping packets for other reasons. For example, if BRO > encounters TCP overlapping, Does it drop all the packets? No. It always delivers the byte stream as soon as it is in sequence. Subsequent overlapping-but-inconsistent packets generate the event mentioned above. > Choose some of > them? Always the first seen. > Are these actions log somewhere? I'm not sure what there is to log. Overlapping that's consistent is not any sort of necessarily peculiar phenomenon. Inconsistent overlapping leads to events. > The same with fragmentations issues. > Where can I check the portion of fragments that where reassembled? No, this isn't instrumented. > how many > frames discarded, etc? I don't know what you mean by discarded here. > 8. I am not seeing any difference in bro logs when I analyze 2 pcap files. Again, you need to send along traces and command lines.     Vern ---
**Author:** ![Veronica\_Estrada](https://avatars.discourse-cdn.com/v4/letter/v/ba9def/32.png) [@Veronica\_Estrada](https://community.zeek.org/u/Veronica_Estrada)\ **Post date:** [December 6, 2010, 2:04pm UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/4 "2010-12-06T14:04:18Z")
Thank you very much! This information helps me a lot. Regarding fragmentation (question 1), I am running bro with this command: bro $files todai-nets -f “tcp or udp or icmp” dpd\_conn\_logs=T dpd detect-protocols dyn-disable detect-protocols-http proxy ssh irc-bot brolite print-globals capture-loss The variable $files contains 300 consecutive captured files (pcap) of 4 Gb each. In this experiment I am not getting any “fragment” event. On the contrary, fragmentation events appeared when bro was run using a subset of the aforementioned files. I don’t have a record about the Bro parameters set for this case. Basically the difference here is how the input data to Bro was previously processed. In this case, we aggregated these small pcap files in bigger ones (80Gb). Every 20 files we applied ipsumdump to generate a bigger pcap file and feed Bro system so now, Bro triggers this type of events: excessively\_small\_fragment, fragment\_inconsistency, fragment\_overlap, fragment\_size\_inconsistency, fragment\_with\_DF What would be the reason for not getting these fragment events in the first experiment? Regarding the other questions about overlapping, I am preparing an e-mail with detail of the traces with problems. I cannot publish this on the list. Thank you again, Veronica ---
**Author:** ![Vern](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/vern/32/630_2.png) [@Vern](https://community.zeek.org/u/Vern)\ **Post date:** [December 6, 2010, 5:51pm UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/5 "2010-12-06T17:51:44Z")
> Regarding fragmentation (question 1), I am running bro with this command: > > bro $files todai-nets -f "tcp or udp or icmp" dpd\_conn\_logs=T dpd > detect-protocols dyn-disable detect-protocols-http proxy ssh irc-bot brolite > print-globals capture-loss What happens when you use   bro ... -f "tcp or udp or icmp or (ip[6:2] & 0x3fff != 0)" ... ? > The variable $files contains 300 consecutive captured files (pcap) of 4 Gb > each. In this experiment I am not getting any "fragment" event. Again, we really need trace snippets to diagnose problems like these. You should extract a small subset of the trace that you believe should cause behavior different from what Bro does.     Vern ---
**Author:** ![Veronica\_Estrada](https://avatars.discourse-cdn.com/v4/letter/v/ba9def/32.png) [@Veronica\_Estrada](https://community.zeek.org/u/Veronica_Estrada)\ **Post date:** [December 7, 2010, 1:23pm UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/6 "2010-12-07T13:23:29Z")
Thank you very much for the help. Some feedback: 1. After including the filter -f "tcp or udp or icmp or (ip[6:2] & 0x3fff != 0)" the fragmentations events appeared. Now, I am re-processing the files but loading only weird and matching unfragmented datagrams only to accelerate the process. Command: bro -r $files -f (ip[6:2] & 0x3fff != 0)" weird All fragment events are handle by flow weird. When is invoked flow weird handler? 2. Related with a retransmission inconsistency. I found the remix inconsistency events in the notice file. 4. "Traffic to ports for which there's an analyzer that uses the byte stream. You can also control this using tcp\_reassembler\_ports\_orig and tcp\_reassembler\_ports\_resp." How can I redef these variables? I tried to redef this variables on my start policy but all I get are errors ((port and 21): error, arithmetic mixed with non-arithmetic). const tcp\_reassembler\_ports\_orig: set[port] = {} &redef; const tcp\_reassembler\_ports\_resp: set[port] = {} &redef; Could you please illustrate us with an example, for instance include port 21 & 80 in the tcp\_reassebler\_ports\_orig? ---
**Author:** ![Veronica\_Estrada](https://avatars.discourse-cdn.com/v4/letter/v/ba9def/32.png) [@Veronica\_Estrada](https://community.zeek.org/u/Veronica_Estrada)\ **Post date:** [December 8, 2010, 8:59am UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/7 "2010-12-08T08:59:46Z")
8. By the way, before sending you the trace we did more research and we suspect that the problem is generated by the machine used for the capture. So we are going to check with the vendor. Thanks for the help. ---
**Author:** ![Vern](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/vern/32/630_2.png) [@Vern](https://community.zeek.org/u/Vern)\ **Post date:** [December 9, 2010, 5:06am UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/8 "2010-12-09T05:06:43Z")
> All fragment events are handle by flow weird. When is invoked flow > weird handler? It's only used for packets that are so broken that Bro can't reliably associate them with a connection. > How can I redef these variables? I tried to redef this variables on > my start policy but all I get are errors ((port and 21): error, If you want TCP port 21 then you specify it as "21/tcp", not just "21".     Vern ---
**Author:** ![Veronica\_Estrada](https://avatars.discourse-cdn.com/v4/letter/v/ba9def/32.png) [@Veronica\_Estrada](https://community.zeek.org/u/Veronica_Estrada)\ **Post date:** [December 9, 2010, 6:03am UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/9 "2010-12-09T06:03:21Z")
And now it works! 🙂 ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:39pm UTC](https://community.zeek.org/t/fragmentation-and-tcp-overlapping-issues/1720/10 "2022-05-06T15:39:16Z")