# File Extraction Gaps **URL:** \ **Category:** Zeek\ **Created:** [May 29, 2018, 2:55pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326 "2018-05-29T14:55:38Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Weasel\_Gary\_W\_CIV\_DI](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Weasel\_Gary\_W\_CIV\_DI](https://community.zeek.org/u/Weasel_Gary_W_CIV_DI)\ **Post date:** [May 29, 2018, 2:55pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/1 "2018-05-29T14:55:38Z")
Hey Bro List, So I seem to be running into a problem with file extraction (or perhaps just file analysis in general). I have a basic extraction script running pulling out EXEs that are seen coming across HTTP and for some reason, there are consistently a large number of file gaps in the file it sees. I have a custom log outputting the fuid for any file\_gap event ([https://www.bro.org/sphinx/scripts/base/bif/event.bif.bro.html#id-file\_gap](https://www.bro.org/sphinx/scripts/base/bif/event.bif.bro.html#id-file_gap)), and I seem to get a wildly varying number of file gap events for a given file. In my example, I am curling an exe to a server, where that traffic is spanned to my Bro sensor (the exe in question is 1 MB in size). If I curl repeatedly, Bro sees all the files, but the number of file gap events varies wildly (anywhere from 2 or 3 to over 100). The part that gets me, if I tcpdump alongside Bro, and pull the files out of pcap, they're all intact and hash correctly, so I know I'm getting all the packets on wire. Bro and PF\_RING report 0 packet loss. Does anyone have anything that could shed light on what's going on here? Thanks, - Gary ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [May 29, 2018, 4:51pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/2 "2018-05-29T16:51:54Z")
Seems either Bro behaves differently in offline vs. live usage (not what I'd expect to see in this case, though I can't rule it out for sure) or it's not actually seeing the same packet input in the live deployment as it was in offline usage. Maybe to pursue whether the later is true you could have Bro write out the packets that it actually saw with the `-w \` command line flag and examine the resulting pcap to see if it looks like you expect. - Jon ---
**Author:** ![Weasel\_Gary\_W\_CIV\_DI](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Weasel\_Gary\_W\_CIV\_DI](https://community.zeek.org/u/Weasel_Gary_W_CIV_DI)\ **Post date:** [May 29, 2018, 5:22pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/3 "2018-05-29T17:22:01Z")
So I just tested running bro in command line mode (i.e. not using broctl), fed it my usual policy files and dumped to pcap. The extracted\_files folder showed the same story, lots of file gaps, all different hashes for the same file. When I loaded up the pcap in wireshark and extracted all the files, all files hash correctly. So this tells me, at some point Bro is getting all the data. Something is just messing up for some reason when it comes to the file analysis and/or file extraction modules. ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [May 29, 2018, 5:57pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/4 "2018-05-29T17:57:25Z")
Would be most helpful to get a pcap, scripts, and command that can reproduce what you see. Or else steps one could follow to try to reproduce a pcap that may exhibit the same problem. Otherwise, only guess I have is to check for anything unusual at the TCP-layer -- the file analysis/extraction is dependent on the TCP reassembly process, so if the sequence of events at the TCP level lead Bro to believe it missed part of the TCP stream, that also manifests as a gap event in any associated file analysis. You could go as far as looking at the contents/ordering of things in a tcp\_packet event handler as a sanity check, though there may also be residuals in weird.log that you could simply check (I don't recall particular names to look for off the top of my head). - Jon ---
**Author:** ![BortolameottiR](https://avatars.discourse-cdn.com/v4/letter/b/51bf81/32.png) [@BortolameottiR](https://community.zeek.org/u/BortolameottiR)\ **Post date:** [May 29, 2018, 6:57pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/5 "2018-05-29T18:57:05Z")
Bro should write in weird.log something like : inflate\_failed. This error should represent (tell me if I am wrong) that bro failed to decompress (inflate/deflate of http) the file correctly. By cross-checking the connection ids you should be able to verify if that was the problem. It happened to me in different settings (with a manipulated pcap) that file reconstruction did not work properly. I was not able to fix it though. R. ---
**Author:** ![Weasel\_Gary\_W\_CIV\_DI](https://avatars.discourse-cdn.com/v4/letter/w/c77e96/32.png) [@Weasel\_Gary\_W\_CIV\_DI](https://community.zeek.org/u/Weasel_Gary_W_CIV_DI)\ **Post date:** [May 30, 2018, 2:43pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/6 "2018-05-30T14:43:33Z")
So here's what I did for reproducibility sake. I installed CentOS 7 (1804) minimal install, and manually compiled and installed bro-2.5.3, all default settings. I then made the following policy changes listed below. Interestingly, I don't get as many corrupt files as I was previously seeing (only 25-40% instead of 80-90%), but the number of file gaps are still pretty large (well over 100). Makes me wonder if this is some sort of bug dependent on performance? Musing out loud with that. To /bro/share/bro/site/local.bro I added at the end (does not include horizontal rules made here) --- local.bro ----------------------------------------------- @load policy/custom/extract-files.bro @load policy/custom/file-gap.bro ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:45pm UTC](https://community.zeek.org/t/file-extraction-gaps/5326/7 "2022-05-06T15:45:49Z")
Closed on May 6, 2022, 3:45pm UTC