Hey Bro List,
So I seem to be running into a problem with file extraction (or perhaps just file analysis in general). I have a basic extraction script running pulling out EXEs that are seen coming across HTTP and for some reason, there are consistently a large number of file gaps in the file it sees. I have a custom log outputting the fuid for any file_gap event (https://www.bro.org/sphinx/scripts/base/bif/event.bif.bro.html#id-file_gap), and I seem to get a wildly varying number of file gap events for a given file.
In my example, I am curling an exe to a server, where that traffic is spanned to my Bro sensor (the exe in question is 1 MB in size). If I curl repeatedly, Bro sees all the files, but the number of file gap events varies wildly (anywhere from 2 or 3 to over 100). The part that gets me, if I tcpdump alongside Bro, and pull the files out of pcap, they're all intact and hash correctly, so I know I'm getting all the packets on wire. Bro and PF_RING report 0 packet loss.
Does anyone have anything that could shed light on what's going on here?
Seems either Bro behaves differently in offline vs. live usage (not
what I'd expect to see in this case, though I can't rule it out for
sure) or it's not actually seeing the same packet input in the live
deployment as it was in offline usage. Maybe to pursue whether the
later is true you could have Bro write out the packets that it
actually saw with the `-w <pcap file>` command line flag and examine
the resulting pcap to see if it looks like you expect.
So I just tested running bro in command line mode (i.e. not using broctl), fed it my usual policy files and dumped to pcap.
The extracted_files folder showed the same story, lots of file gaps, all different hashes for the same file.
When I loaded up the pcap in wireshark and extracted all the files, all files hash correctly.
So this tells me, at some point Bro is getting all the data. Something is just messing up for some reason when it comes to the file analysis and/or file extraction modules.
Would be most helpful to get a pcap, scripts, and command that can
reproduce what you see. Or else steps one could follow to try to
reproduce a pcap that may exhibit the same problem. Otherwise, only
guess I have is to check for anything unusual at the TCP-layer -- the
file analysis/extraction is dependent on the TCP reassembly process,
so if the sequence of events at the TCP level lead Bro to believe it
missed part of the TCP stream, that also manifests as a gap event in
any associated file analysis. You could go as far as looking at the
contents/ordering of things in a tcp_packet event handler as a sanity
check, though there may also be residuals in weird.log that you could
simply check (I don't recall particular names to look for off the top
of my head).
Bro should write in weird.log something like : inflate_failed. This error
should represent (tell me if I am wrong) that bro failed to decompress
(inflate/deflate of http)
the file correctly. By cross-checking the connection ids you should be able
to verify if that was the problem.
It happened to me in different settings (with a manipulated pcap) that
file reconstruction did not work properly. I was not able to fix it though.
So here's what I did for reproducibility sake.
I installed CentOS 7 (1804) minimal install, and manually compiled and installed bro-2.5.3, all default settings.
I then made the following policy changes listed below. Interestingly, I don't get as many corrupt files as I was previously seeing (only 25-40% instead of 80-90%), but the number of file gaps are still pretty large (well over 100). Makes me wonder if this is some sort of bug dependent on performance? Musing out loud with that.
To /bro/share/bro/site/local.bro I added at the end (does not include horizontal rules made here)
--- local.bro -----------------------------------------------