Extract complete files

Zeek
1

Hello,

I am experimenting with the Files framework in bro 2.4 beta. I would like to extract HTTP files, *without* missing_bytes.
Can anyone please help me on this?

Thanks,
Albert

2

Hello,

I am experimenting with the Files framework in bro 2.4 beta. I would like to extract HTTP files, *without* missing_bytes.
Can anyone please help me on this?

Thanks,
Albert

3

Hi Albert,

it’s hard to help without any context, so just some hints: It took me some time to find the -C switch to ignore wrong checksums in bro. Without it the traffic did not reach the extraction layer. Also it’s always a good idea to compare bro with other tools. Make sure wireshark does show the complete http session.

Franky

4

Hi Frank,

Perhaps I didn’t explain myself properly. I meant extracting only complete files (or removing incomplete ones). There might be file gaps because of improper taping…
I attached the Files::ANALYZER_EXTRACT and Files::ANALYZER_SHA256 in the file_sniff event. The event_hash triggers only for complete files, but the file gets extracted anyway.

Albert

5

Albert,

You have a chicken and egg problem. Specifically, you're not going to be certain of how many bytes are missing at the time you have to determine whether or not you'll be extracting the file. Instead, you'll have to extract all files and then later remove the files that aren't the ones that you want. This is similar to how the issue of 'how do I name the file after the hash' is solved.

I have some examples of that here in the plugins directory: https://github.com/hosom/bro-file-extraction/

While it isn't precisely what you want... you'll be able to piece together the hashing examples into removing files from the filesystem that show as having missing bytes.

If you're seeing a large number of missing bytes in files consistently, there are likely other problems occurring.

Thanks,

Stephen

6

Hi Stephen,

Your rename-after-hash example is a life saviour!

Thanks a bunch,
Albert