surgical file extraction

Zeek
1

Good day all,

I am new to creating custom bro scripts, so please bear with me.

I am looking to perform file extraction only on HTTP conversations to either a specific URL or destination IP address. I have some command and control traffic POST traffic that I want to capture with bro, as opposed to what I am using now. I am currently having to run TCPDumps on my sensors for traffic destined to a specific IP, then manually cut out the POST requests, redo the pcap, then run it through bro… tedious.

Here is what I have, but cant get it to work (mainly due to a lack of bro script knowledge):

global mime_to_ext: table[string] of string = {

[“text/plain”] = “txt”,

[“text/html”] = “html”,

};

Event file_sniff(f: fa_file, meta: fa_metadata)

{

if ( f$source != “HTTP” )

return;

if ( meta$rx_hosts != “123.123.123.123” )

return;

if ( ! meta?$mime_type )

return;

if ( meta$mime_type !in mime_to_ext )

return;

local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[meta$mime_type]);

print fmt(“Extracting file %s”, fname);

Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);

}

I get the warning that the meta$rx_hosts isn’t valid. I have tried a “redef record” statement, but it made it much worse. Am I even in the ballpark here? I wasnt sure if I needed to use some other means to identify the URL, then start file extraction.

Appreciate any help.

2

You’re close! The field rx_hosts doesn’t exist in the meta record. If you want to get the destination IP, then you can do something similar to the first example on this page: https://www.bro.org/sphinx-git/frameworks/file-analysis.html

In your event, you’d do something like this …

for ( cid in f$conns )
{
if ( f$conns[cid]$id$resp_h == 123.123.123.123 )
{
print “found IP 123.123.123.123”;
}
}

This should get you started: http://try.bro.org/#/trybro/saved/27817

Josh

3

Josh,

That worked great. Thanks a ton for the help. Any idea why when I load my new file extraction script, that I verified works great on standalone packet captures, I get the below error:

error in /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro, line 15: syntax error, at or near “module”

I validated that if I comment out my @load line for the new file extraction script this error goes away in a “broctl check” check.

4

You may be missing a semi-colon somewhere near the end of your script. Bro’s error reporting will accidentally report the problem on the first line of the next script in this case on accident.

  .Seth

5

Seth,

Thanks, but I have pulled up quite a few other examples and dont see where I am missing a semi-colon. I am wondering if its because I am running 32+ worker nodes and the script isnt built correctly to utilize the clustering (shot in the dark). Here is my script as I have it now. I was thinking I may want to move it to use a URL instead of the IP, especially if I start finding CnC comms to load balancer sites:

global mime_to_ext: table[string] of string = {
[“text/plain”] = “txt”,
[“text/html”] = “html”,
};
event file_sniff(f: fa_file, meta: fa_metadata)
{
for ( cid in f$conns )
{
if ( f$conns[cid]$id$resp_h != 123.123.123.123 )
return;

if ( f$source != “HTTP” )
return;

if ( ! meta?$mime_type )
return;

if ( meta$mime_type !in mime_to_ext )
return;

local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[meta$mime_type]);
print fmt(“Extracting file %s”, fname);
Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);
}

6

Brandon,

It may be the extraneous “,” at the end of your mime_to_ext table.

Kevin

7

You are missing a right curly brace at the end. If you go through and clean up your indentation it should quickly become clear what’s missing. :slight_smile:

  .Seth

8

Yeah, I think it was that missing closing bracket. I was stuck with using “vi” on one box, but once I opened it with “vim” it was clear. Taking the comma out in the table was just good practice, so thanks for that guys. I ran a check, and deployed it. Now I will have to monitor my test box to see what kind of performance hit this makes.

Really appreciate the help. I am trying to show the benefits of using Bro in my work environment, and this is a huge win if I can get it working well.