File Extraction

Hello Bro community,

I’m new to Bro and using version 2.3.2 and want to extract all the exe’s seen on the network. In bro-file-extract we are using the file-extract.bro script to try to parse for the exe’s (partial of script):

global ext_map:table[string] of string = {
[“application/x/dosexec”] = “exe”,

redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro";
redef FileExtract::default_limit = 314572800;
redef FileExtract::prefix = “/var/log/netlogs/bro/file-extracts/”;

We also have the file-extract-http-local.bro set to extract on our network:

global http_extract_file_ignore: set [subnet] = {
10.0.0.0/8,
};

We think the problem is that load.bro has the file extract commented out under bro-icmp:
#@load ./file-extract-http-local.bro
#@load ./file-extract-types.bro
@load ./bro-file-extract
When I tried to enable these Bro failed the scripts check with errors like:
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7:can’t find base/protocols/http/file-ident
I continued to receive these errors and had to back out of removing the comments

Under bro-file-extract load.bro looks correct:
@load ./file-extract

What I’m getting in /var/log/netlogs/bro/file-extracts are entries like:
HTTP-F7K52nSzN3h7GNM31.exe
These files occur occasionally I’m not sure what they are.

Thanks for your help,

Al

Hi Al,

I'm new to Bro and using version 2.3.2 and want to extract all the exe's
seen on the network. In bro-file-extract we are using the file-extract.bro
script to try to parse for the exe's (partial of script):

First - is there any reason for you to still use 2.3.2? File handling (and
a lot of other things) have become more robust in 2.4.

In any case...

global ext_map:table[string] of string = {
["application/x/dosexec"] = "exe",

you probably want application/x-dosexec here, not x/dosexec. That might
already be enough to fix this.

redef FileExtract::prefix="/var/log/netlogs/bro/file-extracts.bro";

This line seems superfluous and wrong, especially since it is redef-ed
again two lines later.

redef FileExtract::default_limit = 314572800;
redef FileExtract::prefix = "/var/log/netlogs/bro/file-extracts/";

We also have the file-extract-http-local.bro set to extract on our network:

global http_extract_file_ignore: set [subnet] = {
                 10.0.0.0/8,
};

The following seems to talk about files that you modified locally and that
do not ship with the Bro distribution. As such, it is really hard to give
feedback about it.

We think the problem is that _load_.bro has the file extract commented out
under bro-icmp:
#@load ./file-extract-http-local.bro
#@load ./file-extract-types.bro
@load ./bro-file-extract
When I tried to enable these Bro failed the scripts check with errors like:
internal warning in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
6: Discarded extraneous Broxygen comment: Modified from base scripts to
extract only from external hosts
fatal error in
/usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line
7:can't find base/protocols/http/file-ident
I continued to receive these errors and had to back out of removing the
comments

Under bro-file-extract _load_.bro looks correct:
@load ./file-extract

What I'm getting in /var/log/netlogs/bro/file-extracts are entries like:
HTTP-F7K52nSzN3h7GNM31.exe
These files occur occasionally I'm not sure what they are.

I hope this helps,
Johanna

Thanks Johanna,

I made your recommended change and am still getting the error, see detail below:

file-extract.bro script

global ext_map:table[string] of string = { [“application/x/dosexec”] =
“exe”,
you probably want application/x-dosexec here, not x/dosexec. That might already be enough to fix this.

Changed:

file-extract.bro

global ext_map: table[string] of string = {

[“application/x-dosexec”] = “exe”,
[“text/plain”] = “txt”,
[“image/jpeg”] = “jpg”,
[“image/png”] = “png”,
[“text/html”] = “html”,
} &default ="";

Uncomment #@load ./file-extract-http-local.bro and #@load ./file-extract-types.bro:

load.bro

File extractions (/application/.*) – This has changed significantly in 2.2

@load ./file-extract-http-local.bro
@load ./file-extract-types.bro
@load ./bro-file-extract

I get this error again:

manager scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

proxy scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

enm1-eth1-httpproxy scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

enm2-eth2-httpinternal scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

enm3-eth3-collector scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

enm4-eth5-dns scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

enm5-eth6-syslog scripts failed.

internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

Here’s the script that it’s failing on:

file-extract-http-local.bro
@load base/protocols/http/main
@load base/protocols/http/file-ident
@load base/utils/files
module HTTP;
export {

Pattern of file mime types to extract from HTTP response entity bodies.

const extract_file_types_local = /NO_DEFAULT/ &redef;

The on-disk prefix for files to be extracted from HTTP entity bodies.

const extraction_prefix_local = “http-item” &redef;

redef record Info += {

On-disk file where the response body was extracted to.

extraction_file_local: file &log &optional;

Indicates if the response body is to be extracted or not. Must be

set before or by the first :bro:id:http_entity_data event for the

content.

extract_file_local: bool &default=F;
};
}

Define local sources to ignore file extract

global http_extract_file_ignore: set[subnet] = {

172.16.0.0/12, # Internal FRS, trusted destination
10.0.0.0/8, # Internal FRS, trusted destination
};

event http_entity_data(c: connection, is_orig: bool, length: count, data: string) &priority=-5
{

Client body extraction is not currently supported in this script.

if ( is_orig )
return;

We do not want to extract files from internal to internal hosts

if ( c$id$resp_h in http_extract_file_ignore )
return;

if ( c$http$first_chunk )
{
if ( c$http?$mime_type &&
extract_file_types_local in c$http$mime_type )
{
c$http$extract_file_local = T;
}

if ( c$http$extract_file_local )
{
local suffix = fmt("%s_%d.dat", is_orig ? “orig” : “resp”, c$http_state$current_response);
local fname = generate_extraction_filename(extraction_prefix_local, c, suffix);

c$http$extraction_file_local = open(fname);
enable_raw_output(c$http$extraction_file_local);
}
}

if ( c$http?$extraction_file_local )
print c$http$extraction_file_local, data;
}

event http_end_entity(c: connection, is_orig: bool)
{
if ( c$http?$extraction_file_local )
close(c$http$extraction_file_local);
}
Ideas? Thanks!

Thanks Johanna,

Adding additional information:

We are going to upgrade from 2.3.2 but have not yet.

I made your recommended change and am still getting the error, see detail below:

file-extract.bro script

global ext_map:table[string] of string = { [“application/x/dosexec”] =
“exe”,
you probably want application/x-dosexec here, not x/dosexec. That might already be enough to fix this.

Changed:

file-extract.bro
global ext_map: table[string] of string = {
[“application/x-dosexec”] = “exe”,
[“text/plain”] = “txt”,
[“image/jpeg”] = “jpg”,
[“image/png”] = “png”,
[“text/html”] = “html”,
} &default ="";

Un-comment #@load ./file-extract-http-local.bro and #@load ./file-extract-types.bro:

load.bro

File extractions (/application/.*) – This has changed significantly in 2.2

@load ./file-extract-http-local.bro
@load ./file-extract-types.bro
@load ./bro-file-extract

I get this error again:

manager scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident
proxy scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident
enm1-eth1-httpproxy scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident
enm2-eth2-httpinternal scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident
enm3-eth3-collector scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident
enm4-eth5-dns scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident
enm5-eth6-syslog scripts failed.
internal warning in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 6: Discarded extraneous Broxygen comment: Modified from base scripts to extract only from external hosts
fatal error in /usr/local/bro/share/bro/site/./custom/./file-extract-http-local.bro, line 7: can’t find base/protocols/http/file-ident

Here’s the script that it’s failing on:

file-extract-http-local.bro
@load base/protocols/http/main
@load base/protocols/http/file-ident
@load base/utils/files
module HTTP;
export {

Pattern of file mime types to extract from HTTP response entity bodies.

const extract_file_types_local = /NO_DEFAULT/ &redef;

The on-disk prefix for files to be extracted from HTTP entity bodies.

const extraction_prefix_local = “http-item” &redef;

redef record Info += {

On-disk file where the response body was extracted to.

extraction_file_local: file &log &optional;

Indicates if the response body is to be extracted or not. Must be

set before or by the first :bro:id:http_entity_data event for the

content.

extract_file_local: bool &default=F;
};
}

Define local sources to ignore file extract

global http_extract_file_ignore: set[subnet] = {
192.168.2.0.0/24, # Internal Seminal1, trusted destination
192.168.1.0/24, # Internal Seminal2, trusted destination
};

event http_entity_data(c: connection, is_orig: bool, length: count, data: string) &priority=-5
{

Client body extraction is not currently supported in this script.

if ( is_orig )
return;

We do not want to extract files from internal to internal hosts

if ( c$id$resp_h in http_extract_file_ignore )
return;

if ( c$http$first_chunk )
{
if ( c$http?$mime_type &&
extract_file_types_local in c$http$mime_type )
{
c$http$extract_file_local = T;
}

if ( c$http$extract_file_local )
{
local suffix = fmt("%s_%d.dat", is_orig ? “orig” : “resp”, c$http_state$current_response);
local fname = generate_extraction_filename(extraction_prefix_local, c, suffix);

c$http$extraction_file_local = open(fname);
enable_raw_output(c$http$extraction_file_local);
}
}

if ( c$http?$extraction_file_local )
print c$http$extraction_file_local, data;
}

event http_end_entity(c: connection, is_orig: bool)
{
if ( c$http?$extraction_file_local )
close(c$http$extraction_file_local);
}

Ideas? Thanks!

Al B.
Seminal Networks

I did not look through all of your script - the big reason that Bro
currently complains is that you try to load a nonexisting script (there is
no base/protocols/http/file-ident).

Johanna