File Extraction

Zeek
1

Hey Bro members,

Some questions about File Extraction for Bro on my Red Hat 6.5 server.

File Extraction

  1. I’ve configured Bro appropriately to extract “exe” mime types from the HTTP protocol. It works great. However, the “files.log” only contains MD5 and SHA1 entries for some of the files, not all of them. How do I fix this so that all of the extracted files have the MD5 and SHA1 entries?

  2. I have analysts that need access to the files (/var/data/bro/extracted), but I’ve noticed that bro creates the files with random permissions, either 644 or 600… so they can only access the ones with 644. How do I ensure bro extracts the file with the 644 permission set on all of them? (see below example)

-rw-r–r–. 1 root root 703736 Nov 7 04:29 HTTP-FzedfU1k233I0Kiwn8.exe.dead
-rw-------. 1 root root 358799 Nov 5 04:07 HTTP-FzFPDF3EF77DEUSjdf.exe.dead
-rw-------. 1 root root 26121658 Nov 6 03:17 HTTP-FzhwqG33dNtUHZraZ4.exe.dead
-rw-------. 1 root root 249856 Nov 5 00:00 HTTP-FZi4XxyXiaoBquRu.exe.dead
-rw-r–r–. 1 root root 332536 Nov 28 14:21 HTTP-FZikQY3r8a7gXtLlee.exe.dead
-rw-r–r–. 1 root root 24306 Nov 12 05:02 HTTP-FzjIxe2MR9Uj8S8j27.exe.dead
-rw-------. 1 root root 94568 Nov 6 04:00 HTTP-FzJjxg23F3HPqtRbC2.exe.dead

  1. Is there a way to tell bro to run as a different user / group other than root? I didn’t see any options for it in the bro --help. I would assume I would have to give broctl and bro binaries / modules the ownership and executable rights by another user, then have bro start up as that new user, but wanted to see if there was an easier way. Otherwise I’d have to change the default install configuration each time I upgrade.

Thanks!
JW

2

I doubt.

Bro needs to run as root because it captures network traffic using libpcap.

3

1. I've configured Bro appropriately to extract "exe" mime types from the HTTP protocol. It works great. However, the "files.log" only contains MD5 and SHA1 entries for some of the files, not all of them. How do I fix this so that all of the extracted files have the MD5 and SHA1 entries?

Hm. Could you show a line from files.log where a file was extracted but you did get hashes? I suspect this is because the file ended up having trouble being transferred which will cause the hash analyzers to stop hashing (could be due to a packet drop even).

2. I have analysts that need access to the files (/var/data/bro/extracted), but I've noticed that bro creates the files with random permissions, either 644 or 600... so they can only access the ones with 644. How do I ensure bro extracts the file with the 644 permission set on all of them? (see below example)

This is weird. I haven’t seen Bro creating extracted files with different permissions before. Are these all dropped directly into the location by Bro or copied there later? All files from the same Bro process or from multiple Bro processes?

3. Is there a way to tell bro to run as a different user / group other than root? I didn't see any options for it in the bro --help. I would assume I would have to give broctl and bro binaries / modules the ownership and executable rights by another user, then have bro start up as that new user, but wanted to see if there was an easier way. Otherwise I'd have to change the default install configuration each time I upgrade.

Unfortunately, this is an area that hasn’t seen much attention for too long. We’ve been meaning to spend some time adding privilege dropping to Bro but it hasn’t happened yet. This would make Bro similar to many other tools that are run by root, but then drop their privileges to another user account at runtime, and in our case this would happen after Bro opens any interfaces that it needs to sniff on.

Is there anyone out there in Bro-land that is interested in digging into the core enough to add privilege dropping? :slight_smile:

  .Seth

4

As for running bro as non-root, I’ve always created a bro user/group, chown’d the bro directory and files to that user and group, and use setcap as below.

setcap cap_net_raw,cap_net_admin=eip /path/to/bro/bin

Though not privilege dropping, it will still give you the added security and peace of mind that you aren’t running as root. I’ve been doing this for several years now and never had any issues with it. Albeit, this has always been on a linux based server.
​

5

That’s definitely the alternative. I used to do the same thing on FreeBSD, but on there you make changes to the devd.conf so that your lower-privileged user has permission to open the /dev/bpf* devices.

Ultimately I think that using a privilege dropping mechanism is probably the easiest way for most people because it doesn’t require any special configuration to the OS to make it work. You would just configure broctl to run Bro as a certain user or run Bro with a certain flag (depending on if you use BroControl or run Bro directly).

  .Seth

6

1.)
Here is a sample of the files.log where there are entries that have no MD5 or SHA1:

1417723198.683744 FAvAKu2jFR7eGbRsH9 23.235.40.193 143.45.2.4 CYzgUS3Y6uS9dk3YM9 HTTP 0 MD5,SHA1 image/png - 2.677866 F F 60072
7 731951 131224 0 F - - - - -
1417723198.684489 FBn4gNzWGTRPkoTm3 23.235.40.193 143.45.97.186 CHjRXfnjhGudwL2re HTTP 0 MD5,SHA1 image/png - 2.677447 F F 59842
3 731951 133528 0 F - - - - -
1417723201.725784 FTegh14C2bs0OBTXil 143.45.170.74 184.75.174.163 Ch6xdv40tNxq4gWOb7 HTTP 0 MD5,SHA1 text/plain - 0.000000 T T 44 44 0 0 F - bc443e340953993a069985719f1cac76 8b1c00142caf938a917ef5cd04a2977993c7edd3 - -

Packet drop might be an issue, this sensor is hit pretty hard, and we see roughly 1-5% packets dropped on tcpdump for example. One thing that sticks out now as I look at this log is that it says “image/png” and “text/plain” for mime_type. I configured the script I wrote to only extract exe’s:

#make a lookup hash of mimetypes to file extensions
global ext_map: table[string] of string = {
 [“application/x-dosexec”] = “exe”,
 [“text/plain”] = “txt”,
 [“image/jpeg”] = “jpg”,
 [“text/html”] = “html”,
} &default=””;
#create an event to handle new files
event file_new(f: fa_file)
{
  #ignore files with no mimetype and get only exe’s
  if (! f?$mime_type || f$mime_type != “application/x-dosexec”)
  return;
  

  #otherwise it passes and continues on, so it IS an exe file  
  #lets capture it and put it in a directory, name it,& analyze it
  local ext = “”;
  if ( f?$mime_type )
   ext = ext_map[f$mime_type];
     #add <dot>.dead to the end to ensure file cannot be executed accidentally
  local fname = fmt(“/var/data/bro/extracted/%s-%s.%s.dead”, f$source, f$id, ext);
  Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);

}

2.)

All are sent to the directory by bro (based on script above that has the file_new event handler).
As far as the number of bro processes… I honestly don’t know. I start bro via broctl via cron:

@reboot root sleep 90 ; /opt/bro/bin/broctl start

However, if I look at the process table, it shows 3 processes:
]# ps -ef | grep -i bro
root 2733 1 0 Nov13 ? 00:00:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i em1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto
root 3425 2733 39 Nov13 ? 8-06:46:02 /opt/bro/bin/bro -i em1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto
root 4073 3425 13 Nov13 ? 2-19:47:05 /opt/bro/bin/bro -i em1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto

3.)
Sounds good, I’ll keep an eye out.

Let me know on the other 2, thanks!

JW

7

Very interesting Marcus, I’ll take a look at the setcap option, thanks!