Speed up bro execution

Zeek
1

Hello,

I am handling rather big pcap files in the size of 500gb and bro execution takes a few hours to complete. For this reason I am looking for ways to speed up the execution.

I want to keep only specific logs files with the goal of making my bro execution faster. For my research I want to keep the following files: conn.log, ssl.log, x509.log, dns.log, http.log

From what I understood this command should do the trick: bro -r <pcap_file_name> -b base/protocols/ssl base/protocols/dns base/protocols/conn base/protocols/http

However, with the addition of base/protocol/ssl I also get the tunnel.log and files.log which I do not need. Is there a way to exclude these files from logging?

Moreover, I have a rather powerful machines with 8 cores and 8gb of RAM does anyone know a way to fully utilize that when using bro?

Thanks all,
Mike

2

There are two ways you may be able to address the issues of maximizing processor use, but both would take some additional work. May be worth it if you need to do this a lot.

  1. Setup a TCPREPLAY server that forwards the traffic to the bro system and use the normal fanout options (AF_PACKET, PF_RING, etc) to maximize processor usage. You could theoretically do the replay and listening on the same box, but tcpreplay would likely bog down one of the procs (2 more for the manager/proxy, leaving 5 for Bro analysis).

2.Divide up your 500gb PCAP into smaller chunks (e.g. different pcaps based on SRC/DST) and run “bro -r /path/to/pcap” multiple times in parallel. You’d have to point each run to a different log-file path and combine them later. Best to write a script for this if you often have to run large pcaps offline.