Writing logs to both ACII and JSON

Zeek
1

Hello all,

Apologies in advance if this is an uninformed question - is it possible to configure Bro to write logs to both ASCII and JSON outputs (in different directories, preferably)? There’s another active thread on the mailing list at the moment about using multiple logger instances in Bro 2.5 which got me thinking that maybe this problem could be addressed by running multiple logger instances - one for ASCII logs, and one for JSON. If I understand the architecture correctly, I’d love to see a single manager instance duplicate all the data to send to both logger instances, and have one write ASCII and one to JSON. Is this a possibility? If so, how would I go about configuring this? I should note that my bro-knowledge is pretty limited to loading scripts from git hub and some very basic whitelisting, so unfortunately I’m not very comfortable modifying or writing bro code.

My organization relies on ASCII logs for plain text retention and all our normal 'nix plain text searching utilities, but I’ve been experimenting with Graylog and importing Bro JSON logs to Graylog is too easy and flexible for us to find the time to write grok parsers for our ASCII logs. We’re not prepared to not write logs to ASCII in prod, so I’m hopeful that there’s an almost easy way to get logs in both formats. For additional context we run Bro on Security Onion, and we’re currently running 2.4.1 in prod but plan to upgrade to 2.5 soon. I do have a test environment with Security Onion and Bro 2.5 available to me.

Any advice / steps on how to achieve this would be much appreciated!

Thanks,

James Gordon

2

Hi James,

Apologies in advance if this is an uninformed question - is it possible to
configure Bro to write logs to both ASCII and JSON outputs (in different
directories, preferably)?

some time ago I have written a small script that should fit your needs:

Using path_json you should also be able to log into a different directory.

I hope this helps,
Jan

3

Jan, + re-adding the bro mailing list because email is hard and I accidentally removed it - and in case there’s a bug impacting this script in v 2.5,

I tested this script on my physical security onion box, as well a security onion VM and a CentOS VM both with fresh installs of Bro 2.5. I tested with live network traffic and with a pcap and consistently get different results in my JSON log dir every time I run bro against the pcap.

When I run bro against a pcap, I get the following error:
“expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path]”

It looks like that line refers back to the json path. I have the json path defined as: const path_json = “/nsm/bro/logs/json/” &redef; - is this the correct way to define the log path?

Here’s some examples of the inconsistencies I see (this is reproduceable on all three systems). I’ll run the same pcap through Bro twice and we’ll get a different number of JSON logs, and different entries in the files - but ASCII logs always turn out the same.

root@sensor:/home/sensor/test# /opt/bro/bin/bro -r test.pcap /opt/bro/share/bro/site/local.bro
expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path]

root@sensor:/home/sensor/test# ls
capture_loss.log dhcp.log files.log loaded_scripts.log packet_filter.log ssl.log test.pcap weird.log
conn.log dns.log http.log notice.log reporter.log stats.log tunnel.log x509.log
root@sensor:/home/sensor/test# ls | wc -l
16

root@sensor:/home/sensor/test# cat conn.log | wc -l
1631

root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/
dhcp-json.log tunnel-json.log x509-json.log
root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/ | wc -l
3

As you can see there was no JSON conn log generated - so i’ll compare the dhcp logs:

root@sensor:/home/sensor/test# cat dhcp.log | wc -l
11
root@sensor:/home/sensor/test# cat /nsm/bro/logs/json/dhcp-json.log | wc -l
2

Some of the lines (8) in the ASCII file are headers so this log only missed one entry. It still missed logging all 1631 connections in the pcap to conn.log. I’ll clear out the logs now and try again, and we’ll get a different number types of json logs created.

root@sensor:/home/sensor/test# rm .log
root@sensor:/home/sensor/test# rm /nsm/bro/logs/json/

root@sensor:/home/sensor/test# /opt/bro/bin/bro -r test.pcap /opt/bro/share/bro/site/local.bro
expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path]

root@sensor:/home/sensor/test# ls
capture_loss.log dhcp.log files.log loaded_scripts.log packet_filter.log ssl.log test.pcap weird.log
conn.log dns.log http.log notice.log reporter.log stats.log tunnel.log x509.log
root@sensor:/home/sensor/test# ls | wc -l
16
root@sensor:/home/sensor/test# cat conn.log | wc -l
1631
root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/
capture_loss-json.log files-json.log packet_filter-json.log weird-json.log
conn-json.log loaded_scripts-json.log reporter-json.log x509-json.log
dhcp-json.log notice-json.log tunnel-json.log
root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/ | wc -l
11
root@sensor:/home/sensor/test# cat /nsm/bro/logs/json/conn-json.log | wc -l
1622

This time it logged all the connections, but it failed to even create http, ssl, stats, or dns json logs.

This script is exactly the functionality I need, I just can’t seem to get it working correctly. I don’t begin to understand why I get different results every time I run the same pcap through Bro.

Thanks!

James Gordon

4

When I run bro against a pcap, I get the following error:
"expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path]"

It looks like that line refers back to the json path. I have the json path defined as: const path_json = "/nsm/bro/logs/json/" &redef; - is this the correct way to define the log path?

No.. as the error message says there is a problem with filter$path being missing, not path_json.

This script is exactly the functionality I need, I just can't seem to get it working correctly. I don't begin to understand why I get different results every time I run the same pcap through Bro.

Because the script does this:

  for ( id in Log::active_streams )
    {
    if ( (enable_all_json || (id in include_json)) && (id !in exclude_json) )
      {
      local filter = Log::get_filter(id, "default");
      filter$name = string_cat(filter$name, "_json");
      filter$path = string_cat(path_json, filter$path, "-json");
      filter$config = config_json;
      filter$interv = interv_json;
      Log::add_filter(id, filter);
      }
    }

and Log::active_streams is a hash table populated at startup and the iteration order is random:

$ cat b.bro ;echo one:;bro b.bro |head; echo two:; bro b.bro |head
event bro_init() {
    for ( id in Log::active_streams )
        print id;
}
one:
Weird::LOG
PacketFilter::LOG
Conn::LOG
NetControl::SHUNT
DNS::LOG
FTP::LOG
SIP::LOG
SNMP::LOG
Syslog::LOG
DPD::LOG
two:
DPD::LOG
Software::LOG
IRC::LOG
RFB::LOG
SSL::LOG
KRB::LOG
SOCKS::LOG
Syslog::LOG
Log::UNKNOWN
DHCP::LOG

It's failing on one of your log files because filter$path is not set. Once that happens the event aborts and everything after that does not get json added.

The loop needs to check filter?$path before trying to use it.

You also probably have something broken (or at least weird) in your configuration because this error does not occur on a stock 2.5 config, so it's probably useful to figure out which of your logs has no path for some reason.

5

When I run bro against a pcap, I get the following error:
"expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field
value missing [Log::filter$path]"

I've just tested the script using 2.4.1 and 2.5 on try.bro.org
(Try Zeek) and locally using 2.5 with a
different path for JSON-logs. Unfortunately I am unable to reproduce
this error.

Maybe we can shed some light on this if we know which log doesn't
provide a path. Can you try to replace line 34 with:

if ( filter?$path )
    filter$path = string_cat(path_json, filter$path, "-json");
else
    Reporter::error(fmt("Path missing for %s", id));

That should provide some hint on which logs don't define a filter path.
If you can share your test pcap that might be of interest, too. One
thing I could imagine would be some kind of timing issue. Maybe playing
with the events &priority has influence on your results.

Jan

6

Looks like I was wrong about this being a ‘standard’ config - I fired this script up on a new VM and it worked with no issues. I pulled my local.bro off one of the machines I was testing with earlier and I had the SMB analyzer enabled. I enabled SMB on the new VM, added your lines from your last post and I have the following entries in reporter.log:

0.000000 Reporter::ERROR Path missing for SMB::MAPPING_LOG /usr/local/bro/share/bro/test/./add-json.bro, line 35

0.000000 Reporter::ERROR Path missing for SMB::CMD_LOG /usr/local/bro/share/bro/test/./add-json.bro, line 35

0.000000 Reporter::ERROR Path missing for SMB::FILES_LOG /usr/local/bro/share/bro/test/./add-json.bro, line 35

Any ideas on how to fix this (preferably), or hard exclude the SMB files that cause issues?

I ran with SMB disabled for about half an hour on a very slow network and everything worked as expected. Threw some test pcaps that I pulled off a Security Onion machine and those also ran well and logged as expected.

Thanks!

James Gordon

7

0.000000 Reporter::ERROR Path missing for SMB::MAPPING_LOG
/usr/local/bro/share/bro/test/./add-json.bro,
line 35

0.000000 Reporter::ERROR Path missing for SMB::CMD_LOG
/usr/local/bro/share/bro/test/./add-json.bro,
line 35

0.000000 Reporter::ERROR Path missing for SMB::FILES_LOG
/usr/local/bro/share/bro/test/./add-json.bro,
line 35

Using the SMB-Analyzer I was able to reproduce the issue: The
SMB-Analyzer does not set path, which is indeed optional but used for
all the other logs by convention.

Any ideas on how to fix this (preferably), or hard exclude the SMB files
that cause issues?

I have fixed the script but I need some more testing (just noticed that
path_func wasn't supported as well). For now, you can use exclude_json
to exclude SMB::MAPPING_LOG, SMB::CMD_LOG and SMB::FILES_LOG.

Jan

8

Yup, you are right. This looks like an oversight, the path should have
been set for all the create_stream calls. I will fix this in master in a
few minutes - thanks for finding this :slight_smile:

Johanna

9

Using the SMB-Analyzer I was able to reproduce the issue: The
SMB-Analyzer does not set path, which is indeed optional but used for
all the other logs by convention.

Yup, you are right. This looks like an oversight, the path should have
been set for all the create_stream calls. I will fix this in master in a
few minutes - thanks for finding this :slight_smile:

Thanks a lot for the quick fix! This way the handling of streams is more
consistent across the streams. I will also update my script once I find
some time for testing, as not specifying the path is generally valid
(cf.
base/frameworks/logging/main.bro — Bro 2.6.1 documentation).

Jan

10

Jan + all,

Thanks for your help on all this! The script is working great with the exclusion of SMB logs. Apologies for all the confusion on my side - I’m not much of a programmer, but use Bro daily as a vital data source at my job. Anything to enhance the data we get is always good, and JSON makes it much easier to ingest into other sources.

I’ve already come across another Bro script that the add-json.bro script doesn’t seem to agree with, but will unload that script as it doesn’t provide much value for my org. I look forward to seeing an updated version that can handle these stray log files though!

Thanks again!

James Gordon

11

Hi James,

I've already come across another Bro script that the add-json.bro script
doesn't seem to agree with, but will unload that script as it doesn't
provide much value for my org. I look forward to seeing an updated version
that can handle these stray log files though!

meanwhile I have updated the script. It should work with SMB using Bro
2.5 and supports path functions. I hope the new version also works with
the third-party script you mentioned.

As additional JSON-logging seems to be a quite common requirement, I
have added the script as a package for bro-pkg. Thanks to Johanna it's
already merged! If you have configured bro-pkg, the following will
install the script:

bro-pkg install add-json

The package is located at GitHub - J-Gras/add-json: Enables additional JSON-logging for Zeek.. In case
you encounter any problems, please let me know.

Best regards,
Jan