# Writing logs to both ACII and JSON **URL:** \ **Category:** Zeek\ **Created:** [January 9, 2017, 3:27pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583 "2017-01-09T15:27:31Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![James\_Gordon](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@James\_Gordon](https://community.zeek.org/u/James_Gordon)\ **Post date:** [January 9, 2017, 3:27pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/1 "2017-01-09T15:27:31Z")
Hello all, Apologies in advance if this is an uninformed question - is it possible to configure Bro to write logs to both ASCII and JSON outputs (in different directories, preferably)? There’s another active thread on the mailing list at the moment about using multiple logger instances in Bro 2.5 which got me thinking that maybe this problem could be addressed by running multiple logger instances - one for ASCII logs, and one for JSON. If I understand the architecture correctly, I’d love to see a single manager instance duplicate all the data to send to both logger instances, and have one write ASCII and one to JSON. Is this a possibility? If so, how would I go about configuring this? I should note that my bro-knowledge is pretty limited to loading scripts from git hub and some very basic whitelisting, so unfortunately I’m not very comfortable modifying or writing bro code. My organization relies on ASCII logs for plain text retention and all our normal 'nix plain text searching utilities, but I’ve been experimenting with Graylog and importing Bro JSON logs to Graylog is too easy and flexible for us to find the time to write grok parsers for our ASCII logs. We’re not prepared to not write logs to ASCII in prod, so I’m hopeful that there’s an almost easy way to get logs in both formats. For additional context we run Bro on Security Onion, and we’re currently running 2.4.1 in prod but plan to upgrade to 2.5 soon. I do have a test environment with Security Onion and Bro 2.5 available to me. Any advice / steps on how to achieve this would be much appreciated! Thanks, James Gordon ---
**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\ **Post date:** [January 9, 2017, 4:03pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/2 "2017-01-09T16:03:58Z")
Hi James, > Apologies in advance if this is an uninformed question - is it possible to > configure Bro to write logs to both ASCII and JSON outputs (in different > directories, preferably)? some time ago I have written a small script that should fit your needs: > Using path\_json you should also be able to log into a different directory. I hope this helps, Jan ---
**Author:** ![James\_Gordon](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@James\_Gordon](https://community.zeek.org/u/James_Gordon)\ **Post date:** [January 11, 2017, 9:31pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/3 "2017-01-11T21:31:46Z")
Jan, + re-adding the bro mailing list because email is hard and I accidentally removed it - and in case there’s a bug impacting this script in v 2.5, I tested this script on my physical security onion box, as well a security onion VM and a CentOS VM both with fresh installs of Bro 2.5. I tested with live network traffic and with a pcap and consistently get different results in my JSON log dir every time I run bro against the pcap. When I run bro against a pcap, I get the following error: “expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path]” It looks like that line refers back to the json path. I have the json path defined as: const path\_json = “/nsm/bro/logs/json/” &redef; - is this the correct way to define the log path? Here’s some examples of the inconsistencies I see (this is reproduceable on all three systems). I’ll run the same pcap through Bro twice and we’ll get a different number of JSON logs, and different entries in the files - but ASCII logs always turn out the same. root@sensor:/home/sensor/test# /opt/bro/bin/bro -r test.pcap /opt/bro/share/bro/site/local.bro expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path] root@sensor:/home/sensor/test# ls capture\_loss.log dhcp.log files.log loaded\_scripts.log packet\_filter.log ssl.log test.pcap weird.log conn.log dns.log http.log notice.log reporter.log stats.log tunnel.log x509.log root@sensor:/home/sensor/test# ls | wc -l 16 root@sensor:/home/sensor/test# cat conn.log | wc -l 1631 root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/ dhcp-json.log tunnel-json.log x509-json.log root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/ | wc -l 3 As you can see there was no JSON conn log generated - so i’ll compare the dhcp logs: root@sensor:/home/sensor/test# cat dhcp.log | wc -l 11 root@sensor:/home/sensor/test# cat /nsm/bro/logs/json/dhcp-json.log | wc -l 2 Some of the lines (8) in the ASCII file are headers so this log only missed one entry. It still missed logging all 1631 connections in the pcap to conn.log. I’ll clear out the logs now and try again, and we’ll get a different number types of json logs created. root@sensor:/home/sensor/test# rm _.log root@sensor:/home/sensor/test# rm /nsm/bro/logs/json/_ root@sensor:/home/sensor/test# /opt/bro/bin/bro -r test.pcap /opt/bro/share/bro/site/local.bro expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path] root@sensor:/home/sensor/test# ls capture\_loss.log dhcp.log files.log loaded\_scripts.log packet\_filter.log ssl.log test.pcap weird.log conn.log dns.log http.log notice.log reporter.log stats.log tunnel.log x509.log root@sensor:/home/sensor/test# ls | wc -l 16 root@sensor:/home/sensor/test# cat conn.log | wc -l 1631 root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/ capture\_loss-json.log files-json.log packet\_filter-json.log weird-json.log conn-json.log loaded\_scripts-json.log reporter-json.log x509-json.log dhcp-json.log notice-json.log tunnel-json.log root@sensor:/home/sensor/test# ls /nsm/bro/logs/json/ | wc -l 11 root@sensor:/home/sensor/test# cat /nsm/bro/logs/json/conn-json.log | wc -l 1622 This time it logged all the connections, but it failed to even create http, ssl, stats, or dns json logs. This script is exactly the functionality I need, I just can’t seem to get it working correctly. I don’t begin to understand why I get different results every time I run the same pcap through Bro. Thanks! James Gordon ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [January 11, 2017, 11:12pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/4 "2017-01-11T23:12:54Z")
> When I run bro against a pcap, I get the following error: > "expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field value missing [Log::filter$path]" > > It looks like that line refers back to the json path. I have the json path defined as: const path\_json = "/nsm/bro/logs/json/" &redef; - is this the correct way to define the log path? No.. as the error message says there is a problem with filter$path being missing, not path\_json. > This script is exactly the functionality I need, I just can't seem to get it working correctly. I don't begin to understand why I get different results every time I run the same pcap through Bro. Because the script does this:   for ( id in Log::active\_streams )     {     if ( (enable\_all\_json || (id in include\_json)) && (id !in exclude\_json) )       {       local filter = Log::get\_filter(id, "default");       filter$name = string\_cat(filter$name, "\_json");       filter$path = string\_cat(path\_json, filter$path, "-json");       filter$config = config\_json;       filter$interv = interv\_json;       Log::add\_filter(id, filter);       }     } and Log::active\_streams is a hash table populated at startup and the iteration order is random: $ cat b.bro ;echo one:;bro b.bro |head; echo two:; bro b.bro |head event bro\_init() {     for ( id in Log::active\_streams )         print id; } one: Weird::LOG PacketFilter::LOG Conn::LOG NetControl::SHUNT DNS::LOG FTP::LOG SIP::LOG SNMP::LOG Syslog::LOG DPD::LOG two: DPD::LOG Software::LOG IRC::LOG RFB::LOG SSL::LOG KRB::LOG SOCKS::LOG Syslog::LOG Log::UNKNOWN DHCP::LOG It's failing on one of your log files because filter$path is not set. Once that happens the event aborts and everything after that does not get json added. The loop needs to check filter?$path before trying to use it. You also probably have something broken (or at least weird) in your configuration because this error does not occur on a stock 2.5 config, so it's probably useful to figure out which of your logs has no path for some reason. ---
**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\ **Post date:** [January 11, 2017, 11:22pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/5 "2017-01-11T23:22:08Z")
> When I run bro against a pcap, I get the following error: > "expression error in /opt/bro/share/bro/test/./add-json.bro, line 34: field > value missing [Log::filter$path]" I've just tested the script using 2.4.1 and 2.5 on try.bro.org ([Try Zeek](http://try.bro.org/#/trybro/saved/115989)) and locally using 2.5 with a different path for JSON-logs. Unfortunately I am unable to reproduce this error. Maybe we can shed some light on this if we know which log doesn't provide a path. Can you try to replace line 34 with: if ( filter?$path )     filter$path = string\_cat(path\_json, filter$path, "-json"); else     Reporter::error(fmt("Path missing for %s", id)); That should provide some hint on which logs don't define a filter path. If you can share your test pcap that might be of interest, too. One thing I could imagine would be some kind of timing issue. Maybe playing with the events &priority has influence on your results. Jan ---
**Author:** ![James\_Gordon](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@James\_Gordon](https://community.zeek.org/u/James_Gordon)\ **Post date:** [January 12, 2017, 1:14am UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/6 "2017-01-12T01:14:48Z")
Looks like I was wrong about this being a ‘standard’ config - I fired this script up on a new VM and it worked with no issues. I pulled my local.bro off one of the machines I was testing with earlier and I had the SMB analyzer enabled. I enabled SMB on the new VM, added your lines from your last post and I have the following entries in reporter.log: 0.000000 Reporter::ERROR Path missing for SMB::MAPPING\_LOG /usr/local/bro/share/bro/test/./add-json.bro, line 35 0.000000 Reporter::ERROR Path missing for SMB::CMD\_LOG /usr/local/bro/share/bro/test/./add-json.bro, line 35 0.000000 Reporter::ERROR Path missing for SMB::FILES\_LOG /usr/local/bro/share/bro/test/./add-json.bro, line 35 Any ideas on how to fix this (preferably), or hard exclude the SMB files that cause issues? I ran with SMB disabled for about half an hour on a _very_ slow network and everything worked as expected. Threw some test pcaps that I pulled off a Security Onion machine and those also ran well and logged as expected. Thanks! James Gordon ---
**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\ **Post date:** [January 12, 2017, 10:00am UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/7 "2017-01-12T10:00:59Z")
> 0.000000 Reporter::ERROR Path missing for SMB::MAPPING\_LOG > /usr/local/bro/share/bro/test/./add-json.bro, > line 35 > > 0.000000 Reporter::ERROR Path missing for SMB::CMD\_LOG > /usr/local/bro/share/bro/test/./add-json.bro, > line 35 > > 0.000000 Reporter::ERROR Path missing for SMB::FILES\_LOG > /usr/local/bro/share/bro/test/./add-json.bro, > line 35 Using the SMB-Analyzer I was able to reproduce the issue: The SMB-Analyzer does not set path, which is indeed optional but used for all the other logs by convention. > Any ideas on how to fix this (preferably), or hard exclude the SMB files > that cause issues? I have fixed the script but I need some more testing (just noticed that path\_func wasn't supported as well). For now, you can use exclude\_json to exclude SMB::MAPPING\_LOG, SMB::CMD\_LOG and SMB::FILES\_LOG. Jan ---
**Author:** ![johanna](https://avatars.discourse-cdn.com/v4/letter/j/50afbb/32.png) [@johanna](https://community.zeek.org/u/johanna)\ **Post date:** [January 12, 2017, 12:44pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/8 "2017-01-12T12:44:36Z")
Yup, you are right. This looks like an oversight, the path should have been set for all the create\_stream calls. I will fix this in master in a few minutes - thanks for finding this 🙂 Johanna ---
**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\ **Post date:** [January 12, 2017, 2:31pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/9 "2017-01-12T14:31:47Z")
> > Using the SMB-Analyzer I was able to reproduce the issue: The > > SMB-Analyzer does not set path, which is indeed optional but used for > > all the other logs by convention. > > Yup, you are right. This looks like an oversight, the path should have > been set for all the create\_stream calls. I will fix this in master in a > few minutes - thanks for finding this 🙂 Thanks a lot for the quick fix! This way the handling of streams is more consistent across the streams. I will also update my script once I find some time for testing, as not specifying the path is generally valid (cf. [base/frameworks/logging/main.bro — Bro 2.6.1 documentation](https://www.bro.org/sphinx/scripts/base/frameworks/logging/main.bro.html#type-Log::Filter)). Jan ---
**Author:** ![James\_Gordon](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@James\_Gordon](https://community.zeek.org/u/James_Gordon)\ **Post date:** [January 12, 2017, 6:07pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/10 "2017-01-12T18:07:24Z")
Jan + all, Thanks for your help on all this! The script is working great with the exclusion of SMB logs. Apologies for all the confusion on my side - I’m not much of a programmer, but use Bro daily as a vital data source at my job. Anything to enhance the data we get is always good, and JSON makes it much easier to ingest into other sources. I’ve already come across another Bro script that the add-json.bro script doesn’t seem to agree with, but will unload that script as it doesn’t provide much value for my org. I look forward to seeing an updated version that can handle these stray log files though! Thanks again! James Gordon ---
**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\ **Post date:** [January 15, 2017, 2:36pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/11 "2017-01-15T14:36:02Z")
Hi James, > I've already come across another Bro script that the add-json.bro script > doesn't seem to agree with, but will unload that script as it doesn't > provide much value for my org. I look forward to seeing an updated version > that can handle these stray log files though! meanwhile I have updated the script. It should work with SMB using Bro 2.5 and supports path functions. I hope the new version also works with the third-party script you mentioned. As additional JSON-logging seems to be a quite common requirement, I have added the script as a package for bro-pkg. Thanks to Johanna it's already merged! If you have configured bro-pkg, the following will install the script: bro-pkg install add-json The package is located at [GitHub - J-Gras/add-json: Enables additional JSON-logging for Zeek.](https://github.com/J-Gras/add-json). In case you encounter any problems, please let me know. Best regards, Jan ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/writing-logs-to-both-acii-and-json/4583/12 "2022-05-06T15:44:28Z")