problem with bro json log format

So, I am not sure whatgs going on, but when I do:

python -m json.tool < $somelog

I get

Extra data: line 2 column 1 - line 3 column 1 (char 507 - 1011)

All I did was turn json format logging on in ascii writer conf. All of my bro logs cant seem to be parsed by json.tool…

json.tool tries to read the entire log file as a single json record when it consists of one json record per line.

Use jq instead: https://stedolan.github.io/jq/

Yep, I had just gone down that route. :slight_smile: I had mistakenly believed that json.tool did more than one record at once. Thanks for the fast response Justin!

Hi,

>
> So, I am not sure whatgs going on, but when I do:
>
> python -m json.tool < $somelog
>
> I get
>
> Extra data: line 2 column 1 - line 3 column 1 (char 507 - 1011)
>
> All I did was turn json format logging on in ascii writer conf. All
> of my bro logs cant seem to be parsed by json.tool....

json.tool tries to read the entire log file as a single json record
when it consists of one json record per line.

Use jq instead: https://stedolan.github.io/jq/

I would propose an alternative sticking to base python:

import json
with open('conn.log') as conn:
  for line in conn:
    print(json.loads(line))

or bash:

for line in $(cat conn.log); do echo $line | python -m json.tool; done

Franky

I would propose an alternative sticking to base python:

import json
with open('conn.log') as conn:
for line in conn:
   print(json.loads(line))

This would be closer to what jq does by default:

import json
import pprint
import sys

for line in sys.stdin:
    pprint.pprint(json.loads(line))

or bash:

for line in $(cat conn.log); do echo $line | python -m json.tool; done

$(cat conn.log) will try to expand to the entire contents of the conn log and blow up..

while read line; do echo $line | python -m json.tool;done < conn.log

would work, but since it runs python for each log line it won't be very fast :slight_smile: