# Bro logs from JSON to TSV **URL:** \ **Category:** Zeek\ **Created:** [January 3, 2018, 9:13pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138 "2018-01-03T21:13:59Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![ElBadry\_Shaker\_Moust](https://avatars.discourse-cdn.com/v4/letter/e/eb8c5e/32.png) [@ElBadry\_Shaker\_Moust](https://community.zeek.org/u/ElBadry_Shaker_Moust)\ **Post date:** [January 3, 2018, 9:13pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/1 "2018-01-03T21:13:59Z")
Greetings, Hope my email finds you well. I was wondering if someone can help me figure out how to transform existing Bro logs from JSON format to TSV format. The TSV format is what Bro uses by default to write log files. Thanks in advance! Sincerely, **Moustafa ElBadry** , Information Security Analyst, Office of Information Security **Oregon State University** | Information Services | 541-737-4545 ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [January 3, 2018, 9:25pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/2 "2018-01-03T21:25:01Z")
Do you want the exact TSV format with the #fields and #types header, or just TSV in general? This is a somewhat strange thing to want to do - since working with the data in JSON format is generally easier.. What exactly are you trying to accomplish after you convert the logs? ---
**Author:** ![ElBadry\_Shaker\_Moust](https://avatars.discourse-cdn.com/v4/letter/e/eb8c5e/32.png) [@ElBadry\_Shaker\_Moust](https://community.zeek.org/u/ElBadry_Shaker_Moust)\ **Post date:** [January 3, 2018, 9:38pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/3 "2018-01-03T21:38:26Z")
I want the exact TSV format. We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro’s default TSV format. Moustafa ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [January 4, 2018, 8:35pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/4 "2018-01-04T20:35:43Z")
Ah, I see now. You have a few of options here. You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem. There is an open issue for RITA to support json: [https://github.com/ocmdev/rita/issues/146](https://github.com/ocmdev/rita/issues/146) A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses. For awk stuff you can swap out bro-cut for jq or [https://github.com/JustinAzoff/json-cut](https://github.com/JustinAzoff/json-cut) json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs. For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do     zcat notice.\* | bro-cut note msg | awk ...     zcat notice.\* | json-cut note msg | awk ... For jq you use something like     zcat notice.\* | jq -r '[.note, .msg]|@tsv' | awk ... If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that 🙂 ---
**Author:** ![ElBadry\_Shaker\_Moust](https://avatars.discourse-cdn.com/v4/letter/e/eb8c5e/32.png) [@ElBadry\_Shaker\_Moust](https://community.zeek.org/u/ElBadry_Shaker_Moust)\ **Post date:** [January 5, 2018, 4:34pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/5 "2018-01-05T16:34:58Z")
Great. Thanks Justin for sharing this. Definitely helps us a lot. Moustafa     \>     \> I want the exact TSV format.     \>     \> We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro’s default TSV format.     \>     \> Moustafa          Ah, I see now. You have a few of options here.          You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem.          There is an open issue for RITA to support json: [https://github.com/ocmdev/rita/issues/146](https://github.com/ocmdev/rita/issues/146)          A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses.          For awk stuff you can swap out bro-cut for jq or [https://github.com/JustinAzoff/json-cut](https://github.com/JustinAzoff/json-cut)          json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs.          For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do              zcat notice.\* | bro-cut note msg | awk ...         zcat notice.\* | json-cut note msg | awk ...          For jq you use something like              zcat notice.\* | jq -r '[.note, .msg]|@tsv' | awk ...          If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that 🙂 ---
**Author:** ![ElBadry\_Shaker\_Moust](https://avatars.discourse-cdn.com/v4/letter/e/eb8c5e/32.png) [@ElBadry\_Shaker\_Moust](https://community.zeek.org/u/ElBadry_Shaker_Moust)\ **Post date:** [January 8, 2018, 7:27pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/6 "2018-01-08T19:27:26Z")
Hello, I have a follow up question on this. Justin, you mentioned that I could tell bro to write out the logs in both formats (TSV and JSON) at the same time. How can I do this? And can I have the TSV logs saved in one directory and the JSON logs saved in another directory? Is the ascii.bro file located at /usr/local/bro/share/bro/base/frameworks/logging/writers/ the right file where we can configure bro to write in two different formats? Thanks a lot for your help. I really appreciate it! Moustafa     Great. Thanks Justin for sharing this. Definitely helps us a lot.          Moustafa              \>         \> I want the exact TSV format.         \>         \> We currently have our Bro cluster writing logs in JSON. There are couple of network traffic analytics tools like RITA (Real Intelligence Threat Analytics) and some AWK scripts that we want to use. The problem is that the tools we want to use work only with Bro’s default TSV format.         \>         \> Moustafa                  Ah, I see now. You have a few of options here.                  You could just tell bro to write out the logs in both formats at the same time. For older logs there is only a script for bro that can re-log to json, but not the other way, most people have the opposite problem.                  There is an open issue for RITA to support json: [https://github.com/ocmdev/rita/issues/146](https://github.com/ocmdev/rita/issues/146)                  A tool to convert the json logs back into the TSV format could be written, but ultimately that would be a waste of time. Better to update RITA to support json instead of writing more tools to work with the tsv format that only bro uses.                  For awk stuff you can swap out bro-cut for jq or [https://github.com/JustinAzoff/json-cut](https://github.com/JustinAzoff/json-cut)                  json-cut it doesn't support all the options that bro-cut supports and may be a bit buggy, but it's easier to extract a few fields from a json log as TSV and 2x faster than jq. If I can find a nice, small json library for C we can probably update bro-cut to natively support the json logs.                  For now, to extract note and msg from a stream of notice logs with bro-cut and json-cut you just do                      zcat notice.\* | bro-cut note msg | awk ...             zcat notice.\* | json-cut note msg | awk ...                  For jq you use something like                      zcat notice.\* | jq -r '[.note, .msg]|@tsv' | awk ...                  If the awk scripts are hardcoding top level field numbers like $3 and $5 instead of using bro-cut... they should not do that 🙂 ---
**Author:** ![ElBadry\_Shaker\_Moust](https://avatars.discourse-cdn.com/v4/letter/e/eb8c5e/32.png) [@ElBadry\_Shaker\_Moust](https://community.zeek.org/u/ElBadry_Shaker_Moust)\ **Post date:** [January 9, 2018, 4:38pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/7 "2018-01-09T16:38:21Z")
Great. Thanks for sharing this. I really appreciate it! Moustafa     Look at add-JSON:          [https://gist.github.com/J-Gras/f9f86828f9e9d9c0b8f0908bc3573bb0](https://gist.github.com/J-Gras/f9f86828f9e9d9c0b8f0908bc3573bb0)          That will log JSON output to the path you define in path\_json, and should retain the standard logging as well. Add-JSON is also available as a bro package.          I've been able to get the log rotation to work for this script, though. I ended up creating a cron job that stops bro once a day, purges the JSON logs, and restarts. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:45pm UTC](https://community.zeek.org/t/bro-logs-from-json-to-tsv/5138/8 "2022-05-06T15:45:29Z")