# problem ingesting bro json logs into splunk **URL:** \ **Category:** Zeek\ **Created:** [July 14, 2016, 1:33pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242 "2016-07-14T13:33:50Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![philosnef](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@philosnef](https://community.zeek.org/u/philosnef)\ **Post date:** [July 14, 2016, 1:33pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/1 "2016-07-14T13:33:50Z")
We are getting a spurious sourcetype when ingesting bro json logs into splunk. Specifically, we are getting a sourcetype of bro\_00. There is no log file named this, and the splunkforwarder is just pushing the raw logs for indexing into splunk. There is no massaging of the log data. Anyone know why this sourcetype is popping up? ---
**Author:** ![Brandon\_Lattin2](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@Brandon\_Lattin2](https://community.zeek.org/u/Brandon_Lattin2)\ **Post date:** [July 14, 2016, 2:14pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/2 "2016-07-14T14:14:25Z")
Do you have the Splunk installed? ([https://splunkbase.splunk.com/app/1617/](https://splunkbase.splunk.com/app/1617/)) The TA will dynamically create sourcetypes based on the log name. # Dynamic source typing based on log filename # Match: conn.log, bro.conn.log, # md5.bro.conn.log, whatever.conn.log [BroAutoType] DEST\_KEY = MetaData:Sourcetype SOURCE\_KEY = MetaData:Source REGEX = ([a-zA-Z0-9-]+)(?:.[0-9-]_)?(?:.[0-9:-]_)?.log FORMAT = sourcetype::bro\_$1 WRITE\_META = true ---
**Author:** ![philosnef](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@philosnef](https://community.zeek.org/u/philosnef)\ **Post date:** [July 14, 2016, 2:23pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/3 "2016-07-14T14:23:32Z")
There are no 00.log files in Bro, so the automatic generation of the sourcetype bro\_00 makes no sense. It does not follow the standard sourcetype pinning that all the other log files generate. find . -name “00\*” in the parent logs directory reports zero logs of this type. This only occured when we moved off of Bro standard log format to JSON format. Do you have the Splunk installed? ([https://splunkbase.splunk.com/app/1617/](https://splunkbase.splunk.com/app/1617/)) The TA will dynamically create sourcetypes based on the log name. # Dynamic source typing based on log filename # Match: conn.log, bro.conn.log, # md5.bro.conn.log, whatever.conn.log [BroAutoType] DEST\_KEY = MetaData:Sourcetype SOURCE\_KEY = MetaData:Source REGEX = ([a-zA-Z0-9-]+)(?:.[0-9-]_)?(?:.[0-9:-]_)?.log FORMAT = sourcetype::bro\_$1 WRITE\_META = true ---
**Author:** ![Gross\_Brett](https://avatars.discourse-cdn.com/v4/letter/g/a698b9/32.png) [@Gross\_Brett](https://community.zeek.org/u/Gross_Brett)\ **Post date:** [July 14, 2016, 3:33pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/4 "2016-07-14T15:33:57Z")
We’ve used bro and splunk at our organization for a couple years now. We utilize the Splunk props and transforms configs to ingest the bro log in the format we want or with the additional attributes and aliases. **transforms.conf:** [remove\_hash\_comments] REGEX = ^#.\* DEST\_KEY = queue FORMAT = nullQueue [bro\_conn\_extractions] DELIMS = “\t” FIELDS = ts,uid,id.orig\_h,id.orig\_p,id.resp\_h,id.resp\_p,proto,service,duration,orig\_bytes,resp\_bytes,conn\_state,local\_orig,local\_resp,missed\_bytes,history,orig\_pkts,orig\_ip\_bytes,resp\_pkts,resp\_ip\_bytes,tunnel\_parents,orig\_cc,resp\_cc,sensorname **Props.conf:** [bro\_conn] REPORT-bro\_conn\_extract = bro\_conn\_extractions TRANSFORMS-sourcetype = remove\_hash\_comments SHOULD\_LINEMERGE = false TRUNCATE = 0 KV\_MODE = none MAX\_TIMESTAMP\_LOOKAHEAD = 20 TIME\_FORMAT = %s.%6N **Inputs.conf** : [monitor:///your\_log\_path/log/bro/conn.log] index = bro\_conn sourcetype = bro\_conn \_TCP\_ROUTING = primary\_indexers brett ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [July 14, 2016, 3:59pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/5 "2016-07-14T15:59:48Z")
Ah, that's for the tab delimited logs, not the json logs though. I actually did it that way for years, I even have a python program that helps you generate the config: [https://github.com/JustinAzoff/bro\_scripts/blob/2.0/generate\_splunk\_configs.py](https://github.com/JustinAzoff/bro_scripts/blob/2.0/generate_splunk_configs.py) But, I wouldn't use this method - the splunk TA app for bro is better. As far as I know the transforms/props method only does the field lookups at search time, not at index time like the TA app configures. Whenever the bro logs change and a column is added or removed, all those search time field lookups break. ---
**Author:** ![Brandon\_Lattin2](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@Brandon\_Lattin2](https://community.zeek.org/u/Brandon_Lattin2)\ **Post date:** [July 14, 2016, 4:06pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/6 "2016-07-14T16:06:43Z")
The Bro TA is assuming TSV extractions. The move to JSON probably is causing the Splunk auto-sourcetyper to do some funky things. [source::…bro._.log] SHOULD\_LINEMERGE = false TRUNCATE = 0 MAX\_TIMESTAMP\_LOOKAHEAD = 20 TIME\_FORMAT = %s.%6N TRANSFORMS-BroAutoType = BroAutoType, TrashComments INDEXED\_EXTRACTIONS = TSV FIELD\_HEADER\_REGEX = ^#fields\t(._) FIELD\_DELIMITER = \t FIELD\_QUOTE = \t ---
**Author:** ![Drew\_Dixon](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@Drew\_Dixon](https://community.zeek.org/u/Drew_Dixon)\ **Post date:** [July 14, 2016, 4:14pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/7 "2016-07-14T16:14:44Z")
Sorry hope I’m not hijacking- quick question very closely related to this…is the Splunk app for Bro that Brandon linked to here supposed to parse out all the various bro 2.4.1 log types’ fields correctly? In other words, is the latest version of the Splunk app fro Bro/TA supposed to work properly for parsing out Bro log fields with they way the log fields/columns etc. are now in Bro 2.4.1? I think the Splunk Add-on for Bro IDS was written for Bro 2.1 or 2.2…do changes that were made in subsequent versions of Bro such as 2.4.1 break the fields being parsed out in Splunk when using this Splunk Add-on for Bro/Bro TA in Splunkbase? Or does Splunk need to update the add-on to work properly with Bro 2.4.1? Thank you, -Drew ---
**Author:** ![philosnef](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@philosnef](https://community.zeek.org/u/philosnef)\ **Post date:** [July 14, 2016, 4:16pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/8 "2016-07-14T16:16:24Z")
The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices… The Bro TA is assuming TSV extractions. The move to JSON probably is causing the Splunk auto-sourcetyper to do some funky things. [source::…bro._.log] SHOULD\_LINEMERGE = false TRUNCATE = 0 MAX\_TIMESTAMP\_LOOKAHEAD = 20 TIME\_FORMAT = %s.%6N TRANSFORMS-BroAutoType = BroAutoType, TrashComments INDEXED\_EXTRACTIONS = TSV FIELD\_HEADER\_REGEX = ^#fields\t(._) FIELD\_DELIMITER = \t FIELD\_QUOTE = \t ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [July 14, 2016, 4:30pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/9 "2016-07-14T16:30:34Z")
Odd, I'd expect it to be about the same. The indexed data should be the same, and even though every json record includes the field names, they compress well. It's possible that the bro app indexing the fields individually is what makes the indexes larger... if you do something like     id\_resp\_p=6379 (or whatever the field shows up as for you) does that find the records immediately, or does it have to scan through all the data? without individual field indexes you would have to do something like     6379 id\_resp\_p=6379 and hope that speeds it up, if you're trying to do something like     id\_orig\_p=80 Then this will be pretty slow:     80 id\_orig\_p=80 ---
**Author:** ![Brandon\_Lattin2](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@Brandon\_Lattin2](https://community.zeek.org/u/Brandon_Lattin2)\ **Post date:** [July 14, 2016, 4:33pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/10 "2016-07-14T16:33:55Z")
Drew, It should work just fine, assuming TSV headers are present, as it keys off the headers for the extractions. ---
**Author:** ![Brandon\_Lattin2](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@Brandon\_Lattin2](https://community.zeek.org/u/Brandon_Lattin2)\ **Post date:** [July 14, 2016, 4:38pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/11 "2016-07-14T16:38:35Z")
I would assume your index volume (license usage) is significantly greater though? You’re right, the raw to indexed ratio is abysmal with TSV (we’re getting 0.29:1). ---
**Author:** ![philosnef](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@philosnef](https://community.zeek.org/u/philosnef)\ **Post date:** [July 14, 2016, 4:44pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/12 "2016-07-14T16:44:01Z")
It is the TS IDX files in Splunk that grow out of control when using the Bro TSV app. Hope this helps for anyone interested. > The problem with the Spunk app is that indexing is occuring at time of ingest. This causes the indices of the Bro data to grow extremely fast. Using json and not the Bro app means that the data is indexed by Splunk, resulting in far smaller indices on the splunk indexing servers. This is specifically why we moved away from TSV and to JSON, since it was nuking disk storage for those indices… Odd, I’d expect it to be about the same. The indexed data should be the same, and even though every json record includes the field names, they compress well. It’s possible that the bro app indexing the fields individually is what makes the indexes larger… if you do something like id\_resp\_p=6379 (or whatever the field shows up as for you) does that find the records immediately, or does it have to scan through all the data? without individual field indexes you would have to do something like 6379 id\_resp\_p=6379 and hope that speeds it up, if you’re trying to do something like id\_orig\_p=80 Then this will be pretty slow: 80 id\_orig\_p=80 ---
**Author:** ![Brandon\_Lattin2](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@Brandon\_Lattin2](https://community.zeek.org/u/Brandon_Lattin2)\ **Post date:** [July 14, 2016, 4:54pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/13 "2016-07-14T16:54:49Z")
I do wonder if it’s even faster having the pre-search-time extractions in the tsidx files. I suppose if you’re going for a specific IP, the bloom filters may help? I’ve been really hesitant to move to JSON, simply because of the added raw volume impact on licensing. Bro is already over 250GB/day for us using TSV files. ---
**Author:** ![philosnef](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@philosnef](https://community.zeek.org/u/philosnef)\ **Post date:** [July 14, 2016, 5:03pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/14 "2016-07-14T17:03:13Z")
Heh. We have a multi-TB license for splunk… Bro is one of the largest consumers of that license… I do wonder if it’s even faster having the pre-search-time extractions in the tsidx files. I suppose if you’re going for a specific IP, the bloom filters may help? I’ve been really hesitant to move to JSON, simply because of the added raw volume impact on licensing. Bro is already over 250GB/day for us using TSV files. ---
**Author:** ![Steve\_Brant](https://avatars.discourse-cdn.com/v4/letter/s/858c86/32.png) [@Steve\_Brant](https://community.zeek.org/u/Steve_Brant)\ **Post date:** [July 14, 2016, 11:07pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/15 "2016-07-14T23:07:41Z")
I’ve attached a modified version of the Splunk TA for bro, that accommodates bro logs in json format. Let me know if you have any problem with it. Thanks, Steve [TA-bro\_json.tar.gz](https://community.zeek.org/uploads/short-url/1Dgoe4lIvnN3OGYGX5TK3EBxQKG.gz) (4.14 KB) ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:43pm UTC](https://community.zeek.org/t/problem-ingesting-bro-json-logs-into-splunk/4242/16 "2022-05-06T15:43:50Z")