So I’m logging my Bro in JSON format on my manager node. I have Splunk ingesting the log files through the Splunk TA from Github : https://github.com/jahshuah/splunk-ta-bro-json
Everything is working fine except I’m only getting sporadic http.log entries. Looking in the Splunk logs, it appears that the http.log file is large enough that Splunk isn’t finished indexing it, when it gets rotated/compressed out and the new 1/2 hour files starts to fill.
Splunk doesn’t seem to do any file locking(a good thing), but the file goes away before its finished with it. The machine seems to have plenty of resources, and I’ve turned off the index thruput limit on the splunk heavy forwarder. So I’m not sure if I can make Splunk go any faster.
Are there any bro settings that would help here? I thought about rotating the logs more frequently but if volume is the issue that won’t really help. Is there a way to have bro not compress/remove the file immediately?
Or anyone tackled this problem and found a different/splunk solution?
Jeffrey Collyer
Information Security Engineer
University of Virginia
434-297-6317