Get the license usage down in Splunk when indexing Bro logs

Zeek
1

We only do filtering on conn logs, as they are the heaviest (in our environment at least), before indexing it in Splunk.
Also, if you are ingesting files.log as well, then you can build some similar filters in props and transforms for the
mime-type you can ignore (like plain/text etc), that will also reduce some of the volume indexed by your Splunk cluster.
I do not know much about the cloud deployment, hence can’t comment on that.

Regards,
Fatema.

2

Fatema,

Trying to filter out on types in the the files.log as well sounds like a great idea.

We’re a bit more limited as to what we can do ourselves when it comes to cloud Splunk but I’m sure they’re more than happy to sell some PS time if need be. :wink:

Once again - many thanks for a very helpful suggestion.

Cheers, Mike

3

Hey Mike,

So just out of curiosity, I ran a quick search on the files.log for the past hour, to see

the top most logged mime-types, and here is the top 12 mime-types in the file:

2630139 application/pkix-cert
366285 text/plain
259828 -
258732 image/gif
175465 text/html
142375 image/jpeg
116151 application/xml
103263 text/json
70691 image/png
48208 application/ocsp-response
18720 application/ocsp-request
16267 application/javascript

The Splunk filter can be easily built to ignore or filter the logs with these mime-types (of-course, only if you don’t

want them in Splunk):

In profs.conf:
[bro_files_sourcetype]
TRANSFORMS-null= bro_files_setnull

In transforms.conf:
[brol_files_setnull]
REGEX = (application/pkix-cert|text/plain|image/gif|text/html|image/jpeg)
DEST_KEY = queue
FORMAT = nullQueue

You can add more mime-types in the above REGEX, to filter and send to null queue in Splunk.

I added top five, just to give you an idea of how it can be implemented.

Hope this helps. :slight_smile:

Thanks,

Fatema.

4

Fatema,

Many thanks yet again.

Just for your comparison - here’s the top 12 from our current files.log, which is about 50 minutes old and about to rotate out.

1271142 application/pkix-cert
182904 text/plain
54187 text/json
49992 application/xml
46694 text/html
30609 -
19430 image/png
9421 image/jpeg
4778 image/gif
3664 application/x-dosexec
3049 text/ini
1304 application/vnd.ms-cab-compressed

Getting rid of just the ‘application/pkix-cert’ would probably save us lots of license here so your tip is really good.

Cheers, Mike