We only do filtering on conn logs, as they are the heaviest (in our environment at least), before indexing it in Splunk.
Also, if you are ingesting files.log as well, then you can build some similar filters in props and transforms for the
mime-type you can ignore (like plain/text etc), that will also reduce some of the volume indexed by your Splunk cluster.
I do not know much about the cloud deployment, hence can’t comment on that.
Trying to filter out on types in the the files.log as well sounds like a great idea.
We’re a bit more limited as to what we can do ourselves when it comes to cloud Splunk but I’m sure they’re more than happy to sell some PS time if need be.
Once again - many thanks for a very helpful suggestion.
So just out of curiosity, I ran a quick search on the files.log for the past hour, to see
the top most logged mime-types, and here is the top 12 mime-types in the file:
The Splunk filter can be easily built to ignore or filter the logs with these mime-types (of-course, only if you don’t
want them in Splunk):
REGEX = (application/pkix-cert|text/plain|image/gif|text/html|image/jpeg)
DEST_KEY = queue
FORMAT = nullQueue
You can add more mime-types in the above REGEX, to filter and send to null queue in Splunk.
I added top five, just to give you an idea of how it can be implemented.
Hope this helps.
Many thanks yet again.
Just for your comparison - here’s the top 12 from our current files.log, which is about 50 minutes old and about to rotate out.
Getting rid of just the ‘application/pkix-cert’ would probably save us lots of license here so your tip is really good.