Mike,
How much data are we talking about? Have you done the analysis to see what logs are actually causing you problems? I am currently ingesting somewhere in the neighborhood of 50GB of bro logs a day but at one point it was a lot more. After doing some digging we found out that our sensor was saturated and dropping a ton of packets which had the bro wierd log and conn log going through the roof because it some connections where appearing 3 or 4 times due to it thinking they were different connections.
Troy
Troy,
Many thanks for your reply.
For this one host we’re looking at around 170 - 190GB of data per day according to Splunk. Which feels like a lot, but could also be the ‘true’ value for a host with our traffic and all logs turned on if you see what I mean?
So far we’ve not looked at duplicates in the logs - and I’ll have to check the size of the weird.log when I get in to work - so that’s definitely worth looking at.
If you don’t mind me asking, if we find that we have a similar issue like yours with dropped packets etc - what steps did you go through in order to work on this issue? Spread the load out on more cores? Lessen the amount of scripts run? All of the above, none of the above or something completely different?
Cheers, Mike