I hate to ask about dropped packets again, as I feel it is a re-occurring question… but I can’t seem to figure out if this is a configuration issue or just not enough hardware. I see people asking about massive pipes, but I am not looking at a very large pipe here. using broctl’s capstats, I show:
kpps .3
mbps .6
The node that is dropping is a physical machine that doesn’t appear to have issues with memory or CPU. There are 2 bro processes in top, and both generally are at or under 20% CPU utilization. The box keeps a load average around .30, which generally would not make me believe it was over taxed.
An example notice:
1393944936.832292 - - - - - - - - - PacketFilter::Dropped_Packets 11 packets dropped after filtering, 207913 received, 207913 on link - - - - - ids-1 Notice::ACTION_LOG 3600.000000 F - - - - -
I was having issues with missed bytes, but that was resolved by turning off offload settings on my nic. I rarely see any missed bytes now.
I have tried adjusting some settings, but I saw no improvements.
If it is of any help, I notice that these drops seem to happen 5 minute
intervals. When it is happening a lot, they are 5 minutes apart, but
sometimes it is 10 minutes, or 25. Always an increment of 5 minutes though.
Is there something in the cron jobs that could case me to have drops?
PacketFilter::stats_collection_interval is 5 mins by default. You're seeing it reported every 5 minutes because that's the reporting interval.
If you look into the percentage of traffic you're seeing reported as lost, it's actually 0.005% which isn't really that bad. Granted, it doesn't explain *why* you had a few packets reported as lost but in the grand scheme of things it's really not that bad.
Well, that explains the interval. The example I posted was one of the smallest loss amounts, but what is an acceptable or expected loss level? In the last 6 hours my setup does have a 1448 out of 206031, or .7%. So .005% seems small, is .7% small?
I am running the default local.bro, and it does have misc/scan loaded. I turn that off and see if I still see loss, but if 1% or less is considered normal loss even for a low load and small traffic, then I guess it is less important.
Well, that explains the interval. The example I posted was one of the smallest loss amounts, but what is an acceptable or expected loss level? In the last 6 hours my setup does have a 1448 out of 206031, or .7%. So .005% seems small, is .7% small?
For most people anything under 1-2% is pretty acceptable. You could probably work to get rid of that last bit of loss (which I have no clue why that's happening on such a small link) but it's probably more worth your time to start using Bro to achieve some goals within your organization rather than make sure you aren't dropping any packets at all.
I am running the default local.bro, and it does have misc/scan loaded. I turn that off and see if I still see loss, but if 1% or less is considered normal loss even for a low load and small traffic, then I guess it is less important.
Are you running in standalone mode? I guess I forgot to ask that, but it could be significant.
This is running in a cluster setup. The nodes that are dropping are actually physical machines, while I had a couple other nodes as VM’s (which see a tiny amount of traffic). The VM’s haven’t dropped anything, but with the small traffic they see that didn’t surprise me. I was just surprised when I keep getting notice log entries for the physical machines dropping, when I expected them to be able to handle a much bigger load.
I think I will take your advice and focus more getting some use out of my setup. I didn’t expect loss when the box doesn’t seem loaded, but if less than 1-2% loss isn’t going to negatively impact my usage, I probably don’t need to spend time trying to tweak it away.
Mine will vary between sub-1-percent and up to about 9%, for Reasons I’ll clarify in another list mail (once I get things sorted out so I’m back to under 1% loss). I’m not *pleased* about anything over 2-3%, but I can accept that. Anything under 1 and I’m happy; it’s very likely noise, given we’re a large network and see a ton of weird traffic that I don’t expect Bro to be able to fully account for.
For reference, I’ve got 8 workers going on an overloaded box that sees a floor of 350Mbps and a ceiling of about 3.5Gbps (although I’ve seen sustained peaks up to 6 and short peaks up to 12).
