Bro drop packets while not using CPU at full capacity

Zeek
1

Hi All
I use Bro for my PhD research, I add scripts in Bro and then see the CPU and packet drop rate, using @load stats.bro. I’m using Bro 2.3 with standard libcap.
I use tcpreplay from Machine A to replay the pre-captured traffic into Bro multi-core machine B through port mirror switch. I replay the traffic from 100 to 1000 Mbps , When reach 200 Mbps and onward, packet start drop and increases. Surprisingly, the CPU is not fully utilized, CPU still 40% usage. What we know is that drop packet resulted from CPU full load, but in our case CPU still less than 50%, so My question, what is the cause of this packet drop? Is it normal?

Best regards
Aidaros

2

Is it possible that the CPU has two cores and Bro is consuming 100% of one core? Some tools average the core utilization to report "CPU usage".

3

Thanks Dave,
I couldn’t get what you mean. How stats.bro calculate CPU usage, is it per core utilization? My bro machine is quad-core with hypertheading enabled, means 8 logical cores. So, if one core is fully utilized then stats should report 12.5% (100/8), not 40% or 60% as in my case. How my Bro report 60% CPU with 20% drop packet rate reported? Is there any reason that make packet drop?
Anyone could clarify please.

4

Is there any reason you aren’t using 2.4.x? Step one would be to use that I would think. 2.4.x fixed a great many bugs I believe.

Cheers,

JB



**From:**aidaros.dev@gmail.com
**Sent:**June 25, 2016 7:15 AM
**To:**bro@pingtrip.com
**Cc:**bro@bro.org
**Subject:**Re: [Bro] Bro drop packets while not using CPU at full capacity

|

  • |

Thanks Dave,
I couldn’t get what you mean. How stats.bro calculate CPU usage, is it per core utilization? My bro machine is quad-core with hypertheading enabled, means 8 logical cores. So, if one core is fully utilized then stats should report 12.5% (100/8), not 40% or 60% as in my case. How my Bro report 60% CPU with 20% drop packet rate reported? Is there any reason that make packet drop?
Anyone could clarify please.

5

I started my experiments when Bro 2.3 was the latest stable version. All my results are based on 2.3, I can not shift to newer version now.
Anyone can clarify why Packet are dropping while no fully CPU utilization.?

Best regards

6

I’ve been troubleshooting my clusters recently. I’m seeing some drops in the kernel using drop watch. Previously I’ve seen loss from spans when approaching link saturation

7

Thanks Neslog and Slawek, for reply.

In my experiments, I do not use cluster. My main question now, what is the cause of drop packets reported from the following stats.bro

info$pkts_dropped = ns$pkts_dropped - last_ns$pkts_dropped;

Is it packet drop by: 1) Bro, 2)libpcap 3)kernel OS ?

Regards

8

Hi,

I had such a problem before, there was no cpu or other problem but bro was dropping the packets. It was saying packets are larger than expected MTU, so it was dropping then. It turned out that some NICs offload the reassembly of traffic into bigger packets so that fewer packets are passed up to stack. Actually the intention is to reduce burden on CPU, but it also causes bro to drop packets. So it may be a similar problem in your case. I solved the problem changing interface settings, you can try the following settings. I hope this will solve your problem.

ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 ufo off
ethtool -K eth0 gso off
ethtool -K eth0 gro off
ethtool -K eth0 lro off

Best regards,

Mehmet Leblebici

27 Haziran 2016 Pazartesi tarihinde, Neslog <neslog@gmail.com> yazdı: