High drop rates on recent builds

On recent builds from the master branch, I'm seeing anomalously high drop rates.

From notice.log

458745 packets dropped after filtering, 747736 received, 288991 on link
262140 packets dropped after filtering, 581026 received, 318886 on link
524280 packets dropped after filtering, 826789 received, 302509 on link

If I flip some of these values around to make sense (so that dropped+received=onlink), I'm still left with really bad capture rates. I'm running on DAG cards, and I can confirm that they, too, think that I'm madly dropping packets on the floor, even though CPU utilization for the Bro processes is hovering around 3%.

By reverting back to the 2.3.1 tag, these problems go away.

Any thoughts? Is anyone else seeing this?

v/r

John Donaldson


Hm, I've actually heard a similar story from one or two other people. I'm not sure where to start debugging this one though. Fortunately the diffs between 2.3 and 2.3.1 are pretty minimal so it might take someone digging through those by hand to see if there is something that changed in there that could impact some people.

  .Seth

Any idea at what time this started?

Two candidates I can think of:

commit e9692958f05fb17bc946a04a78313cc5dd0922de

Robin,

It looks like this problem existed prior to or on 22 September, which was when I most recently rebuilt, and, looking back, started seeing the odd rates. I know that I rebuilt on the 9th, as well, because of the (since-resolved) issues with 3caecad265438b and specifying DAG streams. I'm stepping through commits now.

John Donaldson

If you’re able to test and provide feedback on latest master again, 9cd85be308 fixes a problem that caused the main loop to spin more frequently than it used to.

- Jon

No success. It jumps right back up to around 50% loss.

John Donaldson