Zeek + PF_Ring Issue

Zeek
1

Hi Zeekers!

I need to resolve a problem attached to Zeek when its configured to work with PF_Ring.

The thing is that we receive between 1.0 and 2.5 GB/s in a fiber interface. Also when we lauch the command “Zeekctl top” to check the Cpu usage and the traffic managed in each worker, we see that the sum of the traffic of all workers is greater than the traffic we receive through the interface.

This makes me think that we have something badly configured in PF_Ring or somehow Zeek is generating some kind of loop.

For example, receiving 2Gb/s, i execute “Zeekctl top” and the result is the next one:

Name Type Host Pid VSize Rss Cpu Cmd

logger logger localhost 11474 3G 118M 50% zeek

manager manager localhost 11520 589M 98M 25% zeek

proxy-1 proxy localhost 11565 610M 113M 18% zeek

worker-1-1 worker localhost 11693 1G 570M 62% zeek

worker-1-2 worker localhost 11701 1G 574M 62% zeek

worker-1-3 worker localhost 11711 1G 573M 68% zeek

worker-1-4 worker localhost 11713 1G 572M 50% zeek

worker-1-5 worker localhost 11718 3G 2G 106% zeek

worker-1-6 worker localhost 11719 1G 567M 62% zeek

worker-1-7 worker localhost 11726 1G 579M 68% zeek

worker-1-8 worker localhost 11732 1G 575M 56% zeek

worker-1-9 worker localhost 11733 1G 571M 68% zeek

worker-1-10 worker localhost 11735 1G 558M 62% zeek

Hope someone of you can help me to resolve this.

Really thank you.

Best Regards!

2

Jorge,

Have you checked for duplicate events in Zeek? I recall when I set up Zeek with PF_RING, I followed the instructions at https://www.zeek.org/documentation/load-balancing.html and only followed the instructions through the “Using PF_RING” paragraph. In my case I was pinning to four CPUs, and what I found was that I was getting four copies of the all sniffed network traffic onto my Zeek environment, one going to each worker. The symptom that tipped me off is that I would see was four “conn” events for a given connection, each with all the same source/dest/byte counts/etc. but each had a different UID. I suspect that if I continued on to additional paragraphs I would have been able to get past this problem (note how in the paragraph “Using PF_RING+DNA with symmetric RSS” it says “You can sniff each packet only once”… don’t we always want that? :slight_smile: ) Alas, I’m not 100% sure of the solution as I started using a different Zeek approach instead. Hope it helps though.

3

I'm not certain if it's the exact root cause, but does the advice on
PCAP_PF_RING_CLUSTER_ID at
https://www.ntop.org/guides/pf_ring/thirdparty/bro.html apply?

...Bro needs to setup a pf_ring kernel cluster in order to split the traffic across the processes (otherwise your get duplicated data).

- Darren

4

Can you run bro-doctor: https://packages.bro.org/packages/view/1251f948-f435-11e9-9321-0a645a3f3086 (works with zeek, just didn’t change the name). that will likely tell you what is wrong. You’re probably not actually using pf_ring and should use the native plugin and not the pcap wrapper.

5

I have ran bro-doctor as you said and certainly I saw interesting things, for example:

6

I have ran bro-doctor as you said and certainly I saw interesting things, for example:

###################################################################

Checking if connections are unevenly distributed across workers

###################################################################

error: The distribution of connections across workers seems uneven:

worker-1-5: 462 connections

worker-1-4: 890 connections

worker-1-7: 874 connections

worker-1-6: 4122 connections

worker-1-1: 432 connections

worker-1-3: 930 connections

worker-1-2: 907 connections

worker-1-9: 451 connections

worker-1-8: 435 connections

worker-1-10: 497 connections

Interesting indeed. If you look at your conn log can you tell anything about all those connections that worker-1-6 is seeing?

Let me know what do you think about the report.

I have checked about the PF_Ring plugin but it gives me an error, im not sure if im following the last update of this plugin.
https://github.com/ntop/bro-pf_ring

you should be able to zkg install bro-pf_ring… or install it manually with ./configure && make && sudo make install. are you getting an error when you do that?

Also doing a further investigation it seems that the script that is overcharguing the cpu is the weird.zeek ¿Is there a way to disable this script?

Do you say that because you have a lot of entries in the weird log? that points to traffic issues that need to be fixed… disabling the weird logs will just ignore the problem. What are the top weirds that you are seeing?

cat /usr/local/zeek/logs/current/weird.log |zeek-cut name|sort|uniq -c|sort -rn

What did you see as the result from this check?

Checking if many recent connections have a SAD or had history

7

Enviado desde Outlook<http://aka.ms/weboutlook>