BRO with PF_Ring multiplies log records

Zeek
1

Hello BRO community,

Need a bit of a help with pf_ring load balancing

I have the following setup:

  • One server with one 10G interface
  • bro 2.5.4 compiled and installed with pf_ring support
  • latest pf_ring installed
  • I want multiple bro processes to capture the traffic with pf_ring balancing
    node.cfg is below:

[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy-1]
type=proxy
host=localhost

[worker-1]
type=worker
host=localhost
interface=pf_ring::p2p2
lb_method=pf_ring
lb_procs=4

Now when I run such config on the test traffic, it looks like each bro process receives all the traffic, with no flow balancing between processes. For example, each http request in my pcap file is recorded 4 times in http.log (and I run 4 processes). When I put lb_procs=1 everything is fine again.

What am I doing wrong with this balancing?

2

From of one of Justin’s posts a while back (as I have struggled with this numerous times) - this may or may not be the issue, but putting it out there if it is as it has the same symptoms.

[root@bro-dev ~]# broctl config | grep pfring
pfringclusterid = 21
pfringclustertype = 4-tuple
ringfirstappinstance = 0

if you have pfringclusterid set to 0, that’s the problem that was just fixed. You can easily workaround that by adding

PFRINGClusterID = 21

to your /usr/local/bro/etc/broctl.cfg

Mark

3

Mark, thanks for your response

I’ve tried setting PFRINGClusterID as you advised. Unfortunately that hasn’t changed anything - I still see that each event is multiplied by number of processes (x4 in my case). So each flow hits all the processes, with no actual balancing between them.

Anything else I could check?

4

Can you confirm that

    broctl config | grep pfring

outputs something like

pfringclusterid = 21
pfringclustertype = 4-tuple

There was a problem with this configuration the bro-pf_ring plugin, but I got that fixed last September.
Did you install the plugin recently?

5

If you have a p2p2 interface, does that mean that card is a Myricom card? Have you loaded any of the SNF (Sniffer 10g+ drivers up)? I’m very familiar with those, but I’ve observed that to be the interface on occaisions. If so, I’m not sure how PF_RING, Myrcom/SNF and Bro play together.

Mark

6

Mark,

The card is Intel X520 10G. Standard ixgbe drivers are loaded

Justin, I confirm that pfringclusterid setting correctly applied:

/opt/bro/bin/broctl config | grep pfring

pfringclusterid = 21
pfringclustertype = 4-tuple
pfringfirstappinstance = 0

Still same problem. Will try bro+pf_ring on another server, will see if the problem will reproduce itself

7

Yasha,

You’ve probably already checked but are Bro and pf_ring communicating with each other? If you execute the following command (while Bro is running)…

cat /proc/net/pf_ring/info

…what is the value in the “Total rings” field? My understanding is this is the number of processes utilizing pf_ring, it should be 4 if you have 4 Bro processes, I believe. We had the pfringclusterid=0 issue and that is how we found it. Pf_ring was loaded but the total rings were 0, even though Bro was running and should have been utilizing pf_ring.

Output of cat /proc/net/pf_ring/info on our Bro host:

PF_RING Version : 7.2.0 ( 7.2.0-stable:1fb19fb55a8c6e6525899c938a9f90be9dac7a64)

Total rings : 30

Standard (non ZC) Options

Ring slots : 32768

Slot version : 17

Capture TX : No [RX only]

IP Defragment : No

Socket Mode : Standard

Cluster Fragment Queue : 52

Cluster Fragment Discard : 0

Sounds like your issue might be something else, but thought I’d throw this out there just in case. :slight_smile:

Thanks,

Carl