Using native PF_RING plugin with broctl

Zeek
1

Hi all,

I am trying to use Bro's PF_RING plugin with broctl, using a simple bro
cluster on a single host.

Here is an extract of my 'node.cfg' file:

[worker]
type=worker
host=localhost
interface=pf_ring::eth0
lb_method=pf_ring
lb_procs=8
pin_cpus=0,1,2,3,4,5,6,7

When I used the deploy command, I got the following error: "fatal error:
type of packet source 'pf_ring' no recognized, or mode not supported"

Here is the output of the deploy command:

[BroControl] > deploy
...
starting ...
starting manager ...
starting proxy ...
starting worker-1
...
starting worker-8
worker-1 terminated immediately after starting; check output with "diag"
...
worker-8 terminated immediately after starting; check output with "diag"

And when running "diag":

[BroControl] > diag

==== stderr.log
fatal error: type of packet source 'pf_ring' no recognized, or mode not
supported

However I do not have any problem running bro as a standalone process
with local commands such as:

$/usr/local/bro/bin/bro -i pf_ring::eth0
listening on eth0

and:

$/usr/local/bro/bin/bro -N | grep PF
Bro::PF_RING - Packet acquisition via PF_RING (dynamic, version 1.0)

This tends to prove Bro plugin has been installed and works.

I think Broctl is launching Bro binary without the right settings for
the plugin to be found/to work correctly. Am I missing something with
configuration files ?
May be the environment variables aren't properly set?

Does anyone use bro's PF_RING plugin with a cluster configuration
without issues?

Thanks,

Rémi

2

I think you just need “interface=eth0”. It knows to use pf_ring because of the next line.

3

When I set 'interface=eth0' and 'lb_method=pf_ring', the broctl deploy
command works, bro starts, but PF_RING is not used. Indeed, all workers
receive the same packets (i.e no load-balancing is performed).

When I cat the file /proc/net/pf_ring/info, the total number of rings
used is 0. Moreover, when I put a breakpoint within Source.cc:192
(PcapSource::ExtractNextPacket), I can see the call to the libpcap
function pcap_next(), which should never be called.

4

How did you install bro? Do you have more than one version of bro installed?

What environment variables are you referring to?

5

I installed bro using the following commands:

./configure --prefix=/usr/local/bro/
make
sudo make install

Then, bro is started using broctl install, followed by broctl start.

I have only one version of bro installed, the stable version 2.5
(Released Nov 16, 2016).

I was referring to the environment variables such as PATH, BROPATH or
CLUSTER_NODE, contained in ${PREFIX}/spool/worker-X/.env_vars, and
generated by the wrapper script ${PREFIX}/share/broctl/scripts/run-bro.

I forgot to mention that without using the native PF_RING plugin, I am
able to use PF_RING with the dedicated libpcap, such as explained here:
https://www.bro.org/sphinx/configuration/index.html#pf-ring-cluster-configuration.

Therefore, I don't think that this issue is related to the pf_ring
network driver or something like that, but rather to bro or broctl that
does not set the right configuration to enable the plugin.

Rémi

6

When you built Bro did you also configure/make/make install the pf_ring plugin? My recollection is that the plugins are not automatically built when you build bro. They should be in the <path-to-bro-source>/aux/plugins/ in the source tree. They typically install into <path-to-bro>/lib/bro/plugins/.

~Gary

7

There is a glitch that Justin and I worked through a few weeks ago with the PF_RING (native) support in Bro 2.5. Try adding these two items to your /opt/bro/etc/broctl.conf file:

PFRINGClusterID = 21
PFRINGClusterType = 6-tuple

The issue is there is some broken login in a file that defaults the ClusterID to zero (0) and at the same time that doesn’t split the packets out of the PF_RING interface as it should, which results in all workers getting all packets (as you described). So long as the PFRINGClusterID is anything aside from zero (0), I believe it will fix the issue. I add the ClusterType just as a safety net to ensure you get decent distribution of packets. Other modes (5-tuple, 4-tuple or 2-tuple) should also be valid. FYI - the 6th tuple is VLAN, so it may be more benificial to use 5-tuple in some environments.

If those items are in your broctl.conf file, then I’m a bit lost, but I’ve been fighting with PF_RING over the past few weeks and this has allowed for a repeatable process for myself.

This is also assuming you use in the node.cfg:
interface = eth0
lb_method = pf_ring

Additionally, the PF_RING module (new in 2.5) I believe suffers from the same glitch or possibly another one. I know roughly how to fix it, but have time to validate and push back upstream.

Mark

8

Are there people out there that are using the pf_ring plugin to successfully load balance traffic? I just checked the source to that plugin and I don't see where it sets up a load balanced ring. (I haven't worked on this plugin at all)

  .Seth

9

###pfring
git clone https://github.com/ntop/PF_RING.git

cd PF_RING/kernel
make
sudo make install

cd PF_RING/userland/lib
./configure --prefix=/opt/pfring
make
sudo make install

cd ../libpcap
./configure --prefix=/opt/pfring
make
sudo make install

cd ../tcpdump
./configure --prefix=/opt/pfring
make
sudo make install

modprobe pf_ring enable_tx_capture=1 min_num_slots=32768

10

I can see from this thread that a number of people think they are using the plugin, but are not actually using it.

interface = eth0 # pf_ring libpcap wrapper
interface = pf_ring::eth0 # native bro pf_ring plugin

11

I believe that's because I got the info from here:

https://www.bro.org/documentation/load-balancing.html

Maybe add info from https://www.bro.org/sphinx/components/bro-plugins/pf_ring/README.html to the above? I will try out the native on next device reboot.

James

12

When you built Bro did you also configure/make/make install the pf_ring
plugin? My recollection is that the plugins are not automatically built
when you build bro. They should be in the
<path-to-bro-source>/aux/plugins/ in the source tree. They typically
install into <path-to-bro>/lib/bro/plugins/.

Are there people out there that are using the pf_ring plugin to successfully load balance traffic? I just checked the source to that plugin and I don't see where it sets up a load balanced ring. (I haven't worked on this plugin at all)

I can see from this thread that a number of people think they are using the plugin, but are not actually using it.

interface = eth0 # pf_ring libpcap wrapper
interface = pf_ring::eth0 # native bro pf_ring plugin

Indeed, this is what I try to underline with this thread, I also believe
there is a glitch with the native PF_RING plugin.

I think that the example pointed by James Lay is using PF_RING through
the libpcap, but NOT with the native plugin. It can be proved by
breaking within Source.cc:192, the PcapSource::ExtractNextPacket() and
the underlaying pcap_next() function, should never be called, if the
plugin is properly used.

I would suggest using a libpcap compiled without PF_RING support, to
avoid confusion. This is actually how I test the plugin.

13

I vote for updated documentation for the new plugin. What James posted is how I would set it up, which would use pf_ring enabled libpcap.