PF_RING's greatest feature is the ability to do per-flow clustering so
that many sniffers can run in parallel on their own CPU and each
receive a portion of the traffic which has been load-balanced. Snort
and Suricata both support this, and we've come to rely on it as we
don't have access to hardware load balancing. Is there a way to do
this in Bro?
Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. I'll work on merging in our fixed branch today and I'll reply again once it's updated.
.Seth
Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. I'll work on merging in our fixed branch today and I'll reply again once it's updated.
What I'm confused about is where the docs refer to a "node." Can this
be just another instance of Bro, with all instances on the same
machine? I don't want to run Bro on separate machines, I want to run
a lot of Bros on the same machine, and I want each instance to see
only some of the flows.
I think it doesn't work like that 
It appears that PF_RINGs clustering doesn't use virtual interfaces for its clustering.
Bro currently uses libpcap exclusively for packet capture and AFAIK libpcap doesn't use PF_RING clustering.
I guess there are two possible ways to get this support to Bro:
a) directly: add a new PktSrc() that uses the PF_RING API directly and add the command line argument to specify the clustering information. BroControl will likely need to be updated as well. If somebody does that, than the same framework can probably be used for Endace's load-balancing packet capture as well.
b) use a modified libpcap: modify libpcap so that it uses PF_RING clusters and add some magic to the way libpcap parses interface names passed to it. Say that the monitoring interface it eth0, then libpcap could interpret an interface like "eth0::1" (*) as the first cluster stream from eth0 and setup PF_RING accordingly. You'd probably need a small external program that sets up the cluster then.
Although b) is a little hacky, it would also enable any libpcap based application to make use of PF_RING and it likely quicker to integrate with Bro and BroControl (since neither Bro nor BroControl need to be changed for that)
(*) I guess one could even encode even more information into this "virtual" interface name, like the cluster id: "eth0::CLUSTERID::STREAM"
cu
Gregor
I just got this branch merged so you'll have to do:
git pull
git submodule update --init --recursive
Also, there seems to be some problem with installing this update overtop of an old install. You will probably want to backup any config and scripts files you have and delete your entire installation directory.
.Seth
But will this work if I don't have virtual interfaces?
I think it doesn't work like that
It appears that PF_RINGs clustering doesn't use virtual interfaces for its clustering.
Arg. Well that's annoying.
a) directly: add a new PktSrc() that uses the PF_RING API directly and add the command line argument to specify the clustering information. BroControl will likely need to be updated as well. If somebody does that, than the same framework can probably be used for Endace's load-balancing packet capture as well.
I keep thinking that we should something along these lines eventually. I'd like to support the upcoming FreeBSD netmap API directly as well since it seems like there are some performance benefits to that over using their libpcap wrapper.
.Seth
The PF_RING API is incredibly simple. I bet you could get this
working in just a few hours, and that would be an enormous win for the
community.
Unless you are using myricom cards 
It hates those.
You can divide the work up by using a pcap filters and specifying a different aux script for each work node. I have set up one node to monitor a tagged 10 GigE VLAN that I use to peer with I2, NLR and the internet using a TAP. Each worker mentors 1/8 the traffic on the I2, NLR and Internet connections.
The second node monitors traffic from a mirrored 10 GigE interface. Each works monitors 1/4 the traffic.
I haven't see any drops out of bro.
I use the following filter:
q1of8.bro:
redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 1" };
q2of8.bro:
redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 2" };
..
q8of8.bro:
redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 7" };
Here the node file:
Heh, the hack rises it's head again (including command line stuff in the 'interface' option). We really need to get multiple interface support in broctl soon.
I do actually have a question though, why have you included -C? Using the -C flag when running in production on live traffic is actually pretty bad to do since you leave yourself open to trivial evasion. You should be seeing valid checksums anywhere you'd be sniffing anyway, the checksum problem only happens when a host offloads the checksum calculation to the NIC and you're sniffing traffic on your local machine.
Let me know when you are ready to upgrade to the next release when we get it out too because there is an easier way to do your per-node script. I *think* that what you're currently doing should continue to work though. Thanks for documenting the filter trick for the list.
.Seth
Good point about the -C. I set it because I was getting checksum errors.
I now know that the source was due to the small snap len that bro uses in combination with a standard optimization in the new 10 GiGE card that will present several continues packets as one to pcap. So 10 1500 packets could turn into one 15000 buy packet.
I patched bro server weeks ago. I forgot remove the -C parameter.
I just removed on my bro cluster are it not complaining about checksum.
I don't mind fixing my configs if you give me a more standard way to do something. When the next version coming out?
I now know that the source was due to the small snap len that bro uses in combination with a standard optimization in the new 10 GiGE card that will present several continues packets as one to pcap. So 10 1500 packets could turn into one 15000 buy packet.
We'll be integrating the permanent-ish fix for this soon. The default snaplen is changing to 65535 along with having a -s option in case you need to shorten the snaplen. I think a lot of people have been running into that packet merging thing lately.
I don't mind fixing my configs if you give me a more standard way to do something. When the next version coming out?
Our aim right now is a beta next month.
.Seth
Ah, very cool! This may work for me--I will have to play around with
it and learn the cluster setup.
Vlan interfaces often have wrong checksums on them, depending on the
setup, so you may initially have needed that and then changed
something.