Hi everyone,
We would like to deploy a Bro Cluster at a 10 Gbps at about 35% peak usage.
We already have a splitter in place and are discussing options for a
front-end that can merge both traffic directions and load balance sessions
to Bro workers based on session hash and MAC rewriting. Ideally we would
like some equipment that supports multi-port mirroring so that we can add
other monitoring tools in addition to the Bro Cluster (e.g., Snort,
TimeMachine or other Storage).
Robin mentioned to me that people are using Arista and CPacket switches for
this kind of setup. After looking at their webpages the Arista 7150 seems
like a possibility for us (I see on the web page the San Diego SDSC and
Cornell use the larger 7500 series) and CPacket's cVu240NG may be another
(although there is less information about CPacket products online).
Does anyone have experience with these products? Do those models make sense
for the description above?
Any recommendations or things to consider for people without prior
experience in such setups?
Thanks!
Juan Caballero
Assistant Research Professor
IMDEA Software Institute
Madrid, Spain
(about a 10 gig cluster expecting 3-4 gig peaks, wanting a way to more easily manage the distribution of traffic to monitoring systems, asking specifically about Arista and CPacket)
Does anyone have experience with these products? Do those models make sense
for the description above?
We’ve been using an Arista 7150S for exactly this purpose, and our requirements are pretty similar (10 gig links, expected peaks are 4-4.5 gig, although we’ve seen 6-7). I can’t speak to CPacket. But yes, the Arista will fit exactly into what you want and pricing on them is pretty good.
We don’t do load balancing on the switch though, we’re doing it on the NICs. I have a machine with an Endace DAG (older box, pre-Arista) and several with Intel x520 NICs using PF_RING. There should be several others on the list feeding Aristas to Intels, although I’ll let them speak about their own experiences. I’m finding I get better performance out of the DAG box, although not 10,000 dollars better performance (which is about the price gap).
Any recommendations or things to consider for people without prior
experience in such setups?
Whatever you pick, figure out a way to integrate monitoring it into your environment, or accept that you’ll get to monitor it yourself. And unless your NOC is heavily invested in these things too, chances are it will be the latter, so you’ll get to play sysadmin too. 
Mike
We are using Arista 7150S’s but are in the process up upgrading to 7280SE’s. The 7280SE’s are their next-gen platform and a much needed feature request that we have (MPLS label popping) is roadmap for the 7280SE They will be evaluating it for there 7150S but no commitment yet). cPacket already has MPLS label popping but is considerably more expensive. We’re happy enough with the Arista’s to stay that route.
Hi Juan,
We use both the cPacket (cVue 240) and Arista (7150s) and both are quite capable of handling the traffic you suggest. In our older setups we use a custom cPacket device to do MAC re-writing from 10G input to 1G Bro worker nodes. As Mike mentioned, load-balancing traffic to workers on a multi-core box with specialized NIC driver is a more common and often more cost effective configuration these days. We’re currently ramping up our 100G Bro cluster with a combination of Arista hardware and collection of Myricom 10G workers on FreeBSD. I would suggest that you use the device you choose to aggregate, filter and distribute your traffic to the different tools and then experiment with running a Bro cluster on a single box. I think with the traffic volumes you mention you should be able to monitor everything with a single 10G card and multiple worker threads.
One thing not to forget is that you’ll need 1 port for each direction of “input” traffic on these devices to monitor full duplex taps, so make sure you take that into account when counting ports. The cVue is a very nice piece of hardware with great flexibility, however, the cost is not comparable with the Arista. The Arista feature set is quite good and they have been receptive to our feature requests. We’re also very excited to be using Arista’s API which lets us do dynamic shunting based on feedback from Bro. If you have specific questions, let me know and I’d be happy to answer them.
Thank you,
Vince
Hi Vincent,
Thanks a lot for your feedback. Indeed, we plan using a multi-core machine with one Bro worker per core ( plus 1-2 cores for other stuff) and distribute traffic to them either using an Endance card (already available), Myricom cards, or PF-RING. I wasn’t sure if our current machine would be enough that is why I was thinking to support multiple machines, but starting with a single machine sounds like a great idea. From the answers to my questions the Arista may be a cost-effective option for an initial deployment.
Juan
We don’t do load balancing on the switch though, we’re doing it on the NICs.
It seems that load-balancing on the NICs is the preferred approach by different groups, so we will definitely give that a try. Thanks!
We may try an Endace DAG that we have around, but would like to experiment with Intel x520+PF_RING as well
figure out a way to integrate monitoring it into your environment, or accept that you’ll get to monitor it yourself.
Right, thanks for the advice. I am counting we will have to do some sys admin work ourselves at least at the start
Thanks!
Juan