# Bro with 10Gb NIC's or higher **URL:** \ **Category:** Zeek\ **Created:** [January 7, 2015, 10:04am UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415 "2015-01-07T10:04:01Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Yahoo](https://avatars.discourse-cdn.com/v4/letter/y/898d66/32.png) [@Yahoo](https://community.zeek.org/u/Yahoo)\ **Post date:** [January 7, 2015, 10:04am UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/1 "2015-01-07T10:04:01Z")
Does anyone have experience with higher speed NIC’s and Bro? Will it sustain 10Gb speeds or more provide the hardware is spec’d appropriately? regards, Coen ---
**Author:** ![Mike\_Patterson](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@Mike\_Patterson](https://community.zeek.org/u/Mike_Patterson)\ **Post date:** [January 8, 2015, 3:28pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/2 "2015-01-08T15:28:55Z")
Succinctly, yes, although that provision is a big one. I'm running Bro on two 10 gig interfaces, an Intel X520 and an Endace DAG 9.2X2. Both perform reasonably well. Although my hardware is somewhat underspecced (Dell R710s of differing vintages), I still get tons of useful data. If your next question would be "how should I spec my hardware", that's quite difficult to answer because it depends on a lot. Get the hottest CPUs you can afford, with as many cores. If you're actually sustaining 10+Gb you'll probably want at least 20-30 cores. I'm sustaining 4.5Gb or so on 8 3.7Ghz cores, but Bro reports 10% or so loss. Note that some hardware configurations will limit the number of streams you can feed to Bro, eg my DAG can only produce 16 streams so even if I had it in a 24 core box, I'd only be making use of 2/3 of my CPU. Mike ---
**Author:** ![John\_Donnelly](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@John\_Donnelly](https://community.zeek.org/u/John_Donnelly)\ **Post date:** [January 8, 2015, 4:31pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/3 "2015-01-08T16:31:37Z")
How does one know if bro is dropping (10%) of messages ? ---
**Author:** ![Mike\_Patterson](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@Mike\_Patterson](https://community.zeek.org/u/Mike_Patterson)\ **Post date:** [January 8, 2015, 4:32pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/4 "2015-01-08T16:32:12Z")
capture\_loss log, not enabled by default. ---
**Author:** ![Brandon\_Lattin1](https://avatars.discourse-cdn.com/v4/letter/b/a587f6/32.png) [@Brandon\_Lattin1](https://community.zeek.org/u/Brandon_Lattin1)\ **Post date:** [January 8, 2015, 4:41pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/5 "2015-01-08T16:41:01Z")
Turn on the capture-loss script by adding the following to your local.bro: @load misc/capture-loss ---
**Author:** ![Donaldson\_John](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@Donaldson\_John](https://community.zeek.org/u/Donaldson_John)\ **Post date:** [January 9, 2015, 6:03pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/6 "2015-01-09T18:03:38Z")
I'd agree with all of this. We're monitoring a few 10Gbps network segments with DAG 9.2X2s, too. I'll add in that, when processing that much traffic on a single device, you'll definitely not want to skimp on memory. I'm not sure which configurations you're using that might be limiting you to 16 streams -- we're run with at least 24 streams, and (at least with the 9.2X2s) you should be able to work with up to 32 receive streams. v/r John Donaldson ---
**Author:** ![Mike\_Patterson](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@Mike\_Patterson](https://community.zeek.org/u/Mike_Patterson)\ **Post date:** [January 9, 2015, 6:20pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/7 "2015-01-09T18:20:17Z")
You're right, it's 32 on mine. I posted some specs for my system a couple of years ago now, I think. 6-8GB per worker should give some headroom (my workers usually use about 5 apiece I think). Mike ---
**Author:** ![Aashish\_Sharma1](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Aashish\_Sharma1](https://community.zeek.org/u/Aashish_Sharma1)\ **Post date:** [January 9, 2015, 7:00pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/8 "2015-01-09T19:00:54Z")
While, we at LBNL continue to work towards a formal documentation, I think I'd reply then causing further delays: Here is the 100G cluster setup we've done: - 5 nodes running 10 workers + 1 proxy each on them - 100G split by arista to 5x10G - 10G on each node is further split my myricom to 10x1G/worker with shunting enabled !! Note: Scott Campbell did some very early work on the concept of shunting       ([http://dl.acm.org/citation.cfm?id=2195223.2195788](http://dl.acm.org/citation.cfm?id=2195223.2195788)) We are using react-framework to talk to arista written by Justin Azoff. With Shunting enabled cluster isn't even truly seeing 10G anymore. oh btw, Capture\_loss is a good policy to run for sure. With above setup we get ~ 0.xx % packet drops. (Depending on kind of traffic you are monitoring you may need a slightly different shunting logic) Here is hardware specs / node: - Motherboard-SM, X9DRi-F - Intel E5-2643V2 3.5GHz Ivy Bridge (2x6-=12 Cores) - 128GB DDRIII 1600MHz ECC/REG - (8x16GB Modules Installed) - 10G-PCIE2-8C2-2S+; Myricom 10G "Gen2" (5 GT/s) PCI Express NIC with two SFP+ - Myricom 10G-SR Modules On tapping side we have - Arista 7504 (gets fed 100G TX/RX + backup and other 10Gb links) - Arista 7150 (Symetric hashing via DANZ - splitting tcp sessions 1/link - 5 links to nodes on Bro side: 5 nodes accepting 5 links from 7150 Each node running 10 workers + 1 proxy Myricom spliting/load balancing to each worker on the node. Hope this helps, let us know if you have any further questions. Thanks, Aashish ---
**Author:** ![Mike\_Reeves](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Mike\_Reeves](https://community.zeek.org/u/Mike_Reeves)\ **Post date:** [January 9, 2015, 7:11pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/9 "2015-01-09T19:11:54Z")
In all of the 10G deployments I have done I always do multiple boxes behind a flow based load balancer. That way I can use commodity boxes without special NICs and keep them at a reasonable price point. The bang for the buck goes down when you talk 4 x 12 core HT processors etc. vs a dual 10 core HT. You also get the ability to have some fault tolerance where if you have hardware issues you are not blind. I have a few deployments that are going from 10G to 100G and the only thing we have to change is the inbound interfaces on the LB gear. The other positive is as usage goes up I can add additional capacity incrementally instead of having to re-solution. Thanks Mike ---
**Author:** ![John\_Donnelly](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@John\_Donnelly](https://community.zeek.org/u/John_Donnelly)\ **Post date:** [January 9, 2015, 7:31pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/10 "2015-01-09T19:31:16Z")
Hi, What is the name of the log and where is it located at ? ---
**Author:** ![Michal\_Purzynski1](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@Michal\_Purzynski1](https://community.zeek.org/u/Michal_Purzynski1)\ **Post date:** [January 9, 2015, 8:01pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/11 "2015-01-09T20:01:33Z")
Do you really see and can handle 1Gbit/sec of traffic per core? I'm curious. I would say, with a 2.6Ghz CPU my educated guess would be somewhere about 250Mbit/sec / core with Bro. Of course configuration is everything here, I'm just looking into "given you do it right, that's what's possible". ---
**Author:** ![Aashish\_Sharma1](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Aashish\_Sharma1](https://community.zeek.org/u/Aashish_Sharma1)\ **Post date:** [January 9, 2015, 8:26pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/12 "2015-01-09T20:26:01Z")
> Do you really see and can handle 1Gbit/sec of traffic per core? I’m curious. Haven’t measured if a core can handle 1Gbit/sec but I highly highly doubt. What saves us is the shunting capability - basically bro identifies and cuts off the rest of the big flows by placing a src,src port - dst, dst-port ACL on arista while continuing to allow control packets (and dynamically removes ACL once connection\_ends) So each core doesn’t really see anything more then 20-40 Mbps (approximation) (Notes for self, it would be good to get these numbers in a plot) Thanks, Aashish ---
**Author:** ![Oehlert\_Samuel](https://avatars.discourse-cdn.com/v4/letter/o/43a26b/32.png) [@Oehlert\_Samuel](https://community.zeek.org/u/Oehlert_Samuel)\ **Post date:** [January 9, 2015, 9:02pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/13 "2015-01-09T21:02:41Z")
Capture\_loss.log and it should be with all your other logs once you turn it on. Remember to install, check, and restart brocontrol to get it turned on. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [January 9, 2015, 9:29pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/14 "2015-01-09T21:29:58Z")
After you add the following to local.bro of course… @load misc/capture-loss   .Seth ---
**Author:** ![John\_Donnelly](https://avatars.discourse-cdn.com/v4/letter/j/3ec8ea/32.png) [@John\_Donnelly](https://community.zeek.org/u/John_Donnelly)\ **Post date:** [January 9, 2015, 9:37pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/15 "2015-01-09T21:37:46Z")
Ok - I found it it : / - along with “weird” How I can I specify another directory ? What do the fields mean ? root@x64-01:/# cat cap\* 1420832673.023244,900.000068,bro,0,0,0.0 1420833573.023279,900.000035,bro,0,6,0.0 1420833727.951157,154.927878,bro,0,0,0.0 1420833885.693988,154.676438,bro,0,0,0.0 ---
**Author:** ![Aubrey\_Wells](https://avatars.discourse-cdn.com/v4/letter/a/82dd89/32.png) [@Aubrey\_Wells](https://community.zeek.org/u/Aubrey_Wells)\ **Post date:** [January 9, 2015, 9:42pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/16 "2015-01-09T21:42:21Z")
My file has the headers in it: $ head -8 capture\_loss.log #separator \x09 #set\_separator , #empty\_field (empty) #unset\_field - #path capture\_loss #open 2015-01-09-16-33-16 #fields ts ts\_delta peer gaps acks percent\_lost #types time interval string count count double ---
**Author:** ![Li\_Yee\_Ting](https://avatars.discourse-cdn.com/v4/letter/l/ee59a6/32.png) [@Li\_Yee\_Ting](https://community.zeek.org/u/Li_Yee_Ting)\ **Post date:** [January 9, 2015, 9:44pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/17 "2015-01-09T21:44:18Z")
Hi, i concur with Aashish; the biggest help is the shunting of large flows (and possibly encrypted flows). we have a Cisco Nexus 3172 (6x40gbps + 48x10gbps copper) load balancing to 6 x Dell 620s (E5-2695 v2 @ 2.40GHz x 24); each with Intel X540-AT2’s (2x10gbp copper) running 20 workers each (with pfring/dna)… sustaining about 5gbps… and we still see packet loss \>5% on some workers due to the elephant flows in our environment. Yee. ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [January 9, 2015, 9:45pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/18 "2015-01-09T21:45:03Z")
> How I can I specify another directory ? What do you mean? > What do the fields mean ? It’s documented:   [https://www.bro.org/sphinx/scripts/policy/misc/capture-loss.bro.html#type-CaptureLoss::Info](https://www.bro.org/sphinx/scripts/policy/misc/capture-loss.bro.html#type-CaptureLoss::Info) > root@x64-01:/# cat cap\* > 1420832673.023244,900.000068,bro,0,0,0.0 > 1420833573.023279,900.000035,bro,0,6,0.0 > 1420833727.951157,154.927878,bro,0,0,0.0 > 1420833885.693988,154.676438,bro,0,0,0.0 That last number is the estimated percent of packet loss. Unnnnnfortunately, I think I know enough to guess that your traffic is heavily leaning toward DNS and capture-loss relies on having a lot of TCP available so in your case the numbers might be misleading.   .Seth ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [January 10, 2015, 3:11am UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/19 "2015-01-10T03:11:14Z")
Yes, this has been high on our radar for quite a while. I suspect we’re getting closer and closer to getting this into Bro in a generic manner. Unfortunately it just takes a lot of time and experiences to get it right.   .Seth ---
**Author:** ![Kelly\_Kerby](https://avatars.discourse-cdn.com/v4/letter/k/4491bb/32.png) [@Kelly\_Kerby](https://community.zeek.org/u/Kelly_Kerby)\ **Post date:** [January 12, 2015, 5:43pm UTC](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415/20 "2015-01-12T17:43:58Z")
Hello, We at UT Austin are fairly new to Bro and new to the list (been following, but never posted), but I thought I'd share my experience. We have had good luck monitoring our traffic which sustains ~17-20 Gbps during peak hours with 2 devices made by a company called Netronome. The traffic is distributed between the 2 clustered devices using an integrated load balancer which evenly spreads the traffic across all the processors which have been pinned to corresponding bro workers. We see very little traffic loss - random ~2-3% drops per Bro instance with the occasional larger ~10% drop. Our configuration: - 2 clustered devices 40 cores each with 32 workers and 4 proxies - Primary device with 2 10 gig cards Hope this is helpful. -Kelly UT Austin [Next page](https://community.zeek.org/t/bro-with-10gb-nics-or-higher/3415.md?page=2)