My understanding is that 4,000+ CPU cores would be necessary to support this throughput. In the recent meeting from CERN I recall seeing someone describe 200Gbps, which would imply 8,000+ CPU cores. Is this accurate, or am I doing a conversion incorrectly?
I am basing this purely on this quote, from
https://docs.zeek.org/en/stable/cluster/
“The rule of thumb we have followed recently is to allocate approximately 1 core for every 250Mbps of traffic that is being analyzed. However, this estimate could be extremely traffic mix-specific. It has generally worked for mixed traffic with many users and servers. For example, if your traffic peaks around 2Gbps (combined) and you want to handle traffic at peak load, you may want to have 8 cores available (2048 / 250 == 8.2). ”
These rules aren’t current anymore and frankly, have never been accurate.
Your Zeek speed depends on the traffic you have, if you have some elephant flows (and how you deal with them), scripts you run, etc. I remember pushing between 5-10Gbit/sec through a server with 24 cores (not threads), with room to spare.
You will also need memory, and depending on scripts you intend to write, that might be quite a lot. We run with 192GB / server.
Do you have 100Gbit of traffic or 100Gbit interfaces?
Either way, you’re gonna build yourself a cluster with a packet broker in front of it. Arista works well, other people use different brands, depending on your needs and your budget.
Give those tuning guides I wrote with Suricata developers a read, while on it, they apply to Zeek as well. Of course Suricata can process way more traffic per core, than Zeek, because the processing it does is way simpler.
https://github.com/pevma/SEPTun
https://github.com/pevma/SEPTun-Mark-II
Thanks for the details. I am aware of MarkII and am reading through it.
How as a community can we update that clustering documentation? If it’s not accurate it could very easily turn people away
I agree. If you keep notes as you build your cluster please share them.
Updating cluster docs may be another thing to add here.
https://blog.zeek.org/2019/04/google-season-of-docs.html
-AK
The guidelines are bit off these days, a single core does more work
than it used to. However, your math is off by a factor of 10.
Sticking with the rule that one core can do 250mbps, then you need 4
cores to handle 1gbps, 40 to handle 10 gbps, and 400 to handle
100gbps. Not 4000.
Whoops, thanks for fixing the arithmetic 
. Do we have to details on to how modern CPUs benchmark?