I Need Guidance on Optimizing Zeek for High-Traffic Environments

Hello guys! :smiling_face_with_three_hearts:

I am managing a network security infrastructure for a mid-sized enterprise experiencing rapid growth. We rely heavily on Zeek to monitor our network traffic, but the increasing volume is putting pressure on our current setup.

To ensure optimal performance and reliability, I’m seeking expert advice on enhancing our Zeek deployment. Specifically, I’m interested in best practices for:

  • Hardware configuration: Identifying the ideal CPU, memory, and storage specifications to handle high-throughput traffic.
  • Zeek tuning: Optimizing Zeek configurations and scripts for peak performance under heavy load.
  • Log management: Efficiently storing and managing the vast amount of generated logs without impacting system performance.
  • Zeek clustering: Implementing a robust Zeek cluster to distribute the workload effectively.

I also check this: https://community.zeek.org/t/monitoring-of-intra-virtual-machines-network-traffic-on-same-physical-hosalteryx But I have not found any solution. Could anyone provide me the best solution for this? I would appreciate any insights, recommendations, or resource suggestions from those who have successfully managed similar challenges.

Thanks in advance! :innocent:

Respected community member :blush:

Hi, you might wanna try a zeek upgrade to zeek 7.0.3 (at the time of this reply) in a development environment to test the performance and see if it fits your needs.

Generally, intel processers provide the best performance for network packet filtering. There’re other considerations such as configurations. You may want to look at such as using AF_PACKET for flow-based load balancing. It’s prefered over PF_RING and provides better performance at 10gbps speeds.
Refer to zeek documentation: Zeek Cluster Setup — Book of Zeek (v7.0.3)
Also refer to Security Onion manual: https://docs.securityonion.net/_/downloads/en/2.4/pdf/

Ensure you have the same MTU on your monitoring interfaces (on all zeek nodes) as the monitoring/ mirroring ports they are connected to. It sounds like a lot but you have to get the configurations correct to realise the best performance benefits.
Happy hacking and hunting.

This kind of question also in general is very hard to answer for us - as it is difficult to give generic advice.

That being said, there recently were two talks in our new webinar series, where the speakers talked about the hardware configuration of their system, with some mentions of the other topics that you noted.

These talks are Zeek@Meta: Scale, Log Enrichment and Detections and How Zeek Helps Secure Open Science.

I hope the information in them is helpful to you.