Zeek is consuming 100% RAM/memory

Zeek
1

Hi, I am trying to process 10Gbps rate. For that i have created 70workers. i can see that after 30mins zeek workers are consuming 500GB RAM. I dont understand why ??
Please suggest.

2

@Vijay - welcome!

70 workers might be a bit too much for 10Gbps, but memory usage should not approach 500GB for all workers.

Can you confirm that you’re using AF_PACKET or PF_RING for load balancing purposes? If not, it might be that all workers are seeing the same traffic.

3

Thanks awelzel for replying, i am not using AF_Packet or PF_Ring. Here i am using Napatech 10G smart NIC card which itself is load balancing the traffic. I can see it in profiling tool of napatech and also in zeekctl netstats output.

4

Thanks for the info about using Napatech. One cross-check would be to look into conn.log and see if you have multiple entries for the same five-tuple (or same community-id) during some time period. If there are duplicates, possibly all workers are seeing the same traffic. If there are no duplicates, should be okay.

There have been a recent reports around DNS lookups causing memory growth.

This will be fixed in 6.0.1, but you could try commenting out the following lines in local.zeek and see if it significantly improves memory behavior on your end:

@load policy/frameworks/notice/extend-email/hostnames
@load frameworks/files/detect-MHR

Otherwise, if you’re comfortable, memory profiling with jeprof may show what’s going on: Jemalloc Memory Profiling.

Please share which Zeek version you’re running. Also, is memory growth steady or suddenly spikes up? If you monitor using Prometheus node_exporter or process_exporter, could you share graphs?

Thanks,
Arne