Zeek memory is increasing constantly

Zeek
1

Hi,
I am using zeek 5.0.10 downloaded on 4th Oct 2023.

Issue: Zeek memory keeps increasing over the time.
Description :
i am using Napatech card and distributing the traffic to 70 zeek worker and generating logs on nvme disk. I have observed that memory taken by each zeek worker and manager keeps increasing.(zeekctl top).

Zeek version: 5.0.10
Total worker: 70 with one manager and one proxy.
Link rate: 10Gbps
Zeek output: log files on disk
Plugin: Using Zeek-Napatech plugin.

2

Hello @Vijay , would it be possible for you to try Zeek 6.0.1?

There have been improvements around unbounded state growth in SMB and DCE_RPC.

Alternatively you could try to install the following packages and see if it alleviates memory growth:

GitHub - corelight/zeek-smb-clear-state: reduce amount of tracked smb state
GitHub - corelight/zeek-ssl-clear-state: Clear SSL State earlier to reduce memory usage

We also have a troubleshooting section explaining how to use jemalloc to track down memory leaks:

Troubleshooting ā€” Book of Zeek (git/master)

The latter would be most interesting to us for Zeek 6.0 as Zeek 5.0 is almost end of life.

Hope that helps,
Arne

1 Like
3

Thanks @awelzel ,
I will try with these packages,
Also,Memory spike issue seems to be related with one more issue i have observed that zeek is taking a lot of memory during packet processing and log generation, but when there is no traffic on link, zeek is not releasing the memory it malloced earlier.
Could you please suggest some solution for it or ā€œclear-stateā€ solution will resolve this issue as well.

4

It might help, but you should really consider moving to Zeek 6.0.x as it contains more thorough fixes for the state growth. Sometimes itā€™s also the glibc memory allocator causing fragmentation and compiling Zeek with jemalloc may show improvements, too.

If none of these help, jemalloc memory profiling is the next step.

Hope that helps,
Arne

5

Hi @awelzel ,
I installed both the packages zeek-ssl-clear-state and smb clear-state. Seems it didnā€™t help much.
Zeek is still consuming 90% of RAM in 12 hours.
I am trying to install jemalloc profiling for zeekctl but no packages had been installed using ā€˜zkg install zeek-jemalloc-profilingā€™.
Could you please suggest some other way to resolve this memory issue?

Also, What do you suggest how many zeek worker are enough to process 10Gbps link rate ?
Zeek version : 5.0.10

6

I am trying to install jemalloc profiling for zeekctl but no packages had been installed using ā€˜zkg install zeek-jemalloc-profilingā€™.

Could you provide more details - commands executed and their output?

Have you seen the information on the troubleshooting page?

Also, What do you suggest how many zeek worker are enough to process 10Gbps link rate ?
Zeek version : 5.0.10

[You should upgrade to 6.0.x :slight_smile: - 5.0.10 does not receive maintenance anymore. If you compile from source, compile with libjemalloc instead of glibc as described on the troubleshooting page.]

Itā€™s difficult to say just based on 10Gbps. It heavily depends on the traffic mix youā€™re monitoring. Youā€™ll need to test and iterate on the system and Zeek configuration.

In general, no worker process should saturate a single CPU (show 100% CPU usage in top). Iā€™d start with 8 workers, configure CPU pinning with zeekctl so each worker has a dedicated CPU available. Then check CPU usage at peak times, check for packet drops in stats.log and see if capture_loss log / conn.logā€™s history looks clean. Double the number of workers if they are overloaded, possibly reduce the number of workers if they are barely used. For this approach, you need at least as many CPUs in the system as you configure workers plus a few CPUs for logger, manager and proxies.

If you have very spiky traffic, also look at packet buffers used by the NIC, kernel and for AF_PACKET.
You can check usage of Zeek processes with top or zeekctl top. For better visibility over time, recording usage/memory of workers historically is very useful, too. In a Prometheus environment, you could use process-exporter to collect Zeek processes information on a regular basis.

Hope this helps,
Arne

7

Hi @awelzel , As per the suggestion, I installed zeek_v6.0.2 now using source code compliation method, later i tried to integrate it with Napatech using GitHub - napatech/zeek_plugin . But plugin compliation failed at configure step. Same plugin was working with zeek_4.0.9 and zeek_v5.0.10.
I just need to check here with you - Is zeek_6.0.x is compatible with zeek-napatech plugin or not ? Or is there any other different way to integrate zeek with Napatech ?

8

I just need to check here with you - Is zeek_6.0.x is compatible with zeek-napatech plugin or not ?

Thereā€™s been recently a ticket opened that that it is not.

I donā€™t have the Naptech software installed, but in the best case this is mostly around prefixing header paths with ā€œzeek/ā€ if you want to give it a shot.

Or is there any other different way to integrate zeek with Napatech ?

The plugin seems a good route. Napatech does provide an adapated libpcap which might work, too. Not sure anyone has tried.