Zeek memory Limit

Zeek
1

Hello everyone,
I’m running Zeek 6.0.1 with the DPDK ESnet plugin. and trying to limit Zeek’s memory usage. I’m currently testing two approaches: Zeek’s Memlimit and setting memory limits via systemd (systemctl).
But i got result as follow

  • with MemLimit=15728640 (15GB i think) in zeekctl.cfg
[root@localhost ~]# /opt/zeek/bin/zeekctl deploy
checking configurations ...
installing ...
removing old policies in /storage/meta/spool/installed-scripts-do-not-touch/site ...
removing old policies in /storage/meta/spool/installed-scripts-do-not-touch/auto ...
creating policy directories ...
installing site policies ...
generating cluster-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
stopping ...
stopping worker ...
stopping proxy ...
stopping manager ...
stopping logger ...
starting ...
starting logger ...
(zeek still initializing)
starting manager ...
(manager still initializing)
starting proxy ...
(proxy still initializing)
starting worker ...
(worker-1 still initializing)
  • Limit with systemctl
    Zeek always auto restart so i gave it up

Is there any other practical method to limit Zeek’s memory usage besides Memlimit and systemd (systemctl) limits?

2

Hey @trong- shameless systemd plug - there isn’t much documentation or tutorials yet, but if you’re open to exploring a non-Zeekctl cluster: With Zeek 8.1, there’s a way to have an opinionated systemd-native Zeek deployment:

https://github.com/zeek/zeek/tree/master/tools/systemd-generator

You can set worker_memory_max in the configuration file. systemd will then limit the memory of each individual Zeek worker using cgroups. You can also use drop-in unit files to, e.g., set a memory limit on the zeek-workers.sliceto cap the limit for all workers together.

This is independent of Zeekctl. It does away with the cron job for restarting crashed Zeek processes, stderr/stdout of the individual Zeek processes is readily available via journalctl and any information about process restarts is also tracked by systemd. zeekctl top can be replaced by systemd-cgtop zeek.slice. I like it a lot.

In the following just released video you can see a glimpse of it running on my own laptop (1.5x speed recommended :slight_smile: ):

https://www.youtube.com/watch?v=EeW_Oo-xNdQ

This doesn’t provide easy support for a multi-host cluster and assumes customization is done within Zeek scripts or with separate service files that are managed separately, so definitely a bit more advanced. It’s tailored for a fairly specific and opinionated Zeek deployment on a single host. It’s also fairly new, so any feedback and experiences would be very welcome :slight_smile:

Hope this helps,
Arne

1 Like
3

Hi @awelzel,
Thanks for the information.

I noticed that you created a plugin that allows limiting Zeek’s memory usage:

However, it seems that this approach does not work as expected when using the esnet-dpdk plugin.
Here is my config

[root@localhost etc]# tail -20 zeekctl.cfg
systemd.enabled = true
systemd.etc_unit_path = /etc/systemd/system
systemd.user = zeek
systemd.group = zeek
systemd.lib_unit_path = /usr/lib/systemd/system
systemd.logger_memory_max = 4G
systemd.logger_nice = -19
systemd.loggers_memory_max = 8G
systemd.manager_memory_max = 4G
systemd.manager_nice = -19
systemd.memory_max = 8G
systemd.proxies_memory_max = 8G
systemd.proxy_memory_max = 4G
systemd.proxy_nice = -19
systemd.restart = always
systemd.restart_sec = 1
systemd.start_limit_interval_sec = 0
systemd.worker_memory_max = 4G
systemd.worker_nice = -19
systemd.workers_memory_max = 16G
4

Hey @trong,

I noticed that you created a plugin that allows limiting Zeek’s memory usage:

The zeekctl-systemd plugin is a bit older and was a first idea when I hadn’t learned about systemd-generators yet. I’d recommend you use zeek-systemd-generator / zeek.conf instead.

However, it seems that this approach does not work as expected when using the esnet-dpdk plugin.
Here is my config

You would need to share a few more details (error messages, log output or observations) as of why it is not working. If it’s about the memory limit not being in effect, check the generated unit files and/or use systemctl show for interrogation.

Hope that helps,
Arne

1 Like
5

Thanks @awelzel

I solved the problem. I realized the unit file was missing the necessary fields in CapabilityBoundingSet, so Zeek couldn’t access the dpdk’s memory.

More specifically, it is as follows

[root@localhost etc]# cat /etc/systemd/system/zeek-worker@.service.d/10-capabilities.conf 
[Service]
CapabilityBoundingSet=CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_IPC_LOCK CAP_IPC_OWNER CAP_KILL CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_NICE CAP_SYS_PTRACE CAP_SYS_RESOURCE CAP_SYS_TIME CAP_SYS_TTY_CONFIG CAP_SYS_RAWIO
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW

I’m currently using version 6.0.2 on Rocky9 and I think it will be quite difficult to upgrade to version 8.1.0.

6

. I realized the unit file was missing the necessary fields in CapabilityBoundingSet

Nice! Glad you figured this out. Yeah, the processes are very restricted in their capabilities by default.

I’m currently using version 6.0.2 on Rocky9 and I think it will be quite difficult to upgrade to version 8.1.0.

Could you consider 8.0.x? We do not maintain 6.0.x anymore and with the 8.1 release just being out, 7.0 has left maintenance, too.

What I’ve done in an internal project is to include the tools/systemd-generator directory (and the policy script policy/misc/systemd-generator.zeek) in the 8.0 tree - essentially a backport. It works well.

For 6.0, I’d probably try the following: You can compile the generator independently of Zeek (you’ll have to change cluster_backend_args in zeek.conf to base/frameworks/broker with 6.0 as it defaults to policy/frameworks/cluster/backend/zeromq which is only available starting with 8.0.

In a Zeek checkout:

$ cd tools/systemd-generator
$ mkdir build && cd build && cmake ../ && make -j && make install

This will assume /usr/local/zeek/etc/zeek/zeek.conf as configuration file.

7

thanks @awelzel ,
I’ll try it

1 Like