Using systemd deployment method with PF_RING

Zeek
,
1

Hi,

Zeek-8.1 has added a Systemd generator deployment method and the docs show how to deploy Zeek cluster on a single node.

We have been using zeekctl to deploy Zeek using PF_RING as load balancer till now. However, from the docs it is not clear to me how to translate PF_RING specific node.cfg options (lb_method and lb_procs) to zeek.conf?

Additionally, In our deployment we use use /opt/zeek as prefix with logs going to a separate /zeeklogmount point. We disable PF_RING’s ld_so.conf entries so that all programs that link to libpcap (like tcpdump or snort) do not link to PF_RING’s libpcap. For Zeek, we create a wrapper around zeekctl at /usr/bin/zeekctl explicitly linking it to PF_RING

#!/bin/bash
PATH=/opt/zeek/bin:$PATH LD_LIBRARY_PATH=/opt/zeek/lib64/:/usr/local/lib:$LD_LIBRARY_PATH /usr/bin/python3.11 /opt/zeek/bin/zeekctl $*

And then call zeekctl normally. To replicate the above (only Zeek gets linked to PF_RING’s libraries), how to proceed when using Zeek’s systemd generator?

Any insights will be helpful.

Thanks

2

Hey @redbaron :waving_hand:

We have been using zeekctl to deploy Zeek using PF_RING as load balancer till now. However, from the docs it is not clear to me how to translate PF_RING specific node.cfg options (lb_method and lb_procs) to zeek.conf?

Haven’t tested PF_RING yet, but should mostly be setting a few environment variables. I’ll try it out in the coming days and post here and/or update instructions.

Additionally, In our deployment we use use /opt/zeek as prefix with logs going to a separate /zeeklogmount point.

The log rotation setup for the zeek-systemd-generator is only using a basic zeek-archiver setup, rotating logs from logger-X spool directories into <PREFIX>/var/log/zeek/<date> directories. One thought would be to mount/bind-mount /zeeklogs to /opt/zeek/var/log/zeek/ and see if that would work for you?

Thanks for you interest! Besides @dopheide, you’re part of the early adopter group :slight_smile:

3

@redbaron - mind checking out the proposed changes to the docs?

The following should work:

[zeek]
manager = 1
loggers = 1
proxies = 2

[interface enp8s0]
interface = enp8s0
workers = 4
# Preload PF_RING's libpcap.so library and setup the environment for
# flow-balancing on inner five tuples. Find more environment variables
# recognized by PF_RING's libpcap version here:
#
#    https://www.ntop.org/guides/pf_ring/api/libpcap.html
#
worker_env =
  LD_PRELOAD=/usr/local/lib/libpcap.so
  PCAP_PF_RING_APPNAME=zeek-${interface_section_name}
  PCAP_PF_RING_CLUSTER_ID=42
  PCAP_PF_RING_USE_CLUSTER_PER_INNER_FLOW_5_TUPLE=1

Instead of LD_PRELOAD, can compile Zeek using --with-pcap. Find more details in the proposed docs changes.

Let us know if you have more questions or run into and blockers. Feedback is appreciated!

4

Hi @awelzel

Thanks for looking at this and providing the info.

I am still trying to figure out systemd-generator (first without PF_RING) so I am not sure if I can be called an ‘early adopter’ :slight_smile: .

Looking at the code of systemd-generator, I cannot find any mention of worker_env. Our custom zeek package dumps zeek libs at /opt/zeek/lib64 so when I start zeek-target all zeek processes (except archiver) fail with libbinpac.so not found . In zeekctl, I overcome this using the custom wrapper I mentioned, but when I use the following zeek.conf, the error does not go away. I can’t see any Environment changes in generated units too

interface = enp1s0
workers = 1
loggers = 1
proxies = 1
base_dir = /opt/zeek
archiver = 1
user = root
group = zeek
worker_env =
  LD_LIBRARY_PATH=/opt/zeek/lib64/:$LD_LIBRARY_PATH

The generated worker unit

# cat /run/systemd/generator/zeek-worker@.service 
# Auto-generated, do not edit. Use drop-in files instead!
[Unit]
Description=Zeek Worker %i
SourcePath=/opt/zeek/etc/zeek/zeek.conf
After=zeek-setup.service
After=zeek-manager.service
After=zeek-logger@.service
After=zeek-proxy@.service
StopPropagatedFrom=zeek.target
StartLimitIntervalSec=0

[Service]
SyslogIdentifier=zeek-worker-%i
Type=exec
Nice=0
MemoryMax=
User=root
Group=zeek
WorkingDirectory=/opt/zeek/var/spool/zeek/worker-%i
ReadWritePaths=/opt/zeek/var/spool/zeek/worker-%i 
CapabilityBoundingSet=CAP_NET_RAW
AmbientCapabilities=CAP_NET_RAW
Environment=PATH=/opt/zeek/bin:/usr/local/bin:/usr/bin:/bin
Environment=ZEEKPATH=/opt/zeek/var/spool/zeek/generated-scripts:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
Environment=CLUSTER_NODE=worker-%i
ExecStart=/opt/zeek/bin/zeek -i ${INTERFACE} policy/misc/systemd-generator local frameworks/cluster/backend/zeromq
Slice=zeek-workers.slice
Restart=always
RestartSec=1

If I can get Zeek to start without PF_RING, I will look at the PF_RING based environment changes you mentioned.

5

I am still trying to figure out systemd-generator (first without PF_RING) so I am not sure if I can be called an ‘early adopter’ :slight_smile: .

Then thanks for testing :slight_smile:

Looking at the code of systemd-generator, I cannot find any mention of worker_env

Which version are you testing with? worker_envwill only become available in 9.0, so recommend you build from git using the release/9.0 branch or wait until the rc2 tarball is out.

The worker_env settings will be rendered into individual drop-in files. For example, the first worker of enp8s0 here has the following drop-in unit file generated (they are worker specific, because you can use basic templating for the environment variables).

$ cat /run/systemd/generator/zeek-worker-enp8s0@1.service.d/10-zeek-systemd-generator.conf
[Unit]
SourcePath=/opt/zeek/etc/zeek/cluster/ubuntu-01.zeek.conf

[Service]
Environment=INTERFACE=enp8s0
Environment=LD_PRELOAD=/usr/local/lib/libpcap.so
Environment=PCAP_PF_RING_APPNAME=zeek-enp8s0
Environment=PCAP_PF_RING_CLUSTER_ID=42
Environment=PCAP_PF_RING_USE_CLUSTER_PER_INNER_FLOW_5_TUPLE=1

fail with libbinpac.so not found

Hmm, hmm, hmm… Do you use --binary-package at configure time for your build? Any chance you could drop that? Alternatively, you could add /etc/ld.so.conf/zeek.conf that includes /opt/zeek/lib64 and run ldconfig, but maybe you opted against doing that?

6

Ah that explains it. I am on 8.2.2, hence couldn’t find it. I’ll wait for 9.0 (stable) for further testing.

For libbinpac.so I usually rely on setting LD_LIBRARY_PATH but I can also add and ld.so for zeek. That too I’ll try when trying to build RPM for 9.0 but my guess is that LD_LIBRARY_PATH set via env should work too?

7

I’ll wait for 9.0 (stable) for further testing.

Thanks!

LD_LIBRARY_PATH set via env should work too?

Yeah, it should. It’s just a bit unusual, maybe? And if you just want to run /opt/zeek/bin/zeek , you always need to set LD_LIBRARY_PATH?

There’s a mention in the docs about file-based capabilities: If you use setcap on the zeek executable, theLD_LIBRARY_PATH, LD_PRELOAD and some other dynamic linker environment variables as well as MALLOC_CONF will be ignored, that’s why I’d think using ld.so.confor dropping --binary-package might be easiest.

8

Thanks @awelzel for those pointers. I think ld.so.conf path is the best.

Only problem is that I need LD_LIBRARY_PATH for PF_RING . I had once (Zeek-3.0 or so) tried using --with-pcap when building Zeek and providing PF_RING’s libpcap but it had caused some problems (don’t remember as it was long back). I don’t need PF_RING’s libpcap to get used for other tools (there’s a bunch of them on the sensor), just for Zeek so LD_LIBRARY_PATH or LD_PRELOAD was what I chose. Maybe I’ll give it a try with 9.0 when I try to overhaul my archaic build process.