Hello, I’m fairly new to Zeek and I’m trying to install and configure a Zeek cluster as a Proof of Concept for enterprise deployment. The environment consists of 3 hosts - 1 manager and 2 workers (Zeek has been compiled with PF_RING to leverage load balancing capabilities).
The installation is successful, zeekctl deploy didn’t yield any errors, and all nodes appear as ‘running’; however I cannot see any logs (HTTP, DNS, SSL etc.). The workers don’t seem to be working.
Digging a little bit, the logger process yields a report.log, in which the following entry shows up repeatedly: {"ts":1586866086.934979,"level":"Reporter::WARNING","message":"SumStat key request for the j1158rc4kei SumStat uid took longer than 1 minute and was automatically cancelled.","location":"/usr/local/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226"}.
I’ve found a similar issue here but I made sure that scan.zeek policy is commented out.
Also, the manager process outputs the following in the stderr.log: warning in /usr/local/zeek/share/zeek/base/frameworks/netcontrol/./cluster.zeek, line 117: &default on parameter 'msg' has no effect (not a event declaration). Not sure if this is even relevant, but I did not recall seeing this when I installed zeek as a standalone.
Could you help shed a light on this?
I’m sharing as much information as possible from the cluster below:
node.cfg:
[manager]
type=manager
host=nids
[proxy-1]
type=proxy
host=nids
[logger]
type=logger
host=nids
[worker-1]
type=worker
host=192.168.2.31
interface=ens3
lb_method=pf_ring
lb_procs=3
pin_cpus=0,1,2
[worker-2]
type=worker
host=192.168.2.36
interface=ens3
lb_method=pf_ring
lb_procs=3
pin_cpus=0,1,2
zeekctl status:
Name Type Host Status Pid Started
logger logger nids running 12620 14 Apr 11:52:04
manager manager nids running 12668 14 Apr 11:52:05
proxy-1 proxy nids running 12715 14 Apr 11:52:07
worker-1-1 worker 192.168.2.31 running 24440 14 Apr 11:52:08
worker-1-2 worker 192.168.2.31 running 24436 14 Apr 11:52:08
worker-1-3 worker 192.168.2.31 running 24439 14 Apr 11:52:08
worker-2-1 worker 192.168.2.36 running 24619 14 Apr 11:52:08
worker-2-2 worker 192.168.2.36 running 24617 14 Apr 11:52:08
worker-2-3 worker 192.168.2.36 running 24616 14 Apr 11:52:08
zeekctl top:
Name Type Host Pid VSize Rss Cpu Cmd
logger logger nids 12620 1G 107M 0% zeek
manager manager nids 12668 678M 108M 0% zeek
proxy-1 proxy nids 12715 676M 106M 0% zeek
worker-1-1 worker 192.168.2.31 24440 683M 112M 0% zeek
worker-1-2 worker 192.168.2.31 24436 683M 112M 0% zeek
worker-1-3 worker 192.168.2.31 24439 683M 113M 0% zeek
worker-2-1 worker 192.168.2.36 24619 685M 115M 0% zeek
worker-2-2 worker 192.168.2.36 24617 683M 113M 0% zeek
worker-2-3 worker 192.168.2.36 24616 684M 114M 0% zeek
zeekctl config:
bindir = /usr/local/zeek/bin
capstatspath = /usr/local/zeek/bin/capstats
cfgdir = /usr/local/zeek/etc
commandtimeout = 60
commtimeout = 10
compresscmd = gzip
compressextension = gz
compresslogs = 1
compresslogsinflight = 0
configchksum = cc8e3228f42668759783d0165ac9181f751e6e76
confignodechksum = 29aa08b5f6adaf65cfe2f550452d9abd7a76a699
controltopic = zeek/control
crashexpireinterval = 0
croncmd =
cronenabled = True
debug = 0
debuglog = /usr/local/zeek/spool/debug.log
defaultstoredir = /usr/local/zeek/spool/stores
env_vars =
global-hash-seed = a776fc25
hash-nodecfg = 05042402823ed87a824dd5042ad63f8f679b6761
hash-zeekctlcfg = 583b8364fa01143dead8af7fbbcdb01fc98762f2
havenfs = 0
helperdir = /usr/local/zeek/share/zeekctl/scripts/helpers
keeplogs =
lb_custom.interfaceprefix =
lb_custom.interfacesuffix =
libdir = /usr/local/zeek/lib
libdir64 = /usr/local/zeek/lib64
libdirinternal = /usr/local/zeek/lib/zeekctl
localnetscfg = /usr/local/zeek/etc/networks.cfg
lockfile = /usr/local/zeek/spool/lock
logdir = /usr/local/zeek/logs
logexpireinterval = 0
logexpireminutes = 0
logger-crashed = False
logger-expect-running = True
logger-host = nids
logger-pid = 12620
logger-port = 47763
logrotationinterval = 3600
mailalarmsinterval = 86400
mailalarmsto = root@localhost
mailarchivelogfail = 1
mailconnectionsummary = True
mailfrom = Zeek <zeek@nids>
mailhostupdown = True
mailreceivingpackets = 1
mailreplyto =
mailsubjectprefix = [Zeek]
mailto = root@localhost
makearchivename = /usr/local/zeek/share/zeekctl/scripts/make-archive-name
manager-crashed = False
manager-expect-running = True
manager-host = nids
manager-pid = 12668
manager-port = 47764
memlimit = unlimited
mindiskspace = 5
nodecfg = /usr/local/zeek/etc/node.cfg
os = Linux
pcapbufsize = 128
pcapsnaplen = 9216
pfringclusterid = 21
pfringclustertype = 4-tuple
pfringfirstappinstance = 0
pin_command = taskset -c
plugindir = /usr/local/zeek/lib/zeekctl/plugins
pluginzeekdir = /usr/local/zeek/lib/zeek/plugins
policydir = /usr/local/zeek/share/zeek
policydirsiteinstall = /usr/local/zeek/spool/installed-scripts-do-not-touch/site
policydirsiteinstallauto = /usr/local/zeek/spool/installed-scripts-do-not-touch/auto
postprocdir = /usr/local/zeek/share/zeekctl/scripts/postprocessors
prefixes = local
proxy-1-crashed = False
proxy-1-expect-running = True
proxy-1-host = nids
proxy-1-pid = 12715
proxy-1-port = 47765
savetraces = 0
scriptsdir = /usr/local/zeek/share/zeekctl/scripts
sendmail = /usr/sbin/sendmail
sitepluginpath =
sitepolicypath = /usr/local/zeek/share/zeek/site
sitepolicyscripts = local.zeek
spooldir = /usr/local/zeek/spool
standalone = False
statefile = /usr/local/zeek/spool/state.db
staticdir = /usr/local/zeek/share/zeekctl
statsdir = /usr/local/zeek/logs/stats
statslog = /usr/local/zeek/spool/stats.log
statslogenable = True
statslogexpireinterval = 0
statuscmdshowall = False
stoptimeout = 60
stopwait = 0
test.enabled = False
test.foo = 1
time = /usr/bin/time
timefmt = %d %b %H:%M:%S
timemachinehost =
timemachineport = 47757/tcp
tmpdir = /usr/local/zeek/spool/tmp
tmpexecdir = /usr/local/zeek/spool/tmp
tracesummary = /usr/local/zeek/bin/trace-summary
version = 2.1.0-11
worker-1-1-crashed = False
worker-1-1-expect-running = True
worker-1-1-host = 192.168.2.31
worker-1-1-pid = 24440
worker-1-1-port = 47766
worker-1-2-crashed = False
worker-1-2-expect-running = True
worker-1-2-host = 192.168.2.31
worker-1-2-pid = 24436
worker-1-2-port = 47767
worker-1-3-crashed = False
worker-1-3-expect-running = True
worker-1-3-host = 192.168.2.31
worker-1-3-pid = 24439
worker-1-3-port = 47768
worker-2-1-crashed = False
worker-2-1-expect-running = True
worker-2-1-host = 192.168.2.36
worker-2-1-pid = 24619
worker-2-1-port = 47769
worker-2-2-crashed = False
worker-2-2-expect-running = True
worker-2-2-host = 192.168.2.36
worker-2-2-pid = 24617
worker-2-2-port = 47770
worker-2-3-crashed = False
worker-2-3-expect-running = True
worker-2-3-host = 192.168.2.36
worker-2-3-pid = 24616
worker-2-3-port = 47771
zeek = /usr/local/zeek/bin/zeek
zeekargs =
zeekbase = /usr/local/zeek
zeekctlconfigdir = /usr/local/zeek/spool
zeekport = 47762
zeekscriptdir = /usr/local/zeek/share/zeek
zeekversion = 3.2.0-dev.391
zeekctl diag:
[logger]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== reporter.log
{"ts":1586865786.930556,"level":"Reporter::WARNING","message":"SumStat key request for the 5dLj9RAlW1g SumStat uid took longer than 1 minute and was automatically cancelled.","location":"/usr/local/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226"}
{"ts":1586865786.930556,"level":"Reporter::WARNING","message":"SumStat key request for the JXG5gNSXhlj SumStat uid took longer than 1 minute and was automatically cancelled.","location":"/usr/local/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226"}
{"ts":1586866086.934979,"level":"Reporter::WARNING","message":"SumStat key request for the j1158rc4kei SumStat uid took longer than 1 minute and was automatically cancelled.","location":"/usr/local/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226"}
{"ts":1586866086.934979,"level":"Reporter::WARNING","message":"SumStat key request for the 8eFeFUPsW01 SumStat uid took longer than 1 minute and was automatically cancelled.","location":"/usr/local/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226"}
==== stderr.log
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=logger
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[manager]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
warning in /usr/local/zeek/share/zeek/base/frameworks/netcontrol/./cluster.zeek, line 117: &default on parameter 'msg' has no effect (not a event declaration)
warning in /usr/local/zeek/share/zeek/base/frameworks/netcontrol/./cluster.zeek, line 125: &default on parameter 'msg' has no effect (not a event declaration)
warning in /usr/local/zeek/share/zeek/base/frameworks/netcontrol/./cluster.zeek, line 133: &default on parameter 'msg' has no effect (not a event declaration)
warning in /usr/local/zeek/share/zeek/base/frameworks/netcontrol/./cluster.zeek, line 143: &default on parameter 'msg' has no effect (not a event declaration)
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p manager local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=manager
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[proxy-1]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p proxy-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=proxy-1
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[worker-1-1]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
listening on ens3
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i ens3 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=worker-1-1
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[worker-1-2]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
listening on ens3
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i ens3 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=worker-1-2
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[worker-1-3]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
listening on ens3
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i ens3 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-3 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=worker-1-3
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[worker-2-1]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
listening on ens3
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i ens3 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=worker-2-1
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[worker-2-2]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
listening on ens3
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i ens3 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=worker-2-2
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log
[worker-2-3]
No core file found.
Zeek 3.2.0-dev.391-debug
Linux 4.15.0-36-generic
Zeek plugins: (none found)
==== No reporter.log
==== stderr.log
listening on ens3
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i ens3 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-3 local.zeek zeekctl base/frameworks/cluster zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=worker-2-3
==== .status
RUNNING [net_run]
==== No prof.log
==== No packet_filter.log
==== No loaded_scripts.log