Zeek crashes after zeekctl cron

Zeek
1

Hello,

Whenever I do zeekctl cron directly or by a cronjob. The running zeek process crashes and I have the following error in stderr.log.

error in /opt/zeek/spool/installed-scripts-do-not-touch/auto/standalone-layout.zeek, line 6: Failed to listen on INADDR_ANY:27760 (Broker::listen(Broker::default_listen_address, Broker::default_port, Broker::default_listen_retry)) fatal error: errors occurred while initializing

I am pretty confident that there is no other process than zeek that listen on 27760, I checked with netstat

Here is my zeekctl.cfg:

MailTo = root@localhost
MailConnectionSummary = 1
MinDiskSpace = 5
MailHostUpDown = 1
LogRotationInterval = 3600
LogExpireInterval = 1 day
StatsLogEnable = 1
StatsLogExpireInterval = 0
StatusCmdShowAll = 0
CrashExpireInterval = 0
SitePolicyScripts = /opt/zeek/share/zeek/site/local.zeek
LogDir = /opt/zeek/logs
SpoolDir = /opt/zeek/spool
BrokerDBDir = /opt/zeek/spool/brokerstore
CfgDir = /opt/zeek/etc
ZeekPort = 27760
2

Hello @Lamorale ,

I am pretty confident that there is no other process than zeek that listen on 27760, I checked with netstat

Any chance you have other Zeek processes running on they system? What does your netstat output look like? Does -p tell you the pid? Is it possible you have runaway Zeek processes?

netstat -n --listen --ip -p | grep 277

Hope that helps,
Arne

3

It might be caused by runaway zeek processes.
Also I am surprised that the zeekctl does not kill the zeek processes.

Here are the steps that I do:

  1. kill all zeek processes
    pkill -f zeek

  2. check nothing is listening to the port
    netstat -n --listen --ip -p | grep 277

  3. start zeek
    /opt/zeek/bin/zeekctl start
    everything is fine so far, some logs are created in /opt/zeek/spool/zeek/conn.log

  4. stop zeek
    /opt/zeek/bin/zeekctl stop

At that point I am very surpised to still see some zeek processes doing ps aux | grep zeek:
root 185757 0.0 0.0 6788 3152 ? S 15:45 0:00 bash /opt/zeek/share/zeekctl/scripts/run-zeek...
root 185763 31.6 9.0 3393160 739104 ? Sl 15:45 1:20 /opt/zeek/bin/zeek -i ...

From this point if I try to start again with zeekctl it fails with the error message:
Failed to listen on INADDR_ANY:27760...

I think the problem comes from zeek processes not being killed properly by zeekctl

I am not too much well versed into how zeek works but I guess:

/opt/zeek/share/zeekctl/scripts/run-zeek process is the process that manages zeek and that takes orders from zeekctl command. If yes, should this process be killed when zeekctl stop?

/opt/zeek/bin/zeek process is the main zeek process. This one should definitely be killed by zeekctl sop

What exactly is ZeekPort? Is it the port that the main zeek process will listen to get its orders from run-zeek?

Why is zeek-run necessary? Cannot zeekctl directly communicate with zeek?

I am running a simple single node zeek.

Thanks in advance

4

Yeah, what you describe should not happen. Do you have another Zeek installation on the same system and possibly a crontab installation that restarts workers? What does zeekctl ps.zeek tell you? Here’'s it’s code with a description of the output.

What exactly is ZeekPort? Is it the port that the main zeek process will listen to get its orders from run-zeek?

It’s the “base” port used to allocate port numbers for all processes Broker communication. You will see port 27761, 27762, 27663, etc, allocated.

Why is zeek-run necessary? Cannot zeekctl directly communicate with zeek?

zeekctl communicates with Zeek via the broker port, zeek-run should mostly sets-up the environment and records some information in the working directory.

Does zeekctl diag indicate anything unusual after starting the nodes?