Zeek crashes after zeekctl cron

Zeek
1

Hello,

Whenever I do zeekctl cron directly or by a cronjob. The running zeek process crashes and I have the following error in stderr.log.

error in /opt/zeek/spool/installed-scripts-do-not-touch/auto/standalone-layout.zeek, line 6: Failed to listen on INADDR_ANY:27760 (Broker::listen(Broker::default_listen_address, Broker::default_port, Broker::default_listen_retry)) fatal error: errors occurred while initializing

I am pretty confident that there is no other process than zeek that listen on 27760, I checked with netstat

Here is my zeekctl.cfg:

MailTo = root@localhost
MailConnectionSummary = 1
MinDiskSpace = 5
MailHostUpDown = 1
LogRotationInterval = 3600
LogExpireInterval = 1 day
StatsLogEnable = 1
StatsLogExpireInterval = 0
StatusCmdShowAll = 0
CrashExpireInterval = 0
SitePolicyScripts = /opt/zeek/share/zeek/site/local.zeek
LogDir = /opt/zeek/logs
SpoolDir = /opt/zeek/spool
BrokerDBDir = /opt/zeek/spool/brokerstore
CfgDir = /opt/zeek/etc
ZeekPort = 27760
2

Hello @Lamorale ,

I am pretty confident that there is no other process than zeek that listen on 27760, I checked with netstat

Any chance you have other Zeek processes running on they system? What does your netstat output look like? Does -p tell you the pid? Is it possible you have runaway Zeek processes?

netstat -n --listen --ip -p | grep 277

Hope that helps,
Arne

3

It might be caused by runaway zeek processes.
Also I am surprised that the zeekctl does not kill the zeek processes.

Here are the steps that I do:

  1. kill all zeek processes
    pkill -f zeek

  2. check nothing is listening to the port
    netstat -n --listen --ip -p | grep 277

  3. start zeek
    /opt/zeek/bin/zeekctl start
    everything is fine so far, some logs are created in /opt/zeek/spool/zeek/conn.log

  4. stop zeek
    /opt/zeek/bin/zeekctl stop

At that point I am very surpised to still see some zeek processes doing ps aux | grep zeek:
root 185757 0.0 0.0 6788 3152 ? S 15:45 0:00 bash /opt/zeek/share/zeekctl/scripts/run-zeek...
root 185763 31.6 9.0 3393160 739104 ? Sl 15:45 1:20 /opt/zeek/bin/zeek -i ...

From this point if I try to start again with zeekctl it fails with the error message:
Failed to listen on INADDR_ANY:27760...

I think the problem comes from zeek processes not being killed properly by zeekctl

I am not too much well versed into how zeek works but I guess:

/opt/zeek/share/zeekctl/scripts/run-zeek process is the process that manages zeek and that takes orders from zeekctl command. If yes, should this process be killed when zeekctl stop?

/opt/zeek/bin/zeek process is the main zeek process. This one should definitely be killed by zeekctl sop

What exactly is ZeekPort? Is it the port that the main zeek process will listen to get its orders from run-zeek?

Why is zeek-run necessary? Cannot zeekctl directly communicate with zeek?

I am running a simple single node zeek.

Thanks in advance

4

Yeah, what you describe should not happen. Do you have another Zeek installation on the same system and possibly a crontab installation that restarts workers? What does zeekctl ps.zeek tell you? Here’'s it’s code with a description of the output.

What exactly is ZeekPort? Is it the port that the main zeek process will listen to get its orders from run-zeek?

It’s the “base” port used to allocate port numbers for all processes Broker communication. You will see port 27761, 27762, 27663, etc, allocated.

Why is zeek-run necessary? Cannot zeekctl directly communicate with zeek?

zeekctl communicates with Zeek via the broker port, zeek-run should mostly sets-up the environment and records some information in the working directory.

Does zeekctl diag indicate anything unusual after starting the nodes?

5

First of all thanks for your response and sorry for not answering earlier to your message, the problem actually disappeared with the time and I did not investigate further because things were working…

But the problem is back

Here are the steps I perform:

  1. Kill all seek process

  2. Check with netstat port usage

  3. Start zeek with /opt/zeek/bin/zeekctl start

    From this point I see 2 zeek processes running:
    ps aux | grep zeek
    shows
    root 24177 0.0 0.0 7036 2956 ? S 15:00 0:00 bash /opt/zeek/share/zeekctl/scripts/run-zeek -1 -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto
    and
    root 24183 39.6 4.9 3175000 602060 ? Sl 15:00 0:21 /opt/zeek/bin/zeek -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto

Zeek is actively writing connection logs in /opt/zeek/logs/current directory as expected
(conn.log, etc…)

However:
/opt/zeek/bin/zeekctl status
shows
zeek standalone localhost crashed

But
/opt/zeek/bin/zeekctl ps.zeek
shows
(-) root 24183 24177 42.7 5.3 3329432 658168 ? S 15:00:18 00:01:15 zeek

The zeekctl status should not show crashed as zeek process is running and actively producing logs.

Now when I stop zeek process with zeekctl:

/opt/zeek/bin/zeekctl stop

That does no stop the process.

Indeed
ps aux | grep zeek
shows
root 24177 0.0 0.0 7036 2956 ? S 15:00 0:00 bash /opt/zeek/share/zeekctl/scripts/run-zeek -1 -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto
and
root 24183 41.0 5.4 3342672 671284 ? Sl 15:00 1:55 /opt/zeek/bin/zeek -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto

/opt/zeek/bin/zeekctl ps.zeek
(-) root 24183 24177 40.2 5.5 3356112 684488 ? S 15:00:18 00:02:29 zeek

If from this point I do zeekctl start it fails because of port already in use which is quite understandable

Then I kill manually all zeek process and repeat steps 1,2 and 3 until a zeek process is running and writing output logs

If I do zeekctl cron,
zeek process stop writing logs in /opt/zeek/logs/current but the process still exist.

Here is the output of zeekctl diag from this point:

[zeek]

No core file found.

Zeek 6.0.3
Linux 6.1.0-17-amd64

Zeek plugins: (none found)

==== reporter.log

==== stderr.log
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
listening on ens224

error in /opt/zeek/spool/installed-scripts-do-not-touch/auto/standalone-layout.zeek, line 6: Failed to listen on INADDR_ANY:27760 (Broker::listen(Broker::default_listen_address, Broker::default_port, Broker::default_listen_retry))
fatal error: errors occurred while initializing

==== stdout.log
max memory size             (kbytes, -m) unlimited
data seg size               (kbytes, -d) unlimited
virtual memory              (kbytes, -v) unlimited
core file size              (blocks, -c) unlimited

==== .cmdline
-i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=

==== .status
TERMINATED [fatal_error]

==== No prof.log

==== packet_filter.log

==== loaded_scripts.log
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Here is my zeekctl.cfg:

MailTo = root@localhost

MailConnectionSummary = 1

MinDiskSpace = 5

MailHostUpDown = 1

###############################################
# Logging Options

LogRotationInterval = 3600

LogExpireInterval = 1 day

StatsLogEnable = 1

StatsLogExpireInterval = 0

StatusCmdShowAll = 0

CrashExpireInterval = 0

SitePolicyScripts = /opt/zeek/share/zeek/site/local.zeek

LogDir = /opt/zeek/logs

SpoolDir = /opt/zeek/spool

BrokerDBDir = /opt/zeek/spool/brokerstore

CfgDir = /opt/zeek/etc

ZeekPort = 27760

It seems that communication between zeekctl and zeek process does not work as expected. Any help much appreciated.