# Zeek crashes after zeekctl cron **URL:** \ **Category:** Zeek\ **Created:** [October 19, 2023, 9:47am UTC](https://community.zeek.org/t/zeek-crashes-after-zeekctl-cron/7177 "2023-10-19T09:47:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Lamorale](https://avatars.discourse-cdn.com/v4/letter/l/bc79bd/32.png) [@Lamorale](https://community.zeek.org/u/Lamorale)\ **Post date:** [October 19, 2023, 9:47am UTC](https://community.zeek.org/t/zeek-crashes-after-zeekctl-cron/7177/1 "2023-10-19T09:47:55Z")
Hello, Whenever I do `zeekctl cron` directly or by a cronjob. The running zeek process crashes and I have the following error in stderr.log. `error in /opt/zeek/spool/installed-scripts-do-not-touch/auto/standalone-layout.zeek, line 6: Failed to listen on INADDR_ANY:27760 (Broker::listen(Broker::default_listen_address, Broker::default_port, Broker::default_listen_retry)) fatal error: errors occurred while initializing` I am pretty confident that there is no other process than zeek that listen on 27760, I checked with netstat Here is my zeekctl.cfg: ```auto MailTo = root@localhost MailConnectionSummary = 1 MinDiskSpace = 5 MailHostUpDown = 1 LogRotationInterval = 3600 LogExpireInterval = 1 day StatsLogEnable = 1 StatsLogExpireInterval = 0 StatusCmdShowAll = 0 CrashExpireInterval = 0 SitePolicyScripts = /opt/zeek/share/zeek/site/local.zeek LogDir = /opt/zeek/logs SpoolDir = /opt/zeek/spool BrokerDBDir = /opt/zeek/spool/brokerstore CfgDir = /opt/zeek/etc ZeekPort = 27760 ``` ---
**Author:** ![awelzel](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/awelzel/32/609_2.png) [@awelzel](https://community.zeek.org/u/awelzel)\ **Post date:** [October 19, 2023, 10:55am UTC](https://community.zeek.org/t/zeek-crashes-after-zeekctl-cron/7177/2 "2023-10-19T10:55:24Z")
Hello @Lamorale , > I am pretty confident that there is no other process than zeek that listen on 27760, I checked with netstat Any chance you have other Zeek processes running on they system? What does your netstat output look like? Does `-p` tell you the pid? Is it possible you have runaway Zeek processes? `netstat -n --listen --ip -p | grep 277` Hope that helps, Arne ---
**Author:** ![Lamorale](https://avatars.discourse-cdn.com/v4/letter/l/bc79bd/32.png) [@Lamorale](https://community.zeek.org/u/Lamorale)\ **Post date:** [October 19, 2023, 2:00pm UTC](https://community.zeek.org/t/zeek-crashes-after-zeekctl-cron/7177/3 "2023-10-19T14:00:52Z")
It might be caused by runaway zeek processes. Also I am surprised that the zeekctl does not kill the zeek processes. Here are the steps that I do: 1. kill all zeek processes `pkill -f zeek` 2. check nothing is listening to the port `netstat -n --listen --ip -p | grep 277` 3. start zeek `/opt/zeek/bin/zeekctl start` everything is fine so far, some logs are created in /opt/zeek/spool/zeek/conn.log 4. stop zeek `/opt/zeek/bin/zeekctl stop` At that point I am very surpised to still see some zeek processes doing `ps aux | grep zeek`: `root 185757 0.0 0.0 6788 3152 ? S 15:45 0:00 bash /opt/zeek/share/zeekctl/scripts/run-zeek...` `root 185763 31.6 9.0 3393160 739104 ? Sl 15:45 1:20 /opt/zeek/bin/zeek -i ...` From this point if I try to start again with zeekctl it fails with the error message: `Failed to listen on INADDR_ANY:27760...` I think the problem comes from zeek processes not being killed properly by zeekctl I am not too much well versed into how zeek works but I guess: `/opt/zeek/share/zeekctl/scripts/run-zeek` process is the process that manages zeek and that takes orders from zeekctl command. If yes, should this process be killed when zeekctl stop? `/opt/zeek/bin/zeek` process is the main zeek process. This one should definitely be killed by `zeekctl sop` What exactly is ZeekPort? Is it the port that the main zeek process will listen to get its orders from run-zeek? Why is zeek-run necessary? Cannot zeekctl directly communicate with zeek? I am running a simple single node zeek. Thanks in advance ---
**Author:** ![awelzel](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/awelzel/32/609_2.png) [@awelzel](https://community.zeek.org/u/awelzel)\ **Post date:** [October 23, 2023, 3:32pm UTC](https://community.zeek.org/t/zeek-crashes-after-zeekctl-cron/7177/4 "2023-10-23T15:32:31Z")
> [@Lamorale](#): > > Also I am surprised that the zeekctl does not kill the zeek processes. > … > From this point if I try to start again with zeekctl it fails with the error message: > `Failed to listen on INADDR_ANY:27760...` Yeah, what you describe should not happen. Do you have another Zeek installation on the same system and possibly a crontab installation that restarts workers? What does `zeekctl ps.zeek` tell you? Here’'s it’s [code](https://github.com/zeek/zeekctl/blob/master/ZeekControl/plugins/ps.py) with a description of the output. > What exactly is ZeekPort? Is it the port that the main zeek process will listen to get its orders from run-zeek? It’s the “base” port used to allocate port numbers for all processes [Broker communication](https://docs.zeek.org/en/master/frameworks/broker.html). You will see port 27761, 27762, 27663, etc, allocated. > Why is zeek-run necessary? Cannot zeekctl directly communicate with zeek? zeekctl communicates with Zeek via the broker port, `zeek-run` should mostly sets-up the environment and records some information in the working directory. Does `zeekctl diag` indicate anything unusual after starting the nodes? ---
**Author:** ![Lamorale](https://avatars.discourse-cdn.com/v4/letter/l/bc79bd/32.png) [@Lamorale](https://community.zeek.org/u/Lamorale)\ **Post date:** [January 30, 2024, 2:38pm UTC](https://community.zeek.org/t/zeek-crashes-after-zeekctl-cron/7177/5 "2024-01-30T14:38:51Z")
First of all thanks for your response and sorry for not answering earlier to your message, the problem actually disappeared with the time and I did not investigate further because things were working… But the problem is back Here are the steps I perform: 1. Kill all seek process 2. Check with netstat port usage 3. Start zeek with /opt/zeek/bin/zeekctl start Zeek is actively writing connection logs in `/opt/zeek/logs/current` directory as expected (conn.log, etc…) However: `/opt/zeek/bin/zeekctl status ` shows `zeek standalone localhost crashed ` But `/opt/zeek/bin/zeekctl ps.zeek` shows `(-) root 24183 24177 42.7 5.3 3329432 658168 ? S 15:00:18 00:01:15 zeek ` The zeekctl status should not show `crashed` as zeek process is running and actively producing logs. Now when I stop zeek process with zeekctl: `/opt/zeek/bin/zeekctl stop ` That does no stop the process. Indeed ` ps aux | grep zeek` shows `root 24177 0.0 0.0 7036 2956 ? S 15:00 0:00 bash /opt/zeek/share/zeekctl/scripts/run-zeek -1 -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto` and `root 24183 41.0 5.4 3342672 671284 ? Sl 15:00 1:55 /opt/zeek/bin/zeek -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto` `/opt/zeek/bin/zeekctl ps.zeek` `(-) root 24183 24177 40.2 5.5 3356112 684488 ? S 15:00:18 00:02:29 zeek` If from this point I do zeekctl start it fails because of port already in use which is quite understandable Then I kill manually all zeek process and repeat steps 1,2 and 3 until a zeek process is running and writing output logs If I do `zeekctl cron`, zeek process stop writing logs in `/opt/zeek/logs/current` but the process still exist. Here is the output of `zeekctl diag` from this point: ```auto [zeek] No core file found. Zeek 6.0.3 Linux 6.1.0-17-amd64 Zeek plugins: (none found) ==== reporter.log ==== stderr.log bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) listening on ens224 error in /opt/zeek/spool/installed-scripts-do-not-touch/auto/standalone-layout.zeek, line 6: Failed to listen on INADDR_ANY:27760 (Broker::listen(Broker::default_listen_address, Broker::default_port, Broker::default_listen_retry)) fatal error: errors occurred while initializing ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i ens224 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek /opt/zeek/share/zeek/site/local.zeek zeekctl zeekctl/standalone zeekctl/auto ==== .env_vars PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins CLUSTER_NODE= ==== .status TERMINATED [fatal_error] ==== No prof.log ==== packet_filter.log ==== loaded_scripts.log bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) ``` Here is my zeekctl.cfg: ```auto MailTo = root@localhost MailConnectionSummary = 1 MinDiskSpace = 5 MailHostUpDown = 1 ############################################### # Logging Options LogRotationInterval = 3600 LogExpireInterval = 1 day StatsLogEnable = 1 StatsLogExpireInterval = 0 StatusCmdShowAll = 0 CrashExpireInterval = 0 SitePolicyScripts = /opt/zeek/share/zeek/site/local.zeek LogDir = /opt/zeek/logs SpoolDir = /opt/zeek/spool BrokerDBDir = /opt/zeek/spool/brokerstore CfgDir = /opt/zeek/etc ZeekPort = 27760 ``` It seems that communication between zeekctl and zeek process does not work as expected. Any help much appreciated.