# Logging Issue after upgrade to LTS 5.0.0 **URL:** \ **Category:** Zeek\ **Tags:** cluster, upgrade\ **Created:** [July 19, 2022, 8:08pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613 "2022-07-19T20:08:24Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![akash](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@akash](https://community.zeek.org/u/akash)\ **Post date:** [July 19, 2022, 8:08pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/1 "2022-07-19T20:08:24Z")
I upgraded my zeek cluster to LTS 5.0 last week,since then the logs have stopped being generated after 6-7 hours of restart. Whenever I restart the cluster it logs for a few hours and then stops logging. There are no errors in the stderr log and diag output, everything seems normal, unable to figure it out. ---
**Author:** ![johanna](https://avatars.discourse-cdn.com/v4/letter/j/50afbb/32.png) [@johanna](https://community.zeek.org/u/johanna)\ **Post date:** [July 20, 2022, 11:52am UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/2 "2022-07-20T11:52:59Z")
Huh, this is weird, I never have heard of this specific error case before. Is the output of `zeekctl status` and `zeekctl peerstatus` also normal? Could you also check the output of `cluster.log` and `broker.log` - they might have information about this. Finally, can you try stopping/starting the logger node (leaving the other nodes running) and see if this gets logs flowing again? (That obviously is no great fix, but might help determining if the problem is solely on the side of the logger). ---
**Author:** ![redbaron](https://avatars.discourse-cdn.com/v4/letter/r/3bc359/32.png) [@redbaron](https://community.zeek.org/u/redbaron)\ **Post date:** [July 21, 2022, 11:13am UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/3 "2022-07-21T11:13:34Z")
Can you check output of `reporter.log`? Sometimes errors are reported there too. ---
**Author:** ![akash](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@akash](https://community.zeek.org/u/akash)\ **Post date:** [July 21, 2022, 2:21pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/4 "2022-07-21T14:21:43Z")
Thanks for the response. There are no errors in any logs, except one in reporter.log. zeekctl status, peer status all seems normal. _Reporter::ERROR no such index (Notice::tmp\_notice\_storage[Notice::uid]) /opt/zeek/share/zeek/policy/frameworks/notice/extend-email/hostnames.zeek, line 39_ _local.zeek_ _# Extend email alerting to include hostnames_ _@load policy/frameworks/notice/extend-email/hostnames_ _I’ve some local additions in local.zeek that too were working fine before the upgrade. I’ve restarted the cluster with the default configuration and it’s logging so far._ ---
**Author:** ![Tim\_Wojtulewicz](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/tim_wojtulewicz/32/594_2.png) [@Tim\_Wojtulewicz](https://community.zeek.org/u/Tim_Wojtulewicz)\ **Post date:** [July 28, 2022, 4:29pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/5 "2022-07-28T16:29:02Z")
Has this reproduced again? ---
**Author:** ![akash](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@akash](https://community.zeek.org/u/akash)\ **Post date:** [July 28, 2022, 5:00pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/6 "2022-07-28T17:00:08Z")
Hi Tim, Yes, it’s happening again. The cluster stopped logging/generating logs after a couple days. I see only these 3 logs, stats.log stderr.log stdout.log with no information. On manually stopping the cluster, it kills each worker process, normal stop is not happening. Due to lack of errors in logs, I’ve not been able to figure out the root cause of the issue. ---
**Author:** ![earlesswondercat](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/earlesswondercat/32/616_2.png) [@earlesswondercat](https://community.zeek.org/u/earlesswondercat)\ **Post date:** [July 28, 2022, 8:07pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/7 "2022-07-28T20:07:38Z")
Just so it’s in public (we’ve had some discussion on Slack) I’m seeing the same thing. Oddly, I have two clusters seeing the same traffic, same configuration, only one cluster is seeing this. Difference between clusters is the one seeing this is production has a physical manager + two physical worker systems (16 workers apiece); the testing one has a VM manager + one physical worker system (16 workers). In the production cluster I have commented out the _extend-email/hostnames.zeek_ from local.zeek and done a deploy. Previously I would see the no logs generated starting anywhere from 3 hours to 3 days after a deploy, so we’ll see. ---
**Author:** ![earlesswondercat](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/earlesswondercat/32/616_2.png) [@earlesswondercat](https://community.zeek.org/u/earlesswondercat)\ **Post date:** [July 28, 2022, 9:39pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/8 "2022-07-28T21:39:57Z")
That made no difference, it fell over again within about 2 hours of starting. ---
**Author:** ![akash](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@akash](https://community.zeek.org/u/akash)\ **Post date:** [July 28, 2022, 9:47pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/9 "2022-07-28T21:47:22Z")
Found an ‘out of memory’ error on all the nodes. The cluster was initially configured with jemalloc. Out of memory: Killed process 2566999 (zeek) total-vm:14135852kB, anon-rss:11044432kB, file-rss:129972kB, shmem-rss:0kB, UID:501 pgtables:25956kB oom\_score\_adj:0 oom\_reaper: reaped process 2566999 (zeek), now anon-rss:0kB, file-rss:131072kB, shmem-rss:0kB ---
**Author:** ![earlesswondercat](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/earlesswondercat/32/616_2.png) [@earlesswondercat](https://community.zeek.org/u/earlesswondercat)\ **Post date:** [July 28, 2022, 9:48pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/10 "2022-07-28T21:48:13Z")
I do occasionally have a worker OOM, I also use jemalloc. But - only on the production cluster, not the testing one. ☹ ---
**Author:** ![earlesswondercat](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/earlesswondercat/32/616_2.png) [@earlesswondercat](https://community.zeek.org/u/earlesswondercat)\ **Post date:** [July 28, 2022, 9:49pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/11 "2022-07-28T21:49:18Z")
I should say - the OOMs aren’t usually associated with the logs stopping. But if a worker crashes from an OOM, it doesn’t seem to restart correctly. ---
**Author:** ![earlesswondercat](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/earlesswondercat/32/616_2.png) [@earlesswondercat](https://community.zeek.org/u/earlesswondercat)\ **Post date:** [August 4, 2022, 5:45pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/12 "2022-08-04T17:45:47Z")
Tim and I spent some time with my production cluster again, and verified that everything _should_ be working, it just isn’t. Worker hosts are sending no traffic to the manager host. I’ve made my prod cluster be like my dev, with just one worker host now, and we’ll see if it repeats. ---
**Author:** ![Tim\_Wojtulewicz](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/tim_wojtulewicz/32/594_2.png) [@Tim\_Wojtulewicz](https://community.zeek.org/u/Tim_Wojtulewicz)\ **Post date:** [August 8, 2022, 5:07pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/13 "2022-08-08T17:07:36Z")
@akash If you’re not using them, could you try disabling the `known-*` scripts? Add the following to local.zeek: ```auto redef Known::use_host_store = F; redef Known::use_service_store = F; redef Known::use_cert_store = F ; ``` We’ve had some problems with those scripts recently, specifically in the round of testing for 5.0. We thought we’d ironed out all of the issues but early testing on @earlesswondercat’s system seems to indicate that they might be a culprit. ---
**Author:** ![akash](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@akash](https://community.zeek.org/u/akash)\ **Post date:** [August 8, 2022, 6:40pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/14 "2022-08-08T18:40:45Z")
Hello Tim, Thank you for the response. I reverted back to version 4 last week. We do use these scripts, however, I can test if disabling these scripts help in v5. ---
**Author:** ![awelzel](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/awelzel/32/609_2.png) [@awelzel](https://community.zeek.org/u/awelzel)\ **Post date:** [August 24, 2022, 4:02pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/15 "2022-08-24T16:02:40Z")
@akash - could you share a few details about your setup and infrastructure? Are you running workers, loggers, proxies on separate hosts or are all nodes co-located on the same physical host? We have a potential lead, but so far could only reproduce when workers and loggers are located on separate hosts. ---
**Author:** ![Scot](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@Scot](https://community.zeek.org/u/Scot)\ **Post date:** [August 24, 2022, 6:15pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/16 "2022-08-24T18:15:37Z")
This issue sounds like the issue I have run into. Posted in another thread with details there. Short description: Ubuntu 20.04 nodes. One node runs manager, logger, proxy. Other node is configured with four workers. Running zeek 5.0.0. I only get the following log files generated on the manager node: zeek@elkstack:/opt/zeek/logs/current$ ls -l total 12 -rw-rw-r-- 1 zeek zeek 1047 Aug 24 18:06 reporter.log -rw-rw-r-- 1 zeek zeek 1360 Aug 24 18:10 stats.log -rw-rw-r-- 1 zeek zeek 0 Aug 23 19:55 stderr.log -rw-rw-r-- 1 zeek zeek 188 Aug 23 19:55 stdout.log After some period of time the workers end up in crashed state. Estimate several hours before this happens. stats.log looks like data was collected. netstats, peerstats, and capstats look right. packets look like they are being seen. [ZeekControl] \> netstats worker-zeek-ch-1: 1661364847.281574 recvd=84983 dropped=36 link=85267 worker-zeek-ch-2: 1661364847.295818 recvd=85059 dropped=79 link=85259 worker-zeek-ch-3: 1661364847.308851 recvd=85114 dropped=29 link=85425 worker-zeek-ch-4: 1661364847.323851 recvd=85574 dropped=60 link=85835 [ZeekControl] \> netstats worker-zeek-ch-1: 1661364853.923669 recvd=236101 dropped=36 link=236400 worker-zeek-ch-2: 1661364853.937458 recvd=236191 dropped=79 link=236377 worker-zeek-ch-3: 1661364853.949551 recvd=236145 dropped=29 link=236362 worker-zeek-ch-4: 1661364853.963070 recvd=236599 dropped=60 link=236763 But no log files created and after some time the workers crash. This was a new install of the zeek software on both nodes. ---
**Author:** ![Scot](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@Scot](https://community.zeek.org/u/Scot)\ **Post date:** [August 24, 2022, 6:27pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/17 "2022-08-24T18:27:01Z")
I tried modifying local.zeek as mentioned in this thread. deployed after the change there is one difference noted so far. In the logs directory “current” a new file was created, loaded\_scripts.log was not there before. zeek@elkstack:/opt/zeek/logs/current$ ls -l total 44 -rw-rw-r-- 1 zeek zeek 697 Aug 24 18:22 broker.log -rw-rw-r-- 1 zeek zeek 742 Aug 24 18:22 cluster.log -rw-rw-r-- 1 zeek zeek 31014 Aug 24 18:22 loaded\_scripts.log -rw-rw-r-- 1 zeek zeek 0 Aug 24 18:22 stderr.log -rw-rw-r-- 1 zeek zeek 188 Aug 24 18:22 stdout.log zeek@elkstack:/opt/zeek/logs/current$ will monitor to see it starts logging data as expected. ---
**Author:** ![Scot](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@Scot](https://community.zeek.org/u/Scot)\ **Post date:** [August 24, 2022, 6:32pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/18 "2022-08-24T18:32:31Z")
Reporter.log file was created. Contents may point to an issue: zeek@elkstack:/opt/zeek/logs/current$ cat reporter.log #separator \x09 #set\_separator , #empty\_field (empty) #unset\_field - #path reporter #open 2022-08-24-18-28-04 #fields ts level message location #types time enum string string 1661365684.150128 Reporter::WARNING SumStat key request for the sMKulHLKb58 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226 1661365684.150128 Reporter::WARNING SumStat key request for the eWcinLxo4t1 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226 zeek@elkstack:/opt/zeek/logs/current$ ---
**Author:** ![akash](https://avatars.discourse-cdn.com/v4/letter/a/a5b964/32.png) [@akash](https://community.zeek.org/u/akash)\ **Post date:** [August 24, 2022, 6:41pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/19 "2022-08-24T18:41:35Z")
@awelzel We’ve 5 worker nodes each on a separate physical host, while logger, manager and proxy on one physical host. The cluster is working fine after I reverted to version 4. ---
**Author:** ![ThomasBrown](https://avatars.discourse-cdn.com/v4/letter/t/71e660/32.png) [@ThomasBrown](https://community.zeek.org/u/ThomasBrown)\ **Post date:** [May 9, 2023, 7:16pm UTC](https://community.zeek.org/t/logging-issue-after-upgrade-to-lts-5-0-0/6613/20 "2023-05-09T19:16:47Z")
I do use jemalloc and occasionally have a worker OOM. However, only on the production cluster; the testing cluster was not affected[.](https://mybankinginformation.com/citibank-routing-number/)