Zeek 5.0 cluster setup

Zeek
1

Just setup a zeek cluster using zeek 5.0.0. On Ubunto 20.04

Have the main system running the manager, proxy, and logger.

Collection system is configured for four workers.

Have the following output showing status of the systems:

[ZeekControl] > status
Name Type Host Status Pid Started
logger-1 logger elkstack.hollywood.local running 406359 23 Aug 19:55:27
manager manager elkstack.hollywood.local running 406418 23 Aug 19:55:28
proxy-1 proxy elkstack.hollywood.local running 406467 23 Aug 19:55:30
worker-zeek-ch-1 worker zeek-ch.hollywood.local running 15193 23 Aug 19:55:31
worker-zeek-ch-2 worker zeek-ch.hollywood.local running 15192 23 Aug 19:55:31
worker-zeek-ch-3 worker zeek-ch.hollywood.local running 15191 23 Aug 19:55:31
worker-zeek-ch-4 worker zeek-ch.hollywood.local running 15194 23 Aug 19:55:31
[ZeekControl] >

[ZeekControl] > capstats
Interface kpps mbps (10s average)

zeek-ch.hollywood.local/enp1s0f0np0 17.3 106.9
[ZeekControl] > netstats
worker-zeek-ch-1: 1661285242.494510 recvd=12810030 dropped=113 link=12810247
worker-zeek-ch-2: 1661285242.509398 recvd=12810283 dropped=65 link=12810550
worker-zeek-ch-3: 1661285242.524727 recvd=12810659 dropped=65 link=12810848
worker-zeek-ch-4: 1661285242.538205 recvd=12810710 dropped=26 link=12810812

[ZeekControl] > peerstatus
logger-1
1661285299.311641 peer=afa392dd-d474-5446-9533-1d6d78276345 host=10.1.5.104 status=Broker::CONNECTED
1661285299.311641 peer=b67a71bf-0cce-59f1-a7ba-2820a1766c2a host=10.1.5.104 status=Broker::CONNECTED
1661285299.311641 peer=7cc7782d-702d-5cf0-a878-ca40e6aef42d host=10.1.5.104 status=Broker::CONNECTED

manager

1661285299.326808 peer=003723c7-649b-5237-892c-b133599a70c1 host=10.1.5.104 status=Broker::CONNECTED
1661285299.326808 peer=b67a71bf-0cce-59f1-a7ba-2820a1766c2a host=10.1.5.104 status=Broker::CONNECTED
1661285299.326808 peer=928fb345-d118-536b-9be7-be0951dbc745 host=10.1.5.104 status=Broker::CONNECTED

proxy-1

1661285299.342610 peer=7f0a13e6-dd4f-5fa5-a571-3075f1d3a08b host=10.1.5.104 status=Broker::CONNECTED
1661285299.342610 peer=7cc7782d-702d-5cf0-a878-ca40e6aef42d host=10.1.5.104 status=Broker::CONNECTED
1661285299.342610 peer=928fb345-d118-536b-9be7-be0951dbc745 host=10.1.5.104 status=Broker::CONNECTED

worker-zeek-ch-1
1661285299.354560 peer=62c80d70-1840-50b4-bc6b-524db782bc75 host=10.1.161.104 status=Broker::CONNECTED
1661285299.354560 peer=4462e147-dc1a-5dbf-954d-bdc2bb22f69c host=10.1.161.104 status=Broker::CONNECTED

worker-zeek-ch-2
1661285299.370548 peer=4ce0d0b2-c666-57d7-9072-428566dbe340 host=10.1.161.104 status=Broker::CONNECTED
1661285299.370548 peer=4f505812-0af6-5470-a407-1d5beb4bd197 host=10.1.161.104 status=Broker::CONNECTED

worker-zeek-ch-3
1661285299.384764 peer=2a89be73-6a97-509f-81a1-dcb00c479324 host=10.1.161.104 status=Broker::CONNECTED
1661285299.384764 peer=67a0f57f-e5f2-5c12-80de-6d890a98d85b host=10.1.161.104 status=Broker::CONNECTED

worker-zeek-ch-4
1661285299.400222 peer=e56a3030-05dc-5f81-a209-794ac90f4378 host=10.1.161.104 status=Broker::CONNECTED
1661285299.400222 peer=ef7bc908-ca13-522c-af3d-0691c222e009 host=10.1.161.104 status=Broker::CONNECTED

[ZeekControl] >

It all looks like it should be working. However I am not getting any logs generated other than the following:

root@elkstack:/opt/zeek/logs/current# ls -l
total 16
-rw-rw-r-- 1 zeek zeek 760 Aug 23 20:08 broker.log
-rw-rw-r-- 1 zeek zeek 1483 Aug 23 20:11 reporter.log
-rw-rw-r-- 1 zeek zeek 1360 Aug 23 20:10 stats.log
-rw-rw-r-- 1 zeek zeek 0 Aug 23 19:55 stderr.log
-rw-rw-r-- 1 zeek zeek 188 Aug 23 19:55 stdout.log
root@elkstack:/opt/zeek/logs/current#

The stats.log file appears show packets collected. But no log files created.

root@elkstack:/opt/zeek/logs/current# cat stats.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path stats
#open 2022-08-23-20-00-27
#fields ts peer mem pkts_proc bytes_recv pkts_dropped pkts_link pkt_lag events_proc events_queued active_tcp_conns active_udp_conns active_icmp_conns tcp_conns udp_conns icmp_conns timers active_timers files active_files dns_requests active_dns_requests reassem_tcp_sizereassem_file_size reassem_frag_size reassem_unknown_size
#types time string count count count count count interval count count count count count count count count count count count count count count count count count count
1661284827.733631 logger-1 108 0 0 - - - 826 340 0 0 00 0 0 1687 45 0 0 0 0 0 0 0 0
1661284829.131527 manager 110 0 0 - - - 826 345 0 0 0 00 0 1506 49 0 0 0 0 0 0 0 0
1661284830.531431 proxy-1 108 0 0 - - - 826 340 0 0 0 00 0 1385 44 0 0 0 0 0 0 0 0
1661285127.734043 logger-1 108 0 0 - - - 331 333 0 0 00 0 0 1683 45 0 0 0 0 0 0 0 0
1661285129.131942 manager 110 0 0 - - - 342 341 0 0 0 00 0 1508 52 0 0 0 0 0 0 0 0
1661285130.531945 proxy-1 108 0 0 - - - 331 333 0 0 0 00 0 1382 43 0 0 0 0 0 0 0 0
1661285427.734219 logger-1 108 0 0 - - - 339 337 0 0 00 0 0 1685 47 0 0 0 0 0 0 0 0
1661285429.132044 manager 110 0 0 - - - 341 341 0 0 0 00 0 1501 48 0 0 0 0 0 0 0 0
1661285430.532142 proxy-1 108 0 0 - - - 338 337 0 0 0 00 0 1384 44 0 0 0 0 0 0 0 0
root@elkstack:/opt/zeek/logs/current#

After a while (hours) the worker processes crash.

Any ideas on what was missed during setup?

Thank you.

2

I saw your reply on the other thread. Replying here as I didn’t make an immediate connection. You don’t seem to get any logs from the workers at all. One hunch might be that worker->logger connectivity is impaired.

Could you ensure the zeek worker host is able to connect to the manager, logger and proxy ports by running the following on the worker host in a shell:

# Check logger connectivity
$ nc -v -z <ip of manger/logger/proxy>  47761
# Check manager connectivity
$ nc -v -z <ip of manger/logger/proxy> 47762
# ... (47763... for the following proxies)

Maybe there’s some firewall setup preventing workers to connect to loggers?

Otherwise, zeekctl diag or looking into /opt/zeek/spool/<worker...> on the worker host may give some clues - look for stdout/stderr output specifically.

3

Thank you for the reply. Have tried the tests you recommended.

Results below.

Connectivity appears to be good between the workers and the main system.

zeek@zeek-ch:~$ nc -v -z elkstack.hollywood.local 47761

Connection to elkstack.hollywood.local 47761 port [tcp/*] succeeded!

zeek@zeek-ch:~$ nc -v -z elkstack.hollywood.local 47762

Connection to elkstack.hollywood.local 47762 port [tcp/*] succeeded!

zeek@zeek-ch:~$ nc -v -z elkstack.hollywood.local 47763

Connection to elkstack.hollywood.local 47763 port [tcp/*] succeeded!

zeek@zeek-ch:~$

First thing I checked with the firewall settings under Ubuntu. And by default the firewall is not enabled.

I did make the changes suggested in the other thread. The workers have not crashed since then.

Still no additional log files.

From the machine running the workers:

zeek@zeek-ch:/opt/zeek/spool$ ls -l

total 48

drwxr-xr-x 2 zeek zeek 4096 Jan 28 2015 brokerstore

drwxrwsr-x 4 zeek zeek 4096 Jul 28 21:19 installed-scripts-do-not-touch

-rw-r–r-- 1 zeek zeek 12288 Jul 28 18:16 state.db

drwxrws— 58 zeek zeek 4096 Aug 24 19:38 tmp

drwxrwsr-x 2 zeek zeek 4096 Aug 3 21:22 worker-1

drwxrwsr-x 2 zeek zeek 4096 Aug 24 19:38 worker-zeek-ch-1

drwxrwsr-x 2 zeek zeek 4096 Aug 24 19:38 worker-zeek-ch-2

drwxrwsr-x 2 zeek zeek 4096 Aug 24 19:38 worker-zeek-ch-3

drwxrwsr-x 2 zeek zeek 4096 Aug 24 19:38 worker-zeek-ch-4

-rw-rw-r-- 1 zeek zeek 2751 Aug 24 19:38 zeekctl-config.sh

zeek@zeek-ch:/opt/zeek/spool$ cd worker-zeek-ch-1

zeek@zeek-ch:/opt/zeek/spool/worker-zeek-ch-1$ ls

stderr.log stdout.log

zeek@zeek-ch:/opt/zeek/spool/worker-zeek-ch-1$ ls -l

total 8

-rw-rw-r-- 1 zeek zeek 26 Aug 24 19:38 stderr.log

-rw-rw-r-- 1 zeek zeek 188 Aug 24 19:38 stdout.log

zeek@zeek-ch:/opt/zeek/spool/worker-zeek-ch-1$ cat stderr.log

listening on enp1s0f0np0

zeek@zeek-ch:/opt/zeek/spool/worker-zeek-ch-1$ cat stdout.log

max memory size (kbytes, -m) unlimited

data seg size (kbytes, -d) unlimited

virtual memory (kbytes, -v) unlimited

core file size (blocks, -c) unlimited

zeek@zeek-ch:/opt/zeek/spool/worker-zeek-ch-1$

And the netstats and capstats reports look like packets are being captured.

[ZeekControl] > netstats

worker-zeek-ch-1: 1661426596.285219 recvd=491498494 dropped=117 link=491498840

worker-zeek-ch-2: 1661426596.300317 recvd=491498886 dropped=131 link=491499123

worker-zeek-ch-3: 1661426596.314334 recvd=491499174 dropped=150 link=491499459

worker-zeek-ch-4: 1661426596.330405 recvd=491499453 dropped=150 link=491499721

[ZeekControl] > capstats

Interface kpps mbps (10s average)

4

Remainder of the diag output.

[ZeekControl] > netstats
worker-zeek-ch-1: 1661426596.285219 recvd=491498494 dropped=117 link=491498840
worker-zeek-ch-2: 1661426596.300317 recvd=491498886 dropped=131 link=491499123
worker-zeek-ch-3: 1661426596.314334 recvd=491499174 dropped=150 link=491499459
worker-zeek-ch-4: 1661426596.330405 recvd=491499453 dropped=150 link=491499721
[ZeekControl] > capstats
Interface kpps mbps (10s average)

zeek-ch.hollywood.local/enp1s0f0np0 13.6 95.9
[ZeekControl] >

Diag output below:

[ZeekControl] > diag
[logger-1]

No core file found.

Zeek 5.0.0
Linux 5.4.0-122-generic

Zeek plugins: (none found)

==== reporter.log
#open 2022-08-25-11-04-49
#fields ts level message location
#types time enum string string
1661425488.134852 Reporter::WARNING SumStat key request for the 1x1w5SRa0u2 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226
1661425488.134852 Reporter::WARNING SumStat key request for the 65iumpwx79l SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226
1661425788.160373 Reporter::WARNING SumStat key request for the c0pNiJleY19 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226
1661425788.160373 Reporter::WARNING SumStat key request for the 8QHkJgE6YE1 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226
1661425788.160373 Reporter::WARNING SumStat key request for the Vz8XKQb9MTi SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226
1661426088.184013 Reporter::WARNING SumStat key request for the SBToQ9ds6r2 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226
1661426088.184013 Reporter::WARNING SumStat key request for the fR7pa6SFoQ1 SumStat uid took longer than 1 minute and was automatically cancelled. /opt/zeek/share/zeek/base/frameworks/sumstats/./cluster.zeek, line 226

==== stderr.log

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p logger-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/opt/zeek/bin:/opt/zeek/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/zeek/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=logger-1

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[manager]

No core file found.

Zeek 5.0.0
Linux 5.4.0-122-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p manager local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/opt/zeek/bin:/opt/zeek/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/zeek/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=manager

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[proxy-1]

No core file found.

Zeek 5.0.0
Linux 5.4.0-122-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-U .status -p zeekctl -p zeekctl-live -p local -p proxy-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/opt/zeek/bin:/opt/zeek/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/zeek/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=proxy-1

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[worker-zeek-ch-1]

No core file found.

Zeek 5.0.0
Linux 5.4.0-124-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log
listening on enp1s0f0np0

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i enp1s0f0np0 -U .status -p zeekctl -p zeekctl-live -p local -p worker-zeek-ch-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=worker-zeek-ch-1

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[worker-zeek-ch-2]

No core file found.

Zeek 5.0.0
Linux 5.4.0-124-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log
listening on enp1s0f0np0

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i enp1s0f0np0 -U .status -p zeekctl -p zeekctl-live -p local -p worker-zeek-ch-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=worker-zeek-ch-2

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[worker-zeek-ch-3]

No core file found.

Zeek 5.0.0
Linux 5.4.0-124-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log
listening on enp1s0f0np0

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i enp1s0f0np0 -U .status -p zeekctl -p zeekctl-live -p local -p worker-zeek-ch-3 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=worker-zeek-ch-3

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[worker-zeek-ch-4]

No core file found.

Zeek 5.0.0
Linux 5.4.0-124-generic

Zeek plugins: (none found)

==== No reporter.log

==== stderr.log
listening on enp1s0f0np0

==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited

==== .cmdline
-i enp1s0f0np0 -U .status -p zeekctl -p zeekctl-live -p local -p worker-zeek-ch-4 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

==== .env_vars
PATH=/opt/zeek/bin:/opt/zeek/share/zeekctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ZEEKPATH=/opt/zeek/spool/installed-scripts-do-not-touch/site::/opt/zeek/spool/installed-scripts-do-not-touch/auto:/opt/zeek/share/zeek:/opt/zeek/share/zeek/policy:/opt/zeek/share/zeek/site:/opt/zeek/share/zeek/builtin-plugins
CLUSTER_NODE=worker-zeek-ch-4

==== .status
RUNNING [run_loop]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

[ZeekControl] >

5

@Scot - thanks for double checking the firewall and connectivity.

I’m not sure what’s going on. As a further test, could you you migrate logger-1 to the zeek-ch.hollywood.local box (so that it runs on the same system as the workers).

Then, on the zeek-ch.hollywood.local.box look into /opt/zeek/logs/current. If that works, then it might actually be related to the other thread about logging and 5.0.

Otherwise, my only thoughts are:

  • On the worker host, if possible, capture a pcap from the monitoring interface and run zeek -r <pcap> in offline mode to verify some logs are created. I suspect that works, your stats.log not showing any entries for workers is more likely the culprit.
  • Cross check with Zeek 4.2 or Zeek 4.0 if that works in your environment.
6

Setup as you suggested, put logger on the monitoring system.

[ZeekControl] > status

Name Type Host Status Pid Started

logger-1 logger zeek-ch.hollywood.local running 26928 25 Aug 14:04:34

manager manager elkstack.hollywood.local running 448494 25 Aug 14:04:35

proxy-1 proxy elkstack.hollywood.local running 448543 25 Aug 14:04:36

worker-zeek-ch-1 worker zeek-ch.hollywood.local running 27018 25 Aug 14:04:38

worker-zeek-ch-2 worker zeek-ch.hollywood.local running 27022 25 Aug 14:04:38

worker-zeek-ch-3 worker zeek-ch.hollywood.local running 27023 25 Aug 14:04:38

worker-zeek-ch-4 worker zeek-ch.hollywood.local running 27024 25 Aug 14:04:38

[ZeekControl] >

7

Hey - now that logger-1 is running on zeek-ch, do you see logs produced on that system in /opt/zeek/logs/current ?

8

Yes, I see logs created on the monitor server.

-rw-rw-r-- 1 zeek zeek 2674 Aug 29 06:00 tunnel.05:00:00-06:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 3018 Aug 29 07:00 tunnel.06:00:00-07:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 2617 Aug 29 08:00 tunnel.07:00:00-08:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 2704 Aug 29 09:00 tunnel.08:00:00-09:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 2675 Aug 29 10:00 tunnel.09:00:00-10:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 2881 Aug 29 11:00 tunnel.10:00:00-11:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4579054 Aug 29 01:00 weird.00:00:00-01:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4648035 Aug 29 02:00 weird.01:00:00-02:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4332318 Aug 29 03:00 weird.02:00:00-03:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4454148 Aug 29 04:00 weird.03:00:00-04:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4607502 Aug 29 05:00 weird.04:00:00-05:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4396381 Aug 29 06:00 weird.05:00:00-06:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4415583 Aug 29 07:00 weird.06:00:00-07:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4407511 Aug 29 08:00 weird.07:00:00-08:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4308858 Aug 29 09:00 weird.08:00:00-09:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 4449260 Aug 29 10:00 weird.09:00:00-10:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 5380639 Aug 29 11:00 weird.10:00:00-11:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 1778 Aug 29 01:00 x509.00:00:00-01:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 3551 Aug 29 02:00 x509.01:00:00-02:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 861 Aug 29 03:00 x509.02:00:00-03:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 2739 Aug 29 04:00 x509.03:00:00-04:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 1040 Aug 29 05:00 x509.04:00:00-05:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 2100 Aug 29 06:00 x509.05:00:00-06:00:00.log.gz

-rw-rw-r-- 1 zeek zeek 1059 Aug 29 07:00 x509.06:00:00-07:00:00.log.gz

image001.png

9

Hmm, well. Not sure that’s great, as it’s surprising. In the stats.log, do you continue to see entries for the manager/proxies that are still running on the zeek-ch hosts? When moving back the logger to zeek-ch is logging impaired again?

Did you do any notable configuration changes in node.cfg, zeekctl.cfg or local.zeek?

Otherwise, I’m a bit out of ideas. Testing on Zeek 4.0 or 4.2 would be helpful to us to determine if it’s a Zeek 5.0 issue, or something about your environment/infrastructure. Would that be possible?

10

Note: when logger was on zeek-ch monitor server (typically workers only) no current directory was created. The archive directories were being created and log files for the standard stuff was being collected on the monitor server.

Just moved logger back to the main system (no workers). Current folder was created and the five files that appeared before were created. Will wait for a few hours to see if any of the other log files are created.

No notable changes to the config files other than adding the setcap plugin so that is set on all remote nodes when zeek is reloaded.

Have not added any other plugins yet to the system.

11

Was hoping to setup these new servers using 5.0.

If 5.0 does not work I will schedule time to remove it and replace with 4.2 if that is the best release to downgrade to.

Thank you.

12

Do you have an update or have some other findings?

It is not the best release to downgrade to - running 5.0 should be preferred. Trying 4.2 would primarily give a data point if a previous release works as expected.

13

Agreed, I would rather run 5.0 or newer if available.

I tried putting a worker on the manager server to see what that would do. It created logs for traffic on the port for the new worker. But I don’t see any traffic in the logs from the workers on the other node in the cluster.

[ZeekControl] > status
Name Type Host Status Pid Started
logger-1 logger elkstack.hollywood.local running 632747 01 Sep 10:51:12
manager manager elkstack.hollywood.local running 632797 01 Sep 10:51:14
proxy-1 proxy elkstack.hollywood.local running 632846 01 Sep 10:51:15
worker-elkstack-1 worker elkstack.hollywood.local running 632894 01 Sep 10:51:16
worker-zeek-ch-1 worker zeek-ch.hollywood.local running 199766 01 Sep 10:51:16
worker-zeek-ch-2 worker zeek-ch.hollywood.local running 199764 01 Sep 10:51:16
worker-zeek-ch-3 worker zeek-ch.hollywood.local running 199763 01 Sep 10:51:16
worker-zeek-ch-4 worker zeek-ch.hollywood.local running 199762 01 Sep 10:51:16
[ZeekControl] >

How does the logger work in a cluster?

Does it establish communications from the manager/logger to nodes with workers?

Or do the workers establish the connection back to the logger to send their data?

It appears that the elkstack server has connectivity to the zeek-ch server, status works, capstats works, netstats works.

But the data that should be going to the logs is not showing up on the server with logger on it.

So far the logs/error files I have looked at don’t show anything.