Cluster configuration zeekctl status hangs

Zeek
1

Install Zeek 3.0 on Centos 8.

Have been working through the setup of zeek using two machines in a cluster.

The cluster appears to be working.

I can zeekctl install and zeekctl start the cluster.

On the remote machine I see the workers start up.

On the local machine the services and workers appear to startup.

Remote machine:

zeek 25985 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 3 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-3-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 25986 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 2 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-3-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 25990 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 4 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-4-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 25992 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 5 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-4-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 26012 25985 9 08:58 ? 00:01:31 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-3-2 local.zee zeekctl base/frameworks/cluster zeekctl/auto

zeek 26013 25986 9 08:58 ? 00:01:31 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-3-1 local.zee zeekctl base/frameworks/cluster zeekctl/auto

zeek 26016 25992 9 08:58 ? 00:01:31 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-4-2 local.zee zeekctl base/frameworks/cluster zeekctl/auto

zeek 26017 25990 9 08:58 ? 00:01:30 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-4-1 local.zee zeekctl base/frameworks/cluster zeekctl/auto

Local (manager) machine:

zeek 8314 1 0 08:57 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek -1 -U .status -p zeekctl -p zeekctl-live -p local -p logger local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8320 8314 5 08:57 ? 00:00:58 /opt/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p logger local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8361 1 0 08:57 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek -1 -U .status -p zeekctl -p zeekctl-live -p local -p manager local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8367 8361 5 08:57 ? 00:00:59 /opt/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p manager local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8406 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek -1 -U .status -p zeekctl -p zeekctl-live -p local -p proxy-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8412 8406 1 08:58 ? 00:00:21 /opt/zeek/bin/zeek -U .status -p zeekctl -p zeekctl-live -p local -p proxy-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8471 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 2 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8474 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 3 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8477 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 5 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8479 1 0 08:58 ? 00:00:00 /usr/bin/bash /opt/zeek/share/zeekctl/scripts/run-zeek 4 -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8499 8471 17 08:58 ? 00:03:09 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8502 8474 21 08:58 ? 00:03:47 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-1-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8503 8477 17 08:58 ? 00:03:09 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-2 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8504 8479 18 08:58 ? 00:03:17 /opt/zeek/bin/zeek -i af_packet::eno1 -U .status -p zeekctl -p zeekctl-live -p local -p worker-2-1 local.zeek zeekctl base/frameworks/cluster zeekctl/auto

zeek 8593 3011 6 08:58 pts/0 00:01:04 /usr/bin/python3.6 /opt/zeek/bin/zeekctl status

The problem is that when I run zeekctl status that request hangs:

[zeek@heimdallr etc]$ zeekctl status

Warning: ZeekControl plugin uses legacy BroControl API. Use

‘import ZeekControl.plugin’ instead of ‘import BroControl.plugin’

Getting process status …

Getting peer status …

Only way to resolve this is to kill process 8593.

Any ideas on why this is hanging?

Secondary problem with a work around available:

Also have to follow the following steps for the cluster to work.

  1. zeekctl install

  2. setcap cap_net_raw=eip /opt/zeek/bin/zeek (on the remote peer)

  3. zeekctl start

Attempts to use zeekctl deploy does not work as the setcap command needs to be run on the remote peer after the install is completed.

Running zeek 3.0.

2

The problem is that when I run zeekctl status that request hangs:

[zeek@heimdallr etc]$ zeekctl status

Warning: ZeekControl plugin uses legacy BroControl API. Use

‘import ZeekControl.plugin’ instead of ‘import BroControl.plugin’

Getting process status …

Getting peer status …

Only way to resolve this is to kill process 8593.

Any ideas on why this is hanging?

Odd that it’s even doing that… did you change this option in zeekctl.cfg?

Show all output of the zeekctl status command. If set to 1, then all output

is shown. If set to 0, then zeekctl status will not collect or show the peer

information (and the command will run faster).

StatusCmdShowAll = 0

The default is to skip the “peer status” stuff, which causes zeekctl to connect to each worker on the broker port. You may have firewall rules or something preventing this from working. Does the zeekctl netstats command also hang?

Secondary problem with a work around available:

Also have to follow the following steps for the cluster to work.

  1. zeekctl install

  2. setcap cap_net_raw=eip /opt/zeek/bin/zeek (on the remote peer)

  3. zeekctl start

Attempts to use zeekctl deploy does not work as the setcap command needs to be run on the remote peer after the install is completed.

This should do what you want: https://github.com/PingTrip/broctl-setcap

3

Justin,

That option did resolve the status problem I was seeing.

What peer data is it trying to pull that causes it to hang?

Now get the expected results:

[zeek@heimdallr etc]$ zeekctl status

Warning: ZeekControl plugin uses legacy BroControl API. Use

‘import ZeekControl.plugin’ instead of ‘import BroControl.plugin’

Name Type Host Status Pid Started

logger logger 10.1.1.15 running 18323 03 Dec 10:26:15

manager manager 10.1.1.15 running 18370 03 Dec 10:26:16

proxy-1 proxy 10.1.1.15 running 18415 03 Dec 10:26:17

worker-1-1 worker 10.1.1.15 running 18505 03 Dec 10:26:19

worker-1-2 worker 10.1.1.15 running 18501 03 Dec 10:26:19

worker-2-1 worker 10.1.1.15 running 18506 03 Dec 10:26:19

worker-2-2 worker 10.1.1.15 running 18507 03 Dec 10:26:19

worker-3-1 worker 10.1.7.186 running 28032 03 Dec 10:26:19

worker-3-2 worker 10.1.7.186 running 28033 03 Dec 10:26:19

worker-4-1 worker 10.1.7.186 running 28035 03 Dec 10:26:19

worker-4-2 worker 10.1.7.186 running 28036 03 Dec 10:26:19

Will try the other fix shortly.

Thank you!

Scot

4

Justin,

Was able to get that setcap script to work.

Required editing to get paths correct and remove extras that were not required.

But it does work now!

Thank you.