I’ve been playing with some scripts in a local testing environment, and I’ve noticed that sometimes, a given Bro process won’t restart properly. I’ve seen this before in the past, but I’ve only started paying attention to it recently. In all cases where I’ve paid attention it’s been the worker process.
Below is a process listing I’ve just taken, where a ‘broctl restart’ was issued earlier in the day. This is running Bro 2.3.2, in a very simple configuration: one node is running manager, proxy, and worker; and there are two additional worker nodes in the cluster. Installation is done from source. In this case, none of the workers in the cluster were stopped properly.
Is this a known issue? Is there some kind of logging somewhere I can look at to see what’s going on? As far as I can tell, the broctl restart is succeeding without issue, but the old bro workers just aren’t exiting.
# ps auxw | grep -i bro
bro 6590 0.0 0.0 19596 1700 ? S Apr02 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth0 -U .status -p broctl -p broctl-live -p local -p worker-0 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
bro 6605 5.2 3.1 154976 122300 ? S Apr02 829:09 /opt/bro/bin/bro -i eth0 -U .status -p broctl -p broctl-live -p local -p worker-0 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
bro 6610 4.2 1.3 90300 53320 ? SN Apr02 672:40 /opt/bro/bin/bro -i eth0 -U .status -p broctl -p broctl-live -p local -p worker-0 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
root 12066 0.0 0.0 10464 948 pts/0 S+ 20:50 0:00 grep --color=auto -i bro
bro 22790 0.0 0.0 17992 1592 ? S 15:47 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
bro 22799 2.3 1.6 767060 62336 ? Sl 15:47 7:04 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
bro 22805 8.5 1.6 98588 61928 ? SN 15:47 25:47 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
bro 22822 0.0 0.0 17992 1596 ? S 15:47 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p proxy-0 local.bro broctl base/frameworks/cluster local-proxy broctl/auto
bro 22862 2.1 1.4 87424 54852 ? R 15:47 6:40 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy-0 local.bro broctl base/frameworks/cluster local-proxy broctl/auto
bro 22863 8.5 1.6 100556 63864 ? RN 15:47 26:02 /opt/bro/bin/bro -U .status -p broctl -p broctl-live -p local -p proxy-0 local.bro broctl base/frameworks/cluster local-proxy broctl/auto
bro 22890 0.0 0.0 17996 1600 ? S 15:47 0:00 bash /opt/bro/share/broctl/scripts/run-bro -1 -i eth0 -U .status -p broctl -p broctl-live -p local -p worker-0 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
bro 22900 6.0 2.1 115420 82684 ? S 15:47 18:28 /opt/bro/bin/bro -i eth0 -U .status -p broctl -p broctl-live -p local -p worker-0 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
bro 22905 4.2 1.3 90444 53700 ? SN 15:47 12:44 /opt/bro/bin/bro -i eth0 -U .status -p broctl -p broctl-live -p local -p worker-0 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto