Warning: "Bro node ... possibly still running"

Zeek
1

Issue #1: My node.cfg file specifies “type=standalone”, but I get a BroCtl warning that “Bro node ‘worker-1’ possibly still running on host…”.

Operating on Bro 2.4.1 and BroControl 1.4.

Background:

I configured a local cluster with one manager, one proxy, and two workers. Worker-1 is monitoring eth1, and worker-2 is monitoring eth2. The host was suffering too much packet loss, as indicated in the notice.log with the messages “PacketFilter::Dropped_Packets” and “CaptureLoss::Too_Much_Loss”. Therefore, I backed down from a local cluster, to just a standalone configuration in node.cfg. First, monitored only eth1 for a few days to observe packet loss, and then changed to monitor only eth2 for a few days. When I edit node.cfg and then run broctl, I get the following warnings:

Warning: broctl node config has changed (run the broctl “deploy” command)

Warning: Bro node “worker-1” possibly still running on host “localhost” (PID www)

Warning: Bro node “worker-2” possibly still running on host “localhost” (PID xxx)

Warning: Bro node “proxy” possibly still running on host “localhost” (PID yyy)

Warning: Bro node “manager” possibly still running on host “localhost” (PID zzz)

This is very curious that broctl “remembers” the previous node.cfg settings. Of course, none of the PIDs are valid anymore, because those processes were terminated when I changed from a cluster to standalone. But for some reason, broctl believes these processes might still be running. Where does BroCtl store this information?

Issue #2: Originally, when I changed node.cfg back to standalone, and then ran BroCtl “deploy” to implement the new configuration, the original manager, proxy, and worker processes were not terminated. BroCtl left these processes running, and then started a new set of processes for the new config. I discovered this a few days later because the notice.logs had entries from “bro” (standalone), and still was getting entries from “worker-1” and “worker-2” even though the cluster configuration was removed two days prior. I would run BroCtl “nodes” and it would correctly show that Bro is standalone monitoring eth1 only. I was confused. Finally, I ran process list on the host, and it revealed the original manager, proxy, and workers were all still running. To clear the situation, I ran BroCtl “stop”, then ran “kill -9” on every Bro-related PID, and then ran BroCtl “deploy”. This cleared away the issue of “worker-1” and “worker-2” from writing to the notice.logs; however, I still observe Issue #1, where BroCtl gives the warning messages that “Warning: Bro node … possibly still running".

I have a crontab to run BroCtl “cron” every five minutes. Does BroCtl “cron” affect how various configs are “remembered”? Should I disable that crontab item before making any changes to node.cfg and/or before running BroCtl “deploy”?

Thanks!

Mark I. Fernandez

2

In order to prevent this problem, you should run "broctl stop"
before removing (or renaming) any nodes in your node.cfg.

3

Daniel,

Thank you. To clarify, I should run broctl stop before I even edit the node.cfg file? I did not do so the first time. Bro was still running, I edited node.cfg, then ran broctl deploy. Indeed, while I was troubleshooting this issue, I tried every variation. I would run broctl stop, then edit node.cfg, then broctl deploy. This had no effect on the original manager, proxy and worker processes; and the only way to terminate these processes was to run "kill -9". Even earlier this morning, I ran broctl stop, edited node.cfg, and when I ran broctl, it gave the warnings.

Now that the damage is done, how do I undo this condition? I believe the system is monitoring and logging as intended, but for trust and confidence in the system state, I would like to clear away these warnings. Any advice on how to do so?

Mark