all broctl instances are running yet broctl status shows stopped

Zeek
1

Hi all,

When running bro release 2.5 in cluster mode (manager,proxy and several workers) I have a strange issue :

new logs are written to spool/manager and according to pgrep -c bro all instances are running, yet “broctl status” shows that all instances are stopped.

For some reason about an hour after using the “broctl deploy”, this issue occurs again.

Any thoughts on what might cause this behavior ?

I see no fix for this on newer versions.

Thank you

B

2

I've seen this as well, with the additional behavior that if I attempt a "broctl start", the processes all immediately crash.

I have had some success with doing "broctl stop", manually killing the remaining (unmanaged?) processes on each listener, *then* doing a "broctl start".

N.B. We are woefully behind, at release 2.3 .

   -g

3

When this happens, what does the output of "broctl diag" look like?

4

Hi

Thanks for your replies

broctl diag returns "HINT : Run broctl deploy to get started.

All the rest of the output is not populated

broctl deploy solves this issue, but I do not want to restart my cluster every hour

B

5

OK, that is what I expected. You have two different copies of Bro
installed on your system (doesn't matter if they are the same
version or not), and I recommend removing one of them to
avoid confusion.

This problem could happen, for example, if you have two copies
of Bro installed and you run "sudo broctl deploy", but then later
you run "broctl status" and this actually runs the other copy (on
most systems, "sudo" uses a different PATH than normal users).
Each installation of Bro includes its own scripts, config files,
executables, state file, etc.

6

Hi Daniel,

Thanks

I have deleted another bro environment on that server and doubled checked that there are no other broctl\bro executable besides the work_dir and build_dir.

Yet this issue still occures.

I run bro with a specific user and on “top” I see that bro is running under that user, yet “./bin/broctl status” still returns that all instances are stopped.

Any suggestions ?

Thanks again

B

7

When you run "broctl diag", it will output the contents of several files
in the Bro working directory (this is the directory where bro is
running). For example, it will show you the contents of the
".status" file and "stdout.log", and several other files.

If you don't see anything in the output, but you are sure that
bro is running (and producing logs), then that means
bro is running in a different directory.

Each installation of bro uses its own directory paths for
locations of the config files, working directory, executables, etc.
You can see these by running "broctl config". You can check
if the output of "broctl config | grep spooldir" is the
parent directory of the directory where you are seeing bro
producing log files.