bro cluster setup questions

Zeek
1

A while back I asked some questions about setting up Bro clusters in containers. Let me explain my setup a little and then I can ask my questions.

I have a cluster of like 40 machines. We have a ton of traffic that will be coming into that cluster and distributed to the NICs on all host hosts. So my plan was to have a worker on each host processing traffic and then sending that traffic to either a logger or a manager (given the size of the cluster, I think a separate logger is warranted). I also would have one ore more proxies.

We are using Mesos/Marathon to deploy containers onto the cluster. I can in theory pin containers to given hosts, although I would prefer to let Marathon manage that if I can. From my experience though, having a manager container and a logger container running on the same host means that Bro will ignore the logger and just use the manager (i.e., I find the logs being collected on the manager).

I’ve been initializing the cluster by having sshd running inside each of the containers and then running broctl install and deploy from inside the manager (and I have the sshd listening on 2022 instead of 22 since we already have sshd running on the machines the containers are running on - I have an ssh config inside of the manager container that makes 2022 the default port to use).

This will work fine as long as I only have one bro container on each host in my cluster. If I need to run more than one on any server, then it no longer makes sense. When I asked about this a while back, someone had mentioned that you can initialize the cluster without ssh. How do you do that exactly? Put the node.cfg on each of the hosts and then run broctl install on each? I’m a little confused on how each node knows what type of host it is supposed to be, other than consulting the node.cfg file and seeing the host/type relationship (but if you have the logger and proxy on the same host, how do they know which is which?).

The cluster configuration documentation (https://www.bro.org/sphinx-git/configuration/index.html) and the broctl documentation (https://www.bro.org/sphinx-git/components/broctl/README.html) doesn’t really make it clear to me how this works.

2

Yeah, you really should not be using broctl for what you are doing.

All broctl does is generate the /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro and use ssh to start the appropriate bro processes.

To start the processes broctl just sets the env var

CLUSTER_NODE=whatever

And runs the run-bro script with a bunch of options

#loggers
/usr/local/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/frameworks/cluster broctl/auto
#manager
/usr/local/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster broctl/auto
#proxy
/usr/local/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local -p proxy-2 local.bro broctl base/frameworks/cluster broctl/auto
#worker
/usr/local/bro/share/broctl/scripts/run-bro 15 -i myricom::p1p1:6 -U .status -p broctl -p broctl-live -p local -p worker-1 local.bro broctl base/frameworks/cluster broctl/auto

The main way they differ is by which prefix(-p) they are told to load, but I believe that doesn't actually do anything these days, so it would just boil down to 2 variations:

#loggers, managers, and proxies
/usr/local/bro/share/broctl/scripts/run-bro -1 -U .status -p broctl -p broctl-live -p local local.bro broctl base/frameworks/cluster broctl/auto
#workers
/usr/local/bro/share/broctl/scripts/run-bro -1 -i ethwhatever -U .status -p broctl -p broctl-live -p local local.bro broctl base/frameworks/cluster broctl/auto

The main tricky part is you'd probably have to tweak some of the stock cluster scripts so that the cluster can handle dynamic nodes that aren't listed in the cluster_layout.