multiple workers per cluster node

Zeek
1

Hopefully quick question. How would you go about configuring Bro cluster
nodes to each run dual clients (one per input interface)?

Ie, all of my systems have input sources on eth4 and eth5. Instead of
bonding those together and running a single Bro thread on bond0, I'd
rather have two. Something is getting super confused when I try to do it:

For each worker I have this:
[nids-21a]
type=worker
host=10.142.148.21
interface=eth4

[nids-21b]
type=worker
host=10.142.148.21
interface=eth5

[BroControl] > start
starting manager ...
starting proxy-1 ...
starting nids-21a ...
starting nids-21b ...
starting nids-22a ...
starting nids-22b ...
starting nids-23a ...
starting nids-23b ...
starting nids-24a ...
starting nids-24b ...
(nids-22a still initializing)
(nids-21b still initializing)
(nids-23b still initializing)
(nids-21a still initializing)

What's strange is that it seems to fail unevenly. Fails totally on 21,
partially on 22 and 23, but works on 24. It's always the same nodes
failing.

Thanks,
-Dop

2

Instead of

For each worker I have this:
[nids-21a]
type=worker
host=10.142.148.21
interface=eth4

[nids-21b]
type=worker
host=10.142.148.21
interface=eth5

Try:

For each worker I have this:
[nids-21]
type=worker
host=10.142.148.21
interface=eth4 -Ieth5

If you node had motile nodes you can write a pcap filter to split the ip space into multiples of 2,4 or 8 and run 2, 4, or 8 instance on the node.

This set up allow one bro instance to see by sides of the same flow and will allow you to take advanced of all the cpu on a node.

Bill Jones

3

This should work fine, I run 4 workers on one machine without any
issues.

It sounds like maybe you have some filesystem issues preventing bro from
starting.

What do you have in /usr/local/bro/spool/ for each of the failing nodes?
Is there anything in the stdout or stderr logs?

/usr/local/bro/spool/debug.log may also have useful info

I would focus on the machine that it starts partially on.

4

Do you get a crash message when a worker fails?

  .Seth

5

Thanks everyone for the replies and suggestions. Apparently I just forgot
to run 'install' after changing the node config which is embarrassing, but
I still find it interesting that they all reacted differently.

For future reference, all of the instances that fail show:

/usr/local/bro/share/bro/broctl/cluster-worker.remote.bro, line 14
(BroCtl::workers[WORKER]): run-time error, no such index
/usr/local/bro/share/bro/broctl/cluster-worker.remote.bro, line 13
($host=BroCtl::manager$ip, $p=BroCtl::manager$p,
$events=Remote::manager_events, $connect=T, $sync=F, $retry=1.0 min,
$class=BroCtl::workers[WORKER]$tag): run-time error, uninitialized list
value
/usr/local/bro/share/broctl/scripts/run-bro: line 73: 27140 Segmentation
fault (core dumped) nohup $tmpbro $@

-Dop

6

Try name the works

[worker-1]

[worker-2]

...

Bill Jones