First time Bro Cluster Spin up

Zeek
1

Hi.

I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). “fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied”

Thoughts on what I am missing?

Config:

I have an Endace DAG8.1SX set up to run with 22 streams.

1 Server set up to have 22 instances of BRO workers on it

1 server set up as the manager and prox

Using libpcap-1.6.2

dag-5.2.0.

bro version 2.3.1

TS:

  1. I have run the following command on both the worker and the manager/proxy servers.

sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro

  1. I can launch bro manually on the worker if I use sudo

acns-bro@endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42

listening on dag0:42, capture length 8192 bytes

^C1416254260.140036 received termination signal

1416254260.140036 209 packets received on interface dag0:42, 0 dropped

  1. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group

FULL OUTPUT of DIAG:

BroControl] > diag worker-21

[worker-21]

Bro 2.3.1

Linux 3.13.0-39-generic

==== No reporter.log

==== stderr.log

[dag_open] dag_clone dagfd for dagiom: Permission denied

fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied

==== stdout.log

max memory size (kbytes, -m) unlimited

data seg size (kbytes, -d) unlimited

virtual memory (kbytes, -v) unlimited

core file size (blocks, -c) unlimited

==== .cmdline

-i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

==== .env_vars

PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site

CLUSTER_NODE=worker-21

==== .status

TERMINATED [atexit]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

Scott Dawson
ACNS Network Security

Colorado State University
970-297-3712

“chop wood carry water”

2

Dawson,

Do you have any other processes listening to those streams? Only one process can attach to each stream, and we usually see those types of errors when a Bro worker tries to attach to a stream that is already in use. This also happens if a process doesn’t cleanly release the stream, and, in our experience, requires a full reboot to clear.

v/r

3

You must be running this on Linux as a user other than root (good for you). Run this command on each of the cluster nodes:

setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro && setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats

I'd give credit if I could recall where I found this, I found it on a website somewhere. I blame Seth Hall.

Note that you'll have to do this every time you push cluster config changes as well.

-- KS

Keith Schoenefeld
Information Security Analyst
Baylor University
254-710-6667

4

Thank you both John and Keith for the suggestions. I rebooted the server for good measure and re-ran the setcap command, with the same results in broctl.

worker-21 terminated immediately after starting; check output with “diag”

……

Diag worker-21

…

….

==== stderr.log

fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied

…….

Is anyone using an Endace card in there cluster? I am starting to suspect that the permissions issue has to do with how I am configuring and launching the capture card. I am going to reach out to their support organization and see if there is a way to launch the card as a non root user.

I am sure I can get all this working with root but I am trying to avoid that.

Thanks again

5

It was the Endace Capture card.

6

Just FYI, I run a DAG 9.2 here - in the past, I’ve run into situations where a crashing worker (or Snort process) will “lock” its stream. I usually do a cold power off in those situations, as a reboot didn’t always seem to clear it. I don’t know if that’s what you ran into, but it’s something to keep in mind anyway.

It’s not clear to me - did you manage to get it working as non-root then?

Mike

7

Mike,

Yes I did, or at least I don’t have to type sudo in front of the commands to launch and configure the endace card or bro anymore :slight_smile: .
I had to edit the file dagload and change the "group" from root to the group I was using.
I than had to change ownership on /dev/dag* and /usr/loca/bin/dag* files to the group also.

scott

"chop wood carry water"