Bro hanging on some sensors

I have several SecurityOnion sensors and most are working ok. There are a couple that I see the below problem on with Bro.

The /nsm/bro/spool/manager/communication.log file shows the below in it on each of the problem sensors:
    1402520922.886012 manager parent - - - info warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012 manager parent - - - info warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012 manager parent - - - info warning: cannot increase SO_SNDBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012 manager parent - - - info warning: cannot increase SO_RCVBUF socket buffer size from 16384K (1024K was requested)
    1402520922.886012 manager parent - - - info communication started, parent pid is 3646, child pid is 3660

I only ever see these files created in the Bro log working directory:
    communication.log
    loaded_scripts.log
    eporter.log
    stderr.log
    stdout.log

Most of the sensors are configured exactly the same both software and hardware wise; so I'm not seeing a correlation there as yet. I've tried rebooting and using broctl commands and so far no resolution. Many time "broctl check" will hang. I have all the latest patches on SO installed. Any help would be appreciated.

The only major change I've made in the last month is to add a few Intel feeds.

Thanks,
Brian

Hi Brian,

How exactly did you add your Intel feeds?

I use a cron job that runs every 30 minutes to download the intel files to:
/opt/bro/share/bro/policy/. The cron job uses the mal-dnssearch script.

In each sensor's /opt/bro/share/bro/site/local.bro file I have the below:

# load intelligence framework
@load policy/frameworks/intel/seen
@load policy/frameworks/intel/do_notice
#@load policy/integration/collective-intel
redef Intel::read_files += {
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
        "/opt/bro/share/bro/policy/xxx.intel",
};

In the reporter.log file I am now seeing the below warning on the four sensors
having this issue:
    0.000000 Reporter::WARNING SumStat key request for the
3Mntn3EPhU3 SumStat uid took longer than 1 minute and was automatically
cancelled. /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
209
    0.000000 Reporter::WARNING SumStat key request for the
2HAva5N4Kqf SumStat uid took longer than 1 minute and was automatically
cancelled. /opt/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line
209

Thanks,
Brian

Did these sensors work correctly before you added the Intel files?

Have you tried removing the Intel files from the equation? If not,
please try commenting out the entries you added to
/opt/bro/share/bro/site/local.bro and then running:
sudo broctl install
sudo broctl restart

Are you running OnionSalt? Do you have OnionSalt configured to
automatically restart Bro? There are known issues with doing so,
which is why OnionSalt does not do so by default. Please see:
https://code.google.com/p/security-onion/wiki/Salt#Features

This may not be an issue strictly related to Bro, so we may need to
move this conversation to the security-onion mailing list:
https://code.google.com/p/security-onion/wiki/MailingLists

Yeah they were working up to about a week ago. I commented out the intel
files on one and that fixed it. Odd, as they should all be the same with the
replication from the SO server policy directory. The majority of the sensors
are working fine with the same intel files and I'm seeing new results in ELSA
every day. Hmmm

I do not have Bro restarting via Onion-Salt. I was under the understanding
that Bro would periodically sense the file changes and re-read the intel
files.

Thanks,
Brian

Thank you,
Brian Kellogg
Security Analyst; IT Governance, Risk, and Compliance
500 Paul Clark Drive, Olean, NY 14760
T: (716) 375-3186 | F: (716) 375-3557
www.dresser-rand.com NYSE: DRC

Bringing energy and the environment into harmony®
IMPORTANT NOTICE:
This email may be confidential, may be legally privileged, and is for the intended recipient only. Unauthorized access, disclosure, copying, distribution, or reliance on any of it by anyone else is prohibited and may be a criminal offense. Please delete if obtained in error and email confirmation to the sender.

Bro will sense additions but won’t remove old stuff unless it is restarted check out “Putting it all together” in the blog post below.

http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html

Thanks

I’m not seeing any rhyme or reason to this. I’ve tried commenting out one intel feed at a time and the problem will disappear on some sensors and start on others. I’ve commented them all out and enabled one at a time with the same results.

The only thing that works is to disable the intel framework entirely so far.

When you comment out intel feeds, are you installing the new config
before restarting?
sudo broctl install
sudo broctl restart

Are you sure your intel feeds are formatted properly?
http://www.bro.org/sphinx-git/frameworks/intel.html

Any other errors or warnings in reporter.log, stderr.log, or stdout.log?

Yep, I run the install prior to the restart. The format in the files is set by the mal-dnssearch script; I checked the files to ensure they are tab delimited and they are. I get no errors about Bro not seeing the intel files in the correct format. I have seen these before when creating our own custom intel file. Our custom intel file is maintained via a python script to ensure that tab delimiting is always maintained.

The one consistent thing I see is that when I stop, install, and then start Bro, Bro starts ok and all the appropriate logs are created. If I stop and restart Bro again then the only logs I see in the “current” directory are: communication, loaded_scripts, reporter, stderr, and stdout.

I have the intel config loaded at the very end of the local.bro file if that makes a difference.

There are not errors that I see in reporter.log, stderr.log, or stdout.log as shown below.

drc@nsm-che:/nsm/bro/logs/current$ cat stderr.log

/opt/bro/share/bro/securityonion/./bpfconf.bro, line 103: BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf

/opt/bro/share/bro/securityonion/./bpfconf.bro, line 103: BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf

drc@nsm-che:/nsm/bro/logs/current$ cat stdout.log

unlimited

unlimited

unlimited

drc@nsm-che:/nsm/bro/logs/current$ cat reporter.log

#separator \x09

#set_separator ,

#empty_field (empty)

#unset_field -

#path reporter

#open 2014-06-17-14-20-55

#fields ts level message location

#types time enum string string

0.000000 Reporter::WARNING Template value remaining in BPFConf filename: /etc/nsm/{{hostname}}-{{interface}}/bpf-bro.conf /opt/bro/share/bro/securityonion/./bpfconf.bro, line 99

0.000000 Reporter::INFO BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103

0.000000 Reporter::INFO BPFConf filename set: /etc/nsm/nsm-che-eth0/bpf-bro.conf /opt/bro/share/bro/securityonion/./bpfconf.bro, line 103

Yep, I've seen this issue before. I'm not sure if it's an issue with
the Security-Onion-specific scripts that we load into Bro, or if it
could be an issue with Bro itself.

Has anybody else seen this issue on a vanilla Bro installation (not
using Security Onion)?

The other consistent thing I see is that with the Intel framework disabled I'll have to stop and start Bro usually two times before I start seeing all of the logs generated but usually only once. When I have the Intel framework enable I can stop and start Bro a number of times with only those five log files being generated each time. And again, on some of the sensors Bro will work with the Intel framework enabled and they all are using the same Intel files replicated via the "policy" directory Security Onion replication.

As a temporary test (perhaps on a non-production machine), could you
comment out the Security-Onion-specific scripts in
/opt/bro/share/bro/site/local.bro and see if that makes any
difference?

#@load securityonion
#@load file-extraction
#@load apt1

I know there were some issues previously with the hostname/interface
scripts in /opt/bro/share/bro/securityonion/ that resulted in a timing
issue. Some of the issues were fixed, but perhaps some other issues
remain.

Commenting out apt1 seems to make it more reliable. I can at least load one
intel file now; sometimes two.

The odd thing is that the more I play with this the more unreliable it gets.
I thought I saw a pattern at one point as I commented out securityonion and
that allowed me to load two intel files. I then uncommented securityonion and
it still worked with two intel files. The it once worked with four intel
files. I then tried to add a fifth and that broke it. So I went back to four
and Bro would still not generate more than the five log files. Then I was
back to two intel files to get Bro working after a couple more restarts. Then
I uncommented apt1 and it still worked with two intel files loaded.

So far, the most reliable way of getting an intel file loaded is by commenting out apt1 and only enabling one intel file.


I suspect what's going on is some sort of resource overload issue right at Bro startup and that's causing some other "downstream" stuff to fail. If I get a chance sometime I may write a new script for you to load intelligence with so that it never tries to load more than one intel file at a time. (or someone else could write that!)

Thanks for reporting the issue. Hopefully we can figure it out together. :wink:

  .Seth

Can you point me in the direction to start looking at this myself? I'll see what I can dig around and find, thanks.

You can take a look at the base/frameworks/intel/input.bro script. Right now that creates all of the input streams at the same time which causes them all to start reading as fast as they can. It might make more sense (at the moment since we don't have a limiter mechanism on the input framework) to wait until one file is fully read before creating the next input stream.

That input.bro script is an incredibly basic script and it should be possible to create another script and keep it outside of the intel framework that uses the intel framework api to import intelligence data by a different mechanism.

.Seth