Odd log problem - logs get archived as empty

Zeek
1

We are running “bro version 2.2” compiled from source on CentOS 6.5.

broctl.cfg for the logging parts is:

LogDir = /usr/local/bro/logs
LogRotationInterval = 3600

LogExpireInterval = 14

MinDiskSpace = 5

bro is run from broctl…

log files in /usr/local/bro/logs/current is fine:

-rw-r-----. 1 root root 121 Mar 6 23:23 .cmdline

-rw-r-----. 1 root root 14820 Mar 6 23:44 communication.log
-rw-r-----. 1 root root 391 Mar 6 23:28 dhcp.log
-rw-r-----. 1 root root 826073 Mar 6 23:45 dns.log
-rw-r-----. 1 root root 355 Mar 6 23:38 dpd.log
-rw-r-----. 1 root root 314 Mar 6 23:23 .env_vars
-rw-r-----. 1 root root 961 Mar 6 23:39 files.log
-rw-r-----. 1 root root 14907 Mar 6 23:23 loaded_scripts.log
-rw-r-----. 1 root root 226 Mar 6 23:23 packet_filter.log
-rw-r-----. 1 root root 5 Mar 6 23:23 .pid
-rw-r-----. 1 root root 58 Mar 6 23:23 .startup
drwx------. 3 root root 4096 Mar 6 23:23 .state
-rwx------. 1 root root 18 Mar 6 23:23 .status
-rw-r-----. 1 root root 46 Mar 6 23:23 stderr.log
-rw-r-----. 1 root root 30 Mar 6 23:23 stdout.log
-rw-r-----. 1 root root 330 Mar 6 23:33 tunnel.log

but when they get cycled out to the normal log archive they become empty blank files.

ls -al /usr/local/bro/logs/2014-03-06
-rw-r-----. 1 root root 2 Mar 6 01:00 communication.00:00:00-01:00:00.log.

-rw-r-----. 1 root root 2 Mar 6 02:00 communication.01:00:00-02:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 03:00 communication.02:00:00-03:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 04:00 communication.03:00:00-04:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 05:00 communication.04:00:00-05:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 06:00 communication.05:00:00-06:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 07:00 communication.06:00:00-07:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 08:00 communication.07:00:00-08:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 09:00 communication.08:00:00-09:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 10:00 communication.09:00:00-10:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 11:00 communication.10:00:00-11:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 12:00 communication.11:00:00-12:00:00.log.
-rw-r-----. 1 root root 2 Mar 6 13:00 communication.12:00:00-13:00:00.log.

etc… etc… etc…

I know at one point this was working, but we went back today to look for something and noticed this problem.

Any ideas where to start to look for reasons why this might be happening?

Just for giggles, local.bro looks like this:

@load misc/loaded-scripts
@load tuning/defaults

event bro_init()
{
Log::disable_stream(HTTP::LOG);
Log::disable_stream(Syslog::LOG);
Log::disable_stream(Conn::LOG);
Log::disable_stream(SMTP::LOG);
Log::disable_stream(Weird::LOG);
Log::disable_stream(SSL::LOG);
Log::remove_default_filter(DNS::LOG);
Log::add_filter(DNS::LOG, [$name=“new-default”,
$include=set(“ts”,“id.orig_h”,“id.orig_p”,“id.resp_h”,“id.resp_p”,“proto”,“trans_id”,“query”,“qclass_name”,“qtype_name”,“rcode_name”,“AA”,“answers”,“RA”,“RD”,“TTL”)]);
}

Thanks!

2

Run

    broctl config | grep compress

What are the variables set to?

looks like your compresscmd and compressextension options are blank.

3

broctl config | grep compress

compresscmd = gzip -9
compressextension = gz
compresslogs = 1

If the variables are blank, wouldn’t, worst case, it copy the files in and just have them be big?

4

not sure.. the command it runs is:

    nice ${compresscmd} <$1 >$dest.${compressextension}

if compresslogs is not 1, then it just runs

    nice cp $1 $dest

Your logs have a '.' at the end so it is clearly trying to do something,
but not having the right variables there.

You should have one or more 'broctl-config.sh' files

something like:

    /usr/local/bro/spool/broctl-config.sh

try

    grep compress /usr/local/bro/spool/broctl-config.sh

you should get the same output.

5

grep compress /usr/local/bro/spool/broctl-config.sh

compresslogs=“1”

That is interesting. So it’s missing the two lines:
compresscmd = gzip -9
compressextension = gz

I’ll add those and restart and see what happens

Side note - this is an upgrade from 2.1 to 2.2

And i think/thought it was working in 2.1

6

So I added those two lines, restarted bro “broctl restart” waited a number of minutes, then restarted it again, the logs moved into the archive directory, but still end up emtpry and with the dot at the end.

I’m heading home for the night, but I’ll keep reading and checking out some things.

Thanks!

7

You need to do "broctl install" after upgrading (that will update
your broctl-config.sh file).

8

Right right… I so rarely change that; I forgot. Thank you and I’ll test that tomorrow.

9

Thanks for your help everyone… I did the broctl install and then restart and then at the next restart/rotate, things compressed like normal.

I’ll have to do a test install and see if those two lines are in the broctl-config.sh by default.

10

Sounds like we should add a check to broctl that when the version of
either itself of Bro changes, it suggests doing an "install" if not
done yet.

Robin

11

Well so I went to do our other bro installs and noticed that the one with the 2.2 fresh install already had the commands but the ones we upgraded from 2.1 to 2.2 did not. So eh… I don’t know.

Plus, the ‘broctl config’ knew about the commands, but they weren’t in the .sh file. So I’m not sure how/why that was different. But that is probably totally be me not knowing how Bro works in detail.

I’m just helpful for the quick response of this list and that it wasn’t something simple that I had missed.

12

The way broctl works now is that you need to do a
"broctl install" whenever you change the bro or broctl configuration
(i.e., config files or policy scripts). Upgrading bro/broctl
would be considered a configuration change.

13

Plus, the 'broctl config' knew about the commands, but they weren't in the .sh
file. So I'm not sure how/why that was different. But that is probably totally
be me not knowing how Bro works in detail.

The why would be: 'broctl install' was not ran after upgrading bro.

I'm just helpful for the quick response of this list and that it wasn't
something simple that I had missed.

Looks like there are things that can be done in broctl to check for
this, but in the future just running broctl install after an ugprade
will make this not happen again.

14

You can also roll your own...in a cron job at say 23:58 (you lose the email stats without broctl however)

#!/bin/bash
killall bro
mkdir /media/backup/bro/logarchive/`date --date=today +%m-%d-%Y`
mv /media/backup/bro/current/*.log /media/backup/bro/logarchive/`date --date=today +%m-%d-%Y`
cd /media/backup/bro/logarchive/`date --date=today +%m-%d-%Y`
/usr/local/bin/gz.pl
cd /media/backup/bro/current
/usr/local/bin/startbro
exit

cat /usr/local/bin/gz.pl

#!/usr/bin/perl
@zips = `ls *.log | more`;
foreach (@zips) {
system("gzip $_");
}
exit;

James

15

Please don’t use that “script” James posted. (No offense.) You’ll lose data for the time that bro is off. (Maybe you wont care. Maybe you will.)

I wouldn’t recommend killing a process just to rotate it’s log, as was suggested.

If you must “roll your own” logs, I’d recommend logrotate (add a config in /etc/logrotate.d/). That’s what it is there for. Short of that you can rotate your own logs using some kind of copy-truncate or copy-rm procedure. But be careful of the many gotcha’s in writing your own, one of which is avoiding data loss.

Easiest thing is stick with broctl or logrotate.

~Ben