My Bro cluster is happily flagging and accumulating data - but:
1. The last two hourly cycles left uncompressed logfiles in
/opt/app/bro/logs/current:
:
-rw-r--r-- 1 bro bro 73529 Jan 8 11:00 reporter-15-01-08_10.00.00.log
-rw-r--r-- 1 bro bro 749059 Jan 8 11:00 tunnel-15-01-08_10.00.00.log
-rw-r--r-- 1 bro bro 2474781 Jan 8 11:00 weird-15-01-08_10.00.00.log
-rw-r--r-- 1 bro bro 17062559659 Jan 8 10:00 conn-15-01-08_09.00.00.log
-rw-r--r-- 1 bro bro 2260979370 Jan 8 10:00 files-15-01-08_09.00.00.log
-rw-r--r-- 1 bro bro 4942559737 Jan 8 10:00 http-15-01-08_09.00.00.log
: etc.
:
2. No gzip processes were in evidence;
3. Figuring it might be the appropriate proverbial kick in the pants, I
did a "broctl restart", which ran cleanly - and to all appearances,
*erased* the older uncompressed files in question.
I now have a hole where the data from 10:00-12:00 today used to be - can anyone shed light on what's going on here?
I've seen this sort of thing happen when one or more log files get really large, typically dns.log, conn.log, or http.log. It seems to partially rotate the logs, perform the initial rename, but not compress them and do the second renaming (you'll notice that the naming convention differs from the already moved and compressed logs). You may also notice you stop getting connection summaries when this happens as well. I suspect that part of the post-processing never happens or never completes.
Many times I find the logs end up really big due to one or more misbehaving hosts, such as open DNS resolvers participating in a DDOS or a compromised host aggressively scanning/attacking something. Best bet is to manually move the logs that missed rotation(and compress them) then restart bro. If you restart bro without moving the logs the logs that didn't fully get processed on prior rotations first I've noticed those logs tend to simply get deleted. The only work around I've found is to address the traffic that is causing the logs to explode.
Not good. Broctl should always compress and archive any left-over
logs, not simply delete them on restart. If it doesn't, that's a bug.
For fixing that it would be very helpful to have a way to reproduce
the problem. If anybody knows how to do that, please file a
corresponding ticket.
Has this happened more than once? Have you tried looking for
unarchived log files in /opt/app/bro/spool/tmp ?
If there were any unarchived log files in logs/current, then when
you do a broctl stop (or broctl restart), I would expect that they
would get moved into a subdirectory of spool/tmp/ (assuming that
you're using a recent version of broctl).
I have anecdotal information, but no direct experience beyond the one time. I used a 'find' command to search throughout my installation for the lost files, to no effect.
I would have expected the same thing, but found them gone instead. FWIW, broctl reports itself as v1.3 (in a v2.3 Bro deployment).