Hi,
In my /usr/local/bro/etc/broctl.cfg I’ve specified:
LogExpireInterval = 14
Additionally in /etc/cron.d/bro I’ve specified:
0-59/5 * * * * /usr/local/bro/bin/broctl cron
However I’ve found that I have more daily directories present that 14 days…What configuration options should I be checking to troubleshoot this problem?
Thanks,
Jesse
Have you run broctl install && broctl check? I always forget to do
that after modifying LogExpireInterval.
Tyler
Hi Tyler,
Thanks for the response.
Yes, I have ensured that these have been run…I’ve also tried just running ‘broctl cron’ manually, but I get no output and it never seems to quit (or at least, outlasts my patience)…Any other hints?
Cheers,
Jesse
broctl cron typically doesn’t give output. If it is hanging, you should check for other instances of broctl cron running and kill them. They will sometimes log jam. I haven’t figured out why that happens.
Tyler
Thanks again Tyler.
Unfortunately, that was not the case here (although I have observed the same previously). Disabling the cron job and running it manually with strace, we end up hanging after a few minutes:
read(8, “W”, 1) = 1
read(8, “r”, 1) = 1
read(8, “i”, 1) = 1
read(8, “t”, 1) = 1
read(8, “e”, 1) = 1
read(8, " ", 1) = 1
read(8, “p”, 1) = 1
read(8, “a”, 1) = 1
read(8, “c”, 1) = 1
read(8, “k”, 1) = 1
read(8, “e”, 1) = 1
read(8, “t”, 1) = 1
read(8, “s”, 1) = 1
read(8, " ", 1) = 1
read(8, “t”, 1) = 1
read(8, “o”, 1) = 1
read(8, " ", 1) = 1
read(8, “f”, 1) = 1
read(8, “i”, 1) = 1
read(8, “l”, 1) = 1
read(8, “e”, 1) = 1
read(8, “\n”, 1) = 1
wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(8, “\n”, 1) = 1
wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(8, “~”, 1) = 1
read(8, “~”, 1) = 1
read(8, “~”, 1) = 1
read(8, “\n”, 1) = 1
wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(32, “0”, 1) = 1
read(32, “\n”, 1) = 1
wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(32,
Any other hints?
Cheers,
Jesse
I've typically seen this when the stats.log file grows too large. Try deleting or moving <prefix>/spool/stats.log and then do a full install/restart in broctl.
.Seth
I’ll try that, but my stats files are 45 - 107 lines long…Is that too large?
Cheers,
Jesse
Oh, probably not. Maybe there is some other file I'm trying to think of... It's been a while since I've seen this problem, I used to see it all the time when the boxes I ran were under high load though.
.Seth
Have you tried looking at the <install_prefix>/spool/debug.log file?
If that file doesn't exist, then uncomment the "Debug = 1" line
in your broctl.cfg file.
Turned debug on, install && check’ed everything, ensured ‘broctl cron’ is in cron, ensured two jobs not running, but still my logs don’t expire…
More hints anyone?
Cheers,
Jesse
When "broctl cron" runs, it will output a line (in debug.log)
that indicates it is starting, followed by a lot of output
showing its progress, and finally a line indicating it
is done. It looks something like this:
12 Feb 14:29:30 [command] cron
<...lots of output here...>
12 Feb 14:30:37 [main] cron done
If "broctl cron" is hanging, then it might be
useful to check the debug.log to see how far it
gets before it hangs.
So my solution is a two-part one…The easy part first: the cron job wasn’t running.
In the document (http://www.bro-ids.org/documentation/quickstart.html) I used for setup, it was suggested to put the following into crontab:
0-59/5 * * * * /usr/local/bro/bin/broctl cron
However since I was putting this under /etc/cron.d/bro (on a RHEL 6 box), what I actually need is:
0-59/5 * * * * root /usr/local/bro/bin/broctl cron
If you don’t specify the user, it won’t run (or complain).
Derp.
So once that was fixed, I ran into a second issue: after the first instance ran, subsequent runs would not be able to get the lock (generating a “cannot get lock” error email). Looking at the issue further, it appears that there are is a capstats instance running at 99%. Killing that job allowed the initial cron to finish, and my directories were deleted as expected.
root 21847 21661 0 11:35 ? 00:00:00 bash /usr/local/bro/share/broctl/scripts/helpers/run-cmd /usr/local/bro/bin/capstats -i p2p1;p2p2;p2p3;p2p4 -I 5 -n 1
root 21848 21847 99 11:35 ? 00:11:29 /usr/local/bro/bin/capstats -i p2p1
Seeing that the bash script /usr/local/bro/share/broctl/scripts/helpers/run-cmd is the dead simple:
echo 0 # Fix me. Should be real return code.
eval $@ 2>&1
echo ~~~
I can surmise the problem: Because my interface specification requires the use of ‘;’, bash is breaking the command up before it should and capstats doesn’t know it should quit…The format I’m using (p2p1;p2p2;p2p3;p2p4) is making use of PF_RING to listen to all these interfaces simultaneously. For snort I have to quote it to prevent it being broken up and I suspected something similar is required here as well.
Sure enough, patching control.py to quote the interfaces allows the job to finish and the stats to get written, with one error message in the stats file: “1360775469.46 proxy-1 error error proxy-1: unexpected capstats output: capstats [Options] -i interface”.
So while this apparently fixes my issue:
— control.py 2013-02-13 12:08:00.514656601 -0500
+++ control_mod.py 2013-02-13 12:09:38.382663593 -0500
@@ -808,7 +808,7 @@
for (addr, interface) in hosts.keys():
node = hosts[addr, interface]
I cannot speak to how this might affect others, the system in general, or where else this issue might crop up. I suspect that anywhere that involves bash + interface names is likely to suffer unexpected results due to this PF_RING style invocation…
I plan to let this run for now, perhaps others might comment on the advisability of this?
Cheers,
Jesse
Woah! PF_RING lets you sniff multiple interfaces that way? If you give that same value to tcpdump (while using the pf_ring libpcap wrapper) does it work there too?
.Seth
I can surmise the problem: Because my interface specification requires the use of ‘;’, bash is breaking the command up before it should and capstats doesn’t know it should quit…The format I’m using (p2p1;p2p2;p2p3;p2p4) is making use of PF_RING to listen to all these interfaces simultaneously. For snort I have to quote it to prevent it being broken up and I suspected something similar is required here as well.
Woah! PF_RING lets you sniff multiple interfaces that way? If you give that same value to tcpdump (while using the pf_ring libpcap wrapper) does it work there too?
.Seth
That is my understanding. Anything built against PF_RING’s libpcap can use the notation…However, now that I’ve put it out on the internet and it’s not apparently common knowledge, I’m doubting myself… 
As a reference, straight from (one of) the horses mouths: http://lists.ntop.org/pipermail/ntop-misc/2012-August/003128.html
Cheers,
Jesse
And you are using the pf_ring aware drivers?
.Seth
Yes, but not the DNA drivers…
I'm curious how you're getting things working with semicolons in the
interface name. Do you have a line like this in your node.cfg:
interface=p2p1;p2p2;p2p3;p2p4
I seem to remember trying to quote it at first, but then found that this worked:
interface=p2p1;p2p2;p2p3;p2p4
Cheers,
Jesse