So, it happened again, this morning around 6:55am.
Bro stopped at that time, don’t really know why.
I got to know about this when I wanted to analyse traffic for a particular IP around 11 and found out that we don’t have any logs after 7am logged by BRO
I quickly checked the status of bro on manager, and found that bro isn’t running.
I restarted bro from manager and all but one worker came up online, and bro started normally, running with remaining nodes in the cluster.
This have happened before, when one of the workers will become unreachable and bro stops.
I don’t really know what happens first,i.e whether worker becomes offline first and then bro stops, or vise versa.
I tried looking for some errors on the workers as well as on manager in :
dir but nothing useful, only some warnings in stderr.log like following:
warning in /usr/local/bro/2.4.1/share/bro/site/connStats.bro, line 39: dangerous assignment of double to integral (ConnStats::out$EstinboundConns = ConnStats::result[EstinboundConns]$sum)
warning in /usr/local/bro/2.4.1/share/bro/site/connStats.bro, line 40: dangerous assignment of double to integral (ConnStats::out$EstoutboundConns = ConnStats::result[EstoutboundConns]$sum)
listening on em1, capture length 8192 bytes
1477133753.104159 processing suspended
1477133753.104159 processing continued
1477133759.776854 Failed to open GeoIP Cityv6 database: /usr/share/GeoIP/GeoIPCityv6.dat
1477133759.776854 Failed to open GeoIPv6 Country database: /usr/share/GeoIP/GeoIPv6.dat
Is there anywhere else I can look also to diagnose the issue?
Is there any reason, bro will stop entirely if one of the workers become offline for some reason?
Or the issue is completely else, and I am looking in completely wrong direction.
I have two crons currently in bro’s crontab:
$ crontab -l
0-59/5 * * * * /usr/local/bro/default/bin/broctl cron
55 6 * * * /usr/local/bro/bin/restart-bro
restart-bro is a small script that looks like this:
The reason, I think, for having bro restart every morning at 6:55 is we pull down the intel feeds every morning at 6:45
that updates the files that bro monitors as input feeds for intel framework.
And I thought that Bro would not pick up new/updated input feeds unless restarted.
Is that would be something causing bro to not restart?
You shouldn't have to restart bro for it to pull in updates from intel files.
It's suspicious that you say bro crashed at 7am and that cron job runs at 6:55.
It's possible that something went wrong during the restart and bro just ended up stopped. I could see 'broctl restart' leaving the cluster in an inconsistent state if it gets interrupted.
I'd just remove that job (since intel files should auto update on their own) or try changing the time it runs at to 6:57, which should at least avoid it running at the same time as cron.
Hmm, that kinda makes sense.
Disabled the cron job of restart-bro, and will keep a check on bro on
manager for future.
While Bro should pick up new intel without a restart, 2.4.1 will never
delete any intel that has been ingested. If you are using large volatile
feeds that might become a problem. With 2.5 the intel framework allows
to expire intel. I would be curious to know if you are experiencing any
corresponding problems with 2.4.1.
