# Bro workers die **URL:** \ **Category:** Zeek\ **Created:** [April 22, 2013, 8:04pm UTC](https://community.zeek.org/t/bro-workers-die/2657 "2013-04-22T20:04:42Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Michal\_Purzynski](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Michal\_Purzynski](https://community.zeek.org/u/Michal_Purzynski)\ **Post date:** [April 22, 2013, 8:04pm UTC](https://community.zeek.org/t/bro-workers-die/2657/1 "2013-04-22T20:04:42Z")
Hi, me again. Bro is a new one, from the SVN, but I had the same results with 2.1 stable. broctl start starting manager ... starting proxy ... starting nsm1-eth4-1 ... starting nsm1-eth5-1 ... starting nsm1-eth5-10 ... starting nsm1-eth5-11 ... starting nsm1-eth5-12 ... starting nsm1-eth5-2 ... starting nsm1-eth5-3 ... starting nsm1-eth5-4 ... starting nsm1-eth5-5 ... starting nsm1-eth5-6 ... starting nsm1-eth5-7 ... starting nsm1-eth5-8 ... starting nsm1-eth5-9 ... (nsm1-eth5-12 still initializing) (nsm1-eth5-9 still initializing) (nsm1-eth5-10 still initializing) (nsm1-eth5-11 still initializing) (nsm1-eth4-1 still initializing) And after a while Name Type Host Status Pid Peers Started nsm1-eth4-1 worker \ crashed nsm1-eth5-10 worker \ crashed nsm1-eth5-11 worker \ crashed nsm1-eth5-12 worker \ crashed nsm1-eth5-9 worker \ crashed manager manager \ running 44798 9 22 Apr 19:27:37 proxy proxy \ running 44845 9 22 Apr 19:27:39 nsm1-eth5-1 worker \ running 45048 2 22 Apr 19:27:41 nsm1-eth5-2 worker \ running 45057 2 22 Apr 19:27:41 nsm1-eth5-3 worker \ running 45060 2 22 Apr 19:27:41 nsm1-eth5-4 worker \ running 45063 2 22 Apr 19:27:41 nsm1-eth5-5 worker \ running 45066 2 22 Apr 19:27:41 nsm1-eth5-6 worker \ running 45067 2 22 Apr 19:27:41 nsm1-eth5-7 worker \ running 45068 2 22 Apr 19:27:41 nsm1-eth5-8 worker \ running 45069 2 22 Apr 19:27:41 Two more questions: 1. does Bro use pf\_ring by default with a configuration like this? 2. how can i change the load balancing method? I need to spread things more evenly. cat /opt/bro/etc/node.cfg [manager] type=manager host=\ [proxy] type=proxy host=\ [nsm1-eth4] type=worker host=\ interface=eth4 lb\_method=pf\_ring lb\_procs=1 [nsm1-eth5] type=worker host=\ interface=eth5 lb\_method=pf\_ring lb\_procs=12 ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [April 22, 2013, 8:21pm UTC](https://community.zeek.org/t/bro-workers-die/2657/2 "2013-04-22T20:21:30Z")
> 1. does Bro use pf\_ring by default with a configuration like this? Yes, it's the lb\_method=pf\_ring that enables it. > 2. how can i change the load balancing method? I need to spread things > more evenly. What do you want to change it to? I think it's doing 4-tuple or 5-tuple by default right now. One problem you will encounter is a issue with pf\_ring cluster\_id choice. You will be running two pf\_ring clusters on the same host (i'm assuming that nsm1 is the same physical host) and pf\_ring doesn't like that. It does something weird like trying to stick packets from both NICs into the same queue. We have it fixed for our next release (that did get merged into master, right Daniel?) but it's a problem right now. You are sending us enough information to determine why you're seeing crashes though. Could you send the output from broctl diag nsm1-eth5-1 (assuming that's a host that is currently crashed)? Thanks,   .Seth ---
**Author:** ![Michal\_Purzynski](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Michal\_Purzynski](https://community.zeek.org/u/Michal_Purzynski)\ **Post date:** [April 22, 2013, 8:51pm UTC](https://community.zeek.org/t/bro-workers-die/2657/3 "2013-04-22T20:51:41Z")
> > 1. does Bro use pf\_ring by default with a configuration like this? > > Yes, it's the lb\_method=pf\_ring that enables it. > > > 2. how can i change the load balancing method? I need to spread things > > more evenly. > > What do you want to change it to? I think it's doing 4-tuple or 5-tuple by default right now. OK, I might be wrong on that, it has helped in a big way for snort. > One problem you will encounter is a issue with pf\_ring cluster\_id choice. You will be running two pf\_ring clusters on the same host (i'm assuming that nsm1 is the same physical host) and pf\_ring doesn't like that. It does something weird like trying to stick packets from both NICs into the same queue. We have it fixed for our next release (that did get merged into master, right Daniel?) but it's a problem right now. I'm running the SVN code, so you think it does not choose a unique cluster id for eth4 and another for eth5? How can i fix it? > You are sending us enough information to determine why you're seeing crashes though. Could you send the output from broctl diag nsm1-eth5-1 (assuming that's a host that is currently crashed)? > > Thanks, > >    .Seth broctl diag nsm1-eth5-1 [nsm1-eth5-1] Bro 2.1-386 ==== No reporter.log ==== stderr.log listening on eth5, capture length 8192 bytes 1366658863.663940 processing suspended 1366658863.664006 processing continued 1366658869.682828 Failed to open GeoIP database: /usr/share/GeoIP/GeoIPCity.dat 1366658869.682828 Fell back to GeoIP Country database 1366658869.682828 Failed to open GeoIP database: /usr/share/GeoIP/GeoIPCityv6.dat ==== stdout.log unlimited ==== .cmdline -i eth5 -U .status -p broctl -p broctl-live -p local -p nsm1-eth5-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env\_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bro/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER\_NODE=nsm1-eth5-1 ==== .status RUNNING [net\_run] ==== No prof.log ==== No packet\_filter.log ==== No loaded\_scripts.log ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [April 22, 2013, 8:58pm UTC](https://community.zeek.org/t/bro-workers-die/2657/4 "2013-04-22T20:58:19Z")
> I'm running the SVN code, so you think it does not choose a unique cluster id for eth4 and another for eth5? How can i fix it? I don't know if that's fixed in master yet (i'm assuming you're running git master). > broctl diag nsm1-eth5-1 That shows the process is running fine. You need to do that for a worker that is crashed.   .Seth ---
**Author:** ![Michal\_Purzynski](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Michal\_Purzynski](https://community.zeek.org/u/Michal_Purzynski)\ **Post date:** [April 22, 2013, 9:01pm UTC](https://community.zeek.org/t/bro-workers-die/2657/5 "2013-04-22T21:01:50Z")
Right, just noticed the stack traces. root@nsm1:/nsm/bro/logs/current# broctl diag nsm1-eth5-9 [nsm1-eth5-9] Bro 2.1-386 core [New LWP 54717] [Thread debugging using libthread\_db enabled] Using host libthread\_db library "/lib/x86\_64-linux-gnu/libthread\_db.so.1". Core was generated by `/opt/bro/bin/bro -i eth5 -U .status -p broctl -p broctl-live -p local -p nsm1-e'. Program terminated with signal 11, Segmentation fault. #0 AsBool (this=0x0) at scan.l:1074 Thread 1 (Thread 0x7f3af913d740 (LWP 54717)): #0 AsBool (this=0x0) at scan.l:1074 #1 do\_atif (expr=\) at scan.l:686 #2 0x000000000051c95e in yyparse () at parse.y:1203 #3 0x00000000004c615e in main (argc=18, argv=\) at /home/michal/bro/src/main.cc:801 ==== No reporter.log ==== stderr.log error in /opt/bro/share/bro/base/frameworks/cluster/./main.bro, line 136: no such index (Cluster::nodes[Cluster::node]) warning in /opt/bro/share/bro/base/frameworks/notice/./cluster.bro, line 23: non-void function returns without a value: Cluster::local\_node\_type /opt/bro/share/broctl/scripts/run-bro: line 60: 54717 Segmentation fault (core dumped) nohup $mybro $@ ==== stdout.log unlimited ==== .cmdline -i eth5 -U .status -p broctl -p broctl-live -p local -p nsm1-eth5-9 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env\_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER\_NODE=nsm1-eth5-9 ==== .status INITIALIZING [main] ==== No prof.log ==== No packet\_filter.log ==== No loaded\_scripts.log You have new mail in /var/mail/root root@nsm1:/nsm/bro/logs/current# broctl diag nsm1-eth4-1 [nsm1-eth4-1] Bro 2.1-386 core [New LWP 54008] [Thread debugging using libthread\_db enabled] Using host libthread\_db library "/lib/x86\_64-linux-gnu/libthread\_db.so.1". Core was generated by `/opt/bro/bin/bro -i eth4 -U .status -p broctl -p broctl-live -p local -p nsm1-e'. Program terminated with signal 11, Segmentation fault. #0 AsBool (this=0x0) at scan.l:1074 Thread 1 (Thread 0x7ff6c1d0f740 (LWP 54008)): #0 AsBool (this=0x0) at scan.l:1074 #1 do\_atif (expr=\) at scan.l:686 #2 0x000000000051c95e in yyparse () at parse.y:1203 #3 0x00000000004c615e in main (argc=18, argv=\) at /home/michal/bro/src/main.cc:801 ==== No reporter.log ==== stderr.log error in /opt/bro/share/bro/base/frameworks/cluster/./main.bro, line 136: no such index (Cluster::nodes[Cluster::node]) warning in /opt/bro/share/bro/base/frameworks/notice/./cluster.bro, line 23: non-void function returns without a value: Cluster::local\_node\_type /opt/bro/share/broctl/scripts/run-bro: line 60: 54008 Segmentation fault (core dumped) nohup $mybro $@ ==== stdout.log unlimited ==== .cmdline -i eth4 -U .status -p broctl -p broctl-live -p local -p nsm1-eth4-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env\_vars PATH=/opt/bro/bin:/opt/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin BROPATH=/nsm/bro/spool/installed-scripts-do-not-touch/site::/nsm/bro/spool/installed-scripts-do-not-touch/auto:/opt/bro/share/bro:/opt/bro/share/bro/policy:/opt/bro/share/bro/site CLUSTER\_NODE=nsm1-eth4-1 ==== .status INITIALIZING [main] ==== No prof.log ==== No packet\_filter.log ==== No loaded\_scripts.log root@nsm1:/nsm/bro/logs/current# ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [April 22, 2013, 9:05pm UTC](https://community.zeek.org/t/bro-workers-die/2657/6 "2013-04-22T21:05:59Z")
Have you run "broctl install" since you last changed your node.cfg file?   .Seth ---
**Author:** ![Michal\_Purzynski](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@Michal\_Purzynski](https://community.zeek.org/u/Michal_Purzynski)\ **Post date:** [April 22, 2013, 9:15pm UTC](https://community.zeek.org/t/bro-workers-die/2657/7 "2013-04-22T21:15:37Z")
root@nsm1:~# broctl status 2\>&1 | grep nsm1 | grep worker | wc -l 13 root@nsm1:~# broctl status 2\>&1 | grep nsm1 | grep running | wc -l 13 Awesome! Thank you, i didn't know I'm supposed to 🙂 Now on to the traffic filtering (which is ignored) but I've separated that into another post. ---
**Author:** ![William\_Jones](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@William\_Jones](https://community.zeek.org/u/William_Jones)\ **Post date:** [April 23, 2013, 3:38pm UTC](https://community.zeek.org/t/bro-workers-die/2657/8 "2013-04-23T15:38:41Z")
Seth, The only time I am seeing dropped packets are during attempts to us TACC to amplify dos attach very aggressive port scans. In both cases bro workers are being overloaded by 500kk to 1000k incoming packets. It looks like a single worker can only handle 30K packets/sec before it reaches 100 percent cpu usage. Is there any effort going into bro development to handle these cases. My only work around that I have now is to block aces to common ports at the boarder router and opening host to vetted hosts. Bill Jones ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:40pm UTC](https://community.zeek.org/t/bro-workers-die/2657/9 "2022-05-06T15:40:57Z")