# Cluster setup **URL:** \ **Category:** Zeek\ **Created:** [October 29, 2013, 6:28am UTC](https://community.zeek.org/t/cluster-setup/2876 "2013-10-29T06:28:12Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 29, 2013, 6:28am UTC](https://community.zeek.org/t/cluster-setup/2876/1 "2013-10-29T06:28:12Z")
Hi All, Here is what I am trying to achieve: Incoming traffic on Host-A should be sent to worker Host-B (and to more workers in future). Here is how my config looks like in node.cfg Manager: Host-A Proxy: Host-A Worker1: Host-B (which is 10.73.149.31) I have bro installed on all machines. Now, I start broctl on Host-A: -bash-4.2$ sudo broctl Password: Welcome to BroControl 1.1 Type “help” for help. [BroControl] \> install removing old policies in /usr/local/spool/installed-scripts-do-not-touch/site … done. removing old policies in /usr/local/spool/installed-scripts-do-not-touch/auto … done. creating policy directories … done. installing site policies … done. generating cluster-layout.bro … done. generating local-networks.bro … done. generating broctl-config.bro … done. updating nodes … warning: host 10.73.149.31 is not alive done. [BroControl] \> install removing old policies in /usr/local/spool/installed-scripts-do-not-touch/site … done. removing old policies in /usr/local/spool/installed-scripts-do-not-touch/auto … done. creating policy directories … done. installing site policies … done. generating cluster-layout.bro … done. generating local-networks.bro … done. generating broctl-config.bro … done. updating nodes … done. [BroControl] \> start starting manager … starting proxy-1 … starting worker-1 … cannot create working directory for worker-1 \<\<-- not sure why I get this message. [BroControl] \> Do I need to do anything on Worker-1?? Do I need to put it in some special mode? Any help/pointers would be appreciated. Cheers, Hiren ---
**Author:** ![Daniel\_Thayer](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@Daniel\_Thayer](https://community.zeek.org/u/Daniel_Thayer)\ **Post date:** [October 29, 2013, 6:50am UTC](https://community.zeek.org/t/cluster-setup/2876/2 "2013-10-29T06:50:51Z")
Did you verify that you can ssh from host-A to host-B without having to type a password? Next, on host-B, verify that the partition where /usr/local/spool is located is not mounted read-only and that there is some free disk space (broctl is trying to create a directory in /usr/local/spool on host-B). ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 29, 2013, 5:09pm UTC](https://community.zeek.org/t/cluster-setup/2876/3 "2013-10-29T17:09:07Z")
> Did you verify that you can ssh from host-A to host-B without > having to type a password? Just set that up. > Next, on host-B, verify that the partition where /usr/local/spool > is located is not mounted read-only and that there is some free > disk space (broctl is trying to create a directory > in /usr/local/spool on host-B). Checked this too. Still, [BroControl] \> install removing old policies in /usr/local/spool/installed-scripts-do-not-touch/site ... done. removing old policies in /usr/local/spool/installed-scripts-do-not-touch/auto ... done. creating policy directories ... done. installing site policies ... done. generating cluster-layout.bro ... done. generating local-networks.bro ... done. generating broctl-config.bro ... done. updating nodes ... warning: host 10.73.149.31 is not alive done. [BroControl] \> What does that mean? I still cannot get worker-1 to work properly. in "top" (inside broctl) also, worker-1 is shown \ Do I need to setup anything on worker-1? Cheers, Hiren ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 29, 2013, 5:50pm UTC](https://community.zeek.org/t/cluster-setup/2876/4 "2013-10-29T17:50:43Z")
Ah, I realized that I had to do this as "root" because broctl is run as root 🙂 Set that up and now: [BroControl] \> install removing old policies in /usr/local/spool/installed-scripts-do-not-touch/site ... done. removing old policies in /usr/local/spool/installed-scripts-do-not-touch/auto ... done. creating policy directories ... done. installing site policies ... done. generating cluster-layout.bro ... done. generating local-networks.bro ... done. generating broctl-config.bro ... done. updating nodes ... warning: host 10.73.149.31 is not alive done. [BroControl] \> check manager is ok. proxy-1 is ok. worker-1 is ok. [BroControl] \> start starting manager ... starting proxy-1 ... starting worker-1 ... worker-1 terminated immediately after starting; check output with "diag" [BroControl] \> diag worker-1 [worker-1] ==== No reporter.log ==== stderr.log error in /usr/local/share/bro/base/frameworks/cluster/\_\_load\_\_.bro, line 16: can't open cluster-layout ==== stdout.log unlimited 536870912 unlimited ==== .cmdline -i bce1 -U .status -p broctl -p broctl-live -p local -p worker-1 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env\_vars PATH=/usr/local/bin:/usr/local/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/y/bin:/root/bin BROPATH=/usr/local/spool/installed-scripts-do-not-touch/site::/usr/local/spool/installed-scripts-do-not-touch/auto:/usr/local/share/bro:/usr/local/share/bro/policy:/usr/local/share/bro/site CLUSTER\_NODE=worker-1 ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet\_filter.log ==== No loaded\_scripts.log [BroControl] \> Trying to determine what is causing this. cheers, Hiren ---
**Author:** ![Daniel\_Thayer](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@Daniel\_Thayer](https://community.zeek.org/u/Daniel_Thayer)\ **Post date:** [October 29, 2013, 6:01pm UTC](https://community.zeek.org/t/cluster-setup/2876/5 "2013-10-29T18:01:28Z")
Exit from broctl, then verify that you can ping the worker machine from the manager machine. If that works, then do another "broctl install" and make sure you don't see any error or warning messages. ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 29, 2013, 6:04pm UTC](https://community.zeek.org/t/cluster-setup/2876/6 "2013-10-29T18:04:31Z")
I am not able to find/understand what is causing this problem. Anyone with some clue? Thanks in advance, Hiren ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 29, 2013, 6:16pm UTC](https://community.zeek.org/t/cluster-setup/2876/7 "2013-10-29T18:16:42Z")
Alright, So its looking for a file: cluster-layout.bro I could see that on manager node at: /usr/local/spool/installed-scripts-do-not-touch/auto/ But it was not available on same location in worker-1 node. (please let me know if there is a better way to this) I scp'ed that file there and then [BroControl] \> install [BroControl] \> start worked. top also shows all the nodes: manager, proxy-1 and worker-1 active. Now is time for actual traffic. Cheers, Hiren ---
**Author:** ![Azoff\_Justin](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@Azoff\_Justin](https://community.zeek.org/u/Azoff_Justin)\ **Post date:** [October 29, 2013, 6:21pm UTC](https://community.zeek.org/t/cluster-setup/2876/8 "2013-10-29T18:21:42Z")
This is from:     def isAlive(host):         ...         (success, output) = runLocalCmd(os.path.join(config.Config.scriptsdir, "is-alive") + " " + util.scopeAddr(host))         ...         if not success and not config.Config.cron == "1":             util.warn("host %s is not alive" % host) which just runs the is-alive script which runs     ping -c 1 -W 1 host so, if that is failing you're running a restrictive iptables policy or have disabled icmp? ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [October 30, 2013, 2:24pm UTC](https://community.zeek.org/t/cluster-setup/2876/9 "2013-10-30T14:24:10Z")
Thanks for pointing that out, I've seen that on quite a few hosts (CentOS) where the default iptables setup interferes with BroControl.   .Seth ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 30, 2013, 11:43pm UTC](https://community.zeek.org/t/cluster-setup/2876/10 "2013-10-30T23:43:24Z")
Thanks for all the hints and help guys. I have a question about parallelism. If I am consuming live traffic on a 10G intel nic with 8 queues in it on my manager node, can I setup bro to send data to 8 separate workers such that, queue:1 traffic goes to a worker:1 and so on...? Is that possible? Or I am thinking it wrong? Cheers, Hiren ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [October 31, 2013, 1:03pm UTC](https://community.zeek.org/t/cluster-setup/2876/11 "2013-10-31T13:03:37Z")
Yes, in that case you would just put your workers and your manager on the same physical host. .Seth ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 31, 2013, 4:33pm UTC](https://community.zeek.org/t/cluster-setup/2876/12 "2013-10-31T16:33:50Z")
> > I have a question about parallelism. If I am consuming live traffic on > > a 10G intel nic with 8 queues in it on my manager node > > Yes, in that case you would just put your workers and your manager on the same physical host. But then would one host be able to keep up doing everything? I somehow cannot direct a queue’s traffic to a specific worker? Thanks a lot for your help, Hiren ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [October 31, 2013, 4:51pm UTC](https://community.zeek.org/t/cluster-setup/2876/13 "2013-10-31T16:51:03Z")
You already said you had 8 queues on your NIC though? I guess I assumed you had something like PF\_Ring configured to split your traffic to multiple processes. .Seth ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [October 31, 2013, 7:20pm UTC](https://community.zeek.org/t/cluster-setup/2876/14 "2013-10-31T19:20:35Z")
Right. So (afaik) in FreeBSD we do not have PF\_RING like functionality where there is an PF\_RING application sdk and applications can choose which queue it wants to listen to. Intel NIC (that I am using) definitely can distribute traffic in 8 queues it has but question for me is, how do I distribute it to the application/workers. I also have a larger question here though. How do people usually do something that I want to do: Have a box tap continuous traffic on a 10G card and let workers parse/interpret it (via bro). Do they have PF\_RING setup which blindly ports queue:1 traffic to worker:1 and bro (using PF\_RING's sdk) will do the parsing? Simple aim here is load-distribution. Thanks a lot of tolerating my (possibly) stupid questions 🙂 Cheers, Hiren ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [November 1, 2013, 12:51am UTC](https://community.zeek.org/t/cluster-setup/2876/15 "2013-11-01T00:51:50Z")
> Right. So (afaik) in FreeBSD we do not have PF\_RING like functionality > where there is an PF\_RING application sdk and applications can choose Ah, generally right now people are only doing load balancing on FreeBSD with Myricom NICs and the Myricom Sniffer driver. > which queue it wants to listen to. Intel NIC (that I am using) > definitely can distribute traffic in 8 queues it has but question for > me is, how do I distribute it to the application/workers. In FreeBSD at the moment you don't. It's possible that if you have netmap enabled you might be able to use that in some fashion, but generally those FlowDirector based queues on the high end Intel NICs aren't actually exposed in userland. If you are talking about RSS (receive side scaling), then that's insufficient unless you have RX and TX RSS (I'm a little confused about this, but I read something recently that seemed to indicate this might be a thing on some NICs) because both directions of each connection need to go to each process. > Do they have PF\_RING setup which blindly ports queue:1 traffic to > worker:1 and bro (using PF\_RING's sdk) will do the parsing? Typically people run PF\_Ring in mode 0 which is actually not exposing hardware load balanced traffic. It's collecting all of the traffic and load balancing it in the core. I don't have much of a suggestion right now for FreeBSD beyond Myricom though.   .Seth ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [November 1, 2013, 3:36pm UTC](https://community.zeek.org/t/cluster-setup/2876/16 "2013-11-01T15:36:49Z")
> > Right. So (afaik) in FreeBSD we do not have PF\_RING like functionality > > where there is an PF\_RING application sdk and applications can choose > > Ah, generally right now people are only doing load balancing on FreeBSD with Myricom NICs and the Myricom Sniffer driver. This is not an option for me but I will surely see how they are doing it. > > which queue it wants to listen to. Intel NIC (that I am using) > > definitely can distribute traffic in 8 queues it has but question for > > me is, how do I distribute it to the application/workers. > > In FreeBSD at the moment you don't. It's possible that if you have netmap enabled you might be able to use that in some fashion, but generally those FlowDirector based queues on the high end Intel NICs aren't actually exposed in userland. If you are talking about RSS (receive side scaling), then that's insufficient unless you have RX and TX RSS (I'm a little confused about this, but I read something recently that seemed to indicate this might be a thing on some NICs) because both directions of each connection need to go to each process. Yeah, tricky part is the userland association. But I am also not too clear on RSS detail. That looks like the only option I have. I need to dig deeper. > > Do they have PF\_RING setup which blindly ports queue:1 traffic to > > worker:1 and bro (using PF\_RING's sdk) will do the parsing? > > Typically people run PF\_Ring in mode 0 which is actually not exposing hardware load balanced traffic. It's collecting all of the traffic and load balancing it in the core. Here, core == bro's core? I really appreciate you taking time out and responding to my questions, Seth 🙂 cheers, Hiren ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [November 1, 2013, 5:34pm UTC](https://community.zeek.org/t/cluster-setup/2876/17 "2013-11-01T17:34:39Z")
Sorry, totally wrong word. I meant kernel. 🙂   .Seth ---
**Author:** ![hiren\_panchasara](https://avatars.discourse-cdn.com/v4/letter/h/fbc32d/32.png) [@hiren\_panchasara](https://community.zeek.org/u/hiren_panchasara)\ **Post date:** [November 1, 2013, 6:36pm UTC](https://community.zeek.org/t/cluster-setup/2876/18 "2013-11-01T18:36:35Z")
Ah, okay. So instead of card, let kernel do all the load balancing (probably with assistance from PF\_RING). Thank you, Hiren ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:41pm UTC](https://community.zeek.org/t/cluster-setup/2876/19 "2022-05-06T15:41:20Z")