# Bro 2.6-beta plans **URL:** \ **Category:** Development\ **Tags:** development\ **Created:** [September 5, 2018, 10:35pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443 "2018-09-05T22:35:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 5, 2018, 10:35pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/1 "2018-09-05T22:35:57Z")
There's no significant code changes/features planned to get added to the master branch from now until the 2.6-beta gets released (maybe in about a week). Until that happens, please help test the latest master branch and provide any feedback about how it's working if you can. - Jon ---
**Author:** ![Michael\_Dopheide](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@Michael\_Dopheide](https://community.zeek.org/u/Michael_Dopheide)\ **Post date:** [September 5, 2018, 10:43pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/2 "2018-09-05T22:43:17Z")
To be clear, Cluster::relay\_rr() is gone forever? I’ll need to rewrite some policies, but also update the blog to be correct. Dop ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 6, 2018, 3:28pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/3 "2018-09-06T15:28:09Z")
Yes. - Jon ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 6, 2018, 7:41pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/4 "2018-09-06T19:41:48Z")
I just got 2 clusters upgraded from fa7fa5aa to 452eb0cb And now everything is broken.. cpu and memory are through the roof across the board, as well as network traffic, but it's not logging much. I may have created a message loop replacing the relay\_rr stuff, but it's kind of hard to tell. I'll do some more testing but so far this is the first issue I've ran into in months. I guess one observation is that it is really hard to tell what bro/broker are doing. Before you could minimally tcpdump the communication and see what events were being sent back and forth, but now that is encrypted. ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 6, 2018, 8:11pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/5 "2018-09-06T20:11:41Z")
I tested an almost stock local.bro (a few additional things disabled) and saw the same thing. fa7fa5aa is fine, but with 452eb0cb everything is working really hard to do something. The most noticeable thing is the network traffic on the manager changing from being almost idle at 300KiB TX and 3MiB RX and jumping to 20MiB TX and 12MiB RX ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 6, 2018, 8:14pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/6 "2018-09-06T20:14:51Z")
> I just got 2 clusters upgraded from > > fa7fa5aa to > 452eb0cb > > And now everything is broken.. > > cpu and memory are through the roof across the board, as well as network traffic, but it's not logging much. > > I may have created a message loop replacing the relay\_rr stuff, but it's kind of hard to tell. The recent forwarding changes would be my main suspicion and, at least in the default scripts, there's no communication patterns that actually make use of the automatic forwarding, so can you check if adding "redef Broker::forward\_messages = F;" to site/local.bro makes a difference? If it does fix things, then yeah, either I missed a forwarding loop in the default scripts or potentially you introduced one when replacing relay\_rr (feel free to point me at stuff to look over). (Generally may want to just leave message forwarding turned off due to these types of dangers if that's what it turns out to be...). > I guess one observation is that it is really hard to tell what bro/broker are doing. Before you could minimally > tcpdump the communication and see what events were being sent back and forth, but now that is encrypted. You can redef Broker::disable\_ssl=T. I don't recall how readable the non-encrypted communications are, but I think I did it at least once or twice and still was able to spot event names. - Jon ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 6, 2018, 8:19pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/7 "2018-09-06T20:19:16Z")
Thanks for that, I'll start looking into it, but still would be helpful if you could try disabling message forwarding (or disable ssl + look at some captured traffic to see if you can understand what might be happening). Thanks. - Jon ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 6, 2018, 8:34pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/8 "2018-09-06T20:34:52Z")
Yeah, that fixed it! I re-enabled that and then disabled ssl and I am looking at the comm stuff going to the logger, which should just be logs This seems to work for basic quick analysis: [root@bro40-dev ~]# tcpdump -n -i em1 port 47761 -A|sed "s/\.\.\.\.\./\n/g"|egrep -io broker.\* |head -n 10000|sort|uniq -c|sort -nr tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes tcpdump: Unable to write output: Broken pipe    8842 broker::topic+broker::internal\_command+@u32.bro/known/certs/\<$\>/data/clone    1124 broker::topic+broker::internal\_command+@u32.bro/known/hosts/\<$\>/data/clone       8 broker::internal\_command+@u32.bro/known/certs/\<$\>/data/clone       5 broker::topic+broker::internal\_command+@ ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 6, 2018, 11:40pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/9 "2018-09-06T23:40:00Z")
Thanks, there was an unintended forwarding loop in data store communication. It's fixed in master now, but I've also just reverted to generally disabling the forwarding mechanisms by default (any specific/advanced use-cases can always choose to selectively enable it, but the default cluster config doesn't need it for anything currently). For anyone doing testing, please pull the latest change from master and give feedback on that version. - Jon ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 7, 2018, 8:41pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/10 "2018-09-07T20:41:16Z")
Latest master works much better. One thing I'm still seeing when I switch from an old version to latest master is that huge spike in Content switches/interrupts and cpu spent in the kernel. On one test worker box (that is receiving a full 10G and a bit overloaded) before: context switches 200k/sec, interrupts 150k/sec after: context switches 1mil/sec, interrupts 700k/sec Before, cpu maxed out but spending 60% in user and 30% in system After, cpu maxed out but spending 12% in user and 80% in system On the manager side, cpu went up a little, but not by much. I tried redef Broker::max\_threads=2; but that didn't make much of a difference. ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 7, 2018, 10:58pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/11 "2018-09-07T22:58:50Z")
I just updated the default tuning parameters for CAF's scheduling policy and exposed them all in Bro [1] and get within 10% of the number of context switches that 2.5.5 had in a very simple test case. Let me know how that goes for everyone. [1] [https://github.com/bro/bro/blob/4bd6da71864b4ab68b372954c6268a023d69e52b/scripts/base/frameworks/broker/main.bro#L64-L94](https://github.com/bro/bro/blob/4bd6da71864b4ab68b372954c6268a023d69e52b/scripts/base/frameworks/broker/main.bro#L64-L94) ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 8, 2018, 3:20pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/12 "2018-09-08T15:20:05Z")
That helped a little with the context switches, but not much with the cpu load. Do many of those options do anything? I tried looking in the CAF source to figure out how they are used, and it looks like they are all defined in libcaf\_core/caf/actor\_system\_config.hpp as   // -- work-stealing parameters -----------------------------------------------   size\_t work\_stealing\_aggressive\_poll\_attempts CAF\_DEPRECATED;   size\_t work\_stealing\_aggressive\_steal\_interval CAF\_DEPRECATED;   size\_t work\_stealing\_moderate\_poll\_attempts CAF\_DEPRECATED;   size\_t work\_stealing\_moderate\_steal\_interval CAF\_DEPRECATED;   size\_t work\_stealing\_moderate\_sleep\_duration\_us CAF\_DEPRECATED;   size\_t work\_stealing\_relaxed\_steal\_interval CAF\_DEPRECATED;   size\_t work\_stealing\_relaxed\_sleep\_duration\_us CAF\_DEPRECATED; ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 8, 2018, 4:10pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/13 "2018-09-08T16:10:35Z")
Scratch that.. the deprecation refers to something else, they are still used here: work\_stealing::worker\_data::worker\_data(scheduler::abstract\_coordinator\* p)     : rengine(std::random\_device{}()),       // no need to worry about wrap-around; if `p-\>num_workers() \< 2`,       // `uniform` will not be used anyway       uniform(0, p-\>num\_workers() - 2),       strategies{{         {CONFIG("aggressive-poll-attempts", aggressive\_poll\_attempts), 1,          CONFIG("aggressive-steal-interval", aggressive\_steal\_interval),          timespan{0}},         {CONFIG("moderate-poll-attempts", moderate\_poll\_attempts), 1,          CONFIG("moderate-steal-interval", moderate\_steal\_interval),          CONFIG("moderate-sleep-duration", moderate\_sleep\_duration)},         {1, 0, CONFIG("relaxed-steal-interval", relaxed\_steal\_interval),          CONFIG("relaxed-sleep-duration", relaxed\_sleep\_duration)}}} {   // nop } So that strategies is an array of structs of [attempts, step size, steel interval, sleep interval] so as I understand it, the current default is 5 polls, 1 step, interval 4, 0 sleep 5 polls, 1 step, interval 2, 16msec sleep 1 poll, 0 step, interval 1, 64msec sleep Which means if a thread has no work to do it will loop for 5 times with no sleep, then 5 more times with 16msec sleep, then once with 64ms sleep. This doesn't seem that bad.. worse case is 11 attempts every 149ms or ~70/second. I went as far as trying redef Broker::aggressive\_polls = 1; redef Broker::moderate\_polls = 1; redef Broker::moderate\_sleep = 100msec; redef Broker::relaxed\_sleep = 200msec; but that didn't make a noticeable change, so the scheduler may not be the problem. Will have to do some more testing. ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 10, 2018, 4:09pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/14 "2018-09-10T16:09:42Z")
I did some more testing and profiling and figured out what is going on.. The new version is much more efficient, so it's spending a lot less time in user space and a lot more time in the kernel fetching packets. On the managers, overall CPU using is a bit lower, but I think some of that is from removing all uses of &synchronized, especially for large input tables. If anything there's just a slight overhead in the bro-myricom plugin in how it uses snf\_ring\_recv to receive a single packet instead of snf\_ring\_recv\_many to grab multiple packets like how af\_packet works. I'd look into fixing it but we are moving to intel 40g cards anyway. ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 12, 2018, 2:17pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/15 "2018-09-12T14:17:18Z")
Hi again! Just finished the migration to master across the board, and it’s looking REALLY good. No crashes, memory is stable, cpu is pretty good. The only thing I noticed is the manager userspace CPU utilization on one cluster jumped up a bit after the switch. This graph shows the system and userspace utilization across an 8 physical node cluster. All the lower lines are the system usage and the higher lines are userspace. The Y axis is cpu seconds per second, so 4 is full 4 cores worth of usage. After the switchover: - the worker userspace cpu utilization was unchanged - the worker system cpu utilization increased a bit, but I believe that is due to more time spent in the myricom driver waiting for packets - the manager system cpu utilization dropped a bunch - the manager userspace cpu increased 1-3x The manager box in this cluster only runs the manager and logger processes, no proxies. It also has something like 20 idle cores, so this isn’t a problem at all, but could affect people who run a cluster-in-a-box. ![](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/076ee4263af726a334706e3190d973765bc881ab.jpeg) I do seem to be seeing a bunch of reporter errors like Reporter::ERROR string with embedded NUL: “\x00\x00\x00\x00OPTIONS” but I’m not sure if that is a new thing. ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 12, 2018, 3:14pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/16 "2018-09-12T15:14:34Z")
> Just finished the migration to master across the board, and it's looking REALLY good. Great, thanks for helping test and provide performance data. > The manager box in this cluster only runs the manager and logger processes, no proxies. It also has something like 20 idle cores, > so this isn't a problem at all, but could affect people who run a cluster-in-a-box. An idea in this type of situation could be to tune Broker::max\_threads per node type. E.g. leave at 1 for workers and bump to ~4 for manager/logger since there's idle cores on their host and they're inherently in a less-scalable/centralized location. That may not lower overall cpu usage, but may help prevent some bottlenecks in the processing of remote messages. Particularly the work of processing data store communication should distribute among threads, potentially each data store could be processing messages independently on separate threads. (The default scripts have 3 stores, one for each known-\* script). > I do seem to be seeing a bunch of reporter errors like > > Reporter::ERROR string with embedded NUL: "\\x00\\x00\\x00\\x00OPTIONS" Ok, not necessarily that bad, but would be nice to find where that's coming from to handle it more properly - Jon ---
**Author:** ![Alan\_Commike](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@Alan\_Commike](https://community.zeek.org/u/Alan_Commike)\ **Post date:** [September 12, 2018, 4:57pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/17 "2018-09-12T16:57:52Z")
For those that haven’t yet dug into all that is Broker, this brings up a question on the general architecture. Are there still parent/child processes handling comms/work? I’d assume the fork is gone, so the new code is dropping a process but gaining a thread, in essence a wash. Is there a mechanism today for per node type tuneables? …alan ---
**Author:** ![Jon\_Siwek](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Jon\_Siwek](https://community.zeek.org/u/Jon_Siwek)\ **Post date:** [September 12, 2018, 7:26pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/18 "2018-09-12T19:26:29Z")
> Are there still parent/child processes handling comms/work? No. Single process, configurable number of threads (default 1). > Is there a mechanism today for per node type tuneables? One should be able to use an @if directive [1] to tune differently per-node if they want. - Jon [1] [https://www.bro.org/sphinx-git/script-reference/directives.html](https://www.bro.org/sphinx-git/script-reference/directives.html) ---
**Author:** ![JA1](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@JA1](https://community.zeek.org/u/JA1)\ **Post date:** [September 13, 2018, 9:57pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/19 "2018-09-13T21:57:54Z")
Could anyone update Bro on docker to include 2.5.5, 2.6-beta and master? [https://hub.docker.com/r/broplatform/bro/tags/](https://hub.docker.com/r/broplatform/bro/tags/) We use this with our internal trybro instance which is fantastic for quickly collaborating and testing scripts. 🙂 ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [September 14, 2018, 1:54pm UTC](https://community.zeek.org/t/bro-2-6-beta-plans/5443/20 "2018-09-14T13:54:02Z")
Working on it now 🙂 There isn't an official 2.6 beta, but I will build and upload 2.5.5 and current master. [Next page](https://community.zeek.org/t/bro-2-6-beta-plans/5443.md?page=2)