# early performance comparisons of CAF-based run loop
**URL:** https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771
**Category:** Development
**Tags:** development
**Created:** [April 12, 2017, 12:41am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771 "2017-04-12T00:41:23Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 12, 2017, 12:41am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/1 "2017-04-12T00:41:23Z")
I recently got a minimal CAF-based run loop for Bro working, did crude performance comparisons, and wanted to share.
The approach was to measure average time between calls of net\_packet\_dispatch() and also the average time it takes to analyze a packet. The former attempts to measure the overhead imposed by the loop implementation and the later just gives an idea of how significant a chunk of time that is in relation to Bro’s main workload. I found that the overhead of the loop can be ~5-10% of the packet processing time, so it does seem worthwhile to try and keep the run loop overhead low.
Initial testing of the CAF-based loop showed the overhead increased by ~1.8x, but there was still a major difference in the implementations: the standard Bro loop only invokes its IOSource polling mechanism (select) once every 25 cycles of the loop, while the CAF implementation’s polling mechanism (actor/thread scheduling + messaging + epoll) is used for every cycle/packet. As one would expect, by just trivially spinning the main process() function in a loop for 25 iterations, the overhead of the CAF-based loop comes back into line with the standard run loop.
To try and better measure the actual differences related to the polling mechanism implementation, I quickly hacked Bro’s standard runloop to select() on every packet instead of once every 25th and found that the overhead measures +/- 10% within the 1.8x overhead increase of the initial CAF-based loop. So is the cost of the extra system call for epoll/select per packet the main thing to avoid? Sort of. I again hacked Bro’s standard loop to be able to use either epoll or poll instead of select and found that those do better, with the overhead increase being about 1.3x (still doing one “poll” per packet) in relation to the standard run loop. Meaning there is some measurable trend in polling mechanism performance (for sparse # of FDs/sources): poll comes in first, epoll second, with CAF and select about tied for third.
Takeaways:
(1) Regardless of runloop implementation or polling mechanism choices, performing the polling operation once per packet should probably be avoided. In concept, it’s an easy way to get a 2-5% speedup in relation to total packet processing time.
(2) Related to (1), but not in the sense of performance, is that even w/ a CAF-based loop it still seems somewhat difficult to reason about the reality of how IOSources are prioritized. In the standard loop, the priority of an IOSource is a combination of its “idle” state, the polling frequency, and a timestamp, which it often chooses arbitrarily as the “time of last packet”, just so that it gets processed with higher priority than subsequent packets. Maybe the topic of making IOSource prioritization more explicit/well-defined could be another thread of discussion, but my initial thought is that the whole IOSource abstraction may be over-generalized and maybe not even needed.
(3) The performance overhead of a CAF-based loop doesn’t seem like a showstopper for proceeding with it as a choice for replacing the current loop. It’s not significantly worse than the current loop (provided we still throttle the polling ratio when packet sources are saturated), and even using the most minimal loop implementation of just poll() would only be about a 1% speedup in relation to the total packet processing workload.
Just raw data below, for those interested:
I tested against the pcaps from [http://tcpreplay.appneta.com/wiki/captures.html](http://tcpreplay.appneta.com/wiki/captures.html)
(I was initially going to use tcpreplay to test performance against a live interface, but decided reading from a file is easier and just as good for what I wanted to measure).
Numbers are measured in “ticks”, which are equivalent to nanoseconds on the test system.
Bro and CAF are both compiled w/ optimizations.
bigFlows.pcap, 1 “poll" per packet
---
### Author:  [@Slagell\_Adam\_J](https://community.zeek.org/u/Slagell_Adam_J)
#### Post date: [April 12, 2017, 6:35pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/2 "2017-04-12T18:35:38Z")
Justin asked an interesting question today, how does this affect performance on the manager? That is where we are feeling a lot of pain with select().
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 13, 2017, 2:05am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/3 "2017-04-13T02:05:52Z")
If you mean the select() that’s in the process fork’d by the old RemoteSerializer code, you’d still see the same problems with the CAF-based runloop. But that code is irrelevant once Broker takes its place. i.e. to answer that question, you need to design a communication stress test using Broker-based Bros as that’s more relevant than just changing the main loop.
Eventually, I can also imagine the Broker-based communication being more tightly integrated into the CAF-based runloop helping improve performance over the current Broker integration method. Either way, what needs to be measured is how CAF’s multiplexer performs in relation to Bro’s communication patterns, but maybe still want to wait for the Broker improvements to wrap up before looking into doing those tests.
In the near-term, I can make a totally separate code branch that simply replaces select() with epoll. Then, if Justin were to test it and find it alleviates performance pains on the manager, it could potentially get merged into bro/master ahead of the any of the pending broker/caf/runloop projects since it should be a trivial and safe change to do. Let me know.
- Jon
---
### Author:  [@Clark\_Gilbert](https://community.zeek.org/u/Clark_Gilbert)
#### Post date: [April 13, 2017, 3:19am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/4 "2017-04-13T03:19:02Z")
$0.02 USD:
As I recall, Bro’s per-packet processing overhead can vary significantly as a result of timers and triggers that execute on a situational basis. Also, relative overhead of packet ingest is going to vary based on the set of loaded scripts in addition to the specific trace used to run the tests. That’s not trying to argue that these results are not useful / interesting, but instead _only_ that the specific percentages might not be representative of the general case (just because I’m convinced that there really is not a general case to objectively measure).
Also … if the overhead of the polling / ingest itself turns out to be a huge problem at high rates, one idea would be to separate that and pass packets (in bulk) through a ring / high-speed IPC to the process that needs to ingest them. That’s worked pretty well for me in DPDK, and has the benefit of being able to distribute packets from one ingest to multiple processors (which is something I’ve had to do for process-heavy workloads … which I would argue is something that Bro tends to be).
Along those lines, rather than spending much time on packet ingest mechanics in bro (or pieces thereof), one idea might be to instead focus on integrating packet bricks as a standard ingest / distribution mechanic for everything packet-related in the general case. The idea would be that fetching packets from bro (and its related processes) would become less about calls to epoll and select, and more about high-speed IPC that went out of its way to avoid kernel-space entirely. The nice thing about that is that it’d be a little easier to standardize on the bro side of things, and would take a step toward separating bro as a scripting / event engine from bro as a (relative) monolith.
Of course, the down side is that packet bricks could add some serious (mandatory) complexity to bro, so maybe it’s not the right answer … but maybe a more lightweight, specialized distribution channel might be doable, or maybe there would be a way to embed packet bricks inside of an application in the event that folks didn’t want to run the two separately, or … etc.
As always, just for what it’s worth 🙂
-Gilbert
---
### Author:  [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)
#### Post date: [April 13, 2017, 3:19am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/5 "2017-04-13T03:19:47Z")
> > Justin asked an interesting question today, how does this affect performance on the manager? That is where we are feeling a lot of pain with select().
>
> If you mean the select() that’s in the process fork’d by the old RemoteSerializer code, you’d still see the same problems with the CAF-based runloop. But that code is irrelevant once Broker takes its place. i.e. to answer that question, you need to design a communication stress test using Broker-based Bros as that’s more relevant than just changing the main loop.
Yep, that select stuff. My question was mostly about the different workloads in a bro cluster.
Something that may be optimized for a worker dealing with 1 pktsrc and 2 peers may not be as optimal for a logger/manager that has no pktsrc but 100+ worker connections. I've often wondered if the event loop should have a hint somewhere about which kind of process is running so it can optimize for throughput vs multiplexing many peers.
> Eventually, I can also imagine the Broker-based communication being more tightly integrated into the CAF-based runloop helping improve performance over the current Broker integration method. Either way, what needs to be measured is how CAF’s multiplexer performs in relation to Bro’s communication patterns, but maybe still want to wait for the Broker improvements to wrap up before looking into doing those tests.
>
> In the near-term, I can make a totally separate code branch that simply replaces select() with epoll. Then, if Justin were to test it and find it alleviates performance pains on the manager, it could potentially get merged into bro/master ahead of the any of the pending broker/caf/runloop projects since it should be a trivial and safe change to do. Let me know.
Ah.. I had actually started trying to do that a long time ago, but gave up because broker was going to replace all of that code anyway.
> **[Commits · bro/bro](https://github.com/bro/bro/commits/topic/jazoff/select-to-poll)**
>
> Mirror of https://github.com/zeek/zeek. Contribute to bro/bro development by creating an account on GitHub.
from what I recall the first commit seemed to work but the second broke something.
The thing that always stood out to me was that the manager would run select across all the worker sockets, and then loop over each worker and run CanRead, which just ran select again on each individual FD.
One issue a few people have run into on the manager is that select returns EINVAL and deadlocks bro if you give it a FD larger than 1024, which you currently hit on around a 200 node cluster (socket + flares use 4 or 5 FDs per worker).
---
### Author:  [@Slagell\_Adam\_J](https://community.zeek.org/u/Slagell_Adam_J)
#### Post date: [April 13, 2017, 4:29pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/6 "2017-04-13T16:29:37Z")
That might be useful. I would like Robin’s thoughts, too.
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 13, 2017, 7:52pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/7 "2017-04-13T19:52:38Z")
I agree, the specific numbers here aren’t generalizable, but I think that’s ok and we can still infer that the different runloop implementation doesn’t raise any obvious performance concern. That being due to (1) with the tests using the default set of Bro scripts, I’d expect it to be more common for users to have more complicated scripts and highly customized deployments such that the relative overhead decreases further and becomes more irrelevant than the tests show and (2) even if the specific pcaps tested were at either end of the spectrum in terms of how much work is required to process them, it still shows that the relative overhead differences are minimal.
i.e. I think we’d only be in trouble interpreting the results if the tests showed a significant relative overhead difference. Since then we don’t know if the given pcaps where just “easy” ones for Bro to process.
- Jon
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 13, 2017, 8:12pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/8 "2017-04-13T20:12:48Z")
Yeah, I’ve thought the same and related with takeaway (2) that I mention in the original post. It seems like it would be nice to have a more well-defined system for specifying IOSource prioritization or at least between packet sources and other io sources. Then, since it’s hard to nail down settings that are going to work for all deployments in general, it would also have ways to tune it via scripts so it would be open for a user to tweak settings that may improve for their particular manager, logger, worker, or whatever Bro node they have.
But at the moment, I don’t think there’s a whole lot of info on exactly what tweaks can be made to optimize for the “no pktsrc, but lots of remote comm.” case, and it may be best to wait for the broker integration to become fully realized before investigating that.
- Jon
---
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [April 14, 2017, 1:32pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/9 "2017-04-14T13:32:11Z")
Nice, thanks for the doing these measurements! I haven't looked at the
code yet, but some quick thoughts on your results and some of the
other comments this thread, and then some suggested next steps at the
end.
- Agree that overall your numbers suggest that all these mechanisms
are fine performancewise, assuming we keep the optimization to batch
packets between polls/selects to avoid the
one-system-call-per-packet overhead.
- I don't think we should spend time anymore on improving the old
communication code. We're getting close to retire that now and a
number of its issues (like selects in the child process) will just
go away with that. Let's focus on the new setting where Broker/CAF
will be doing all communication.
- Regarding optimizing for different use cases: I would prefer
avoiding having lots of knobs to configure the specifics of the
loop. We have these magic values in the current I/O loop where
nobody knows how to pick them because it's hard to understand their
impact; and where folks have played with them, it was always hard
conclude much about them beyond any specific setting. What we could
try instead is a loop that adjusts itself based on load patterns: if
the load is heavy on packets, build larger batches to process
between polls; if input comes from lots different sources, increase
the polling; etc. Any heuristic here would need to stay quite simple
(otherwise we'd again end up not being able to predict much), but I
think that'd be worth a try.
- Gilbert's point on high-performance IPC is a good one. I don't think
we want to switch to direct memory access as our main model for the
time being at least, but it does pose the question if/how can
integrate packet sources that either don't need or don't support
select/poll. (Which, in a nod to history, accounts for some of the
complexities of the current loop because many years ago some pcaps
didn't support select)
In terms of next steps, we need to see if these results hold across
different OSs, and also with live traffic. The two questions are (1)
does the new loop function on all platforms with both low- and
high-volume live traffic (presumably it will but that needs double
checking, given the history of weird OS-specific effects); and (2)
does performance match the measurements shown so far? If we can
confirm that on at least Linux and FreeBSD for, say, the two most
recent major releases of each and also consider common alternative
capturing solutions (pfring, netmap, afnet?), I'd be pretty
comfortable switching.
Robin
---
### Author:  [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)
#### Post date: [April 14, 2017, 2:56pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/10 "2017-04-14T14:56:09Z")
> If we can confirm that on at least Linux and FreeBSD for, say, the two
> most recent major releases of each and also consider common
> alternative capturing solutions (pfring, netmap, afnet?), I'd be
> pretty comfortable switching.
Just a quick comment here regarding FreeBSD: the native polling
mechanism is kqueue, and CAF still lacks support for it [1].
Fortunately, this is a rather straight-forward task.
Matthias
[1] [Add support for kqueue in multiplexer · Issue #533 · actor-framework/actor-framework · GitHub](https://github.com/actor-framework/actor-framework/issues/533)
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 14, 2017, 5:32pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/11 "2017-04-14T17:32:35Z")
> - I don't think we should spend time anymore on improving the old
> communication code. We're getting close to retire that now and a
> number of its issues (like selects in the child process) will just
> go away with that. Let's focus on the new setting where Broker/CAF
> will be doing all communication.
If people are hitting the 1024 FD hard-limit in the old comm. code’s select(), that would indeed go away with the change to Broker. But I think the way Broker is integrated in the parent’s main loop still relies on a select(), with the number of FDs it monitors scaling with the number of peers. i.e. there may still be critical errors w/ large Bro clusters even using Broker as the communication system, just this time the problem manifests in the main loop.
Just mentioning it in case you didn’t account for the real fix also requiring the CAF-based loop being fully realized in addition to Broker — I’m less certain about the timeline of finishing up the CAF-based loop compared to just patching in a temporary stopgap of patching out the select() calls. (Also don’t have a sense of the frequency/urgency of the problem).
> - Regarding optimizing for different use cases: I would prefer
> avoiding having lots of knobs to configure the specifics of the
> loop. We have these magic values in the current I/O loop where
> nobody knows how to pick them because it's hard to understand their
> impact; and where folks have played with them, it was always hard
> conclude much about them beyond any specific setting. What we could
> try instead is a loop that adjusts itself based on load patterns: if
> the load is heavy on packets, build larger batches to process
> between polls; if input comes from lots different sources, increase
> the polling; etc.
That seems like a Good Idea.
> it does pose the question if/how can
> integrate packet sources that either don't need or don't support
> select/poll
I think that’s just a matter of making sure the main loop “spins” at an appropriate frequency, which might change dynamically, dependent on loading pattern optimizations, as per the above idea.
Maybe you could even think of reading an offline pcap file as a source that doesn’t need select/poll. Pedantically, regular files also don't “support” select(), at least not w/ the same intention (nonblocking IO), but it just happens to work fine in the current runloop implementation.
So since I’ve been able to get the CAF-based loop working on offline pcap files (it does not rely on polling the FD of the open file since it didn't work anyway w/ CAF's epoll-based multiplexer on Linux), it may be fair to say that other packet sources that don’t require/support poll-ability should also be possible to integrate.
- Jon
---
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [April 19, 2017, 7:55am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/12 "2017-04-19T07:55:30Z")
Oh, sounds like that would be high-priority task then before we'd
consider moving to a CAF-based loop?
Robin
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 19, 2017, 3:24pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/13 "2017-04-19T15:24:08Z")
It still falls back to ‘poll’ on non-Linux. If more performance tests are to be done in realistic conditions (live traffic + cluster communication) on various platforms, you’d likely find out at that point whether it’s a high-priority task. I expect ‘poll’ might still be usable if the common case is only going to max out around the order of 100s of peers.
- Jon
---
### Author:  [@Matthias\_Vallentin1](https://community.zeek.org/u/Matthias_Vallentin1)
#### Post date: [April 19, 2017, 8:40pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/14 "2017-04-19T20:40:53Z")
> Oh, sounds like that would be high-priority task then before we'd
> consider moving to a CAF-based loop?
I've added kqueue support in topic/kqueue, but it's still missing the
final touch. (Unit tests are still failing.) Hopefully it's not a long
way from here.
Matthias
---
### Author:  [@robin](https://community.zeek.org/u/robin)
#### Post date: [April 20, 2017, 5:40pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/15 "2017-04-20T17:40:36Z")
> Just mentioning it in case you didn’t account for the real fix also
> requiring the CAF-based loop being fully realized in addition to
> Broker
Yeah, true, I was thinking that eventually we will have this all solved.
> (Also don’t have a sense of the frequency/urgency of the problem).
I think that's the main question. So far I haven't gotten the sense
that this really affects a lot of people, so I see the priority as
rather low given our limited cycles for development and testing. If
it's a more pressing problem, we can reconsider of course.
> So since I’ve been able to get the CAF-based loop working on offline
> pcap files (it does not rely on polling the FD of the open file since
> it didn't work anyway w/ CAF's epoll-based multiplexer on Linux), it
> may be fair to say that other packet sources that don’t
> require/support poll-ability should also be possible to integrate.
I need to think about that argument ... Did you try reading from files
while also doing communication (that would be pseudo-realtime mode),
or was the pcap the only source of input?
Robin
---
### Author:  [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)
#### Post date: [April 21, 2017, 3:13am UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/16 "2017-04-21T03:13:37Z")
Tested:
\* pcap
\* pcap + script doing DNS queries
\* live interface
\* live interface + script doing DNS queries
Untested:
\* anything with remote communication
\* pseudo-realtime
- Jon
---
### Author:  [@system](https://community.zeek.org/u/system)
#### Post date: [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/early-performance-comparisons-of-caf-based-run-loop/4771/17 "2022-05-06T15:44:49Z")