Broker performance tuning

Zeek
,
4

Hey @biswa61 - I’ve tested with the latest Zeek master and a 6.1.1 container image (you should really update to 7.1) and event throughput appears fine. I’m getting ~6.3k events per second with the following test program:

# test.zeek
global hello: event(c: count);

event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string)
        {
        print "peer_added", endpoint, msg;
        }

event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string)
        {
        print "peer_lost", endpoint, msg;
        }

global c = 0;
event tick()
        {
        Broker::publish("/topic/test", hello, c);
        ++c;
        schedule 0.1msec { tick() };
        }

event zeek_init()
        {
        Broker::listen("127.0.0.1", 9999/tcp);
        event tick();
        }

#!/usr/bin/env python
# client.py
#
# export PYTHONPATH=/opt/zeek-dev-prod2/lib/zeek/python/
import broker

def main():
    print(broker)
    cfg = broker.Configuration()
    cfg.max_threads = 16

    with broker.Endpoint(cfg) as ep, \
        ep.make_subscriber("/topic/test") as sub, \
            ep.make_status_subscriber(True) as ss:
                print("peer")
                ep.peer("127.0.0.1", 9999)

                while True:
                    st = ss.poll()
                    if st:
                        print(st)

                    su = sub.get(0.0000001);
                    if su is None:
                        continue

                    print(su)


if __name__ == "__main__":
    main()

# Terminal 1
$ zeek test.zeek

# Terminal 2
$ taskset -c 1 python3 client.py | pv -l > /dev/null
43,3k 0:00:07 [6,32k/s] [                                                 <=>                                                                                                                                                                                                                                                                                                                ]

If you see drastically less, is this maybe a debug build? Do you use some custom Python version? Is the Python process running at 100% CPU? If so, could you profile it?

If you get also many thousand events per second, then the processing bottleneck is probably somewhere else.

Hope this helps,
Arne