@biswa - one more follow up, you may want to consider or experiment running with redef io_poll_interval_default = 1. In zeek -r mode, broker communication is processed every 100 packets after a continue_processing() call. You may see more “interactive” communication behavior by lowering the poll_interval, at the cost of more polling system calls.
You’ll obviously experiment with a suitable setting in your environment, but wanted to point this out as I’ve been stumbling over this lately, too.
@biswa, could you say a bit more about what your Python endpoint actually does with the received events? Is it a plain “sink” for the subscribed events and you’re using Python mainly for convenience? Or are you also sending events back into Zeek?
The discussion in this thread is very valuable (as you can tell, you might be among the first people to combine pcap processing with a Broker-peered Python script) but I’m wondering whether the overall model is going to work for you even once we overcome the startup issues.
Hi @Christian
I am using python broker both as subscriber and publisher. This broker daemon is not just for sinking, but the goal is to do few computationally intensive analysis and creating offline databases for further query, in the python broker with the information I receive from zeek and generate new events and send back to zeek to dump in it’s log file. However I am using pcap just because right now I don’t have live setup with me. That’s just coincidence you can say, but the positive side is that pcap testing exposed the bug.
Let me know if you need any other clarification.
Regards,
Biswa
Thanks Subhajit — that all makes sense. If you want a quick & dirty replay setup to work around the bug, you could do something like
# ip link add dummy0 type dummy
# ifconfig dummy0 up
Then run zeek on dummy0, and tcpreplay your pcap onto that interface. This assumes Linux and is a bit of a hack, but also has advantages until you run “properly” on live traffic — you get to control when traffic flows, and Zeek naturally keeps running once you’ve processed the pcap.
Like Arne said, kudos for your use-case. If your Python endpoint does something generally useful, consider sharing it!
If your Python endpoint does something generally useful, consider sharing it!