Hi Zeek,
Hope you’re all doing well.
I am using an Input framework to provide some dynamic input to bro.
Here’s how the infra looks like,
cat /usr/local/bro/share/bro/base/protocols/file_port_list
#fields port_num
60000/tcp
8080/tcp
49154/tcp
55907/tcp
49152/tcp
49153/tcp
8000/tcp
5357/tcp
type Portsx: record {
port_num: port;
};
global file_port_list: set[port] = set();
redef Communication::nodes += {
[“python”] = [$host = 127.0.0.1, $events = /config_update/, $connect=F, $ssl=F]
};
event bro_init()
{
Input::add_table([$source=file_port_list_loc, $name=“file_port_list”, $idx=Portsx, $destination=file_port_list]);
}
event bro_done()
{
Input::remove(“file_port_list”);
}
event config_update()
{
Input::force_update(“file_port_list”);
}
It works really well. But when I terminate bro process, it goes into defunct state and throws an error on the console.
Error: received signal while waiting for thread /usr/local/bro/share/bro/base/protocols/file_port_list/Input::READER_ASCII, aborting all …
Anything am I doing wrong??
Thanks a lot for looking…!!
Regards,
Nabil
Try deleting this part:
event bro_done()
{
Input::remove("file_port_list");
}
Hi Justin,
I didn’t try that yet.
I got caught up in some other side effect. Bro’s CPU usage goes 100% when I use input framework(or maybe my understanding is incorrect). I have no idea what triggered this.
I tested with and without two lines(integrating bro’s input framework) in the bro script.
test.bro (On which I observe 100% CPU usage):
event bro_init()
{
What version are you using? You should try comparing against one of
the latest 3.0.x or 3.1.x releases since there's specifically things
addressed in them that might explain that behavior. E.g.
https://github.com/zeek/broker/pull/97
- Jon
I am using 2.6.x version.
Hi Jon,
Even in the latest zeek version which is 3.1.3, it reaches 100% when I integrate Input framework,
cat /tmp/file_port_list
#fields port_num
60000/tcp
8080/tcp
49154/tcp
55907/tcp
49152/tcp
49153/tcp
8000/tcp
5357/tcp
80/tcp
cat test.zeek
global file_port_list: set[port] = set();
global file_port_list_loc: string = “/tmp/file_port_list”;
type Portsx: record {
port_num: port;
};
event zeek_init() &priority=5
{
suspend_processing();
Input::add_table([$source=file_port_list_loc, $name=“file_port_list”, $idx=Portsx, $destination=file_port_list]);
Input::remove(“file_port_list”);
}
Regards,
Nabil
Are you missing a matching call to `continue_processing()` ?
event Input::end_of_data(name: string, source: string)
{
if ( name == "file_port_list" )
continue_processing();
}
- Jon
Hi Jon,
I want to keep bro in suspended mode until a signal through broccoli framework kicks in.
I see bro utilizing 100% CPU when in suspended state. And normal percentage CPU(roughly 15-20%) usage when in continued state.
Why bro is taking up so much CPU when in suspended state?
cat test.bro
I see bro utilizing 100% CPU when in suspended state. And normal percentage CPU(roughly 15-20%) usage when in continued state.
Why bro is taking up so much CPU when in suspended state?
From a historical explanation: I don't know, but maybe because it
wasn't meant for general usage or only for some niche use-cases where
some short-term high CPU usage was acceptable (the main use-case I
know is currently in Zeek's test suite to cause determinism in some
input/IPC tests that otherwise have subtle races).
From a technical explanation: likely it's because the suspend/continue
functions don't remove the PktSrc file descriptor from the IO/polling
loop so it's always considered "ready" but also never processes
anything to drive it forward out of that "ready" state due to being
"suspended".
The technicalities of suspend/continue performing an
unregister/register the PktSrc FD are changeable if you want to make a
GitHub issue for that enhancement (also may help if you provide more
explanation of your use-case and the importance of this to it).
Though might not expect any changes related to this happening until
Zeek 3.2 or later -- 3.0.x has a different IO loop that's trickier to
change, 3.1.x is a bit easier to change, but this isn't a clear-cut
"regression" that suits a patch release, and 2.6.x or before are no
longer supported.
- Jon
Thanks a lot for the thorough explanation.
I really appreciate it.
From the above explanation, it means that, bro is expecting to be continued soon after the suspend call to tackle subtle race cases as you mentioned and there’s no rest in between(suspend & continue) which can stabilize the CPU load(basically it polls for it to be continued again). I hope it cleared my understanding of how suspend/continue_processing() calls works.
My use case is I want bro not to process any packets from any source(netmap pipe or libpcap) until I somehow sends a signal for bro to be continued. The use case is achieved even now but at a higher cost of CPU being loaded.
if you want to make a GitHub issue for that enhancement (also may help if you provide more explanation of your use-case and the importance of this to it).
Okay…
Thanks a lot again !!
Regards,
Nabil