Bro logs - enable_local_logging and remove_default_filter

Zeek
1

Hi all,

Can you please help to explain how to disable local logging ? I am using the KafkaWriter Bro plugin for many years now without a problem but after an upgrade to Bro 2.6 there is a problem.

The logs that are excluded from sending to Kafka are the logs that are being written to disk. In Bro config language that means the logs that are not explicitly defined in KafkaLogger::logs_to_send.

Example from local.bro for KafkaLogger:

redef KafkaLogger::logs_to_send( CaptureLoss::LOG, etc… )

Historically I modify the KafkaLogger plugin slightly to support disabling the writing of logs to disk by adding a function call to “Log::remove_default_filter” for each log. With Bro 2.6 this no longer seems to work the way it once did.

So I check the documentation at https://www.bro.org/sphinx-git/scripts/base/frameworks/logging/main.bro.html and see remove_default_filter still exists and also notice two variables that might be relevant to my issue.

Log::enable_local_logging: bool &redef | If true, local logging is by default enabled for all filters. |

But when I try to set Log::enable_local_logging=0 within the KafkaLogger plugin loop for each log I get an error.

Thanks in advance.

-Hovsep

2

Hi Hovsep,

[...]

Historically I modify the KafkaLogger plugin slightly to support disabling
the writing of logs to disk by adding a function call to
"Log::remove_default_filter" for each log. With Bro 2.6 this no longer
seems to work the way it once did.

I just looked and I did not really see any big way in which this changed.
Could you perhaps provide a code-snippet that does not work anymore?

I also just tried a minimal example script and Log::remove_default_filter
seems to work as expected.

[...]

But when I try to set Log::enable_local_logging=0 within the KafkaLogger
plugin loop for each log I get an error.

This is probably a misunderstanding. Log::enable_local_logging is not a
per-log setting - so there is nothing to loop over.

If you do a

redef Log::enable_local_logging = F;

The setting will persist. That being said, you will very probably not want
to enable this, it means something slightly different than what you
expect. Remote logging means that a log is sent to a remote Bro
instance; local logging means that logging is performed by the current
node. If you set enable_local_logging to false on a node, it will not
output any kind of logs directly itself - this includes sending logs to
Kafka - from a Bro point of view, these are local logs (the logging is
performed by the local node).

By default this is set to "T" in standalone mode; in clusters the setting
is "T" on Logger nodes and "F" on all other nodes. Which is very probably
like you want it.

I hope this helps,
Johanna

3

I just looked and I did not really see any big way in which this changed.
Could you perhaps provide a code-snippet that does not work anymore?

I modify the KafkaLogger script (logs-to-kafka.bro) and add Log::remove_default_filter before the call to Log::add_filter.

I also just tried a minimal example script and Log::remove_default_filter
seems to work as expected.

It works for some of the logs except:

ls -l bro/logs/current/
total 511992
-rw-r–r-- 1 bro bro 1032325 Dec 18 07:15 broker.log
-rw-r–r-- 1 bro bro 666385163 Dec 18 07:15 conn.log
-rw-r–r-- 1 bro bro 12994 Dec 18 07:15 dce_rpc.log
-rw-r–r-- 1 bro bro 223181005 Dec 18 07:15 files.log
-rw-r–r-- 1 bro bro 5780 Dec 18 07:15 smb_files.log
-rw-r–r-- 1 bro bro 3283 Dec 18 07:15 smb_mapping.log
-rw-r–r-- 1 bro bro 5077483 Dec 18 07:15 stderr.log
-rw-r–r-- 1 bro bro 187 Dec 13 14:45 stdout.log

[…]

But when I try to set Log::enable_local_logging=0 within the KafkaLogger
plugin loop for each log I get an error.

This is probably a misunderstanding. Log::enable_local_logging is not a
per-log setting - so there is nothing to loop over.

Ok, thanks.