Zeek 3.1.2 and Kafka - No data flow

Hello Zeeks

Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ? I am trying to modernize the metron-kafka plugin and have partial success. My problem seems to be with script-land referencing.

The logger node is loading the plugin OK and connects to the Kafka broker. The broker IP is redef information found from site/local.zeek.

$ bin/zeekctl diag logger-1

No core file found.

Zeek 3.1.2-debug

Zeek plugins:
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)

==== No reporter.log

==== stderr.log
%7|1587948661.341|RECV|rdkafka#producer-3| […kafka messages…] …

But the worker node has a problem referencing existing variable declaration. The logs-to-kafka.bro script expects it. There is also suspicion with the Zeek plugins info that is different from the logger node and maybe the problem.

$ bin/zeekctl diag worker-1-1

No core file found.

Zeek 3.1.2-debug

Zeek plugins: (none found) <<< ??? Normal for worker node ???

==== No reporter.log

==== stderr.log

error in /opt/zeek/spool/installed-scripts-do-not-touch/site/custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro, line 24: unknown identifier logs_to_send, at or near “logs_to_send”

The configuration is not default and explained below:

The Kafka logger was installed to site/custom_plugins/APACHE_KAFKA

share/zeek/site/local.zeek uses:

@load custom_plugins/APACHE_KAFKA/scripts/Apache/Kafka

lib/zeek/plugins/custom_plugins is a symlink to share/zeek/site/custom_plugins

Using the lib symlink seems to be the only way to load the plugin, then the @load statement brings redef customizations and scripts. This works ok for the logger node but not the worker who cannot interface with the plugin ?

Another idea is have non-logger nodes bypass loading logs-to-kafka.bro but this isn’t fully understood.



I have not run it on 3.1.2 yet but I recommend making your changes to the plugin and running the end to end testing script at https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh

It was meant to help isolate issues when making changes to the plugin. Also, we welcome PRs against the project so please feel free to contribute. Thanks,

Jon Zeolla

Were you able to get this working? I’m planning to work on the bro to zeek cutover for the plugin soon.

I have the plugin working with 3.1.2 here - feedback is welcome.

It is not working yet for me and was set aside to fix another time.

Very glad to hear about pull 44, I will test !


Master officially supports Zeek 3.1 now so please let me know if you run into any issues.

I tested this successfully today. Thank you again for your work and assistance.