# Zeek 3.1.2 and Kafka - No data flow

**URL:** <https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103>\
**Category:** Zeek\
**Created:** [April 27, 2020, 1:10am UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103 "2020-04-27T01:10:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hovsep\_Levi](https://avatars.discourse-cdn.com/v4/letter/h/977dab/32.png) [@Hovsep\_Levi](https://community.zeek.org/u/Hovsep_Levi)\
**Post date:** [April 27, 2020, 1:10am UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/1 "2020-04-27T01:10:58Z")

</div>

Hello Zeeks

Has anyone succeeded to enable Kafka plugin with Zeek 3.1.2 ? I am trying to modernize the metron-kafka plugin and have partial success. My problem seems to be with script-land referencing.

The logger node is loading the plugin OK and connects to the Kafka broker. The broker IP is redef information found from site/local.zeek.

$ bin/zeekctl diag logger-1  
[logger-1]

No core file found.

Zeek 3.1.2-debug

Zeek plugins:  
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)

==== No reporter.log

==== stderr.log  
%7|1587948661.341|RECV|rdkafka#producer-3| […kafka messages…] …

But the worker node has a problem referencing existing variable declaration. The logs-to-kafka.bro script expects it. There is also suspicion with the Zeek plugins info that is different from the logger node and maybe the problem.

$ bin/zeekctl diag worker-1-1  
[worker-1-1]

No core file found.

Zeek 3.1.2-debug

Zeek plugins: (none found) \<\<\< ??? Normal for worker node ???

==== No reporter.log

==== stderr.log

error in /opt/zeek/spool/installed-scripts-do-not-touch/site/custom\_plugins/APACHE\_KAFKA/scripts/Apache/Kafka/./logs-to-kafka.bro, line 24: unknown identifier logs\_to\_send, at or near “logs\_to\_send”

The configuration is not default and explained below:

The Kafka logger was installed to site/custom\_plugins/APACHE\_KAFKA

share/zeek/site/local.zeek uses:

@load custom\_plugins/APACHE\_KAFKA/scripts/Apache/Kafka

lib/zeek/plugins/custom\_plugins is a symlink to share/zeek/site/custom\_plugins

Using the lib symlink seems to be the only way to load the plugin, then the @load statement brings redef customizations and scripts. This works ok for the logger node but not the worker who cannot interface with the plugin ?

Another idea is have non-logger nodes bypass loading logs-to-kafka.bro but this isn’t fully understood.

TIA

/hovsep

---

<div class="post-metadata">

**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\
**Post date:** [April 27, 2020, 10:39am UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/2 "2020-04-27T10:39:29Z")

</div>

I have not run it on 3.1.2 yet but I recommend making your changes to the plugin and running the end to end testing script at [https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run\_end\_to\_end.sh](https://github.com/apache/metron-bro-plugin-kafka/blob/master/docker/run_end_to_end.sh)

It was meant to help isolate issues when making changes to the plugin. Also, we welcome PRs against the project so please feel free to contribute. Thanks,

Jon Zeolla

---

<div class="post-metadata">

**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\
**Post date:** [April 29, 2020, 7:35pm UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/3 "2020-04-29T19:35:01Z")

</div>

Were you able to get this working? I’m planning to work on the bro to zeek cutover for the plugin soon.

---

<div class="post-metadata">

**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\
**Post date:** [May 4, 2020, 5:01pm UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/4 "2020-05-04T17:01:39Z")

</div>

I have the plugin working with 3.1.2 [here](https://github.com/apache/metron-bro-plugin-kafka/pull/44) - feedback is welcome.

---

<div class="post-metadata">

**Author:** ![Hovsep\_Levi](https://avatars.discourse-cdn.com/v4/letter/h/977dab/32.png) [@Hovsep\_Levi](https://community.zeek.org/u/Hovsep_Levi)\
**Post date:** [May 7, 2020, 5:12pm UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/5 "2020-05-07T17:12:51Z")

</div>

It is not working yet for me and was set aside to fix another time.

Very glad to hear about pull 44, I will test !

/Hovsep

---

<div class="post-metadata">

**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\
**Post date:** [May 7, 2020, 6:17pm UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/6 "2020-05-07T18:17:15Z")

</div>

Master officially supports Zeek 3.1 now so please let me know if you run into any issues.

---

<div class="post-metadata">

**Author:** ![Hovsep\_Levi](https://avatars.discourse-cdn.com/v4/letter/h/977dab/32.png) [@Hovsep\_Levi](https://community.zeek.org/u/Hovsep_Levi)\
**Post date:** [May 26, 2020, 10:14am UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/7 "2020-05-26T10:14:39Z")

</div>

I tested this successfully today. Thank you again for your work and assistance.

/hovsep

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\
**Post date:** [May 6, 2022, 3:47pm UTC](https://community.zeek.org/t/zeek-3-1-2-and-kafka-no-data-flow/6103/8 "2022-05-06T15:47:14Z")

</div>