# which kafka plugin to use? **URL:** \ **Category:** Zeek\ **Created:** [August 15, 2017, 3:48pm UTC](https://community.zeek.org/t/which-kafka-plugin-to-use/4989 "2017-08-15T15:48:53Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Erich\_M\_Nahum](https://avatars.discourse-cdn.com/v4/letter/e/958977/32.png) [@Erich\_M\_Nahum](https://community.zeek.org/u/Erich_M_Nahum)\ **Post date:** [August 15, 2017, 3:48pm UTC](https://community.zeek.org/t/which-kafka-plugin-to-use/4989/1 "2017-08-15T15:48:53Z")
`> The original kafka plugin, hosted at [https://github.com/bro/bro-plugins](https://github.com/bro/bro-plugins)` `> , is now gone.` `D'oh, I now see it is also available in aux/plugins/kafka` `> When trying to build from the git tree at [https://github.com/g-clef/](https://github.com/g-clef/)` `> KafkaLogger,` `> I get the following build error:` `>` `> [33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux-` `> x86_64.dir/src/AddingJson.cc.o` `> /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h:` `> No such file or directory` `> compilation terminated.` `> CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe` `> for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/` `> AddingJson.cc.o' failed` `Perhaps this is useful to Aaron Gee-Clough. I forgot to mention that` `I'm using Ubuntu 16.04 running apt-get upgrade periodically.` `> I see there's now a Metro fork of the kafka plugin at` `>` `> [https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka](https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka)` `>` `> but I am reluctant to try it based on email comments that it is beta.` `>` `> Any comments/suggestions?` `While I can use the version in the bro source, I guess my question still stands:` `what's the long-term outlook for kafka support?` `-Erich` ---
**Author:** ![William\_Arbaugh](https://avatars.discourse-cdn.com/v4/letter/w/cdc98d/32.png) [@William\_Arbaugh](https://community.zeek.org/u/William_Arbaugh)\ **Post date:** [August 15, 2017, 4:08pm UTC](https://community.zeek.org/t/which-kafka-plugin-to-use/4989/2 "2017-08-15T16:08:20Z")
> `>` > `> Any comments/suggestions?` > > `While I can use the version in the bro source, I guess my question still stands:` > `what's the long-term outlook for kafka support?` For what it’s worth, we use filebeat to shoot our bro logs into Kafka. ---
**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\ **Post date:** [August 15, 2017, 4:25pm UTC](https://community.zeek.org/t/which-kafka-plugin-to-use/4989/3 "2017-08-15T16:25:28Z")
To clarify, the Metron project developed the kafka plugin for its own uses and then contributed it into bro-plugins. Recently I worked with the initial creator of the plugin to unify all of the updates that have happened to it over the years (in a way that complies with its LICENSE) [here](https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka). I’m in the process of porting it to be a bro package and moving it to [https://github.com/apache/metron-bro-plugin-kafka](https://github.com/apache/metron-bro-plugin-kafka) which will be its final resting point. I’m currently battling through some CentOS 6 → 7 upgrades in Metron, and then upgrading bro to 2.5.1 (from 2.4) in Metron (and all of the associated automation/testing), and then finally I will be publishing the kafka plugin module and submitting a PR to [https://github.com/bro/packages](https://github.com/bro/packages). Some very, **very** early movement towards packaging the kafka plugin can be found [here](https://github.com/JonZeolla/metron-bro-plugin-kafka) (caution, it almost definitely does not work - I’m trying to figure out how to handle the librdkafka dependancy in the package, any feedback would be helpful). I would /love/ to have this ready to go for brocon (which is my goal). Jon ---
**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\ **Post date:** [August 15, 2017, 4:52pm UTC](https://community.zeek.org/t/which-kafka-plugin-to-use/4989/4 "2017-08-15T16:52:29Z")
For what it’s worth, I’m currently using the plugin available under [https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka](https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka) in my production bro environment, which is an 8 node cluster with \> 25,000 events per second and it’s working just fine for me, but I would love to get others to test it. I’m not making any changes to the core kafka plugin itself for the move, just packaging it and incrementing some version numbers - the real heavy lift is within Metron itself, not the bro plugin. Jon ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:45pm UTC](https://community.zeek.org/t/which-kafka-plugin-to-use/4989/5 "2022-05-06T15:45:12Z")