> The original kafka plugin, hosted at [https://github.com/bro/bro-plugins](https://github.com/bro/bro-plugins)
> , is now gone.
D'oh, I now see it is also available in aux/plugins/kafka
> When trying to build from the git tree at [https://github.com/g-clef/](https://github.com/g-clef/)
> KafkaLogger,
> I get the following build error:
>
> [ 33%] Building CXX object CMakeFiles/Kafka-KafkaWriter.linux-
> x86_64.dir/src/AddingJson.cc.o
> /usr/src/KafkaLogger/src/AddingJson.cc:3:20: fatal error: config.h:
> No such file or directory
> compilation terminated.
> CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/build.make:80: recipe
> for target 'CMakeFiles/Kafka-KafkaWriter.linux-x86_64.dir/src/
> AddingJson.cc.o' failed
Perhaps this is useful to Aaron Gee-Clough. I forgot to mention that
I'm using Ubuntu 16.04 running apt-get upgrade periodically.
> I see there's now a Metro fork of the kafka plugin at
>
> [https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka](https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka)
>
> but I am reluctant to try it based on email comments that it is beta.
>
> Any comments/suggestions?
While I can use the version in the bro source, I guess my question still stands:
what's the long-term outlook for kafka support?
-Erich
>
> Any comments/suggestions?
While I can use the version in the bro source, I guess my question still stands:
what's the long-term outlook for kafka support?
For what it’s worth, we use filebeat to shoot our bro logs into Kafka.
To clarify, the Metron project developed the kafka plugin for its own uses and then contributed it into bro-plugins. Recently I worked with the initial creator of the plugin to unify all of the updates that have happened to it over the years (in a way that complies with its LICENSE) here.
I’m in the process of porting it to be a bro package and moving it to https://github.com/apache/metron-bro-plugin-kafka which will be its final resting point. I’m currently battling through some CentOS 6 → 7 upgrades in Metron, and then upgrading bro to 2.5.1 (from 2.4) in Metron (and all of the associated automation/testing), and then finally I will be publishing the kafka plugin module and submitting a PR to https://github.com/bro/packages. Some very, very early movement towards packaging the kafka plugin can be found here (caution, it almost definitely does not work - I’m trying to figure out how to handle the librdkafka dependancy in the package, any feedback would be helpful).
I would /love/ to have this ready to go for brocon (which is my goal).
Jon
For what it’s worth, I’m currently using the plugin available under https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka in my production bro environment, which is an 8 node cluster with > 25,000 events per second and it’s working just fine for me, but I would love to get others to test it. I’m not making any changes to the core kafka plugin itself for the move, just packaging it and incrementing some version numbers - the real heavy lift is within Metron itself, not the bro plugin.
Jon