I'm currently at my wits end dealing with the Kafka plugin, I'm having great difficulty stopping it from crashing.
When I use the library of librdkafka as prescribed from https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086 (librdkafka-0.11.5), my logger crashes immediately after startup. When using an alternative version of librdkafka (librdkakfa1-0.11.4_confluent4.1.3) the logger doesn't immediately crash but within a minute of starting it usually does.
The stderr.log says the same every time, /run-bro: line 110: <pid> Segmentation fault nohup "$mybro" "$@"
I have downloaded the most recent version of https://github.com/apache/metron-bro-plugin-kafka and still experience this.
I am building an RPM (running CentOS) for the Kafka plugin and installing that way, since the box is offline and unable to reach bro-packages. When I tried to use librdkafka-0.11.5 I've also built an RPM for that.
The following is my only added configuration
redef Kafka::logs_to_send = set(Conn::LOG);
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "172.16.0.40.9092"
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
The interesting thing to note: the logger does not crash if no logs are being sent (i.e. I comment out the logs_to_send line).
The only other plugins I'm running are Bro::AF_Packet and Corelight::CommunityID.
Anyone have any insight or doing something different?