kafka plugin silently fails

Zeek
1

Howdy,

I’m currently using the latest kafka package manager on Bro 3.0.2. The plugin is configured to send to two kafka brokers. Unfortunately, it seems to work for some time and then quietly stops sending to the second broker.

Here’s my zeek config for kafka:

@load packages/metron-bro-plugin-kafka
redef Kafka::topic_name = “”;
redef Kafka::kafka_conf = table(
[“metadata.broker.list”] = “broker1:9092, broker2:9092”
);

event zeek_init()
{
local protocol_list = table(
[“conn” ] = Conn::LOG,
[“dhcp” ] = DHCP::LOG,
[“dns” ] = DNS::LOG,
[“ftp” ] = FTP::LOG,
[“http”] = HTTP::LOG,
[“ssl”] = SSL::LOG,
[“x509”] = X509::LOG
);
for (proto, log_id in protocol_list ) {
local this_filter: Log::Filter = [
$name = “kafka-” + proto,
$writer = Log::WRITER_KAFKAWRITER,
$config = table(
[“metadata.broker.list”] = “broker1:9092, broker2:9092”
),
$path = proto
];
Log::add_filter(log_id, this_filter);
}
}

Does anyone see anything wrong with my config? It works fine for a single broker.

I notice the failure since all the netstat entries disappear for broker2, and it stops receiving data. Broker1 is fine.

Thanks,

-Erich

2

I didn’t take a close look but I believe this is a known bug. Take a look at https://github.com/apache/metron-bro-plugin-kafka/pull/40

I followed up on that PR to see if we can get it merged.