Kafka plugin causes logger to segfault

Zeek
1

All,

I'm currently at my wits end dealing with the Kafka plugin, I'm having great difficulty stopping it from crashing.

When I use the library of librdkafka as prescribed from https://packages.zeek.org/packages/view/7388aa77-4fb7-11e8-88be-0a645a3f3086 (librdkafka-0.11.5), my logger crashes immediately after startup. When using an alternative version of librdkafka (librdkakfa1-0.11.4_confluent4.1.3) the logger doesn't immediately crash but within a minute of starting it usually does.

The stderr.log says the same every time, /run-bro: line 110: <pid> Segmentation fault nohup "$mybro" "$@"

I have downloaded the most recent version of https://github.com/apache/metron-bro-plugin-kafka and still experience this.

I am building an RPM (running CentOS) for the Kafka plugin and installing that way, since the box is offline and unable to reach bro-packages. When I tried to use librdkafka-0.11.5 I've also built an RPM for that.

The following is my only added configuration

@load Apache/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(Conn::LOG);
redef Kafka::kafka_conf = table(
        ["metadata.broker.list"] = "172.16.0.40.9092"
);
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;

The interesting thing to note: the logger does not crash if no logs are being sent (i.e. I comment out the logs_to_send line).

The only other plugins I'm running are Bro::AF_Packet and Corelight::CommunityID.

Anyone have any insight or doing something different?

v/r
Gary

2

Hi,

You don't say what version you're running, but with 2.5 and 2.6 I use these
lines along with the kafka config:

### JSON LOGGING
@load tuning/json-logs
# Set the log separator
redef Log::default_scope_sep = "_";
# Set the time in iso format
redef LogAscii::json_timestamps = JSON::TS_ISO8601;

Your kafka config looks close to mine (I leave the topic_name field blank.)
My kafka emitter has been running on Centos 6, Centos 7 and RHEL7 systems
for about a year.
Can you manually connect to your broker from the zeek box? I have had
issues in the past when the logger was happy but other things in the pipe to
zookeeper and kafka were unhappy.

Pat

3

172.16.0.40.9092 doesn’t appear to be an IP address to me. Did you mean 172.16.0.40:9092?

4

That was a typo when copying over into the email. It's a colon in the actual config.

I'm running bro 2.6.1.

It turns out there was something wrong with the Kafka pipeline, and after we resolved those issues, the logger stopped crashing with the confluent version of librdkafka, but still crashes immediately with the regular version (the version prescribed by zeek packages).

v/r
Gary

5

Are you able to turn debug on[1] and share the details? If you need to bring this off list for sensitivity reasons feel free to contact me directly.

1: https://github.com/apache/metron-bro-plugin-kafka/blob/master/README.md#debug

Jon Zeolla