Bro Logs Sending to Kafka

Hi all,

Apologies if this is not the place to post this question. If it is not, please re-direct me to the correct location.


I’ve added the following code (below – host:port says an actual host and port number) to local.bro in order to send 2 different logs (DNS and CONN) to 3 different Kafka topics.

However, the conn logs being sent to one of the kafka topics is sending messages very slowly (~single digit messages per batch) while the other conn logs are sending to the other kafka topic very quickly (~2 thousand message per batch). Why is this happening?

To follow up, is there an optimal batch size to in order to send thousands of batched message to different Kafka topics. I’ve been able to max out at around 2-3K per batch even if I increase the max ms to 30000.

To run: sudo bro –i lo /path/to/local.bro


@load Bro/Kafka/logs-to-kafka.bro

#redef Kafka::logs_to_send = set(Conn::LOG);

#redef LogAscii::include_meta T;

#redef LogAscii::separator “,”;

redef Kafka::topic_name = “”;

#redef Kafka::tag_json = T;

#redef Kafka::max_wait_on_shutdown = 3000;

redef Kafka::kafka_conf = table(

[“”] = “host:port”, #host:port

[“debug”] = “generic, broker, topic, msg”, #debugging lines

[“queue.buffering.max.messages”] = “5000”, #send in batches of this number

[“”] = “3500” #wait this many ms until sending to topic


#init function to name kafka topic (path), set KAFKAWRITER, and receive only LOG files

event bro_init()


#Conn logs sent straight to

Log::add_filter(Conn::LOG, [

$name = “kafka-conn”,


$path = “conn-logs-batches”


#Conn logs sent to

Log::add_filter(Conn::LOG, [

$name = “kafka-conn-kate”,


$path = “conn-logs-kate”


#DNS logs to DGA

Log::add_filter(DNS::LOG, [

$name = “kafka-dns”,


$path = “dns-logs-dga”



Thank you,