Hi all,
Apologies if this is not the place to post this question. If it is not, please re-direct me to the correct location.
Question:
I’ve added the following code (below – host:port says an actual host and port number) to local.bro in order to send 2 different logs (DNS and CONN) to 3 different Kafka topics.
However, the conn logs being sent to one of the kafka topics is sending messages very slowly (~single digit messages per batch) while the other conn logs are sending to the other kafka topic very quickly (~2 thousand message per batch). Why is this happening?
To follow up, is there an optimal batch size to max.ms in order to send thousands of batched message to different Kafka topics. I’ve been able to max out at around 2-3K per batch even if I increase the max ms to 30000.
To run: sudo bro –i lo /path/to/local.bro
Code:
@load Bro/Kafka/logs-to-kafka.bro
#redef Kafka::logs_to_send = set(Conn::LOG);
#redef LogAscii::include_meta T;
#redef LogAscii::separator “,”;
redef Kafka::topic_name = “”;
#redef Kafka::tag_json = T;
#redef Kafka::max_wait_on_shutdown = 3000;
redef Kafka::kafka_conf = table(
[“metadata.broker.list”] = “host:port”, #host:port
[“debug”] = “generic, broker, topic, msg”, #debugging lines
[“queue.buffering.max.messages”] = “5000”, #send in batches of this number
[“queue.buffering.max.ms”] = “3500” #wait this many ms until sending to topic
);
#init function to name kafka topic (path), set KAFKAWRITER, and receive only LOG files
event bro_init()
{
#Conn logs sent straight to
Log::add_filter(Conn::LOG, [
$name = “kafka-conn”,
$writer = Log::WRITER_KAFKAWRITER,
$path = “conn-logs-batches”
]);
#Conn logs sent to
Log::add_filter(Conn::LOG, [
$name = “kafka-conn-kate”,
$writer = Log::WRITER_KAFKAWRITER,
$path = “conn-logs-kate”
]);
#DNS logs to DGA
Log::add_filter(DNS::LOG, [
$name = “kafka-dns”,
$writer = Log::WRITER_KAFKAWRITER,
$path = “dns-logs-dga”
]);
}
Thank you,
David