# send all logs to kafka **URL:** \ **Category:** Zeek\ **Created:** [April 3, 2017, 7:09am UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753 "2017-04-03T07:09:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Youzha](https://avatars.discourse-cdn.com/v4/letter/y/f0a364/32.png) [@Youzha](https://community.zeek.org/u/Youzha)\ **Post date:** [April 3, 2017, 7:09am UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/1 "2017-04-03T07:09:37Z")
hi, i’m trying to using bro kafka plugin to send the bro logs into kafka. i’ve a problem to send all the logs type to kafka. i’ve set this into my local.bro : **@load Bro/Kafka/logs-to-kafka.bro** **redef Kafka::logs\_to\_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES\_LOG, Weird::LOG, Notice::LOG);** but when i check on kafka topic. there are only **http, conn, & dns**. i’ve check in my bro logs dir and there are so many types of log. ![](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/5bc775af876beda568550c03691a2c050cefb0bf.png) is there any config that i missed? pls help. Best Regards, Tukang\_Cangkul ---
**Author:** ![JonZeolla](https://avatars.discourse-cdn.com/v4/letter/j/ccd318/32.png) [@JonZeolla](https://community.zeek.org/u/JonZeolla)\ **Post date:** [April 3, 2017, 1:03pm UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/2 "2017-04-03T13:03:02Z")
Are you sending all of those logs to the same topic? Some of your kafka-related bro configs are missing in the above email, can you send everything? For instance, Kafka::kafka\_conf, Kafka::topic\_name (if used), etc. How are you verifying that they are properly getting onto kafka? I’ve never sent anything other than http, conn, and dns to kafka before, but I feel like that should work. I could be wrong. Jon ![Screenshot from 2017-04-03 14:08:41.png](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/5bc775af876beda568550c03691a2c050cefb0bf.png) ![Screenshot from 2017-04-03 14:08:41.png](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/5bc775af876beda568550c03691a2c050cefb0bf.png) ---
**Author:** ![Youzha](https://avatars.discourse-cdn.com/v4/letter/y/f0a364/32.png) [@Youzha](https://community.zeek.org/u/Youzha)\ **Post date:** [April 3, 2017, 2:15pm UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/3 "2017-04-03T14:15:26Z")
hi Zeolla, yeah i sending all the logs to the same topic (bro topic). maybe i do something wrong about the writing of config **set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES\_LOG, Weird::LOG, Notice::LOG); ?** **maybe there are case sensitive words? or anything else? can you give me some lists of the logs that i can use?** ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [April 3, 2017, 2:29pm UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/4 "2017-04-03T14:29:47Z")
http,dns,conn are all high volume log files compared to known services, weird, and notice. Based on your file sizes it looks like you only had a few notice and known services log entries, so is it possible that you just missed them among the large volume of conn and dns log entries? Also, your weird log looks to be very large, you should do a     cat weird.log |bro-cut name|sort|uniq -c|sort -nr|head -n 10 to see why you have so many weird entries. ---
**Author:** ![Youzha](https://avatars.discourse-cdn.com/v4/letter/y/f0a364/32.png) [@Youzha](https://community.zeek.org/u/Youzha)\ **Post date:** [April 3, 2017, 2:38pm UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/5 "2017-04-03T14:38:10Z")
Hi Azoff, I’ve running bro with that config about 2 days and the picture that i sent before is just a current log dir. This is the result of the command : ```auto cat weird.log |bro-cut name|sort|uniq -c|sort -nr|head -n 10 ``` ![](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/5f8fad9d09fde40f43457e0da3fba9d00c6a0dfb.png) ---
**Author:** ![Youzha](https://avatars.discourse-cdn.com/v4/letter/y/f0a364/32.png) [@Youzha](https://community.zeek.org/u/Youzha)\ **Post date:** [April 3, 2017, 2:49pm UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/6 "2017-04-03T14:49:53Z")
Sorry, i’ve missed to answer your question before . This is all of my config to bro-kafka . **@load Bro/Kafka/logs-to-kafka.bro** **redef Kafka::logs\_to\_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::ServicesInfo, Weird::LOG, Notice::LOG, SSH::LOG, SMTP::LOG, DHCP::LOG);** **redef Kafka::topic\_name = “bro”;** **redef Kafka::tag\_json = T;** **redef Kafka::kafka\_conf = table([“metadata.broker.list”] = “hostname:6667”);** I can verifying that they are getting onto kafka or not by using this command : **bin/kafka-console-consumer.sh --bootstrap-server hostname:6667 --topic bro --from-beginning |grep weird** ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/send-all-logs-to-kafka/4753/7 "2022-05-06T15:44:47Z")