send all logs to kafka

Zeek
1

hi,

i’m trying to using bro kafka plugin to send the bro logs into kafka.
i’ve a problem to send all the logs type to kafka.

i’ve set this into my local.bro :

@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG);

but when i check on kafka topic. there are only http, conn, & dns.
i’ve check in my bro logs dir and there are so many types of log.

is there any config that i missed?
pls help.

Best Regards,

Tukang_Cangkul

2

Are you sending all of those logs to the same topic? Some of your kafka-related bro configs are missing in the above email, can you send everything? For instance, Kafka::kafka_conf, Kafka::topic_name (if used), etc.

How are you verifying that they are properly getting onto kafka? I’ve never sent anything other than http, conn, and dns to kafka before, but I feel like that should work. I could be wrong.

Jon

Screenshot from 2017-04-03 14:08:41.png

Screenshot from 2017-04-03 14:08:41.png

3

hi Zeolla,

yeah i sending all the logs to the same topic (bro topic).

maybe i do something wrong about the writing of config set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::SERVICES_LOG, Weird::LOG, Notice::LOG); ?

maybe there are case sensitive words? or anything else? can you give me some lists of the logs that i can use?

4

http,dns,conn are all high volume log files compared to known services, weird, and notice.

Based on your file sizes it looks like you only had a few notice and known services log entries, so is it possible that you just missed them among the large volume of conn and dns log entries?

Also, your weird log looks to be very large, you should do a

    cat weird.log |bro-cut name|sort|uniq -c|sort -nr|head -n 10

to see why you have so many weird entries.

5

Hi Azoff,

I’ve running bro with that config about 2 days and the picture that i sent before is just a current log dir. This is the result of the command :

cat weird.log |bro-cut  name|sort|uniq  -c|sort -nr|head -n 10

6

Sorry,

i’ve missed to answer your question before .
This is all of my config to bro-kafka .

@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, CONN::LOG, Known::ServicesInfo, Weird::LOG, Notice::LOG, SSH::LOG, SMTP::LOG, DHCP::LOG);
redef Kafka::topic_name = “bro”;
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table([“metadata.broker.list”] = “hostname:6667”);

I can verifying that they are getting onto kafka or not by using this command :

bin/kafka-console-consumer.sh --bootstrap-server hostname:6667 --topic bro --from-beginning |grep weird