is there a bro script to ignore duplicated logs?

Zeek
1

How many workers do you have in your cluster?
Many months ago, we had split-ed connections issue, where Seth provided a script to add the worker node to conn.log to see where exactly packets are being processed,i.e. which nodes.

You can run this script and see if the duplicate connections are happening on which workers and go from there:

$ cat add-node-to-conn.bro

##! Add the name of the current node to conn.log

@load base/protocols/conn

export {
redef record Conn::Info += {

The name of the node where this connection was analyzed.

node: string &log &optional;
};
}

event connection_state_remove(c: connection) &priority=2
{
c$conn$node = peer_description;
}

2

Most likely you have separate workers parsing the same traffic.

Can you load the script fatema told you about and also a capture loss script and report results for the hour or so?

If each worker sees the same duplicate traffic then the amount of packets processed will be very similar.

3

This can be further automated by using Justins bro-doctor script available as a package: https://github.com/ncsa/bro-doctor

Jan

4

We had a similar issue and it turned out that the SPAN the network engineer configured was capturing a trunk such that any inter-VLAN traffic was being analyzed multiple times.

5

Yeah, if the traffic was duplicated 2 (maybe 3) times this could be the problem. In this case it's duplicated 12 times which almost definitely points to an lb_procs=12 in node.cfg and load balancing not working properly.

6

Hi All,

@fatema bannatwala im having 2 different interfaces each with 6 workers using af_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3 and kept monitoring one interface only. on conn.log im getting below:

1536746523.249570 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp dns 11.342418 264 88 SF T T 0 Dd 6 432 2 144 (empty) worker-em1-1

on dns.log

1536746526.252543 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252634 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283534 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252543 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746526.252634 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283534 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

1536746530.283625 CbQGeN3yuYTKQd6xE 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C_INTERNET 12 PTR - - F

ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.



Try Bro
try.bro.org
Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output.
|

  • |

the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script.

 Log::add_filter(DNS::LOG, [
        $name = "kafka-dns",
        $writer = Log::WRITER_KAFKAWRITER,
        $pred(rec: DNS::Info) = { return ! (( |rec$uid| == |rec$uid| )); },
        $config = table(
            ["metadata.broker.list"] = "localhost:9092"
        )
    ]);

but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code.

thanks

7

Can you post your node.cfg? It looks like you may have told bro to load balance, but you are not actually using the af_packet plugin.

as in, you have

interface=em1
lb_method=custom
lb_procs=6

instead of

interface=af_packet::em1
lb_method=custom
lb_procs=6

8

Hi Justin,

It is actually set as af_packet::em1

MA’AN ABUSHAQRA
Dubai, UAE
+971501201752

9

Hi Justin,

It is actually set as af_packet::em1

Oh :frowning:

What did you mean by

ive also checked the http://try.bro.org/ and ran the exercies , and found the same issue.

Do you mean you uploaded a pcap file and that showed the same duplicated logs?
If so, how did you obtain that pcap?

10

No I didn’t upload a pcap, the provided pcaps on the website show duplicate dns UIDs. I suspect that it’s a duplicated packets issue as I’ve analyzed some traffic on wireshark and it had no duplicates.

I’d appreciate it if anyone can assist with this,

Thanks

MA’AN ABUSHAQRA
Dubai, UAE

11

Oh, yes, I see what you are saying now.

The repeated entries i see on the pcap on try.bro.org are from netbios queries that are all broadcast queries for WORKGROUP. I believe those are actually just repeated broadcasts with the same 5 tuple which is what causes the duplicates.

At this point with your traffic I would stop bro and run a simple tcpdump to generate a pcap file while you generate some known traffic (like a small file download over HTTP) and then inspect the resulting pcap file.

You could also set lb_procs=1 to ensure that there is only one bro process running to rule out any issue with af_packet load balancing.

One weird thing I see is that for your conn.log entry:

1536746523.249570
CbQGeN3yuYTKQd6xE
10.1.196.178 52851
10.190.129.250 53
udp dns
11.342418 264
88 SF
T T
0 Dd
6 432
2 144
(empty) worker-em1-1

The 6 and 2 are orig_pkts and resp_pkts. For a simple DNS lookup, you would expect orig_pkts and resp_pkts to both be one.. one packet for the query and then one packet for the response. But your bro worker somehow saw 6 packets for the query and 2 packets for the response.