# is there a bro script to ignore duplicated logs? **URL:** \ **Category:** Zeek\ **Created:** [October 4, 2018, 12:57pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485 "2018-10-04T12:57:14Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![fatema\_bannatwala](https://avatars.discourse-cdn.com/v4/letter/f/5f9b8f/32.png) [@fatema\_bannatwala](https://community.zeek.org/u/fatema_bannatwala)\ **Post date:** [October 4, 2018, 12:57pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/1 "2018-10-04T12:57:14Z")
How many workers do you have in your cluster? Many months ago, we had split-ed connections issue, where Seth provided a script to add the worker node to conn.log to see where exactly packets are being processed,i.e. which nodes. You can run this script and see if the duplicate connections are happening on which workers and go from there: $ cat add-node-to-conn.bro ##! Add the name of the current node to conn.log @load base/protocols/conn export { redef record Conn::Info += { ## The name of the node where this connection was analyzed. node: string &log &optional; }; } event connection\_state\_remove(c: connection) &priority=2 { c$conn$node = peer\_description; } ---
**Author:** ![Michal\_Purzynski1](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@Michal\_Purzynski1](https://community.zeek.org/u/Michal_Purzynski1)\ **Post date:** [October 4, 2018, 1:26pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/2 "2018-10-04T13:26:03Z")
Most likely you have separate workers parsing the same traffic. Can you load the script fatema told you about and also a capture loss script and report results for the hour or so? If each worker sees the same duplicate traffic then the amount of packets processed will be very similar. ---
**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\ **Post date:** [October 4, 2018, 1:33pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/3 "2018-10-04T13:33:00Z")
This can be further automated by using Justins bro-doctor script available as a package: [https://github.com/ncsa/bro-doctor](https://github.com/ncsa/bro-doctor) Jan ---
**Author:** ![ericooi](https://avatars.discourse-cdn.com/v4/letter/e/ecae2f/32.png) [@ericooi](https://community.zeek.org/u/ericooi)\ **Post date:** [October 4, 2018, 1:42pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/4 "2018-10-04T13:42:12Z")
We had a similar issue and it turned out that the SPAN the network engineer configured was capturing a trunk such that any inter-VLAN traffic was being analyzed multiple times. ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [October 4, 2018, 2:21pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/5 "2018-10-04T14:21:55Z")
Yeah, if the traffic was duplicated 2 (maybe 3) times this could be the problem. In this case it's duplicated 12 times which almost definitely points to an lb\_procs=12 in node.cfg and load balancing not working properly. ---
**Author:** ![MAAN\_ABU\_SHAQRA](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@MAAN\_ABU\_SHAQRA](https://community.zeek.org/u/MAAN_ABU_SHAQRA)\ **Post date:** [October 18, 2018, 7:07am UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/6 "2018-10-18T07:07:38Z")
Hi All, [@fatema bannatwala](mailto:fatema.bannatwala@gmail.com) im having 2 different interfaces each with 6 workers using af\_packet as a load balancer, ive tried the script provided and it showed that both interfaces are streaming same dns/http logs. so i disabled one interface and reduced workers to 3 and kept monitoring one interface only. on conn.log im getting below: 1536746523.249570 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52851 10.190.129.250 53 udp dns 11.342418 264 88 SF T T 0 Dd 6 432 2 144 (empty) worker-em1-1 on dns.log 1536746526.252543 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746526.252634 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283534 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52851 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283625 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746526.252543 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746526.252634 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283534 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283625 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52852 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283625 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283625 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F 1536746530.283625 **CbQGeN3yuYTKQd6xE** 10.1.196.178 52853 10.190.129.250 53 udp 59985 - 100.2.116.192.in-addr.arpa 1 C\_INTERNET 12 PTR - - F ive also checked the [http://try.bro.org/](http://try.bro.org/#/?example=basics-switches-solution-switch-exercise) and ran the exercies , and found the same issue. > [Try Bro](http://try.bro.org/#/?example=basics-switches-solution-switch-exercise) > [try.bro.org](http://try.bro.org) > Hello World. Welcome to our interactive Bro tutorial. Click run and see the Bro magic happen. You may need to scroll down a bit to get to the output. > | > > - | the problem is im using the apapche kafka plugin with apache metron and seeing huge amount of duplicate DNS events and UIDs. i have tried to filter out duplicates in the local.bro using below script. ```auto Log::add_filter(DNS::LOG, [ $name = "kafka-dns", $writer = Log::WRITER_KAFKAWRITER, $pred(rec: DNS::Info) = { return ! (( |rec$uid| == |rec$uid| )); }, $config = table( ["metadata.broker.list"] = "localhost:9092" ) ]); ``` but got nothing because all dns entries are duplicated. can anyone help me with the syntax of the above code. thanks ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [October 18, 2018, 1:54pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/7 "2018-10-18T13:54:36Z")
Can you post your node.cfg? It looks like you may have told bro to load balance, but you are not actually using the af\_packet plugin. as in, you have interface=em1 lb\_method=custom lb\_procs=6 instead of interface=af\_packet::em1 lb\_method=custom lb\_procs=6 ---
**Author:** ![MAAN\_ABU\_SHAQRA](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@MAAN\_ABU\_SHAQRA](https://community.zeek.org/u/MAAN_ABU_SHAQRA)\ **Post date:** [October 18, 2018, 1:58pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/8 "2018-10-18T13:58:20Z")
Hi Justin, It is actually set as af\_packet::em1 MA’AN ABUSHAQRA Dubai, UAE +971501201752 ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [October 18, 2018, 2:00pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/9 "2018-10-18T14:00:24Z")
> Hi Justin, > > It is actually set as af\_packet::em1 Oh ☹ What did you mean by > ive also checked the [http://try.bro.org/](http://try.bro.org/) and ran the exercies , and found the same issue. Do you mean you uploaded a pcap file and that showed the same duplicated logs? If so, how did you obtain that pcap? ---
**Author:** ![MAAN\_ABU\_SHAQRA](https://avatars.discourse-cdn.com/v4/letter/m/f9ae1b/32.png) [@MAAN\_ABU\_SHAQRA](https://community.zeek.org/u/MAAN_ABU_SHAQRA)\ **Post date:** [October 18, 2018, 2:05pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/10 "2018-10-18T14:05:31Z")
No I didn’t upload a pcap, the provided pcaps on the website show duplicate dns UIDs. I suspect that it’s a duplicated packets issue as I’ve analyzed some traffic on wireshark and it had no duplicates. I’d appreciate it if anyone can assist with this, Thanks MA’AN ABUSHAQRA Dubai, UAE ---
**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\ **Post date:** [October 18, 2018, 2:29pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/11 "2018-10-18T14:29:59Z")
Oh, yes, I see what you are saying now. The repeated entries i see on the pcap on [try.bro.org](http://try.bro.org) are from netbios queries that are all broadcast queries for WORKGROUP. I believe those are actually just repeated broadcasts with the same 5 tuple which is what causes the duplicates. At this point with your traffic I would stop bro and run a simple tcpdump to generate a pcap file while you generate some known traffic (like a small file download over HTTP) and then inspect the resulting pcap file. You could also set lb\_procs=1 to ensure that there is only one bro process running to rule out any issue with af\_packet load balancing. One weird thing I see is that for your conn.log entry: 1536746523.249570 CbQGeN3yuYTKQd6xE 10.1.196.178 52851 10.190.129.250 53 udp dns 11.342418 264 88 SF T T 0 Dd 6 432 2 144 (empty) worker-em1-1 The 6 and 2 are orig\_pkts and resp\_pkts. For a simple DNS lookup, you would expect orig\_pkts and resp\_pkts to both be one.. one packet for the query and then one packet for the response. But your bro worker somehow saw 6 packets for the query and 2 packets for the response. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:46pm UTC](https://community.zeek.org/t/is-there-a-bro-script-to-ignore-duplicated-logs/5485/12 "2022-05-06T15:46:07Z")