Duplicate DNS packets

Zeek
1

Hello fellow Zeekers,

I am new to the mailing list and fairly new to Zeek.
I am having an issue where DNS traffic is duplicated. It seem fairly obvious to me that the issue is that the manager is sending a single “session” to all of the workers defined in node.cfg.

Example duplicate logs (sanitized a bit):

user1@site1bro:~$ awk -F ‘\t’ ‘{ if($1 == “1558556089.463824”) print $0;}’ dns.date-time.log
1558556089.463824 Ce6WGH1tX7fUQCJkEb 10.1.1.1 49675 10.5.5.5 53 udp 58613 - yahoo.uservoice.com 1 C_INTERNET 1 A - - F F T F 0 SITE1BRO-4
1558556089.463824 CxhWh33b65uCcQlUR2 10.1.1.1 49675 10.5.5.5 53 udp 58613 - yahoo.uservoice.com 1 C_INTERNET 1 A - - F F T F 0 SITE1BRO-8
1558556089.463824 CNBy3ykdFSvXydiW7 10.1.1.1 49675 10.5.5.5 53 udp 58613 - yahoo.uservoice.com 1 C_INTERNET 1 A - - F F T F 0 SITE1BRO-9
1558556089.463824 CV6w2f3NKeaAwhAvJf 10.1.1.1 49675 10.5.5.5 53 udp 58613 - yahoo.uservoice.com 1 C_INTERNET 1 A - - F F T F 0 SITE1BRO-7
1558556089.463824 Cc5rcP3N92OGHYUKA2 10.1.1.1 49675 10.5.5.5 53 udp 58613 - yahoo.uservoice.com 1 C_INTERNET 1 A - - F F T F 0 SITE1BRO-6

My node.cfg file:

[manager]

type=manager

host=10.10.10.10

[proxy-1]

type=proxy

host=10.10.10.10

[SITE1BRO]

type=worker

host=10.10.10.10

interface=eth5

lb_method=pf_ring

lb_procs=10

pin_cpus=2,3,4,5,6,7,8,9,10,11

Other info:

  • The span feed is clean of duplicates (validated with multiple packet captures)

  • Other logs are generally not duplicated, and I suspect that this only happens with UDP traffic

  • I’ve tried changing the LB type in the broctl.cfg file to 2-tuple, 5-tuple, and round-robin (4-tuple is default) but none of those resolved the issue

  • I’ve tried installing the latest dev version of pf_ring to no avail

  • From previously archived threads, it appears that this is not a new issue, and that it also happens with af_packet … which is what I was going to try next :frowning:

Any insights as to how I can fix, or at least filter these duplicates before they are written to file and/or sent to Kafka would be greatly appreciated.

KCL

2

Hello fellow Zeekers,

I am new to the mailing list and fairly new to Zeek.
I am having an issue where DNS traffic is duplicated. It seem fairly obvious to me that the issue is that the manager is sending a single “session” to all of the workers defined in node.cfg.

not quite, the manager doesn’t send any traffic, the workers read it directly, but you are correct in that all of the workers are seeing the same traffic

Other info:

  • The span feed is clean of duplicates (validated with multiple packet captures)

  • Other logs are generally not duplicated, and I suspect that this only happens with UDP traffic

  • I’ve tried changing the LB type in the broctl.cfg file to 2-tuple, 5-tuple, and round-robin (4-tuple is default) but none of those resolved the issue

  • I’ve tried installing the latest dev version of pf_ring to no avail

  • From previously archived threads, it appears that this is not a new issue, and that it also happens with af_packet … which is what I was going to try next :frowning:

Your problem is that you are not actually using pf_ring to load balance, you’re just running 10 workers all seeing 100% of the traffic. This isn’t really an issue it’s just a common misconfiguration.

The easiest way to fix this is to install https://packages.bro.org/packages/view/1bafeed3-c141-11e8-88be-0a645a3f3086 And not try to use the PF ring libpcap which is where your problem is (It may be installed but you’re not actually using it)

Using af_packet https://packages.bro.org/packages/view/74610004-4fb7-11e8-88be-0a645a3f3086 It’s probably easier anyway and that does not have this problem

3

Justin,

Thanks for taking the time to reply and thanks for the information. I’ll work on this next week and reply to the list.

Kurtis Lawson

4

Just a follow up.

The AF_Ring plugin was a quick and easy solution for my duplication problem. No more duplicates and performance is good, even at sensors with multi-gigabit traffic. Thanks for your help Justin!

KCL