Duplicate packets in Zeek

Hello. I recently discovered that our Zeek instance was generating duplicate events. This problem was identified by comparing zeek-conn logs and logs captured by our perimeter firewall. In the zeek-conn logs, we observed two distinct log events sharing the same Timestamp, Source IP, Destination IP, Source Port, Destination Port, but with unique connection UIDs. Additionally, in the zeek-http logs, two events with partial field information were generated for the same zeek-conn and Timestamp. For example, the first event captured the HTTP Method, while the second event captured fields such as HTTP Connections Status and HTTP Content Length. Both of these events had a “-” (dash) for the field that the other event was capturing. Similarly, zeek-dns logs were also capturing field values in chunks. For instance, one event would capture query type while other would capture Domain. For each of these duplicated events, we noted a corresponding event in zeek-con logs, exacerbating the EPS spike.

Further investigation led me to a post where a Zeek community user talked about a similar issue, attributing it to misconfigurations in the AF_PACKET plugin’s “fanout mode” setting. This setting dictates the various modes for load balancing packets between a given set of network interfaces (if I understood it correctly). I am currently working to identify the correct configuration that ensures a single log entry for a specific event, rather than two logs with partial information/fields.

Any insights or leads on this matter would be greatly appreciated.

Hi there,

From the symptoms you’re describing, it sounds like you have multiple Zeek workers that are seeing only parts of a given flow. If you were seeing more consistent packet duplication, you’d see different failure modes (for example, most likely largely correct TCP logs, since its reassembler will abstract away much of the duplication). Do your conn logs indicate lots of gaps, or lots of retransmissions?

Regarding AF_PACKET, the biggie to keep in mind is that a fanout group ID establishes a consistent set of packets for each of the workers in that group, so each can meaningfully process the packets. If you monitor multiple interfaces, you need to provide a unique fanout ID for the workers on a given interface.

In the unlikely event that you are running a very old kernel, you can take a look at this helper tool to confirm that your fanouts are working correctly: GitHub - JustinAzoff/can-i-use-afpacket-fanout: Validate if afpacket PACKET_FANOUT_HASH is working properly — the documentation doesn’t spell it out, but you’re looking for a roughly even share of flows across the reported workers.

Best,
Christian