how to merge rx and tx from different pcaps / slightly off-topic

Zeek
1

Hi!

Sorry if this is off-topic, but I hope to find the right audience here.

I want to create bro-logs of around 900 Gb of data in 20.000 pcaps.
Capturing was done on different interfaces for upstream and downstream (rx/tx).

Because of the large number of files I cannot merge them in one step (“to many open files”),
so I merged them to one pcap per day with mergecap. After that Bro is called like this:

mergecap -F pcap -w - *.pcap | bro -r - foo.bro

foo.bro reads:

redef bits_per_uids = 128;
redef ignore_checksums = T;
redef Log::default_rotation_interval = 1day;

No real service logs are written, except for a weird.log full of:

connection_originator_SYN_ack
data_after_reset
data_before_established
inappropriate_FIN
possible_split_routing
simultaneous_open
SYN_after_close
SYN_after_reset
SYN_inside_connection
SYN_seq_jump
TCP_ack_underflow_or_misorder
TCP_seq_underflow_or_misorder
window_recision

It looks like Bro not seeing the data in the correct order. But from what I read in mergecap
source in merge_read_packet() this should work as intended: “Read the next packet,
in chronological order, from the set of files to be merged.”

I am thankful for any ideas.

Bye,

Franky

2

It looks like Bro not seeing the data in the correct order. But from what I
read in mergecap source in merge_read_packet() this should work as intended:
"Read the next packet, in chronological order, from the set of files to be
merged."

You could give this a shot:

    ipsumdump --collate -r *.pcap -w merged.pcap

Unlike mergecap, ipsumdump does not assume packets are sorted within the
trace.

    Matthias

3

I ran into some problems trying to process pcaps. One is the checksums issue but I see you’ve already handled that. The other seems like it might possibly be related:

If you don’t specify --pseudo-realtime, BRO will apparently run connection timers based on the current wall clock time, comparing the wall clock with the start time recorded in conjunction with the packets in the pcap. This means it may see a connection start, then immediately expire it as having passed the session time limit. [What? That session is six months old!]

(This seems fundamentally broken to me, but it’s also quite likely that I didn’t fully understand the code and/or that there’s some good reason for it to work this way; in any case, the --pseudo-realtime switch seems to make it behave more sanely – for this particular case anyway.)

Cheers.

4

That’s actually not how Bro works, it uses the timestamps in the packets to drive it’s packet clock forward. Could you show how you’re running Bro? It sounds to me like you’re replaying traffic to and interface and then sniffing it.

.Seth

5

Hi,

6

Seth, Thanks for the clarification.

Uggh… It appears that shady stuff my plugin is doing is responsible for my problem.

I think the problem is that I have opened a live pkt src from within my plugin, but then also trying to read a pcap. Maybe I’ve seeded BRO with a later timestamp than those in the pcap? Having a hard time following the timer logic.

Is it possible to instantiate a per-PktSrc timer?

Anyway, sorry to be spewing misinformation.

7

Uggh... It appears that shady stuff my plugin is doing is responsible for my problem.

Is your plugin posted anywhere?

I think the problem is that I have opened a live pkt src from within my plugin, but then also trying to read a pcap. Maybe I've seeded BRO with a later timestamp than those in the pcap? Having a hard time following the timer logic.

You’re doing both in your plugin? That definitely isn’t a supported model.

Is it possible to instantiate a per-PktSrc timer?

I assume you mean a per-pktsrc clock? (since timers have a meaning and are something different in Bro). If you meant clock, then no, a Bro process has the notion of a singular clock.

  .Seth