I'm having trouble understanding some log entries from my conn.log. I already learned from this mailing list that bro cannot surely detect who initiated a connection if it does not see the initial connection setup, which seems logical to me.
But if I look to my conn.log file, I find entries like these:
1446190221.687738 Cbu3fj3FYdODxvLF1h 87.152.221.xxx 50993 192.168.100.yyy 36709 tcp ssl 122.745965 1238 5340 S1 F T 0 ShAD
ad 20 2050 19 6112 (empty)
1446190138.746769 CykNrp4VEfzbrJ2vm6 87.152.221.xxx 50993 192.168.100.yyy 36679 tcp ssl 223.406750 1384 18908 S1 F T 0 ShAD ad 39 2956 36 20360 (empty)
It looks like our IMAP server (87.152.221.xxx running on port 50993) initiated a connection to my notebook (192.168.100.yyy). That should not be possible due to lack of port forwarding for this connection.
So my first guess is that bro didn't see the initial connection setup (midstream traffic, OTH state). But I took a look into the documentation
You should really be looking at the history field:
history: string &log &optional
Records the state history of connections as a string of letters. The meaning of those letters is:
Letter Meaning
s a SYN w/o the ACK bit set
h a SYN+ACK (“handshake”)
a a pure ACK
d packet with payload (“data”)
f packet with FIN bit set
r packet with RST bit set
c packet with a bad checksum
i inconsistent packet (e.g. SYN+RST bits both set)
If the event comes from the originator, the letter is in upper-case; if it comes from the responder, it’s in lower-case. Multiple packets of the same type will only be noted once (e.g. we only record one “d” in each direction, regardless of how many data packets were seen.)
So any connection that starts with D or d means bro missed the initial syn handshake (Sh)
Bro is configured to listen to a bridge interface (br0). But I also have running a dumpcap process writing all packets to pcap files. Interestingly, if I feed the corresponding pcap file to bro (bro -r file.pcap), I get source and destination in the right order:
I ran the command you mentioned for my pcap file and checked the conn.log written in my current directory. But this does not seem to change anything, source and destination for the connection I was watching at are still twisted.
we are still having problems with messed up src and dst ip/port of some connections.
What we did up to now:
- Disabled bridging and used a single interface (eth1) for packet capturing, as suggested by Seth Hall
- Disabled offloading by running "ethtool -K eth1 tso off ufo off gso off gro off lro off" in the interface's post-up script
- Changed hardware from a "PC Engines APU 1C4" embedded board to a "Thomas Krenn LESv2" system
- Check the history field in conn.log for "ShA" flags, as suggested by Justin S. Azoff
I found several entries like this in conn.log:
1457953693.259152 CaUcf02Z9xpSSHPz24 10.85.1.1 50993 10.85.1.104 41023 tcp ssl 302.736749 987 7059 SF T T 0 ShADadFfR 23 2167 16 7899 (empty)
According to conn.log, this is a connection from 10.85.1.1 (internal IP of our server) port 50993 (the IMAP-TLS port the servers uses) to 10.85.1.104 (a notebook computer) port 41023, lasting for 302 seconds. My understanding is that bro saw the whole connection from establishment to termination (according to the history field).
In fact, the notebook established a connection to the IMAP service of our server. So src ip and port and dst ip and port are twisted in conn.log.
The problem is reproducible (at least in our environment). I captured a pcap file on the same machine running bro, using the same interface and stripped it down so that it contains only the above mentioned connection - 39 frames and about 15 kB (see attachment). If I inject this pcap file in the network using
tcpreplay -i eth1 twisted.pcapng
bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in conn.log. If instead I read the pcap file using "bro -r", conn.log shows a connection from 10.85.1.104 => 10.85.1.1 (correct!).
Does anyboby have a further idea what we could do to track the problem down?
bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in
conn.log. If instead I read the pcap file using "bro -r", conn.log shows
a connection from 10.85.1.104 => 10.85.1.1 (correct!).
Do both log lines differ only in receiver/originator? If there are
packets missing in your replayed test it is likely that there is an
issue with capturing the traffic.
Do both log lines differ only in receiver/originator? If there are
packets missing in your replayed test it is likely that there is an
issue with capturing the traffic.
You are right, they only differ in receiver/originator.
Thanks for the hint, but I don't think that packet loss is the problem here, because conn.log says that bro saw the initial connection setup (ShA flags in history field). That should be sufficient to tell who's the originator.
I also repeated playback several times with different speeds, the result is reproducible.
In my case, it's not UDP, but TCP. Due to the handshake at the beginning of a TCP connection, initiator and receiver of the connection can be distinguished, and conn.log says that bro saw the initial connection setup (ShA flags in history field).
that sounds very promising! Could you please tell me what setup you
used? (OS/Distribution, bro version, command to playback the pcap file?)
I tried that on Fedora 23 with a recent Bro master. To playback I used:
# tcpreplay -t -i <interface> <pcap-file>
I am using Open vSwitch for my virtual testing network and utilized the
OVS monitoring interface I created in this context for replaying the
traffic.