Problem with connections in S1 and SF state

Dear list,

I'm having trouble understanding some log entries from my conn.log. I already learned from this mailing list that bro cannot surely detect who initiated a connection if it does not see the initial connection setup, which seems logical to me.

But if I look to my conn.log file, I find entries like these:

1446190221.687738 Cbu3fj3FYdODxvLF1h 87.152.221.xxx 50993 192.168.100.yyy 36709 tcp ssl 122.745965 1238 5340 S1 F T 0 ShAD
ad 20 2050 19 6112 (empty)
1446190138.746769 CykNrp4VEfzbrJ2vm6 87.152.221.xxx 50993 192.168.100.yyy 36679 tcp ssl 223.406750 1384 18908 S1 F T 0 ShAD ad 39 2956 36 20360 (empty)

It looks like our IMAP server (87.152.221.xxx running on port 50993) initiated a connection to my notebook (192.168.100.yyy). That should not be possible due to lack of port forwarding for this connection.

So my first guess is that bro didn't see the initial connection setup (midstream traffic, OTH state). But I took a look into the documentation

You should really be looking at the history field:

history: string &log &optional
Records the state history of connections as a string of letters. The meaning of those letters is:

Letter Meaning
s a SYN w/o the ACK bit set
h a SYN+ACK (“handshake”)
a a pure ACK
d packet with payload (“data”)
f packet with FIN bit set
r packet with RST bit set
c packet with a bad checksum
i inconsistent packet (e.g. SYN+RST bits both set)
If the event comes from the originator, the letter is in upper-case; if it comes from the responder, it’s in lower-case. Multiple packets of the same type will only be noted once (e.g. we only record one “d” in each direction, regardless of how many data packets were seen.)

So any connection that starts with D or d means bro missed the initial syn handshake (Sh)

Justin, thanks for the hint. I should indeed have checked the history field.

But even for connections that do not start with d or D in the history field, I see the same behaviour. Source and destination is still twisted:

1447675087.121817 CjRCD61gNErucciPb8 87.144.16.xxx 50993 192.168.100.yyy 26577 tcp ssl 83.596659 1432 2619 S1 F T 0 ShADad 18 2164 15 3231 (empty)

Bro is configured to listen to a bridge interface (br0). But I also have running a dumpcap process writing all packets to pcap files. Interestingly, if I feed the corresponding pcap file to bro (bro -r file.pcap), I get source and destination in the right order:

1447675087.121817 C2AvJf3WgcdiBlYfS4 192.168.100.yyy 26577 87.144.16.xxx 50993 tcp ssl 83.596659 1432 2619 S1 - - 0 ShADad 18 2164 15 3231 (empty)

Does anybody have an explanation for this?

Thanks,
Sven

Sven,

Try running the pcap through your local policy scripts and check the output:

bro -r file.pcap local

I don’t otherwise have a specific clue why this could happen, but it’s best to compare the same process.

-Derek

Derek,

thank you very much for your reply.

I ran the command you mentioned for my pcap file and checked the conn.log written in my current directory. But this does not seem to change anything, source and destination for the connection I was watching at are still twisted.

Best regards, Sven

Dear list,

we are still having problems with messed up src and dst ip/port of some connections.

What we did up to now:
- Disabled bridging and used a single interface (eth1) for packet capturing, as suggested by Seth Hall
- Disabled offloading by running "ethtool -K eth1 tso off ufo off gso off gro off lro off" in the interface's post-up script
- Changed hardware from a "PC Engines APU 1C4" embedded board to a "Thomas Krenn LESv2" system
- Check the history field in conn.log for "ShA" flags, as suggested by Justin S. Azoff

I found several entries like this in conn.log:
1457953693.259152 CaUcf02Z9xpSSHPz24 10.85.1.1 50993 10.85.1.104 41023 tcp ssl 302.736749 987 7059 SF T T 0 ShADadFfR 23 2167 16 7899 (empty)

According to conn.log, this is a connection from 10.85.1.1 (internal IP of our server) port 50993 (the IMAP-TLS port the servers uses) to 10.85.1.104 (a notebook computer) port 41023, lasting for 302 seconds. My understanding is that bro saw the whole connection from establishment to termination (according to the history field).

In fact, the notebook established a connection to the IMAP service of our server. So src ip and port and dst ip and port are twisted in conn.log.

The problem is reproducible (at least in our environment). I captured a pcap file on the same machine running bro, using the same interface and stripped it down so that it contains only the above mentioned connection - 39 frames and about 15 kB (see attachment). If I inject this pcap file in the network using

tcpreplay -i eth1 twisted.pcapng

bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in conn.log. If instead I read the pcap file using "bro -r", conn.log shows a connection from 10.85.1.104 => 10.85.1.1 (correct!).

Does anyboby have a further idea what we could do to track the problem down?

Thanks and best regards, Sven

twisted.pcapng (14.5 KB)

Hi Sven,

bro shows a connection from 10.85.1.1 => 10.85.1.104 (wrong!) in
conn.log. If instead I read the pcap file using "bro -r", conn.log shows
a connection from 10.85.1.104 => 10.85.1.1 (correct!).

Do both log lines differ only in receiver/originator? If there are
packets missing in your replayed test it is likely that there is an
issue with capturing the traffic.

Best regards,
Jan

Read this thread:

http://thread.gmane.org/gmane.comp.security.detection.bro/9211

It might help.

James

Hi Jan,

Do both log lines differ only in receiver/originator? If there are
packets missing in your replayed test it is likely that there is an
issue with capturing the traffic.

You are right, they only differ in receiver/originator.

Thanks for the hint, but I don't think that packet loss is the problem here, because conn.log says that bro saw the initial connection setup (ShA flags in history field). That should be sufficient to tell who's the originator.

I also repeated playback several times with different speeds, the result is reproducible.

Thanks,
Sven

Thanks James.

In my case, it's not UDP, but TCP. Due to the handshake at the beginning of a TCP connection, initiator and receiver of the connection can be distinguished, and conn.log says that bro saw the initial connection setup (ShA flags in history field).

Thanks,
Sven

Hi Sven,

I also repeated playback several times with different speeds, the result
is reproducible.

I ran Bro on your pcap and replayed it into my test environment. I could
not reproduce the receiver/originator issue you described.

Best regards,
Jan

Hi Jan,

that sounds very promising! Could you please tell me what setup you used? (OS/Distribution, bro version, command to playback the pcap file?)

Thanks!
Sven

Hi Sven,

that sounds very promising! Could you please tell me what setup you
used? (OS/Distribution, bro version, command to playback the pcap file?)

I tried that on Fedora 23 with a recent Bro master. To playback I used:
# tcpreplay -t -i <interface> <pcap-file>
I am using Open vSwitch for my virtual testing network and utilized the
OVS monitoring interface I created in this context for replaying the
traffic.

Best regards,
Jan

Thanks Jan, I'll give that setup a try and will report back.

Best regards,
Sven