I am running Bro version 2.5.2, and encountering an issue where an established MySQL session after idling for some time (approximately 10 minutes), the MySQL analyzer events are no longer generated.
The MySQL session itself is still alive and still responding to queries, but the analyzer event mysql_command_request stop generating. I’ve also check packet capture and seen no obvious oddities and the packets are the same as when it was working. I am only issuing a ‘Use DB’ command for testing.
I suspect there is a timeout somewhere or a limit somewhere that I’m not aware of, as I am still fairly new to Bro itself.
Does anyone have any ideas on how I can continue to debug this issue?
I tried the same scenario with Bro 2.5.5 and the same problem with MySQL long session is still reproducible. I am attaching the pcap here which should show the same issue.
It is a straight forward MySQL session using mysql command. There is a long period of idling between packet 23 and packet 24, and no mysql events can be generated after packet 24.
The last working command with MySQL event generated is packet 21.
Any suggestions on how to debug this issue would be greatly appreciated. I am not very familiar with how Bro passes received data into the analyzers for the message parsing, but I would like to find out whether it is that the parsing is failing or if Bro is not picking up the data at all.
Thanks in advance for any guidance anyone can provide.
There were a couple potential issues that's working around when I
tested with Bro's git/master branch.
The first was that the responses may be malformed -- Wireshark appears
to report that it's malformed MySQL and Bro also indicates a parsing
problem in dpd.log, showing that the LengthEncodedIntegerLookahead
field was expecting to read 8 bytes, but only 6 were available/sent.
The default behavior is for Bro to disable analyzers that encounter
such protocol violations, but the DPD::ignore_violations customization
prevents that. Someone may ultimately want to check into whether the
packet is legit following the MySQL protocol, because then the parser
in Bro should be updated to understand it better.
The second issue was related to the long idle period that you
mentioned. The default timeout for TCP sessions in Bro is 5mins, so
when eventually new MySQL data is seen, it instantiates a new
session/analyzer, however, the MySQL analyzer currently does not
handle "partial connections" and everything from then on is just
skipped. You can look into tuning the tcp_inactivity_timeout setting
to potentially workaround that, or else it would be better if anyone
wanted to investigate how much it would take to get the MySQL
parser/analyzer able to pick up and analyze sessions already mid-way
in progress.
Thanks Jon for the tips. I did see that Wireshark is reporting malformed response data, but since Bro was able to produce events initially I did not dig into it too much. My MySQL was installed through Ubuntu which is
mysqld Ver 5.7.23-0ubuntu0.16.04.1 for Linux on x86_64 ((Ubuntu))
I will follow your tips and see if I can find more information and report back.
I don't know how difficult this would be, but personally I've been putting off too much work into these sorts of efforts with the existing binpac analyzers because Spicy should have a mechanism to make analyzers "re-synching". I did it with SMB and it was a little tricky to get right.