Hello again,
Half a day more of GDB have unearthed the root cause of this problem. I
am posting this, since this is behavior that potential developers of Bro
analyzers will want to know about. Or possibly, it is a bug.
In my binpac grammar, at one point I need to decide which rule to take
based on the state of both upflow and downflow. The first payload packet
(handshake) comes from upflow and, while processing, I check if a bit is
set. The second packet processed is the handshake from downflow where
the bit is also checked. For any subseqent packets/messages a different
rule is taken depending on if both bits are set or not.
However, Bro does something (at least for me) unexpected. After
processing the upflow handshake, it starts processing the next message
in this flow, although the data is not there yet. Since the downflow
message has not been processed yet the flag is not set, and Bro decides
(somehow correctly) that the next message is going to be of the standard
type. It creates a message object, and sets the FlowBuffer frame size to
4, since this is how long the next message is at least supposed to be.
Then it stops processing since the data isn't actually there, and starts
working on the downflow.
When the processing comes back to the upflow, the downflow handshake has
been parsed and the flag set. Therefore, Bro does not reuse the message
object, but creates a new one for the other type of message. However,
the state in FlowBuffer (namely the frame size) is not reset, and the
first step in parsing the new message is to skip the 4 bytes from the
(actually nonexistent) previous message which effectively swallows the
first 4 bytes of the current message - and that causes the parsing to
use the wrong fields and in the end causes an out_of_bounds exception
that stops the processing.
I my grammar, I did not expect processing of messages before they arrive
and it should work perfectly fine when the messages are processed in the
same order they appear in the dump file. I'm curious if the observed
behavior is some sort of optimization that you just need to consider
when writing binpac grammars (and even more wondering if for some cases
it's possible to write the grammar in the required way) or if protocols
where both sides contribute to the state have not been considered when
designing binpac. I have noticed that the Bro wiki mentions the
$context.flow macro but not the $context.connection macro which is also
there. For now, I am rewriting my grammar in the hope that I can
circumvent my problem, but I would welcome any comments or
clarifications from the authors or savvy developers on that issue.
greetz Martin