&expire_func functionality/verification with pcap

Building a little off my previous question, I have a structure my_table defined:

global my_table: table[string] of vector of HTTP::Info &write_expire = 30secs &expire_func=process;

and my_table will get written to in the connection_state_remove event, which should then call the expire_func 30s later.

I have tried triggering the functionality two ways:

  • Having Bro read in a 1GB test.pcap, waiting for minutes (with exit_only_after_terminate=T), then CTRL-C to exit
  • Having Bro listen on a dummy interface and tcpthrow the test.pcap against it, waiting for minutes then CTRL-C to exit

It seems to work for a subset of the connections but not all of them. My hunch is that Bro’s connection state table has no strict time-based removal process, so the connection_state_remove event will not be triggered unless I throw more data at it. My second thought is that it does get triggered at the end for the CTRL-C, but then shuts down before the expire_func fires 30secs later.

If my hunches correct please let me know, as then it should theoretically work with Bro on the wire as new data comes in. But for testing purposes, is there any way to either force flushes of the connection table or ensure that Bro waits long enough after the CTRL-C to handle the expire_func?

Best,

Jereme Lamps

I believe that's correct. The combination of the exit_only_after_terminate setting and reading a pcap is not particularly well supported because it's not needed for most circumstances. It's also conceptually hard to pull off cleanly because Bro's packet clock is driven by incoming packet timestamps. The reason you aren't seeing those connections expire is because as far as Bro is concerned time has stopped the moment that packets stop coming in.

   .Seth

Pulling the thread a little more:

I waited a few minutes after tcpthrowing the test.pcap and then tcpthrew a small pcap consisting of three pings. This was unfortunately not enough to trigger the connection_state_remove event for any of the other flows. I also tried creating a pcap that consisted of test.pcap followed by 3 pings 4 minutes later and running that through Bro with no luck. Do you think there is something additional going on underneath the hood?

In the conn.log I can verify the “ts” jumped ~4minutes between the last flow from the first pcap and the starting flow the second pcap.

Best,

Jereme

    > * Having Bro read in a 1GB test.pcap, waiting for minutes (with
    > exit_only_after_terminate=T), then CTRL-C to exit
    > * Having Bro listen on a dummy interface and tcpthrow the
    > test.pcap against it, waiting for minutes then CTRL-C to exit
    >
    > It seems to work for a subset of the connections but not all of them.
    > My hunch is that Bro’s connection state table has no strict
    > time-based removal process, so the connection_state_remove event will
    > not be triggered unless I throw more data at it
    
    I believe that's correct. The combination of the
    exit_only_after_terminate setting and reading a pcap is not particularly
    well supported because it's not needed for most circumstances. It's
    also conceptually hard to pull off cleanly because Bro's packet clock is
    driven by incoming packet timestamps. The reason you aren't seeing
    those connections expire is because as far as Bro is concerned time has
    stopped the moment that packets stop coming in.
    
       .Seth