Per connection byte and packet counting

Hi,

I've played around with adding support to count packets and (IP-)bytes
per connection and direction. (Sort of like in addition to conn.log's
osize and rsize, which are based on sequence numbers).
This adds 4 counters per connection.

Basically, there are two ways to implement this:

a) hard coded into the transport analyzers that keep track of the
   connection record (which includes osize and rsize). I've done that
   and added the counters to the "endpoint" record val. (A connection
   record val has an orig and resp "endpoint" record val).

   Advantage: every connection event, i.e., the majority of events, now
   has access to these counters whenever the event is generated.

   Disadvantage: one cannot turn it off and it will result in a memory
   overhead per connection (see below).

b) implement it as a child-analyzer to the transport analyzers and add
   this child-analyzer to the initial analyzer tree. One can configure
   whether to enable/disable this size-analyzer via policy scripts. I've
   done that too.

   Advantage: can be switched on and off. No overhead when off.

   Disadvantage / Questions:
   * How to pass the information to policy scripts? (see below)
   * Slightly slower due to calls to an additional analyzer (and
     virtualized method lookups)

I've implemented both variants to check their performance. Here are
the results using a quite large Time Machine trace and only loading conn
and weird. The trace has some 14.8 M connections total.

  * trunk: 26.5min--27min runtime, 900MB virtual mem, 890MB residual.
  * b: 29.25min runtime, 952MB virtual mem, 940MB residual.

WRT variant b) (separate analyzer): So far I've implemented an event
when the analyzer is Done, i.e., when the connection ends. So, the
policy script can collect the counter values only after the connection
ends. However, I think it might be helpful, if one could access these
counters at any time.
* Is there any way to do so? E.g., by having a table that's indexed by
  connid that can be read from the policy layer but is updated by the
  event engine? Can this be done from an analyzer, or are there some
  hacks required?
* Other ideas to make the counters available to the policy script.
* Do you guys think it's important to have access to the counters at
  any time, or is it sufficient to just get them when the connection
  ends?

Any thoughts or comments?
Gregor

I've played around with adding support to count packets and (IP-)bytes
per connection and direction.

Awesome!

b) implement it as a child-analyzer to the transport analyzers and add
  this child-analyzer to the initial analyzer tree. One can configure
  whether to enable/disable this size-analyzer via policy scripts. I've
  done that too.

  Advantage: can be switched on and off. No overhead when off.

I like this (option b).

  * Slightly slower due to calls to an additional analyzer (and
    virtualized method lookups)

I think the fact that it's slightly slower is mitigated by it being optional.

* Other ideas to make the counters available to the policy script.

Couldn't it just be in the connection record? Maybe you could add the extra data to the endpoint record? Making it look something like this...

type endpoint: record {
  size: count;
  state: count;
  counted_size: count &optional;
  counted_packets: count &optional;
};

That should be accessible from the core (since size and state are currently being filled in from the core) and it makes it available at arbitrary times since it's in the connection record.

* Do you guys think it's important to have access to the counters at
any time, or is it sufficient to just get them when the connection
ends?

Yes, I would really like to access these counters at any time. I would even say it's a requirement for this feature (for me at least).

  .Seth

Oh, and it would be awesome if it was possible to use these values in "when" statements. Currently, these endpoint record values can't be used in when statements, but it would make for really simple code in some cases. For example, in my ssh-ext script I have to set timers to regularly check back into connections to see if a byte threshold has been crossed like this:

event ssh_watcher(c: connection)
  {
  local id = c$id;
  # don't go any further if this connection is gone already!
  if ( !connection_exists(id) )
    {
    delete active_ssh_conns[id];
    return;
    }

  # run the code to check the size of the connection
  event check_ssh_connection(c, F);
  if ( c$id in active_ssh_conns )
    schedule +15secs { ssh_watcher(c) };
  }

event protocol_confirmation(c: connection, atype: count, aid: count)
  {
  if ( atype == ANALYZER_SSH )
    {
    local tmp: ssh_ext_session_info;
    active_ssh_conns[c$id]=tmp;
    schedule +15secs { ssh_watcher(c) };
    }
  }

It would much easier to do...

event protocol_confirmation(c: connection, atype: count, aid: count)
  {
  if ( atype == ANALYZER_SSH )
    {
    when ( c$resp$counted_size > 5120 )
      {
      # This is a heuristically derived "successful" SSH connection.
      }
    }
  }

Of course, I don't know why this doesn't currently work or if it's something that could even be reasonably implemented but it would be *really* nice. :slight_smile: If it did work, it would be one less "gotcha" in the scripting language.

  .Seth

Couldn't it just be in the connection record? Maybe you could add the extra data to the endpoint record? Making it look something like this...

type endpoint: record {
  size: count;
  state: count;
  counted_size: count &optional;
  counted_packets: count &optional;
};

If I do this, then I basically have variant (a). (with the addition that
it's slower than (a)).

The advantage of (b) is that it has no memory overhead if the counters
are not used, but if I add them to the connection record, this advantage
is gone.

The only way for (b) would be to have a global table, indexed by connid,
that yields the counters. But I think this could be painful, because I
would have to update this table from the event engine for each received
packet.

cu
Gregor

If I do this, then I basically have variant (a). (with the addition that
it's slower than (a)).

Heh, the worst of both worlds. :slight_smile:

The advantage of (b) is that it has no memory overhead if the counters
are not used, but if I add them to the connection record, this advantage
is gone.

Bro allocates memory for unused optional elements?

Oh, I guess I hadn't thought about this enough anyway. My thought was that you'd be able to give some signal back to the core with a BiF call or something which would then enable the counting for the connection but that doesn't work because by the time you get your first event (like connection_established) you've already missed several packets. Hm.

The only way for (b) would be to have a global table, indexed by connid,
that yields the counters. But I think this could be painful, because I
would have to update this table from the event engine for each received
packet.

That seems sort of hacky too.

  .Seth

That should be accessible from the core (since size and state are currently being filled in from the core) and it makes it available at arbitrary times since it's in the connection record.

Oh, and it would be awesome if it was possible to use these values in "when" statements. Currently, these endpoint record values can't be used in when statements, but it would make for really simple code in some cases. For example, in my ssh-ext script I have to set timers to regularly check back into connections to see if a byte threshold has been crossed like this:

[...]

Hmm. Actually this currently only works for connection records due to
the way the connection record is allocated in the event engine. Each
connection instance (C++ class instance) has a conn_val Val* that get's
allocated once is then updated and passed to each event. If you save
this record in the policy layer, it will get updated "in the background"
by the event engine. However, it will only be updated if the event
engine calls BuildConnVal(), which is in general only done if a event
for this connection is generated. So, if the protocol_confirmation()
event is the only event for this connection that's generated, then the
connection value would not be updated.

Note, that somebody could also generate a connection event and just pass
a newly allocated connection record to it. I this case, the above
approach would also break (e.g., if you use this connection record
further down the line, it would never changes its values).

It would much easier to do...
[...]

Of course, I don't know why this doesn't currently work or if it's something that could even be reasonably implemented but it would be *really* nice. :slight_smile: If it did work, it would be one less "gotcha" in the scripting language.

(partially guessing, since I haven't dug too deep into how "when"
statements work).

I think the problem is that the connection record (and thus the conn_id)
is only valid in the scope of the function. The when statement clones
the local stack frame. So for everything that's in the local scope, you
only get a static copy. I think this would work, if you keep a global
table with "conn_ids_to_watch", add the conn_id to this table and then
use the table for the "when" statement. Maybe this code works:

table conns_to_watch[conn_id] of connection;

event protocol_confirmation(c: connection, atype: count, aid: count)
{
   if ( atype == ANALYZER_SSH )
       {
       conns_to_watch[c$id] = c
       when ( conns_to_watch[c$id]$resp$counted_size > 5120 )
           {
           # ....
           }
       }
}

You probably don't need to save to full connection record in the table.
I'm not sure what happens when the connection get's removed before then
when triggers.....

cu
Gregor

> counted_size: count &optional;
> counted_packets: count &optional;
> };

If I do this, then I basically have variant (a). (with the addition that
it's slower than (a)).

I don't understand this. &optional fields will be null pointers if the
field isn't present. So it's just 4 bytes each when not in use, rather
than a full Val record (plus 4 bytes to point to it). I also don't
see why it would be slower, other than the smidgen of cycles it takes to
initial the null pointers.

The only way for (b) would be to have a global table, indexed by connid,
that yields the counters.

Yuck. But first let's understand the issue above.

    Vern

However, it will only be updated if the event
engine calls BuildConnVal(), which is in general only done if a event
for this connection is generated.

Yep. Though with the optional approach, the event engine could know to
build the connection value for every packet if there's a pending "when"
that cares (at least, I think it can tell when this is the case - Robin
is the definitive viewpoint here).

Note, that somebody could also generate a connection event and just pass

I don't see any need to worry about that. No one is supposed to be
generating connection events. Connection records are linked with the event
engine and thus shouldn't be created separately.

I think the problem is that the connection record (and thus the conn_id)
is only valid in the scope of the function.

This is also a question for Robin, as it involves the particulars of just
how "when" is implemented uner the hood.

    Vern

Actually, I was wrong with the &optional fields. I thought they would
always need to allocate memory in the background.
So, I did some more test and here are the results:

* 901 MB ... baseline trunk

For these, I added two optional counts to the endpoint record, and set
them to NULL:

* 903 MB ... baseline w/ optionals. No counting what-so-ever

In addition, I now added counting (so far still directly in the
TCP_Analyzer and UDP_Analyzer).

* 908 MB ... Counters as uint64_t directly in UDP_Analyzer and
             TCP_Endpoint, but keep optional fields in endpoint
             record as NULL.

* 905 MB ... Counters in their own struct, pointers to this struct
             in UDP_Analyzer and TCP_Endpoint. But pointers are not
             allocated. Optionals are NULL as well.

* 917 MB ... Counters as structs as above, but this time allocated.
             Optionals are NULL (this case is rather pointless, as
             we do count here, but we don't report it)

* 970 MB ... Counters as structs, allocated. Report values in endpoint
             record.

I haven't yet tested the "Counting in separate Child Analyzer, reporting
via connection record's endpoint record". I have to figure out an
elegant way to do this. (See other mail).

BTW, a question, it seems that for optionals I can either assign a NULL
pointer:
  rv->Assign(n, NULL)
or not assign anything at all. The policy-layer seems to handle both
variants in the same way.

cu
Gregor

BTW, a question, it seems that for optionals I can either assign a NULL
pointer:
  rv->Assign(n, NULL)
or not assign anything at all. The policy-layer seems to handle both
variants in the same way.

I think because they're equivalent. All the above does is assign the
nth field to a NULL pointer, and presumably it's initialized to that
anyway.

    Vern

* 901 MB ... baseline trunk
* 903 MB ... baseline w/ optionals. No counting what-so-ever

This looks good enough to me, seems fine adding the fields.

I haven't yet tested the "Counting in separate Child Analyzer, reporting
via connection record's endpoint record".

Conceptually this would be my preferred approach. See my other mail
for thoughts on how to implement it. But it would be interesting to
see memory usage and potential run-time ffects for this case before
deciding.

Robin