Per connection byte and packet counting

Development
1

Hi,

I've played around with packet and byte counting for connections some
more. I think I've got a pretty neat version:

* Counting is done in a separate analyzer that can be added to the
  initial analyzer tree based on a global boolean const.

* Reporting is done via the endpoint records in the connection record (
  c$orig, c$resp). The endpoint records have to new &optional fields for
  number of packets and number of bytes.

* Updating the counters in the endpoint is done by:
  + Removing UpdateEndpointVal() from TransportAnalyzer
  + Adding a UpdateConnVal(conn_val) to Analyzer.cc. When a Conn object
    needs to update the conn_val (e.g., to generate an event) it calls
    the root analyzer's UpdateConnVal(). This call is then "forwarded"
    to every analyzer in the tree. I.e., every analyzer could update the
    conn_val, e.g., by adding additional, optional fields to the
    connection record.

In terms of speed and memory consumption this is surprisingly
lightweight. Runtime as reported by time, memory consumption from top.
I've only loaded conn and weird:

* Baseline. Current git master: (3 runs)
     2m42s 900MB
     2m37s 900MB
     2m37s 900MB

* My version, with counting disabled (i.e., counting analyzer not
  instantiated, no the optional fields in endpoint are NULL):
     2m34s 898MB
     2m36s 898MB
     2m36s 898MB
  It seems to be faster and using less memory(!). There is an
  explanation why it needs slightly less memory: the git-master Conn.cc
  had two RecordVal* for orig_endp, and resp_endp that were passed to
  UpdateEndpointVal. Since I now only pass the conn_val to
  UpdateConnVal, I don't need these pointers in Conn.cc anymore.
  I've checked the testsuite it my version passes it.

* My version with counting enabled:
     2m40s 1000MB
     2m41s 1000MB
     2m41s 1000MB
     2m41s 1000MB
  So: no speed penalty and about 100MB for the analyzer instance and
  the additional fields in endpoint (these are responsible for most of
  the memory. Vals are expensive memory-wise).

* As an additional experiment, I made the history field in conn_val
  &optional, so that it's only allocated, if record_state_history=T.
  Results:
    with counting: 2m39s 862MB
    wihtout counting: 2m33s 962MB
  So I think it might make sense to also integrate this change.
  (Currently the history is always tracked and always part of the
  connection record, but it is only logged in conn.log if
  record_state_history=T).

Finally, a longer trace with way more data and more policy scripts (conn
weird dpd http-request htt-reply dns):
    master / baseline 88m38 2906MB
    no counting 88m37 2907MB
    with counting 87m47 2963MB

So, all in all I would say: we should go with this version, as this
seems to be the ideal solution: overhead when disabled, only memory
overhead when enabled (and this overhead is inevitable!).

In addition, I would suggest to make c$history &optional and only
allocate and assign a Val* to it if record_state_history=T, since this
seems to safe quite a bit of memory!

cu
Gregor

2

This sounds awesome!

The only question I have is what is the $history field used for? This is second time I've seen it referred to today, but I have no idea what it's for. :slight_smile:

  .Seth

3

It records the state history of (TCP) connections. When
record_state_history=T it will add an additional column to conn.log.
It's probably mostly used for measurements rather than live security
deployment. The "history" is a string of letters:

-8<------
s == a SYN w/o the ACK bit set
h == a SYN+ACK ("handshake")
a == a pure ACK
d == packet with payload ("data")
f == packet with FIN bit set
r == packet with RST bit set

I also think there is a "c" which means we saw a packet with a bad
checksum and "i" for "inconsistent" packets---basically just weird crap
like both SYN and RST bits lit up.

If the code is in upper case it means the event comes from the
originator and lower case then means the responder.

Also, there is compression. We only record one "d" in each direction,
for instance. I.e., we just record that data went in that direction.
This history is not meant to encode how much data that happened to be.
->8------

cu
Gregor

4

Yes, this all sounds really good.

One more thought: does expect_connection() work with this (see,
e.g., ftp.bro for how it's used). Then one could enable it for
individual future sessions (like file transfers).

In principle, it seems an enable_analyzer() function (which we still
don't have; there's just disable_analyzer()) could be nice for this
too. However, it would only get a chance to add the counting once a
few packets have already passed so not sure how useful it would
really be.

Robin

5

One more thought: does expect_connection() work with this (see,
e.g., ftp.bro for how it's used). Then one could enable it for
individual future sessions (like file transfers).

Don't think so, since it needs to be a child *packet* analyzer of
TCP_Analyzer.

In principle, it seems an enable_analyzer() function (which we still
don't have; there's just disable_analyzer()) could be nice for this
too. However, it would only get a chance to add the counting once a
few packets have already passed so not sure how useful it would
really be.

In addition to enable_analyzer() and add_to_initial_tree() function
would also be nice, so that users can select analyzers that they want to
run on all connections.

6

Hi,

since there were no more comments. I'll wrap up these changes and make
them ready for a merge.

I will also make the connection history in the conn record an optional
field, since this will save quite a bit of memory.

cu
Gregor