I've played around with packet and byte counting for connections some
more. I think I've got a pretty neat version:
* Counting is done in a separate analyzer that can be added to the
initial analyzer tree based on a global boolean const.
* Reporting is done via the endpoint records in the connection record (
c$orig, c$resp). The endpoint records have to new &optional fields for
number of packets and number of bytes.
* Updating the counters in the endpoint is done by:
+ Removing UpdateEndpointVal() from TransportAnalyzer
+ Adding a UpdateConnVal(conn_val) to Analyzer.cc. When a Conn object
needs to update the conn_val (e.g., to generate an event) it calls
the root analyzer's UpdateConnVal(). This call is then "forwarded"
to every analyzer in the tree. I.e., every analyzer could update the
conn_val, e.g., by adding additional, optional fields to the
In terms of speed and memory consumption this is surprisingly
lightweight. Runtime as reported by time, memory consumption from top.
I've only loaded conn and weird:
* Baseline. Current git master: (3 runs)
* My version, with counting disabled (i.e., counting analyzer not
instantiated, no the optional fields in endpoint are NULL):
It seems to be faster and using less memory(!). There is an
explanation why it needs slightly less memory: the git-master Conn.cc
had two RecordVal* for orig_endp, and resp_endp that were passed to
UpdateEndpointVal. Since I now only pass the conn_val to
UpdateConnVal, I don't need these pointers in Conn.cc anymore.
I've checked the testsuite it my version passes it.
* My version with counting enabled:
So: no speed penalty and about 100MB for the analyzer instance and
the additional fields in endpoint (these are responsible for most of
the memory. Vals are expensive memory-wise).
* As an additional experiment, I made the history field in conn_val
&optional, so that it's only allocated, if record_state_history=T.
with counting: 2m39s 862MB
wihtout counting: 2m33s 962MB
So I think it might make sense to also integrate this change.
(Currently the history is always tracked and always part of the
connection record, but it is only logged in conn.log if
Finally, a longer trace with way more data and more policy scripts (conn
weird dpd http-request htt-reply dns):
master / baseline 88m38 2906MB
no counting 88m37 2907MB
with counting 87m47 2963MB
So, all in all I would say: we should go with this version, as this
seems to be the ideal solution: overhead when disabled, only memory
overhead when enabled (and this overhead is inevitable!).
In addition, I would suggest to make c$history &optional and only
allocate and assign a Val* to it if record_state_history=T, since this
seems to safe quite a bit of memory!