new Bro pre-release available - 0.7a90

A new Bro pre-release is now available from:

The main theme of this release is better state management, especially for
use in high-volume sites. There are now a number of ways to track the
resources consumed by Bro while it runs, and to reduce the amount of memory
Bro consumes. Adding

  @load print-resources

will log a summary of the resources consumed by Bro when it exits, and

  @load reduce-memory

will change a number of default values in order to significantly diminish
the amount of memory Bro requires.

Other ways to lower the required memory:

  - The global "maintain_http_sessions" now controls whether
    http.bro tracks multiple HTTP connections between the same
    hosts as a single HTTP session. Doing so costs a considerable
    amount of state (so set to F for reducing the memory impact
    of HTTP analysis).

  - The global "suppress_scan_checks", if true, turns off address
    and port scan detection. (You can't achieve this by simply
    not @load'ing scan.bro, because it's loaded by some of the
    default policy scripts.) Turning it off can save a lot of memory.

  - Note, the ident.bro is also expensive in terms of state-holding.

A summary of other changes with respect to the last snapshot:

  - New library functions:

    - resource_usage() returns a record detailing real time,
      CPU time, memory, other getrusage info, and the current
      and maximum number of TCP/UDP/ICMP connections, and
      timers and fragments.

    - val_size() returns the size in bytes needed to represent
      a given value (which can be a record, a table, or a
      simple constant, for example). It's not fully accurate
      but is in the right ballpark.

    - global_sizes() return a table mapping every global
      variable to its size (per val_size()). Useful for
      tracking which ones are growing large over time.

  - You can now control a number of timers related to Bro's connection
    management. Setting them to lower values generally decreases the
    amount of state Bro has to keep (see reduce-memory.bro), though
    this can also make it easier for an attacker to evade detection:

    tcp_SYN_timeout: interval
      Controls how long Bro waits after seeing the
      beginning of a connection (whether due to a SYN
      or not; the timer is misnamed) before checking
      whether it elicited any reply.

    tcp_session_timer: interval
      After a connection has closed, wait this long for
      further activity before checking whether to time
      out its state.

    tcp_connection_linger: interval
      When checking a closed connection for further
      activity, Bro should consider it inactive if there
      hasn't been any for this long. It also complains
      if the connection is reused before this much time
      has elapsed.

    tcp_attempt_delay: interval
      Bro waits this long upon seeing an initial SYN
      before timing out the connection attempt.

    tcp_close_delay: interval
      Upon seeing a normal connection close, Bro flushes
      state after this much time.

    tcp_reset_delay: interval
      Upon seeing a RST, Bro flushes state after this
      much time.

    tcp_partial_close_delay: interval
      Bro generates a connection_partial_close event this
      much time after one half of a partial connection
      closes, assuming there has been no subsequent

    non_analyzed_lifetime: interval
      If a connection belongs to an application that you
      aren't analyzing, Bro times it out after this
      interval. If 0 secs (default), then it doesn't
      time it out.

    dns_session_timeout: interval
    ntp_session_timeout: interval
    rpc_timeout: interval
      Bro waits this long before timing out a DNS/NTP/RPC

    max_timer_expires: count
      The maximum number of timers to expire after
      processing each new packet. The value trades off
      spreading out the timer expiration load with
      possibly having to hold state longer. A value of 0
      means "process all expired timers with each new

  - Two new flags control other facets of Bro's connection management,
    and thus state-holding:

    partial_connection_ok: bool
      Whether Bro should analyze connections for which
      it doesn't see the beginning, only the middle.
      This can be very expensive to do in the face of
      stealth-scanning, which looks like a bunch of
      partial connections.

      Note, the HTTP analyzer has been modified to
      now always skip partial connections. This should
      instead be user controllable.

    tcp_SYN_ack_ok: bool
      If true, Bro instantiates connection state when
      it sees a SYN ack but not the initial SYN (even
      if partial_connection_ok is false). The intent
      behind this knob (which is not well tested) is
      to allow you to filter out initial SYNs and only
      react to SYN acks. This keeps Bro from holding
      state during SYN scans and SYN flooding, except
      for when the destination responds.

  - Some other miscellaneous thresholds that you can now modify
    from your policy script:

    heartbeat_interval: count
      How often to generate net_stats_update() events.
      This timer really isn't needed any more, since
      you can use "schedule" to achieve the same effect.

    tcp_storm_thresh: count
      If Bro sees this many FINs/RSTs in a row, it
      flags them as a "storm".

    tcp_storm_interarrival_thresh: interval
      The FINs/RSTs must come with this much time or less
      between them.

  - The state management for line-oriented applications like HTTP
    requests has been improved.

  - The HTTP analyzer now expands %hex sequences. If anyone has
    a Unicode expander to contribute, that'd be terrific.

  - The Code Red detection is more robust (fewer false positives).

  - A new redefinable variable, skip_services, lists applications
    that should not be analyzed for purposes of detecting scans.
    (Default: ident) The point of having this (poorly named) hook
    is so that code-red.bro can add "http" to it, to avoid reporting
    the zillions of Code Red scans that a site can see.

  - Bro now matches regular expressions using lazy DFA evaluation.
    The upshot of this is (1) it no longer maintains .bro-RE-cache.v1,
    (2) it starts up fast regardless of whether you've added new
    regular expressions, (3) you can afford to add lots of regular
    expressions, and (4) it's actually a bit faster.

  - The list of "hot_ids" has been expanded with a number of other
    common root-privileged accounts.

- Vern