Hi,
I was wondering what some major differences are between the concept of a ‘connection’ in Bro and a a ‘flow’ in NetFlow. Or are they essentially the same concept? If this requires a detailed answer, a reference would be very helpful!
Thank you!
Netflow connections are generally logged and a new connection recorded if they exceed 30 minutes. That’s one.
We set our routers to export flows after one minute if they’re still in progress (it’ll continue to send a flow export every minute until it’s complete). More info here:
https://www.manageengine.com/products/netflow/help/cisco-netflow/cisco-ios-netflow.html (“ip flow-cache timeout active 1” is the command to use)
This means that, AFAIUI, Netflow can be made to be more timely than Bro. Bro will only output a bro_conn when the flow has been deemed to have finished.
Also, Netflow exports are unidirectional – you get separate flow exports for A->B and B->A. With Bro, a bro_conn logs traffic in both directions.
alec
The fun part about Bro is that it's a scripting language and we can do whatever we want! 
Here's a script that I wrote in Broala a while ago that we're releasing under the BSD license.
https://github.com/broala/bro-long-connections
I think I will need to do a bit more work on this to make it more like flow cutting, but at the very least it now makes active connections visible. Any feedback would be appreciated.
Thanks!
.Seth
Have you tested it with loooots of connections? How hard it is on the memory and CPU?
It hasn't been tested very extensively, but I wouldn't expect it to have much trouble with either memory or CPU since it's just riding on top of the existing connection state mechanism.
.Seth
Imma test this out Seth thank you...I'll report findings here.
James
Awesome! Thanks.
.Seth
Yea this worked well:
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn_long
#open 2016-09-02-13-33-11
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string]
1472843591.734071 CVFKX14EPP6mzgoKta 192.168.1.3 61648 172.217.3.174 443 tcp - 1193.913203 0 23239 OTH T F0 had 0 0 565 52825 (empty)
I see this in the script\main.bro:
## The default duration that you are locally
## considering a connection to be "long".
const default_durations = Durations(10min, 30min, 1hr, 12hr, 24hrs, 3days) &redef;
I'd like to see an example of redefing this to a different time. Also, a whitelist of IP's not to be included would be next. I have a lot of use cases...truth be told I'm "kind of" doing something similar with grep/sed/awk and the current conn_log for tracking "unusual" long sessions. For example, a netblock, say 172.16.1.0/24 is dedicated to VPN connections, which I expect to be longer as they are a constant session, so i'd want to ignore those in my conn_long file. Thanks Seth!
James
const default_durations = Durations(10min, 30min, 1hr, 12hr, 24hrs,
3days) &redef;I'd like to see an example of redefing this to a different time.
redef LongConnection::default_durations = LongConnection::Durations(30sec, 1min, 1hr, 10hrs, 1day);
Also, a whitelist of IP's not to be included would be next. I have a lot of
use cases...truth be told I'm "kind of" doing something similar with
grep/sed/awk and the current conn_log for tracking "unusual" long
sessions.
Except that you unfortunately aren't seeing connections "live" before the connection has completed.
For example, a netblock, say 172.16.1.0/24 is dedicated to
VPN connections, which I expect to be longer as they are a constant
session, so i'd want to ignore those in my conn_long file.
Ah, interesting point. It sort of sounds like you're starting to use the log for detection with this change though. Are you sure you want to do that? Would it make more sense if we added some other behavior that actually detected something that you're interested in? Alternately you could use a logging filter that filters out connections involving the hosts on your VPN. Here's one you can start with....
const ignore_for_long_connections: set[subnet] &redef;
event bro_init()
{
local filt = Log::get_filter(LongConnection::LOG, "default");
filt$pred = function(rec: Conn::Info): bool
{
return rec$id$orig_h !in ignore_for_long_connections &&
rec$id$resp_h !in ignore_for_long_connections;
};
Log::add_filter(LongConnection::LOG, filt);
}
.Seth
Ah there you go...yea a logging filter would be best...I'll give that a whirl in my test environment. And in this case I'm really interested in sessions that are over a certain time, possibly over longer than a day, and at a very small throughput (can you say data exfiltration?). So I have a subset of internal IP's that are known to have long sessions..anything else I wanna see. Thanks again Seth!
James