I am using BRO v 1.5.1 to analyze off-line pcap files. When I run BRO
on 4Gb pcap file, one CPU core always reaches 100% but the server
still has more 15 idle cores.
The analysis uses brolite, dpd and detect-protocols.
I am afraid BRO is loosing packets. By the way, how can I measure
packet dropping?
The capture-loss generates this notice:
no=CaptureLossSummary na=NOTICE_ALARM_ALWAYS msg=estimated\ rate\ \=\
0.0082201 tag=@36-6fb3-4a
Are this events or bytes? WHy indicates tag? I cannot find any
reference to this tag in any of the other logs. By reading the
documentation, it seems you don't recommend this metric.
Instead, I will be happy to know the number of packets that BRO
processed. I cannot find where is this number logged.
Best regards
Veronica Estrada
Nakao's Laboratory
Univ. of Tokyo
Maybe Bro somehow behaves differently, but I don't think it's possible
for anything reading a pcap file to drop packets since the program
will read them as it has buffer available.
To utilize mutliple cores, try splitting the pcap into multiple files
with something like splitcap (splitcap.sourceforge.net) and running
parallel Bro processes. I can't tell you whether this will be faster
in the end, but I suspect it would be, especially if you have splitcap
write its splits to stdout and have Bro read from stdin, avoiding any
disk writing. This may take a bit of bash scripting, but I think it's
probably possible. Does anyone have something done like this already
to send to Veronica?
This may be possible. I just Googled and saw there is a program called
tcprelay that can be used to feed a pcap into an Ethernet interface.
You could use tcprelay to feed the pcap into the Click! Modular Router
and have Click! load balance the traffic to a Bro cluster with many
workers to utilize all your cores.
The cluster is quite easy to set up, and the Click! interface config is
pretty easy as well. I have a cluster of seven workers running on seven
of the cores in my server. The eighth runs Click!. The cluster manager
and proxy run on a recycled lab workstation with a big hard drive.
If you are interested, I can send a copy of my Click! configuration. It
is a modified version of Justin's that was posted to the list a while back.
Using this type of setup, you could run the Bro manager, proxy(ies) as
well as 12 or 13 worker processes and Click! all on the same server.
The only reason I moved my manager and proxy off was to have more
workers processing traffic. I think this will work with FreeBSD or
Linux. Click! kernel mode requires Linux, but I don't think the load
balancing uses kernel drivers.
You can grep the notice.log for Dropped to see how much traffic is not
being processed. I don't recall the script that logs that, but it is
probably drop.bro. I think it is on by default with the cluster config.
I am using BRO v 1.5.1 to analyze off-line pcap files. When I run BRO
on 4Gb pcap file, one CPU core always reaches 100% but the server
still has more 15 idle cores.
Right, because unless you set up a cluster, Bro runs as a single process.
I am afraid BRO is loosing packets. By the way, how can I measure
packet dropping?
It will not lose packets when reading from a trace off-line.
The capture-loss generates this notice:
no=CaptureLossSummary na=NOTICE_ALARM_ALWAYS msg=estimated\ rate\ \=\
0.0082201 tag=@36-6fb3-4a
That's its estimate of what was lost during the original packet capture
(i.e., the recording of the trace).
Are this events or bytes?
Unless you redef CaptureLoss::report_byte_based_estimates=T, the estimates
are events.
WHy indicates tag?
The tag is for use when operating a cluster; you can ignore it here.
By reading the
documentation, it seems you don't recommend this metric.
Right - events are more reliable than byte-based loss estimations.
Instead, I will be happy to know the number of packets that BRO
processed. I cannot find where is this number logged.
This isn't reported, but per the above, it will be the entire trace file
without any loss during its reading of it.