To proxy or not to proxy...

Zeek
1

Hi folks,

So in my continuing pursuit of perfecting my Bro setup I found that adding a proxy on every box that also runs workers keeps bro much happier then a single manager/proxy box with one or more worker(s) boxes.

Prior to adding the additional proxies bro workers would die due to resource constraints.

Are other folks doing this?

Cheers,
Harry

2

I'm currently running a separate box that has the manager and proxies on it, but I did just as you describe at one point and it seemed to work fairly well. You may want to reduce your worker count a bit to leave enough CPUs for the proxies. Out of curiosity are you pinning your workers to dedicated CPU cores? If you are not it could be that your workers are bouncing between cores due to hyper-threading which can cause them to stomp all over each other. I found pinning workers to cores helped tremendously when it came to worker health.

~Gary

3

Same here: I have a proxy for every 10 workers on each of the physical box (which runs workers) in the cluster.

Ah! regarding CPU pinning:

fairly well. You may want to reduce your worker count a bit to leave
enough CPUs for the proxies. Out of curiosity are you pinning your
workers to dedicated CPU cores? If you are not it could be that your
workers are bouncing between cores due to hyper-threading which can
cause them to stomp all over each other. I found pinning workers to
cores helped tremendously when it came to worker health.

I agree completely!

Also, Make sure that you have enough cores to run workers on.

With respect to CPU pinning, on *FreeBSD*, CPUs are numbered as :
P = physical core
H = Hyperthread core

0/1 = P/H
2/3 = P/H
4/5 = P/H
...
...
11/12=P/H

You certainly don't want to pin_cpu on FreeBSD as 0,1,2,3 but instead pin_cpu=0,2,4,6,8.... (or 1,3,5,7...)

However, I beleive Linux does it different. While I have not yet looked at a Linux's box, I believe its scheme for hex-core processor is

0/6=P/H
1/7=P/H
2/8=P/H
..
..
5/12=P/H

so you might want to pin_cpu on linux as: pin_cpu=01,2,3,4,5 or (6,7,8,9,10,11,12)

Make sure you leave a few cores alone for proxy and other tasks when pinning.

Oh, btw, we have found no noticible difference in performance at all, when you pin a bro process on only physical core vs only hyperthreded cores. But make sure you don't pin bro processes on both P/H at the same time.

Now, it would be great if someone can confirm the linux side of the story. or shed more light on cpu_pinning.

Aashish

4

Great discussion and pointers. I’m working on a similar performance tuning and stabilization effort.

I took a closer look to verify and I can confirm Aashish’s statement about the numbering of cores on Linux. We’re running CENTOS 6.2. This box has 2 hyperthreaded hex-core procs. All physical cores are assigned sequential ID’s in socket/core order, then hyperthreaded cores are assigned sequential ID’s in socket/core order.

Here’s what we end up with:

__socket0 (P/H)__
0/12
1/13
2/14
3/15
4/16
5/17

__socket1 (P/H)__
6/18
7/19
8/20
9/21
10/22
11/23

Adam