Core affinity on AMD Opteron 6276

Hi everyone,

I’ve been told this is the right place to share experience and maybe a script for our Opteron setup, and get feedback if this is the right thing to do.

We’re trialling Bro on an AMD Opteron 6276 in our campus network and did not find much useful information on the net on how to configure this particular setup. It’s running fine now and we’re waiting for students to return to uni so we can test under a higher load (they need to watch some videos…).

Anyway, the 6276 is a strange machine in that AMD markets it as a 64-core machine - 4 sockets, 16 cores each. However, cores are paired within one socket, and each pair shares resources and data lines: CPU freq regulation, FPU, L2+L3, and instruction fetch and decode circuitry (!). This makes it quite unusual - under Linux, you will find that different methods to count the cores give you a different number, sometimes 32, sometimes 64.

For our experiments, we chose to use /proc/cpuinfo to determine which cores are pairs - they should share “physical ID” and “core ID”. The attached hacky script generates a node_cluster.cfg that places 32 workers on 32 cores that should not be paired (logger, manager, proxy are going to sit on another host).

As far as I can tell, we are not experiencing packet loss at all with the out-of-the-box scripts loaded, but we’re at just 2Gbit load atm, and come the new semester we’ll know more. The moment I add another core, I have one worker that is experiencing loss, so I am guessing this is the upper limit.

I’d be happy to receive feedback if this is a reasonable setup and the right thing to do.

Ralph

create_config_from_file.py (1.36 KB)