Possible system tweak for reducing memory usage

TL;DR: It's possible that transparent huge pages and bro do not get along, try doing a

    # on all nodes
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    # then
    broctl restart

There are ways to make that permanent if it helps.

I've been doing some research to try to figure out why some people have more memory issues than others. I think the kernel feature Transparent Huge Pages (THP) and bro may not get along well. It's supposed to help performance for memory allocations, but many services recommend disabling it (mongodb, redis, mysql). For example:

Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.

However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP on Linux machines to ensure best performance with MongoDB.

Bro memory allocations can best described as unpredictable, especially on 'custer in a box' deployments.

On our systems, disabling it drops bro worker memory usage by 20% and manager/logger usage by even more, but since we only have one of those it's harder to compare. For workers I disabled THP on half the nodes, and the post bro restart memory usage is consistently lower.

This was always a RHEL6/CentOS6 requirement for applications like you stated.

Which OS are you noticing the issue and the performance gains on?

Centos 7.3

Can confirm that this has always been a performance improvement in our environment. We have historically used Ubuntu 14.04 and 16.04.

I think pf_ring ZC requires hugepages so this fix benefits a subset of Linux deployments.

huge pages or transparent huge pages?

Oops. Only hugepages, not transparent hugepages. Nevermind what I said.