Lo All,
Started receiving this error after adding a worker yesterday. If I remove the worker and deploy, no issues.
Communications.log: “accept failed, Too many open files 24”
Running CentOS6. Bro 2.4.1. 17 Physical worker systems. 150 total worker processes.
When adding the 18th worker (6 additional worker processes) logs slow to a crawl and the communications log is filled with the failure message.
I’ve experimented with limits.conf and set a high soft and hard limits of open files. Also tried doubling the defaults, and many different combinations to no avail. Most of these caused bro to hang and stop logging. Others had no affect on the problem.
Has anyone had to deal with this issue or have some ideas? Is there some hidden setting in bro where I can set open file limits?
There was an issue with file handles being left open when they weren't needed which will be fixed in 2.5 (and is fixed in the 2.5 beta). Also, in 2.5 you can run a cluster with a dedicated "logger" node which should improve some of the cluster behavior with such a large cluster.
I would recommend jumping in and trying out the beta.
.Seth
Has anyone had to deal with this issue or have some ideas? Is there some hidden setting in bro where I can set open file limits
There was an issue with file handles being left open when they weren't needed which will be fixed in 2.5 (and is fixed in the 2.5 beta). Also, in 2.5 you can run a cluster with a dedicated "logger" node which should improve some of the cluster behavior with such a large cluster.
I would recommend jumping in and trying out the beta.
The 2 file descriptor leakage fixes were committed just after the 2.5 beta was released
commit 520ed43eae4ce7bcd8bb22cfd9cb6d138c4a4fd7
Author: Daniel Thayer <dnthayer@illinois.edu>
James,
I tried all those options. Bro would hang when I changed the soft limit, and no matter what hard limit I set, it still produced errors. I’m going to do a parallel install of 2.5 and see if that fixes it. I have already tested, but was waiting for the full release.
Thanks for you responses.
You bet...sorry it didn't help. Would like to know what the resolution was once you find it.
James