Hello, new here
I’m using bro 2.2 and I connect to bro using broccoli to receive events.
I can manage connecting to bro-worker and receive events, not sure if it’s the correct way to receive event from bro but connecting to the manager port didn’t retrieve any event whatsoever,
the problem is that when I receive events at speeds higher than 2Mbps the parent of the bro-worker (not the broccoli application) memory expands rapidly and can reach 10Gb in a minute.
Disconnecting the broccoli application immediately frees all memory of the worker (10Gb to 100Mb in less than a second).
Daniel.
I'm using bro 2.2 and I connect to bro using broccoli to receive events.
I can manage connecting to bro-worker and receive events, not sure if it's the correct way to receive event from bro but connecting to the manager port didn't retrieve any event whatsoever,
Could you give more information about what events you are sending around? Are you receiving events from Bro or sending them to Bro?
What configuration have you done in Bro to send or receive these events?
the problem is that when I receive events at speeds higher than 2Mbps the parent of the bro-worker (not the broccoli application) memory expands rapidly and can reach 10Gb in a minute.
Another interesting number might be events per second. I'm even a little unclear what you mean by 2Mbps. Do you mean that the data rate of your connection between your broccoli application and Bro is 2Mbps?
.Seth
Could you give more information about what events you are sending around? Are you receiving events from Bro or sending them to Bro?
I’m capturing HTTP events, only receiving, not sending
What configuration have you done in Bro to send or receive these events?
I’ve used the default settings, same configurations as in the examples that come with broccoli
I was sending out many HTTP requests which causes raising of many events per request/response, capstats showed I was at 2~3 Mbps transfer rate on the interface but when I checked the transfer rate of events between bro and my broccoli client I was at 600~700Mbps, the events seem to be too large, even when using compact events, and my broccoli client ended up using 100% cpu of the core it was on, maybe enabling parallelism of this section could give better results at events processing.
I’ve figured out the memory expansion problem, it seems that the “ChunkQueue” in “ChunkedIO” does not have a limit and I was sending events at higher speeds than my broccoli client could process so the queue just kept growing.I updated the queue so it will drop chunks when it reaches a certain limit of chunks in the queue and now the memory stays steady at 1.5GB even at high speeds of events.
This is a temporary fix in my opinion, a more robust communication framework is needed such as using an external queue (such as ActiveMQ / ZeroMQ) for transferring events/chunks.
Daniel.
I was sending out many HTTP requests which causes raising of many events per request/response
Generally, I wouldn't recommend sending around protocol based events. Sending anything with a connection record that needs serialized and deserialized is probably not a good idea.
Why are you sending so much data by the way? You may have approached the problem with a suboptimal design.
I've figured out the memory expansion problem, it seems that the "ChunkQueue" in "ChunkedIO" does not have a limit and I was sending events at higher speeds than my broccoli client could process so the queue just kept growing.
I was sort of curious if that's what was going on. Nice to have an answer to that. 
This is a temporary fix in my opinion, a more robust communication framework is needed such as using an external queue (such as ActiveMQ / ZeroMQ) for transferring events/chunks.
There is already major overhaul of Bro's communication system underway.
.Seth
Why are you sending so much data by the way? You may have approached the problem with a suboptimal design
I’m extracting information about HTTP requests/responses going through the network and I’m using an external database to save some of that data so I couldn’t just use Bro scripting so using broccoli was a nice solution at that time. If you have any suggestions how I could implement my application without using broccoli It’d be great.
There is already major overhaul of Bro’s communication system underway
Is there a place I can find more information about that?
Another problem I had is that I tried upgrading to Bro 2.3 but I couldn’t receive any event through broccoli like I was receiving with Bro 2.2 no matter what configuration I was using on the bro client side, should have I enabled it on the Bro side somehow?
Why are you sending so much data by the way? You may have approached the problem with a suboptimal design
I'm extracting information about HTTP requests/responses going through the network and I'm using an external database to save some of that data so I couldn't just use Bro scripting so using broccoli was a nice solution at that time.
Ah. You could write a logging writer. We do have an SQLite writer already and there is a PostgreSQL writer in the pipeline. Alternately, you could write to a log on disk and then have some other process read that file in and pass it to the database.
There is already major overhaul of Bro's communication system underway
Is there a place I can find more information about that?
Not really yet. It's in the early implementation phase still and there is no timeline on when it will be functional yet.
Another problem I had is that I tried upgrading to Bro 2.3 but I couldn't receive any event through broccoli like I was receiving with Bro 2.2 no matter what configuration I was using on the bro client side, should have I enabled it on the Bro side somehow?
Are you positive that you're running all of the same scripts that you were and that you're using Broccoli from Bro 2.3? I'm not sure off the top of my head if there were any compatibility changes between the two releases or not, but it's certainly possible.
.Seth