Question about how to transfer bro data to java

Hi all,

I have a question about transferring data in Bro script to java. I currently have a tree-like data structure kept by multi-level tables in Bro script. I would like to transfer this data structure to java environment for further use (maybe using multi-level hashmaps in java to keep it). What is the most efficient way to do this?

The way I can come up with is to simply traverse the tree and write all the data in the tree in some certain formats to a text file and read that file in java to construct the structure again from scratches. But this seems extremely inefficient. Do you know any better way to do this?

Thanks a lot for your answers.


That’s probably not a bad way to go. No matter what you do you’re going to end up with Bro serializing and then your java code deserializing all of your data.


Just a few more thoughts, for what they're worth. Might be a reason the following are not good ideas: haven't thought through them too much. Just listing the first few things that come to mind.

One alternative might be to build a plugin to define a bro script method that would serialize the table data into something Java could understand, send it to Java somehow, and then have Java deserialize it on the receiving end. If you're looking for simplicity, I think JSON would definitely be easy to deserialize into nested HashMaps on the Java end, and would *probably* be pretty straightforward to serialize on the Bro end. That's not going to be the fastest format to work with, however.

Something that might be even faster (and harder) would be to build a shared memory region, then build a plugin method that would serialize the data into a form that made sense for Java, then bulk copy the table data into that shared memory space. From there, the Java application could map the shared memory segment into its own space directly, so there wouldn't need to be a deserialization step involved. The synchronization could be a challenge, though, I think. Also, pulling tabular data from Java mapped ByteBuffer objects (or equivalent) could be a challenge, depending on the type of data contained in the table.

A slightly more complex twist might be to extend bro's tables to support being backed by shared memory, the idea being that they could be mapped into other processes directly. I do think this would be *very* hard to get exactly right ... and there would still be some additional overhead in the form of copying / synchronization required to make this kind of IPC work in the first place. It's hard to say exactly how well (or not) this would perform, I think.

If the Java environment can reasonably run in the bro process space, then another option might be to launch a JVM from within bro. Write a script extension that would write the table data into a queue, and let a separate Java thread read / process the data being fed to it through said queue.

Note that, last I knew, Java doesn't support shared memory directly ... [1] and [2] might be interesting reading that would be related to this subject and offer practicable workarounds, however.

If the intermediate file option is the most attractive, might consider looking into trying to write to / read from a RAM disk (if that hasn't been done already). The performance *could* be better ... or not.

Anyway, hope something in there is interesting / useful, and good luck :slight_smile:



Hi Gilbert and Seth,

Thanks a lot for so many thoughts. I guess I will start with writing the data into a file and see how it works.