# Unique connection ID for bro \<-\> logging framework **URL:** \ **Category:** Development\ **Tags:** development\ **Created:** [December 4, 2010, 2:35am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740 "2010-12-04T02:35:34Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Gregor\_Maier](https://avatars.discourse-cdn.com/v4/letter/g/94ad74/32.png) [@Gregor\_Maier](https://community.zeek.org/u/Gregor_Maier)\ **Post date:** [December 4, 2010, 2:35am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/1 "2010-12-04T02:35:34Z")
Hi, I was wondering whether it would make sense to assign each connection an ID that's unique for this bro run. This ID can just be a 64-bit counter that gets incremented on every new connection. Why: If we add this ID to log outputs, it would be much easier to correlate activity across logs (e.g., find the connection in http.log, alarm.log, and conn.log, without having to match 5-tuples and timestamps) I think this would be a rather nice (and very easy to implement) feature. Cluster considerations: maybe add a nodeID or something to the connection ID. E.g., in the high-order 8 or 16 bits. Thoughts? Comments? cu Gregor ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [December 4, 2010, 2:42am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/2 "2010-12-04T02:42:32Z")
> I was wondering whether it would make sense to assign each connection an > ID that's unique for this bro run. This ID can just be a 64-bit counter > that gets incremented on every new connection. That's an interesting idea. > Why: If we add this ID to log outputs, it would be much easier to > correlate activity across logs (e.g., find the connection in http.log, > alarm.log, and conn.log, without having to match 5-tuples and timestamps) My only question is under what circumstance you do that activity correlation activity within a single connection? I'm unable to think of a single time when I've needed to do something like that where I wasn't able to just search for the single IP address that I was interested in because I was interested in anything that IP address was referenced in and not just that single connection.   .Seth ---
**Author:** ![Gregor\_Maier](https://avatars.discourse-cdn.com/v4/letter/g/94ad74/32.png) [@Gregor\_Maier](https://community.zeek.org/u/Gregor_Maier)\ **Post date:** [December 4, 2010, 3:04am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/3 "2010-12-04T03:04:32Z")
> > Why: If we add this ID to log outputs, it would be much easier to > > correlate activity across logs (e.g., find the connection in http.log, > > alarm.log, and conn.log, without having to match 5-tuples and timestamps) > > My only question is under what circumstance you do that activity correlation activity within a single connection? I'm unable to think of a single time when I've needed to do something like that where I wasn't able to just search for the single IP address that I was interested in because I was interested in anything that IP address was referenced in and not just that single connection. Some examples: \* I want to count the number of HTTP request per connection \* I do per connection stats (e.g., number of packets, number of   bytes, retransmissions, RTTs), store them in their own log files   and then want to correlate with the conn.log or the http.log \* Easier debugging / analysis:   I can just grep for the connectionID, instead of   having to map between different connection formattings (e.g., notices   have origIP:origPort -\> respIP:respPort but when I want to grep for   them in conn.log, I have to do some awk to get there) \* ... I guess I have more a measurement point of view here.... cu Gregor ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [December 4, 2010, 4:07am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/4 "2010-12-04T04:07:57Z")
> \* I want to count the number of HTTP request per connection Ah, ok. Now that you mention it, I have done searches for that before too. 🙂 > (e.g., notices > have origIP:origPort -\> respIP:respPort but when I want to grep for > them in conn.log, I have to do some awk to get there) If the logging framework proceeds in the direction that Robin and I have been outlining, most of this trouble will go away. > I guess I have more a measurement point of view here.... Yeah, makes sense. I just wasn't understanding that before. 🙂 Reading the things you need to do does remind me that Justin Azoff and I need to get back to the metrics framework we've been talking about. It could help you output logs with a lot of the measurement type data you are looking to get instead of having to do post-processing on the existing logs. Getting back to your question though, it's an interesting idea but I wonder if it will still be necessary once the "normal" logging output changes. At the very least, if you output tab separated value data, you should be able to do something like this.... cat whatever.log | grep "1.2.3.4\35231\4.3.2.1\80" The binary log output may make that even easier too.   .Seth ---
**Author:** ![Gregor\_Maier](https://avatars.discourse-cdn.com/v4/letter/g/94ad74/32.png) [@Gregor\_Maier](https://community.zeek.org/u/Gregor_Maier)\ **Post date:** [December 4, 2010, 6:53am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/5 "2010-12-04T06:53:48Z")
> Getting back to your question though, it's an interesting idea but I wonder if it will still be necessary once the "normal" logging output changes. At the very least, if you output tab separated value data, you should be able to do something like this.... > > cat whatever.log | grep "1.2.3.4\35231\4.3.2.1\80" In general yes, as long as the 5-tuple isn't reused. (I can basically do this right now, if I use awk to reorder the connection-tuple so that I can grep for it. Might thought was that having a single numeric ID might make life easier.) > The binary log output may make that even easier too. Being able to use grep, sed, awk, and co. is still very nice, so I'll probably end up using a binary to ascii converter quite frequently. cu gregor ---
**Author:** ![Gregor\_Maier](https://avatars.discourse-cdn.com/v4/letter/g/94ad74/32.png) [@Gregor\_Maier](https://community.zeek.org/u/Gregor_Maier)\ **Post date:** [December 6, 2010, 5:29pm UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/6 "2010-12-06T17:29:24Z")
BTW, the addl field in conn.log is sometimes used for something similar. E.g., http.bro will create a unique ID for each HTTP-session and put this session ID into the connections addl.... cu gregor ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [December 10, 2010, 10:32am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/7 "2010-12-10T10:32:31Z")
Two more thoughts on htis: - generally, I see that such an ID could be quite helpful. While with the new logging, filtering for 4-tuples should be quite easy, having a per-conn ID gets us more crisp semantics and generally simplifies things. - however, I think there'd need to be one more piece to that story: the IDs should be unique across Bro runs. Otherwise crunching information from a big log archive woulnd't be much better than it is today. But that would probably mean we'd need to go beyond 64-bit integers, perhaps to a string prefixed with something likely to be unique. Robin ---
**Author:** ![Gregor\_Maier](https://avatars.discourse-cdn.com/v4/letter/g/94ad74/32.png) [@Gregor\_Maier](https://community.zeek.org/u/Gregor_Maier)\ **Post date:** [December 10, 2010, 7:06pm UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/8 "2010-12-10T19:06:53Z")
> - however, I think there'd need to be one more piece to that story: > the IDs should be unique across Bro runs. Otherwise crunching > information from a big log archive woulnd't be much better than it > is today. But that would probably mean we'd need to go beyond > 64-bit integers, perhaps to a string prefixed with something likely > to be unique. We can probably keep a 64 bit counter internally and also add a bro\_instance\_ID, that's globally unique across Bro runs. For logging, we can then log the 64 bit counter and the instance\_ID, or concatenate the two (I would guess that the instance\_ID will be handy in other situations too). Doesn't the cluster already have/need something like that? In order to generate such an instance\_ID, we could: a) make sure it's truly globally unique, e.g., by using a    cryptographically secure, long (128 bit, maybe even 160 or more)    random number. Possibly from an entropy pool (can we use OpenSSL for    that?) b) the user supplies a "hostID", we can then add time and PID    and hash all that together to get the instance ID, e.g.,    md5(hostID + PID + gettimeofday())    (this should probably be fairly tolerant even if the hostID gets    reused across machines). cu Gregor ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [December 13, 2010, 8:50am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/9 "2010-12-13T08:50:54Z")
> We can probably keep a 64 bit counter internally and also add a > bro\_instance\_ID, that's globally unique across Bro runs. For logging, we > can then log the 64 bit counter and the instance\_ID, or concatenate the > two (I would guess that the instance\_ID will be handy in other > situations too). Doesn't the cluster already have/need something like that? There's a global peer\_description (string) that if set will be used as prefix for IDs in logs; see prefixed\_id() in bro.init. The cluster sets that differently for each a node. However, the cluster currently also doesn't give unique IDs across runs, just unique across nodes within a single run). > In order to generate such an instance\_ID, we could: My main concern is not wasting too many bytes for these IDs, as I imagine they would be included in pretty much every log entry. On the other hand, I don't think we need to be 100% sure that the IDs are unique as long as the probability of a collision is small. Seems that a single 64-bit int should be able to achieve that already if we hash all information in. > b) the user supplies a "hostID", we can then add time and PID >    and hash all that together to get the instance ID, e.g., >    md5(hostID + PID + gettimeofday()) I generally like this, and the hostID can be the peer\_description. But I think we can hash into 64-bit instead and probably take a simpler hash function as well. And then we can just add the 64-bit counter to that value. Robin ---
**Author:** ![Gregor\_Maier](https://avatars.discourse-cdn.com/v4/letter/g/94ad74/32.png) [@Gregor\_Maier](https://community.zeek.org/u/Gregor_Maier)\ **Post date:** [December 13, 2010, 7:29pm UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/10 "2010-12-13T19:29:41Z")
> > b) the user supplies a "hostID", we can then add time and PID > >    and hash all that together to get the instance ID, e.g., > >    md5(hostID + PID + gettimeofday()) > > I generally like this, and the hostID can be the peer\_description. > But I think we can hash into 64-bit instead and probably take a > simpler hash function as well. And then we can just add the 64-bit > counter to that value. I'd prefer to keep the counter and the runID separate. E.g., by making the runID n-bits and the counter 64-n bits and the OR-ing them together. OTOH, I don't think we need to worry too much about wasting bytes by using a say, 32 bit runID + 64 bit unique-ID. We now use ACSII for logging, if we move to binary (and possibly compression), then this will save \*way\* more memory than our ID will ever add. (we could also just add the run-id once per log-file or log-group inside a file, but might be too cumbersome) cu gregor ---
**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\ **Post date:** [December 17, 2010, 9:31am UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/11 "2010-12-17T09:31:28Z")
> I'd prefer to keep the counter and the runID separate. E.g., by making > the runID n-bits and the counter 64-n bits and the OR-ing them together. What's the advatnage of keeping them separate? > using a say, 32 bit runID + 64 bit unique-ID. We now use ACSII for > logging, if we move to binary (and possibly compression), then this will > save \*way\* more memory than our ID will ever add. Well, that's right, but still no excuse for waste if we don't need it. 🙂 (I'm also thingking about potentially piping logs into a DB and perhaps indexing on such an ID as a key.) Robin ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:39pm UTC](https://community.zeek.org/t/unique-connection-id-for-bro-logging-framework/1740/12 "2022-05-06T15:39:18Z")
Closed on May 6, 2022, 3:39pm UTC