printing stream columns

Zeek
1

I’m trying to print the record type for each log stream at startup. Something like:

for ( id in Log::active_streams ) {
local stream = Log::active_streams[id];
print stream$path, stream$columns;

}

doesn’t work because $columns is a record type, and gets stringified “”.

Is there a way to do this in zeek script?

Thanks,
Henri

2

Hi Henri,

Great question.
The logging framework is extremely flexible and allows for log stream columns to dynamically change during run time. This means at startup, the bro_init() event, Zeek may not know all the columns of all the logs. Here’s a script I wrote for you which sort of answers your question. If you have more questions about it, just reach back out to the list.

-AK

function pfunk(rec: any): bool {
print type_name(rec);
return T;
}

event bro_init() {
for (id in Log::active_streams) {
for (fname in Log::get_filter_names(id)) {
local filter: Log::Filter;
filter = Log::get_filter(id, fname);
filter$pred = pfunk;
Log::add_filter(id, filter);
}
}
}

3

Zeek 3.0 should give better descriptions for types. This was the
relevant patch which is not in any 2.6.x version:

  https://github.com/bro/bro/commit/1f450c05102be6dd7ebcc2c5901d5a3a231cd675

This script may also help demonstrate things related to what you're
trying to do:

  https://gist.github.com/jsiwek/f843b3321f4227b6ec32d110424ebf70

It prints field descriptions of all logs either to stdout or a CSV
file. Example command:

  ZEEK_ALLOW_INIT_ERRORS=1 zeek print-log-info.bro PrintLogs::csv=F

Sample of output:

known_hosts.log | Hosts with complete TCP handshakes
  ts: time - The timestamp at which the host was detected.
  host: addr - The address that was detected originating or responding
to a TCP connection.

- Jon

4

Thanks Jon and Anthony for the quick responses! print-log-info.bro looks promising for what I’m trying to do.

5

I’ve been playing around with Jon’s script and am getting close to what I want, but still have one outstanding issue related to nested records. Currently they show up as a single feed with a type “record foo” (such as “record conn_id” or “record FTP::ExpectedDataChannel”).

I’d like to be able to peek into nested records to get the inner fields that will show up in the logs. It doesn’t seem like there’s a way to do record introspection given a string representation of the record type name, but if I’d be delighted to be told I’m missing something.

Thanks for any pointers!

6

No, didn't look like there was a way to do that, but I've made a
PR/patch that should make recursive introspection possible via
something like `record_fields("conn_id")` for any arbitrary record
type name:

https://github.com/zeek/zeek/pull/675

- Jon

7

Ooh that looks great. Thanks!

8

Now that this patch is merged (thanks again) I’ve upgraded my Zeek script and the record_fields changes work great.

I still have one outstanding issue which is that for a container type, record_field$type_name is just the container name (such as “vector” or “set”). I don’t see a way to get the type of the container elements from zeek script, but once again would be delighted to be corrected.

And if there’s currently no way, I’m happy to put up a PR, but I could use some guidance on how to expose this in Zeek (e.g. a new field on record_field?).

Thanks,
Henri

9

Would be great if you want to try making a PR. The first way to do it
that comes to mind is just alter that "record_field$type_name" to
better describe containers in a format like "vector of XXX",
"set[XXX]" or "table[XXX] of YYY". This should be the relevant code
to modify:

https://github.com/zeek/zeek/blob/b86a8acc2b84089efbbe51216a4f4d1d57a4f430/src/Type.cc#L845-L850

- Jon

10

Cool, that’s exactly the place i was looking (wasn’t sure if changing this might break existing scripts… but since this is all quite new, probably best to make the change soon). I’ll get the PR up soon.