Help with Bro & ES

Zeek
1

I recently installed Bro, and I am trying to get it to work with elastic search (with Kibana as a front end.) I have alerts getting to ES and it shows up in Kibana, but it is a mix of unintelligible json messages. For example, some don’t have timestamps:

{
"_index": “bro-201510191500”,
"_type": “loaded_scripts”,
"_id": “AVCBw07WiyISA4W_6X0j”,
"_score": 1,
"_source": {
"name": " /usr/local/bro/share/bro/base/bif/bro.bif.bro"
}
}

Can anyone provide guidance, or suggest resources on organizing and sorting alerts/messages such that I can use it with ES/Kibana? I am not sure if I missed something in configuration and I am having a tough time finding resources online for further suggestions.

Thanks!

2

I recently installed Bro, and I am trying to get it to work with elastic search (with Kibana as a front end.) I have alerts getting to ES and it shows up in Kibana, but it is a mix of unintelligible json messages. For example, some don't have timestamps:

...

  "_type": "loaded_scripts",
  
The loaded_scripts.log is 'special' and does not have timestamps. How do entries from things like the conn.log or http.log look?

3

I mean… I think they look ok. Again, I understand that I have to learn how to organize the information in such a way that will make sense. This is an example of a conn message:

{
"_index": “bro-201510191500”,
"_type": “conn”,
"_id": “AVCBxqIWiyISA4W_6X6I”,
"_score": 1,
"_source": {
"ts": 1445286221580,
"uid": “CG7qWz2Xgs7J8LcO5d”,
“id.orig_h”: “..9.119”,
"id.orig_p": 123,
“id.resp_h”: “..1.3”,
"id.resp_p": 123,
"proto": “udp”,
"duration": 0.0002,
"orig_bytes": 0,
"resp_bytes": 48,
"conn_state": “SHR”,
"local_orig": false,
"local_resp": false,
"missed_bytes": 0,
"history": “Cd”,
"orig_pkts": 0,
"orig_ip_bytes": 0,
"resp_pkts": 1,
"resp_ip_bytes": 76,
"tunnel_parents": []
}
}

4

I’m working on elastic for a while
I changed a bit on the elasticsearch source,recompile after doing this.
sed -i “s/JSON::TS_MILLIS/JSON::TS_ISO8601/g” bro/aux/plugins/elasticsearch/src/ElasticSearch.cc

my conn.log looks like this in kibana check ts difference

{
“_index”: “bro-201509160700”,
“_type”: “conn”,
“_id”: “AVCLfROKixyabuRJCOlt”,
“_score”: null,
“_source”: {
“_timestamp”: 1442388234879,
“ts”: “2015-09-16T07:16:54.185442Z”,
“uid”: “Cv7R6a19zHzfu1H6U4”,
“id.orig_h”: “192.168.1.122”,
“id.orig_p”: 49428,
“id.resp_h”: “192.168.102.97”,
“id.resp_p”: 514,
“proto”: “udp”,
“duration”: 360.241984,
“orig_bytes”: 32096,
“resp_bytes”: 0,
“conn_state”: “S0”,
“missed_bytes”: 0,
“history”: “D”,
“orig_pkts”: 191,
“orig_ip_bytes”: 37444,
“resp_pkts”: 0,
“resp_ip_bytes”: 0,
“tunnel_parents”: []
}

The next chalenge are coordinates from geoip …

5

Justin,

I understand that “loaded scripts” is special, but it is creating issues with the index because there are no timestamps. Is there a recommended method for handling these messages?

6

The easiest way might be to just disable the loaded scripts log by adding this to local.bro:

event bro_init()
{
Log::disable_stream(LoadedScripts::LOG);
}

7

One doesn't need to disable it entirely, you can just not send it to ES by using

redef LogElasticSearch::excluded_log_ids += {LoadedScripts::LOG};

8

Thanks Mike and Justin,

Please excuse my ignorance… but what are the implications of doing this? I would prefer Justin’s method just in case I need to dive in… but could someone explain the significance of loaded_scripts?

9

Open it up and look at it.. It's literally just a log file created when bro starts that lists out all the scripts you have loaded. Other than for occasional troubleshooting it has no value whatsoever.