Bro -> Elastic Search -> Kibana Issues | Default Analyzed Fields

Zeek
1

I am having an issue when trying to process Bro data through Elastic Search and Kibana.

Specifically, I am doing basic quantitative statistics such as pulling the Top 5 Originating IP’s addresses by id_orig_h. However, Elastic Search and Kibana breaks apart this field as shown below in the screenshot linked below:

http://i.imgur.com/m3BH6LP.png

Basically, for some strings, the default Elastic Search analyzer will segment them into different pieces based on the “.” character. For example, 130.85.12.20 will be broken apart into 130, 85, 12, and 80. This is because in the Core Type mappings, the not index attribute is not explicitly set to no (it defaults to yes). There is no way to adjust this for existing fields.

Here is the current mapping created by Bro:

“id.resp_h” : {
“type” : “string”
}

It should be:

“id.resp_h” : {
“type” : “string”
“index”: “not_analyzed”

}

Suggestions?

2

Real Quick Update:

My initial analysis was incorrect. IPv4 fields are processed correctly. The issue is with IPv6 fields. Same concept, different trigger. Here is a picture illustrating the problem:

http://i.imgur.com/pdxRbmX.png

3

Arash,
I imagine that is because elasticsearch currently does not support IPv6 fields.

4

Exactly, which is why I suggest disabling string analyzation in Elastic Search for this field (and all IP fields). Otherwise, the IPv6 entries are ruining the results.

5

Does anyone have any suggestions on working around this? I was thinking of using LogStash but I’m worried about scalability with that solution.

I understand Bro wasn’t explicitly made to interface with ElasticSearch and that this might be a process.

6

Thanks Jay, it worked.

I used index templates as suggested by Jay, https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html, to override the mapping for all string fields in Bro. I do not know the complete ramifications of this but it works as a bandaid fix for now.

I got the entire mapping by doing the following: curl -XGET ‘http://localhost:9200/bro-*/_mapping/?pretty=true’

Then I modified it via the index template linked here: http://pastebin.com/qRFCzaMT

Warning, Bro might change its ElasticSearch mapping / field names at any time and this index template should only be used as a reference. I fully expect the current ElasticSearch writer to change so to take full advantage of ElasticSearch and Kibana, it is very much a work in progress.