Bro -> Elasticsearch -> Kibana4beta -> GeoLocation

Zeek
1

To be more clear
I use this bro script for geo location

##! Add geo_location for the originator and responder of a connection
##! to the connection logs.

module Conn;

export
{
redef record Conn::Info +=
{
orig_loc: geo_location &optional &log;
resp_loc: geo_location &optional &log;
};
}

event connection_state_remove(c: connection)
{
local orig_loc = lookup_location(c$id$orig_h);
if (orig_loc?$longitude && orig_loc?$latitude)
c$conn$orig_loc= orig_loc;
local resp_loc = lookup_location(c$id$resp_h);
if (resp_loc?$longitude && resp_loc?$latitude)
c$conn$resp_loc= resp_loc;
}

Produces this output in json example

{
“ts”: “2013-04-26T12:12:02.341149Z”,
“uid”: “C0GaiXWHKY4Uj0qke”,
“id.orig_h”: “83.161.249.149”,
“id.orig_p”: 49318,
“id.resp_h”: “68.232.35.139”,
“id.resp_p”: 443,
“proto”: “tcp”,
“conn_state”: “SHR”,
“missed_bytes”: 0,
“history”: “f”,
“orig_pkts”: 0,
“orig_ip_bytes”: 0,
“resp_pkts”: 1,
“resp_ip_bytes”: 67,
“tunnel_parents”: [],
“orig_loc.country_code”: “NL”,
“orig_loc.latitude”: 52.366699,
“orig_loc.longitude”: 4.9,
“resp_loc.country_code”: “US”,
“resp_loc.region”: “CA”,
“resp_loc.city”: “Santa Monica”,
“resp_loc.latitude”: 34.011902,
“resp_loc.longitude”: -118.468201
}

According to the elasticsearch documentation i need an output like

“resp_loc”: {
“lat”: 52.366699,
“long”:4.9
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html

2

You should be able to customize how ElasticSearch stores the data via an explicit mapping: https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html (i.e., you would add lat and long as objects under the resp_loc object).

Alternatively, if all you want is to present a geo-point, there are some other ways to represent that: https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

–Vlad

3

Hi Vlad,

I’m interested in how I can present the json the way elastic wants it.

Regards,

Daniel

4

I’m not sure if this is what you are looking for but just in case it’s useful.

BRO_connlog

if [type] == “BRO_connlog” {

grok {

match => [ “message”, “(?(.?))\t(?(.?))\t(?<id.orig_h>(.?))\t(?<id.orig_p>(.?))\t(?<id.resp_h>(.?))\t(?<id.resp_p>(.?))\t(?(.?))\t(?(.?))\t(?(.?))\t(?<orig_bytes>(.?))\t(?<resp_bytes>(.?))\t(?<conn_state>(.?))\t(?<local_orig>(.?))\t(?<missed_bytes>(.?))\t(?(.?))\t(?<orig_pkts>(.?))\t(?<orig_ip_bytes>(.?))\t(?<resp_pkts>(.?))\t(?<resp_ip_bytes>(.?))\t(?<tunnel_parents>(.))\t(?<orig_cc>(.))\t(?<resp_cc>(.))\t(?<orig_longitude>(.))\t(?<orig_latitude>(.))\t(?<resp_longitude>(.))\t(?<resp_latitude>(.))” ]

}

mutate {

‘coords’ will be kept, ‘tmplat’ is temporary.

Both of these new fields are strings.

add_field => [ “resp_coords”, “%{resp_longitude}”,

“tmplat”, “%{resp_latitude}” ]

}

mutate {

Merge ‘tmplat’ into ‘coords’

merge => [ “resp_coords”, “tmplat” ]

}

mutate {

Convert our new array of strings back to float

convert => [ “resp_coords”, “float” ]

Delete our temporary latitude field

remove => [ “tmplat” ]

}

}

I just used the resp but could duplicate for orig.

Kind regards,
Andy
Andrew.Ratcliffe@NSWCSystems.co.uk
CISSP, GCIA, GCIH, GPEN, GWAPT, CSTA, CSTP, CWSA, GCFE
Blog.InfoSecMatters.net

5

I use the elasticsearch plugin in bro. I know logstash works fine but its
very cpu intensive. Thanx anyway.

6

Technically it can be done, but it would require changes to the JSON formatter (in the core). This is actually a pretty reasonable request (and I like the idea a lot!). It might not be too much work to implement it, it just needs to be done.

  .Seth

7

The funny thing is that elasticsearch stores the data internal
like the bro output is.

quote from the object document

Internally, this document is indexed as a simple, flat list of key-value pairs, something like this:

{
  "region":             "US",
  "manager.age":        30,
  "manager.name.first": "John",
  "manager.name.last":  "Smith"
}

Maybe this is an elasticsearch problem …
To make it all work ElasticSearch.cc has to change to do
the geopoint mapping. And maybe stop analyse strings like
user_agent to avoid chopping of the result in the first word.
This could be solved by using url formatted strings you want
to show in graphs etc (no spaces).
The last thing is some naming collisions elasticsearch is
confused about, like version in ssh & socks, but thats easy
to change in their main scripts.

Daniel

8

Hi All,

The problem was solved like this.
The geoip script adds the geo_location

With the proper mapping kibana shows
geoip data ;).
Check
https://hub.docker.com/r/danielguerra/bro-debian-elasticsearch/

##! Add geo_location for the originator and responder of a connection
##! to the connection logs.

module Conn;

export
{
redef record Conn::Info +=
{
geo_location: string &optional &log;
};
}

event connection_state_remove(c: connection)
{
local resp_loc = lookup_location(c$id$resp_h);
if (resp_loc?$longitude && resp_loc?$latitude)
#geo location is just a cat lat,long
c$conn$geo_location= cat(resp_loc$latitude,",",resp_loc$longitude);
}

add the mapping before reading data with

curl -XPUT elasticsearch:9200/_template/fixstrings_bro -d '{
“template”: “bro-*”,
“mappings” : {

“conn” : {
“geo_location” : { “type” : "geo_point” }
}
}}’

9

Hi,

The docker image is currently not working (should have checked first). It works fine on my laptopbut docker has some troubles with it. You can build the dockerfile yourself
https://github.com/danielguerra69/bro-debian-elasticsearch