A little more confusion with Intel

Zeek
1

So I’m testing something completely unrelated to this issue, but I’ve run into something interesting. First off following this works:

https://www.bro.org/current/solutions/intel/index.html

my test intel-1.bro:
@load frameworks/intel/seen

redef Intel::read_files += {
fmt("%s/intel-1.dat", @DIR)
};

my intel-1.dat file (whitespace=tab):
#fields indicator indicator_type meta.source
fetchback.com Intel::DOMAIN my_special_source
yahoo.com Intel::DOMAIN testdomain

I’ve carved out the dns request for fetchback.com from the exercise packet capture, which I’m including. Testing line below works just fine:

bro -C -r exercise-traffic-fetch-dns.pcap intel-1.bro

I see lot’s of good stuff:
conn.log
1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 udp dns 0.200354 31 99 SF - - 0 Dd 1 59 1 127 (empty)

dns.log
1258565309.806483 CVifWt1zc5YSG0Vhc9 192.168.1.103 53856 192.168.1.1 53 udp 4438 0.200354 fetchback.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 69.71.52.52 1800.000000 F

intel.log
1258565309.806483 CmeOAzpOmlw26nOEi 192.168.1.103 53856 192.168.1.1 53 fetchback.com Intel::DOMAIN DNS::IN_REQUEST bro Intel::DOMAIN my_special_source - - -

however running against the included yahoodns.pcap here’s what I get:
conn.log
1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp dns 0.003246 31 124 SF - - 0 Dd 1 59 1 152 (empty)

dns.log
1516289219.143906 CFXRMB4RJIFYSdw72a 192.168.1.2 62196 192.168.1.1 53 udp 3285 0.003246 www.yahoo.com 1 C_INTERNET 1 A 0 NOERROR F F TT 0 atsv2-fp.wg1.b.yahoo.com,98.138.252.38,98.138.252.39,98.139.180.180,206.190.39.43 1320.000000,39.000000,39.000000,39.000000,39.000000 F

and no intel.log. What’s different here? Would love to know what I’m missing…thank you.

James

yahoodns.pcap (295 Bytes)

exercise-traffic-fetch-dns.pcap (274 Bytes)

2

What do you have your local_nets set to?

3

In this particular test I haven’t set it for either run. Thanks Michael.

James

4

Maybe tab formatting in the intel.dat file?

I think you will get Reporter errors, so the first IOC works, but the second one does not because it cannot be parsed.

5

Thanks…both entries are tabbed formatted…still digging on my end via trace files.

James

6

www.yahoo.com is not yahoo.com

You need an intel::seen even that uses https://github.com/sethhall/domain-tld to get that to match. I thought someone wrote a package that did this, but apparently not.

7

Wait what? So…it looks like www.yahoo.com matches, but yahoo.com doesn’t?

That kinda nukes the whole match any bad host with domain :wink: Thank you can lend a hand Seth? Thanks.

James

8

So mystery #1 solved: Intel::DOMAIN != tld domain; cool Next up…with these same files only having this in the intel-1.dat file:

192.168.1.1 Intel::ADDR testip

the above is tabbed formatted correctly. Now…running against both pcaps I get no intel hits. Here is the only entries in the trace file that show Intel::ADDR:

}]’, tpe = ‘Input::EVENT_NEW’, item = ‘[indicator=192.168.1.1, indicator_type=Intel::ADDR, meta=[source=testip, desc=, url=]]’)
0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:469 function called: Intel::insert(item = ‘[indicator=192.168.1.1, indicator_type=Intel::ADDR, meta=[source=testip, desc=, url=]]’)
0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:400 Builtin Function called: to_lower(str = ‘192.168.1.1’)
0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:400 Function return: 192.168.1.1
0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:404 Builtin Function called: to_addr(ip = ‘192.168.1.1’)
0.000000 /opt/bro/share/bro/base/frameworks/intel/./main.bro:404 Function return: 192.168.1.1
0.000000 /opt/bro/share/bro/base/frameworks/input/./main.bro:248 event called: Input::end_of_data(name = ‘intel-/home/user/dev/bro/intel/./intel-1.dat’, source = ‘/home/user/dev/bro/intel/./intel-1.dat’)
0.000000 /opt/bro/share/bro/base/utils/./exec.bro:102 Builtin Function called: split_string1(str = ‘intel-/home/user/dev/bro/intel/./intel-1.dat’, re = ‘/^?(_)$?/’)
0.000000 /opt/bro/share/bro/base/utils/./exec.bro:102 Function return: [intel-/home/user/dev/bro/intel/./intel-1.dat]
1516289219.143906 /opt/bro/share/bro/base/misc/find-checksum-offloading.bro:62 event called: ChecksumOffloading::check()
1516289219.143906 /opt/bro/share/bro/base/misc/find-checksum-offloading.bro:29 Builtin Function called: get_net_stats()
1516289219.143906 /opt/bro/share/bro/base/misc/find-checksum-offloading.bro:29 Function return: [pkts_recvd=1, pkts_dropped=0, pkts_link=0, bytes_recvd=73]
1516289219.143906 /opt/bro/share/bro/base/frameworks/packet-filter/./main.bro:157 event called: filter_change_tracking()
1516289219.143906 /opt/bro/share/bro/base/bif/event.bif.bro:88 event called: new_connection(c = '[id=[orig_h=192.168.1.2, orig_p=62196/udp, resp_h=192.168.1.1, resp_p=53/udp], orig=[size=31, state=1, num_pkts=0, num_bytes_ip=0, flow_label=0, l2_addr=48:4d:7e:a3:53:5e], resp=[size=0, state=0, num_pkts=0, num_bytes_ip=0, flow_label=0, l2_addr=00:08:e3:ff:fc:04], start_time=1516289219.143906, duration=0.0, service={

Here too, is there something I’m missing? In testing a different packet captures using TCP, I get intel…so does the Intel framework not support UDP? Thank you.

James

2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png

9

The intel framework doesn't know anything about tcp or udp. The default scripts for connections only alert on tcp connections though:

https://github.com/bro/bro/blob/master/scripts/policy/frameworks/intel/seen/conn-established.bro

10

Ah....Ok thanks again Justin. Seth should I put in a feature request for both TLD and UDP for the Intel framework? Thanks.

James

11

That's probably something that can be addressed with a package. In general you can have a look at https://github.com/bro/bro/tree/master/scripts/policy/frameworks/intel/seen to get an idea of how the intel framework gathers its information.

Jan

12

Yes this would be a nice to have.

13

Yes this would be a nice to have.

I put together a POC for effective TLDs but haven't tested deploying.
During the weekend I should be able to polish it a bit. If someone
already wants to give it a try:
bro-pkg install GitHub - J-Gras/intel-seen-more: Additional seen-triggers for Zeek's intelligence framework.

Jan

14

That looks just like what I had in mind..

It makes sense that the type would be different, but I could see some people expecting it to just use the normal Intel::DOMAIN so
existing feeds match.

The more I think about this, there's also the similar calls to seen() for via HTTP::IN_HOST_HEADER, SSL::IN_SERVER_NAME, and X509::IN_CERT

Maybe the intel framework itself needs to have an option to use the effective TLD when looking up Intel::DOMAINs inside of seen()

15

Hi James! :slight_smile:

Right now the Intel framework is only for doing complete strings matches for all of the string types (which Intel::DOMAIN is) so you don’t get the substring matching like you want. Robin and I talked about this a couple of years ago as something that we wanted to address in Bro and Robin did a small prototype of a library that would make it possible by globbing. The idea was that you’d be able to have intelligence items that looked like this… “yahoo.com" or "www..yahoo.*”. The initial implementation didn’t perform acceptably well and we haven’t had time to get back to that work yet.

Right now if you are interested in looking for “www.yahoo.com” you will have to insert that specifically as an intelligence item. I’m not sure that the example you’ve given is something that people encounter in typical operational usage (although if I’m wrong, someone please let me know!).

.Seth

2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png

16

It makes sense that the type would be different, but I could see some people expecting it to just use the normal Intel::DOMAIN so
existing feeds match.

While that's certainly true, a couple of people might already rely on Intel::DOMAIN matching the complete domain.

The more I think about this, there's also the similar calls to seen() for via HTTP::IN_HOST_HEADER, SSL::IN_SERVER_NAME, and X509::IN_CERT

Yep, I will just add corresponding scripts to the package.

Maybe the intel framework itself needs to have an option to use the effective TLD when looking up Intel::DOMAINs inside of seen()

In that case the framework should report both: The effective and the complete domain. However, using a separate type would be more flexible as users could decide case by case or even add both.

Given that the effective_domain function is already available as a package, I would vote for an additional package. In theory even the intel framework itself could be made a package.

Jan

17

Hi Seth,

It’s actually the inverse of what I’m seeing. In my tests if I have Intel::DOMAIN yahoo.com and I did a "dig www.yahoo.com", the domain intel would not match because the dns request was for “www.yahoo.com”, not yahoo.com. Does that make sense? Thank you.

James

2018-01-18 10_35_16-fetchtrace.txt - Visual Studio Code.png

18

Yeah, if we had a more comprehensive matcher for the intel framework then you'd have a lot of options open for you. I suppose that my main point was that at the moment you will have to just include the exact domain that you want to match on.

Do you have a large list where you'd like to watch for any hits on the effective second level domain like you're describing here?

   .Seth

19

Thanks Seth,

I basically modified this for bro use:

https://isc.sans.edu/forums/diary/Tracking+Newly+Registered+Domains/23127/

It's basically a list of domain names that have been newly registered. Does that help?

James

20

Ahh! I understand your use case better now. We could use the effective-tld package to create a new "seen" injector for the intel framework that pokes effective TLDs into the intel framework. I don't know what the overhead effects of this would be, but it might not be too bad.

   .Seth