Issues with intel framework

Zeek
1

Hi all,

I’m running Bro on a fairly new distributed security onion setup - we have 3 sensors running bro in our environment. We’re running bro version 2.4.1. I’ve been wanting to add Critical Stack’s intel agent into our stack, but wanted to test how the intel framework works before we set it all up. To test the framework, I added a test domain (www.reddit.com) and two test IP addresses to intel.dat. I also downloaded and formatted a list of known malicious domains using the instructions found here: pintumbler.org - Bro Agent for Sguil - Now supports Intel.log, and saved this file in $bropath/share/bro/intel/intel_domans.dat. To test this list, I appended an entry for www.linux.com. All these files are tab delimited. We’re having two large reliability issues with the intel framework:

1 - My rule for the reddit.com domain in intel.dat fires sporadically, and it seems like only for certain subnets / end users. I can not get that alert to trip for me by browsing to reddit.com, despite seeing these connections in http.log and conn.log. The rules for Intel::ADDR in intel.dat never fire, even though we do see connections to those addresses in conn.log. Where can I look for what may be causing this unreliable intel alerting? Obviously intel.dat is loaded correctly as reddit.com generates intel hits for *some* users. I haven’t done anything to whitelist IP’s or subnets from alerting (I don't even know where to do this).

2 - This could be the exact same problem as #1, but I don’t seem to be getting any alerts from my intel_domains.dat file that I created. I tested this by adding www.linux.com as an intel rule, and it could just be that none of my user base that seems to be capable of generating alerts is visiting linux.com. I want to verify that I loaded this new file correctly.

The attached text file contains redacted information from our bro logs. I have my IP, and other endpoints that either have or should be generating alerts, and our proxy IP defined at the top. The first section should demonstrate that my IP is not generating intel hits even though the logs are present, and that others are generating intel hits. The second section is evidence of the Intel::ADDR rules that fail to fire at all. The third section is the __load__.bro file which I assume is all I need to modify in order to load my new intel_domains.dat file, created following the link above.

Please let me know if any more specific information could help pinpoint this issue and I’d be happy to provide! I’m excited to use Bro to identify (potential) compromise and hunt for other interesting things in my environment, but don’t want to jump the gun and implement intel when it doesn’t seem to be working reliably for us.

Thanks,

James Gordon

Intel Issues - Redacted.txt (7.61 KB)

2

Hi all,

I’m running Bro on a fairly new distributed security onion setup - we have 3 sensors running bro in our environment. We’re running bro version 2.4.1. I’ve been wanting to add Critical Stack’s intel agent into our stack, but wanted to test how the intel framework works before we set it all up. To test the framework, I added a test domain (www.reddit.com) and two test IP addresses to intel.dat. I also downloaded and formatted a list of known malicious domains using the instructions found here: http://www.pintumbler.org/words/broagentforsguil-nowsupportsintellog, and saved this file in $bropath/share/bro/intel/intel_domans.dat. To test this list, I appended an entry for www.linux.com. All these files are tab delimited. We’re having two large reliability issues with the intel framework:

1 - My rule for the reddit.com domain in intel.dat fires sporadically, and it seems like only for certain subnets / end users. I can not get that alert to trip for me by browsing to reddit.com, despite seeing these connections in http.log and conn.log.

The good news is that bro appears to be working properly, but intel is just not doing what you want (which is much easier to fix than bro not working properly!)

It looks like http requests that are not matching are the proxy requests, correct? There is a small discrepancy in the code that logs http requests and the code that feeds into the intel system:

The logging code does this:

event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=5
    {
    ...
        else if ( name == "HOST" )
            # The split is done to remove the occasional port value that shows up here.
            c$http$host = split_string1(value, /:/)[0];

While the intel code uses 'value' as is without stripping the port. If your client is sending the host header as www.reddit.com:443 that would cause this to not match.

If you add an entry to your intel file for www.reddit.com:443 does that fix that problem?

If it does, it's a simple fix to intel/seen/http-headers.bro to remove the port as well. I filed an issue to look into this:

https://bro-tracker.atlassian.net/browse/BIT-1695

The rules for Intel::ADDR in intel.dat never fire, even though we do see connections to those addresses in conn.log.

The default behavior for connections in intel is that it only alerts for established connections. The two log entries you have there are failed connections. You can easily have it alert on failed inbound and failed outbound connections if you wanted though. I thought we had a script for that somewhere.. It would basically be the same as the ./policy/frameworks/intel/seen/conn-established.bro but using connection_attempt instead of connection_established.. something like

event connection_attempt(c: connection)
  {
  Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
  Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
  }

Or maybe just only one of orig/resp.. it likely depends on exactly what you care about :slight_smile:

3

Thanks Justin!

Adding an entry for www.reddit.com:443 seemed to do the trick. I’ll experiment with removing the port from the http-headers.bro file in a test environment so that we don’t have to add duplicate entries for all https sites. Unfortunately, I added another entry in the intel_domains.dat file for www.linux.com:443, and that still is not firing intel hits. I have '@load intel' in my local.bro file, and I modified intel’s __load__.bro to include the new intel_domains.dat (as found in the original attachment). Is that all that should be necessary to load a new .dat into the intel framework?

Also, thanks for explaining the way connections are alerted on! I see value in alerting on failed outbound connections that we’re suspicious about.
This is probably a dumb question, but should I replace $bropath/policy/frameworks/intel/seen/conn-established.bro with a conn-attempted.bro, or will it be necessary to use both if I want to be alerted on both successful and attempted connections?

Thanks again,

James Gordon

4

If I recall, the Intel framework is based on a seen script. You could easily draft your own extension script to strip port numbers, paths, or schema off HTTP Host headers using the url decompose script.

Examples of Intel extensions are floating around on GitHub.

-AK

5

Thanks Justin!

Adding an entry for www.reddit.com:443 seemed to do the trick. I’ll experiment with removing the port from the http-headers.bro file in a test environment so that we don’t have to add duplicate entries for all https sites. Unfortunately, I added another entry in the intel_domains.dat file for www.linux.com:443, and that still is not firing intel hits.

Is www.linux.com or www.linux.com:443 showing up anywhere? It should be in at least some of the dns/http/ssl logs

I have '@load intel' in my local.bro file, and I modified intel’s __load__.bro to include the new intel_domains.dat (as found in the original attachment). Is that all that should be necessary to load a new .dat into the intel framework?

Don't modify any of the installed scripts. If you want to load an additional intel file use

redef Intel::read_files += {
  "/some/filename.dat"
};

in your local.bro or another script that you load from your local.bro

To avoid hardcoding the full path you can use

redef Intel::read_files += {
  fmt("%s/filename.dat", @DIR)
};

Also, thanks for explaining the way connections are alerted on! I see value in alerting on failed outbound connections that we’re suspicious about.
This is probably a dumb question, but should I replace $bropath/policy/frameworks/intel/seen/conn-established.bro with a conn-attempted.bro, or will it be necessary to use both if I want to be alerted on both successful and attempted connections?

Thanks again,

James Gordon

You'll need both. Create a intel-conn-attempted.bro file next to your local.bro and include the code there.

6

You'll need both. Create a intel-conn-attempted.bro file next to your local.bro and include the code there.

I guess this use case might be quite common. I would vote for adding a
conn-attempted.bro to the seen scripts but require explicit loading.

Jan