Removing IP from Intel Framework?

All,

I'm working with the intel framework and enjoying it, but have hit a bit of a problem: I can successfully add new IPs to watchlists in the framework, but I can't remove them without restarting bro. I'd like to be able to do this to handle false-positives, for example.

The fact that new watchlist entries are flagged says to me that I'm doing the "create the file then move it into place" bit properly...I don't know what's up with removing entries, though.

I'm running bro 2.3 (the 06/16/14 release), and am invoking the intel framework like this:

@load frameworks/intel/seen
@load frameworks/intel/do_notice

redef Intel::read_files += {
  "/opt/bro/etc/internalList.dat",
};

internalList.dat looks like:

#fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in
targetDomain.blah Intel::DOMAIN internal_monitoring https://internalsite/campaign?arg1=text&arg2=some%20more%20text T -

Any ideas?

Thanks.

Aaron

Unfortunately there is no way to remove them without restarting.

I'm hoping that I can get a repository up on github today/tonight that makes your statement incorrect. :slight_smile:

  .Seth

Thats awesome Seth!

Thanks

Tom

"Life is too short to spend time with people who suck the happy out of you."

https://github.com/sethhall/intel-ext

This repository adds two features.
  - You can extend your intel log (now named intel_ext.log).
  - You can whitelist items.

These features will likely be integrated into Bro at a future date. I'm trying to use this ext repository as a way to vet features for the intel framework before integrating them right into the main distribution.

If you want to start whitelisting intel items at runtime, you should create a new intel file with an extra "meta.whitelist" field and set the field value to "T" (there is a test that shows this). As you add elements to this intel file, those items won't show up in your intel_ext.log.

The intel file will look something like this...

#fields indicator indicator_type meta.source meta.whitelist
bro.org Intel::DOMAIN my_whitelist T

You should probably maintain this as a separate file and make sure that you are giving the source as something distinct from where the data comes from originally (it's "my_whitelist" in my example).

Have fun! :slight_smile:

  .Seth

W00t, thanks a lot, testing ASAP.

Just to clarify a couple things ...

Do in-line indicator changes require a restart? That is, if my intel
file is deployed with indicator blah.org and a whitelist value of F,
then later I update that value to T, do I need to restart for that
change to be picked up? IIRC you still would need to restart for the
value change to be read.

The whitelisting also wouldn't decrease any processing requirements of
the Intel framework since the initial indicator match is still
occurring, right?

Josh

Do in-line indicator changes require a restart? That is, if my intel
file is deployed with indicator blah.org and a whitelist value of F,
then later I update that value to T, do I need to restart for that
change to be picked up? IIRC you still would need to restart for the
value change to be read.


I wouldn't recommend setting a whitelist value in your normal intel datasets. I would maintain it as a separate file as I recommended in my previous email.

The whitelisting also wouldn't decrease any processing requirements of
the Intel framework since the initial indicator match is still
occurring, right?

Having fewer items being matched really doesn't change your processing time overhead so there isn't really an optimization to be made there. It primarily just uses less memory at runtime but you wouldn't notice that either unless you have some sort of monstrous whitelist file. The only case where I could see it really helping would be if you are having a really huge number of hits, but I still suspect most people wouldn't notice.

  .Seth