Intel framework troubleshooting on Bro 2.5

Are there any tricks to use when debugging the Intel framework that would show parsing errors ?

The problem we have is when combining multiple intel files one bad file seems to corrupt the entire lot.

http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html

Following that guide works fine, there are a number of intel hits on Tor activity within minutes of restarting Bro. When we add in the giant list of intel from CriticalStack the Tor intel hits no longer trigger which suggests an issue with that file. Commenting out the tor.intel file sort of narrows it down to the CriticalStack file but it also suggests that a bad intel file somehow corrupts previously read files. (they both use a list of Tor exit nodes from the Suricata project.)

I’ve checked the files for correct headers, no spaces, and tab formatting all which seem to be OK.

#---- from local.bro file ----#

@load frameworks/intel/seen
@load frameworks/intel/do_notice

Load custom intel feed

@load local-intel.bro

#---- local-intel.bro file ------#

[bro@mgr /opt/bro]$ less /opt/bro/share/bro/site/local-intel.bro
const feed_directory = “/opt/bro/feeds”;

redef Intel::read_files += {

feed_directory + “/tor.intel”,

feed_directory + “/critical-stack/master-public.bro.dat”
};

-Hovsep

First step would be to check reporter.log and stderr.log on the manager.

Nothing stands out. Looking at base/frameworks/intel/input.bro is there a way to hook Input::add_event and have those events written to a log file ? I tried moving a new intel file into place but didn’t notice anything in reporter.log or stderr.

ex: cp master-public.bro.dat master-public.bro.dat.new && mv master-public.bro.dat.new master-public.bro.dat

Nothing stands out. Looking at base/frameworks/intel/input.bro is there a
way to hook Input::add_event and have those events written to a log file ?

You could use the Intel::read_entry event. For validation of the files
have a look at GitHub - mixmodeai/bro_intel_linter: Bro Intel Feed Linter.

Can you reproduce the issue running a standalone deployment or against a
pcap and is that issue new in Bro 2.5?

Jan

Thanks, that linter is finding errors. I just started using CriticalStack with Bro 2.5 so I can’t comment on prior issues.

If the linter is working as expected then it appears the problem is with a few URIs from PhishTank with odd URL encoding, maybe they are mistakenly being interpreted as tabs during parsing or corrupting some internal state within Bro.

bro@mgr:/opt/bro/feeds % bro_intel_linter/intel_linter.py -f master-public.bro.dat
WARNING: Line 1263 - Invalid entry “bjcurio.com/js/index.htm?\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82???\xc3\x83?\xc3\x82?\xc3\x83?\xc3\x82??%” for column “indicator”
WARNING: Line 4501 - Invalid entry “generalfil.es/download/gs4eb28030h17i0/windows%20live%20messenger%208.5%20%20%20patch%20anti-atualizaai\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbdi\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbdai\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbdi\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbdo%20%20%20messenger%20plus!%20liv.html” for column “indicator”
WARNING: Line 12438 - Invalid entry “www.alhotocaia.com.br/Templates/11632/simplestyle_5/style/-6327-40825785664-3357953/index.html?A?A?A?%20I?A?A?A?A\xef\xbf\xbd\xef\xbf\xbd1A?A\xef\xbf\xbd\xef\xbf\xbdA?A??” for column “indicator”
ERROR: Line 13902 - Indicator type “Intel::ADDR” possible issue with indicator: “2400:8901::f03c:91ff:feb0:bdb0”
ERROR: Line 13902 - Details - Invalid IP address

If the linter is working as expected then it appears the problem is with a
few URIs from PhishTank with odd URL encoding, maybe they are mistakenly
being interpreted as tabs during parsing or corrupting some internal state
within Bro.

Theoretically malformed items should not affect previously loaded intel
files. If you can reproduce this issue and provide something like a
minimal example it would be good to open a ticket.

Jan