Hi,
I have been using Bro’s intel framework to input my intelligence feed and get matches in intel.log. I seem to only be getting hits on domains but not IPs or URLs. I have also tried it on the mal-dnssearch feeds with the same results. Is there any particular reason why this would happen? How can I fix it?
Thanks!
Disha Bora
Associate Product Manager
iSIGHT Partners
Hi Disha,
Can you include your intel.dat file and the script you are using to load it please?
-AK
Hi,
for URLs there is an important detail I missed the first time, when I used the intel framework. The documentation says: Intel::URL - A complete URL without the prefix “http://”.
However, IPs worked for me without any problem. Did you see any errors in the logs regarding the intel-files you use? Depending on how you generate your feeds the intel linter (https://github.com/packetsled/bro_intel_linter) might be helpful for you.
Best regards,
Jan
Hi,
There are no errors as far as I can tell. I use a python script to generate my feeds but I also tried it on the mal-dnssearch feeds which are pre-formatted. In both cases, I am only successful on domains not IPs. Both have the same tab separation and header delineation.
Thanks,
Disha
I think the most common gotcha for IP addresses is that they will only appear in intel.log when they are a part of a successful TCP connection. Unsuccessful TCP connections and non-TCP connections will not appear in the log.
Josh
This is true. I had to write a new script that will match for uncompleted connections.
However and as you might expect, it is costly with performance.
Thanks for pointing that out Josh! I do want to point out the reasoning behind that as a default decision. My thinking was that spoofed packets could cause false hits and generally non-responded-to probes coming from intelligence hosts aren’t all that interesting. Is there general agreement that that’s the right approach or should single packets hit on intelligence items by default?
I think there is a high bar to clear with adding that as default behavior, but I’d like to hear from people actively doing incident response on their thoughts.
I should also point out that we can certainly include a script that matches on non-complete connections with Bro even if we don’t load it by default.
.Seth
I think you’re right, the default behavior is probably the best for most users.
There are a few cases where I’d want to see an indicator match regardless of the connection status— for example, if the IP indicator in question is associated with a known command and control server or an attack group who may utilize the server as a C2, then I’d want to see any connection attempt from my network to that IP address.
It also depends on how users are using intel.log. For example, if a team treats every indicator in intel.log as a traditional alert (that someone then has to triage), then you want to minimize the resource impact that has on your team (e.g., I don’t want my team chasing down indicators that amount to nothing)— in this case, the default behavior works to our favor. If a team uses intel.log as a source of indicator-related information, then you may want to see every IP indicator match regardless of connection status— seeing every IP indicator match increases your chances of catching things like attacker infrastructure changes or infrastructure roll over.
There’s a lot of granularity that can be worked into how indicators are handled (the intel-ext script is great for that), but it’s very dependent on the objectives of any one particular team. Adding a policy script to enable IP indicator matches for any connection would be a good start, but the default behavior is a great approach!
Josh
I perform IR daily as a core function. I find immense value in seeing all attempts, so long as the performance cost isn’t prohibitive.
Even if the connection doesn’t complete, it shows intent. There may not be a great deal to look at, but it puts it on my radar and generally inspires me to look at other traffic on that local host. It is possible that something else of interest is happening, but the IOCs haven’t made it into a feed.
I'm interested on non-complete connections being logged; CrowdStrike has a script to enable this already and I use it. It’s a way of finding compromised hosts out there that have lost contact with their CnC for whatever reason. But, I do not run a lot of indicators in Bro. Most open source intel lists create too many FPs to be used as a good primary indicator of badness in my experience. An option to enable it per indicator/indicator type or both may be a nice option.
-Brian
IN_ORIG / IN_RESP may help with this
seen IN_RESP in a failed outbound connection to a known phishing site, useful to know
seen IN_ORIG in a failed incoming port 22 connection from a known ssh scanner, probably just noise.
seen in IN_RESP in a failed outbound port 22 connection to that same known ssh scanner, useful to know
Which I guess would mean something like
event connection_i_forget(c: connection) {
if(!Site::is_local_addr(c$id$resp_h)) {
Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
}
}
I would have to agree with the majority here. I’d like the availability to turn it on per sensor or not. Some places are just more sensitive than others.
Also I think looking for every connection outbound is very useful because it can show a compromised host on the network.
Could it be added as a flag in the intel feed data file? Included in the Metadata items - granted, this is a little more actionable than metadata, but there is already meta.do_notice. Could there be a meta.log_any_conn (or something like that). It could default to the full connection state and then those who wanted to see any hits, could turn change the option for those items they want any notification regardless of state.
I think it’s valuable to provide the options, but can see both sides of the fence. Just like others, however, I’m not sure what the performance penalty would be on enabling something like this by default.
Mark
Something that I’d add to Justin’s approach is quasi-state for non-TCP. I’d have to think how best to write the event for a bit, but basically apply the same logic to ICMP/UDP. But also catch if seen IN_ORIG of a UDP connection and there is any response at all. Maybe extend the Crowdstrike script and apply is_local_addr filter.
Something like (could be glitches, haven’t tested this yet):
# Source: https://gist.github.com/dcode/dfe6026fd9865fb8e1ab
@load base/frameworks/intel
@load policy/frameworks/intel/seen/where-locations
event connection_state_remove(c: connection)
{
if ( c$conn?$proto && ( c$conn$proto != tcp || ( c$conn?$history && c$conn$proto == tcp && "h" !in c$conn$history ) ) )
{
if ( !Site::is_local_addr(c$id$resp_h) )
{
Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
}
else if ( Site::is_local_addr(c$id$orig_h) && c$resp_pkts > 0 )
{
Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
}
}
}
Of course, you could definitely parameterize this behavior like known-hosts so it’s easier to configure for incident responders.
Generally speaking, I favor grabbing more data over grabbing a portion of the data, so I prefer to see all of the IP indicators in every connection, then process the output outside of Bro. That said, the suggestions of checking for non-TCP / unestablished TCP IN_RESP connections makes sense if a sensor is resource constrained.
I also like the idea of getting more granular with indicator-related activity (I do a bit of this, just not shared publicly), but I wonder if those specific use cases should be their own policy scripts instead of default behavior. More indicator-related policy scripts could give users better options.
Josh