Emerging Threats signatures on Bro ids ?

Zeek
1

Hi,

Anyone interested for supporting / converting Emerging Threats [ET] signatures on Bro IDS ?

- convert on regexp bro format (if threats are easy)

- or better convert to a bro powerful language... (more complex threats)

Not a automatic converter, need (long long) review all signatures for understand threats and use better (bro) converter...

What do you think ?

Im interested if anyone are running futur bro+ET direct feedback... (FP, FN, performance....)

Happy Detect with Bro, Suricata and Snort.
Regards
Rmkml

http://twitter.com/rmkml

2

Your best bet would be to try to convert the ET USER_AGENTS signatures
and modify them for inclusion in
https://github.com/grigorescu/bro-scripts/blob/9d59a7a482b068304a2d33a3c9c8dc696c176650/scripts/http-exe-bad-attributes.bro
. That would be a good start.

3

Hi Rmkml,

As Martin said, that would be a good start, and would provide everyone with some very useful data. Longterm, however, I think that this is a perfect fit for the upcoming intelligence framework. As I understand it, the goal of that framework is to separate the scripting layer from the intelligence layer (so, you have a user-agent analyzer script, which reads good or bad user agents from the intelligence layer. Your script stays nice and clean, and your intelligence can just be presented in a logical way, and be processed by the script into something useful).

Unfortunately, Emerging Threats doesn't present the intelligence in a logical way, and it's preprocessed for Snort. What I'd love to see is ET just provide *data*, and then you have a script to convert it to a format Snort understands, Bro processes it into something it understands, and so on.

tl;dr: I think it'd be very useful to have this data, but I don't think anyone should sink too much time into it until the intel framework comes out.

  --Vlad

4

You hit that perfectly. I'm working hard on getting the intelligence framework ready for some people to start testing soon hopefully. It's in memory tuning now to reduce worker memory usage on clusters.

  .Seth

5

Hi,

Ok first alpha release on yesterday update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_10aug.bro
(contains only 13 signatures)

Im interested if you have comments/feedback/flame/performance/FP/FN please.

Tested on bro v2.0 with:
  bro -C -r test.pcap et_bro2_10aug

Futur work:
I have a small pb on this bro powerful language:
-I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.

Regards
Rmkml

http://twitter.com/rmkml

6

Hi,

ok please found second alpha release update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_11aug.bro
  (contains only 37 signatures, fixed one bug on previous rule set)

Im always interested if you have comments/feedback/flame/performance/FP/FN please.

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...

Regards
Rmkml

http://twitter.com/rmkml

7

Have you tried running Bro on live traffic with this script? I looked through it briefly and it seems like it would severely impact performance.

  .Seth

8

Hi,

ok please found third alpha release update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_12aug.bro
  (contains only 54 signatures, begin User-Agent sigs)

Im always interested if you have comments/feedback/flame/performance/FP/FN please.
Enable or disable variable in bro script reduce number sigs (et_currents, et_enabled, et_dns, et_trojan...).

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...
4) how to match POST http_method with argument ?

Regards
Rmkml

http://twitter.com/rmkml

9

Hi,

ok please found Four alpha release update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_13aug.bro
   -sorry, no new signatures
   -moved print to notice alarm realtime mode
   -disabled by default sig performance penalty with et_performancepenalty variable

Im always interested if you have comments/feedback/flame/performance/FP/FN please.
Enable or disable variable in bro script reduce number sigs (et_currents, et_enabled, et_dns, et_trojan...).

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...
4) how to match POST http_method with argument ?

Regards
Rmkml

http://twitter.com/rmkml

10

Hi,
Im continue to update (user-agent actually) converting (open-gpl) Emerging Threats signatures:
  http://88.191.140.111/et_bro2_14aug_pb.bro
It's work.

but when I de-comment/enable these two lines on et_bro2_14aug_pb.bro:
  228: else if ( (/[gG][oO][oO][gG][lL][eE][bB][oO][tT]/ in c$http$user_agent) && sid2015529 && (et_currents || et_useragent) )
  229: NOTICE([$conn=c, $note=EmergingThreats, $msg=fmt("[1:2015529:1] ET CURRENT_EVENTS Googlebot User-Agent Outbound (likely malicious)")]);

bro produce an error:
  bro20 -C -r testbro.pcap et_bro2_14aug_pb
  error in policy/et_bro2_14aug_pb.bro, line 229: memory exhausted, at or near "("

Only for test, continue to enabled lines 228 and 229, but comment/disable previous lines 224 and 225, bro fire on my test...
maybe it's a internal memory related pb on bro ?

Anyone known this pb and how to fix please?
Regards
Rmkml

http://twitter.com/rmkml

11

Hi,

ok please found Five alpha release update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_14aug.bro
   -bypassed "memory exhausted" pb with if loop (w/o else if) at this time
   -72 new signatures (total 124)
   -use different pattern matching regexp or not
   -disabled by default sig performance penalty with et_performancepenalty variable

Im always interested if you have comments/feedback/flame/performance/FP/FN please.
Enable or disable variable in bro script reduce number sigs (et_useragent, et_malware, et_currents, et_dns, et_trojan...).

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...
4) how to match POST http_method with argument ?

Regards
Rmkml

http://twitter.com/rmkml

12

I think there are some fundamental issues at play here, and integrating the EmergingThreats signatures in this manner is probably not the right way to go.

Some comments on the script:

    - That et_performancepenalty variable won't really help since just handling the packet_content event is what will cause most of the overhead.

    - Your other variables such as sid2014029, et_trojan, and et_useragent are not actually providing any benefit because you aren't ordering the conditions correctly to allow for short circuiting.

    - You are accessing c$http$user_agent inside of http_header event handlers which will have several problems by itself. That variable won't be filled out until the user-agent header is seen which means you will be filling the reporter log with a lot of error messages and possibly causing memory leaks because the event handler will be failing due to an attempt to access a null field in the record. It's also something that could just be checked at a later point (like in http_message_done).

Those three big issues were just from a quick glance through the code. There are better and more flexible ways of approaching this, but a big part of the problem is the way the intelligence from emerging threats is distributed is not suitable as-is for Bro right now.

The best thing that could come out of our community now is guidance for EmergingThreats on how to provide their data in a way that is less product specific. The signatures are written for Snort and Suricata; trying to jam them into Bro without thinking hard about the problem is probably going to be a waste of effort unfortunately.

  .Seth

13

I think there are some fundamental issues at play here, and integrating the EmergingThreats signatures in this manner is probably not the right way to go.

Some comments on the script:

    - That et_performancepenalty variable won't really help since just handling the packet_content event is what will cause most of the overhead.

    - Your other variables such as sid2014029, et_trojan, and et_useragent are not actually providing any benefit because you aren't ordering the conditions correctly to allow for short circuiting.

    - You are accessing c$http$user_agent inside of http_header event handlers which will have several problems by itself. That variable won't be filled out until the user-agent header is seen which means you will be filling the reporter log with a lot of error messages and possibly causing memory leaks because the event handler will be failing due to an attempt to access a null field in the record. It's also something that could just be checked at a later point (like in http_message_done).

Those three big issues were just from a quick glance through the code. There are better and more flexible ways of approaching this, but a big part of the problem is the way the intelligence from emerging threats is distributed is not suitable as-is for Bro right now.

The best thing that could come out of our community now is guidance for EmergingThreats on how to provide their data in a way that is less product specific. The signatures are written for Snort and Suricata; trying to jam them into Bro without thinking hard about the problem is probably going to be a waste of effort unfortunately.

We're definitely all for that! We already distribute our rules in
forms other than Snort and Suricata for a few customers. We're set up
well on the backend to manage multiple languages.

I have to claim significant ignorance of how the Bro language looks
and is structured. But very eager to dive in and see what we can do.

We tried this years ago as I mentioned, and the impact we had on
performance wasn't good. What is essentially the approach we need to
take to put the same intel into a form Bro can use effectively? Do we
have to not think "one bro sig == one suricata sig"?

Thanks Seth!

Matt

14

So, here's the intel feed I'd want:
{
  host:<some bad hostname pattern, e.g. 'example.com'>
  uri: <some bad URI pattern, e.g. 'in.cgi'>
  uri_params: <array of URI parameters that constitutes "badness",
e.g. [ 'id', 'os', 'affid' ]
  headers: <hash of header content of badness, e.g. { 'user-agent': 'Presto' },
  etc...
}

As you can probably see, yara would be a great fit for something like this.

15

It's even a bit further than that I'm afraid. The problem is that in the case of many of your rules you have some intelligence in them, but it's encoded with the implicit assumption that you are just scanning a byte stream (in most cases at least).

Since I work best in very concrete term, I'll give some examples of signatures and how they could be reapplied into general intelligence that we could more easily consume…

alert tcp $EXTERNAL_NET any -> $SMTP_SERVERS 25 (msg:"ET CURRENT_EVENTS DHL Spam Inbound"; flow:established,to_server; content:"name=|22|DHL"; nocase; content:".zip|22|"; within:68; nocase; pcre:"/name=\x22DHL(\s|_|\-)?[a-z0-9\-_\.\s]{0,63}\.zip\x22/i"; reference:url,doc.emergingthreats.net/2010148; classtype:trojan-activity; sid:2010148; rev:12;)

What that rule is really doing is looking for file names matching the regular expression…
  /^DHL(\s|_|\-)?[a-z0-9\-_\.\s]{0,63}\.zip$/

The first version of the intelligence framework in Bro won't support regular expressions unfortunately, but hopefully it will eventually. The data would be included into Bro like this (this is made up right now, just to get the idea across):

  [$pattern=/^DHL(\s|_|\-)?[a-z0-9\-_\.\s]{0,63}\.zip$/, $subtype=Intel::FILENAME, $expected_in=Intel::EMAIL]

If you had a full filename to match it might look like this…

  [$str="DHL.zip", $subtype=Intel::FILENAME, $expected_in=Intel::EMAIL]

By feeding in intelligence this way we can suddenly reuse that information to start doing these matches in other protocols and in ways that you didn't originally expect.

Another example:

#alert http $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS TROJAN Likely TDSS Download (197.exe)"; flow:established,to_server; content:"GET "; depth:4; nocase; uricontent:"/codec/197.exe"; nocase; reference:url,doc.emergingthreats.net/2010056; classtype:trojan-activity; sid:2010056; rev:3;)

This would be:
  [$glob="*/codec/197.exe", $subtype=Intel::URL, $expected_in=Intel::URL]
  
I will say there are plenty of examples in your set now that we don't yet have a great answer for, but we're considering how to make those work as well.

  .Seth

16

Haha. That's actually fairly nice and similar to how Bro's existing signature language works already (we have a number of special keywords besides "payload"), but the problem that I see is where you know a file name that you know is being used by malicious actors and you'd like to watch for that filename anywhere. You're really looking to just insert the filename as intelligence and watch everywhere that file names are found.

You do make a good point though that reapplying our signature language to intelligence correlation might be a good idea.

  .Seth

17

If you start with the intel that filename x=bad, then you could have
an internal Bro generation that extends the filename to URI. So, some
of the URI information would be specific to URI patterns etc., but
some would be autogenerated at runtime by Bro:

pseudo-spec:
{
  type: FILE,
  filename:foo
}

Synthesizes this additional check:

{
  type: HTTP,
  uri: foo
}

18

Hi,

ok please found Six alpha release update (open-gpl) Emerging Threats signatures, I have switched to bro signature language:
  http://88.191.140.111/et_bro2_16aug.sig

You can start bro like this:
  bro -i eth0 -s et_bro2_16aug.sig

Features:
-previously used bro powerful language, now use bro signature language!
-update to last Emerging Threats 16 Aug 2012
-contains only 111 sig at this time, work in progress
-bro signature language use regular expression (like juniper/onesecure) need rewrite signature
-remember bro tcp reassembly only first 1k for performance reason, check dpd_buffer_size

Im always interested if you have comments/feedback/flame/performance/FP/FN please.

Futur work:
1) use Dynamic Port Detection (not static port)
1) use local_net / external_net
2) split signature per category files
3) find case insensitive more "simplify" regexp ?
4) create on new dns parser like dns-request on bro signature language

More information on Bro Signature Language:
  http://bro-ids.org/documentation/signatures.html

Regards
Rmkml

http://twitter.com/rmkml

19

Hi,
ok please new release update (open-gpl) Emerging Threats signatures to Bro Signature langage:
  http://88.191.140.111/et_bro2_17aug.sig

You can start bro like this:
  bro -i eth0 -s et_bro2_17aug.sig

Features:
-update to last Emerging Threats 17 Aug 2012
-contains only 269 sig at this time, work in progress
-bro signature language use regular expression (like juniper/onesecure) need rewrite ET signature
-remember bro tcp reassembly only first 1k for performance reason, check dpd_buffer_size

Im always interested if you have comments/feedback/flame/performance/FP/FN please.

Futur work:
1) use Dynamic Port Detection (not static port)
2) use local_net / external_net
3) split signatures per category files
4) find case insensitive more "simplify" regexp ?
5) create on new dns parser like dns-request on bro signature language

More information on Bro Signature Language:
http://bro-ids.org/documentation/signatures.html

Regards
Rmkml

http://twitter.com/rmkml

20

Could you please stop announcing every update you do to these signatures on our mailing list? You are free to start your own mailing list and announce these there for all who are interested.

The signatures you are writing aren't actually something that can be deployed on most users Bro installations due to performance issues and hold very little interest for much of the community due to that. Additionally, the signatures already exist and are tested for other platforms. Anyone running your signatures would have to trust that you converted them into Bro scripts and signatures correctly.

Thanks,
  .Seth