# http-ext-identified-files

**URL:** <https://community.zeek.org/t/http-ext-identified-files/1851>\
**Category:** Zeek\
**Created:** [April 1, 2011, 6:20pm UTC](https://community.zeek.org/t/http-ext-identified-files/1851 "2011-04-01T18:20:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Will](https://avatars.discourse-cdn.com/v4/letter/w/90ced4/32.png) [@Will](https://community.zeek.org/u/Will)\
**Post date:** [April 1, 2011, 6:20pm UTC](https://community.zeek.org/t/http-ext-identified-files/1851/1 "2011-04-01T18:20:33Z")

</div>

Hey Seth,

I recently updated one of our bro boxes focusing on http traffic with your ‘ext’ scripts. I am drawing blank on a few things.

1. The old way of flagging via ‘HTTP\_WatchedMIMEType’ appears to have gone away in lieu of ‘add si$tags[“identified-files”];’  
The new ‘redef signature\_files += “http-ext-identified-files.sig”;’ looks to have replaced the magic\_mime. I see that by default, file type in that signature file are ignored.

For the time being, I have added the ‘NOTICE’ and ‘HTTP\_WatchedMIMEType’ back in because I really want email alerts for any watched\_mime\_types not coming from a whitelisted location.

So, what is the correct way to generate alerts? My botched version is below.

1. By adding the ‘NOTICE’ back in below, I think I botched the ‘add si$tags[“identified-files”];’, because none of the files are getting logged to identified files.

Thanks in advance!

Will

---

<div class="post-metadata">

**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\
**Post date:** [May 11, 2011, 1:18pm UTC](https://community.zeek.org/t/http-ext-identified-files/1851/2 "2011-05-11T13:18:28Z")

</div>

Sorry for not reply earlier. I started a response to your email and never finished it. 🙂

---

<div class="post-metadata">

**Author:** ![Aashish\_SHARMA2](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Aashish\_SHARMA2](https://community.zeek.org/u/Aashish_SHARMA2)\
**Post date:** [May 11, 2011, 5:00pm UTC](https://community.zeek.org/t/http-ext-identified-files/1851/3 "2011-05-11T17:00:56Z")

</div>

> Hello:

---

<div class="post-metadata">

**Author:** ![Will](https://avatars.discourse-cdn.com/v4/letter/w/90ced4/32.png) [@Will](https://community.zeek.org/u/Will)\
**Post date:** [May 12, 2011, 5:19pm UTC](https://community.zeek.org/t/http-ext-identified-files/1851/4 "2011-05-12T17:19:57Z")

</div>

> Hello:
> 
> HTTP\_WatchedMIMEType is declared in bro/share/bro/http-identified-files.bro.
> 
> I think you can make the code work by doing the following changes in the http-ext-identified-files.bro
> 
> 1) Load http-identified-files  
> 2) change "const" to "redef" for the following variables: watched\_mime\_types, ignored\_urls, mime\_types\_extensions, ignored\_signatures  
> 3) Comment out declaration of HTTP\_IncorrectFileType from http-ext-identified-files.bro
> 
> + @load http-identified-files
> 
> - redef enum Notice += {  
> - # This notice is thrown when the file extension doesn't  
> - # seem to match the file contents.  
> - HTTP\_IncorrectFileType,  
> - };
> 
> - const watched\_mime\_types = /application\/x-dosexec/  
> + redef watched\_mime\_types = /application\/x-dosexec/
> 
> - const ignored\_urls = /^http:\/\/(au\.|www\.)?download\.windowsupdate\.com\/msdownload\/update/ &redef;  
> + redef ignored\_urls = /^http:\/\/(au\.|www\.)?download\.windowsupdate\.com\/msdownload\/update/ ;
> 
> - redef mime\_types\_extensions: table[string] of pattern = {  
> + const mime\_types\_extensions: table[string] of pattern = {
> 
> - const ignored\_signatures += /^matchfile-/ &redef;  
> + redef ignored\_signatures += /^matchfile-/;
> 
> Aashish
> 
> > Sorry for not reply earlier. I started a response to your email and never finished it. 🙂
> > 
> > > 1. The old way of flagging via 'HTTP\_WatchedMIMEType' appears to have gone away
> > 
> > Hm, I wonder why I removed that? There will be a solution for this problem in the next release.
> > 
> > Did you end up figuring out what was wrong with this?

Yes, pretty close to what Aashish describes to do above. Though I  
don't see what changing the ignored\_signatures file does, because it  
already looks redef'd. Our "whitelist" is larger and slightly more  
custom to our environment, but otherwise just as below. The  
mis-matched file type is great for when a file is down loaded with a  
random string and doesn't have a "watched" mime type, i.e. a php file  
named "WJ4JR874".

Here is what we are using and seems to be working seemlessly:

@load global-ext  
@load http-ext  
@load http-reply  
@load http-body  
@load signatures  
redef signature\_files += "http-ext-identified-files.sig";

module HTTP;

export {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;redef enum Notice += {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# This notice is thrown when the file extension doesn't  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# seem to match the file contents.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HTTP\_IncorrectFileType,

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Generated when we see a MIME type we flagged for watching.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HTTP\_WatchedMIMEType,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;};

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# MIME types that you'd like this script to identify and log.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const watched\_mime\_types = /application\/x-dosexec/  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /application\/x-executable/  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /application\/octet-stream/  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /application\/x-compressed/  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /application\/x-msdownload/ &redef;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# URLs included here are not logged and notices are not thrown.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Take care when defining regexes to not be overly broad.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const ignored\_urls =  
/^http:\/\/(au\.|www\.)?download\.windowsupdate\.com\/msdownload\/update/  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /^http:\/\/.\*\.adobe\.com\//  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /^http:\/\/.\*\.cisco\.com\//  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /^http:\/\/.\*\.hp\.com\//  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /^http:\/\/.\*\.macromedia\.com\//  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /^http:\/\/.\*\.microsoft\.com\//  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> /^http:\/\/.\*\.sun\.com\// &redef;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Create regexes that \*should\* in be in the urls for specifics  
mime types.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Notices are thrown if the pattern doesn't match the url for  
the file type.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const mime\_types\_extensions: table[string] of pattern = {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;["application/x-dosexec"] = /\.([eE][xX][eE]|[dD][lL][lL])/,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} &redef;  
}

# Don't delete the http sessions at the end of the request!  
redef watch\_reply=T;

# Ignore the signatures used to match files  
redef ignored\_signatures += /^matchfile-/;

# This script uses the file tagging method to create a separate file.  
event bro\_init()  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Add the tag for log file splitting.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;LOG::define\_tag("http-ext", "identified-files");  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

event signature\_match(state: signature\_state, msg: string, data: string)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Only signatures matching file types are dealt with here.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ( /^matchfile/ !in state$id ) return;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Not much point in any of this if we don't know about the  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# HTTP-ness of the connection.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ( state$conn$id !in conn\_info ) return;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;local si = conn\_info[state$conn$id];  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Set the mime type seen.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;si$mime\_type = msg;  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;local defanged\_url = gsub(si$url, /\./, "[.]");  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;local message = fmt("%s %s", msg, defanged\_url);  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ( ignored\_urls !in si$url )  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ( watched\_mime\_types in msg )  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;NOTICE([$note=HTTP\_WatchedMIMEType,  
$msg=message, $conn=state$conn, $method=si$method, $URL=si$url]);  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# Add a tag for logging purposes.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;add si$tags["identified-files"];  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ( msg in mime\_types\_extensions &&  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mime\_types\_extensions[msg] !in si$url )  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;NOTICE([$note=HTTP\_IncorrectFileType,  
$msg=message, $conn=state$conn, $method=si$method, $URL=si$url]);  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;event file\_transferred(state$conn, data, "", msg);  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

Thanks to both!

-Will

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\
**Post date:** [May 6, 2022, 3:39pm UTC](https://community.zeek.org/t/http-ext-identified-files/1851/5 "2022-05-06T15:39:31Z")

</div>