# File extraction after checking hash.

**URL:** <https://community.zeek.org/t/file-extraction-after-checking-hash/4386>\
**Category:** Zeek\
**Created:** [October 3, 2016, 6:49pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386 "2016-10-03T18:49:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fatema\_bannatwala](https://avatars.discourse-cdn.com/v4/letter/f/5f9b8f/32.png) [@fatema\_bannatwala](https://community.zeek.org/u/fatema_bannatwala)\
**Post date:** [October 3, 2016, 6:49pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/1 "2016-10-03T18:49:38Z")

</div>

Hi,

I was reading about the Files framework of Bro, and know that there are file analyzers available that can be attached to files that Bro sees on the network connection.  
I am currently extracting all the ‘application/x-dosexec’ files from http connections, and realized that  
there are lot of files that are just duplicates (i.e with same hashes).

Hence was thinking to write some bro script that would use Files analysis FW and checkes the hash of the file first against a set of hashes already seen (extracted) by Bro and will skip the extraction of that file if it’s present in the set of hashes.

I tried adding the files::add\_analyzer(f, Files::ANALYZER\_EXTRACT,…) in file\_new event, file\_sniff event and file\_state\_removed event(except it didn’t work here), but turns out that file\_hash event triggers later than all these events and hashes get calculated after the file extraction analyzer has run.

Hence wanted to ask is it possible to add Files::ANALYZER\_EXTRACT AFTER Files::ANALYZER\_MD5 analyzer so that I can get the hash first to compare against the set before making a decision to extract the file?

Thanks,  
Fatema.

---

<div class="post-metadata">

**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\
**Post date:** [October 4, 2016, 2:36am UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/2 "2016-10-04T02:36:30Z")

</div>

Unfortunately not. Since we don't know the hash of the file when we see the beginning we can't yet determine that we don't want to extract the file. Sort of a chicken and egg problem. 🙂

&nbsp;&nbsp;.Seth

---

<div class="post-metadata">

**Author:** ![fatema\_bannatwala](https://avatars.discourse-cdn.com/v4/letter/f/5f9b8f/32.png) [@fatema\_bannatwala](https://community.zeek.org/u/fatema_bannatwala)\
**Post date:** [October 4, 2016, 12:34pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/3 "2016-10-04T12:34:15Z")

</div>

Thanks Seth for confirming!  
I think we can go through the extractions afterwards and write some sort of script to delete the dups. 🙂  
And same for hashes, asked Wes Young about querying limit to cif server (REN-ISAC) for hashes.  
I know that we can query the cif server for a given hash, and get back the results with cif confidence rate and other respective fields.  
Hence will be writing some scripts to get unique hashes and malware execs from traffic 🙂

Thanks!  
Fatema.

---

<div class="post-metadata">

**Author:** ![fatema\_bannatwala](https://avatars.discourse-cdn.com/v4/letter/f/5f9b8f/32.png) [@fatema\_bannatwala](https://community.zeek.org/u/fatema_bannatwala)\
**Post date:** [October 4, 2016, 5:39pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/4 "2016-10-04T17:39:59Z")

</div>

So here’s a simple script that will add a column ‘uniq\_hash’ to the files.log file that will show  
whether bro has seen that hash before (in one day duration).

module Uniq\_hashes;

redef record Files::Info += {

## Adding a field column of host and uniq\_hash to show from where

## the file got downloaded and whether seen first time or duplicate.

host: string &optional &log;  
uniq\_hash: bool &optional &log ;  
};

#global uniq\_hashes: set[string] ;  
global uniq\_hashes: set[string] &create\_expire=1day;

event file\_hash(f: fa\_file, kind: string, hash: string)  
{  
print “file\_hash”, f$id, kind, hash;

if(f?$http && f$http?$host)  
f$info$host = f$http$host;

if(hash in uniq\_hashes)  
f$info$uniq\_hash = F;

else  
{  
add uniq\_hashes[hash];  
f$info$uniq\_hash = T;  
}

}

And, then I can grep the hashes with uniq\_hash=T and query the cif server for analysis.  
Also, can script to get the name of the extracted file from the ‘extracted’ field in files.log with uniq\_hash=F  
and delete that file almost realtime, after Bro has extracted that file.

Before I can test it in production, I want to ask if there is a way I can delete the contents of set uniq\_hashes right at the midnight  
so that we can get unique files and hashes on a daily basis logged in files.log? I don’t want that variable to grow out of bound,  
consuming lot of memory, hence thought 1 day should be reasonable period of time to flush the contents of the set and exact time line  
will give an idea of uniq hashes queried daily and no. of execs extracted on daily basis.

Any help appreciated!

Thanks,  
Fatema.

---

<div class="post-metadata">

**Author:** ![Azoff\_Justin\_S](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@Azoff\_Justin\_S](https://community.zeek.org/u/Azoff_Justin_S)\
**Post date:** [October 4, 2016, 6:40pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/5 "2016-10-04T18:40:30Z")

</div>

> And, then I can grep the hashes with uniq\_hash=T and query the cif server for analysis.  
> Also, can script to get the name of the extracted file from the 'extracted' field in files.log with uniq\_hash=F  
> and delete that file almost realtime, after Bro has extracted that file.

Do you know that the intel framework supports hashes? If you export a feed of hashes from CIF you can load that into bro and do the alerting on known hashes bad in real time.

> Before I can test it in production, I want to ask if there is a way I can delete the contents of set uniq\_hashes right at the midnight  
> so that we can get unique files and hashes on a daily basis logged in files.log? I don't want that variable to grow out of bound,  
> consuming lot of memory, hence thought 1 day should be reasonable period of time to flush the contents of the set and exact time line  
> will give an idea of uniq hashes queried daily and no. of execs extracted on daily basis.

You can probably do it using something like this:

global SECONDS\_IN\_DAY = 60\*60\*24;

function midnight(): time  
{  
&nbsp;&nbsp;&nbsp;&nbsp;local now = network\_time();  
&nbsp;&nbsp;&nbsp;&nbsp;local dt = time\_to\_double(now);  
&nbsp;&nbsp;&nbsp;&nbsp;local mn = double\_to\_count(dt / SECONDS\_IN\_DAY) \* SECONDS\_IN\_DAY;  
&nbsp;&nbsp;&nbsp;&nbsp;return double\_to\_time(mn);  
}

function interval\_to\_midnight(): interval  
{  
&nbsp;&nbsp;&nbsp;&nbsp;return midnight() - network\_time();  
}  
event reset\_hashes()  
{  
&nbsp;&nbsp;&nbsp;&nbsp;uniq\_hashes = set(); #I think this is the proper way to clear a set?  
}

event bro\_init()  
{  
&nbsp;&nbsp;&nbsp;&nbsp;print "Time to midnight:", interval\_to\_midnight();  
&nbsp;&nbsp;&nbsp;&nbsp;schedule interval\_to\_midnight() { reset\_hashes()};  
}

I think that might work properly except for the timezone being in UTC, so it might need to be adjusted, or something different altogether

Seth has this plugin: [GitHub - sethhall/bro-approxidate: Approxidate time plugin for Bro](https://github.com/sethhall/bro-approxidate)

which would let you do

local md = approxidate("midnight");

If it was packaged for bro-pkg it would be easier to install though 🙂

The known hosts/services/certs scripts need a framework to do things like this, so 2.6 may end up having this as a built in feature.

---

<div class="post-metadata">

**Author:** ![fatema\_bannatwala](https://avatars.discourse-cdn.com/v4/letter/f/5f9b8f/32.png) [@fatema\_bannatwala](https://community.zeek.org/u/fatema_bannatwala)\
**Post date:** [October 4, 2016, 7:49pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/6 "2016-10-04T19:49:54Z")

</div>

Hi Justin,

> Do you know that the intel framework supports hashes? If you export a feed of hashes from CIF you can load that into bro and do the alerting on known hashes bad in real time.

Yes. And that was the plan, but unfortunately, I couldn’t get the list of the feeds (hashes) pulled down from REN-ISAC , that’s interesting that they provide other feeds but hashes (will ask in REN-ISAC mailing list to confirm).  
But I figured out that you can query their database to get information about a particular hash.  
Also, tried looking for a good open source of feeds for hashes, but couldn’t find it hence don’t have any hash feeds currently in intel ☹

Thank you for the code, works perfect! 🙂  
Made a little tweak, replaced network\_time() with current\_time() function at both the places.  
For some reason I was getting 0.0 as network\_time() value when ran the code in [try.bro.org](http://try.bro.org) with sample http pcap.

Also, added "local mn\_EST = mn + 14400.0; " in midnight() function to get local EST in quick and dirty way. 🙂 (I know the best way to do ii to use Seth’s plugin, will try that next).

Hence, the complete script looks like this now:

module Uniq\_hashes;

redef record Files::Info += {

## Adding a field column of host and uniq\_hash to show from where

## the file got downloaded and whether seen first time or duplicate.

host: string &optional &log;  
uniq\_hash: bool &optional &log ;  
};

global SECONDS\_IN\_DAY = 60_60_24;  
global uniq\_hashes: set[string] ;

function midnight(): time  
{  
local now = current\_time();  
local dt = time\_to\_double(now);  
local mn = double\_to\_count(dt / SECONDS\_IN\_DAY) \* SECONDS\_IN\_DAY;  
local mn\_EST = mn + 14400.0;  
return double\_to\_time(mn\_EST);  
}

function interval\_to\_midnight(): interval  
{  
return midnight() - current\_time();  
}

event reset\_hashes()  
{  
uniq\_hashes = set(); #I think this is the proper way to clear a set?  
}

event file\_hash(f: fa\_file, kind: string, hash: string)  
{  
#print “file\_hash”, f$id, kind, hash;

if(f?$http && f$http?$host)  
f$info$host = f$http$host;

if(hash in uniq\_hashes)  
f$info$uniq\_hash = F;

else  
{  
add uniq\_hashes[hash];  
f$info$uniq\_hash = T;  
}

}  
event bro\_init()  
{ #print “current\_time”, current\_time();  
#print “midnight”, midnight();  
#print “Time to midnight:”, interval\_to\_midnight();  
schedule interval\_to\_midnight() { reset\_hashes()};  
}

Thanks,Fatema.

---

<div class="post-metadata">

**Author:** ![Jan](https://avatars.discourse-cdn.com/v4/letter/j/ce7236/32.png) [@Jan](https://community.zeek.org/u/Jan)\
**Post date:** [October 4, 2016, 8:22pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/7 "2016-10-04T20:22:06Z")

</div>

> > Do you know that the intel framework supports hashes? If you export a
> 
> feed of hashes from CIF you can load that into bro and do the alerting on  
> known hashes bad in real time.
> 
> Yes. And that was the plan, but unfortunately, I couldn't get the list of  
> the feeds (hashes) pulled down from REN-ISAC

If you come up with a feed, using the intel framework should be straight  
forward. We did a POC, extracting files (I think below 100MB) and just  
preserve them in case of an intel hit (see  
[https://github.com/J-Gras/intel-extensions/blob/master/scripts/preserve\_files.bro](https://github.com/J-Gras/intel-extensions/blob/master/scripts/preserve_files.bro)).  
The only thing to set up except extraction and this script is a cron job  
deleting the extracted files that aren't of interest. To avoid dups one  
might want to name the extracted files according to their hash or  
something like that.

Jan

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\
**Post date:** [May 6, 2022, 3:44pm UTC](https://community.zeek.org/t/file-extraction-after-checking-hash/4386/8 "2022-05-06T15:44:06Z")

</div>