Hey all,
So here's the run:
sudo bro -C -r ../captures/email.pcapng /usr/local/bro/share/bro/policy/frameworks/intel/seen/smtp-url-extraction.bro
and list of files generated:
-rw-r--r-- 1 root root 12419 Aug 7 10:18 conn.log
-rw-r--r-- 1 root root 0 Aug 7 10:18 debug.log
-rw-r--r-- 1 root root 12586 Aug 7 10:18 files.log
-rw-r--r-- 1 root root 253 Aug 7 10:18 packet_filter.log
-rw-r--r-- 1 root root 39557 Aug 7 10:18 smtp.log
-rw-r--r-- 1 root root 7936 Aug 7 10:18 ssl.log
-rw-r--r-- 1 root root 8608 Aug 7 10:18 x509.log
For the life of me I'm unable to find where the links might be at. One of the links in the pcap has 88EX336W4062X11N55206638L1122194955 in it...this string shows up no where in any of the logs...is there a step I'm missing with this? Thank you.
James
Ah! Perhaps a poorly named script. That's only extracting the URLs and feeding them into the intel framework.
Would you like a script that extracts and logs them? I ran one of those in production before, it was useful to be able to see what links were flying around for sure.
I'm thinking for fields we could have...
ts
uid
fuid
trans_depth
link
That should provide enough information to link back to the connection it happened over and which "file" (or body content since they're effectively the same in smtp) it was seen within.
.Seth
I would absolutely love a script that would log urls....we all know that quoted-printable and bas364 shenanigans may get missed, but every little bit helps..thanks a bunch Seth.
James
I would absolutely love a script that would log urls....we all know that quoted-printable and bas364 shenanigans may get missed
Much of that should be handled automatically by the mime analyzer (I'm not sure of the limits of that offhand).
, but every little bit helps..thanks a bunch Seth.
I'll see if I can get to it soon.
.Seth
Seth,
+100
I just wanted to add that I think that script that logs SMTP URLs would get a lot of use in our environment as well. It’s been an elusive data point, but one we really would like to have. We’ve been having high-level discussions on how to implement something that does this exact process in our office, so I’d be very interested in using this script once it’s ready as well.
Thanks!
–John
All,
I submitted a pull request last week for this. You could technically grab the script and run it. Since I’m not part of the Bro team though, I can’t promise that this will continue to work.
https://github.com/bro/bro/pull/10
I run a variation of this script in my production environment right now. Keep in mind that it is normally a bad plan to extend an internal Bro module. Since there’s a pretty high demand for it, if you’d like to modify this to not extend the internal SMTP modules and be separate, it is a relatively short task (about 15 minutes).
Lastly, this is provided as-is with no warranty, etc. etc.
Thanks,
Stephen
OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending.
These scripts have been running for > 1 1/2 years so I can say they are fairly stable and should not cause any issues.
1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.
2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro
3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2).
This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate.
This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well.
Please let me know if you have any questions or have issues running these scripts.
Thanks,
Aashish
LBNL
smtp-embedded-url.bro (6.56 KB)
smtp-embedded-url-bloom.bro (7.1 KB)
site.bro (1.85 KB)
Thanks, Aashish.
I've been working on this script for a while this morning just doing general clean up and documentation. Right now I'm getting ready to add cluster support to it. I'll
Stephen, cool if I close your pull request since I think that Aashish's script has more functionality?
.Seth
Aashish,
I'm curious why you suggested only using the bloom filter version of
this script in Bro 2.3-- is there a reason one wouldn't want to use it
in Bro 2.2?
Josh
Bloomfilter code in bro-2.2 or below has had some hash collision issues.
Matthias's fixes became part of bro-2.3 release (from CHANGE log):
- Switch to double hashing for Bloomfilters for better performance. (Matthias Vallentin)
- Bugfix to use full digest length instead of just one byte for Bloomfilter's universal hash function. Addresses BIT-1140. (Matthias Vallentin)
Please see: https://bro-tracker.atlassian.net/browse/BIT-1140
If you run smtp-embedded-url-bloom.bro in bro-2.2 world, You will see a huge number of false positives for "SMTP_Link_in_EMAIL_Clicked"
smtp-embedded-url.bro has exact same functionality, except that it maintains a table of smtp urls and checks http requests against it. So less efficient on memory. I expire the contents of the table in 12hours thus a little limited on visibility too. But still I'd say the code works quite alright, so if you cannot quite immidiately upgrade to bro-2.3, feel free to use: smtp-embedded-url.bro script.
Hope this helps,
Aashish
I didn't realize there were hash collision issues with 2.2 and below--
thanks for the info!
Josh
Please close it and use Aashish's. Mine is actually a variation of his suite, so getting it from him works.