# http incomplete file extraction (Files::ANALYZER\_EXTRACT) **URL:** \ **Category:** Zeek\ **Created:** [March 27, 2015, 1:35pm UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535 "2015-03-27T13:35:29Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Franky](https://avatars.discourse-cdn.com/v4/letter/f/958977/32.png) [@Franky](https://community.zeek.org/u/Franky)\ **Post date:** [March 27, 2015, 1:35pm UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/1 "2015-03-27T13:35:29Z")
Hi! I am relatively new to bro so please excuse me, if I missed the obvious solution. I want to extract files downloaded via http from a pcap-file, but the files I download are never extracted completely. They seem to be truncated at ~1 mb. My bro-script is quite simple: event file\_new(f: fa\_file) {                 Files::add\_analyzer(f, Files::ANALYZER\_EXTRACT); } Are there any other events I have to catch to get the complete file? When I download a test file from [1] with size 3521964 bytes, only 960204 bytes are extracted. I checked with wireshark and tcpflow, that the download was completely captured in the pcap, I tested with Bro 2.3.2 and the current dev version from git. have a nice weekend! Franky [1] [http://ipv4.download.thinkbroadband.com/5MB.zip](http://ipv4.download.thinkbroadband.com/5MB.zip) ---
**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\ **Post date:** [March 27, 2015, 6:41pm UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/2 "2015-03-27T18:41:47Z")
> event file\_new(f: fa\_file) > { >                Files::add\_analyzer(f, Files::ANALYZER\_EXTRACT); > } Nope, that should work. > Are there any other events I have to catch to get the complete file? > > When I download a test file from [1] with size 3521964 bytes, only 960204 bytes are extracted. I checked with > wireshark and tcpflow, that the download was completely captured in the pcap, Could you show me the files.log entry and the associated conn.log entry?   .Seth ---
**Author:** ![Franky](https://avatars.discourse-cdn.com/v4/letter/f/958977/32.png) [@Franky](https://community.zeek.org/u/Franky)\ **Post date:** [March 30, 2015, 6:31am UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/3 "2015-03-30T06:31:19Z")
Hi again! Thanks for the quick reply! Your question for the logs is a valid one, I should have sent them in my initial mail. I was also wondering, why the correct size is in the logs. If data was missing I would at least have exspected a warning or some missing\_bytes. I hope the logs are readable inline in the mail, attachments seem to be filtered. Thanks! Franky conn.log: #fields ts uid id.orig\_h id.orig\_p id.resp\_h id.resp\_p proto service duration orig\_bytes resp\_bytes conn\_state local\_orig local\_resp missed\_bytes history orig\_pkts orig\_ip\_bytes resp\_pkts resp\_ip\_bytes tunnel\_parents #types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string] 1427461795.952391 CiJ3X2Tf0O0EVCX6a 192.168.2.103 32880 80.249.99.148 80 tcp - - - - OTH - - 0 C 0 0 0 0 (empty) 1427461798.647371 CIS6ae2iV8YZoi8wa3 192.168.2.103 37219 173.194.116.186 80 tcp - 0.016545 0 0 OTH - - 0 Ca 0 0 1 52 (empty) 1427461795.983496 CXNI8E2HLRrrW8qOh1 192.168.2.103 32880 80.249.99.148 80 tcp - 2.369540 0 5243156 SHR - - 0 hCadcf 0 0 3637 5422092 (empty) 1427461798.167374 C9kHyj4HJdMN1lMwtd 192.168.2.103 45447 74.125.136.94 80 tcp - 0.044061 0 0 OTH - - 0 Ca 0 0 1 52 (empty) 1427461798.999381 ClgBVx3a9pQkee3uHf 192.168.2.103 34635 173.194.116.169 80 tcp - 0.016103 0 0 OTH - - 0 Ca 0 0 1 52 (empty) #close 2015-03-27-15-02-36 files.log: #fields ts fuid tx\_hosts rx\_hosts conn\_uids source depth analyzers mime\_type filename duration local\_orig is\_orig seen\_bytes total\_bytes missing\_bytes overflow\_bytes timedout parent\_fuid extracted md5 sha1 sha256 #types time string set[addr] set[addr] set[string] string count set[string] string string interval bool bool count count count count bool string string string string string 1427461796.014318 FbVw4P1oMybfKCu0Wg 80.249.99.148 192.168.2.103 CXNI8E2HLRrrW8qOh1 HTTP 0 EXTRACT - - 0.540901 - F 960204 5242880 0 0 F - extract-1427461796.555219-HTTP-FbVw4P1oMybfKCu0Wg - - - http.log: fields ts uid id.orig\_h id.orig\_p id.resp\_h id.resp\_p trans\_depth method host uri referrer user\_agent request\_body\_len response\_body\_len status\_code status\_msg info\_code info\_msg filename tags username password proxied orig\_fuids orig\_mime\_types resp\_fuids resp\_mime\_types #types time string addr port addr port count string string string string string count count count string count string string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] 1427461796.014318 CXNI8E2HLRrrW8qOh1 192.168.2.103 32880 80.249.99.148 80 0 - - - - - 0 960204 200 OK - - - (empty) - - - - - FbVw4P1oMybfKCu0Wg - ---
**Author:** ![Franky](https://avatars.discourse-cdn.com/v4/letter/f/958977/32.png) [@Franky](https://community.zeek.org/u/Franky)\ **Post date:** [March 30, 2015, 9:24am UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/4 "2015-03-30T09:24:18Z")
Hi Kevin, thanks for your mail. I will have a look at the examples. For your hint about the extraction proces: I still doubt that the root of the problem lies here, because other tools successfully extract the files from the same pcap. Franky ---
**Author:** ![Siwek\_Jon](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)\ **Post date:** [March 30, 2015, 4:54pm UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/5 "2015-03-30T16:54:38Z")
In files.log, the value of total\_bytes is just taken from the HTTP Content-Length header. Since the value of seen\_bytes is less than total\_bytes, you can suspect Bro didn’t see the full file for some reason. Do you have a weird.log containing any obvious clues? Else, I may need the original pcap to understand what went wrong. - Jon ---
**Author:** ![Franky](https://avatars.discourse-cdn.com/v4/letter/f/958977/32.png) [@Franky](https://community.zeek.org/u/Franky)\ **Post date:** [April 1, 2015, 9:01am UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/6 "2015-04-01T09:01:44Z")
Hi! > In files.log, the value of total\_bytes is just taken from the HTTP Content-Length header. Since the value of seen\_bytes is less than total\_bytes, you can suspect Bro didn’t see the full file for some reason. Do you have a weird.log containing any obvious clues? Else, I may need the original pcap to understand what went wrong. The weird.log states some “above\_hole\_data\_without\_any\_acks”, but why does it work with tcpflow? Here is what I did: 1. I downloaded the test file: wget [http://ipv4.download.thinkbroadband.com/5MB.zip](http://ipv4.download.thinkbroadband.com/5MB.zip) 2. Gathered the pcap: tcpdump -s0 -i eth0 -w download.pcap port http 3. checked if the file was completely captured with tcpflow: tcpflow -FT -e http -r download.pcap md5sums do match: ~/bro-liste$ md5sum 2015-04-01T07:45:00Z080.249.099.148.00080-192.168.002.103.42716-HTTPBODY-001.zip b3215c06647bc550406a9c8ccc378756 2015-04-01T07:45:00Z080.249.099.148.00080-192.168.002.103.42716-HTTPBODY-001.zip ~/bro-liste$ md5sum 5MB.zip b3215c06647bc550406a9c8ccc378756 5MB.zip 1. run bro (revision 32ae94de9ae36060651240a0ee11838e3e572223) with simple bro-file: ~/bro-liste$ cat extract.bro event file\_new(f: fa\_file) { Files::add\_analyzer(f, Files::ANALYZER\_EXTRACT); } ~/bro-liste$ /usr/local/bro/bin/bro -r download.pcap extract.bro 1427874309.892545 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid TCP checksums, most likely from NIC checksum offloading. 1. Logs from bro and the pcap: (14mb) [http://www.xup.in/dl,19594721/extract.tar.bz2/](http://www.xup.in/dl,19594721/extract.tar.bz2/) Thanks! Franky ---
**Author:** ![Siwek\_Jon](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)\ **Post date:** [April 1, 2015, 3:26pm UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/7 "2015-04-01T15:26:22Z")
> ~/bro-liste$ /usr/local/bro/bin/bro -r download.pcap extract.bro > 1427874309.892545 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid TCP checksums, most likely from NIC checksum offloading. You’ll have to address this problem to get the results you expect. See: [https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums](https://www.bro.org/documentation/faq.html#why-isn-t-bro-producing-the-logs-i-expect-a-note-about-checksums) > The weird.log states some “above\_hole\_data\_without\_any\_acks" In this case, this seems like it’s just a side effect of the bad checksums, but in case you’re interested on how that type of situation can effect file extraction in Bro there’s discussion of how/why here: [https://bro-tracker.atlassian.net/browse/BIT-1255](https://bro-tracker.atlassian.net/browse/BIT-1255) - Jon ---
**Author:** ![Franky](https://avatars.discourse-cdn.com/v4/letter/f/958977/32.png) [@Franky](https://community.zeek.org/u/Franky)\ **Post date:** [April 2, 2015, 7:23am UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/8 "2015-04-02T07:23:36Z")
Thanks to all who answered! The -C switch did the trick. Sometimes warnings should be taken seriously… Have a nice day! Franky ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\ **Post date:** [May 6, 2022, 3:42pm UTC](https://community.zeek.org/t/http-incomplete-file-extraction-files-analyzer-extract/3535/9 "2022-05-06T15:42:33Z")