ActiveHTTP

Zeek
1

I’m testing a new script in 2.5 that uses ActiveHTTP but I’m unable to retrieve the response. With a simple test script of:

when ( local resp = ActiveHTTP::request([$url=“https://www.google.com/”]) )
{
print “Inside the Matrix."
}

I can see the ActiveHTTP request was successful based on the temporary files created:

-rw-r–r-- 1 dave wheel 162 Jan 27 23:43 /tmp/bro-activehttp-HJKhXt6UYXi_body
-rw-r–r-- 1 dave wheel 163 Jan 27 23:43 /tmp/bro-activehttp-HJKhXt6UYXi_headers

But the print statement within the when block never executes. Any ideas what I’m missing?

-Dave

2

I added simple print statements in base/utils/active-http.bro and it doesn’t appear to be entering it’s when() block either. These are the two print statements I added:

print “Entering the ActiveHTTP::Request when() block”;
return when ( local result = Exec::run([$cmd=cmd, $stdin=stdin_data, $read_files=set(bodyfile, headersfile)]) )
{
print “In ActiveHTTP::Request when() block”;

If there is no response line then nothing else will work either.

And the second print doesn’t execute:

$ bro -r test.pcap local …/test.bro

Entering the ActiveHTTP::Request when() block…

I have ‘exit_only_after_terminate’ set to true so it just hangs at this point until I ctrl-c and I see the tmp files deleted.

-Dave

3

If you still have the temp files it means something went wrong along the way. Is bro writing out a reporter.log?

4

Hi Justin,

I responded with a follow-up to my original email and temp files are there because I have ‘exit_only_after_terminate’ set to true, so it pauses until I ctrl-c and the tmp files are then deleted.

-Dave

5

No, the files are there because something went wrong along the way. Is bro writing out a reporter.log?

The code normally works fine, something is broken in your environment.

$ cat b.bro
redef exit_only_after_terminate=T;
when ( local resp = ActiveHTTP::request([$url="https://www.google.com/"]) )
{
  print resp;
  terminate();
}
$ bro --version
bro version 2.5
$ bro b.bro
[code=200, msg=OK\x0d, body=<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en">

6

Interestingly your test script works as expected when run as:

bro b.bro

But if I pass it a PCAP it exhibits the same condition where the when loop isn’t entered:

bro -r test.pcap b.bro

This is the test PCAP I was testing with:

https://github.com/LiamRandall/BroTraining-Montreal/raw/master/signature-framework/1-mswab_yayih/Mswab_Yayih_FD1BE09E499E8E380424B3835FC973A8_2012-03.pcap

-Dave

7

Hi Dave,

But if I pass it a PCAP it exhibits the same condition where the when loop isn’t entered:

bro -r test.pcap b.bro

my guess would be that reading a pcap causes timing problems. Have you
tried processing the pcap using --pseudo-realtime?

Jan

8

I tried with —pseudo-realtime as well as creating a new PCAP to test with but it still exhibits the same behavior. ActiveHTTP successfully makes the request, and receives a response based other the contents of the temp files, but the when() block is never executed.

The reporter.log only has an event for the termination:

#types time enum string string
1485725443.690539 Reporter::INFO received termination signal (empty)

Is anyone able to re-create the same issue or is this limited to my environment?

-Dave

9

I’ve been able to test this in another environment (Debian 8.7 x64) and unlike OS X where the ActiveHTTP conducts a successful request but then doesn’t enter the when{} block, on Debian it errors with the following written to reporter.log:

$ bro --version
bro version 2.5-30

$ bro b.bro

0.000000 Reporter::ERROR curl -s -g -o “/tmp/bro-activehttp-XMayZ2GFnB6_body” -D “/tmp/bro-activehttp-XMayZ2GFnB6_headers” -X “GET” -m 60 “https://www.google.com/” && touch /tmp/bro-activehttp-XMayZ2GFnB6_body |/Input::READER_RAW: Child process exited with non-zero return code 127 (empty)
0.000000 Reporter::WARNING Stream vqz7bJcG1Pg is already queued for removal. Ignoring remove. (empty)
0.000000 Reporter::ERROR /tmp/bro-activehttp-XMayZ2GFnB6_body/Input::READER_RAW: Init: cannot open /tmp/bro-activehttp-XMayZ2GFnB6_body (empty)
0.000000 Reporter::ERROR /tmp/bro-activehttp-XMayZ2GFnB6_body/Input::READER_RAW: Init failed (empty)
0.000000 Reporter::ERROR /tmp/bro-activehttp-XMayZ2GFnB6_body/Input::READER_RAW: terminating thread (empty)
0.000000 Reporter::ERROR /tmp/bro-activehttp-XMayZ2GFnB6_headers/Input::READER_RAW: Init: cannot open /tmp/bro-activehttp-XMayZ2GFnB6_headers (empty)
0.000000 Reporter::ERROR /tmp/bro-activehttp-XMayZ2GFnB6_headers/Input::READER_RAW: Init failed (empty)
0.000000 Reporter::ERROR /tmp/bro-activehttp-XMayZ2GFnB6_headers/Input::READER_RAW: terminating thread (empty)
0.000000 Reporter::INFO received termination signal (empty)
#close 2017-01-30-12-26-47

10

Ok, scratch that error message. The box I was testing on didn’t have curl installed. After installing curl the test script has the same behavior as when run on OS X. Work great by itself but hangs before the when{} block if passed a PCAP.

11

Ok, scratch that error message. The box I was testing on didn’t have curl installed. After installing curl the test script has the same behavior as when run on OS X. Work great by itself but hangs before the when{} block if passed a PCAP.

bro --pseudo-realtime -r
Mswab_Yayih_FD1BE09E499E8E380424B3835FC973A8_2012-03.pcap b.bro

works for me. Takes about one and a half minute (the PCAP covers ~5mins)
to spit out the result.

Jan

12

Thanks Jan, what version of Bro are you running and on which platform?

I have 'bro version 2.5-30’, compiled from Github master, on Debian 8.7 and macOS 10.12.2 and both hang until I ctrl-C, and neither enters the when{} block:

macOS$ time bro -r bro_dev/Mswab_Yayih_FD1BE09E499E8E380424B3835FC973A8_2012-03.pcap b.bro
^C1330843811.964963 received termination signal

real 8m30.316s
user 1m31.343s
sys 6m58.036s

debian$ time bro -r test2.pcap b.bro
^C1330843811.964963 received termination signal

real 2m42.507s
user 1m19.328s
sys 1m23.168s

13

Thanks Jan, what version of Bro are you running and on which platform?

I am using Bro 2.5 on Fedora 23 (4.8 kernel).

I have 'bro version 2.5-30’, compiled from Github master, on Debian 8.7 and macOS 10.12.2 and both hang until I ctrl-C, and neither enters the when{} block:

$ time bro --pseudo-realtime -r
Mswab_Yayih_FD1BE09E499E8E380424B3835FC973A8_2012-03.pcap b.bro
[code=302, msg=Found\x0d, body=...]
1485807420.620682 received termination signal

real 1m0.583s
user 0m26.229s
sys 0m34.185s

Without "--pseudo-realtime" it seems to hang for me, too. Have you tried
using it?

Jan

14

Thanks Jan! So on the --pseudo-realtime option did the trick. I had similar results on Debian as you:

real 1m0.579s
user 0m31.236s
sys 0m29.344s

And similar results on macOS:

real 1m0.568s
user 0m13.238s
sys 0m47.192s

I at least now have a comfort level to continue writing my script (my production Bro boxes are Debian).

-Dave

15

Following on this ticket from the main Bro list, I wonder if we could change the behavior of Bro slightly to make what Dave tried work? I *think* the problem here is that once the packets run out, Bro's internal clock stops moving forward which causes all sorts of trouble for timers and other stuff I'm sure.

What does everyone think about making the clock continue to move forward even after the packet source runs dry? This especially makes sense when someone uses pseudo-realtime because we can keep moving the clock at the rate it was moving (but not jump to current time, we'd just do subtraction based on the time when the packet source ran dry). The main problem I see with this idea is if someone reads a PCAP at full speed, what rate do we make the clock continue ticking?

Does this idea make sense at all? I think we've had too many new Bro programmers get frustrated with this behavior which worries me a little bit.

  .Seth