Running external command line programs

Zeek
1

In case anyone is interested in playing with this *very* early, a couple of days ago I wrote a wrapper for the input framework to execute command line programs and get the result back into Bro in a non-blocking manner. It makes stdout, stderr, and the exit code available once your command is done executing.

The script:
  https://github.com/sethhall/bro-junk-drawer/blob/master/exec.bro

An example use of the script:
  https://github.com/sethhall/bro-junk-drawer/blob/master/exec-test.bro

I'd appreciate feedback if anyone tries it, thanks!

  .Seth

2

I thought I should mention that I did some more updates to make this work better and the current commit that is in my github repository is broken. We're going to be fixing a bug in Bro and likely including this functionality in Bro 2.2.

  .Seth

3

I just got a question asking about getting a working version. You can checkout a commit after you clone the repository that does work like this..

  git checkout edf424

  .Seth

4

Hey Seth,

Having this type of functionality would be awesome! It would “unlock” bro to the point where we would only be limited by our imaginations with what we could make bro do.

I know you mentioned that the current stuff is broken on github, but I gave it a try anyways (I modified the command in exec-test.bro to the date command):

$ bro -r test.pcap exec-test.bro
entering the async whatever
yay!
{
[/tmp/bro-exec-4N1gxc3hF32] = [Thu Feb 21 2013]
}
bro: bro-2.1/src/Trigger.cc:227: bool Trigger::Eval(): Assertion `frame->GetCall()’ failed.
Aborted

So close, and yet so far.

I’m assuming that this is the bug that you mentioned Bro 2.2 will fix. When is Bro 2.2 expected to be released?

-Chris

5

Hey Seth,

Having this type of functionality would be awesome! It would "unlock" bro to the point where we would only be limited by our imaginations with what we could make bro do.

I know you mentioned that the current stuff is broken on github, but I gave it a try anyways (I modified the command in exec-test.bro to the date command):

$ bro -r test.pcap exec-test.bro
entering the async whatever
yay!
{
[/tmp/bro-exec-4N1gxc3hF32] = [Thu Feb 21 2013]
}
bro: bro-2.1/src/Trigger.cc:227: bool Trigger::Eval(): Assertion `frame->GetCall()' failed.
Aborted

So close, and yet so far.

Yep, that's the bug. Try checking out the other commit that I suggested. That should make it work.

Additionally I already have a full module named ActiveHTTP wrapped around it (about 100 lines of code) that uses the curl command line client internally (yes, hacky) but present a very nice and clean API to users. You currently get the body of the response, the response code, the response message, and all of the headers the server returned.

This sort of opens the door to all kinds of crazy stuff though. Someone (you know who you are!) already mentioned the idea of doing an NMAP wrapper so that people could start NMAP scans and get results back into Bro really easily.

I'm assuming that this is the bug that you mentioned Bro 2.2 will fix. When is Bro 2.2 expected to be released?

We aren't quite sure yet, we're furiously working on several big features now. We're

  .Seth

6

Shit, didn't finish that thought. Nothing ground breaking was going to follow. :slight_smile:

  .Seth

7

Hey Seth,

I checked out that commit, and it fixes the error, but there is also no output when I run it.

I modified the new version with some print statements to trace what’s going on:

exec-test.bro

@load ./exec

event bro_init()
{
print “hello”;
Exec::run(“date”, function(r: Exec::Result) {
print “test”;
if ( ! r?$stdout )
{
print “nothing?!?”;
return;
}

for ( i in r$stdout )
{
print r$stdout[i];
print r$stdout;
}
});
}

exec.bro

function run(cmd: string, cb: function(r: Result))
{
print “run”;
local tmpfile = “/tmp/bro-exec-” + unique_id("");
system(fmt(“touch %s_done”, tmpfile));
system(fmt(“touch %s_stdout”, tmpfile));
system(fmt(“touch %s_stderr”, tmpfile));

Sleep for 1 sec before writing to the done file to avoid race conditions

This makes sure that all of the data is read from

system(fmt("%s 2>>%s_stderr 1>> %s_stdout; echo “exit_code:${?}” > %s_done; sleep 1; echo “done” >> %s_done", cmd, tmpfile, tmpfile, tmpfile, tmpfile));

results[tmpfile] = [];
callbacks[tmpfile] = cb;

schedule 1msec { start_watching_files(tmpfile) };
print “run finished”;
print cmd;
print results;
}

When I run that, I get the following output:
$ bro -r test.pcap exec-test.bro
hello
run
run finished
date
{
[/tmp/bro-exec-CRmEOhHjsgk] = [exit_code=0, stdout=, stderr=]
}

Something is happening, though:
$ ls /tmp/
bro-exec-Uvha209VjSk_done bro-exec-Uvha209VjSk_stderr bro-exec-Uvha209VjSk_stdout
$ cat /tmp/bro-exec-Uvha209VjSk_stdout
Thu Feb 21 2013

But, that data never makes it to the output in the bro script.

I’m curious why “test” never gets printed.

8

Bro's shutting down before it gets a chance to. :slight_smile:

When you run Bro, load the frameworks/communication/listen script. That will cause Bro not to shut down right after starting up and will give your script a chance to run.

  .Seht

9

Hmm…I tried it two ways, with no luck:

$ bro -r test.pcap /usr/local/bro-2.1/share/bro/policy/frameworks/communication/listen.bro exec-test.bro
hello
run
run finished
date
{
[/tmp/bro-exec-DdEgoyU0zwf] = [exit_code=0, stdout=, stderr=]
}

and

$ cat exec-test.bro
@load ./exec
@load frameworks/communication/listen

event bro_init()
{
print “hello”;
Exec::run(“date”, function(r: Exec::Result) {
print “test”;
if ( ! r?$stdout )
{
print “nothing?!?”;
return;
}

for ( i in r$stdout )
{
print r$stdout[i];
print r$stdout;
}
});
}

$ bro -r test.pcap exec-test.bro
hello
run
run finished
date
{
[/tmp/bro-exec-f6eBToBcMd6] = [exit_code=0, stdout=, stderr=]
}

Is there another way to load the listen script?

10

Oh, I think it's because you're reading a packet capture. When reading packets from a file you can't enable the communication framework.

Just try taking out the -r argument. You should be able to just put frameworks/communication/listen on the command line too instead of the full file path. Also, Bro won't terminate until you kill it.

  .Seth

11

Cool – that did the trick. This is really good stuff.

I decided to try using this against other bro events, besides just bro_init():

$ cat exec-test.bro
@load ./exec

event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count)
{
Exec::run("./hello", function(r: Exec::Result)
{
if ( ! r?$stdout )
{
print “nothing?!?”;
return;
}

for ( i in r$stdout )
{
print r$stdout[i];
}
});
}

./hello is just a hello world program.

So, when I test that, I see the following.

$ bro -r test.pcap exec-test.bro
Hello World
Hello World
Hello World
nothing?!?
ERROR: 1361476370.202590 no such index (Exec::results[Exec::name]) (././exec.bro, line 25)
[…]

This got me wondering – why would exec-test.bro ever have a case where (! r?$stdout) is true, when I have a program that absolutely returns output every time it’s run? (And then print out “nothing?!?”)

For convenience:

exec.bro

21 event Exec::stdout_line(description: Input::EventDescription, tpe: Input::Event, s: string)
22 {
23 local name = sub(description$name, /[^]*$/, “”);
24
25 if ( ! results[name]?$stdout )
26 results[name]$stdout = vector(s);
27 else
28 results[name]$stdout[|results[name]$stdout|] = s;
29 }

It’s also worth noting that exec.bro really trashes the /tmp directory at this point.

exec.bro

94 function run(cmd: string, cb: function(r: Result))
95 {
96 local tmpfile = “/tmp/bro-exec-” + unique_id("");
97 system(fmt(“touch %s_done”, tmpfile));
98 system(fmt(“touch %s_stdout”, tmpfile));
99 system(fmt(“touch %s_stderr”, tmpfile));
100 # Sleep for 1 sec before writing to the done file to avoid race conditions
101 # This makes sure that all of the data is read from
102 system(fmt("%s 2>>%s_stderr 1>> %s_stdout; echo “exit_code:${?}” > %s_done; sleep 1; echo “done” >> %s_done", cmd, tmpf
103
104 results[tmpfile] = [];
105 callbacks[tmpfile] = cb;
106
107 schedule 1msec { start_watching_files(tmpfile) };
108 }

That chunk of code probably needs something like this:

system(fmt(“rm %s_done”, tmpfile));
system(fmt(“rm %s_stdout”, tmpfile));
system(fmt(“rm %s_stderr”, tmpfile));

I’m just not sure where it should go.

-Chris

12

This got me wondering -- why would exec-test.bro ever have a case where (! r?$stdout) is true, when I have a program that absolutely returns output every time it's run? (And then print out "nothing?!?")

You don't have to do that check if you know your script will have something on stdout. I may even make stdout an empty vector by default (as opposed to null). I can see that argument making sense.

system(fmt("rm %s_done", tmpfile));
system(fmt("rm %s_stdout", tmpfile));
system(fmt("rm %s_stderr", tmpfile));

I'm just not sure where it should go.

You're running old code. :slight_smile: I may not have finished taking care of that yet in the version you're running. Hopefully this will be in Bro's master branch soon but using when statements and just generally being much nicer.

  .Seth

13

Hey Seth,

Cool. I pulled the most recent code from github. It cleans up /tmp, the code looks cleaner – with more when statements – but it also causes bro to consume 100% CPU and the only way I can kill bro is by doing a Ctrl-Z and then a kill -9. I never get output from the command I add to Exec::run, so I added some print statements to try to trace where things go off the tracks:

exec-test.bro

1 @load ./exec
2
3
4 event bro_init()
5 {
6 print “hello”;
7 when ( local result = Exec::run([$cmd=“ls /”]) )
8 {
9 print “it ran?!?”;
10 if ( result?$stdout )
11 print result$stdout;
12 if ( result?$files )
13 print result$files;
14 }
15 }

exec.bro

140 function run(cmd: Command): Result
141 {
142 print “hi”;

Then, when I run the following:
$ bro /usr/local/bro-2.1/share/bro/policy/frameworks/communication/listen.bro exec-test.bro
hello

So, it looks like bro gets hung up before it can get into Exec::run.

-Chris

14

The best chance you have of running that code is if you also build Bro against the topic/jsiwek/ticket946 branch. Follow [1] if you want to track the status of that branch getting merged in to git/master.

    Jon

[1] http://tracker.bro-ids.org/bro/ticket/946