# Problems parallelizing btests

**URL:** <https://community.zeek.org/t/problems-parallelizing-btests/2245>\
**Category:** Development\
**Tags:** development\
**Created:** [March 1, 2012, 1:54am UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245 "2012-03-01T01:54:21Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\
**Post date:** [March 1, 2012, 1:54am UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/1 "2012-03-01T01:54:21Z")

</div>

btest:topic/robin/parallel has a version of btest that can run tests  
in parallel. That works pretty well, except for two issues with Bro's  
standard tests:

- The coverage analysis doesn't like running in parallel, it messes  
&nbsp;&nbsp;&nbsp;up the state file. Jon, do you think we could get that to work  
&nbsp;&nbsp;&nbsp;somehow?

- The communication tests can't be parallelized because they use the  
&nbsp;&nbsp;&nbsp;same port. The btest in the branch supports groups of tests that  
&nbsp;&nbsp;&nbsp;are executed sequentially, which solves the problem. But  
&nbsp;&nbsp;&nbsp;unfortunately that comes at the expenses of losing much of the  
&nbsp;&nbsp;&nbsp;speed-up that parallelizing would otherwise be able to achieve  
&nbsp;&nbsp;&nbsp;(because the communication tests take the longest).

&nbsp;&nbsp;&nbsp;I'm wondering if we could randomize the ports being used in some  
&nbsp;&nbsp;&nbsp;form. But not sure how that would look like.

Other than these, it's actually pretty cool to run the tests in  
parallel. 🙂

Btw, that branch also has a new option to rerun only tests that failed  
last time.

Robin

---

<div class="post-metadata">

**Author:** ![Seth\_Hall3](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Seth\_Hall3](https://community.zeek.org/u/Seth_Hall3)\
**Post date:** [March 1, 2012, 2:21am UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/2 "2012-03-01T02:21:05Z")

</div>

> I'm wondering if we could randomize the ports being used in some  
> &nbsp;&nbsp;form. But not sure how that would look like.

How about we read in the port to use as an environment variable? Btest could just set that before running each test (maybe we could limit it to only set it for communication tests?).

> Other than these, it's actually pretty cool to run the tests in  
> parallel. 🙂

Looking forward to playing with it!

&nbsp;&nbsp;.Seth

---

<div class="post-metadata">

**Author:** ![Slagell\_Adam\_J](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@Slagell\_Adam\_J](https://community.zeek.org/u/Slagell_Adam_J)\
**Post date:** [March 1, 2012, 3:26pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/3 "2012-03-01T15:26:50Z")

</div>

Do they really need to be random or just unique for each test? If the latter, maybe the port could be derived from the test names which themselves could be numbered.

---

<div class="post-metadata">

**Author:** ![Siwek\_Jon](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Siwek\_Jon](https://community.zeek.org/u/Siwek_Jon)\
**Post date:** [March 1, 2012, 4:07pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/4 "2012-03-01T16:07:58Z")

</div>

> - The coverage analysis doesn't like running in parallel, it messes  
> &nbsp;&nbsp;up the state file. Jon, do you think we could get that to work  
> &nbsp;&nbsp;somehow?

Yeah, what I'm thinking is to have Brofiler.cc pass BRO\_PROFILER\_FILE through mkstemp() instead of fopen() and then change that env. var. in the btest.cfg files to use some .XXXXX suffix so that each bro instance writes coverage state to a unique file. Then I've already got a script in testing/scripts/coverage-calc that glues coverage files together. I'll go ahead and try that real quick and commit if it works out.

+Jon

---

<div class="post-metadata">

**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\
**Post date:** [March 1, 2012, 4:46pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/5 "2012-03-01T16:46:14Z")

</div>

> How about we read in the port to use as an environment variable?  
> Btest could just set that before running each test

> Do they really need to be random or just unique for each test? If the  
> latter, maybe the port could be derived from the test names which  
> themselves could be numbered.

Just unique is indeed fine. How about a combination of the two: broctl  
numbers all tests internally and passes the current test's number on  
via an environment variable. The test can then derive a port from  
that, and it's a bit more general in that we might end up using the  
number for other purposes too.

The remaining piece is then using the environment variable to  
configure the port for Bro and Broccoli. Would be nice if we could do  
that centrally somehow, not manually in each test needing it.

---

<div class="post-metadata">

**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\
**Post date:** [March 2, 2012, 5:45pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/6 "2012-03-02T17:45:34Z")

</div>

This branch could use some testing as well, btw. I've also  
restructured things internally a bit. README isn't updated yet but the  
new options are:

&nbsp;&nbsp;-j THREADS, --jobs=THREADS  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;number of threads to run tests in simultaniously; 0  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;disables threading  
&nbsp;&nbsp;-g GROUP, --group=GROUP  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;execute only test of given group, or '-' for those  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;without any group  
&nbsp;&nbsp;-r, --rerun Execute commands for tests that failed last time

(For the Bro tests, one currently needs to remove the  
BRO\_PROFILER\_FILE variable from btest.cfg to make it work.)

Robin

---

<div class="post-metadata">

**Author:** ![Daniel\_Thayer](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@Daniel\_Thayer](https://community.zeek.org/u/Daniel_Thayer)\
**Post date:** [March 5, 2012, 4:15pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/7 "2012-03-05T16:15:36Z")

</div>

It appears that OutputHandlers.py isn't getting installed. When I run  
btest (after doing "python setup.py install"), I see this error:

ImportError: No module named OutputHandlers

---

<div class="post-metadata">

**Author:** ![robin](https://yyz1.discourse-cdn.com/flex011/user_avatar/community.zeek.org/robin/32/599_2.png) [@robin](https://community.zeek.org/u/robin)\
**Post date:** [March 5, 2012, 4:17pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/8 "2012-03-05T16:17:25Z")

</div>

Oops, yeah. I always run it directly from the source directory, which  
is why I didn't notice. Will fix.

Robin

---

<div class="post-metadata">

**Author:** ![Daniel\_Thayer](https://avatars.discourse-cdn.com/v4/letter/d/8dc957/32.png) [@Daniel\_Thayer](https://community.zeek.org/u/Daniel_Thayer)\
**Post date:** [March 6, 2012, 10:48pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/9 "2012-03-06T22:48:29Z")

</div>

It seems that using the "-f" option (without "-b" or "-v") now  
prevents the status message for each test from being output.  
The following patch should fix this bug:

--- a/btest  
+++ b/btest  
@@ -927,11 +927,9 @@ if Options.diagfile:

&nbsp;&nbsp;if Options.verbose:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;output\_handlers += [Verbose(Options, )]

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex011/uploads/zeek/original/1X/f09d732bc2cc7c7cc7e35db67cf4e1d5233ce7a7.png) [@system](https://community.zeek.org/u/system)\
**Post date:** [May 6, 2022, 3:40pm UTC](https://community.zeek.org/t/problems-parallelizing-btests/2245/10 "2022-05-06T15:40:13Z")

</div>

Closed on May 6, 2022, 3:40pm UTC


