Here's an interesting view of Bro. I compiled it with the gcc flags to enable gcov output. After I ran Bro on a short tracefile with a few analyzers enabled, I processed the gcov output files with lcov and created this view of the executed code in that run.
http://www.icir.org/seth/bro-lcov/src/index.html
I think the gcov output could be a good starting point for writing tests since it's pretty easy to find parts of code that aren't executed commonly. At least we could use it to develop the test suite to make sure that we are actually testing as many lines of code as possible. Right now there seems to be lots of code that isn't executed (or at least wasn't with my tiny test).
It should actually work for verifying that we are testing the script execution code too. Check the Expr.cc file and you can see certain script level functionality (driven at the script level at least) that was never executed, like RecordMatchExpr::ExprDescribe[1] (who would have ever thought to test that?).
The gcc/g++ flags to enable gcov output are "-fprofile-arcs -ftest-coverage". Are those something that should be added to the build system? I don't think they need added to the configure script, but maybe to the CMakeLists.txt file?
Here are the docs I followed for getting gcov output and processing with lcov:
http://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html
http://ltp.sourceforge.net/coverage/lcov/readme.php
gcov is probably installed everywhere already, but lcov is a non-gcc package and probably need installed. lcov was in macports though.
.Seth
1. http://www.icir.org/seth/bro-lcov/src/Expr.cc.gcov.html#3744
The gcc/g++ flags to enable gcov output are "-fprofile-arcs
-ftest-coverage". Are those something that should be added to the
build system? I don't think they need added to the configure script,
but maybe to the CMakeLists.txt file?
If that's really all that's needed I think you could get away with just passing the flags through environment variables in the standard way to the configure script, e.g.:
CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" ./configure
Passing the flags only works for the first-time configure of a given build directory.
The lcov doc you referenced also mentioned linking to -lgcov. If something like that's necessary, it probably makes sense to add a --enable-code-coverage option to the configure wrapper that triggers CMake to search for the required stuff and, if it's all there, add the necessary compile and link flags, else fail.
- Jon
CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" ./configure
Passing the flags only works for the first-time configure of a given build directory.
Ah, that would actually work fine then. It seems like it would mostly be used for the automated tests (after we make sure we have tests that hit the code paths in the first place) to make sure that we continue to hit all code paths possible with our tests and we could use it provide assurance that we continue to hit new code paths in our tests.
The lcov doc you referenced also mentioned linking to -lgcov. If something like that's necessary, it probably makes sense to add a --enable-code-coverage option to the configure wrapper that triggers CMake to search for the required stuff and, if it's all there, add the necessary compile and link flags, else fail.
lcov isn't required for gcov, it just generates the nice html output from the raw gcov files but for the automated tests I imagine that we would just use the gcov generated files. It's probably good to avoid stuffing too many options in the configure script anyway.
.Seth