Evaluating compression algorithms for Bro logs and PCAPs

Recently I've heard about Google's new Brotli algorithm outperforming so
many others. I wanted to figure out whether it holds up to his words and
wrote a small benchmark [1]. This benchmark compares various algorithms
along space (compresion ratio) and time (throughput in bytes/sec). As
input I've taken both PCAPs and Bro logs.

Except for the tradeoff plot, I hope the figures speak for themselves.
The tradeoff figure visualizes the space-time spectrum of the
compression algorithm. It plots space savings [2] against throughput
during compression. The further a point sits in the top-right corner,
the more of a best-of-both-worlds algorithm it is. ZSTD appears to have
that property for my tested input.

Brotli compresses at a higher ratio, but is also much slower. So for
high-throughput applications that pump out logs, it may not be
well-suited if a minimal CPU footprint has priority. Even thought Brotli
would have been a nice name for the best algorithm for Bro logs, it only
ranks second after ZPAQ in terms of space savings. However, Brotli is
3 orders of magnitude faster than ZPAQ during decompression, rendering
it a solid choice when you have enough cycles available during


[1] https://github.com/mavam/compbench
[2] https://en.wikipedia.org/wiki/Data_compression_ratio