SHA256 Hash File Analyzer

I was wondering if anyone can tell me why the sha256 hash functionality isn’t turned on by default for the files log.

I am working on something and needed to turn it on. I normally only use Bro to process pcap files offline and have never used it on a live network.

Does it cause performance issues?

Thanks,

Shawn

When I was setting the default behavior a few years ago, I did some very weak testing and noticed that if I had md5 and sha1 turned on, the performance impact was ~1%, but it jumped up somewhere between 3-4% when I enabled SHA256. That measurement should be revisited sometime soon though and perhaps even better measurements done to see if that performance impact is still there.

Generally though, there is nothing in place which is stopping you from enabling SHA256 file hashes.

  .Seth

Thanks for the information. I have it turned on in my offline system, but not sure how to measure performance.

I’m curious if anyone has this turned on at scale, on production systems? If so, can you speak to the performance impacts Seth mentioned below?

Seth,

any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn’t see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16?

Thanks,

ryan

I'm curious if anyone has this turned on at scale, on production systems? If so, can you speak to the performance impacts Seth mentioned below?

Seth,
any thoughts if this would be the same with 2.5 as it was when you originally posted? I didn't see anything specific about it in release notes, so would we be correct to assume the SHA256 analyzer would probably perform the same as what you saw back in Feb 16?

The analyzer really just delegates to openssl to do all the hashing, so you should be able to use openssl to gauge the performance impact:

$ openssl speed md5 sha1 sha256
Doing md5 for 3s on 16 size blocks: 6879766 md5's in 3.00s
Doing md5 for 3s on 64 size blocks: 5066897 md5's in 3.00s
Doing md5 for 3s on 256 size blocks: 2814019 md5's in 3.00s
Doing md5 for 3s on 1024 size blocks: 1016906 md5's in 3.00s
Doing md5 for 3s on 8192 size blocks: 147949 md5's in 3.00s
Doing sha1 for 3s on 16 size blocks: 7763902 sha1's in 3.00s
Doing sha1 for 3s on 64 size blocks: 5420584 sha1's in 3.00s
Doing sha1 for 3s on 256 size blocks: 2965390 sha1's in 3.00s
Doing sha1 for 3s on 1024 size blocks: 1054003 sha1's in 3.00s
Doing sha1 for 3s on 8192 size blocks: 147866 sha1's in 3.00s
Doing sha256 for 3s on 16 size blocks: 4896135 sha256's in 3.00s
Doing sha256 for 3s on 64 size blocks: 2682706 sha256's in 3.00s
Doing sha256 for 3s on 256 size blocks: 1131865 sha256's in 3.00s
Doing sha256 for 3s on 1024 size blocks: 342980 sha256's in 3.00s
Doing sha256 for 3s on 8192 size blocks: 45549 sha256's in 3.00s
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Sep 27 13:37:25 UTC 2016
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md5 36692.09k 108093.80k 240129.62k 347103.91k 403999.40k
sha1 41407.48k 115639.13k 253046.61k 359766.36k 403772.76k
sha256 26112.72k 57231.06k 96585.81k 117070.51k 124379.14k

On a different machine with a different distribution and newer CPUs I get

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md5 50302.28k 175259.63k 373751.13k 536014.85k 632668.98k
sha1 62768.79k 170994.71k 358746.20k 509927.77k 569868.29k
sha256 50775.24k 110530.33k 188262.14k 241865.05k 270240.43k

The 1024 byte block size and below would be the most relevant for bro. Unless you're using jumbo frames bro shouldn't be doing much with blocks larger than 1500.

I would expect to still see a similar performance hit from enabling SHA256, but I don't really know. Someone needs to test it. Justin's point about the performance of OpenSSL is most of the picture, but there is still some additional overhead due to having another analyzer attached to a file so I wouldn't go totally off of the openssl benchmark testing.

  .Seth