Underflow Considered Harmful

Or: How 13 billion became 1.844674e+19 before becoming 0.

After sending amounts totaling over 13 billion thru Sumstats, a value of 0 came out the other end of the sausage factory, but only for one specific data item. Debugging this required lots of well placed print statements, and wild speculation on my part as to what possibly could be broken…

The values being thrown in to be summed are incremental differences from a previous observation, which should be zero or a positive number in the range of several K, so a ‘count’ variable was used. However, for some reason, this value came up negative (or should have) due to (in decreasing likelihood) logic error in script, one of bro’s dark corners, or bro bug. The reason for the negativity is still TBD. But, in the world of unsigned 64-bit values (aka bro ‘count’ variables) there is no negativity, only positivity, and an unsigned underflow creates a number just below 2**64 ~ 1.844674e+19 …

Well, Sumstats tallies in doubles, and naturally this figure (1.844674e+19) dominated the total. In fact, additional increments to this total pushed the total value to be greater than 2**64 (with loss of precision, as doubles only keep 53 bits).

In the processing step at the Sumstats epoch, the value was converted back to a count using the double_to_count() function which the cheatsheet warns returns 0, if the double value is <0.0, but it actually returns 264-double (with a runtime error), and for values > 264 it returns 0 with no runtime error :frowning:

So, there it is, a value that should have been about 13 billion became 1.844674e+19 and then became 0.

A few suggestions:

  1. Conversion routines should saturate at respective minima/maxima of the type being converted to (possibly with runtime error).

  2. Underflow of the ‘count’ type is almost invariably a bug, and should trigger a runtime error. Overflow, similarly, although in practice it seems much less likely to occur as most scripts are dealing with integers considerably less than 2**64. A similar argument could be made for ‘int’. With some operations, it is difficult to detect overflow/underflow, but for simple add and subtract, it is relatively easy.

  3. Documentation to match behavior.

Jim