View Single Post
  #11   Report Post  
Posted to microsoft.public.excel.worksheet.functions
 
Posts: n/a
Default Rounding error in Stdev function result.

I wrote:
having (54%) more bits of precision will always
diminish those effects -- at least until the (intermediate)
result is stored into lower-precision variables.


Well, that's not correct. First, the mantissa is only 23%
more bits of precision (64 bits). Second, I forget that the
original 52-bit mantissa is effectively simply zero-extended.
So the 80-bit FP result is not necessarily better than the
64-bit result. But I don't believe it will ever be worse.