View Single Post
  #8   Report Post  
Posted to microsoft.public.excel.misc
JE McGimpsey JE McGimpsey is offline
external usenet poster
 
Posts: 4,624
Default Standard deviation calculation error...

Thanks for the correction/amplification!

You're right - I should have said that the result is limited by the
combination of the inherent limits of finite precision representation
*and* the quality of implementation of the algorithm(s) involved.



In article ,
Jerry W. Lewis wrote:

While you are correct that this is a direct result of the limitations
of IEEE standard double precision ... and the change in 2003 to a
2-pass algorithm for STDEV, VAR, etc is a welcome numerical
improvement, there is still room for improvement in Excel's
algorithms here.