Standard deviation calculation error...
Thanks for the correction/amplification!
You're right - I should have said that the result is limited by the
combination of the inherent limits of finite precision representation
*and* the quality of implementation of the algorithm(s) involved.
In article ,
Jerry W. Lewis wrote:
While you are correct that this is a direct result of the limitations
of IEEE standard double precision ... and the change in 2003 to a
2-pass algorithm for STDEV, VAR, etc is a welcome numerical
improvement, there is still room for improvement in Excel's
algorithms here.
|