View Single Post
  #3   Report Post  
Posted to microsoft.public.excel.misc
JE McGimpsey JE McGimpsey is offline
external usenet poster
 
Posts: 4,624
Default Standard deviation calculation error...

No, STDEV "works" for fractional as well as whole numbers, but you are
seeing the limitation of IEEE Double Precision Floating Point math,
which is used by XL and most other spreadsheets. There's no patch,
because XL can't determine whether the small rounding error is real or
an artifact.

Nearly all numbers cannot be exactly represented in a finite number of
binary digits, just as the number 1/3 cannot be represented in a finite
number of decimal digits. So you should expect small errors due to
rounding. You can use ROUND() to filter out these errors, e.g.,

=ROUND(STDEV(A2:A61),9)

See

http://cpearson.com/excel/rounding.aspx

for more details



In article ,
Jacky wrote:

Is this due to limitation of the excel formula (STDEV only works for whole
numbers)? Or is there any patch for this error?