View Single Post
  #2   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Bernard Liengme
 
Posts: n/a
Default What is the roundoff error in the subtraction function?

This roundoff error results from the way computers (it is not limited to
Excel) store numbers. The IEEE convention requires the decimal number to be
converted to binary with a fixed number of digits. Within this limit some
decimal numbers do no have an exact binary representation.
For more read:
http://support.microsoft.com/default...kb;en-us;78113
http://support.microsoft.com/default...b;en-us;214118
http://www.cpearson.com/excel/rounding.htm
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"Bill Owens" <Bill wrote in message
...
Why does
1.00000000000000000000-0.99950000000000000000=0.000499999999999945
instead of 0.0005? I have 50,000 records that are evenly spaced by 0.0005
and
they appear that way in the column even when viewed with 20 places,
however,
after subtracting one number from the other the difference is not 0.0005.
I
tried using round off to 4 places and that doesn't resolve the problem.
Any
help is appreciated.