View Single Post
  #2   Report Post  
Posted to microsoft.public.excel.misc
Bernard Liengme
 
Posts: n/a
Default Extra 1 showing up in 15th decimal place

This the result of the way computers (not just Excel) store numbers. We use
decimal numbers (we have 10 digits), computers use binary numbers (a switch
can be on or off). The IEEE convention allows computers to store about 15
decimal places. Some decimal values cannot be exactly represented in this
limit so we get 'round-off' errors.
Rather than test if (a-b) = 0.204, test if ROUND(a-b,3)=0.204
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email


"nsc" wrote in message
...

I'm using Excel 2000 with SR-1.

I enter a value of 9.1173 in cell A1, and 9.3213 in B1.

I set C1 equal to B1 - A1.

I select the entire sheet, format cells as numbers with 4 decimal
places.

I highlight column C, and conditional format it with two conditions. If
the cell value equals .204 the cell should color itself green, if the
cell value does not equal .204 the cell should color itself red.

The cell is colored red.

Expanding the number of decimal places for the entire worksheet shows
me a 1 in the fifteenth decimal place of cell C1.

This happens for a whole range of values that I'm using. The all should
end up being .204 when the subtraction is done, but over half of them
are doing the same thing.

Additionally, if you enter A1 as 24.324 and B1 as 24.528, the
subtraction calculation for C1 will result in .2039999999999970 and it
should be .204.

Anybody know what's going on here?


--
nsc
------------------------------------------------------------------------
nsc's Profile:
http://www.excelforum.com/member.php...o&userid=29648
View this thread: http://www.excelforum.com/showthread...hreadid=493565