View Single Post
  #6   Report Post  
Posted to microsoft.public.excel.misc
joeu2004 joeu2004 is offline
external usenet poster
 
Posts: 2,059
Default Compensating for Excel rounding errors

On Nov 24, 2:56*pm, CB wrote:
It also works if I use ROUND in C1:C3 and change D4 accordingly:
=IF(C4="","",IF(C40.5,"TOO HIGH",IF(C4<-0.5,"TOO LOW","O.K.")))


This is closer to the right approach. I would also round the average,
computed in C4. So the formulas should be (presumably C1 is copied
into C2 and C3):

C1: =IF(A1="","",ROUND(B1-A1,1))

C4: =IF(C1="","",ROUND(AVERAGE(C1:C3),1))

Regarding your first posting, you wrote:
numbers in A1:B3 are entered with only one decimal place


But unfortunately, most of the time, WYSI-not-WYG when it comes to
numbers with decimal fractions. Most such numbers are not stored
exactly internally. This leads to small numerical "errors". This is
a limitation of the technology that Excel relies on -- as do most
applications (but not all).

In your example, the problemmatic numbers are in A3:B3. 16.1 and 15.6
are stored internally as exactly
16.10000000000000142108547152020037174224853515625 and
15.59999999999999964472863211994990706443786621093 75. When they are
subtracted, the result is exactly
0.500000000000001776356839400250464677810668945312 5.

As you can see (and you learned empirically), the result is not
exactly 0.5. In fact, Excel will display it as 0.500000000000002 when
formatted as Number with 15 dp.


Regarding your first work-around, you wrote:
=IF(C4="","",IF(ROUND(C40.5,1),"TOO HIGH",
IF(ROUND(C4<-0.5,1),"TOO LOW","O.K.")))


That is nonsensical. For example, you are rounding the boolean result
of C40.5, which is 0 or 1. I suspect you want to write:

=if(C4="", "", if(round(C4,1)0.5, "TOO HIGH", ...etc...)))

Perhaps you did just that in your spreadsheet, and you simply had a
typo when you entered the formula into your posting. Always cut-and-
paste examples to avoid such mistakes in the future.

HTH.