Handle Floating point rounding errors, efficiently
I like using Currency since most comparisons like this are "usually" not
over 2 places to the right of the decimal point.
Sub Demo()
Dim n As Currency
[A1] = 0
[A2].Formula = "=A1+0.05"
[A2].AutoFill [A2:A21]
n = [A21]
Debug.Print "[A21]=1", n = 1 'Now True
Debug.Print 1 - [A21] '-2.22044604925031E-16
Debug.Print
End Sub
[a21] '-2.22044604925031E-16
For the Op, although cell A21 shows 1.000..., Excel works with only 15
digits. Your math coprocessor is using 17. Excel can not display your
coprocessor's internal number of 1.0000000000000002.
However, when you do the math subtraction, Excel is padding the hidden
ending 2 with zero's in base two. When converted to decimal, the displayed
numbers is not very meaningful.
Therefore, looking at you hidden '2'
? 2*2^-53
2.22044604925031E-16
--
Dana DeLouis
Win XP & Office 2003
"JE McGimpsey" wrote in message
...
Don't know about the fastest, most efficient, but
Const dEPSILON As Double = 1E-10 'set appropriately
If Abs(Cells(21, 1).Value - 1) < dEPSILON Then ...
will work. Using Range references (Cells(21,1)) instead of the evaluate
method ([A21]) will likely speed up your code more than which method of
testing you use.
In article ,
"Peter T" <peter_t@discussions wrote:
The various methods I've tried to eliminate "false" results slow the
process
at least to some extent, inevitably I suppose. My purpose and the point
of
this post is to do exactly what you suggest:
"design your programs to be tolerant of the approximations inherent in
floating point numbers"
But in the fastest most efficient way possible.
|