Rounding is a good way to handle that.
The issue is finite precision calculations involving numbers that
require infinite precision for exact representation. For example,
you would see the same issue in decimal with the VBA Currency data type
(4 decimal places) where
(2/3)-(1/3) = 0.6667-0.3333 = 0.0001
instead of zero.
In your specific case, the binary approximations are
0.349999999999999977795539507496869191527366638183 59375
-0.340000000000000024424906541753443889319896697998 046875
---------------------------------------------------------
0.009999999999999953370632965743425302207469940185 546875
compared to the binary representation for 0.1, which is
0.100000000000000002081668171172168513294309377670 2880859375
I posted VBA code in
http://groups.google.com/group/micro...fb95785d1eaff5
which would give you 28 figures of the decimal equivalent to the binary
approximation of these numbers, but neither that nor full accuracy is
required to roughly predict the degree of approximation in the final
answer. Help for Excel documents Excel's limit of 15 digit accuracy, so
the problem can be viewed as
0.350000000000000???
-0.340000000000000???
---------------------
0.010000000000000???
so it would have been sufficient to round to 15 decimal places, but you
are right that that is often overkill. For example, money calculations
usually have no more than 2 decimal digits of input, and (with the
possible exception of interest calculations) results can usually be
rounded to 2 decimal digits.
Jerry
sportsguy wrote:
i had this problem, where 0.35 - 0.34 < 0.1
so i rounded to the 13th decimal place on all computations, and
all is fine. . .
however, most times, you don't need that many decimals, so round to the
desired level of accuracy. . .
sportsguy