ExcelBanter - Can excel calculate?

64.1 (and most non-integer numbers) cannot be represented exactly in
(IEEE standard) binary instead you get
64.099999999999994315658113919198513031005859375
or roughly
64.1000000000000???
where the uncertain portion (???) can be either positive or negative,
but rounds to the desired number.
64.1000000000000???
-64.0000000000000
--------------------
0.1000000000000???
you got
0.0999999999999943
which, when multiplied by 10 gives ...

The math is correct, but the input numbers are only approximate, so it
should be no surprise that the answer is also only approximate.

C++ appears to give the right answer for one of 4 possible reasons. Either

- you are not displaying full precision in C

- you used more than IEEE double precision (such as the "long double"
data type)

- C is applying a fuzz factor, something like the difference between
=A1-A2 and =(A1-A2) in Excel, for results that are nearly zero. In that
case, you should evaluate what it is doing very carefully, to insure
that it doesn't significantly reduce the precision of your calculations

- C optimized the code to do everything before returning a result from
the processor (practically equivalent to using "long double" for the
calculations and returning the final result as a "double")

Remember, both Excel and C are letting the math coprocessor do the
actual calculation, so fundamentally there is no difference between
arithmetic in C or Excel, providing that you use comparable data types.

Jerry

Karl Itschen wrote:

Strange strange... look at this simple excel-VBA code

Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub

Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64) and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:

tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943

Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl