Can excel calculate?
Binary rounding error. Because data is stored in binary
code in a computer there are certain decimal values that
don't quite work out properly when you perform
mathematical operations on them. Binary is inherently a
bad numbering system because it can really only handle
integer values well. In fact, it's so bad that:
64.1-64=0.999999999999943
64.2-64=0.200000000000003
64.3-64=0.299999999999997
64.4-64=0.400000000000006
64.5-64=0.5 (Hey, this one works!)
64.6-64=0.599999999999994
64.7-64=0.700000000000003
64.8-64=0.799999999999997
64.9-64=0.900000000000006
-----Original Message-----
Strange strange... look at this simple excel-VBA code
Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1
tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))
Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec
End Sub
Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64) and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:
tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943
Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl
.
|