View Single Post
  #1   Report Post  
Posted to microsoft.public.excel.programming
Mike[_31_] Mike[_31_] is offline
external usenet poster
 
Posts: 38
Default Can excel calculate?

Binary rounding error. Because data is stored in binary
code in a computer there are certain decimal values that
don't quite work out properly when you perform
mathematical operations on them. Binary is inherently a
bad numbering system because it can really only handle
integer values well. In fact, it's so bad that:

64.1-64=0.999999999999943
64.2-64=0.200000000000003
64.3-64=0.299999999999997
64.4-64=0.400000000000006
64.5-64=0.5 (Hey, this one works!)
64.6-64=0.599999999999994
64.7-64=0.700000000000003
64.8-64=0.799999999999997
64.9-64=0.900000000000006

-----Original Message-----
Strange strange... look at this simple excel-VBA code

Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub

Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64) and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:

tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943

Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl
.