View Single Post
  #2   Report Post  
Posted to microsoft.public.excel.misc
Harlan Grove Harlan Grove is offline
external usenet poster
 
Posts: 733
Default Single vs Double

Jeff wrote...
I have the code

Sub Test()
Dim X as Single
Dim Y as DOuble
Dim Z as Double

x=.54
Y=300
z=X*Y

End Sub()

Z should equal 162, but it is 162.000006437302

Why is that? . . .


Classic floating point rounding error. 0.54 can't be represented
exactly as a binary fraction.

. . . Does that mean that all the variables have to be double to
get the exact answer?


Not necessarily. If you had declared z as a single, z would have wound
up with the exact value 162 fortuitously.

When using floating point values, you need to check for APPROXIMATE
equality rather than exact equality, e.g.,

Abs(z - 162) < 0.5

rather than

z = 162

This has always been so, and it'll continue to be so for finite
precision floating point.