View Single Post
  #3   Report Post  
Posted to microsoft.public.excel.programming
michdenis michdenis is offline
external usenet poster
 
Posts: 135
Default Type of variables and their effect on result...

Thanks for this explanation. I already knew it !

If your suggestion is true, Why do we get different results
simply by using and defining or not the type of the variables as
suggested by my question ?

More than that, if you put this formula in a cell the
result is good : = Int(37.7266 * 10000)
= 377266

In a VBA window, X = Int(37.7266 * 10000)
X = 377265

Is the binary systems of numbers compute differently in each case
based on the presentation of the formula ?

If anyone has a supplement of information, I would appreciate.

Thanks for your collaboration.










"Ron Rosenfeld" a écrit dans le message de news:
...
On Fri, 29 Dec 2006 18:25:27 -0500, "MichDenis" wrote:

Hello,

3 ways to make a simple operation (same)

My question : Why the result is not the same ? any explination ?

1 - )
X = Int(37.7266 * 10000)
Result = 377265

2 - )
A = 37.7266
B = 10000
X = Int(A * b)
Result = 377266

3 - )
Suppose we use define the type of these variables ::
Dim x As Long
Dim A As Double, B As Integer
A = 37.7266
B = 10000
x = Int(A * B)
Result = 377265

Thanks in advance for your time and your collaboration.





It has to do with the inherent inaccuracies in the IEEE standard for double
precision floating point numbers.

The number 37.7266 cannot be expressed accurately as a binary number. It's
actually the equivalent of something like 37.726599..... so multiplied by 10000
will be 377265.99... and the INT function will return 377265.

However, when you define A as a variant type, I believe the precision
increases, so you wind up with the "correct" answer.

http://www.cpearson.com/excel/rounding.htm discusses rounding errors in Excel.
The concept is equally applicable to VB.




--ron