Home |
Search |
Today's Posts |
#5
![]()
Posted to microsoft.public.excel.programming
|
|||
|
|||
![]()
"Jerry W. Lewis" wrote:
A more direct confirmation of Martin's claim is Sub tryit() Dim x As Double x = 2# ^ 1023 MsgBox x * x / x End Sub If the expression x*x/x were not evaluated in 10-byte registers, the calculation would necessarily overflow. That demonstrates that VBA __sometimes__ relies on the 80-bit FP registers. I wanted to confirm that that explains why my computation of the average, ergo the variance and standard deviation, is "exact" to 64 bits, whereas the computation by AVERAGE() is not. To do that, I believe my recursive example (or any other way to thwart VBA optimization) was necessary to prove that point. This is what judges refer to as an argument "on point". It only works with inline expressions involving explicitly declared Double's and using only the basic 4 arithmetic operations. Thanks for the insight. Question: Does this work on non-Intel processors under Windows or on a Mac? Since the (2006) Mac uses Intel processors, the question is moot :-). Just kidding .... A Google search provides some answers or hints. But since I cannot confirm the information, I don't want to "speculate" here. |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
Unacceptable floating point errors | Excel Discussion (Misc queries) | |||
Convert floating point to Hours and Minutes ?? | Excel Discussion (Misc queries) | |||
Floating point problem?? | Excel Programming | |||
setting a floating decimel point | New Users to Excel | |||
Floating Point Functions | Excel Programming |