Home |
Search |
Today's Posts |
#1
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
64.1 (and most non-integer numbers) cannot be represented exactly in
(IEEE standard) binary instead you get 64.099999999999994315658113919198513031005859375 or roughly 64.1000000000000??? where the uncertain portion (???) can be either positive or negative, but rounds to the desired number. 64.1000000000000??? -64.0000000000000 -------------------- 0.1000000000000??? you got 0.0999999999999943 which, when multiplied by 10 gives ... The math is correct, but the input numbers are only approximate, so it should be no surprise that the answer is also only approximate. C++ appears to give the right answer for one of 4 possible reasons. Either - you are not displaying full precision in C - you used more than IEEE double precision (such as the "long double" data type) - C is applying a fuzz factor, something like the difference between =A1-A2 and =(A1-A2) in Excel, for results that are nearly zero. In that case, you should evaluate what it is doing very carefully, to insure that it doesn't significantly reduce the precision of your calculations - C optimized the code to do everything before returning a result from the processor (practically equivalent to using "long double" for the calculations and returning the final result as a "double") Remember, both Excel and C are letting the math coprocessor do the actual calculation, so fundamentally there is no difference between arithmetic in C or Excel, providing that you use comparable data types. Jerry Karl Itschen wrote: Strange strange... look at this simple excel-VBA code Sub test() Dim tempRound As Double Dim tempInt, tempDec As Double Dim number As Double number = 64.1 tempRound = Round(number, 1) tempInt = Int(tempRound) tempDec = (10 * (tempRound - tempInt)) Debug.Print "tempRound = " & tempRound Debug.Print "tempInt = " & tempInt Debug.Print "tempDec = " & tempDec End Sub Ok this code is supposed to have the integer part of "number" in the "tempInt" variable (ie tempInt=64) and the decimal part rounded to the first decimal value in the "tempDec" variable (ie tempDec=1). BUT it outputs: tempRound = 64.1 tempInt = 64 tempDec = 0.999999999999943 Hmmm anything wrong here perhaps!!!! Any comment?!? Karl |
#2
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
"Jerry W. Lewis" wrote in message ... 64.1 (and most non-integer numbers) cannot be represented exactly in (IEEE standard) binary instead you get 64.099999999999994315658113919198513031005859375 or roughly 64.1000000000000??? where the uncertain portion (???) can be either positive or negative, but rounds to the desired number. 64.1000000000000??? -64.0000000000000 -------------------- 0.1000000000000??? you got 0.0999999999999943 which, when multiplied by 10 gives ... The math is correct, but the input numbers are only approximate, so it should be no surprise that the answer is also only approximate. C++ appears to give the right answer for one of 4 possible reasons. Either - you are not displaying full precision in C This is actually the case, he was using cout which defaults to 6 decimal places. Keith |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
how to calculate e^2 in excel | Excel Discussion (Misc queries) | |||
Excel won't calculate | Excel Discussion (Misc queries) | |||
calculate age in excel so I can add it | Excel Discussion (Misc queries) | |||
How to get Excel to calculate | Excel Discussion (Misc queries) | |||
Can excel calculate? | Excel Programming |