Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 837
Default Can excel calculate?

64.1 (and most non-integer numbers) cannot be represented exactly in
(IEEE standard) binary instead you get
64.099999999999994315658113919198513031005859375
or roughly
64.1000000000000???
where the uncertain portion (???) can be either positive or negative,
but rounds to the desired number.
64.1000000000000???
-64.0000000000000
--------------------
0.1000000000000???
you got
0.0999999999999943
which, when multiplied by 10 gives ...

The math is correct, but the input numbers are only approximate, so it
should be no surprise that the answer is also only approximate.

C++ appears to give the right answer for one of 4 possible reasons. Either

- you are not displaying full precision in C

- you used more than IEEE double precision (such as the "long double"
data type)

- C is applying a fuzz factor, something like the difference between
=A1-A2 and =(A1-A2) in Excel, for results that are nearly zero. In that
case, you should evaluate what it is doing very carefully, to insure
that it doesn't significantly reduce the precision of your calculations

- C optimized the code to do everything before returning a result from
the processor (practically equivalent to using "long double" for the
calculations and returning the final result as a "double")

Remember, both Excel and C are letting the math coprocessor do the
actual calculation, so fundamentally there is no difference between
arithmetic in C or Excel, providing that you use comparable data types.

Jerry

Karl Itschen wrote:

Strange strange... look at this simple excel-VBA code

Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub

Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64) and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:

tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943

Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl


  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 170
Default Can excel calculate?


"Jerry W. Lewis" wrote in message
...
64.1 (and most non-integer numbers) cannot be represented exactly in
(IEEE standard) binary instead you get
64.099999999999994315658113919198513031005859375
or roughly
64.1000000000000???
where the uncertain portion (???) can be either positive or negative,
but rounds to the desired number.
64.1000000000000???
-64.0000000000000
--------------------
0.1000000000000???
you got
0.0999999999999943
which, when multiplied by 10 gives ...

The math is correct, but the input numbers are only approximate, so it
should be no surprise that the answer is also only approximate.

C++ appears to give the right answer for one of 4 possible reasons.

Either

- you are not displaying full precision in C


This is actually the case, he was using cout which defaults to 6 decimal
places.

Keith


Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to calculate e^2 in excel sif Excel Discussion (Misc queries) 3 April 4th 23 02:09 PM
Excel won't calculate Keith Excel Discussion (Misc queries) 2 May 12th 08 09:56 AM
calculate age in excel so I can add it Joe Phillips Excel Discussion (Misc queries) 1 July 11th 07 02:04 PM
How to get Excel to calculate [email protected] Excel Discussion (Misc queries) 3 November 6th 06 03:11 AM
Can excel calculate? Mike[_31_] Excel Programming 3 July 24th 03 02:11 PM


All times are GMT +1. The time now is 08:23 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"