Home |
Search |
Today's Posts |
#1
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
What, exactly, is the problem? Computer floating point
arithmetic is not exact. 0.999999999999943 is very close to 1 (5.7E-12% deviation is negligible for most applications). |
#2
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
Please note that C++ does it right!
#include <iostream #include <math.h using namespace std; int main() { double tempRound; double tempInt, tempDec; double number; number = 64.1; tempRound =floor(number*10+0.5)/10; tempInt = (int)(tempRound); tempDec = 10*(tempRound - tempInt); cout << "tempRound = " << tempRound << endl; cout << "tempInt = " << tempInt << endl; cout << "tempDec = " << tempDec << endl; system("pause"); return 0; } gives the right result: tempRound = 64.1 tempInt = 64 tempDec = 1 Hmmm. Can one trust VBA or not? Karl -----Original Message----- What, exactly, is the problem? Computer floating point arithmetic is not exact. 0.999999999999943 is very close to 1 (5.7E-12% deviation is negligible for most applications). . |
#3
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
On Thu, 24 Jul 2003 at 03:18:29, Karl Itschen (Karl Itschen
) wrote: Please note that C++ does it right! So does VBA if you understand how to manipulate data types. I think C++ may use more bytes to store Dim number, tempRound As Double Dim tempInt As Long Dim tempDec As Long number = 64.111 tempRound = number tempInt = tempRound tempDec = (10 * (tempRound - tempInt)) Debug.Print "tempRound = " & tempRound Debug.Print "tempInt = " & tempInt Debug.Print "tempDec = " & tempDec tempRound = 64.1 tempInt = 64 tempDec = 1 -- Mike |
#4
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
On Thu, 24 Jul 2003 at 03:18:29, Karl Itschen (Karl Itschen
) wrote: Please note that C++ does it right! So does VBA if you know how to manipulate the data types Dim number, tempRound As Double Dim tempInt As Long Dim tempDec As Long number = 64.111 tempRound = number tempInt = tempRound tempDec = (10 * (tempRound - tempInt)) Debug.Print "tempRound = " & tempRound Debug.Print "tempInt = " & tempInt Debug.Print "tempDec = " & tempDec tempRound = 64.111 tempInt = 64 tempDec = 1 I don't know C++, but my guess is that 'cout' has some default value of number of dp to display -- Mike |
#5
Posted to microsoft.public.excel.programming
|
|||
|
|||
Can excel calculate?
"Karl Itschen" wrote in message ... Please note that C++ does it right! #include <iostream #include <math.h using namespace std; int main() { double tempRound; double tempInt, tempDec; double number; number = 64.1; tempRound =floor(number*10+0.5)/10; tempInt = (int)(tempRound); tempDec = 10*(tempRound - tempInt); cout << "tempRound = " << tempRound << endl; cout << "tempInt = " << tempInt << endl; cout << "tempDec = " << tempDec << endl; system("pause"); return 0; } gives the right result: tempRound = 64.1 tempInt = 64 tempDec = 1 Well of course it does The default precison used for doubles by cout is only 6 decimal places Try cout.precision(15); if you want a real comparison. http://msdn.microsoft.com/library/de...recisio n.asp Keith |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
how to calculate e^2 in excel | Excel Discussion (Misc queries) | |||
How can I calculate Ker and Kei using Excel | Excel Worksheet Functions | |||
Excel Won't Calculate | Excel Worksheet Functions | |||
Excel won't calculate | Excel Discussion (Misc queries) | |||
calculate age in excel so I can add it | Excel Discussion (Misc queries) |