Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 21
Default Can excel calculate?

What, exactly, is the problem? Computer floating point
arithmetic is not exact. 0.999999999999943 is very close
to 1 (5.7E-12% deviation is negligible for most
applications).
  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 3
Default Can excel calculate?

Please note that C++ does it right!

#include <iostream
#include <math.h
using namespace std;
int main()
{
double tempRound;
double tempInt, tempDec;
double number;
number = 64.1;

tempRound =floor(number*10+0.5)/10;
tempInt = (int)(tempRound);
tempDec = 10*(tempRound - tempInt);

cout << "tempRound = " << tempRound << endl;
cout << "tempInt = " << tempInt << endl;
cout << "tempDec = " << tempDec << endl;

system("pause");
return 0;
}

gives the right result:

tempRound = 64.1
tempInt = 64
tempDec = 1

Hmmm. Can one trust VBA or not?
Karl
-----Original Message-----
What, exactly, is the problem? Computer floating point
arithmetic is not exact. 0.999999999999943 is very close
to 1 (5.7E-12% deviation is negligible for most
applications).
.

  #3   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 87
Default Can excel calculate?

On Thu, 24 Jul 2003 at 03:18:29, Karl Itschen (Karl Itschen
) wrote:
Please note that C++ does it right!

So does VBA if you understand how to manipulate data types. I think C++
may use more bytes to store

Dim number, tempRound As Double

Dim tempInt As Long
Dim tempDec As Long

number = 64.111
tempRound = number
tempInt = tempRound
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec


tempRound = 64.1
tempInt = 64
tempDec = 1
--
Mike
  #4   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 87
Default Can excel calculate?

On Thu, 24 Jul 2003 at 03:18:29, Karl Itschen (Karl Itschen
) wrote:
Please note that C++ does it right!

So does VBA if you know how to manipulate the data types

Dim number, tempRound As Double

Dim tempInt As Long
Dim tempDec As Long

number = 64.111
tempRound = number
tempInt = tempRound
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

tempRound = 64.111
tempInt = 64
tempDec = 1

I don't know C++, but my guess is that 'cout' has some default value of
number of dp to display
--
Mike
  #5   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 170
Default Can excel calculate?


"Karl Itschen" wrote in message
...
Please note that C++ does it right!

#include <iostream
#include <math.h
using namespace std;
int main()
{
double tempRound;
double tempInt, tempDec;
double number;
number = 64.1;

tempRound =floor(number*10+0.5)/10;
tempInt = (int)(tempRound);
tempDec = 10*(tempRound - tempInt);

cout << "tempRound = " << tempRound << endl;
cout << "tempInt = " << tempInt << endl;
cout << "tempDec = " << tempDec << endl;

system("pause");
return 0;
}

gives the right result:

tempRound = 64.1
tempInt = 64
tempDec = 1


Well of course it does

The default precison used for doubles by cout
is only 6 decimal places

Try

cout.precision(15);

if you want a real comparison.

http://msdn.microsoft.com/library/de...recisio n.asp

Keith




Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to calculate e^2 in excel sif Excel Discussion (Misc queries) 3 April 4th 23 02:09 PM
How can I calculate Ker and Kei using Excel Binder Excel Worksheet Functions 2 March 2nd 09 06:49 AM
Excel Won't Calculate davem97132 Excel Worksheet Functions 1 June 5th 08 01:51 AM
Excel won't calculate Keith Excel Discussion (Misc queries) 2 May 12th 08 09:56 AM
calculate age in excel so I can add it Joe Phillips Excel Discussion (Misc queries) 1 July 11th 07 02:04 PM


All times are GMT +1. The time now is 02:37 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"