Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 5,651
Default Can excel calculate?

On Thu, 24 Jul 2003 01:42:40 -0700, "Karl Itschen"
wrote:

Strange strange... look at this simple excel-VBA code

Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub

Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64) and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:

tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943

Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl


As others have pointed out, the problem is in the 15 digit precision and binary
storage that the specifications call for in storing Double precision numbers.

However, you can get around this limitation, depending on the magnitude of the
number, by using the Decimal data type. For example,

==================
Sub test()
Dim tempRound As Double
Dim tempInt As Double, tempDec As Variant
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (CDec(tempRound) - CDec(tempInt)))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub
=====================
tempRound = 64.1
tempInt = 64
tempDec = 1
===================

See HELP for details on acceptable magnitudes.



--ron
  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 77
Default Can excel calculate?

Just to mention Excel's other method to do a limited amout of arbitrary
precision is the use of Currency.
Depending on your range of numbers, this might work...

Sub Demo()
Dim tempRound As Currency
Dim tempInt As Currency
Dim tempDec As Currency
Dim number As Currency

number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = 10 * (tempRound - tempInt)

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec
End Sub

tempRound = 64.1
tempInt = 64
tempDec = 1


I like Ron's idea. You can also have Excel automatically work at the higher
precision if you want with the following idea...

Sub Demo2()
Dim tempRound
Dim tempInt
Dim tempDec
Dim number

number = CDec(64.1)

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = 10 * (tempRound - tempInt)

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec
End Sub

tempRound = 64.1
tempInt = 64
tempDec = 1

--
Dana DeLouis
Using Windows XP & Office XP
= = = = = = = = = = = = = = = = =


"Karl Itschen" wrote in message
...
Cool thanks, that works fine!
Karl
-----Original Message-----
On Thu, 24 Jul 2003 01:42:40 -0700, "Karl Itschen"


wrote:

Strange strange... look at this simple excel-VBA code

Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub

Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64)

and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:

tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943

Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl


As others have pointed out, the problem is in the 15

digit precision and binary
storage that the specifications call for in storing

Double precision numbers.

However, you can get around this limitation, depending on

the magnitude of the
number, by using the Decimal data type. For example,

==================
Sub test()
Dim tempRound As Double
Dim tempInt As Double, tempDec As Variant
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (CDec(tempRound) - CDec(tempInt)))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub
=====================
tempRound = 64.1
tempInt = 64
tempDec = 1
===================

See HELP for details on acceptable magnitudes.



--ron
.



Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to calculate e^2 in excel sif Excel Discussion (Misc queries) 3 April 4th 23 02:09 PM
How can I calculate Ker and Kei using Excel Binder Excel Worksheet Functions 2 March 2nd 09 06:49 AM
calculate age in excel so I can add it Joe Phillips Excel Discussion (Misc queries) 1 July 11th 07 02:04 PM
how can i calculate uk apr with excel Paul Connolly Excel Worksheet Functions 2 June 10th 05 06:40 PM
Can excel calculate? Mike[_31_] Excel Programming 3 July 24th 03 02:11 PM


All times are GMT +1. The time now is 07:16 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"