Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 38
Default Can excel calculate?

Binary rounding error. Because data is stored in binary
code in a computer there are certain decimal values that
don't quite work out properly when you perform
mathematical operations on them. Binary is inherently a
bad numbering system because it can really only handle
integer values well. In fact, it's so bad that:

64.1-64=0.999999999999943
64.2-64=0.200000000000003
64.3-64=0.299999999999997
64.4-64=0.400000000000006
64.5-64=0.5 (Hey, this one works!)
64.6-64=0.599999999999994
64.7-64=0.700000000000003
64.8-64=0.799999999999997
64.9-64=0.900000000000006

-----Original Message-----
Strange strange... look at this simple excel-VBA code

Sub test()
Dim tempRound As Double
Dim tempInt, tempDec As Double
Dim number As Double
number = 64.1

tempRound = Round(number, 1)
tempInt = Int(tempRound)
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

End Sub

Ok this code is supposed to have the integer part
of "number" in the "tempInt" variable (ie tempInt=64) and
the decimal part rounded to the first decimal value in
the "tempDec" variable (ie tempDec=1).
BUT it outputs:

tempRound = 64.1
tempInt = 64
tempDec = 0.999999999999943

Hmmm anything wrong here perhaps!!!!
Any comment?!?
Karl
.

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
How can I calculate Ker and Kei using Excel Binder Excel Worksheet Functions 2 March 2nd 09 06:49 AM
Excel won't calculate Keith Excel Discussion (Misc queries) 2 May 12th 08 09:56 AM
How to get Excel to calculate [email protected] Excel Discussion (Misc queries) 3 November 6th 06 03:11 AM
how do i calculate in excel % Available conley Excel Discussion (Misc queries) 2 April 21st 06 08:34 PM
how can i calculate uk apr with excel Paul Connolly Excel Worksheet Functions 2 June 10th 05 06:40 PM


All times are GMT +1. The time now is 02:59 PM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"