LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #12   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 4,393
Default A wrong calculation by excel

This so-called error is the result of the computer converting decimals
numbers (base 10) to binary (base 2)
See one or more of these
IEEE 754
Floating-point arithmetic may give inaccurate results in Excel
http://support.microsoft.com/kb/78113/en-us
(Complete) Tutorial to Understand IEEE Floating-Point Errors
http://support.microsoft.com/kb/42980
What Every Computer Scientist Should Know About Floating Point
http://docs.sun.com/source/806-3568/ncg_goldberg.html
http://www.cpearson.com/excel/rounding.htm
Visual Basic and Arithmetic Precision
http://support.microsoft.com/default...NoWebContent=1

--
Bernard V Liengme
Microsoft Excel MVP
http://people.stfx.ca/bliengme
remove caps from email

"Captain" wrote in message
...
Your problem is exactly the same as mine. I found that there was an error
in
the 13th decimal place when subtracting exactly $1,015.00 from $1,183.30!
Get around this error by writing a formula that accepts a calculation as
correct if it matches the expected answer to within $.01 or within .01%,
etc.
example: IF(AND(A1<B1+.01,A1B1-.01),"OK","error").
--
E. Polizzi


"BobD" wrote:

I am using Excel 2003 Professional Edition. If you include the following
formula into a cell, it produces a result of 8.74999999999999 if you
expand
the decimal places to 16. The correct answer is 8.75. The formula is
((17.15-17)*10/6+17)-((8.3-8)*10/6+8). This only happens if you use 17.15
and
8.3. I came across this when using a process to convert times to
decimals
and then subtract one from the other. Anyone have any ideas?
--
BobD



 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Wrong amount or calculation error? Owen[_2_] Excel Discussion (Misc queries) 1 April 11th 07 06:45 PM
DMAX wrong calculation AlexKoL Excel Worksheet Functions 3 August 25th 06 06:43 PM
wrong calculation in excel? formula outcome < cell value?? mcclaud Excel Worksheet Functions 2 August 17th 06 05:20 PM
Excel is totaling wrong Daxy Excel Worksheet Functions 1 July 10th 06 07:38 PM
excel totaling wrong brian Excel Discussion (Misc queries) 5 May 12th 06 03:46 PM


All times are GMT +1. The time now is 03:49 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"