Home |
Search |
Today's Posts |
#12
![]()
Posted to microsoft.public.excel.misc
|
|||
|
|||
![]()
This so-called error is the result of the computer converting decimals
numbers (base 10) to binary (base 2) See one or more of these IEEE 754 Floating-point arithmetic may give inaccurate results in Excel http://support.microsoft.com/kb/78113/en-us (Complete) Tutorial to Understand IEEE Floating-Point Errors http://support.microsoft.com/kb/42980 What Every Computer Scientist Should Know About Floating Point http://docs.sun.com/source/806-3568/ncg_goldberg.html http://www.cpearson.com/excel/rounding.htm Visual Basic and Arithmetic Precision http://support.microsoft.com/default...NoWebContent=1 -- Bernard V Liengme Microsoft Excel MVP http://people.stfx.ca/bliengme remove caps from email "Captain" wrote in message ... Your problem is exactly the same as mine. I found that there was an error in the 13th decimal place when subtracting exactly $1,015.00 from $1,183.30! Get around this error by writing a formula that accepts a calculation as correct if it matches the expected answer to within $.01 or within .01%, etc. example: IF(AND(A1<B1+.01,A1B1-.01),"OK","error"). -- E. Polizzi "BobD" wrote: I am using Excel 2003 Professional Edition. If you include the following formula into a cell, it produces a result of 8.74999999999999 if you expand the decimal places to 16. The correct answer is 8.75. The formula is ((17.15-17)*10/6+17)-((8.3-8)*10/6+8). This only happens if you use 17.15 and 8.3. I came across this when using a process to convert times to decimals and then subtract one from the other. Anyone have any ideas? -- BobD |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
Wrong amount or calculation error? | Excel Discussion (Misc queries) | |||
DMAX wrong calculation | Excel Worksheet Functions | |||
wrong calculation in excel? formula outcome < cell value?? | Excel Worksheet Functions | |||
Excel is totaling wrong | Excel Worksheet Functions | |||
excel totaling wrong | Excel Discussion (Misc queries) |