View Single Post
  #6   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Jerry W. Lewis Jerry W. Lewis is offline
external usenet poster
 
Posts: 837
Default Excel error:ROUND(1000.2555-999,3)=1.255

Excel is performing exactly as it should here.

Excel and almost all other computer software does binary math. Most
terminating decimal fractions (including .2555) are non-terminating binary
fractions that can only be approximated (just as 1/3 can only be approximated
as a decimal fraction). The precision of Excel's approximation is defined by
the IEEE 754 standard, where the decimal representation of the best binary
approximation to 1000.2555 is
1000.255499999999983629095368087291717529296875. It displays as 1000.2555
because (as documented in Help) Excel will display no more than 15 digits,
but when you subtract 999, you correctly get a value that is visibly less
than 1.2555, and which therefore rounds down.

Since you know the operations that produced your 1.25549999999998, you know
that ROUND(1000.2555-999,4) will do no violence to your calculations while
cleaning up the result of the original binary approximations. That will
produce a value that will round as you expected.

Jerry

"liaosheng" wrote:

ROUND(1000.2555-999,3)=1.255
the correct result is 1.256,why?

----------------
This post is a suggestion for Microsoft, and Microsoft responds to the
suggestions with the most votes. To vote for this suggestion, click the "I
Agree" button in the message pane. If you do not see the button, follow this
link to open the suggestion in the Microsoft Web-based Newsreader and then
click "I Agree" in the message pane.

http://www.microsoft.com/office/comm...et.f unctions