#1   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 1
Default Calculation Error?

Sometime the calculation is not accurate, Eg:
11.12 - 10.5 = 0.62

but i get 0.619999999999999

Is it system error or regional setting's problem~~~~
  #2   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 418
Default Calculation Error?

Wei Ming from singapore wrote:
Sometime the calculation is not accurate, Eg:
11.12 - 10.5 = 0.62
but i get 0.619999999999999
Is it system error or regional setting's problem


Neither. The problem is two-fold:

1. The displayed values are probably rounded, if they are the result of
a formula. For example, "11.12" might be any value between 11.115 and
11.1249...9. You can confirm this by formatting Number with many
decimal places (Format Cells Number).

2. Binary computers cannot represent most fractions accurately. So
even if you type "11.12" into a cell and Excel displays "11.120...0",
the value in the computer is not exactly 11.12. Excel makes
adjustments when evaluating some formulas in an attempt to hide these
anomalies. But it is not unusual for the "inaccurate" representation
to have visible consequences.

You can usually compensate by using ROUND() judiciously.

  #3   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 3,440
Default Calculation Error?

Look he

http://support.microsoft.com/kb/78113


--
Kind regards,

Niek Otten
Microsoft MVP - Excel

"Wei Ming from singapore" <Wei Ming from wrote in message
...
| Sometime the calculation is not accurate, Eg:
| 11.12 - 10.5 = 0.62
|
| but i get 0.619999999999999
|
| Is it system error or regional setting's problem~~~~


  #4   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 837
Default Calculation Error?

Specifically, the IEEE double precision binary approximation to 11.12 is
11.11999999999999921840299066388979554176330566406 25
which you can't see directly, because (as documented) Excel displays no more
than 15 significant figures. But you can see the approximation indirectly,
since when you subtract 10.5 (exactly representable), you get
6.199999999999992184029906638897955417633056640625
which Excel displays to 15 figures as
6.19999999999999

Jerry

"Niek Otten" wrote:

Look he

http://support.microsoft.com/kb/78113


--
Kind regards,

Niek Otten
Microsoft MVP - Excel

"Wei Ming from singapore" <Wei Ming from wrote in message
...
| Sometime the calculation is not accurate, Eg:
| 11.12 - 10.5 = 0.62
|
| but i get 0.619999999999999
|
| Is it system error or regional setting's problem~~~~

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
error bars display Amy Charts and Charting in Excel 1 December 22nd 06 11:22 PM
2007 Beta TR calculation error / bug [email protected] Excel Discussion (Misc queries) 1 November 7th 06 07:43 PM
error message when opening excel 2000 Lem Excel Discussion (Misc queries) 3 August 31st 06 11:06 PM
Error when save as CSV with VBA password protection on [email protected] Excel Discussion (Misc queries) 0 August 1st 06 10:50 PM
Standard Error Bars. A S-D Excel Discussion (Misc queries) 1 July 5th 06 11:52 AM


All times are GMT +1. The time now is 07:00 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"