Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 2,059
Default Compensating for Excel rounding errors

PS....

On Nov 24, 4:44*pm, I wrote:
In your example, the problemmatic numbers are in A3:B3. *16.1 and 15.6
are stored internally as exactly
16.10000000000000142108547152020037174224853515625 and
15.59999999999999964472863211994990706443786621093 75.


I did not intend to imply that the numbers in A1:B2 are stored
exactly. In fact, they are not. But coincidentally, their difference
is exactly 0.5, even when we put parentheses around the expression,
which side-steps Excel's attempt to ameliorate such numerical
"errors", and even when we do the computation in VBA.

I want to reiterate that is purely by coincidental. The result of B3-
A3 is much more common.

PS: I also notice that I computed A3-B3, not B3-A3 as you did. That
does not make any difference other than the sign of the result.
  #2   Report Post  
Posted to microsoft.public.excel.misc
CB CB is offline
external usenet poster
 
Posts: 97
Default Compensating for Excel rounding errors



"joeu2004" wrote:

PS....

On Nov 24, 4:44 pm, I wrote:
In your example, the problemmatic numbers are in A3:B3. 16.1 and 15.6
are stored internally as exactly
16.10000000000000142108547152020037174224853515625 and
15.59999999999999964472863211994990706443786621093 75.


I did not intend to imply that the numbers in A1:B2 are stored
exactly. In fact, they are not. But coincidentally, their difference
is exactly 0.5, even when we put parentheses around the expression,
which side-steps Excel's attempt to ameliorate such numerical
"errors", and even when we do the computation in VBA.


I understood your intention. I took programming courses ages and ages ago
and was familiar with how computers stored numbers but that knowledge was
forgotten quite some time ago. This experience has been a refresher. :)


I want to reiterate that is purely by coincidental. The result of B3-
A3 is much more common.

PS: I also notice that I computed A3-B3, not B3-A3 as you did. That
does not make any difference other than the sign of the result.

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Rounding errors when a "5" is the third decimal place using formul Jbagger Excel Discussion (Misc queries) 4 March 28th 07 01:52 AM
How do I correct rounding errors in Excel formulas? C. Van Dam Excel Worksheet Functions 1 August 29th 06 04:37 AM
Excel Throwing Circular Errors When No Errors Exist MDW Excel Worksheet Functions 1 August 10th 06 02:15 PM
Can rounded numbers be summed without rounding errors? chelseab Excel Discussion (Misc queries) 1 February 8th 06 04:26 AM
Rounding Errors Help mattflow Excel Discussion (Misc queries) 2 August 12th 05 08:10 PM


All times are GMT +1. The time now is 02:10 AM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"