LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #1   Report Post  
Posted to microsoft.public.excel.misc
CB CB is offline
external usenet poster
 
Posts: 97
Default Compensating for Excel rounding errors

Hi all,

Im having some issues with the following, which I stumbled across purely by
coincidence. The worksheet was set up by others with more Excel experience
than I so Im not sure as to the best workaround.

FYI, numbers in A1:B3 are entered with only one decimal place. Cells C1:C4
and D4 are calculated. Formulas are noted at the bottom of the post.

The formula in D4 ***appears*** to be incorrect. Since C1:C3 are EXACTLY
-0.5, so is C4. According to the formula, D4 should display O.K. Instead,
it displays Too Low.

I believe the problem lies with the way Excel is storing the numbers (at the
binary level). When troubleshooting, I set cells A1:C3 (and C4) to display
with 30 decimals. I then found the calculation for C3 is not EXACTLY -0.5 as
it should be. Instead, there is a 2 in the fifteenth decimal place.

I tried using the ROUND function in C1:C4 but D4 displays FALSE. D4 still
displays FALSE if I use ROUND in that formula as well.

What am I missing?

Thanks for any and all assistance!

Regards,
Chris



A B C
D
Standard Unit Diff
1 16.3 15.8 -0.5
2 16.2 15.7 -0.5
3 16.1 15.6 -0.5
4 Average Difference -0.5
Too Low

The formula for Diff in C1:C3 is: =IF(A1="","",B1-A1).
The formula for Average Difference in C4 is:
=IF(C1="","",AVERAGE(C1:C3))
The formula for D4 is:
=IF(C4="","",IF(C40.5,"TOO HIGH",IF(C4<-0.5,"TOO LOW",IF(C4-0.5,"O.K."))))



 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Rounding errors when a "5" is the third decimal place using formul Jbagger Excel Discussion (Misc queries) 4 March 28th 07 01:52 AM
How do I correct rounding errors in Excel formulas? C. Van Dam Excel Worksheet Functions 1 August 29th 06 04:37 AM
Excel Throwing Circular Errors When No Errors Exist MDW Excel Worksheet Functions 1 August 10th 06 02:15 PM
Can rounded numbers be summed without rounding errors? chelseab Excel Discussion (Misc queries) 1 February 8th 06 04:26 AM
Rounding Errors Help mattflow Excel Discussion (Misc queries) 2 August 12th 05 08:10 PM


All times are GMT +1. The time now is 09:01 AM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright 2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"