LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #24   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 45
Default Unacceptable floating point errors

Jeff
You can't have a greater degree of accuracy in a result than was present
in the arguments
..30 - .29 does = .01 to 2 significant digits, beyond that the result is
undefined. .01000000000 is just one of many valid ones

..01 is the 2sd representation of any number from .005 to less than .015.

If you have a need for such high precision then you probably need to use
specialist tools rather than a general purpose product like Excel.

cheers
Simon

Jeff in GA wrote:
David, I appreciate your commitment to the notion that there could not
possibly be a solution to the problem, and that we should all accept
incorrect mathematical results.

Meanwhile, even my fourth grade child knows that .30 - .29 = .0100000000000

But yes, David, let's adapt to the machines, and not make the machines adapt
to us.

Also: the customer is always wrong.

Happy now?

...Jeff

"David Biddulph" wrote:

Perhaps you could tell us what the exact binary representation of 0.29 would
be?
--
David Biddulph

"Jeff in GA" wrote in message
...
This is NOT some calculation to the nth place or an atomic weight.

This is: .29 - .28

Instead of returning the correct answer of 0.01, Microsoft Excel returns
0.00999999999999995

This floating point excuse for can not be blamed for poor product quality.

Does Microsoft ever plan to repair its product, or should it only be used
for calculating federal budget numbers?



 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Convert floating point to Hours and Minutes ?? [email protected] Excel Discussion (Misc queries) 1 August 19th 06 02:26 AM
Excel Throwing Circular Errors When No Errors Exist MDW Excel Worksheet Functions 1 August 10th 06 02:15 PM
Converting 2-place decimal value to floating point decimal number with leading zero Kermit Piper Excel Discussion (Misc queries) 3 March 18th 06 06:20 PM
setting a floating decimel point Rose New Users to Excel 2 April 29th 05 06:10 PM
Unresolved Errors in IF Statements - Errors do not show in results Markthepain Excel Worksheet Functions 2 December 3rd 04 08:49 AM


All times are GMT +1. The time now is 10:33 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"