#1   Report Post  
Posted to microsoft.public.excel.misc
Leamsi
 
Posts: n/a
Default Error with decimals

Has anyone notice a problem with up decimal, somehow excel adds up more
decimals.
eg.

cell (A1) 5045.3
cell (A2) 5045.03
cell (A3) =A1-A2

Note that the diference will be .27
But excel brings 0.270000000000437
  #2   Report Post  
Posted to microsoft.public.excel.misc
Bernard Liengme
 
Posts: n/a
Default Error with decimals

Welcome to the world of computer math!
Computers (no just Excel) generally store numbers in binary format (base 2)
rather than decimal (base 10) - we have 10 fingers while a computer knows
only ON and OFF.
The conversion is done following the IEEE protocol which specifies 15 digits
(this is a bit of a simplification). Some decimal numbers (real or floating
point numbers with fractional parts but not integers) cannot be exactly
represented in this binary form. So we get "round-off" errors due to the
conversion to and from binary/decimal
Read more at http://support.microsoft.com/kb/78113/en-us
One way to solve this is to use =ROUND(formula, some number less than 15)
as in =ROUND(A1-A2,10)
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"Leamsi" wrote in message
...
Has anyone notice a problem with up decimal, somehow excel adds up more
decimals.
eg.

cell (A1) 5045.3
cell (A2) 5045.03
cell (A3) =A1-A2

Note that the diference will be .27
But excel brings 0.270000000000437



  #3   Report Post  
Posted to microsoft.public.excel.misc
Jerry W. Lewis
 
Posts: n/a
Default Error with decimals

If you want to expand on Bernard's answer, you might find the functions at
http://groups.google.com/group/micro...06871cf92f8465
to be useful. Using them, you will see that the IEEE standard
approximations to 5045.3 and 5045.03 are
5045.3000000000001818989403545856475830078125
5045.0299999999997453414835035800933837890625
whose difference is
0.27000000000043655745685100555419921875
which Excel correctly reports to its documented limit of 15 digits.

Bottom line: the math is right, but most decimal fractions must be
approximated, leading to approximate results.

Jerry

"Leamsi" wrote:

Has anyone notice a problem with up decimal, somehow excel adds up more
decimals.
eg.

cell (A1) 5045.3
cell (A2) 5045.03
cell (A3) =A1-A2

Note that the diference will be .27
But excel brings 0.270000000000437

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Merging Excel data to word, keeps including all decimals on number Henrik Notlev Excel Discussion (Misc queries) 2 January 27th 06 07:30 AM
Convert $ amount from excel to access? No decimals but 00 at end Nicole Excel Worksheet Functions 1 January 4th 06 04:37 PM
Conditional formatting : amount of decimals belgian11 Excel Discussion (Misc queries) 0 December 25th 05 04:47 PM
Comma format button decimals Steve Ruiter Excel Discussion (Misc queries) 0 November 30th 05 06:51 PM
How do I calculate without decimals in excel? upstate Excel Worksheet Functions 1 August 11th 05 09:26 PM


All times are GMT +1. The time now is 05:13 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"