View Single Post
  #3   Report Post  
Posted to microsoft.public.excel.misc
Bernard Liengme[_3_] Bernard Liengme[_3_] is offline
external usenet poster
 
Posts: 1,104
Default is 0.000000000000000055511151231258=0 in excell?

In short: yes!
Welcome to the world of computers where everything is 0 or 1 (binary or base
2)

Excel (and most computer apps other than COBOL) use what is called the IEEE
convention for converting decimal numbers (base 10 --- because we have 10
fingers) to binary (base 2). This uses a finite number of bytes so we get
what is called round off errors since some decimal numbers (like 0.1) have
no exact binary representation (just as the fraction 1/3 has no exact
decimal representation but is 0.33333333333.....)

The net result is that sometimes when 0 is expected we actually get a number
that is just a tiny bit different. We say that Excel has a 'precision of 15
decimal places'. This means that any two numbers that differ only in the
15th decimal, place are to be considered the same

How to avoid? Replace =A1-B1 by =ROUND(A1,B1,12) and your problem will go
away. I am sure you are not being so precise that this rounding will affect
your results adversely unless you do quantum mechanics.

Further reading:
IEEE 754
Chip's clear explanation
http://www.cpearson.com/excel/rounding.htm

Floating-point arithmetic may give inaccurate results in Excel
http://support.microsoft.com/kb/78113/en-us

(Complete) Tutorial to Understand IEEE Floating-Point Errors
http://support.microsoft.com/kb/42980

What Every Computer Scientist Should Know About Floating Point
http://docs.sun.com/source/806-3568/ncg_goldberg.html
http://www.cpearson.com/excel/rounding.htm

Visual Basic and Arithmetic Precision
http://support.microsoft.com/default...NoWebContent=1

Others:
http://support.microsoft.com/kb/214118

http://docs.sun.com/source/806-3568/ncg_goldberg.html
best wishes

--
Bernard V Liengme
Microsoft Excel MVP
http://people.stfx.ca/bliengme
remove caps from email


"Christian Weller" <Christian wrote in
message ...
In excel I am calculating the time difference between two times
I.e. 6:17:36 AM - 6:17:36 AM = 00:00:00 (12:00:00 am depending on
formatting)
Sometimes I get 0 , but often get =0.000000000000000055511151231258 or
-0.000000000000000055511151231258 (number format with 30 decimal places)

Any ideas why?