Thread: Accuracy
View Single Post
  #3   Report Post  
Posted to microsoft.public.excel.misc
Jim Thomlinson Jim Thomlinson is offline
external usenet poster
 
Posts: 5,939
Default Accuracy

That is not an XL issue but rather it is a computer issue in general. Decimal
numbers will only have a certain degree of accuracy based on processing and
memory limitations of PC's. Google IEEE and floating point for an in depth
discussion of the issue. In general you will get 15 significant digits of
accuracy. The numbers after that or of dubious quality. That being said with
15 digit of accuracy you can measure the distance from here to the sun to
less than a millimeter.
--
HTH...

Jim Thomlinson


"Ad Pronk" wrote:

Using Excel 2003 I "discover" a strange thing. Placing brackets around a term
introduces an inaccuracy. Is this normal?
1-(1/cos(x)*cos(a.t+x) is nil for t=0 as it should be. But I use this term
in a series, so I have to place brackets around the term:
(1-(1/cos(x)*cos(a.t+x))
And surprisingly for t=0 I get the answer 1.1 E(-14).
It's small but I expect at least a value below E(-22) or 0

Reactions are welcomed
Ad Pronk