View Single Post
  #10   Report Post  
Posted to microsoft.public.excel.misc
[email protected] dougaj4@gmail.com is offline
external usenet poster
 
Posts: 10
Default Cosine Function Returns Wrong Answer!

On Jul 6, 9:44 am, Gordon Arnaut
wrote:
Harlan,

Thanks for the heads up about GAMMMALN.

Rounding errors can be annoying. On a density altitude calculator I worked
up, it comes up with a rounding error of about a foot for every couple of
thousand feet increase in altitude. More of an annoyance than an impediment,
but still...

I spent a lot of time going through the formulas trying to get rid of it, to
no avail.

Microsoft needs to do a better job with this product.



Gnumeric may do a better job with Gammaln, but it gives exactly the
same results as Excel with the Cos function.

I'd be interested to see how the errors in your density altitude
calculator came about. I don't see how an error of that magnitude
could be due to rounding at 16 significant figures, unless you are
subtracting very big numbers with a very small difference, in which
case any other standard PC-based package will suffer from similar
problems.