View Single Post
  #12   Report Post  
Posted to microsoft.public.excel.misc
 
Posts: n/a
Default Strange Calculation Error in Excel (2)

Hi Jerry,

Extended precision packages like xlPrecision also do not solve the problem, they just push it farther out (though they do have their uses). <


This is true of some extended precision packages, but not true of
xlPrecision. xlPrecision never converts anything to binary. xlPrecision
does all arithmetic in base 10. Using xlPrecision results in no more
binary conversion errors than doing arithmetic in longhand (i.e.,
pencil and paper).

Also, I may be a little foggy on the definition, but I'm not sure that
it's quite accurate to refer to xlPrecision as "extended" precision.
xlPrecision is *arbitrary* precision in the sense that the underlying
algorithms have no maximum number of significant digits. xlPrecision's
maximum of 32,767 significant digits is simply the result of Excel's
limit of that many characters in a cell. I could easily extend that by
allowing array-entering into multiple cells, but I haven't done that
because I haven't heard of anyone wanting more than 32,767. If I were
to do so, the next limit I would reach is the largest text string
variable allowed, which would be a little over 2 billion significant
digits. Even that could be easily overcome by using arrays. Again, the
reason I haven't done it is because I don't think anyone would be
interested in that many significant digits.


Thanks,

Greg Lovern
http://PrecisionCalc.com
Eliminate Hidden Spreadsheet Errors