View Single Post
  #16   Report Post  
Posted to microsoft.public.excel.misc
 
Posts: n/a
Default Strange Calculation Error in Excel

Hi Fred --

I understand the reason why decimals will not always convert perfectly
to binary, but the similar problem that I kept running into (primarily
using Lotus 1-2-3 Ver 1A -- I've only recently had to start working in
Excel, and so far have had only one occasion to do a work-around in
that program) is that I would get floating point errors when adding
WHOLE NUMBERS that mathematically should sum to zero. Surely whole
numbers should convert perfectly to binary. I frequently needed to
test whether a sum equaled zero, but the best fix I could come up with
was "if(abs([formula])<0.001,[do A],[do B]).

Why adding and subtracting whole numbers would yield these same 15th
decimal place discrepancies is completely beyond my comprehension.