View Single Post
  #2   Report Post  
Harlan Grove
 
Posts: n/a
Default

"GCD_Dilemma" wrote...
How come the following in Excel 2003:
=GCD((1.4-1)*10,10)
yields 1 rather than 2, but
=GCD((0.4)*10,10)
yields the correct answer 2 ?
=GCD(4,10)
also correctly yields 2 !


Because by default Excel uses IEEE double precision floating point math, so
ANY calculation involving fractions other than sums of negative powers of 2
(e.g., 1/2, 1/4, 1/8, 1/16, etc.) is subject to roundoff error in the same
way that representing 1/3 as 0.3333 or any other finite string of 3s to the
right of the decimal point is.

Since GCD and LCM only make sense in the context of integers, ensure that
your arguments to either are integers. For instance,

=GCD(ROUND((1.4-1)*10,0),10)

returns 2, as expected.