LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #2   Report Post  
Harlan Grove
 
Posts: n/a
Default

"GCD_Dilemma" wrote...
How come the following in Excel 2003:
=GCD((1.4-1)*10,10)
yields 1 rather than 2, but
=GCD((0.4)*10,10)
yields the correct answer 2 ?
=GCD(4,10)
also correctly yields 2 !


Because by default Excel uses IEEE double precision floating point math, so
ANY calculation involving fractions other than sums of negative powers of 2
(e.g., 1/2, 1/4, 1/8, 1/16, etc.) is subject to roundoff error in the same
way that representing 1/3 as 0.3333 or any other finite string of 3s to the
right of the decimal point is.

Since GCD and LCM only make sense in the context of integers, ensure that
your arguments to either are integers. For instance,

=GCD(ROUND((1.4-1)*10,0),10)

returns 2, as expected.


 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Averaging function Sarah Excel Discussion (Misc queries) 0 January 18th 05 04:09 PM
Accessing a Function Bill Martin -- (Remove NOSPAM from address) Excel Discussion (Misc queries) 4 January 1st 05 05:46 AM
Function in XL or in VBA for XL that pulls numeric digits from a t Nate Oliver Excel Discussion (Misc queries) 0 December 14th 04 04:57 PM
change function variable prompts?? thinkingfield Excel Worksheet Functions 1 November 8th 04 04:01 PM
SUMIF(AND) FUNCTION Saariko Excel Worksheet Functions 9 October 28th 04 11:52 AM


All times are GMT +1. The time now is 08:30 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"