Home |
Search |
Today's Posts |
#2
![]() |
|||
|
|||
![]()
"GCD_Dilemma" wrote...
How come the following in Excel 2003: =GCD((1.4-1)*10,10) yields 1 rather than 2, but =GCD((0.4)*10,10) yields the correct answer 2 ? =GCD(4,10) also correctly yields 2 ! Because by default Excel uses IEEE double precision floating point math, so ANY calculation involving fractions other than sums of negative powers of 2 (e.g., 1/2, 1/4, 1/8, 1/16, etc.) is subject to roundoff error in the same way that representing 1/3 as 0.3333 or any other finite string of 3s to the right of the decimal point is. Since GCD and LCM only make sense in the context of integers, ensure that your arguments to either are integers. For instance, =GCD(ROUND((1.4-1)*10,0),10) returns 2, as expected. |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
Averaging function | Excel Discussion (Misc queries) | |||
Accessing a Function | Excel Discussion (Misc queries) | |||
Function in XL or in VBA for XL that pulls numeric digits from a t | Excel Discussion (Misc queries) | |||
change function variable prompts?? | Excel Worksheet Functions | |||
SUMIF(AND) FUNCTION | Excel Worksheet Functions |