Home |
Search |
Today's Posts |
#1
![]()
Posted to microsoft.public.excel.misc
|
|||
|
|||
![]()
INT(10^14+0.5) results in exactly 100000000000001. INT(10^14-1+0.5)
results in exactly 99999999999999, as it should. It seems that for all values of 10^14 and higher [1], INT() behaves like ROUND(,0). Why is that? I think this has been discussed before. But I cannot find the previous discussion(s) or any MS KB on the subject. Pointers would be appreciated, as well as an explanation. I don't think this has to do with IEEE 64-bit double floating-point representation. VBA Int() works fine in all cases. I also don't think the problem has to do with Excel 64-bit arithmetic v. VBA 80-bit arithmetic. At least, any such explanation is not obvious to me, having looked at the binary representation. I also tried storing intermediate subexpression results into type Double variables, with no adverse effect. IEEE 64-bit double floating-point can represent integral numbers up to 2^53 accurately. 10^14 requires only 46 of the 52 mantissa bits. On the other hand, I suspect it is no accident that 10^14 is 15 significant digits, the Excel display limit. Does Excel implement INT() by converting the argument to text, then trimming the digits to the right of the decimal point? (Surprise!) End Notes: [1] Obviously, I have not looked at all integers above and below 10^14. I found 10^14 by a binary search between 1 and 13^14. A more accurate statement is: INT() works fine for 20 values in the range between 1 and 10^14-1, and INT() mishaves for 31 values between 10^14 and 13^14. I ass-u-me continuity because the binary search worked as expected. |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
Getting Excel MDI/SDI/Excel Instance to behave more logically | Setting up and Configuration of Excel | |||
Make Cell behave Like Word | Excel Discussion (Misc queries) | |||
How do I get filtering to behave consistently? | Excel Worksheet Functions | |||
Is there a way to count higher than 511 in binary in excel? | Excel Worksheet Functions | |||
How do I ROUND() round off decimals of a column dataset? | Excel Worksheet Functions |