LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #1   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 2,059
Default Why does Excel INT(x) behave like ROUND(x,0) for 10^14 and higher?

INT(10^14+0.5) results in exactly 100000000000001. INT(10^14-1+0.5)
results in exactly 99999999999999, as it should.

It seems that for all values of 10^14 and higher [1], INT() behaves
like ROUND(,0). Why is that?

I think this has been discussed before. But I cannot find the
previous discussion(s) or any MS KB on the subject. Pointers would be
appreciated, as well as an explanation.

I don't think this has to do with IEEE 64-bit double floating-point
representation. VBA Int() works fine in all cases. I also don't
think the problem has to do with Excel 64-bit arithmetic v. VBA 80-bit
arithmetic. At least, any such explanation is not obvious to me,
having looked at the binary representation. I also tried storing
intermediate subexpression results into type Double variables, with no
adverse effect.

IEEE 64-bit double floating-point can represent integral numbers up to
2^53 accurately. 10^14 requires only 46 of the 52 mantissa bits.

On the other hand, I suspect it is no accident that 10^14 is 15
significant digits, the Excel display limit. Does Excel implement
INT() by converting the argument to text, then trimming the digits to
the right of the decimal point? (Surprise!)



End Notes:

[1] Obviously, I have not looked at all integers above and below
10^14. I found 10^14 by a binary search between 1 and 13^14. A more
accurate statement is: INT() works fine for 20 values in the range
between 1 and 10^14-1, and INT() mishaves for 31 values between 10^14
and 13^14. I ass-u-me continuity because the binary search worked as
expected.
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Getting Excel MDI/SDI/Excel Instance to behave more logically mswlogo Setting up and Configuration of Excel 2 April 23rd 23 11:43 AM
Make Cell behave Like Word Ian8276 Excel Discussion (Misc queries) 2 November 26th 07 11:24 PM
How do I get filtering to behave consistently? Steve Excel Worksheet Functions 0 April 25th 07 11:50 AM
Is there a way to count higher than 511 in binary in excel? havocdragon Excel Worksheet Functions 1 October 15th 05 05:14 AM
How do I ROUND() round off decimals of a column dataset? Højrup Excel Worksheet Functions 2 January 12th 05 10:50 AM


All times are GMT +1. The time now is 04:10 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"