In the debug window in Excel 2000, why does ? int( 0.6 * 100) give an answer of 59?

I know it's due to binary storage or something - but how do I correct it?

try:

? cint( 0.6 * 100) = 60
? .6*100 \1 = 60



keepITcool

< email : keepitcool chello nl (with @ and .)
< homepage: http://members.chello.nl/keepitcool


(Roger) wrote:

? int( 0.6 * 100)

I've never encountered that before. In XLv.X:

? int( 0.6 * 100)
60


In article ,
(Roger) wrote:

In the debug window in Excel 2000, why does ? int( 0.6 * 100) give an answer
of 59?

I know it's due to binary storage or something - but how do I correct it?

It appears to be caused by come conversion that is being performed:

? int( 0.6 * 100)
59

? int( 0.6! * 100)
60
? int( 0.6# * 100)
59


0.6 is double
0.6# is double
0.6! is single.
--
Regards,
Tom Ogilvy

"keepitcool" wrote in message
...
try:

? cint( 0.6 * 100) = 60
? .6*100 \1 = 60



keepITcool

< email : keepitcool chello nl (with @ and .)
< homepage: http://members.chello.nl/keepitcool


(Roger) wrote:

? int( 0.6 * 100)

Another Intel error then. I haven't encountered this one either. Could anyone running an
Athlon processor please test this ?

It's easy enough to fix that particular calculation, problem is that INT is used a lot to
remove decimals from calculated big numbers (bigger than integer limits) and it would be
impossible to know when this happens or not. Hmmm. I have to proofread tons of code
because of this, buying Macs to my 3000 collegues is unfortunately no option <g.

Best wishes Harald
Followup to newsgroup only please.

"J.E. McGimpsey" wrote in message
...
I've never encountered that before. In XLv.X:

? int( 0.6 * 100)
60

In article ,
"Harald Staff" wrote:

buying Macs to my 3000 collegues is unfortunately no option <g.

but have you tried??? Bet you could get a substantial discount!<g

? int( 0.6 * 100)
59

AuthenticAMD AMD Athlon(tm) XP 2200+
224MB RAM

Excel 97 SR2

--
Regards,

Tom Ogilvy

Harald Staff wrote in message
...
Another Intel error then. I haven't encountered this one either. Could
anyone running an
Athlon processor please test this ?

It's easy enough to fix that particular calculation, problem is that INT
is used a lot to
remove decimals from calculated big numbers (bigger than integer limits)
and it would be
impossible to know when this happens or not. Hmmm. I have to proofread
tons of code
because of this, buying Macs to my 3000 collegues is unfortunately no
option <g.

Best wishes Harald
Followup to newsgroup only please.

"J.E. McGimpsey" wrote in message
...
I've never encountered that before. In XLv.X:

? int( 0.6 * 100)
60

On Fri, 03 Oct 2003 09:05:09 -0600, "J.E. McGimpsey"
wrote:

I've never encountered that before. In XLv.X:

? int( 0.6 * 100)
60

That's odd.

Excel 2002(10.4524.4219) SP-2
in a worksheet =INT(0.6*100) -- 60

But in the VB6.3 that comes with it, I get 59 in the Immediate window, too.
?int(0.6*100)
59

XP Pro
Processor x86 Family 15 Model 2 Stepping 4 GenuineIntel ~1993 Mhz





--ron

Thanks Tom. Interesting.

Best wishes Harald
Followup to newsgroup only please

"Tom Ogilvy" skrev i melding
...
? int( 0.6 * 100)
59

AuthenticAMD AMD Athlon(tm) XP 2200+
224MB RAM