Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 5,651
Default Is this rounding error?

In the below routine, are the instances where i=0 due to not being able to
express certain values exactly in binary?

And can it be reliably solved for any 'r' by ROUNDing to 15 places?

Or is there something else I'm missing?

Thanks.

===================
Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print ("For r = " & r & ", i = " & i)
Next r

End Sub
================


--ron
  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 131
Default Is this rounding error?

It is a little more complex than just rounding error. Try
this:

Public Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print "r:" & r & " i:" & i _
& " Log(r):" & Log(r) & " Log(r)/Log(r):" _
& Log(r) / Log(r) & " Fix:" & Fix(Log(r) / Log(r))
Next r


End Sub

You will see that when the results of the formula Fix(Log
(r) / Log(r)) are printed directly it gives the result 1
but when it is assigned to the variable i it gives the
result 0! This seems to occur even if I explicitly define
r and i to be either long integers or a double precision
floating point variables!


-----Original Message-----
In the below routine, are the instances where i=0 due to

not being able to
express certain values exactly in binary?

And can it be reliably solved for any 'r' by ROUNDing to

15 places?

Or is there something else I'm missing?

Thanks.

===================
Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print ("For r = " & r & ", i = " & i)
Next r

End Sub
================


--ron
.

  #3   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 837
Default Is this rounding error?

Just a guess, but it is probably due to differences in the way extended
precision is used in evaluating VBA formulas and in the VBA Immediate
window.

I assume that Log(r) always evaluates to the same value (probably by a
call to the Intel Math Coprocessor) for a given r and that x/x evaluates
to 1 for a given x; else there would be all kinds of math discrepancies
that we have not seen.

Under those assumptions, Fix(Log(r)/Log(r)) can only evaluate to zero if
the numerator and denominator represent different levels of precision.
Math in the Coprocessor is done as 10 byte extended precision, but
stored in variables as 8 byte double precision.

If the numerator is rounded down to double precision or the denominator
is rounded up to double precision, with the other term held in extended
precision, then the ratio will be marginally less than 1.

The additional mantissa bits between for the extended precision
representations of exact values of Log(r) a

r extended bits i
2 11110111000 1
3 00111011100 0
4 11110111000 1
5 10000010100 1
6 10110001111 1
7 11111101101 1
8 10111001010 1
9 00111011100 0
10 11110001010 0
11 10000001110 0
12 00100010010 0
13 00101011100 0
14 01001011011 1
15 11010110111 1
16 00000111001 1

These patterns do not seem entirely consistent with my hypothesis. I do
not recall seeing any double precision discrepancies in Log(), but it
may not be exact in extended precision, which might result in consistency.

Jerry

Ron Rosenfeld wrote:

On Fri, 10 Sep 2004 08:37:39 -0700, "K Dales"
wrote:


It is a little more complex than just rounding error. Try
this:

Public Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print "r:" & r & " i:" & i _
& " Log(r):" & Log(r) & " Log(r)/Log(r):" _
& Log(r) / Log(r) & " Fix:" & Fix(Log(r) / Log(r))
Next r


End Sub

You will see that when the results of the formula Fix(Log
(r) / Log(r)) are printed directly it gives the result 1
but when it is assigned to the variable i it gives the
result 0! This seems to occur even if I explicitly define
r and i to be either long integers or a double precision
floating point variables!


I was aware of the differences you mentioned, depending on the assignment.

Also, if you use the Decimal data type, the results seem to work out OK.

For example:

==========================
Sub foo()
Dim r
Dim i

For r = 2 To 10^10
i = CDec(Fix(Log(r) / Log(r)))
If i < 1 Then Debug.Print ("For r = " & r & ", i = " & i)
Next r

End Sub
=======================

seems to always result in a '1'


--ron


  #4   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 5,651
Default Is this rounding error?

On Mon, 13 Sep 2004 12:16:11 GMT, "Jerry W. Lewis"
wrote:

If the numerator is rounded down to double precision or the denominator
is rounded up to double precision, with the other term held in extended
precision, then the ratio will be marginally less than 1.


Well that may be exactly what is happening! It seems as if the denominator may
be being rounded to double precision.

For r = 2 To 20
i = Fix((Log(r)) / (Log(r)))
If i < 1 Then Debug.Print ("For r = " & r & ", i = " & i)
Next r

OR

For r = 2 To 20
i = Fix((Log(r)) / CDbl(Log(r)))
If i < 1 Then Debug.Print ("For r = " & r & ", i = " & i)
Next r

results in:

For r = 3, i = 0
For r = 9, i = 0
For r = 10, i = 0
For r = 11, i = 0
For r = 12, i = 0
For r = 13, i = 0
For r = 17, i = 0

==================

but converting the numerator results in i=1 for all numbers (at least up to
2^10):

i = Fix(CDbl(Log(r)) / (Log(r)))

Very interesting. So I guess this could be categorized as a rounding error of
sorts, although not what we usually run into.

I wonder why the designers would choose to use different precision for the
numerator and denominator...

Thanks for looking at this.


--ron
Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
rounding error pkeegs Excel Discussion (Misc queries) 2 January 13th 10 07:53 AM
My rounding error in a simple IF garron Excel Worksheet Functions 2 August 6th 08 04:11 AM
rounding error? Steve Excel Discussion (Misc queries) 5 June 1st 08 02:58 PM
Percentage rounding error in charts Tracey Excel Discussion (Misc queries) 4 May 14th 05 04:01 AM
Rounding Error Matthew W Excel Programming 5 April 19th 04 10:11 AM


All times are GMT +1. The time now is 12:27 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"