Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 5,651
Default Is this rounding error?

In the below routine, are the instances where i=0 due to not being able to
express certain values exactly in binary?

And can it be reliably solved for any 'r' by ROUNDing to 15 places?

Or is there something else I'm missing?

Thanks.

===================
Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print ("For r = " & r & ", i = " & i)
Next r

End Sub
================


--ron
  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 131
Default Is this rounding error?

It is a little more complex than just rounding error. Try
this:

Public Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print "r:" & r & " i:" & i _
& " Log(r):" & Log(r) & " Log(r)/Log(r):" _
& Log(r) / Log(r) & " Fix:" & Fix(Log(r) / Log(r))
Next r


End Sub

You will see that when the results of the formula Fix(Log
(r) / Log(r)) are printed directly it gives the result 1
but when it is assigned to the variable i it gives the
result 0! This seems to occur even if I explicitly define
r and i to be either long integers or a double precision
floating point variables!


-----Original Message-----
In the below routine, are the instances where i=0 due to

not being able to
express certain values exactly in binary?

And can it be reliably solved for any 'r' by ROUNDing to

15 places?

Or is there something else I'm missing?

Thanks.

===================
Sub foo()
Dim r
Dim i

For r = 2 To 16

i = Fix(Log(r) / Log(r))
Debug.Print ("For r = " & r & ", i = " & i)
Next r

End Sub
================


--ron
.

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
rounding error pkeegs Excel Discussion (Misc queries) 2 January 13th 10 07:53 AM
My rounding error in a simple IF garron Excel Worksheet Functions 2 August 6th 08 04:11 AM
rounding error? Steve Excel Discussion (Misc queries) 5 June 1st 08 02:58 PM
Percentage rounding error in charts Tracey Excel Discussion (Misc queries) 4 May 14th 05 04:01 AM
Rounding Error Matthew W Excel Programming 5 April 19th 04 10:11 AM


All times are GMT +1. The time now is 03:38 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"