Home |
Search |
Today's Posts |
|
#1
![]()
Posted to microsoft.public.excel.programming
|
|||
|
|||
![]()
In the below routine, are the instances where i=0 due to not being able to
express certain values exactly in binary? And can it be reliably solved for any 'r' by ROUNDing to 15 places? Or is there something else I'm missing? Thanks. =================== Sub foo() Dim r Dim i For r = 2 To 16 i = Fix(Log(r) / Log(r)) Debug.Print ("For r = " & r & ", i = " & i) Next r End Sub ================ --ron |
#2
![]()
Posted to microsoft.public.excel.programming
|
|||
|
|||
![]()
It is a little more complex than just rounding error. Try
this: Public Sub foo() Dim r Dim i For r = 2 To 16 i = Fix(Log(r) / Log(r)) Debug.Print "r:" & r & " i:" & i _ & " Log(r):" & Log(r) & " Log(r)/Log(r):" _ & Log(r) / Log(r) & " Fix:" & Fix(Log(r) / Log(r)) Next r End Sub You will see that when the results of the formula Fix(Log (r) / Log(r)) are printed directly it gives the result 1 but when it is assigned to the variable i it gives the result 0! This seems to occur even if I explicitly define r and i to be either long integers or a double precision floating point variables! -----Original Message----- In the below routine, are the instances where i=0 due to not being able to express certain values exactly in binary? And can it be reliably solved for any 'r' by ROUNDing to 15 places? Or is there something else I'm missing? Thanks. =================== Sub foo() Dim r Dim i For r = 2 To 16 i = Fix(Log(r) / Log(r)) Debug.Print ("For r = " & r & ", i = " & i) Next r End Sub ================ --ron . |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
rounding error | Excel Discussion (Misc queries) | |||
My rounding error in a simple IF | Excel Worksheet Functions | |||
rounding error? | Excel Discussion (Misc queries) | |||
Percentage rounding error in charts | Excel Discussion (Misc queries) | |||
Rounding Error | Excel Programming |