View Single Post
  #1   Report Post  
Posted to microsoft.public.excel.programming
joeu2004 joeu2004 is offline
external usenet poster
 
Posts: 2,059
Default How to strore exact double from macro into Excel cell?

I know this is "stoopid" because I'm sure I have done this before with no
problem.

But for some reason, today -- with the stars, sun, moon and earth aligned
such as they are -- the following macro is not doing what I expect.

What am I doing wrong?


Sub doit()
Dim x As Double, s As String, i As Long
For i = -9 To 8
x = 0.28 + i * 2 ^ -54
Debug.Print dbl2dec(x)
Cells(1 + i + 9, 2) = x
Next i
End Sub


dbl2dec() is my function that formats binary floating point exactly. The
debug.print output is what I expect, namely:

i=-9: 0.279999999999999,52704499150968331377953290939331 0546875
i=-8: 0.279999999999999,58255614274094114080071449279785 15625
i=-7: 0.279999999999999,63806729397219896782189607620239 2578125
i=-6: 0.279999999999999,69357844520345679484307765960693 359375
i=-5: 0.279999999999999,74908959643471462186425924301147 4609375
i=-4: 0.279999999999999,80460074766597244888544082641601 5625
i=-3: 0.279999999999999,86011189889723027590662240982055 6640625
i=-2: 0.279999999999999,91562305012848810292780399322509 765625
i=-1: 0.279999999999999,97113420135974592994898557662963 8671875
i=0: 0.280000000000000,02664535259100375697016716003417 96875
i=1: 0.280000000000000,08215650382226158399134874343872 0703125
i=2: 0.280000000000000,13766765505351941101253032684326 171875
i=3: 0.280000000000000,19317880628477723803371191024780 2734375
i=4: 0.280000000000000,24868995751603506505489349365234 375
i=5: 0.280000000000000,30420110874729289207607507705688 4765625
i=6: 0.280000000000000,35971225997855071909725666046142 578125
i=7: 0.280000000000000,41522341120980854611843824386596 6796875
i=8: 0.280000000000000,47073456244106637313961982727050 78125

So "x" is indeed getting the values that I expect.

But the results in B1:B18 are all the same binary representation of the
constant 0.28 (see i=0 above), not the 18 different binary representations
that I generate in the macro.

(C1 is the formula =dbl2dec(B1), which is copied down through C18.)

As a double-check, I do get the 18 different binary representations if
A1:A18 are the values -9 through 8, and I put the following formula into B1
and copy down through B18:

=0.28 + A1 * 2^-54

I think that rules out any user errors w.r.t calculation modes (manual v.
auto; "precision as displayed"; etc).

I know that I can work around the problem by changing the cell assignment in
the macro to:

Cells(1 + i + 9, 2).Formula = "=0.28 + (" & i & ")* 2 ^ -54"

But why can't I return the exact binary representation in cells(...).value
directly?

As I said, I'm certain I have done this before. So I must be having a
"senior moment" and doing something "stoopid".

Please thump me on the head and tell me what it is.

-----

Aha!

I wrote:
As I said, I'm certain I have done this before. So
I must be having a "senior moment" and doing something
"stoopid".


Actually, what I have probably done before is:

Function doit2(i As Long) As Double
Dim x As Double, s As String
x = 0.28 + i * 2 ^ -54
Debug.Print "i=" & i & ": " & dbl2dec(x)
doit2 = x
End Function

where I have =doit3(A1) in B1 and copy down through B18, and A1:A18 are the
values -9 through 8.

And __that__ does work as expected.

So perhaps my question is: why doesn't this work using a macro?

And more to the point: how do I make it work using a macro, other than
storing a formula into the cell?

-----

WAG: When I store into .value in a macro, I wonder if Excel is converting
the double to a numeric string, subject to its display limit of 15
significant digits, then treating it as data entry and converting it back to
a number. Say what?!