View Single Post
  #1   Report Post  
Posted to microsoft.public.excel.misc
joeu2004 joeu2004 is offline
external usenet poster
 
Posts: 2,059
Default Another numeric formatting anomaly

myMod(12346789-0.0000004,1) returns a binary representation which is exactly
0.999999597668647,76611328125. (The comma demarcates the first 15
significant digits.)

But when the cell (A1) is formatted as Number with 15 decimal places, Excel
displays 0.999999597668647 (!).

In contrast, myCstr(A1) returns 0.999999597668648 as expected.

These examples are for Excel 2003 11.5612.5606, part of MS Office Sm Busn Ed
2003 on MS Win XP SP3, and VB 6.3 version 9969, VBA Retail 6.4.8896, Forms3
11.0.5601.

The UDFs a

Function mymod(a As Double, b As Double) As Double
mymod = a - b * Int(a / b)
End Function

Function myCstr(arg As Double) As String
myCstr = CStr(arg)
End Function