Home |
Search |
Today's Posts |
#1
Posted to microsoft.public.excel.misc
|
|||
|
|||
Another numeric formatting anomaly
myMod(12346789-0.0000004,1) returns a binary representation which is exactly
0.999999597668647,76611328125. (The comma demarcates the first 15 significant digits.) But when the cell (A1) is formatted as Number with 15 decimal places, Excel displays 0.999999597668647 (!). In contrast, myCstr(A1) returns 0.999999597668648 as expected. These examples are for Excel 2003 11.5612.5606, part of MS Office Sm Busn Ed 2003 on MS Win XP SP3, and VB 6.3 version 9969, VBA Retail 6.4.8896, Forms3 11.0.5601. The UDFs a Function mymod(a As Double, b As Double) As Double mymod = a - b * Int(a / b) End Function Function myCstr(arg As Double) As String myCstr = CStr(arg) End Function |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
Numeric date to text..formatting question | Excel Discussion (Misc queries) | |||
Numeric formatting | Excel Discussion (Misc queries) | |||
Formatting Numeric Fields from Within MS Query | Excel Discussion (Misc queries) | |||
numeric value is not formatting correctly | Excel Discussion (Misc queries) | |||
Numeric formatting problem | Excel Worksheet Functions |