Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.misc
external usenet poster
 
Posts: 2,059
Default Another numeric formatting anomaly

myMod(12346789-0.0000004,1) returns a binary representation which is exactly
0.999999597668647,76611328125. (The comma demarcates the first 15
significant digits.)

But when the cell (A1) is formatted as Number with 15 decimal places, Excel
displays 0.999999597668647 (!).

In contrast, myCstr(A1) returns 0.999999597668648 as expected.

These examples are for Excel 2003 11.5612.5606, part of MS Office Sm Busn Ed
2003 on MS Win XP SP3, and VB 6.3 version 9969, VBA Retail 6.4.8896, Forms3
11.0.5601.

The UDFs a

Function mymod(a As Double, b As Double) As Double
mymod = a - b * Int(a / b)
End Function

Function myCstr(arg As Double) As String
myCstr = CStr(arg)
End Function

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Numeric date to text..formatting question JHB Excel Discussion (Misc queries) 2 October 4th 08 07:46 PM
Numeric formatting JudgeMental Excel Discussion (Misc queries) 6 May 10th 08 10:19 PM
Formatting Numeric Fields from Within MS Query Paul D. Simon Excel Discussion (Misc queries) 1 January 17th 08 05:16 PM
numeric value is not formatting correctly Fam via OfficeKB.com Excel Discussion (Misc queries) 4 February 2nd 06 04:20 PM
Numeric formatting problem Metolius Dad Excel Worksheet Functions 3 May 22nd 05 07:14 PM


All times are GMT +1. The time now is 04:32 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"