Home |
Search |
Today's Posts |
#1
![]()
Posted to microsoft.public.excel.programming
|
|||
|
|||
![]()
Hi,
The "Floating point" has been extensively discussed in this ng and I understand why it causes errors. But I haven't found an "efficient" way to deal with a simple thing like: If cell-value = 1 Then 'blah Sub test2() [a1] = 0 [a2].Formula = "=A1+0.05" [a2].AutoFill [A2:A21] Debug.Print "[a21]=1", [a21] = 1 'False Debug.Print 1 - [a21] '-2.22044604925031E-16 Dim nDouble As Double, nSingle As Single nDouble = [a21]: nSingle = [a21] Debug.Print "nDouble=1 ", nDouble = 1 'False Debug.Print "nSingle=1 ", nSingle = 1 'True 'Single fixes above but causes problems later nDouble = WorksheetFunction.Round(nDouble / 10, 2) nSingle = WorksheetFunction.Round(nSingle / 10, 2) Debug.Print " 'divide by 10 and round" Debug.Print "nDouble"; nDouble, "nSingle"; nSingle [c1] = nDouble [c2] = nSingle Debug.Print "[c1]=0.1", [c1] = 0.1 Debug.Print "[c2]=0.1", [c2] = 0.1, [c2] - 0.1 End Sub Coercing to a Single appears to resolve my problem, but causes yet further rounding problems downstream. Of course could test with a Single then work with a Double. But this seems like a lot of overhead in a long loop, and even more so in a UDF. Also could use Round or in xl97 Worksheetfunction.round, but the latter in particular is slow. Or maybe check within range +/- 1E-14. Various alternative solutions have been suggested and Tom Ogilvy posted a useful set of links he http://tinyurl.com/662oo However I would be grateful for advice as to the fastest / most efficient way to ignore [say] the 14th decimal in a cell value. TIA, Peter T |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
Unacceptable floating point errors | Excel Discussion (Misc queries) | |||
setting a floating decimel point | New Users to Excel | |||
Handle Compile Errors | Excel Programming | |||
REPOST: How to handle errors | Excel Programming | |||
REPOST: How to handle errors | Excel Programming |