LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 5,600
Default Handle Floating point rounding errors, efficiently

Hi,

The "Floating point" has been extensively discussed in this ng and I
understand why it causes errors. But I haven't found an "efficient" way to
deal with a simple thing like:
If cell-value = 1 Then 'blah

Sub test2()

[a1] = 0
[a2].Formula = "=A1+0.05"
[a2].AutoFill [A2:A21]
Debug.Print "[a21]=1", [a21] = 1 'False
Debug.Print 1 - [a21] '-2.22044604925031E-16

Dim nDouble As Double, nSingle As Single
nDouble = [a21]: nSingle = [a21]
Debug.Print "nDouble=1 ", nDouble = 1 'False
Debug.Print "nSingle=1 ", nSingle = 1 'True

'Single fixes above but causes problems later
nDouble = WorksheetFunction.Round(nDouble / 10, 2)
nSingle = WorksheetFunction.Round(nSingle / 10, 2)
Debug.Print " 'divide by 10 and round"
Debug.Print "nDouble"; nDouble, "nSingle"; nSingle
[c1] = nDouble
[c2] = nSingle
Debug.Print "[c1]=0.1", [c1] = 0.1
Debug.Print "[c2]=0.1", [c2] = 0.1, [c2] - 0.1
End Sub

Coercing to a Single appears to resolve my problem, but causes yet further
rounding problems downstream.

Of course could test with a Single then work with a Double. But this seems
like a lot of overhead in a long loop, and even more so in a UDF. Also could
use Round or in xl97 Worksheetfunction.round, but the latter in particular
is slow. Or maybe check within range +/- 1E-14.

Various alternative solutions have been suggested and Tom Ogilvy posted a
useful set of links he
http://tinyurl.com/662oo

However I would be grateful for advice as to the fastest / most efficient
way to ignore [say] the 14th decimal in a cell value.

TIA,
Peter T


 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Unacceptable floating point errors Jeff in GA Excel Discussion (Misc queries) 32 September 25th 09 11:26 AM
setting a floating decimel point Rose New Users to Excel 2 April 29th 05 06:10 PM
Handle Compile Errors JeffT Excel Programming 1 October 14th 04 01:38 PM
REPOST: How to handle errors Bruccce Excel Programming 1 August 8th 03 03:22 PM
REPOST: How to handle errors Bob Kilmer Excel Programming 0 August 5th 03 11:24 PM


All times are GMT +1. The time now is 12:25 AM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"