Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 1
Default VBA conversion confusion

I suppose this is old hat to Excel VBA pro's but it caught me by
surprise. What is there about the default Variant type that makes the
comparison fail (but only for SOME inputs, d1=0.9,d2=0.1,d3=0,d4=0 is
ok!?). Also, uncommenting the dim statement fixes the issue.

Can anyone please explain? (Excel 2000 9.0.4402 SR-1)

Jeff

===
Function foobar()
' Dim d1 As Double, d2 As Double, d3 As Double, d4 As Double, s as
Double
s = 1
d1 = 0.7
d2 = 0.1
d3 = 0.1
d4 = 0.1
If (d1 + d2 + d3 + d4 = s) Then
foobar = False
Else
foobar = True
End If
End Function
  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 968
Default VBA conversion confusion

Hi Jeff,

looks like you are hitting the binary floating point problem:

any decimal number may not have an exact binary floating point
representation, so doing exact comparisons on decimal numbers is not
recommended.

The solution is to use a tolerance:

If abs(d1 + d2 + d3 + d4 - s)<0.000000001 Then
foobar=True
else
foobar=false
endif


regds
Charles
______________________
Decision Models
FastExcel Version 2 now available.
www.DecisionModels.com/FxlV2WhatsNew.htm

"Jeff" wrote in message
om...
I suppose this is old hat to Excel VBA pro's but it caught me by
surprise. What is there about the default Variant type that makes the
comparison fail (but only for SOME inputs, d1=0.9,d2=0.1,d3=0,d4=0 is
ok!?). Also, uncommenting the dim statement fixes the issue.

Can anyone please explain? (Excel 2000 9.0.4402 SR-1)

Jeff

===
Function foobar()
' Dim d1 As Double, d2 As Double, d3 As Double, d4 As Double, s as
Double
s = 1
d1 = 0.7
d2 = 0.1
d3 = 0.1
d4 = 0.1
If (d1 + d2 + d3 + d4 = s) Then
foobar = False
Else
foobar = True
End If
End Function



Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
lookup confusion HS[_2_] New Users to Excel 1 May 19th 09 10:44 PM
Calculation confusion L Excel Discussion (Misc queries) 2 October 30th 08 12:58 PM
Nested IF confusion! JayPee Excel Worksheet Functions 3 August 7th 08 11:33 PM
MDI Confusion CMoya Excel Discussion (Misc queries) 1 February 21st 08 02:11 PM
Confusion..... Eric @ CMN, Evansville Excel Discussion (Misc queries) 2 December 27th 05 07:15 PM


All times are GMT +1. The time now is 12:09 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"