Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 1
Default incorrect results when working with doubles or singles

Hi everybody,

I am puzzled by the following strange things:

Public Sub test1()
Dim a As Double
a = 16.4
a = a - 0.1

MsgBox a
If (a = 16.3) Then
MsgBox "1"
Else
MsgBox "2"
End If
End Sub

Public Sub test2()
Dim a As Single
a = 16.4
a = a - 0.1

MsgBox a
If (a = 16.3) Then
MsgBox "1"
Else
MsgBox "2"
End If
End Sub

When I run test1 the first MsgBox shows 16.3 and the second shows 2.
This means dat 16.3 doesn't equal to 16.3. Now when i change the
datatype to single (test2) everything works fine. Can anyone explain
me the reason of this?

Public Sub test3()
Dim a As Single
Dim b As Single
a = 16.4

Dim i As Single
For i = 0.1 To 20 Step 0.1
b = a + i
Next i

MsgBox b
End Sub

The result of test3 is: 36.30004. Why isn't it exactly 36.3, and what
to do?

Public Sub test4()
Dim a As Single
Dim b As Single
a = 27.1

Dim i As Single
For i = 0.1 To 5.9 Step 0.1
b = a + i
Next i

MsgBox b
If b = 33 Then
MsgBox "1"
Else
MsgBox "2"
End If
End Sub

When I run test4 the first MsgBox shows 33 and the second one shows 2.
This means that 33 isn't equal to 33. Why not?

Any information about this would be greatly appreciated.

Patrice
  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 4,624
Default incorrect results when working with doubles or singles

Check out

http://cpearson.com/excel/rounding.htm

If you're making exact comparisons with floating point numbers you're
best to either round them or to compare them within a small tolerance,
e.g., instead of

If a = 16.3 Then

use

Const epsilon As Double = 1e-10
Dim a As Double
a = 16.4 - 0.1
MsgBox a = 16.3
If Abs(a - 16.3) < epsilon Then MsgBox "Same"

where epsilon is a sufficiently small number that rounding errors won't
cause a problem

In article ,
(Patrice) wrote:

Hi everybody,

I am puzzled by the following strange things:

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Goalseek gives incorrect results dindigul New Users to Excel 1 July 12th 09 12:14 AM
results incorrect KMC Excel Discussion (Misc queries) 2 July 18th 08 03:14 PM
vlookup gives incorrect results Forrest G. Excel Worksheet Functions 3 December 18th 06 05:40 PM
Very large workbook now giving incorrect results :( [email protected] Excel Discussion (Misc queries) 0 July 17th 06 11:29 PM
How do I prevent incorrect formula results appearing in cell? Marc Todd Excel Worksheet Functions 2 January 26th 05 07:57 AM


All times are GMT +1. The time now is 12:20 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"