Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 1
Default Rounding Defect?

Here is a tough problem. I have an array of singles (the numerica
kind) that I need to search through to find a match for an inpu
number. The input number and the array numbers are decimals of typ
"00.0", but stored as a singles. I search like this:

If Round(Single(i) , 1) = Round(Input , 1) Then
Msgbox(Single(i) & " " & Input)

The numbers in the msgbox are different!!! It matches 50.6 with a
input of 50.5. I have also tried FormatNumber(# , 1), and I trie
Int(10 * #) * .1 but those give the same problem. So, I lined up th
numbers in a spreadsheet to I could look at the result of each roundin
method. Using Int() to convert gave many rounding errors, bu
FormatNumber and Round and the Excel Format Cell Function show the sam
result in the spreadsheet, 50.5. According to the spreadsheet I mad
there is no problem. You can see for yourself:

Array (i).Value What I am looking up
V Format(Array (i),1) V format# (Input,1)
50.09999847 50.1 1.25 50.5
50.20000076 50.2 1.25 50.5
50.29999924 50.3 1.25 50.5
50.40000153 50.4 1.25 50.5
50.5 50.5 1.25 50.5
50.59999847 50.6 1.26 50.5
50.70000076 50.7 1.26 50.5
50.79999924 50.8 1.26 50.5

But just asking the computer to compare using an if statement causes
1.26 to be returned and if I look at the Array value that is bein
matched the computer is matching 50.5 to 50.6. Please help

--
Message posted from http://www.ExcelForum.com

  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 1
Default Rounding Defect?

Wow. Thats the dumbest thing I've ever done. The problem was I di
this

For i = 1 to 1000

if match yadda yadda

i = i + 1 <== recordbook for stupidity

next

--
Message posted from http://www.ExcelForum.com

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
INT defect: Please try this on 2007 for me JoeU2004 Excel Worksheet Functions 26 October 16th 09 03:15 PM
ROUNDING FORMULA =ROUND(B12/$B$10,1) ROUNDING TO HIGH SunshineinFt.Myers[_2_] Excel Worksheet Functions 7 March 5th 09 06:41 PM
AutoFilter defect or my mistake? [email protected] Excel Discussion (Misc queries) 2 June 2nd 07 08:17 PM
I need a formula with rounding up & rounding down to the nearest . Tony Kay Excel Worksheet Functions 3 May 29th 07 11:13 PM
Excel template available for Defect Tracking Maury Brown Excel Discussion (Misc queries) 0 March 28th 05 08:51 PM


All times are GMT +1. The time now is 08:51 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"