![]() |
Rounding Defect?
Here is a tough problem. I have an array of singles (the numerica
kind) that I need to search through to find a match for an inpu number. The input number and the array numbers are decimals of typ "00.0", but stored as a singles. I search like this: If Round(Single(i) , 1) = Round(Input , 1) Then Msgbox(Single(i) & " " & Input) The numbers in the msgbox are different!!! It matches 50.6 with a input of 50.5. I have also tried FormatNumber(# , 1), and I trie Int(10 * #) * .1 but those give the same problem. So, I lined up th numbers in a spreadsheet to I could look at the result of each roundin method. Using Int() to convert gave many rounding errors, bu FormatNumber and Round and the Excel Format Cell Function show the sam result in the spreadsheet, 50.5. According to the spreadsheet I mad there is no problem. You can see for yourself: Array (i).Value What I am looking up V Format(Array (i),1) V format# (Input,1) 50.09999847 50.1 1.25 50.5 50.20000076 50.2 1.25 50.5 50.29999924 50.3 1.25 50.5 50.40000153 50.4 1.25 50.5 50.5 50.5 1.25 50.5 50.59999847 50.6 1.26 50.5 50.70000076 50.7 1.26 50.5 50.79999924 50.8 1.26 50.5 But just asking the computer to compare using an if statement causes 1.26 to be returned and if I look at the Array value that is bein matched the computer is matching 50.5 to 50.6. Please help -- Message posted from http://www.ExcelForum.com |
Rounding Defect?
Wow. Thats the dumbest thing I've ever done. The problem was I di
this For i = 1 to 1000 if match yadda yadda i = i + 1 <== recordbook for stupidity next -- Message posted from http://www.ExcelForum.com |
All times are GMT +1. The time now is 02:02 AM. |
Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
ExcelBanter.com