Home |
Search |
Today's Posts |
#1
Posted to microsoft.public.excel.programming
|
|||
|
|||
Rounding Defect?
Here is a tough problem. I have an array of singles (the numerica
kind) that I need to search through to find a match for an inpu number. The input number and the array numbers are decimals of typ "00.0", but stored as a singles. I search like this: If Round(Single(i) , 1) = Round(Input , 1) Then Msgbox(Single(i) & " " & Input) The numbers in the msgbox are different!!! It matches 50.6 with a input of 50.5. I have also tried FormatNumber(# , 1), and I trie Int(10 * #) * .1 but those give the same problem. So, I lined up th numbers in a spreadsheet to I could look at the result of each roundin method. Using Int() to convert gave many rounding errors, bu FormatNumber and Round and the Excel Format Cell Function show the sam result in the spreadsheet, 50.5. According to the spreadsheet I mad there is no problem. You can see for yourself: Array (i).Value What I am looking up V Format(Array (i),1) V format# (Input,1) 50.09999847 50.1 1.25 50.5 50.20000076 50.2 1.25 50.5 50.29999924 50.3 1.25 50.5 50.40000153 50.4 1.25 50.5 50.5 50.5 1.25 50.5 50.59999847 50.6 1.26 50.5 50.70000076 50.7 1.26 50.5 50.79999924 50.8 1.26 50.5 But just asking the computer to compare using an if statement causes 1.26 to be returned and if I look at the Array value that is bein matched the computer is matching 50.5 to 50.6. Please help -- Message posted from http://www.ExcelForum.com |
#2
Posted to microsoft.public.excel.programming
|
|||
|
|||
Rounding Defect?
Wow. Thats the dumbest thing I've ever done. The problem was I di
this For i = 1 to 1000 if match yadda yadda i = i + 1 <== recordbook for stupidity next -- Message posted from http://www.ExcelForum.com |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
INT defect: Please try this on 2007 for me | Excel Worksheet Functions | |||
ROUNDING FORMULA =ROUND(B12/$B$10,1) ROUNDING TO HIGH | Excel Worksheet Functions | |||
AutoFilter defect or my mistake? | Excel Discussion (Misc queries) | |||
I need a formula with rounding up & rounding down to the nearest . | Excel Worksheet Functions | |||
Excel template available for Defect Tracking | Excel Discussion (Misc queries) |