Home |
Search |
Today's Posts |
#1
Posted to microsoft.public.excel.programming
|
|||
|
|||
Excel precision issues with float/single
If I have a float variable with a value of 9.9 (as seen in the debugger), and
I set it to an Excel cell, the cell value is 9.89999961853027. However, if I have a double variable with the same value of 9.9, setting that to an Excel cell gives me the correct value. (Note that the cell, in it's default size, looks correct, but if you expand the cell, or click on the value and look on the editor area, the value is incorrect.) Am I doing something wrong? How do I fix this? Thanks much! |
#2
Posted to microsoft.public.excel.programming
|
|||
|
|||
Excel precision issues with float/single
Cells hold all numbers as double, so when when the number is converted from
single to double, it takes on the imprecision. I suspect this has to do with how floating point numbers are stored. -- Regards, Tom Ogilvy "Julie" wrote: If I have a float variable with a value of 9.9 (as seen in the debugger), and I set it to an Excel cell, the cell value is 9.89999961853027. However, if I have a double variable with the same value of 9.9, setting that to an Excel cell gives me the correct value. (Note that the cell, in it's default size, looks correct, but if you expand the cell, or click on the value and look on the editor area, the value is incorrect.) Am I doing something wrong? How do I fix this? Thanks much! |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
ieee754 single precision numbers | Excel Discussion (Misc queries) | |||
How to make a key float in an excel wksh when a person scrolls. | Excel Worksheet Functions | |||
Precision in Excel 2002 | Charts and Charting in Excel | |||
Loss of caluclating precision (VBA vs Excel) | Excel Discussion (Misc queries) | |||
Precision displayed does not match precision in cell | Excel Discussion (Misc queries) |