Excel precision issues with float/single
 

If I have a float variable with a value of 9.9 (as seen in the debugger), and
I set it to an Excel cell, the cell value is 9.89999961853027. However, if I
have a double variable with the same value of 9.9, setting that to an Excel
cell gives me the correct value.

(Note that the cell, in it's default size, looks correct, but if you expand
the cell, or click on the value and look on the editor area, the value is
incorrect.)

Am I doing something wrong? How do I fix this? Thanks much!

Excel precision issues with float/single
 

Cells hold all numbers as double, so when when the number is converted from
single to double, it takes on the imprecision. I suspect this has to do with
how floating point numbers are stored.

--
Regards,
Tom Ogilvy


"Julie" wrote:

If I have a float variable with a value of 9.9 (as seen in the debugger), and
I set it to an Excel cell, the cell value is 9.89999961853027. However, if I
have a double variable with the same value of 9.9, setting that to an Excel
cell gives me the correct value.

(Note that the cell, in it's default size, looks correct, but if you expand
the cell, or click on the value and look on the editor area, the value is
incorrect.)

Am I doing something wrong? How do I fix this? Thanks much!