Home |
Search |
Today's Posts |
|
#1
![]()
Posted to microsoft.public.excel.programming
|
|||
|
|||
![]()
Glad the immediate problem is solved. Most terminating decimal fractions are
nonterminating binary fractions that can only be approximated with finite precision. If you want to learn more about the internal representation of numbers, you might find my functions at http://groups.google.com/group/micro...06871cf92f8465 to be useful. D2B(CSng(0.0111773)) shows that the single precision binary representation (24 bits) is 1.01101110010000011111110B-7 D2B(0.0111773) shows that the double precision binary representation (24 bits) is 1.011011100100000111111100111110101001001000000001 1111B-7 You can use B2D to show that while this equals 0.0111773 to 15 digits, it is not exactly 0.0111773. You can use B2D to show that 0.111773 is bracketed by 1.011011100100000111111100111110101001001000000001 11110101101000B-7 and 1.011011100100000111111100111110101001001000000001 11110101101001B-7 Jerry "Frederick Chow" wrote: Hi, Really thanks to all of your advices. The problem is really solved by declaring X to Double! I originally thought that Single data type should hav been enough as it holds astronomical numbers already, without thinking about the decimal points issue. Frederick Chow "Frederick Chow" wrote in message ... Hi all, I have a fairly mysterious problem. It is hard to explain here, but basically what I did is declare a variable X as single, then assign a cell content containing decimal digits to X, then write X back to another cell. Problem is, let say, suppose the original cell contains 0.0111773. After transferring this number to X and then copy back to another cell, that cell now contains 0.0111772995442152. Though the cell content is very near to the original one, but it does cause problem to me. Anybody can guess what actually happened and what is the solution? Thanks a lot. Frederick Chow |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
Unacceptable floating point errors | Excel Discussion (Misc queries) | |||
Convert floating point to Hours and Minutes ?? | Excel Discussion (Misc queries) | |||
setting a floating decimel point | New Users to Excel | |||
Floating Point Functions | Excel Programming | |||
Handle Floating point rounding errors, efficiently | Excel Programming |