Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 16
Default Really thanks to your advices!

Glad the immediate problem is solved. Most terminating decimal fractions are
nonterminating binary fractions that can only be approximated with finite
precision. If you want to learn more about the internal representation of
numbers, you might find my functions at
http://groups.google.com/group/micro...06871cf92f8465
to be useful.

D2B(CSng(0.0111773)) shows that the single precision binary representation
(24 bits) is
1.01101110010000011111110B-7
D2B(0.0111773) shows that the double precision binary representation (24
bits) is
1.011011100100000111111100111110101001001000000001 1111B-7
You can use B2D to show that while this equals 0.0111773 to 15 digits, it is
not exactly 0.0111773. You can use B2D to show that 0.111773 is bracketed by
1.011011100100000111111100111110101001001000000001 11110101101000B-7 and
1.011011100100000111111100111110101001001000000001 11110101101001B-7

Jerry

"Frederick Chow" wrote:

Hi,

Really thanks to all of your advices. The problem is really solved by
declaring X to Double!

I originally thought that Single data type should hav been enough as it
holds astronomical numbers already, without thinking about the decimal
points issue.

Frederick Chow
"Frederick Chow" wrote in message
...
Hi all,

I have a fairly mysterious problem.

It is hard to explain here, but basically what I did is declare a variable
X as single, then assign a cell content containing decimal digits to X,
then write X back to another cell.

Problem is, let say, suppose the original cell contains 0.0111773. After
transferring this number to X and then copy back to another cell, that
cell now contains 0.0111772995442152. Though the cell content is very near
to the original one, but it does cause problem to me. Anybody can guess
what actually happened and what is the solution? Thanks a lot.

Frederick Chow




Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Unacceptable floating point errors Jeff in GA Excel Discussion (Misc queries) 32 September 25th 09 11:26 AM
Convert floating point to Hours and Minutes ?? [email protected] Excel Discussion (Misc queries) 1 August 19th 06 02:26 AM
setting a floating decimel point Rose New Users to Excel 2 April 29th 05 06:10 PM
Floating Point Functions Joel Excel Programming 6 March 30th 05 10:29 PM
Handle Floating point rounding errors, efficiently Peter T Excel Programming 6 December 19th 04 04:34 PM


All times are GMT +1. The time now is 12:51 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"