Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.worksheet.functions
derekcowley
 
Posts: n/a
Default I am getting a strange round off error in excel

when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).


  #2   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Bernard Liengme
 
Posts: n/a
Default I am getting a strange round off error in excel

With XL 2003 I get 0 when both G1 and H1 equal 4.
Note the 0 is left aligned since IMSUM gives an imaginary number in text
format.
When G1 and H1 equal -4 the formula correctly yields 2-2i
What do you get and what version of XL are you using?
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"derekcowley" wrote in message
...
when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).




  #3   Report Post  
Posted to microsoft.public.excel.worksheet.functions
derekcowley
 
Posts: n/a
Default I am getting a strange round off error in excel

I get 2.22044604925031E-016

please see the file at link:
http://www.geocities.com/derekcowley/public/IMerror.xls

"Bernard Liengme" wrote:

With XL 2003 I get 0 when both G1 and H1 equal 4.
Note the 0 is left aligned since IMSUM gives an imaginary number in text
format.
When G1 and H1 equal -4 the formula correctly yields 2-2i
What do you get and what version of XL are you using?
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"derekcowley" wrote in message
...
when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).





  #4   Report Post  
Posted to microsoft.public.excel.worksheet.functions
JMB
 
Posts: n/a
Default I am getting a strange round off error in excel

Using XL2000 I also get 0 with your formula. Are G1 and H1 hardcoded or the
results of a formula? Although Excel may be showing 4 on the spreadsheet
could the actual value be a decimal that is very close to 4?

Also see:
http://www.cpearson.com/excel/rounding.htm

"derekcowley" wrote:

I get 2.22044604925031E-016

please see the file at link:
http://www.geocities.com/derekcowley/public/IMerror.xls

"Bernard Liengme" wrote:

With XL 2003 I get 0 when both G1 and H1 equal 4.
Note the 0 is left aligned since IMSUM gives an imaginary number in text
format.
When G1 and H1 equal -4 the formula correctly yields 2-2i
What do you get and what version of XL are you using?
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"derekcowley" wrote in message
...
when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).





  #5   Report Post  
Posted to microsoft.public.excel.worksheet.functions
JMB
 
Posts: n/a
Default I am getting a strange round off error in excel

BTW - many folks who visit this site will not download files.


http://www.cpearson.com/excel/Attachments.htm


"derekcowley" wrote:

I get 2.22044604925031E-016

please see the file at link:
http://www.geocities.com/derekcowley/public/IMerror.xls

"Bernard Liengme" wrote:

With XL 2003 I get 0 when both G1 and H1 equal 4.
Note the 0 is left aligned since IMSUM gives an imaginary number in text
format.
When G1 and H1 equal -4 the formula correctly yields 2-2i
What do you get and what version of XL are you using?
best wishes
--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"derekcowley" wrote in message
...
when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).







  #6   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Mike Middleton
 
Posts: n/a
Default I am getting a strange round off error in excel

derekcowley -

I get 2.22044604925031E-016 ... <


That's 0.000000000000000222044604925031, and since Excel has 15 significant
digits, that's essentially zero.

- Mike
www.mikemiddleton.com


  #7   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Ron Rosenfeld
 
Posts: n/a
Default I am getting a strange round off error in excel

On Fri, 16 Jun 2006 14:13:01 -0700, derekcowley
wrote:

when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).


In your worksheet, although G1 and H1 appear equal to zero given the limits of
Excel precision, they are not exactly equal to zero.

In particular, if you ROUND the values to 15 decimal places, which is Excel's
precision limit, or if you choose TOOLS/OPTIONS/Calculation/ Precision as
Displayed; you will see the "0" result.


--ron
  #8   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Bernard Liengme
 
Posts: n/a
Default I am getting a strange round off error in excel

This is a good example of where OP did not supply all the data needed to get
a good answer the first time!

--
Bernard V Liengme
www.stfx.ca/people/bliengme
remove caps from email

"Ron Rosenfeld" wrote in message
...
On Fri, 16 Jun 2006 14:13:01 -0700, derekcowley
wrote:

when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).


In your worksheet, although G1 and H1 appear equal to zero given the
limits of
Excel precision, they are not exactly equal to zero.

In particular, if you ROUND the values to 15 decimal places, which is
Excel's
precision limit, or if you choose TOOLS/OPTIONS/Calculation/ Precision as
Displayed; you will see the "0" result.


--ron



  #9   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Ron Rosenfeld
 
Posts: n/a
Default I am getting a strange round off error in excel

On Sun, 18 Jun 2006 10:05:55 -0300, "Bernard Liengme"
wrote:

This is a good example of where OP did not supply all the data needed to get
a good answer the first time!


Concur.

And it's an example what happens when we don't know that we don't know
something; or we think we know something but we're wrong.
--ron
  #10   Report Post  
Posted to microsoft.public.excel.worksheet.functions
Jerry W. Lewis
 
Posts: n/a
Default I am getting a strange round off error in excel

Assuming that G4 and/or H4 are calculated values, then look at =(G1-4) and
=(H1-4). I think that you will find that one or both is not equal to 4.

Note that the parentheses are required in these formulas, else Excel will
return zero if they are equal to 15 digits, even though it will calculate
with the full precision non-difference. Similarly, if the differences
between G1 and H1 and 4 are less than 1E14, then formatting the cell to
display more figures will not help, since Excel (as documented) will display
no more than 15 digits; the rest will arbitrarily be zero if you ask for more.

Jerry

"derekcowley" wrote:

when mixing data types: a real and a complex as in

=IMSUM(-G1/2,IMSQRT(H1))

if G1 and H1 are both exactly equal to 4, the answer whould be 0

see http://www.geocities.com/derekcowley/public/IMerror.xls
and look at location I1 (highlighted).


Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Round off values in Excel harshaputhraya Excel Discussion (Misc queries) 7 May 15th 06 12:39 AM
strange problem with links updates in excel [email protected] Excel Discussion (Misc queries) 1 April 23rd 06 10:59 AM
strange problem: excel file does not open and background freezes i EdH Excel Discussion (Misc queries) 2 April 21st 06 02:00 AM
Why does a spreadsheet sent in Excel round numbers when mailed? artisan New Users to Excel 1 August 17th 05 11:42 AM
In Excel, how do you get it to not automatically round my dollars. Cylie G. Excel Worksheet Functions 2 January 7th 05 10:43 PM


All times are GMT +1. The time now is 11:01 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"