LinkBack Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 215
Default A Serious Round-Off Problem !!

Hello;
The following two sample codes produce two completely different results !!
SUB TEST1()
' ... my code...
SUM = 0.0
FOR I = 1 TO 20000
SUM = SUM + A(I)*B(I)/C(I)
NEXT
RESULT1 = SUM
' ...my code...
END SUB

SUB TEST2()
' ... my code...
SUM = 0.0
FOR I = 1 TO 20000
SUM = SUM + A(I)*B(I)/C(I) /1000.
NEXT
RESULT2 = SUM*1000.
' ...my code
END SUB

Variables: Result1, Result2, SUM, A,B,C are DOUBLE data types
Values of A,B,C vary from very small to very large, in no specific order
For a typical set of analytical data, Result1 = 0.9986, while Result2 =
4.2432, a considerable difference !!

The different results are most likely associated with accumilated round-off
errors known to be associated with floating-point arithmetic precission.

I would very much appreciate your suggestion(s) on how to minimize this
serious round-off error.
Can one use a higher precission than DOUBLE ... either in VBA or FORTRAN ??

Thank you kindly.
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
problem with round down Chris Excel Discussion (Misc queries) 5 September 3rd 09 08:24 PM
Round() problem dindigul Excel Worksheet Functions 2 December 17th 07 04:34 PM
Problem with the round function Patrick Excel Programming 3 January 31st 05 03:58 PM
How do I ROUND() round off decimals of a column dataset? Højrup Excel Worksheet Functions 2 January 12th 05 10:50 AM
Round problem with sorting Zakar Excel Programming 0 May 6th 04 07:56 AM


All times are GMT +1. The time now is 04:31 AM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"