Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 4
Default Help Needed: Calculation error

When I run the following code:

Public Sub test()
Cells(1, 3).Value = Cells(1, 1).Value - Cells(1,
2).Value * 0.000001
End Sub


with input data:
A1=0.035
A2=35000

I expect to get A3=0; however I get A3 = 6.94E-18!

Can anyone explain why and how this can be avoided?

Thank you in advance for your help,

Adrian F.
Ottawa



  #2   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 45
Default Help Needed: Calculation error

The issue is with the level of precision provided by Excel. Excel only
provides precision to 15 decimal places. As a solution, I would use the
ROUND function in VBA to eliminate this problem, such as:

Cells(1, 3).Value = Cells(1, 1).Value - Round(Cells(1, 2).Value * 0.000001,
15)

Note that if you round to 18 decimal places instead of 15 (or whatever level
of precision you need), you will end up with the same answer as if you did
not round.

HTH

--
Michael J. Malinsky
Pittsburgh, PA

"I am a bear of very little brain, and long
words bother me." -- AA Milne, Winnie the Pooh

"Adrian" wrote in message
...
When I run the following code:

Public Sub test()
Cells(1, 3).Value = Cells(1, 1).Value - Cells(1,
2).Value * 0.000001
End Sub


with input data:
A1=0.035
A2=35000

I expect to get A3=0; however I get A3 = 6.94E-18!

Can anyone explain why and how this can be avoided?

Thank you in advance for your help,

Adrian F.
Ottawa





  #3   Report Post  
Posted to microsoft.public.excel.programming
external usenet poster
 
Posts: 27,285
Default Help Needed: Calculation error

Public Sub test()
Cells(1, 3).Value = Format(Cells(1, 1).Value - _
Cells(1, 2).Value * 0.000001,"#0.0000000")
End Sub

It is due to the fact that all decimal numbers can not be represented
exactly using binary. Just like 1/3 can't be exactly represented using
Decimal numbers.

--
Regards,
Tom Ogilvy

"Adrian" wrote in message
...
When I run the following code:

Public Sub test()
Cells(1, 3).Value = Cells(1, 1).Value - Cells(1,
2).Value * 0.000001
End Sub


with input data:
A1=0.035
A2=35000

I expect to get A3=0; however I get A3 = 6.94E-18!

Can anyone explain why and how this can be avoided?

Thank you in advance for your help,

Adrian F.
Ottawa





Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Payroll Calculation - Help Needed Hoov Excel Discussion (Misc queries) 3 February 17th 10 04:53 PM
Formula for the calculation needed to come up with a total Phillse Excel Worksheet Functions 5 April 7th 09 07:05 PM
Calculation with EDATE Help Needed! Danny Boy Excel Worksheet Functions 2 December 30th 08 01:58 AM
Help needed with a calculation question CP Excel Worksheet Functions 6 November 26th 07 04:53 PM
Time Sheet Calculation Help Needed! sax30 Excel Worksheet Functions 2 April 26th 05 08:08 PM


All times are GMT +1. The time now is 04:05 AM.

Powered by vBulletin® Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"