Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1   Report Post  
Erny Meyer
 
Posts: n/a
Default Precision problem with automated formula fill-in

I start with a value of -1 in cell A1 of a worksheet. In
cell A2 I enter the formula =A1+0.1

I repeat the formula downwards. At cell A11 (where I
should read 0), I find an approximation -1.4E-16

Changing the formula to =(A1*10+1)/10 (which is basically
the same) eliminates the problem.

Can this type of problem be avoided through a setup option?
  #2   Report Post  
JE McGimpsey
 
Posts: n/a
Default

You could use Tools/Options/Calculation Precision as displayed... Note
that that is a global setting. Or you could change your formula to

=ROUND(A1+0.1,1)

The problem is that 0.1 cannot be represented exactly in finite binary
digits, much as 1/3 can't be represented exactly in finite decimal
digits (e.g., 0.3333333...)
See

http://cpearson.com/excel/rounding.html

In article ,
"Erny Meyer" wrote:

I start with a value of -1 in cell A1 of a worksheet. In
cell A2 I enter the formula =A1+0.1

I repeat the formula downwards. At cell A11 (where I
should read 0), I find an approximation -1.4E-16

Changing the formula to =(A1*10+1)/10 (which is basically
the same) eliminates the problem.

Can this type of problem be avoided through a setup option?

Reply
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules

Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Excel - formula to calculate colored fill cells within a range wi. MA Excel Worksheet Functions 1 January 7th 05 04:06 PM
Baffling formula problem Ken Schmidt Excel Discussion (Misc queries) 2 December 21st 04 07:52 AM
Identifying the Active Fill Color Steve Conary Excel Discussion (Misc queries) 3 December 9th 04 04:45 AM
Formula Fill properties vfalzone Excel Worksheet Functions 2 November 12th 04 01:05 PM
A different "Fill Color" problem JKD Excel Worksheet Functions 1 November 1st 04 10:21 PM


All times are GMT +1. The time now is 02:15 PM.

Powered by vBulletin® Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright ©2004-2025 ExcelBanter.
The comments are property of their posters.
 

About Us

"It's about Microsoft Excel"