View Single Post
  #5   Report Post  
Posted to microsoft.public.excel.misc
BobD BobD is offline
external usenet poster
 
Posts: 6
Default A wrong calculation by excel

The problem arose because I have an error message built into my spreadsheet
that only appears if the calculated value using the formula is less than
another value calculated by a different means. This then creates a problem
as we are instructing the users that error messages must be cleared, but in
this case, the error message is incorrect. The spreadsheet calculates
working hours and then determines how much are ordinary hours, how much is
overtime, shift penalties etc. The bizaar thing is that it only happens with
a start time of 8.30 and finish at 17.15. If you change either of those two
by even a minute, it doesn't happen. For example, start time of 7.30 and
finish time of 16.15 produces exactly 8.75 hours.
--
BobD


"Stephen Bye" wrote:

As a time value, the difference between 8.74999999999999 and 8.75 is less
than 1 nanosecond. Is that not accurate enough for you? What are you
measuring that requires greater precision?

"BobD" wrote in message
...
The rounding error should not occur. If you do the calculation with a
calculator, it produces an answer of exactly 8.75
--
BobD


"JMB" wrote:

Rounding error. Use Round function to 2 decimal places.

http://cpearson.com/excel/rounding.htm


"BobD" wrote:

I am using Excel 2003 Professional Edition. If you include the
following
formula into a cell, it produces a result of 8.74999999999999 if you
expand
the decimal places to 16. The correct answer is 8.75. The formula is
((17.15-17)*10/6+17)-((8.3-8)*10/6+8). This only happens if you use
17.15 and
8.3. I came across this when using a process to convert times to
decimals
and then subtract one from the other. Anyone have any ideas?
--
BobD