Home |
Search |
Today's Posts |
|
#1
![]()
Posted to microsoft.public.excel.worksheet.functions
|
|||
|
|||
![]()
This is a common occurrence: expecting zero and getting very close to it. It
all has to do with IEEE 754 and the way Excel (and moth other computer apps) convert decimal numbers (numbers using base 10) to binary numbers (numbers using base 2). Only a problem with real numbers (i.e. numbers with decimal places) not with integers How to avoid: round to 12 decimal places =ROUND(COS(RADIANS(90)),12) You handheld calculator does this automatically, so you never see it with that machine For a full explanation: Chip's clear explanation http://www.cpearson.com/excel/rounding.htm Floating-point arithmetic may give inaccurate results in Excel http://support.microsoft.com/kb/78113/en-us (Complete) Tutorial to Understand IEEE Floating-Point Errors http://support.microsoft.com/kb/42980 What Every Computer Scientist Should Know About Floating Point http://docs.sun.com/source/806-3568/ncg_goldberg.html http://www.cpearson.com/excel/rounding.htm Visual Basic and Arithmetic Precision http://support.microsoft.com/default...NoWebContent=1 Others: http://support.microsoft.com/kb/214118 http://docs.sun.com/source/806-3568/ncg_goldberg.html best wishes -- Bernard V Liengme Microsoft Excel MVP http://people.stfx.ca/bliengme remove caps from email "mikelee101" <mikelee101athotmaildotcom wrote in message ... Hello All, I was just curious why I wasn't getting zero for the result of the cosine of a 90 degree angle. This formula =COS(RADIANS(90)) returns 6.12574E-17 when I would have expected 0. Am I misusing a function in there somewhere? Any thoughts would be greatly appreciated. Excel 07, XPPro SP3. Thanks. -- Mike Lee McKinney,TX USA |
#2
![]()
Posted to microsoft.public.excel.worksheet.functions
|
|||
|
|||
![]()
On Sep 14, 7:53�pm, "Bernard Liengme" wrote:
<snip How to avoid: round to 12 decimal places =ROUND(COS(RADIANS(90)),12) You handheld calculator does this automatically, so you never see it with that machine <snip 12 decimal places would give you an accuracy of about 6 inches / 150mm in measuring the distance between the earth & the sun - good enough for government work. Alan Lloyd |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
![]() |
||||
Thread | Forum | |||
How do you convert decimal degrees to degrees minutes seconds | Excel Discussion (Misc queries) | |||
how can i convert degrees/minutes/seconds to decimal degrees? | Excel Discussion (Misc queries) | |||
calculate from degrees to degrees/minutes/seconds on excel? | Excel Worksheet Functions | |||
convert decimal degrees to degrees minutes seconds | Excel Discussion (Misc queries) | |||
converting from digital degrees to degrees minutes seconds | Excel Worksheet Functions |