Home |
Search |
Today's Posts |
#1
Posted to microsoft.public.excel.misc
|
|||
|
|||
Error calculation
Dear Support,
I was tryting to calculate the total variance with expected result equal to 0. But we received a very strange result. We subtract Figure1 data from Figure 2 and receive variances. After this we used Sum function to calculate the total amount and see strange number see below. While if we calculated Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive correct result. Could you please clarify this issue. Regards, Sergey. # Figure 1 1 32400.000000000000000 2 14400.000000000000000 3 24256.400000000000000 4 57.600000000000000 5 21600.000000000000000 6 10800.000000000000000 103514.000000000000000 Figure 2 1 30960.000000000000000 2 14400.000000000000000 3 40010.000000000000000 4 144.000000000000000 5 7200.000000000000000 6 10800.000000000000000 103514.000000000000000 Variance on Totals 0.000000000000000 Variance by line (Figure 2 - Figure1) 1 -1440.000000000000000 2 0.000000000000000 3 15753.600000000000000 4 86.400000000000000 5 -14400.000000000000000 6 0.000000000000000 Sum of Variance -0.000000000012733 |
#2
Posted to microsoft.public.excel.misc
|
|||
|
|||
Error calculation
ITs a binary computation/rounding problem..
http://www.tech-archive.net/Archive/.../msg01520.html TFT "Vyacheslav" wrote: Dear Support, I was tryting to calculate the total variance with expected result equal to 0. But we received a very strange result. We subtract Figure1 data from Figure 2 and receive variances. After this we used Sum function to calculate the total amount and see strange number see below. While if we calculated Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive correct result. Could you please clarify this issue. Regards, Sergey. # Figure 1 1 32400.000000000000000 2 14400.000000000000000 3 24256.400000000000000 4 57.600000000000000 5 21600.000000000000000 6 10800.000000000000000 103514.000000000000000 Figure 2 1 30960.000000000000000 2 14400.000000000000000 3 40010.000000000000000 4 144.000000000000000 5 7200.000000000000000 6 10800.000000000000000 103514.000000000000000 Variance on Totals 0.000000000000000 Variance by line (Figure 2 - Figure1) 1 -1440.000000000000000 2 0.000000000000000 3 15753.600000000000000 4 86.400000000000000 5 -14400.000000000000000 6 0.000000000000000 Sum of Variance -0.000000000012733 |
#3
Posted to microsoft.public.excel.misc
|
|||
|
|||
Error calculation
Look at
=(ROUND(fig1,2)-ROUND(fig2,2)) You will find that the issue is that 24256.4-40010 differs from -15753.6 beyond the 15th digit. As Troy indicated, his happens because the binary representation of most decimal fractions is not exact. The math is right, given the unavoidable binary approximations to the numbers. Since you are only adding an subtracting numbers with a limited number of dcimal places, the simplest solution is to round results. Jerry "Vyacheslav" wrote: Dear Support, I was tryting to calculate the total variance with expected result equal to 0. But we received a very strange result. We subtract Figure1 data from Figure 2 and receive variances. After this we used Sum function to calculate the total amount and see strange number see below. While if we calculated Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive correct result. Could you please clarify this issue. Regards, Sergey. # Figure 1 1 32400.000000000000000 2 14400.000000000000000 3 24256.400000000000000 4 57.600000000000000 5 21600.000000000000000 6 10800.000000000000000 103514.000000000000000 Figure 2 1 30960.000000000000000 2 14400.000000000000000 3 40010.000000000000000 4 144.000000000000000 5 7200.000000000000000 6 10800.000000000000000 103514.000000000000000 Variance on Totals 0.000000000000000 Variance by line (Figure 2 - Figure1) 1 -1440.000000000000000 2 0.000000000000000 3 15753.600000000000000 4 86.400000000000000 5 -14400.000000000000000 6 0.000000000000000 Sum of Variance -0.000000000012733 |
Reply |
Thread Tools | Search this Thread |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
Calculation error? | Excel Discussion (Misc queries) | |||
Calculation Error | Excel Discussion (Misc queries) | |||
Day Calculation Error | Excel Discussion (Misc queries) | |||
Calculation Error? | Excel Discussion (Misc queries) | |||
calculation error | Excel Worksheet Functions |