Error calculation
 

Dear Support,
I was tryting to calculate the total variance with expected result equal to
0. But we received a very strange result. We subtract Figure1 data from
Figure 2 and receive variances. After this we used Sum function to calculate
the total amount and see strange number see below. While if we calculated
Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive
correct result. Could you please clarify this issue. Regards, Sergey.
# Figure 1
1 32400.000000000000000
2 14400.000000000000000
3 24256.400000000000000
4 57.600000000000000
5 21600.000000000000000
6 10800.000000000000000
103514.000000000000000


Figure 2
1 30960.000000000000000
2 14400.000000000000000
3 40010.000000000000000
4 144.000000000000000
5 7200.000000000000000
6 10800.000000000000000
103514.000000000000000

Variance on Totals 0.000000000000000

Variance by line (Figure 2 - Figure1)
1 -1440.000000000000000
2 0.000000000000000
3 15753.600000000000000
4 86.400000000000000
5 -14400.000000000000000
6 0.000000000000000

Sum of Variance -0.000000000012733

Error calculation
 

ITs a binary computation/rounding problem..

http://www.tech-archive.net/Archive/.../msg01520.html

TFT

"Vyacheslav" wrote:

Dear Support,
I was tryting to calculate the total variance with expected result equal to
0. But we received a very strange result. We subtract Figure1 data from
Figure 2 and receive variances. After this we used Sum function to calculate
the total amount and see strange number see below. While if we calculated
Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive
correct result. Could you please clarify this issue. Regards, Sergey.
# Figure 1
1 32400.000000000000000
2 14400.000000000000000
3 24256.400000000000000
4 57.600000000000000
5 21600.000000000000000
6 10800.000000000000000
103514.000000000000000


Figure 2
1 30960.000000000000000
2 14400.000000000000000
3 40010.000000000000000
4 144.000000000000000
5 7200.000000000000000
6 10800.000000000000000
103514.000000000000000

Variance on Totals 0.000000000000000

Variance by line (Figure 2 - Figure1)
1 -1440.000000000000000
2 0.000000000000000
3 15753.600000000000000
4 86.400000000000000
5 -14400.000000000000000
6 0.000000000000000

Sum of Variance -0.000000000012733

Error calculation
 

Look at
=(ROUND(fig1,2)-ROUND(fig2,2))
You will find that the issue is that 24256.4-40010 differs from -15753.6
beyond the 15th digit. As Troy indicated, his happens because the binary
representation of most decimal fractions is not exact. The math is right,
given the unavoidable binary approximations to the numbers.

Since you are only adding an subtracting numbers with a limited number of
dcimal places, the simplest solution is to round results.

Jerry

"Vyacheslav" wrote:

Dear Support,
I was tryting to calculate the total variance with expected result equal to
0. But we received a very strange result. We subtract Figure1 data from
Figure 2 and receive variances. After this we used Sum function to calculate
the total amount and see strange number see below. While if we calculated
Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive
correct result. Could you please clarify this issue. Regards, Sergey.
# Figure 1
1 32400.000000000000000
2 14400.000000000000000
3 24256.400000000000000
4 57.600000000000000
5 21600.000000000000000
6 10800.000000000000000
103514.000000000000000


Figure 2
1 30960.000000000000000
2 14400.000000000000000
3 40010.000000000000000
4 144.000000000000000
5 7200.000000000000000
6 10800.000000000000000
103514.000000000000000

Variance on Totals 0.000000000000000

Variance by line (Figure 2 - Figure1)
1 -1440.000000000000000
2 0.000000000000000
3 15753.600000000000000
4 86.400000000000000
5 -14400.000000000000000
6 0.000000000000000

Sum of Variance -0.000000000012733