Error estimates in the solution of linear systems C. Brezinski
Abstract
Let Ax = b be a system of linear equations whose solution is denoted by x = A?1 b. When solving it by a direct or an iterative method, an approximate solution x is obtained (x denotes the current iterate in the case of an iterative method). In this paper, we propose estimates for the norm of the error e = x ? x. These estimates are valid for arbitrary matrices and for approximations of x obtained by any method.
Let us consider the system of p real linear equations Ax = b. We shall denote its solution by x = A?1b. When solving it by a direct or an iterative method, an approximate solution x is obtained (x denotes the current iterate in the case of an iterative method). Usually the quality of the approximate solution is judged by the norm of the residual vector r = b ? Ax. The error x ? x and the residual are related by r = Ae and, thus, it is not possible to compute the error from the residual. However, it holds 1 kAk kek krk kek 1 ?1 kA k krk 1 where = kAk kA?1k, the norms being the Euclidean ones. Thus, if kAk or kA?1k are known, the quantities krk=kAk and kA?1k krk can be considered as estimates of kek and we have the bounds krk kek kA?1k krk: (1) kAk However, these estimates require the knowledge of the Euclidean norm of A or of its inverse and, moreover, in somes cases, the bounds can be quite large. In this paper, we shall propose other estimates of the norm of the error e = x ? x based on the relation r = Ae. These estimates are valid for arbitrary matrices and for approximations of Laboratoire d'Analyse Numerique et d'Optimisation, Universite des Sciences et Technologies de Lille, 59655{ Villeneuve d'Ascq Cedex, France. E{mail:
[email protected]
1
x obtained by an arbitrary method, direct or iterative. They do not need estimates of the largest and smallest eigenvalues of the matrix nor of its norm, and they provide lower and upper bounds on the norm of the error if the condition number is known. Let us set c0 = (r; r); c1 = (r; Ar) and c2 = (Ar; Ar). We have (e; e) = (A?1r; A?1r) = c?2. In this paper, we shall be concerned with the problem of estimating c?2 . This question was already treated in a number of papers [4, 5, 8, 9, 10, 13] where bounds on c?2 were obtained by quadrature rules in the case where the matrix A is symmetric positive de nite and the conjugate gradient method or a method minimizing some quadratic functional is used. In this paper, c?2 will be estimated from c0; c1 and c2. Our approach is based on the extrapolation of the rst 3 terms the sequence (c ); n = 0; 1; : : : at the point n = ?2. In the rst Section, we shall explain how to derive such estimates and justify our approach. The formulae obtained will be analyzed in the second Section where bounds on the norm of the error will also be given. The last Section is devoted to numerical examples. n
1 Derivation of the estimates Let us consider the singular value decomposition (SVD) of the matrix A A = U V with UU = V V = I and =diag(1; : : :; ) with 1 2 > 0. y being an arbitrary vector, u1; : : : ; u and v1; : : :; v denoting respectively the columns of the matrices U and V , we have T
T
T
p
p
p
p
Ay =
X p
=1
(v ; y)u i
i
i
i
A?1y
=
X p
=1
?1(u ; y)v : i
i
i
i
It follows immediately
c0 = (r; r) = (U r; U r) = T
T
X p
=1
(u ; r)2 i
(2)
i
= (V r; V r) = T
c1 = (r; Ar) =
X p
=1
T
X p
=1
(v ; r)2 i
(3)
i
(u ; r)(v ; r)
(4)
2(v ; r)2
(5)
i
i
i
i
c2 = (Ar; Ar) =
X p
=1
i
i
i
c?1 = (r; A?1r) =
X p
=1
i
2
?1(u ; r)(v ; r) i
i
i
(6)
c?2
= (A?1r; A?1r)
= (e; e) =
X p
=1
?2(u ; r)2: i
(7)
i
i
Approximations of c?2 and c?1 can be obtained by keeping only the rst term in each of the formulae (2{7). So, we shall look for ; and satisfying the interpolation conditions
9 c0 = 2 = 2 > = c1 = > (8) c2 = 22 ; and then extrapolate at the values ?1 and ?2 of the index. In other terms, c?1 and c?2 will be approximated by c?1 ' ?1 and c?2 ' 2?2: Computing the unknowns ; and from the interpolation conditions (8) (which do not have a unique solution), thus leads to the following estimates of c1?22 = kek =
e21 e22 e23 e24 e25
= = = = =
c41=c32 c0c21=c22 c20=c2 c30=c21 c40c2=c41 :
(9) (10) (11) (12) (13)
When A is symmetric positive de nite, the quantity c?1 = (r; A?1r) = (e; Ae) is called the energy norm. From what precedes, we see that it can be estimated, for example, by c20=c1. All these formulae require the computation of Ar. However, in some methods such as Lanczos/Orthores, these products are already needed and, thus, the preceding estimates can be obtained for free. In other cases, they can be computed only from time to time. It is also possible to construct approximations of c?2 using extrapolation formulae with a sum of terms of the form 2?2 instead of a single one. However, such formulae require more matrix{by{vector products (they also sometimes need A ) and, moreover, the improvement over the preceding estimates does not seem so clear. So, we shall not pursue in this direction. T
Remark 1
There are many iterative methods for solving linear systems where the iterates and the residuals are obtained recursively by formulae of the form
x +1 = x + z r +1 = r ? Az k
k
k
k
k
k
k
k
where z is some vector and a parameter. Instead of considering such methods, it is possible to compute the iterates x and the corresponding residuals r by an arbitrary iterative method and, then, to transform them into the new k
k
k
k
3
iterates y and the new corresponding residuals = b ? Ay by formulae similar to the preceding ones, that is k
k
k
y = x + z = r ? Az : k
k
k
k
k
k
k
k
Under some assumptions (see [2]), the sequence (k k) converges faster than the sequence (kr k). In such cases, y is usually a better approximation of the solution x than x . Thus, one can consider y ? x = z as a good approximation of the error x ? x . Thus j j kz k is a good estimate of kx ? x k. Conversely, as showed in [1], estimates of the error form the basis for constructing convergence acceleration methods. Thus, the duality between both questions is completed. k
k
k
k
k
k
k
k
k
k
k
k
Let us now analyze the preceding estimates.
2 Justi cation of the estimates Let us begin by comparing these estimates together. We have (r; Ar)2 (Ar; Ar)(r; r) kAk2(r; r)2 that is
0 c21 c0c2
and it easily follows
Theorem 1
(14)
e1 e2 e3 e4 e5:
We have (with the indexes in the sums always running from 1 to p, the dimension of the system)
X X 2 (v ; r)2 P 2(v ; r)2 12 (v ; r)2 X X 1?2 (u ; r)2 P ?2(u ; r)2 ?2 (u ; r)2 p
i
i
i
i
i
i
i
p
i
that is, using the fact that = kAk kA?1k = 1= , p
c20 c (e; e) 2c2: 0 2 2 We set = c21=(c0c2). From the inequalities (14) and (15), we immediately obtain
4
(15)
Theorem 2
1 1 p 1 p
Since 1, it follows
kek e1 kek p e2 kek
e3 kek p e4 kek : e5
0 p 1 p1 1 : We also have 1 and, thus, for the upper bounds it holds 0 p p :
We know that 1 but nothing can be said about p and compared to 1. For the lower bounds, we have p 1 1 1: 0 p We know that 1= 1 but nothing can be said about 1=(p) and 1=() compared to 1. So, we are not sure that the intervals for kek=e given by the inequalities of Theorem 2 contain 1 except for kek=e3. If A is orthogonal, then = 1 and it follows (16) kek = e1 = pe2 = e3 = p e4 = e5 i
which shows that e3 is exact. So, for these reasons, e3 seems to be the most appropriate estimate of kek and this is con rmed by the numerical examples given in the last Section. If is known, the inequalities of Theorem 2 provide bounds on the norm of the error. In fact, replacing and the e 's by their expressions, it is easy to see that they all lead to the same bounds and we have i
Theorem 3
e3 kek e : 3 These inequalities show that, if A is not too ill{conditioned, the estimates e are good approximations of kek and, in particular, e3. i
5
3 Numerical results Let us now give some numerical results for illustrating the preceding error estimates and bounds. We have also tested the bounds given by (1). On the Figures, these bounds are indicated as normed residuals. In all cases, the solution was chosen randomly and b computed accordingly. The methods tested were Lanczos/Orthodir, the transpose{free version of Lanczos/Orthomin and the coupled implementation of the BiCGSTAB of van der Vorst [14] given in [3], the method of Jacobi and that of Gastinel [6, 7] (which is always convergent but often slowly) and consists of the iterations r) Ar: x +1 = x + (A (rr ;; A r) For ill{conditioned systems, we also made use of the truncated singular value decomposition (TSVD) [11]. As above, let A = U V be the singular value decomposition of A. The TSVD consists of approximating x by x given by k
k
k
T
k
T
T
k
k
k
T
k
X u =1 where = (u ; b). Obviously, we should obtain x = x but, due to the ill{conditioning, we observe that, when k increases, the error begins to decrease and then can become quite large. All methods were started from x0 = 0. k
x =
i
k
i
i
i
i
i
p
Example 1.
Let us consider the pentadiagonal matrix de ned by a =6 i = 1; : : :; p a +1 = 2; a +1 = ?1 i = 1; : : :; p ? 1 a +2 = 1; a +2 = 3 i = 1; : : :; p ? 2: ii
i;i
i
;i
i;i
i
;i
For p = 50, cond(A) = 3:28, kAk = 10:9 and, for Lanczos/Orthodir, we obtain the results of Figure 1 where the bounds given by Theorem 3 are also displayed. The bounds given by (1) are shown in Figure 2. We see that all the estimates are quite good and also the bounds given by (1) and by Theorem 3. The reason is that the system is well{conditioned.
Example 2.
Our second example is concerned with the symmetric pentadiagonal matrix de ned by a = 7:5 i = 1; : : : ; p a +1 = a +1 = 1 i = 1; : : : ; p ? 1 a +2 = a +2 = ?2 i = 1; : : : ; p ? 2: ii
i;i
i
;i
i;i
i
;i
With p = 50, cond(A) = 7:56, kAk = 11:6 and we obtain the results of Figure 3 for the methods of Gastinel and Jacobi. 6
Orthodir: all errors
5
10
0
0
10
10
−5
−5
10
10
−10
−10
10
10
−15
10
Orthodir: bounds
5
10
0
−15
20
40
10
60
0
20
40
60
Figure 1: Example 1 For the same methods, the bounds given by (1) are shown in Figure 4. In this case also, although the condition number of the matrix is bigger than in the rst example, the results are quite good.
Example 3.
Let us now give an example with a convergence behavior not so regular. We consider the circulant matrix whose rst row is 1; : : : ; p. For p = 50, cond(A) = 51, kAk = 1275 and the results of Lanczos/Orthodir and BiCGSTAB are shown on Figure 5. For the BiCGSTAB, all curves have the same aspect but not for Lanczos/Orthodir. On Figure 6, only the exact error and e3 are plotted. We see that e3 is a quite good estimation of the norm of the error.
Example 4.
Let us now give an example with an ill{conditioned matrix. We used the matrix kahan with default parameters from the MATLAB test matrix toolbox [12]. For p = 60, cond(A) = 1:97e +10, kAk = 6:89. The results obtained by the method of Jacobi and the TSVD are given on Figure 7. The behavior of the method of Jacobi seems strange but the estimates of the error are still sharp. Moreover, the lower bound given by (1) is almost attained. On Figure 7:b, we also see that the true error stagnates at the end and that the estimates still decrease. This phenomenon is almost always observed when the true error reaches the computer accuracy and it is due to the discrepancy between the iterative residual which continues to decrease and the actual one. As shown in Figure 8, the upper bounds given by the normed residuals (1) can be attained. Obviously, the lower bound can be reached as we saw for the method of Jacobi and as we shall also see in the next example.
Example 5.
Let us now consider the matrix parter from the matrix toolbox [12]. It is given by a = 1=(i ? j + 0:5) and, for p = 50, cond(A) = 3:04, kAk = 3:14. For Lanczos/Orthomin, Figure 9 ij
7
Orthodir: error and normed residuals
2
10
0
10
−2
10
−4
10
−6
10
−8
10
−10
10
−12
10
−14
10
0
5
10
15
20
25
30
35
40
45
50
Figure 2: Example 1 shows the bounds given by the normed residuals (1) (in this case, we see that the lower bound is attained) and e3. In Figure 9.b, the lowest curve is e3.
Example 6.
We shall now take an arbitrary matrix and m random solutions and then solve the system by a direct method(in fact, with the MATLAB command x = Anb) and compute the mean values of the error estimates. For the parter matrix of dimension p = 50 and for m = 100, we get an exact error of 6:48e ? 15. The estimates are respectively e1 = 4:17e ? 16; e2 = 8:07e ? 16; e3 = 1:58e ? 15; e4 = 3:70e ? 15; e5 = 6:48e ? 15: The bounds given by the normed residuals (1) are 1:51e ? 15 and 4:60e ? 15. Those obtained from e3 are 5:21e ? 16 and 4:82e ? 15. The condition number of this matrix is 3:04. 8
Gastinel: all errors
1
Jacobi: all errors
2
10
10
0
10
0
10
−1
10
−2
−2
10
10
−3
10
−4
10
−4
10
−5
10
0
−6
50
100
150
10
200
0
20
40
60
Figure 3: Example 2 For the orthogonal q matrix orthog of dimension p = 50 of the matrix toolbox [12] whose elements are a = 2=(p + 1) sin ij=(p + 1), we have an exact error (with m = 100) equal to 3:87e ? 15. Since the norm and the condition number of this matrix are both 1, the bounds (1) and those given by e3 are equal to the exact value of the error. We obtain ij
e1 = 1:95e ? 16; e2 = 6:80e ? 16; e3 = 3:87e ? 15; e4 = 6:76e ? 14; e5 = 4:05e ? 12: Since the value of is di erent for each trial, the mean values of the is e 's do not satisfy the equalities (16). i
References [1] C. Brezinski, A new approach to convergence acceleration methods, in Nonlinear Numerical Methods and Rational Approximation, A. Cuyt ed., Reidel, Dordrecht, 1988, pp. 373{405. [2] C. Brezinski, Projection Methods for Systems of Equations, North{Holland, Amsterdam, to appear. [3] C. Brezinski, M. Redivo{Zaglia, Transpose{free coupled product{type Lanczos algorithms for nonsymmetric linear systems, submitted. [4] G. Dahlquist, S.C. Eisenstat, G.H. Golub, Bounds for the error in linear systems of equations using the theory of moments, J. Math. Anal. Appl., 37 (1972) 151{166. [5] G. Dahlquist, G.H. Golub, S.G. Nash, Bounds for the error in linear systems, in Proceedings of the Workshop on Semi{In nite Programming, R. Hettich ed., Springer Verlag, Berlin, 1978, pp. 154{172. 9
Gastinel: error and normed residuals
2
Jacobi: error and normed residuals
2
10
10
0
10
0
10
−2
10
−2
10
−4
10 −4
10
−6
10
−6
10
0
−8
50
100
150
10
200
0
20
40
60
Figure 4: Example 2 [6] N. Gastinel, Procede iteratif pour la resolution numerique d'un systeme d'equations lineaires, C.R. Acad. Sci. Paris, 246 (1958) 2571{2574. [7] N. Gastinel, Sur{decomposition de normes generales et procedes iteratifs, Numer. Math., 5 (1963) 142{151. [8] G.H. Golub, Bounds for matrix moments, Rocky Mt. J. Math., 4 (1974) 207{211. [9] G.H. Golub, G. Meurant, Matrices, moments and quadrature, in Numerical Analysis 1993, D.F. Griths and G.A. Watson eds., Pitman Research Notes in Mathematics vol. 303, Longman Sci. and Tech., Harlow, 1994, pp. 105{156. [10] G.H. Golub, Z. Strakos, Estimates in quadratic formulas, Numerical Algorithms, 8 (1994) 241{268. [11] P.C. Hansen, The truncated SVD as a method for regularization, BIT, 27 (1987) 534{553. [12] N.J. Higham, The test matrix toolbox for MATLAB (Version 3.0), Numerical Analysis Report No. 276, Departments of Mathematics, The University of Manchester, September 1995. [13] G. Meurant, Matrices, moments et quadrature II, Rapport CEA{N{2814, CEN de Limeil{ Valenton, 1996. [14] H.A. van der Vorst, BiCGSTAB: a fast and smoothly converging variant of BiCG for the solution of nonsymmetric linear systems, SIAM J. Sci. Stat. Comput., 13 (1992) 631{644.
10
BiCGSTAB: all errors
4
Orthodir: all errors
5
10
10
2
10
0
10
0
10
−2
10
−5
10
−4
10
−6
10
0
−10
20
40
10
60
0
20
40
60
Figure 5: Example 3
BiCGSTAB: error and e_3
2
10
10
0
0
10
10
−2
−2
10
10
−4
−4
10
10
−6
10
Orthodir: error and e_3
2
0
−6
20
40
10
60
0
Figure 6: Example 3
11
20
40
60
Jacobi: error and normed residuals
20
Jacobi: all errors
10
10
10
5
10
10
10
0
10
0
10
−5
10 −10
10
−10
10
−20
10
0
−15
20
40
10
60
Jacobi: bounds
20
20
40
60
TSVD: all errors
5
10
10
10
0
10
10
0
−5
10
10
−10
−10
10
10
−20
−15
10
10
−30
10
0
0
−20
20
40
10
60
0
Figure 7: Example 4
12
20
40
60
BiCGSTAB: error and normed residuals
12
10
10
10
8
10
6
10
4
10
2
10
0
10
−2
10
−4
10
−6
10
−8
10
0
10
20
30
Figure 8: Example 4
13
40
50
60
2
Orthomin: error and normed residuals
Orthomin: error and e_3
2
10
10
1
10
0
10
0
10
−1
−2
10
10
−2
10
−4
10
−3
10
−4
10
0
−6
20
40
10
60
0
Figure 9: Example 5
14
20
40
60
