Method of Conjugate Gradients - nanoHUB

83 downloads 0 Views 2MB Size Report
Conjugate Gradient Method. 1. Introduction, Notation and Basic Terms. 2. Eigenvalues and Eigenvectors. 3. The Method of Steepest Descent. 4. Convergence ...
Method of Conjugate Gradients Dragica Vasileska

CG Method – Conjugate Gradient (CG) Method M. R. Hestenes & E. Stiefel, 1952 BiCG Method – Biconjugate Gradient (BiCG) Method C. Lanczos, 1952 D. A. H. Jacobs, 1981 C. F. Smith et al., 1990 R. Barret et al., 1994 CGS Method – Conjugate Gradient Squared (CGS) Method (MATLAB Function) P. Sonneveld, 1989 GMRES Method – Generalized Minimal – Residual (GMRES) Method Y. Saad & M. H. Schultz, 1986 R. Barret et al., 1994 Y. Saad, 1996 QMR Method – Quasi–Minimal–Residual (QMR) Method R. Freund & N. Nachtigal, 1990 N. Nachtigal, 1991 R. Barret et al., 1994 Y. Saad, 1996

Conjugate Gradient Method 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Introduction, Notation and Basic Terms Eigenvalues and Eigenvectors The Method of Steepest Descent Convergence Analysis of the Method of Steepest Descent The Method of Conjugate Directions The Method of Conjugate Gradients Convergence Analysis of Conjugate Gradient Method Complexity of the Conjugate Gradient Method Preconditioning Techniques Conjugate Gradient Type Algorithms for NonSymmetric Matrices (CGS, Bi-CGSTAB Method)

1. Introduction, Notation and Basic Terms • The CG is one of the most popular iterative methods for solving large systems of linear equations Ax=b which arise in many important settings, such as finite difference and finite element methods for solving partial differential equa-tions, circuit analysis etc. • It is suited for use with sparse matrices. If A is dense, the best choice is to factor A and solve the equation by backsubstitution. • There is a fundamental underlying structure for almost all the descent algorithms: (1) one starts with an initial point; (2) determines according to a fixed rule a direction of movement; (3) mo-ves in that direction to a relative minimum of the objective function; (4) at the new point, a new direction is determined and the process is repeated. The difference between different algorithms depends upon the rule by which successive directions of movement are selected.

1. Introduction, Notation and Basic Terms (Cont’d) • A matrix is a rectangular array of numbers, called elements. • The transpose of an mn matrix A is the nm matrix AT with elements aijT = aji • Two square nn matrices A and B are similar if there is a nonsingular matrix S such that B=S-1AS • Matrices having a single row are called row vectors; matrices having a single column are called column vectors. Row vector: a = [a1, a2, …, an] Column vector: a = (a1, a2, …, an) • The inner product of two vectors is written as T

n

x y   x i yi i 1

1. Introduction, Notation and Basic Terms (Cont’d) • A matrix is called symmetric if aij = aji . • A matrix is positive definite if, for every nonzero vector x xTAx > 0 • Basic matrix identities: (AB)T=BTAT and (AB)-1=B-1A-1 • A quadratic form is a scalar, quadratic function of a vector with the form 1 T f(x)  x Ax  b T x  c 2 • The gradient of a quadratic form is   f(x)   x 1  1 T 1     A x  Ax  b f ' ( x)     2 2 f(x)   x n 

1. Introduction, Notation and Basic Terms (Cont’d) • Various quadratic forms for an arbitrary 22 matrix: Positive-definite matrix

Singular positive-indefinite matrix

Negative-definite matrix

Indefinite matrix

1. Introduction, Notation and Basic Terms (Cont’d) • Specific example of a 22 symmetric positive definite matrix: 3 2 2 2    A , b , c0  x  2 6   8   2  Contours of the quadratic form

Gradient of the quadratic form

Graph of a quadratic form f(x)

2. Eigenvalues and Eigenvectors • For any nn matrix B, a scalar  and a nonzero vector v that satisfy the equation Bv=v are said to be the eigenvalue and the eigenvector of B. • If the matrix is symmetric, then the following properties hold: (a) the eigenvalues of B are real (b) eigenvectors associated with distinct eigenvalues are orthogonal • The matrix B is positive definite (or positive semidefinite) if and only if all eigenvalues of B are positive (or nonnegative). • Why should we care about the eigenvalues? Iterative methods often depend on applying B to a vector over and over again: (a) If ||1, then Biv=iv will grow to infinity.

2. Eigenvalues and Eigenvectors (Cont’d) • Examples for ||1 =-0.5

2

• The spectral radius of a matrix is: (B)= max|i| =0.7 =-2

2. Eigenvalues and Eigenvectors (Cont’d) • Example: The Jacobi Method for the solution of Ax=b - split the matrix A such that A=D+E - Instead of solving the system Ax=b, one solves the modified system x=Bx+z, where B=-D-1E and z=D-1b - The iterative method is: x(i+1)= Bx(i)+z, or it terms of the error vector: e(i+1)= Be(i) - For our particular example, we have that the eigenvalues and the eigenvectors of the matrix A are: 1=7, v1=[1, 2]T 2=2, v2=[-2, 1]T - The eigenvalues and eigenvectors of the matrix B are: 1=-2/3=-0.47, v1=[2, 1]T 2= 2/3=0.47, v2=[-2, 1]T

2. Eigenvalues and Eigenvectors (Cont’d) • Graphical representation of the convergence of the Jacobi method Eigenvectors of B together with their eigenvalues

Convergence of the Jacobi method which starts at [-2,-2]T and converges to [2, -2]T

The error vector e(0)

The error vector e(1)

The error vector e(2)

3. The Method of Steepest Descent • In the method of steepest descent, one starts with an arbitrary point x(0) and takes a series of steps x(1), x(2), … until we are satis-fied that we are close enough to the solution. • When taking the step, one chooses the direction in which f decreases most quickly, i.e.  f ' (x (i) )  b  Ax (i) • Definitions: error vector: e(i)=x(i)-x residual: r(i)=b-Ax(i) • From Ax=b, it follows that r(i)=-Ae(i)=-f’(x(i)) The residual is actually the direction of steepest descent

3. The Method of Steepest Descent (Cont’d) • Start with some vector x(0). The next vector falls along the solid line x (1)  x ( 0 )  r( 0 ) • The magnitude of the step is determined with a line search procedure that minimizes f along the line:

d T dx (1) f(x (1) )  f ' (x (1) ) 0 d d  r(T1)r( 0 )  0

b  Ax (1) T r(0)  0 b  A x (0)  r(0) T r(0)  0 r(T0 ) r( 0 )   Ar( 0 ) T r( 0 )  0





r(T0 ) r( 0 ) r(T0 ) Ar( 0 )

3. The Method of Steepest Descent (Cont’d) • Geometrical representation (e)

(f)

3. The Method of Steepest Descent (Cont’d) • The algorithm

r(i)  b  Ax (i)  (i) 

T r(i) r(i) T r(i) Ar(i)

Two matrix-vector multiplications are required.

x (i 1)  x (i)   (i) r(i)  e (i 1)  e (i)   (i) r(i) • To avoid one matrix-vector multiplication, one uses r(i 1)  r(i)   (i) Ar(i)

The disadvantage of using this recurrence is that the residual sequence is determined without any feedback from the value of x(i), so that round-off errors may cause x(i) to converge to some point near x.

4. Convergence Analysis of the Method of Steepest Descent • The error vector e(i) is equal to the eigenvector vj of A r(i)   Ae (i)   Av j    j v j r T r(i) 1 (i)  (i)   , e (i 1)  e (i)   (i) r(i)  0 r T Ar(i)  j (i)

• The error vector e(i) is a linear combination of the eigenvectors of A, and all eigenvalues are the same n

e (i)    j v j j1

n

n

r(i)   Ae (i)     j j v j      j v j T

j1

r r(i)  (i) 

(i)

r T Ar(i) (i)



j1

1 , e (i 1)  e (i)   (i) r(i)  0 

3. The Method of Steepest Descent • In the method of steepest descent, one starts with an arbitrary point x(0) and takes a series of steps x(1), x(2), … until we are satis-fied that we are close enough to the solution. • When taking the step, one chooses the direction in which f decreases most quickly, i.e.  f ' (x (i) )  b  Ax (i) • Definitions: error vector: e(i)=x(i)-x residual: r(i)=b-Ax(i) • From Ax=b, it follows that r(i)=-Ae(i)=-f’(x(i)) The residual is actually the direction of steepest descent

3. The Method of Steepest Descent (Cont’d) • Start with some vector x(0). The next vector falls along the solid line x (1)  x ( 0 )  r( 0 ) • The magnitude of the step is determined with a line search procedure that minimizes f along the line:

d T dx (1) f(x (1) )  f ' (x (1) ) 0 d d  r(T1)r( 0 )  0

b  Ax (1) T r(0)  0 b  A x (0)  r(0) T r(0)  0 r(T0 ) r( 0 )   Ar( 0 ) T r( 0 )  0





r(T0 ) r( 0 ) r(T0 ) Ar( 0 )

3. The Method of Steepest Descent (Cont’d) • Geometrical representation (e)

(f)

3. The Method of Steepest Descent (Cont’d) • Summary of the algorithm

r(i)  b  Ax (i) T

 (i) 

r (i) r(i) T

r (i) Ar(i)

x (i 1)  x (i)   (i)r(i) • To avoid one matrix-vector multiplication, one uses instead r(i 1)  r(i)   (i) Ar(i)

• Efficient implementation of the method of steepest descent i0 r  b  Ax   r Tr While i  i max and    2  0 q  Ar   T r q x  x  r If i is divisib le by 50 r  b - Ax else r  r - q endif   r Tr i  i 1

4. Convergence Analysis of the Method of Steepest Descent • The error vector e(i) is equal to the eigenvector vj of A r(i)   Ae (i)   Av j    j v j r T r(i) 1 (i)  (i)   , e (i 1)  e (i)   (i) r(i)  0 r T Ar(i)  j (i)

• The error vector e(i) is a linear combination of the eigenvectors of A, and all eigenvalues are the same n

e (i)    j v j j1

n

n

r(i)   Ae (i)     j j v j      j v j T

j1

r r(i)  (i) 

(i)

r T Ar(i) (i)



j1

1 , e (i 1)  e (i)   (i) r(i)  0 

4. Convergence Analysis of the Method of Steepest Descent (Cont’d) • General convergence of the method can be proven by calculating the energy norm 2

e(i 1)  e T(i 1) Ae (i 1) A

 e(i )

2 A

2

2

 ,   1

2 2 2  j j  j 2 3 2  j j  j  j j  j





• For n=2, one has that 2  1 

( 2   2 )2 (    2 )( 3   2 )



   max /  min ,    2 / 1 Conditioning number

 1  1







4. Convergence Analysis of the Method of Steepest Descent (Cont’d)

Worst-case scenario: =±

The influence of the spectral condition number on the convergence

5. The Method of Conjugate Directions • Basic idea: • Mathematical formulation: 1. Pick a set of orthogonal 1. For each step we choose a search directions d(0), d(1), … , point d(n-1) x(i+1)=x(i)+ (i) d(i) 2. Take exactly one step in 2. To find  (i), we use the fact each search direction to line up with x that e(i+1) is orthogonal to d(i)

d T( i ) e ( i 1)  0 d T( i ) e ( i )   ( i ) d ( i )   0 d T( i ) e ( i )   ( i ) d T( i ) d ( i )  0 (i)  

d T( i ) e ( i ) d T( i ) d ( i )

5. The Method of Conjugate Directions (Cont’d) • To solve the problem of not knowing e(i), one makes the search directions to be A-orthogonal rather then orthogonal to each other, i.e.: d T( i ) Ad ( j)  0

5. The Method of Conjugate Directions (Cont’d) • The new requirement is now that e(i+1) is A-orthogonal to d(i) d T dx (i 1) f(x (i 1) )  f ' (x (i 1) ) 0 d d r(iT1)d (i)  0

d T(i ) Ae (i 1)  0 d T(i ) A e (i )   (i )d (i )   0  (i ) 

d T(i) r(i) d T(i) Ad (i)

If the search vectors were the residuals, this formula would be identical to the method of steepest descent.

5. The Method of Conjugate Directions (Cont’d) • Proof that this process computes x in n-steps - express the error terms as a linear combination of the search directions n 1

e ( 0)    ( j) d ( j) j 0

- use the fact that the search directions are A-orthogonal

5. The Method of Conjugate Directions (Cont’d) • Calculation of the A-ortogonal search directions by a conjugate Gram-Schmidth process 1. Take a set of linearly independent vectors u0, u1, … , un-1 2. Assume that d(0)=u0 3. For i>0, take an ui and subtracts all the components from it that are not A-orthogonal to the previous search directions i 1

u T(i) Ad ( j)

j 0

d T(j) Ad ( j)

d (i )  u (i )   ijd ( j) , ij  

5. The Method of Conjugate Directions (Cont’d) • The method of Conjugate Directions using the axial unit vectors as basis vectors

5. The Method of Conjugate Directions (Cont’d) • Important observations:

1. The error vector is A-orthogonal to all previous search directions 2. The residual vector is orthogonal to all previous search directions 3. The residual vector is also orthogonal to all previous basis vectors. d iT Ae j  d iT rj  u iT rj  0 for i  j

6. The Method of Conjugate Gradients • The method of Conjugate Gradients is simply the method of conjugate directions where the search directions are constructed by conjugation of the residuals, i.e. ui=r(i) • This allows us to simplify the calculation of the new search direction because  1 r(Ti )r(i ) r(Ti )r(i )  T  ij    (i 1) d T Ad r(i 1)r(i 1) ( i 1) ( i 1)  0

i  j1 i  j1

• The new search direction is determined as a linear combination of the previous search direction and the new residual d (i 1)  r(i 1)  id (i)

6. The Method of Conjugate Gradients (Cont’d) • Efficient implementation i0 r  b  Ax dr  new  r T r  0   new While i  i max and  new   2  0 q  Ad    new dTq x  x  d If i is d ivisible b y 50 r  b - Ax else r  r - q endif  old   new  new  r T r    new  old d  r  d i  i 1

The Landscape of Ax=b Solvers Direct A = LU Nonsymmetric Symmetric positive definite

Iterative y’ = Ay

Pivoting LU

GMRES, BiCGSTAB, …

Cholesky

Conjugate gradient

More Robust

More General

More Robust Less Storage (if sparse)

Conjugate gradient iteration x0 = 0, r0 = b, d0 = r0 for k = 1, 2, 3, . . . ák = (rTk-1rk-1) / (dTk-1Adk-1) xk = xk-1 + ák dk-1 rk = rk-1 – ák Adk-1 âk = (rTk rk) / (rTk-1rk-1) dk = rk + âk dk-1

step length approx solution residual improvement search direction

• One matrix-vector multiplication per iteration • Two vector dot products per iteration • Four n-vectors of working storage

7. Convergence Analysis of the CG Method • If the algorithm is performed in exact arithmetic, the exact solution is obtained in at most n-steps. • When finite precision arithmetic is used, rounding errors lead to gradual loss of orthogonality among the residuals, and the finite termination property of the method is lost. • If the matrix A has only m distinct eigenvalues, then the CG will converge in at most m iterations • An error bound on the CG method can be obtained in terms of the A-norm, and after k-iterations:

x  xk

A

 2 x  x0

  ( A )  1 A   ( A )  1  

k

8. Complexity of the CG Method • Dominant operations of Steepest Descent or Conjugate Gradient method are the matrix vector products. Thus, both methods have spatial complexity O(m). • Suppose that after I-iterations, one requires that ||e(i)||A||e(0)||A (a) Steepest Descent: i  12  ln 1



 

(b) Conjugate Gradient: i  1  ln 2

2

 

• The time complexity of these two methods is: (a) Steepest Descent: O ( m ) (b) Conjugate Gradient: O ( m  ) • Stopping criterion: ||r(i)||||r(0)||

9. Preconditioning Techniques References: • J.A. Meijerink and H.A. van der Vorst, “An iterative solution method for linear systems of which the coefficient matrix is a symmetric M-matrix,” Mathematics of Computation, Vol. 31, No. 137, pp. 148-162 (1977). • J.A. Meijerink and H.A. van der Vorst, “Guidelines for the usage of incomplete decompositions in solving sets of linear equations as they occur in practical problems,” Journal of Computational Physics, Vol. 44, pp. 134-155 (1981). • D.S. Kershaw, “The incomplete Cholesky-Conjugate gradient method for the iterative solution of systems of linear equations,” Journal of Computational Physics, Vol. 26, pp. 43-65 (1978). • H.A. van der Vorst, “High performance preconditioning,” SIAM Journal of Scientific Statistical Computations, Vol. 10, No. 6, pp. 1174-1185 (1989).

9. Preconditioning Techniques (Cont’d) • Preconditioning is a technique for improving the condition number of a matrix. Suppose that M is a symmetric, positive-definite matrix that approximates A, but is easier to invert. Then, rather than solving the original system Ax=b one solves the modified system M-1Ax= M-1b • The effectiveness of the preconditioner depends upon the condition number of M-1A. • Intuitively, the preconditioning is an attempt to stretch the quadratic form to appear more spherical, so that the eigenvalues become closer to each other. • Derivation: M  EET , E 1AE T ~ x  E 1b, ~ x  ET x

9. Preconditioning Techniques (Cont’d) • Transformed Preconditioned CG • Untransformed PCG Algorithm ~ ~ (PCG) Algorithm r  E 1r , d  ET d ~ i  0, ~ r  E 1b  E 1AE T ~ x, d  ~ r  ~ rT ~ r,    new

0

new

While i  imax and  new   2 0 ~ ~  E 1AE T d q

   ~new ~ dT q ~ ~ x~ x  d ~ ~ r  E 1b - E 1AE T ~ x or ~ r ~ r - q  old   new  new  ~r T ~r    new  old ~ ~ d~ r  d i  i 1

i  0, r  b  Ax, d  M 1r

 new  rT M -1r ,  0   new While i  imax and  new   2 0 q  Ad    new dT q x  x  d r  b - Ax or r  r - q  old   new  new  rT M -1r    new  old d  M 1r  d i  i 1

9. Preconditioning Techniques (Cont’d) There are sevaral types of preconditioners that can be used: (a) Preconditioners based on splitting of the matrix A, i.e. A=M-N (b) Complete or incomplete factorization of A, e.g. A = LLT + E (c) Approximation of M=A-1 (d) Reordering of the equations and/or unknowns, e.g. Domain Decomposition

9. Preconditioning Techniques (Cont’d) (a) Preconditioning based on splittings of A • Diagonal Preconditioning: M=D=diag{a11, a22, … ann} 3 2 2 2 A , b   , c  0  x     2 6   8   2 

• Tridiagonal Preconditioning • Block Diagonal Preconditioning

9. Preconditioning Techniques (Cont’d) (b) Preconditioning based on factorization of A • Cholesky Factorization (for symmetric and positive definite matrix) as a direct method: A = LLT x = (L-T)(L-1b) where: Lii 

i 1

Aii   Lik k 1 i 1

A ji   L jk Lik L ji 

k 1

, j  i  1, i  2, , n

Lii • Modified Cholesky factorization: A = LDLT

9. Preconditioning Techniques (Cont’d) (b) Preconditioning based on factorization of A • Incomplete Cholesky Factorization: A = LLT + E or A = LDLT + E , where Lij=0 if (i,j)P Simplest choice is: P={(i,j)| aij=0; i,j,=1,2,…,n} => ICCG(0)       ~ ~  ai bi ci   ãi bi c~i   di   LT    D  A                   ~ ~ ~ ~ ~ a~  d 1  a  b 2 d  c~ 2 d , b  b , c~  c , i  1,2, , n i

i

i

i 1 i 1

i m i m 1

i

i T

i

i

• Modified ICCG(0): M  ( L  D )D ( D  L ) bi21 ci2 m  diag ( A )  diag ( M ), d i  ai  d i 1 d i  m

9. Preconditioning Techniques (Cont’d) (c) Preconditioning based on approximation of A • If (J) J= -D-1(L+U)) • The inverse of A can be expressed as: A 1  ( D(I  J )) 1  (I  J ) 1 D 1 



1

k

  ( 1) D (L  U ) D 1 k 0

k



• For k=1, one has that: M 1  D 1  D 1 (L  U ) D 1 • For k=m, one usually uses:

m k 1 1 k 1 M m    k ( 1) D (L  U ) D k 0





9. Preconditioning Techniques (Cont’d) (d) Domain Decomposition Preconditioning Domain I Domain II

A  1 Ax  b,  0  BT  1

0 A2 BT

2

B1   x1   d1      B2  x2  d2     Q   z   f  

M -1 0 0  M  0 0 1  1    1 T M  L 0 M 0  L , where L   0 M 2 0  2    B T BT S   1  0 0 S   1  2   M  0 B 1  1  M   0 M 2 B 2  , S*  S  BT M -1B1  BT M -1B 2 1 1 2 2  BT BT S*   1  2

10. Conjugate Gradient Type Algorithms for Nonsymmetric Matrices T • The Bi-Conjugate Gradient i  0, r  b  Ax, r r*  0 T * * *   ,   p r, p r r r ,  0   new new (BCG) Algorithm was propoWhile i  imax and  new   2 0 sed by Lanczos in 1954. q  Ap • It solves not only the original q*  AT p*  new system Ax=b but also the dual   linear system ATx*=b*. p*T q x  x  p • Each step of this algorithm r  r - q requires a matrix-by-vector r *  r * -  q* product with both A and AT.  old   new *T *   r r new • The search direction pj does  new   not contribute to the solution  old directly. p  r  p

p*  r *   p* i  i 1

10. Conjugate Gradient Type Algorithms for Nonsymmetric Matrices (Cont’d) • The Conjugate Gradient Squared (CGS) Algorithm was developed by Sonneveld in 1984. • Main idea of the algorithm: r j   j ( A )r0 p j   j ( A )r0 r*j   j ( AT )r0* p*j   j ( AT )r*0

i  0, r0  b  Ax 0 , r0* is arbitrary p  r0 , u  r0  new  r*0T r ,  0   new While i  imax and  new   2 0    *Tnew r0 Ap q  u  Ap x  x   (u  q ) r  r -  A (u  q )  old   new  new  r0*T r    new  old u  r  q p  u   (q   p ) i  i 1

10. Conjugate Gradient Type Algorithms for Nonsymmetric Matrices (Cont’d) • The Bi-Conjugate Gradient Stabilized (Bi-CGSTAB) Algorithm was developed by van der Vorst in 1992. • Rather than producing iterates whose residual vectors are of the form r j   2 ( A )r0 j

• it produces iterates with residual vectors of the form r j   j ( A ) j ( A )r0 p j   j ( A ) j ( A )r0

i  0, r0  b  Ax 0 , r0* is arbitrary p  r0  new  r0*T r ,  0   new While i  imax and  new   2 0    *Tnew r0 Ap s  r  Ap sT As  As T As x  x  p   s r  s - As  old   new  new  r0*T r     new   old  p  r   (p  Ap) i  i 1

Complexity of linear solvers Time to solve model problem (Poisson’s equation) on regular mesh

n1/2

n1/3

2D

3D

Sparse Cholesky:

O(n1.5 )

O(n2 )

CG, exact arithmetic:

O(n2 )

O(n2 )

CG, no precond:

O(n1.5 )

O(n1.33 )

CG, modified IC:

O(n1.25 )

O(n1.17 )

CG, support trees: Multigrid:

O(n1.20 ) -> O(n1+ ) O(n1.75 ) -> O(n1.31 ) O(n)

O(n)