The Quasi-Newton Least Squares Method

0 downloads 0 Views 339KB Size Report
Jacobian after each iteration, and satisfies the secant equation. While it has originally ... of the most widely used methods to do so is Newton's method: K′(ps) ds. = −K(ps),. (2) ...... The modified algorithm then reads as follows. Algorithm 3.2.
Working paper: The Quasi-Newton Least Squares Method: A New and Fast Secant Method Analyzed for Linear Systems Rob Haelterman∗

Joris Degroote†

Dirk Van Heule†

Jan Vierendeels‡

Abstract We present a new quasi-Newton method that can solve systems of equations of which no information is known explicitly and which requires no special structure of the system matrix, like positive definiteness or sparseness. The method builds an approximate Jacobian based on inputoutput combinations of a black box system, uses a rank-one update of this Jacobian after each iteration, and satisfies the secant equation. While it has originally been developed for nonlinear equations we analyze its properties and performance when applied to linear systems. Analytically, the method is shown to be convergent in n + 1 iterations (n being the number of unknowns), irrespective of the nature of the system matrix. The performance of this method is greatly superior to other quasi-Newton methods and comparable with GMRes when tested on a number of standardized test-cases.

Disclaimer For the published version see here: http://epubs.siam.org/doi/abs/10.1137/070710469

1

Introduction

In this paper we start from a system of equations given by K(p)

=

0,

(1)

where K : DK ⊂ IRn×1 → IRn×1 has continuous first partial derivatives in DK , a single solution p∗ ∈ DK , and a nonsingular Jacobian K ′ (p∗ ). ∗ Department of Mathematics, Royal Military Academy, Renaissancelaan 30, B-1000 Brussels, Belgium ([email protected], [email protected]). † Department of Flow, Heat, and Combustion Mechanics, Ghent University, St.-Pietersnieuwstraat 41, B-9000 Gent, Belgium ([email protected], [email protected]).

1

We will solve (1) in an iterative way starting from an initial value po . One of the most widely used methods to do so is Newton’s method: K ′ (ps ) ds

=

−K(ps ),

(2)

ps+1

=

p s + θs d s

(3)

(s = 0, 1, 2, . . .), where θs is a scalar parameter. Newton’s method has been studied extensively and exhibits superlinear convergence whenever po is close enough to the exact solution p∗ of (1) as specified by the Newton–Kantorivich theorem [22]; if K ′ (p) is also Lipschitz continuous for all p close enough to p∗ , then the convergence is quadratic. If the matrix of the system is not known, the Jacobian is unavailable or too expensive to compute, then a number of matrix-free methods are at our disposal that use only information derived from the consecutive ˆ s′ ∈ IRn×n of K ′ (ps ) based on iterates and that build an approximation K those values. This approach falls under the general framework of quasiNewton methods. Quasi-Newton methods have been used intensively for solving linear and nonlinear systems and for minimization problems [20]. Their main attraction is that they avoid the cumbersome computation of derivatives for the Jacobians. Recently, interest in quasi-Newton methods has waned, as automatic differentiation has become available [10, 14], except for a recent algorithm by Eirola and Nevanlinna [11, 12], the research performed by Deuflhard (e.g., [8, 9]) and Spedicato, Xia, and Zhang [25], and work on secant-related Krylov–Newton solvers used by Dawson et al. [3]. We are mainly interested in quasi-Newton methods because • we do not have access to the Jacobian as we are working with black box systems, which also makes automatic differentiation impossible; • the cost of a function evaluation is sufficiently high so that numerical differentiation becomes prohibitive. For this reason we will judge performance of the method by the number of function evaluations it needs to converge. In this paper we propose a new quasi-Newton method that has its origins in [29, 30] where an approximate Jacobian was needed for the strong coupling of fluid-structure interaction problems using two commercially available codes (for the structure and the fluid) that are considered black boxes. The approximate Jacobian was constructed based on input-output combinations of each black box in a least squares sense. Introducing a new black box system H such that H(p) = K(p) + p, we are able to apply this method to solve (1) in a matrix-free manner based on the values (pi , H(pi )) (i = 0, 1, 2, . . . , s) that arise during the iteration process when solving (1). It is shown in this paper that the method can be written as a quasi-Newton method with a rank-one update of the approximate Jacobian after each iterate. While the method has its origin as a nonlinear solver, we consider only linear systems in this paper. Studying quasi-Newton methods for linear problems is important not only because many problems are linear or nearly linear but also because the properties of a method in the linear

2

case often define the local convergence behavior of the method in the nonlinear case. This can be understood by observing that, close to a solution of (1) where the Jacobian is nonsingular, the linear approximation of K(p) tends to be dominant. Hence, the generated sequence ps tends to behave like in the linear case. This is the main reason why the local convergence of Newton’s method is quadratic [22]. When studying the analytical properties of the method for linear systems, we will pose H(p) = Ap − b, where A ∈ IRn×n and p, b ∈ IRn×1 and where A−I is assumed to be nonsingular but without further requirements like positive definiteness or sparseness. While most matrix-free solvers assume that Ap can be formed, we assume we can form only H(p). Finding b by computing H(0) is a possibility, but, as we assumed that a “call” of H is very expensive, it is to be avoided. We show that line-searches cannot improve the method’s performance and that it converges in at most n + 1 iterations. This answers one of the questions that Martinez left open in [20], namely, the question of whether Broyden-like methods exist that converge in fewer than 2n steps for linear systems. The performance of this new method is compared with that of other known methods that use rank-one updates and that can be used with black boxes, for instance Broyden’s first and second method [1, 2], the column updating method [17, 19, 20], and the inverse column updating method [16, 18]. We also compare the method with GMRes. Even though our method has not been originally developed as a linear solver, the results show that the method has comparable, but slightly lower, performance to GMRes when the Euclidean norm of the residual is used as the criterion and far better overall performance than the other quasi-Newton methods. The paper is organized as follows: in section 2 we start with some general definitions, conventions, and theorems; in section 3 we analyze the construction of the approximate Jacobian that was proposed in [29] after adaptation to our purposes and present some theorems regarding its properties and convergence; in sections 4 and 5 a review is given of other known quasi-Newton methods and the GMRes method, respectively; in section 6 the relative performance of the different methods is compared on the heat equation, a test-case proposed by Deuflhard [8], and on a number of standardized test-problems from the Matrix Market repository [21].

2

Introductory definitions and theorems

We will use e = p − p∗ , es = ps − p∗ , and so on for the errors. All matrix norms that are used are natural matrix norms, unless otherwise stated. < ·, · > denotes the standard scalar product between vectors. Definition 2.1 A natural matrix norm is a matrix norm induced by a vector norm in the following manner (with M ∈ IRn×m ): kM k = sup x6=0

kM xk or equivalently kM k = sup kM xk. kxk kxk=1

3

(4)

Definition 2.2 (see [7]) Let {xs }s∈IN be a sequence with xs and x∗ ∈ IRn×1 . We say that the sequence {xs } converges superlinearly1 towards x∗ with order α > 1 if ∃ǫ > 0 : kxs+1 − x∗ k ≤ ǫkxs − x∗ kα

(5)

for any arbitrary norm k · k in IRn×1 . Definition 2.3 (see [7]) Let {xs }s∈IN be a sequence with xs and x∗ ∈ IRn×1 . We say that the sequence {xs } converges superlinearly towards x∗ if kxs+1 − x∗ k =0 s→∞ kxs − x∗ k lim

(6)

for any arbitrary norm k · k in IRn×1 . Definition 2.4 (see [7]) Let f : Ω ⊂ IRn×1 → IRm×1 . f is Lipschitz continuous on Ω if ∃ǫ > 0 (the Lipschitz constant) such that ∀p1 , p2 ∈ Ω : kf (p1 ) − f (p2 )k ≤ ǫkp1 − p2 k.

(7)

(If ǫ < 1, then f is called a contraction mapping with respect to the chosen norm.) Definition 2.5 Any n + 1 vectors xo , x1 , . . . , xn ∈ IRn×1 are in general position if the vectors xn − xj (j = 0, . . . , n − 1) are linearly independent. Lemma 2.1 ∀u, v ∈ IRn×1 : det(I + uv T ) = 1 + hu, vi. Proof Let P = I + uv T . For u = 0 or v = 0 the proof is trivial. If u, v 6= 0 and hu, vi 6= 0, then any vector orthogonal to v is a right eigenvector of P (corresponding to an eigenvalue 1) and any multiple of u is also a right eigenvector of P (corresponding to an eigenvalue 1 + hu, vi = 6 0). As there are n − 1 vectors orthogonal to v and as the algebraic multiplicity of an eigenvalue is larger than or equal to its geometric multiplicity, we see that the algebraic multiplicity of the eigenvalue 1 is at least n − 1. As there is another eigenvalue different from 1, the algebraic multiplicity of the eigenvalue 1 must be equal to n − 1. As the determinant of a matrix equals the product of its eigenvalues, we have that det P = 1 + hu, vi. If u, v 6= 0 and hu, vi = 0, then 1 + hu, vi = 1. But then  2 (P − I)2 = uv T = uv T uv T = 0. So the space of generalized eigenvectors corresponding to the eigenvalue 1 has dimension n. Hence, the algebraic multiplicity of the eigenvalue 1 is n and det P = 1.

1 These definitions are based on “q-superlinearity” as detailed in [15]; as we use only this type of convergence criterion, we will simply use the term “superlinear.”

4

Theorem 2.1 (fundamental theorem of linear algebra2 ) Let M ∈ IRn×m ; then N (M ) = (R(M T ))⊥ , where N (M ) is the kernel (or null space) of M , R denotes the range, and (·)⊥ gives the orthogonal complement of a vector-space. For a proof of this theorem we refer to [26]. Theorem 2.2 (the Sherman–Morrison theorem) Let M ∈ IRn×n be nonsingular, and let u, v ∈ IRn×1 be vectors such that v T M −1 u 6= −1; then M + uv T is nonsingular and 

M + uv T

−1

= M −1 −

M −1 uv T M −1 . 1 + v T M −1 u

(8)

For the proof we refer to [24]. Theorem 2.3 Assume that • K : IRn×1 → IRn×1 is differentiable in an open set DK ⊂ IRn×1 ; • the equation K(p) = 0 has a solution p∗ ∈ DK ; • K ′ : DK → IRn×n is Lipschitz continuous with Lipschitz constant κ; • K ′ (p∗ ) is nonsingular. Assume that we use a Newton method (equation (2) and (3)) where we ˆ s = K(ps ) + Es and K ′ (ps ) by K ˆ s′ = K ′ (ps ) + Es ; then replace K(ps ) by K ∗ there exist ǫ, δ, and τ such that if ps ∈ B(p , δ) and kEs k ≤ τ , then the following properties hold: ˆ s′ is nonsingular. 1. K  2. kes+1 k ≤ ǫ kes k2 + kEs kkes k + kEs k . For the proof of this theorem we refer to [15].

Lemma 2.2 Let V ∈ IRn×s be a matrix with column-rank s; then  −1 V V TV V T = Ls LTs ,

(9)

¯ 1 |L ¯ 2 | . . . |L ¯ s ], where L ¯ k is the kth left (normalized) singular with Ls = [L vector of V . Proof We can write the singular value decomposition3 of V as V = LSRT , where the singular values are given by σi (i = 1, . . . , s). According to the conventions of the singular value decomposition we have Sii = σi and Sij = 0 when i 6= j. Then we can write  −1   −1   V V TV V T = LSRT RS T LT LSRT RS T LT . (10)

As L is a unitary matrix this simplifies to  −1   −1   V V TV V T = LSRT RS T SRT RS T LT .

(11)

3 Different conventions exist for the singular value decomposition. We will use the one where L ∈ IRn×n , S ∈ IRn×s , and R ∈ IRs×s and where the singular values are ordered in a nonincreasing way.

5

Because of the special structure of S we have S T S = Λ = diag(σ12 , σ22 , . . . , σs2 ) and we obtain  −1 V V TV V T = LSΛ−1 S T LT (12) = =

LIn,s (LIn,s )T 2

T

L(In,s ) L ,

(13)

(14)

where In,s = diag(1, 1, . . . , 1, 0, 0, . . . , 0). We can further reduce this ex| {z } | {z } s times

n−s times

¯ 1 |L ¯ 2 | . . . |L ¯ s ], where L ¯ k is the kth left pression by introducing Ls = [L singular vector of V :  −1 V V TV V T = Ls LTs , (15) which completes the proof.

Lemma 2.3 Let V ∈ IRn×s be a matrix with column-rank s and M ∈ IRm×n ; then, using the notation of Lemma 2.2,      N M Ls LTs ⊃ (R(V ))⊥ and N M Ls LTs − I ⊃ R(V ). (16)

If M has rank n, then      N M Ls LTs = (R(V ))⊥ and N M Ls LTs − I = R(V ).

(17)

Proof These properties are direct consequences of Theorem 2.1 and Lemma 2.2, as ∀x ∈ R(V ) : Ls LTs x = x and ∀y ∈ (R(V ))⊥ : Ls LTs y = 0.

3 Description of the least squares quasiNewton method 3.1

Construction of the approximate Jacobian

We will now go into more detail on how we construct the approximate ˆ s′ when the exact Jacobian of neither H nor K is availJacobians K able. If we have s + 1 input-output pairs (pi , H(pi )) (i = 0, . . . , s)4 , we can construct s input modes vis = ps − pi (i = 0, . . . , s − 1) and an equal number of output modes wis = H(ps ) − H(pi ) (i = 0, . . . , s − 1). We assume s the pi (i = 0, . . . , s) are in general position. We define Vs = [vos . . . vs−1 ] s s and Ws = [wo . . . ws−1 ]. When we have a new input ps+1 , of which the output is not known, we write ∆ps = ps+1 − ps as a linear combination of the input modes ∆ps

=

s−1 X

s αm vm + ε,

(18)

m=0 4 These input-output pairs can be created in an ad hoc manner or taken from computations that arise in the iteration process.

6

where αm (m = 0, . . . , s − 1) denotes the coordinates of ∆ps with respect to the input modes and ε the part of ∆ps that lies outside R(Vs ). If we want ε to be minimal in the l2 norm5 , we impose ε⊥R(Vs ) with respect to the standard scalar product. If we use α = [αo . . . αs−1 ]T , then ∆ps = Vs α + ε. This leads to VsT ε

=

α

=

VsT (∆ps − Vs α) = 0,  −1 VsT Vs VsT ∆ps .

(19) (20)

We now make a prediction on the output by writing the same linear combination with respect to the output modes:  −1 H(ps+1 ) − H(ps ) ≈ Ws α = Ws VsT Vs VsT (ps+1 − ps ), (21)  −1 H(ps+1 ) ≈ H(ps ) + Ws VsT Vs VsT (ps+1 − ps ). (22)

We see that the expression Ws (VsT Vs )−1 VsT thus fulfills the role of the (approximate) derivative of H with respect to p and, as K(p) = H(p) − I, the choice of the following approximate Jacobian of K is justified:  −1 ˆ s′ = Ws VsT Vs K VsT − I. (23) We will use (23) in the iterative, quasi-Newton scheme ˆ s′ )−1 K(ps ), ps+1 = ps − (K

(24)

starting from an initial value po . Note that the inverse in (24) is rarely computed in practice but is most often replaced by the solution of a linear system. The resulting error, respectively, residual, equation is es+1

=

rs+1

=

ˆ s′ )−1 (A − I)es , e s − (K ˆ s′ )−1 rs . rs − (A − I)(K

(25) (26)

For more details on the background of this method see [29, 30]. The use of a relaxation factor θs for the step-size (i.e., using ps+1 = ps − ˆ s′ )−1 K(ps ), instead of (24)) will be discussed in Theorem 3.2. θs ( K ˆ s′ cannot be conNotethat (24) can be used only for s ≥ 1, because K ˆ o′ as structed earlier. Oftenunderrelaxation). As such, we could define K ′ ˆ Ko = −I to make (24) conform for s = 0. Alternatively, this could be seen as setting the initial approximate Jacobian of H equal to zero, which is as good as any guess, as we don’t know anything about the Jacobian a priori. The complete algorithm, which we will call “quasi-Newton-least squares” (QN-LS) can be described as follows. Algorithm . 3.1 (QN-LS) 1. Startup. Take a starting value po ; compute p1 = H(po ), or alternatively, use underrelaxation to avoid excessive divergence: p1 = po + ω(H(po ) − po ). 5 In

other words, it is a least squares approximation.

7

Set s = 1. 2. Loop until sufficiently converged: a. Compute H(ps ). ˆ s′ based on (po , H(po )), . . . , (ps , H(ps )) b. Construct approximate Jacobian K using (23). ˆ s′ )−1 (H(ps ) − ps ). c. Quasi-Newton step: ps+1 = ps − (K d. Set s = s + 1.

3.2

General theorems regarding QN-LS

The construction in section 3.1 corresponds to the formulation used in Lemmas 2.2 and 2.3, as long as the columns of Vs remain linearly independent. As a result we can establish some properties of the scheme. ˆ s′ Theorem 3.1 Consider the method described in section 3.1. Assume K is nonsingular. Then the following relation for the error holds: ˆ s′ )−1 (A − I))(es − ej ) = 0. ∀j ∈ {0, 1, . . . , s} : (I − (K

(27)

Proof As ∀j ∈ {0, 1, . . . , s} : R(Vj ) ⊂ R(Vs ), Lemma 2.3 allows us to write ∀x ∈ R(Vj ), j = 0, 1, . . . , s, (ALs LTs − I)x x ˆ s′ )−1 (A − I))x (I − (K

=

(A − I)x,

=

(ALs LTs

=

0.

− I)

−1

(28) ′ −1 ˆ (A − I)x = (Ks ) (A − I)x, (29) (30)

ˆ s′ )−1 (A−I)) We can conclude that R(Vs ) is part of the null space of (I −(K (it equals the null-space if A is nonsingular). As es − ej = ps − pj we have (es − ej ) ∈ R(Vs ) from the definition of Vs and the theorem follows. Consequences: 1. As we have that ∀j ∈ {0, 1, . . . , s} : ps − pj = es − ej and thus ˆ s′ )−1 (A − I))(ps − pj ) = 0, it follows that (I − (K ˆ s′ )−1 (H(ps ) − ps ) = pj − (K ˆ s′ )−1 (H(pj ) − pj ). (31) ps+1 = ps − (K This means that the previous iterates contribute only to the solution ˆ s′ . process by creating a better approximate Jacobian K 2. From (25) we obtain es+1

= =

ˆ s′ )−1 (A − I))es (I − (K ˆ s′ )−1 (A − I))eo (I − (K

(32)

=

ˆ s′ )−1 (A − I))(eo − (Vs (VsT Vs )−1 VsT )eo ) (I − (K ˆ s′ )−1 (A − I))eo,⊥ (I − (K s ˆ s′ )−1 (ALs LTs − A)eo,⊥ (K

=

ˆ s′ )−1 Aeo,⊥ , −(K s

= =

s

8

(33)

where eo,⊥ s is the part of eo that is orthogonal to R(Vs ). We see that the error at iteration s + 1 is determined by the content of the initial error eo that is not contained in R(Vs ). It is thus important that Vs captures as many vector-components of eo as quickly as possible. The vector components that it captures are preferably those associated with large eigenvalues of A. Corollary 3.1 Let H(p) = Ap − b. Assume that the method described in ˆ s′ is nonsingular. section 3.1 starts from an initial guess po = 0 and that K Then the quasi-Newton iteration can also be written as ps+1 = (ALs LTs − I)−1 b

(s ≥ 1),

(34)

and the error es+1 is given by es+1 = ((ALs LTs − I)−1 − (A − I)−1 )b Proof If we start from po = 0, then eo = −(A−I) From (32) we get es+1

=

es+1

=

(s ≥ 1). −1

(35)



b as p = (A−I)−1b.

ˆ s′ )−1 (A − I))(A − I)−1 b, −(I − (K ˆ s′ )−1 − (A − I)−1 )b. ((K

(36) (37)

As ps+1 = p∗ + es+1 we get ps+1

=

ˆ s′ )−1 − (A − I)−1 )b, (A − I)−1 b + ((K

(38)

from which (34) follows. Theorem 3.2 Consider the method described in section 3.1 but using a relaxation factor in (24): ˆ s′ )−1 K(ps ); ps+1 = ps − θs (K

(39)

then the choice of θs ∈ IRo is irrelevant for the value of ps+2 .6 Proof We have, by posing es = es,k + es,⊥ , where es,k ∈ R(Vs ) and es,⊥ ∈ (R(Vs ))⊥ , and using Theorem 3.1, es+1

= = = = =

ˆ s′ )−1 (A − I)es e s − θs ( K ˆ s′ )−1 (A − I)es,⊥ − θs es,k es,k + es,⊥ − θs (K

(40)

ˆ s′ )−1 (A − I)es,⊥ (1 − θs )es,k + es,⊥ − θs (K ˆ s′ )−1 Aes,⊥ (1 − θs )es,k + (1 − θs )es,⊥ − θs (K

(42)

(1 − θs )(es,k + es,⊥ + | {z

(41) (43)

ˆ s′ )−1 Aes,⊥ ) − (K ˆ s′ )−1 Aes,⊥ (K

(∗)

}

|

{z

(∗∗)

. (44)

}

The new column vector added to update Vs to Vs+1 will depend on δps = ps+1 − ps = δes = es+1 − es : δes 6 The

=

ˆ s′ )−1 Aes,⊥ ). −θs (es,k + es,⊥ + (K

value of ps+1 will be affected, however.

9

(45)

We can thus conclude that the direction of δes is independent of θs . We see that es+1 has a part that is parallel to δes (part (∗) in (44)) and a remaining part that is independent of θs (part (∗∗)). (∗) will be eliminated completely at the next iteration, according to Theorem 3.1, as it lies in R(Vs+1 ). We can thus conclude that θs will have no effect at the next iteration. Corollary 3.2 Consider the method described in section 3.1. If we use for the first iteration p1 = (1 − θ)po + θH(po ),

(46)

then the value of θ ∈ IR \ {0} has no influence on the following iterations. Proof The proof is analogous to that of Theorem 3.2. This theorem and corollary show that line-searches, which are often part of a (quasi-) Newton method, do not improve the long-term convergence of our algorithm. We note that line-searches require supplemental function evaluations and would thus be very expensive in the framework we imposed. Relaxation can still be applied for practical reasons, however, such as to avoid numerical breakdown due to excessive initial divergence.

3.3

Convergence, error bounds, and invertibility

If we solve (24) exactly, the error will respect the following inequality if the assumptions of Theorem 2.3 are satisfied:  kes+1 k ≤ ǫ kes k2 + kEs kkes k , (47) ˆ s′ − K ′ (ps ). where Es = K Our main concern is thus that kEs k remains bounded and decreases.

Theorem 3.3 Let {Xs }s∈[1,n] be an arbitrary sequence of vectors that are linearly independent ∈ IRn×1 , Vs = [X1 X2 . . . Xs ], and A ∈ IRn×n , and ˆ s′ = AVs (VsT Vs )−1 VsT − I; then let K



ˆ′

ˆ′

(48)

Ks+1 − A + I ≤ K s − A + I . ˆ s′ − A + I. Proof We define Es = K We first note that the rank of Vs is s and that in the limit s → n we will have ˆ n′ = AVn (Vn )−1 (VnT )−1 VnT − I = A − I, K

(49)

and hence kEn k = 0. For s < n Lemma 2.2 gives us ˆ s′ = ALs LTs − I. K

10

(50)

We will decompose any vector x ∈ IRn×1 as x = x⊥s + xks , x⊥s ∈ (R(Vs ))⊥ , xks ∈ R(Vs ). From the definition of the natural matrix norm and Lemma 2.3 we know



  

ˆ′



T T sup x

Ks − A + I = A Ls Ls − I =

A Ls Ls − I (51) n×1 x∈IR ;kxk=1 = sup kAx⊥s k. (52) n×1 x∈IR ;kxk=1 This maximum will be attained for a certain xmax ∈ (R(Vs ))⊥ . Vs+1 = [Vs |Xs+1 ], we have (R(Vs+1 ))⊥ ⊂ (R(Vs ))⊥ and hence kAx⊥s k ≥

sup x∈IR

n×1

;kxk=1

sup x∈IR

n×1

kAx⊥(s+1) k.

As (53)

;kxk=1

′ ˆ s+1 ˆ s′ − A + Ik which completes our proof. It follows that kK − A + Ik ≤ kK

ˆ n′ will be the exact An important consequence of this theorem is that K Jacobian and that therefore the exact solution will be found in at most n + 1 steps if no singularities occur (see section 3.5 on how to deal with them) and if we ignore numerical round-off errors. From Theorem 3.3 we also know that there will be a point in the convergence where kEs k will be small enough to respect the assumptions of ˆ s′ . Theorem 2.3 and thus to ensure nonsingularity of K ˆ s′ Theorem 3.3 also shows that for kEs k to decrease, and hence for K to converge to K ′ (ps ), no specific requirements exist for the columns of Vs (see section 3.1), except that they should be linearly independent (see section 3.5). The chosen method, based on ps and H(ps ), is the most economical, as these results are readily available during the iterations. Theorem 3.4 Let K : DK ⊂ IRn×1 → IRn×1 , where K is continuously differentiable over an open convex set DK ; ∃p∗ ∈ DK : K(p∗ ) = 0 ˆ s′ be a series of nonsingular matrices where K ′ (p∗ ) nonsingular; let K n×n ∈ IR . Furthermore, assume that ∀p ∈ DK kK ′ (p) − K ′ (p∗ )k ≤ ǫkp − p∗ kα

(54)

for some norm k · k and some ǫ, α > 0 and that the quasi-Newton method for the solution K(p) = 0 is given by ˆ s′ )−1 K(ps ). ps+1 = ps − (K

(55) ∗

Then given r ∈ [0, 1], ∃κ = κ(r), and δ = δ(r) such that if kpo − p k ≤ κ ˆ s′ − K ′ (p∗ )k ≤ δ for all s = 0, 1, 2, . . . , the sequence generated by and kK (55) is well defined, converges to p∗ , and satisfies kps+1 − p∗ k ≤ rkps − p∗ kα

(56)

for all s = 0, 1, 2, . . . . For the proof we refer to [5]. We note that in this study K ′ (p) = K ′ (p∗ ) due to the linear nature of the problem and that hence (54) will always be satisfied.

11

Corollary 3.3 Let K : DK ⊂ IRn×1 → IRn×1 , where K is continuously differentiable over an open convex set DK ; ∃p∗ ∈ DK : K(p∗ ) = 0 ˆ s′ be a series of nonsingular matrices where K ′ (p∗ ) nonsingular; let K n×n ∈ IR ; suppose that for some po ∈ DK the sequence {ps } generated by ˆ s′ )−1 K(ps ) ps+1 = ps − (K

(57)



remains in DK and converges to p . Then the sequence converges superlinearly to p∗ if and only if (the Dennis–Mor´e condition)



(K ˆ s − K ′ (p∗ ))(ps+1 − ps ) lim = 0. (58) s→∞ kps+1 − ps k For the proof we refer to [5]. This means that if the assumptions of Corollary 3.3 are satisfied and if ˆ s′ = K ′ (p∗ ), then superlinear convergence is assured. Note that lims→∞ K this is a sufficient but not a necessary condition for superlinear converˆ s′ needs only to converge to K ′ (p∗ ) along the direction of gence, as K δps . The following theorem is thus a direct consequence of Corollary 3.3 and Theorem 3.3. Theorem 3.5 Let K : IRn×1 → IRn×1 : p 7→ K(p) = (A − I)p − b, ˆ s′ be a series of A ∈ IRn×n , b ∈ IRn×1 , and A − I nonsingular; let K n×n nonsingular matrices ∈ IR generated by (23); suppose that for some po ∈ IRn×1 the sequence {ps } is generated by (24); then the sequence converges superlinearly to p∗ . One of the properties of superlinearly convergent schemes is that [4] lim

s→∞

kps+1 − ps k = 1. kps − p∗ k

(59)

This means that we can use kps+1 − ps k ≤ ǫ1 kps k as a stopping criterion; it can be combined with a second criterion given by kK(ps )k ≤ ǫ2 , where ǫ1 and ǫ2 are user-specified tolerances.

3.4

The secant property

The class of quasi-Newton methods encompasses every scheme where the ˆ s′ . Someexact Jacobian K ′ (ps ) in (2) is replaced with an approximation K times the name quasi-Newton has been used to describe the subclass of methods that respect the secant equation [2]7 ˆ s′ · (ps − ps−1 ) = K(ps ) − K(ps−1 ) K

(60)

for all s = 1, 2, . . . . Following the nomenclature of [7] we call the methods respecting (60) secant methods and prove that the proposed algorithm is a secant method. 7 The

secant equation is sometimes called the fundamental equation of quasi-Newton meth-

ods.

12

Theorem 3.6 Consider the method described in section 3.1, and assume ˆ s′ is nonsingular; then the quasi-Newton iteration is a secant method; K i.e., it satisfies the secant equation (60). ˆ s′ = ALs LTs − I; it follows Proof We recall from Lemma 2.2 that K that   ˆ s′ · (ps − ps−1 ) = K ALs LTs − I (ps − ps−1 ) (61)   = A Ls LTs − I (ps − ps−1 ) + (A − I)(ps − ps−1 (62) ). Using Lemma 2.3 we obtain

ˆ s′ · (ps − ps−1 ) = (A − I)(ps − ps−1 ). K

(63)

Under the assumption of linearity we can write K(p) = (A − I)p − b and (60) follows. Apart from (60) it is also easy to show, based on Lemmas 2.2 and 2.3, that ˆ s′ (x − y) = K(x) − K(y), ∀x, y ∈ R(Vs ) : K

(64)

ˆ s′ is correct for every direction in R(Vs ); for every direction meaning that K ˆ s′ v = −v. The latter can be interpreted as setting the v⊥R(Vs ) we have K Jacobian of H equal to zero in every direction of which no information is available. We would like to point out that other methods that respect (64) can be constructed that differ from QN-LS, as it provides only s × n equations for the n2 unknowns of the approximate Jacobian and is thus underdetermined. We would also like to point out that respecting (64) for all previous iterates is clearly a good idea for linear systems as judged from Theorem 3.3. However, this matter should be investigated in more detail for nonlinear systems, as secant properties obtained with points that are far from the actual solution might not be representative of the actual tangent hyperplane and thus hamper convergence. Corollary 3.4 Consider the method described in section 3.1. Then the following expression holds: ′ ˆ s+1 ˆ s′ )−1 )K(ps ). K(ps+1 ) = (I − K (K (65) ˆ s′ )−1 K(ps ) and (from Theorem 3.6) K(ps+1 ) = Proof We have ps+1 = ps −(K ′ ˆ s+1 K(ps ) + K (ps+1 − ps ). Equation (65) follows.

3.5

ˆ′ Obtaining K s+1 by a rank-one update

We will now show that the Jacobian of our method can be obtained by a rank-one update. ˆ s′ is constructed according to (23); then Theorem 3.7 Suppose that K ′ ′ ˆ ˆ Ks+1 is linked to Ks by the following expression (for s < n): ′ ˆ s+1 K

=

ˆ s′ + AL ¯ s+1 L ¯ Ts+1 , K

¯ s+1 is the (s + 1)th left singular vector of Vs+1 . where L

13

(66)

Proof ∀x ∈ IRn×1 we have according to Lemma 2.2 that ′ ˆ s+1 ˆ s′ )x (K −K ′ ˆ s+1 ˆ s′ )x (K −K

=

(ALs+1 LTs+1 − I)x − (ALs LTs − I)x,

(67)

=

A(Ls+1 LTs+1 − Ls LTs )x.

(68)

As R(Vs ) ⊂ R(Vs+1 ) we have ′ ˆ s+1 ˆ s′ )x (K −K

¯ s+1 L ¯ Ts+1 x. AL

=

(69)

As this is true ∀x ∈ IRn×1 we obtain ′ ˆ s+1 ˆ s′ K −K

=

¯ s+1 L ¯ Ts+1 , AL

(70)

which completes our proof. ˆ s′ is constructed according to (23). If Corollary 3.5 Suppose that K ¯ Ts+1 (K ˆ s′ )−1 AL ¯ s+1 L

6=

−1,

(71)

ˆ s′ )−1 AL ¯ s+1 L ¯ Ts+1 (K ˆ s′ )−1 (K , ¯ T (K ˆ s′ )−1 AL ¯ s+1 1+L

(72)

′ ˆ s+1 then K is nonsingular and ′ ˆ s+1 (K )−1

=

ˆ s′ )−1 − (K

s+1

¯ s+1 is the (s + 1)th left singular vector of Vs+1 . where L Proof To prove this theorem we apply Theorem 2.2 (the Sherman–Morrison ˆ s′ , u = AL ¯ s+1 , and v = L ¯ s+1 . Then actheorem) where we put M = K ′ ˆ s+1 cording to Theorem 3.7 we have K = M + uv T and (72) follows. Corollary 3.6 Consider the method described in section 3.1. Then the following expressions hold: ¯ s+1 L ¯ Ts+1 δps , K(ps+1 ) = AL

(73)

with δps = ps+1 − ps for s = 0, 1, 2, . . . . Proof From Theorem 3.7 we have ′ ˆ s+1 ˆ s′ )−1 K (K

¯ s+1 L ¯ Ts+1 (K ˆ s′ )−1 . I + AL

(74)

¯ s+1 L ¯ Ts+1 (K ˆ s′ )−1 K(ps ), −AL T ¯ s+1 L ¯ s+1 δps , AL

(75)

=

Inserting this in (65) gives K(ps+1 )

=

K(ps+1 )

=

(76)

which completes our proof. The results from Theorem 3.7 and Corollary 3.5 mean we can use (66) ˆ s′ and (72) to update (K ˆ s′ )−1 if L ¯ Ts+1 (K ˆ s′ )−1 AL ¯ s+1 6= −1. to update K T ¯ ¯ Note that ALs+1 Ls+1 is a matrix of1 and that, as a consequence, (66) ′ ˆ s′ to obtain K ˆ s+1 can be considered as a rank-one update applied to K . ′ ′ ˆ s+1 ˆ s+1 Both (66) and (72) allow us to compute K and (K )−1 using only matrix-vector and scalar products.

14

′ ′ ˆ s+1 ˆ s+1 To implement the rank-one update for K or (K )−1 in a practical ¯ s+1 and AL ¯ s+1 (without knowing algorithm we need ways to compute L A). ¯ s+1 can be computed using The new singular vector L T ¯ s+1 = δps − Ls Ls δps , L kδps − Ls LTs δps k

(77)

¯ s+1 can be computed using while AL ¯ s+1 AL

=

ˆ s′ + I)δps δps − Ls LTs δps H(ps+1 ) − H(ps ) − (K = (78) kδps − Ls LTs δps k kδps − Ls LTs δps k ˆ s′ δps K(ps+1 ) − K(ps ) − K . T kδps − Ls Ls δps k

A

=

′ ¯ s+1 = ¯ ˆ s+1 ˆ s′ .) (If L 0, we keep K =K We can write the resulting update as T T ˆ′ ′ ˆ s+1 ˆ s′ + (δKs − Ks δps )(δps ) (I − Ls Ls ) , K =K T T h(I − Ls Ls )δps , (I − Ls Ls )δps i

(79)

where δKs = K(ps+1 ) − K(ps ). As (I − Ls LTs )T (I − Ls LTs ) = I − Ls LTs this simplifies to T T ˆ′ ′ ˆ s+1 ˆ s′ + (δKs − Ks δps )((I − Ls Ls )δps ) . K =K hδps , (I − Ls LTs )δps i

(80)

As (I − Ls LTs )δps ⊥δps−1 , (80) shows us that our algorithm satisfies the secant R1 correction; i.e., it satisfies ′ ˆ s+1 K

=

hv, δps−1 i

=

T ˆ′ ˆ s′ + (δKs − Ks δps )v , K hδps , vi 0,

(81) (82)

for which convergence properties have been well-studied [20]. Equation (80) also tells us that ∀z⊥(I − Ls LTs )δps

:

′ ˆ s+1 ˆ s′ z; K z=K

(83)

i.e., the approximate Jacobian changes only in the direction of the newly added orthogonal basis-vector (I − Ls LTs )δps of R(Vs+1 ). This is a welcome property as we know from Theorem 3.6 that the approximate Jacoˆ s′ was correct in every direction that lies in R(Vs ) (which are all bian K orthogonal to (I − Ls LTs )δps ). Similar to (80) the update for the inverse Jacobian can be rewritten as ′ ˆ s+1 (K )−1

=

ˆ s′ )−1 + (K

ˆ s′ )−1 δKs − δps )δpTs (I − Ls LTs )(K ˆ s′ )−1 ((K (84) .

ˆ s′ )−1 δKs , (I − Ls LTs )δps (K

15

3.6

Avoiding singularity

′ ˆ s+1 We have seen in section 3.5 that K becomes singular if

¯ Ts+1 (K ˆ s′ )−1 AL ¯ s+1 = −1. L

(85)

This is not due to the rank-one update of (66) or (72) but would also happen if we used the construction of (23). We might therefore propose ′ ˆ s+1 ˆ s′ if L ¯ Ts+1 (K ˆ s′ )−1 AL ¯ s+1 = −1 and use the updates above to keep K =K otherwise. In that case we have ˆ s′ ps+2 K ˆ s′ ps+1 K

=

ˆ s′ ps+1 − K(ps+1 ), K ˆ s′ ps − K(ps ) K

⇒ ps+2

=

ˆ s′ )−1 (K(ps ) + K(ps+1 )). p s − (K

=

(86) (87) (88) ˆ s′ )−1 (K

If convergence has not been achieved, then K(ps+1 ) 6= 0, and as is ˆ s′ would not exist), we have ps+2 6= ps+1 , which nonsingular (otherwise K allows us to continue the iteration. This approach poses a problem, however, as the secant equation (60) is no longer satisfied; indeed ′ ˆ s−1 K · (ps − ps−1 )

=

(ALs−1 LTs−1 − I)(ps − ps−1 )

6=

K(ps ) − K(ps−1 ).

(89)

To satisfy the secant property as closely as possible, we could use a convex average ′ ˆ s+1 K

=

¯ Ts+1 ), ˆ s′ + θs (K ˆ s′ + AL ¯ s+1 L (1 − θs )K

(90)

where θs is chosen ∈ [0, 1]. We choose the value closest to unity that keeps ′ ˆ s+1 the absolute value of the determinant det(K ) above a certain threshold ˆ s′ )). To help us choose θs , we turn to Lemma 2.1 to σ (say, 0.001 det(K obtain   ′ ¯ Ts+1 ˆ s+1 ˆ s′ I + θs (K ˆ s′ )−1 AL ¯ s+1 L (91) K = K ′ ˆ s+1 ⇒ det(K )

=

ˆ s′ )(1 + θs h(K ˆ s′ )−1 AL ¯ s+1 , L ¯ s+1 i). det(K

(92)

For the update of the inverse approximate Jacobian this becomes ′ ˆ s+1 (K )−1

=

ˆ s′ )−1 + θs (1 − θs )(K

! ¯ Ts+1 (K ˆ s′ )−1 ˆ s′ )−1 AL ¯ s+1 L (K ′ ˆ (Ks ) − (93). ¯ T (K ˆ s′ )−1 AL ¯ s+1 1 + θs L s+1

′ ˆ s+1 ˆ s′ allows Note that writing K as a rank-one update with respect to K us to implement the algorithm without the need to know A or invert the approximate Jacobians, but using only matrix-vector and scalar products. It also allows us to anticipate and avoid numerical breakdown, which is not the case with (23). Tests have shown that this modification is adequate. The modified algorithm then reads as follows. .Algorithm 3.2 As algorithm 3.1, but ˆ s′ )−1 based on (po , H(po )), . . . , 2.b. Construct inverse approximate Jacobian (K (ps , H(ps )) using (93), where θs is chosen as close to unity as ′ ˆ s′ ) > σ det(K ˆ s−1 possible such thatdet(K ) in absolute value.

16

4

Other secant methods

We will compare our method to other quasi-Newton methods that can be implemented for black box systems (i.e., without the need to explicitly know the system matrices), can be used for non-spd systems, and use a rank-one update. Broyden’s first (or “good”), resp., second (or “bad”), method [1, 6] is ′ ˆ s+1,BG a quasi-Newton method where the approximate Jacobian K , resp., ′ ˆ s+1,BB K , is constructed using the following rank-one update: ′ ˆ s+1,BG K

=

′ ˆ s+1,BB K

=

′ ˆ s,BG (δKs − K δps )δpTs ′ ˆ s,BG K + , hδps , δps i ′ ′ ˆ s,BB ˆ s,BB (δKs − K δps )δKsT K ′ ˆ s,BB K + . ˆ′ hδKs , K s,BB δps i

(94) (95)

The column-updating method, resp., inverse column-updating method, is a quasi-Newton method that was introduced by Martinez [17], resp., by ′ ˆ s+1,CU Martinez and Zambaldi [18], where the approximate Jacobian K M, ′ ˆ s+1,ICU M , is constructed using the following rank-one update: resp., K ′ ˆ s+1,CU K M

=

′ ˆ s+1,ICU K M

=

′ T ˆ s,CU (δKs − K M δps )ejs ′ ˆ s,CU , (96) K M + hejs , δps i ′ T ˆ′ ˆ s,ICU (δKs − K M δps )el,s Ks,ICU M ′ ˆ s,ICU ,(97) K M + ˆ′ hel,s , K s,ICU M δps i

where {e1 , e2 , . . . , en } is an orthonormal basis for IRn×1 and where ejs , resp., els , is chosen such that js

=

Argmax{|hej , δps i|; j = 1, . . . , n},

(98)

ls

=

Argmax{|hel , δKs i|; l = 1, . . . , n}.

(99)

These four methods are known to exhibit superlinear convergence [15] and need at most 2n iterations to reach p∗ (Gay’s theorem [13]). No guarantee can be given that the approximate Jacobians are nonsingular nor is the convergence monotone in any sense. Apart from these four methods, we have also tested the algorithm of Eirola and Nevanlinna [11]. While this algorithm shows great potential when counted against the number of iterations, it needs two matrix-vector products with the system matrix AH per iteration, which made it uncompetitive as we judge the performance against the number of times the system matrix is called.

5 The generalized minimal residual (GMRes) method Most formulations of GMRes assume that the user has knowledge of b and can compute the matrix-vector product (A − I)x for any given x ∈ IRn×1 ; in our problem-setting we are working with a black box H (or K) such

17

that ∀x ∈ IRn×1 we can form H(x) = Ax − b without explicitly knowing b. Constructing ro = H(po ) − po thus poses no problem, but finding (A − I)i ro (i = 0, . . . , s) does. In [27] an algorithm was proposed that circumvents this problem (see Algorithm 5.1). Algorithm 1. Take a starting 5.1 (GMRes value po ; (see set do([27])) = po ; compute d1 = H(do ); ro = d1 − do . Set s = 1. 2. Loop until sufficiently converged: a. Compute ds+1 = H(ds ). b. δs = ds − do . c. ρs = (ds+1 − ds ) − ro . P d. Find coefficients {¯ ωi }i=s krs k2 with rs = ro + si=1 ωi ρi . i=1 that minimize Ps e. Optionally: compute ps = po + i=1 ω ¯ i δi . f. Set s = s + 1. Our experience with this variant of GMRes is that it exhibits poor numerical stability. In [27] a remedy was proposed in which the δs (s = 1, 2, . . .) were orthogonalized. As, in our view, it is mainly the conditioning of the ρs (s = 1, 2, . . .) which causes the instability in 2.d. of Algorithm 5.1, we noted only a slight improvement with this modification as it does not guarantee orthogonality of ρs (s = 1, 2, . . .). As we have not found a better alternative in the literature, we therefore propose to use standard GMRes (e.g., [23, 28]) after finding b with a function call K(0). If the initial iterate is po = 0, then this invokes no other cost; otherwise an extra function call needs to be spent.

6

Numerical results

We now compare the method presented in section 3 to Broyden’s good method (“BG”), Broyden’s bad method (“BB”), the column updating method (“CUM”), the inverse column updating method (“ICUM”), and GMRes. For the secant methods of section 4 we start from the initial guess ˆ o′ = −I as is the case for our method. of the approximate Jacobian: K At first, this might seem illogical, but it is done to be better able to compare with our method and as no better guess than the zero matrix is available for the Jacobian of H since we are working with black box systems.

6.1 Test matrices from the Matrix Market Repository The following square non-spd matrices were taken from the Matrix Market Repository [21] and substituted for A − I (the system matrix of K); all come from the original Harwell sparse matrix test collection. 1. SHL 0: simplex method basis matrix; n = 663. 2. STR 0: simplex method basis matrix; n = 363. 3. BP 0: simplex method basis matrix; n = 822.

18

4. FS 541 1, 2, 3, and 4: one stage of FACSIMILE stiff ordinary differential equation (ODE) package, for the atmospheric pollution problem, involving chemical kinetics and two-dimensional transport; n = 541. 5. ARC130: laser problem; n = 130. 6. GENT113: statistical application; n = 113. 7. IBM32: 1971 IBM conference advertisement; n = 32. 8. CURTIS54: stiff biochemical ODEs; n = 54. 9. ASH85: Holland survey; n = 85. 10. ASH292: United Kingdom survey; n = 292. 11. ERIS1176: electrical network; n = 1176. For the vector b we take a vector such that [b]i = 1 (i = 1, . . . , n). All iterations will start from po = 0, and no initial relaxation is used (ω = 1). As a convergence requirementtake a relative reduction of the K(ps ) residual K(p ≤ 10−5 and measure the number of function calls (H(p) o) or K(p)) necessary to obtain convergence, as we assume these are the most expensive part.8 If more than 3n iterations are needed, we break off the iteration and say that the method is nonconvergent (“nc” in the tables); if at some point in the iteration the iterates “blow up” (i.e., the values exceed the largest floating point number of the software9 ), we say it is divergent (“div” in the tables). The results can be found in Table 1.

8 This is not necessarily the case in all of these tests, but as our method is intended to be used in circumstances where each solver call is very expensive we maintain this performance measure. 9 In our tests this was Matlab 7.0.

19

Table 1: Number of iterations required for convergence. Equation K(p) = 0 defined by matrices from the Matrix Market Repository. “div” = divergence; “nc” = nonconvergence after 3n function calls. Matrix

n

SHL0 STR0 BP0 FS 541 1 FS 541 2 FS 541 3 FS 541 4 ARC130 GENT113 IBM32 CURTIS54 ASH85 ASH292 ERIS1176

n n n n n n n n n n n n n n

= 663 = 363 = 822 = 541 = 541 = 541 = 541 = 130 = 113 = 32 = 54 = 85 = 292 = 1176

QN-LS

BG

BB

CUM

ICUM

GMRes

673 366 850 11 250 388 284 28 76 33 49 88 288 293

1337 622 nc 25 1146 div div 54 182 70 72 129 457 809

1023 538 nc 25 div div div 39 174 63 87 163 688 nc

1765 796 nc 27 nc div nc 42 203 72 84 151 618 nc

nc nc div 27 div div div 43 232 75 106 243 nc div

664 364 823 9 203 262 223 11 67 32 48 86 287 267

10

10

70

10

QN−LS BB GMRes

QN−LS 60

10

50

ICUM

10 Relative residual (logscale)

Relative residual (logscale)

BG CUM

5

10

0

10

−5

10

40

10

30

10

20

10

10

10

0

10 −10

10

0

200

400

600 800 Function calls

1000

−10

10

1200

0

500

1000 Function calls

1500

2000

Figure 1: Convergence history of the different solvers for the SHL0 matrix testcase. As an illustrative example we also plot the convergence history of the different methods for the SHL0 matrix (Figure 1). We note that GMRes is the fastest in all test-cases, which is not surprising, as it is well known to be one of the best linear solvers for nonsymmetric systems [23]; it has a slight advantage in these tests because the initial iterate is the zero-vector so that no extra function call is needed to compute b. Still, QN-LS shows very similar performance for the SHL0, STR0, IBM32, CURTIS54, and ASHxxx matrices and only slightly lower performance for the BP0, GENT113, and ERIS1176 matrices. For the other matrices it is still the second fastest solver. We see that QN-LS is reasonably stable, as it does not diverge (unlike the other quasi-Newton methods), and that the maximum number of function calls needed comes close to the theoretical value of n + 2. Of the other quasi-Newton methods the inverse column-updating method is the worst, followed by the column-updating method in all but a few test-

20

cases, while Broyden’s “bad” method seems not so bad after all as it is better than Broyden’s “good” method in 10 test-cases and only worse in 5. All other quasi-Newton methods fail quite often.

6.2 6.2.1

Heat equation The model

The heat equation in one dimension without heat source is given by   ∂T ∂ ∂T ρC − k = 0. (100) ∂t ∂x ∂x T is temperature, ρ is density, C is heat capacity, and k is thermal conductivity. We are working with a fictional material where ρ, C, and k vary linearly from the left boundary to the right boundary in the following way: • Left boundary: ρ = 10−4 kg/m3 , C = 10−7 J/(kg · K), and k = 10−1 W/(m· K). • Right boundary: ρ = 10−2 kg/m3 , C = 10−5 J/(kg · K), and k = 10−3 W/(m· K). We use a finite differencing scheme on n equidistant nodes for the solution of (100), with spacing ∆x, and an implicit time discretization with time step ∆t. We assume that at time ν we have a uniform temperature T ν = T o = 500K. After dividing by T o we get the n linear equations (i = 1, . . . , n):   ν+1 ρ i Ci T ′ i − 1  µ ν+1 ν+1 ν+1 (ki+1 + ki )T ′ i+1 − (ki+1 + 2ki + ki−1 )T ′ i + (ki + ki−1 )T ′ i−1 = 0, − 2 (101) ′ o ∆t where µ = ∆x 2 and T = T /T . We write (101) as K(T ′ ) = 0 for short, where we are solving only the equation for the next time step. We start the iterations from a uniform temperature T ′ = 1; the right boundary condition is T ′ = 1, while the left boundary condition is a function of time, defined as TL′ = 1 + 12 sin πt . 10

6.2.2

Results

In this test the following values of µ were tested 10−4 , 10−3 , . . . , 104 , and we tried n = 100 and n = 1000. We use no initial relaxation, i.e., ω = 1. The convergence criterion is a relative reduction of the residual of 10−5 , and as a performance measure the number of solver calls is used. We break off the iteration if no convergence has been obtained after 3n solver calls. The results can be found in Tables 2 and 3. As an illustrative example we also plot the convergence history of the different methods for the test-case with n = 100 and µ = 10−3 (Figure 2).

21

Table 2: Number of iterations required for convergence of the heat equation. n = 100. “div” = divergence; “nc” = nonconvergence after 3n function calls. µ

QN-LS

BG

BB

CUM

ICUM

GMRes

10−4 10−3 10−2 10−1 1 10 102 103 104

3 15 85 126 115 107 103 101 101

3 nc nc nc nc nc 233 228 226

3 16 81 nc 298 nc nc div div

3 81 nc nc nc nc nc nc nc

3 23 206 nc nc nc div nc div

3 9 31 95 101 101 101 101 101

Table 3: Number of iterations required for convergence of the heat equation. n = 1000. “div” = divergence; “nc” = nonconvergence after 3n function calls. µ

QN-LS

BG

BB

CUM

ICUM

GMRes

10−4

3 16 132 931 2006 1641 1055 1010 1003

3 405 nc nc nc nc div div div

3 17 81 504 2185 nc div div div

3 317 nc nc nc nc nc div div

3 28 366 nc nc nc div div div

3 9 33 141 560 1001 1001 1001 1001

10−3 10−2 10−1 1 10 102 103 104

22

1

1

10

10 QN−LS BB GMRes

0

−1

10

−2

10

−3

10

−4

10

−5

−1

10

−2

10

−3

10

−4

10

−5

10

10

−6

10

QN−LS BG CUM ICUM

0

10

Relative residual (logscale)

Relative residual (logscale)

10

−6

0

2

4

6

8 10 Function calls

12

14

10

16

0

10

1

2

10 10 Function calls (logscale)

3

10

Figure 2: Convergence history of the different solvers for the heat equation with n = 100 and µ = 10−3 . We see that a value µ = 10−4 represents an “easy” case for which all solvers perform equally well with a minimum of function calls. For n = 100 the other quasi-Newton methods need a far higher number of function calls once µ becomes larger; this is especially the case for CUM and ICUM and is even more pronounced for n = 1000. When µ is larger than 10−4 , we see that the number of function calls for GMRes steadily increases until its theoretical value of n + 2, including the extra function call to find b, is reached for µ = 10 (for n = 100) or µ = 100 (for n = 1000). QN-LS also needs more function calls when µ becomes larger but experiences a peak around µ = 1 where more than n + 2 function calls are needed. It also settles around its theoretical value of n+2 for higher values of µ. The reason for this peak is, as yet, not clear.

6.3 6.3.1

ODE boundary value problem The model

In [8] the following ODE boundary value problem was proposed as a testcase: −

d2 u du +β dx2 dx

=

0 on ]0, 1[,

(102)

with boundary conditions u(0) = 1 and u(1) = 0. Equation (102) describes a convection-diffusion problem and is discretized on a uniform grid with step-size h = 1/(n + 1) using first-order upwind discretization for the advection term and second-order central discretization for the diffusion term. This leads to a linear system that can be written as K(p) = AK p − b = 0.

6.3.2

Results

In this test the following values of β were tested: 10−4 , 10−3 , . . . , 105 . As in [8] we take n = 50. po is chosen as po = [1 1 . . . 1]T . We use no initial relaxation, i.e., ω = 1. The convergence criterion is a relative

23

Table 4: Number of iterations required for convergence of the ODE in (102). n = 50. “div” = divergence; “nc” = nonconvergence after 3n function calls. β

QN-LS

BG

BB

CUM

ICUM

GMRes

10−4 10−3 10−2 10−1 1 10 102 103 104 105

72 54 54 67 54 54 14 7 6 5

113 111 112 113 114 115 91 99 101 101

nc nc nc nc nc nc nc nc div div

145 149 147 148 148 149 102 99 101 101

nc nc nc nc nc nc nc nc nc nc

52 52 52 52 52 52 13 6 5 4

5

6

10

10 QN−LS BG GMRes

QN−LS BB

4

10

CUM

0

ICUM Relative residual (logscale)

Relative residual (logscale)

10

−5

10

2

10

0

10

−2

10

−10

10

−4

10

−15

10

0

−6

20

40

60 Function calls

80

100

10

120

0

20

40

60

80 100 Function calls

120

140

160

Figure 3: Convergence history of the different solvers for the boundary value problem with β = 10−3 . reduction of the residual of 10−5 , and as a performance measure the number of solver calls is used. We break off the iteration if no convergence has been obtained after 3n solver calls. The results can be found in Table 4. We see that for this test Broyden’s bad method and ICUM invariably fail, because they either diverge or need more than 3n function calls. Broyden’s good method stays close to its theoretical maximum of function calls (2n); CUM also uses around 2n function calls for high values of β but gradually becomes worse for lower values of β. We point out that for high values of β the initial value po is close to the solution, which explains the trend. QN-LS stays close to its theoretical number of function calls (n + 2) for low values of β and gradually needs less iterations as β is increased. Only for β = 10−4 and β = 10−1 do we notice a deviation. Except for these two values, QN-LS differs by around 2–3 iterations with respect to GMRes, which is clearly the fastest solver. As an illustrative example we also plot the convergence history of the different methods for the test-case with β = 10−3 (Figure 3). We see that, in this particular test, the “failure” of Broyden’s bad method and ICUM is due to the fact that they need (marginally) more than 3n function calls.

24

7

Conclusions

We have transformed a method that was first presented as a coupling technique for nonlinear fluid-structure interaction problems into a quasiNewton method that uses a rank-one update. It can be used to solve problems of which no information about the system matrix is available (black box) and imposes no condition on the nature of the system matrix (like positive definiteness). It is shown that the method analytically converges in n + 1 steps for linear problems if no singularities occur; a fix for the latter has been proposed. Numerical tests show that, with round-off errors taken into account, the number of iterations needed for convergence is very close to this theoretical value. Line-searches, which are common for most quasi-Newton methods, have been shown to have no long term effect on the convergence. The method has been compared to four other quasi-Newton methods and GMRes on a number of test-cases. It steadily outperforms those quasi-Newton methods and is only slightly slower than GMRes.

References [1] C.G. Broyden, A class of methods for solving nonlinear simultaneous equations, Math. Comp., 19 (1965), pp. 577–593. [2] C.G. Broyden, Quasi-Newton methods and their applications to function minimization, Math. Comp., 21 (1967), pp. 368–381. [3] C.N. Dawson, H. Klie, M.F. Wheeler, and C.S. Woodward, A parallel, implicit, cell-centered method for two-phase flow with a preconditioned Newton-Krylov solver, Comput. Geosci., 1 (1997), pp. 215–249. ´, A characterization of superlinear [4] J.E. Dennis and J.J. More convergence and its application to quasi-Newton methods, Math. Comp., 28 (1974), pp. 549–560. ´, Quasi-Newton methods, [5] J.E. Dennis, Jr., and J.J. More motivation and theory, SIAM Rev., 19 (1977), pp. 46–89. [6] J.E. Dennis, Jr., and R.B. Schnabel, Least change secant updates for quasi-Newton methods, SIAM Rev., 21 (1979), pp. 443–459. [7] J.E. Dennis and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice–Hall, Englewood Cliffs, NJ, 1983. [8] P. Deuflhard, R. Freund, and A. Walter, Fast secant methods for the iterative solution of large nonsymmetric linear systems, IMPACT Comp. Sci. Eng., 2 (1990), pp. 244–276. [9] P. Deuflhard, Newton Methods for Nonlinear Problems, Affine Invariance and Adaptive Algorithms, 2nd ed., Springer Ser. Comput. Math. 35, Springer, New York, 2006. [10] L.C.W. Dixon, Automatic Differentiation and Parallel Processing in Optimisation, Technical report 180, The Hatfield Polytechnique, Hatfield, UK, 1987.

25

[11] T. Eirola and O. Nevanlinna, Accelerating with rank-one updates, Linear Algebra Appl., 121 (1989), pp. 511–520. [12] W. Gai, J. Xue, and Y. Qu, A new implementation of the EN method, Appl. Math. Comput., 98 (1999), pp. 199–208. [13] D.M. Gay, Some convergence properties of Broyden’s method, SIAM J. Numer. Anal., 16 (1979), pp. 623–630. [14] G. Kedem, Automatic differentiation of computer programs, ACM Trans. Math. Software, 6 (1980), pp. 150–165. [15] C.T. Kelley, Iterative Methods for Linear and Nonlinear Equations, Frontiers Appl. Math. 16, SIAM, Philadelphia, 1995. [16] V.L.R. Lopes and J.M. Martinez, Convergence properties of the Inverse Column-Updating method, Optim. Methods Softw., 6 (1995), pp. 127–144. [17] J.M. Martinez, A quasi-Newton method with modification of one column per iteration, Computing, 33 (1984), pp. 353–362. [18] J.M. Martinez and M.C. Zambaldi, An inverse column-updating method for solving large-scale nonlinear systems of equations, Optim. Methods Softw., 1 (1992), pp. 129–140. [19] J.M. Martinez, On the convergence of the column-updating method, Comput. Appl. Math., 12 (1993), pp. 83–94. [20] J.M. Martinez, Practical quasi-Newton method for solving nonlinear systems, J. Comput. Appl. Math., 124 (2000), pp. 97–122. [21] Matrix Market Repository, http://math.nist.gov/MatrixMarket/. [22] J.M. Ortega and W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Classics Appl. Math. 30, SIAM, Philadelphia, PA, 2000. [23] Y. Saad, Iterative Methods for Sparse Linear Systems, 2nd ed., SIAM, Philadelphia, 2003. [24] A.H. Sherman and W.J. Morrison, Adjustment of an inverse matrix corresponding to changes in the elements of a given column or a given row of the original matrix, Ann. Math. Statist., 21 (1950), pp. 124–127. [25] E. Spedicato, Z. Xia, and L. Zhang, ABS algorithms for linear equations and optimization, J. Comput. Appl. Math., 124 (2000), pp. 155–170. [26] G. Strang, Linear Algebra and its Applications, 3rd ed., Saunders, Orlando, 1988. [27] E.H. Van Brummelen, C. Michler, and R. De Borst, Interface-GMRES(R) Acceleration of Subiteration for Fluidstructure-interaction Problems, Report DACS–05–001, Delft Aerospace Computational Science, Delft, The Netherlands, 2005. [28] H.A. van der Vorst, Iterative Krylov Methods for Large Linear Systems, Cambridge Monogr. Appl. Comput. Math., Cambridge University Press, New York, 2003.

26

[29] J. Vierendeels, L. Lanoye, J. Degroote, and P. Verdonck, Implicit coupling of partitioned fluid-structure interaction problems with reduced order models, Comput. & Structures, 85 (2007), pp. 970–976. [30] J. Vierendeels, Implicit coupling of partitioned fluid-structure interaction solvers using reduced-order models, in Fluid-Structure Interaction, Modelling, Simulation, Optimization, H.-J. Bungartz and M. Sch¨ afer, eds., Lect. Notes Comput. Sci. Eng. 53, Springer, Berlin, 2006, pp. 1–18.

27