Infeasible start semidefinite programming algorithms via self-dual embeddings Report 97-10
E. de Klerk C. Roos T. Terlaky
Faculteit der Technische Wiskunde en Informatica Faculty of Technical Mathematics and Informatics Technische Universiteit Delft Delft University of Technology
ISSN 0922-5641
Copyright c 1997 by the Faculty of Technical Mathematics and Informatics, Delft, The Netherlands. No part of this Journal may be reproduced in any form, by print, photoprint, microfilm, or any other means without permission from the Faculty of Technical Mathematics and Informatics, Delft University of Technology, The Netherlands. Copies of these reports may be obtained from the bureau of the Faculty of Technical Mathematics and Informatics, Julianalaan 132, 2628 BL Delft, phone +31152784568. A selection of these reports is available in PostScript form at the Faculty’s anonymous ftp-site. They are located in the directory /pub/publications/tech-reports at ftp.twi.tudelft.nl
DELFT UNIVERSITY OF TECHNOLOGY
REPORT Nr. 97{10 Infeasible Start Semidefinite Programming Algorithms via Self{Dual Embeddings.
E. de Klerk, C. Roos, T. Terlaky
ISSN 0922{5641 Reports of the Faculty of Technical Mathematics and Informatics Nr. 97{10 Delft, December 13, 1996 i
E. de Klerk, C. Roos and T. Terlaky, Faculty of Technical Mathematics and Informatics, Delft University of Technology, P.O. Box 5031, 2600 GA Delft, The Netherlands. e{mail:
[email protected],
[email protected],
[email protected]
c 1997 by Faculty of Technical Mathematics and Copyright Informatics, Delft, The Netherlands. No part of this Journal may be reproduced in any form, by print, photoprint, micro lm or any other means without written permission from Faculty of Technical Mathematics and Informatics, Delft University of Technology, The Netherlands. ii
Abstract The development of algorithms for semide nite programming is an active research area, based on extensions of interior point methods for linear programming. As semide nite programming duality theory is weaker than that of linear programming, only partial information can be obtained in some cases of infeasibility, nonzero optimal duality gaps, etc. Infeasible start algorithms have been proposed which yield di erent kinds of information about the solution. In this paper a comprehensive treatment of a speci c initialization strategy is presented, namely self-dual embedding, where the original primal and dual problems are embedded in a larger problem with a known interior feasible starting point. A framework for infeasible start algorithms with the best obtainable complexity bound is thus presented. The information that can be obtained in case of infeasibility, unboundedness, etc., is stated clearly. Important unresolved issues are discussed.
iii
1 Introduction The extension of interior point algorithms from linear programming (LP) to semide nite programming (SDP) became an active research area when Alizadeh [1] and Nesterov and Nemirovskii [11] independently demonstrated the rich possibilities. Most of the algorithms found in the literature require feasible starting points. So-called `big-M' methods (see e.g. [20]) are often employed in practice to obtain feasible starting points. In the LP case an elegant solution for the initialization problem is to embed the original problem in a skew{symmetric self{dual problem which has a known interior feasible solution on the central path [23, 9]. The solution of the embedding problem then yields the optimal solution to the original problem, or gives a certi cate of either infeasibility or unboundedness. In this way detailed information about the solution is obtained. The idea of self-dual embeddings for LP dates back to the 1950's and the work of Goldman and Tucker [8]. With the arrival of interior point methods, the embedding idea was revived to be used in infeasible start algorithms. In spite of the desirable theoretical properties of self-dual embeddings, the idea did not receive immediate recognition in implementations, due to the fact that the embedding problem has a dense column in the coecient matrix. This can lead to ll-in of Choleski factorizations during computation. In spite of this perception, Xu et al. [21] have made a successful implementation for LP using the embedding, and it has even been implemented as an option in the well-known commercial LP solver CPLEX-barrier. The common consensus now is that this strategy promises to be competitive in practice [3] (see also [6] and [17]). A homogeneous embedding of monotone nonlinear complementarity problems is discussed by Andersen and Ye in [4]. For semide nite programming the homogeneous embedding idea was rst developed by Potra and Sheng [14]. The embedding strategy was extended by De Klerk et al. in [5] and independently by Luo et al. [10] to obtain self-dual embedding problems with nonempty interiors. The resulting embedding problem has a known centered starting point, unlike the homogeneous embedding; it can therefore be solved using any feasible path-following interior point method. This is an advantage in the SDP case, where many possible primal-dual algorithms are available, while none has yet emerged as clear favourite. A so{called maximally complementary solution (e.g. the limit of the central path) of the embedding problem yields one of the following alternatives about the original problem pair: (I) an optimal solution with zero duality gap for the original problem is obtained; (II) a ray is obtained for either the primal and/or dual problem (strong infeasibility is 1
detected); (III) a certi cate is obtained that no optimal solution pair with zero duality gap exists and that neither the primal nor the dual problem has a ray. This can only happen if one or both of the primal and dual SDP problems fail to satisfy the Slater regularity condition. Loosely speaking, the original primal and dual problems are solved if a complementary solution pair exists, or if one or both of the problems are strongly infeasible. Unfortunately, some pathological duality e ects can occur for SDP1 which are absent from LP, for example:
A positive duality gap at an optimal primal-dual solution pair; an arbitrarily small duality gap can be attained by feasible primal-dual pairs, but
no optimal pair exists; an SDP problem may have an optimal solution even though its (Lagrangean) dual is infeasible.
In cases like these little or no information could be given in [5]. In this paper we elaborate on the work in [5] in three important respects: (1) It is indicated how to extend p any primal-dual algorithm to solve the embedding problem. In this way O( n + 2 log(1=)) iteration complexity can be obtained for computing an -optimal solution to the embedding problem. (2) The problem of how to decide which variables are zero in a maximally complementary solution of the embedding problem, if only an -optimal solution is known, is discussed. This is important in drawing conclusions about the original problem pair from an -optimal solution of the embedding problem. (3) Solutions to the unresolved duality questions are given. We show how to detect weak infeasibility and unboundedness in general by using extended Lagrange-Slater dual problems [15] in the embedding, where necessary. In this way the optimal value of a given SDP problem can be obtained if it is nite. This solves the open problem posed by Ramana in [16], namely how to use the extended Lagrange-Slater dual problems in an infeasible-start algorithm.
Outline of the paper After some preliminaries in Section 2, a review of recent results concerning the convergence of the central path is given in Section 3, with simpli ed proofs. The embedding strategy 1 Examples of these e ects will be given in Sections 2 and 7, and can also be found in [20] and [10].
2
is discussed thereafter in Section 4. Solution strategies for solving the embedding problem are given in Section 5. In Section 6 it is shown how to interpret an -optimal solution of the embedding problem in order to draw conclusions about the solution of the original problem pair (i.e. to distinguish between the abovementioned cases (I) to (III)). The remaining diculties are highlighted. Remaining duality issues and ways of detecting weak infeasibility are discussed in Section 7. In Section 8 it is shown how extended Lagrange-Slater duals can be used in the embedding strategy instead of Lagrangean duals to give a certi cate of the status of a given problem. Finally, some conclusions are drawn.
2 Preliminaries Problem statement We will consider the semi{de nite programming problem in the standard form. Thus a problem in standard primal form may be written as: (P ) : nd p = inf Tr(CX ) X subject to Tr(AiX ) = bi; i = 1; : : : ; m X 0; where C and the Ai's are symmetric n n matrices, b 2 IRm, and X 0 means X is positive semi{de nite. The Lagrangean dual problem of a problem of the form (P ) takes the standard dual form: (D) : nd d = sup bT y S;y
subject to m
X
yiAi + S = C; S 0: The solutions X and (y; S ) will be referred to as feasible solutions as they satisfy the primal and dual constraints respectively. The values p and d will be called the optimal values of (P ) and (D), respectively. We use the convention that p = ?1 if (P ) is unbounded and p = 1 if (P ) is infeasible, with the analogous convention for (D). The primal and dual feasible sets will be denoted by P and D respectively, and P and D will denote the respective optimal sets, i.e. i=1
n
o
P = fX 2 P : Tr(CX ) = pg and D = (S; y) 2 D : bT y = d : 3
A problem (P ) (resp. (D)) is called solvable if P (resp. D ) is nonempty. We will assume that the matrices Ai are linearly independent. Under this assumption y is uniquely determined for a given dual feasible S .
The duality gap and orthogonality property Recall that the duality gap for (P ) and (D) at solutions X 2 P and (y; S ) 2 D is given by m m Tr(CX ) ? bT y = Tr ( yiAi + S )X ? yiTr(AiX ) = Tr(SX ): !
X
X
i=1
i=1
The optimal duality gap is said to be zero if inf Tr(CX ) = sup bT y: X 2P S;y2D that P and
(1)
Note that this de nition does not imply D are nonempty. A problem (P ) (resp. (D) ) is called strictly feasible if there exists X 2 P with X 0 (resp. (y; S ) 2 D with S 0). Strict feasibility is equivalent to the well-known Slater's constraint quali cation or Slater regularity condition. It is well-known that if both (P ) and (D) are strictly feasible, then P and D are nonempty and the duality gap is zero. If both (P ) and (D) are feasible, and one is strictly feasible, then (1) is also guaranteed to hold. The proof of the following well-known orthogonality property is trivial.
Lemma 2.1 (Orthogonality) Let (X; S ) and (X 0; S 0) be two pairs of feasible solutions. The following orthogonality relation holds:
Tr (X ? X 0 )(S ? S 0 ) = 0:
Feasibility issues To decide about possible infeasibility and unboundedness of the problems (P ) and (D) we need the following de nition.
De nition 2.1 We say that the primal problem (P ) has a ray if there is a symmetric matrix X 0 such that Tr(Ai X ) = 0; 8 i and Tr(C X ) < 0. Analogously, the dual problem (D) has a ray if there is a vector y 2 IRm such that S := ? mi=1 yiAi 0 and bT y > 0. P
4
Primal rays cause infeasibility of the dual problem, et vice versa. Formally one has the following result.
Lemma 2.2 If there is a dual ray y then (P ) is infeasible. Similarly, a primal ray X
implies infeasibility of (D).
Proof:
Let a dual ray y be given. By assuming the existence of a primal feasible X one has m T 0 < b y = Tr(AiX )yi = ?Tr(X S) 0; X
i=1
which is a contradiction. The proof in case of a primal ray proceeds similarly.
2
De nition 2.2 Problem (P) (resp. (D)) is called strongly infeasible if (D) (resp. (P)) has a ray.
Every infeasible LP problem is strongly infeasible, but in the SDP case so-called weak infeasibility is also possible.
De nition 2.3 Problem (P) is weakly infeasible if P = ; and for each > 0 exists an X 0 such that jTr(AiX ) ? bij ; 8i: Similarly, problem (D) is called weakly infeasible if D = ; and for every > 0 exist y 2 IRm and S 0 such that m yiAi + S ? C :
X
i=1
Example 2.1 An example of weak infeasibility is given if (D) is de ned by: nd sup y1
subject to
2
3
2
3
6 4
7 5
6 4
7 5
0 1 1 0 1 0 0 0 where we can construct an `-infeasible solution' by setting 1= 1 ; y1 = ? 1 : S= 1
y1
2
3
6 4
7 5
2
It can be shown [10] that an infeasible SDP problem is either weakly infeasible or strongly infeasible. 5
Complementarity The optimality conditions for (P ) and (D) are Tr(AiX ) = bi; X 0 i = 1; : : : ; m m y A + S = C; S 0 i=1 i i XS = 0
P
9 > > > > = > > > > ;
(2)
Solutions X and S satisfying the last equality constraints are called complementary. Since X and S are symmetric positive semi{de nite matrices the complementarity of X and S (X S = 0) is equivalent to Tr(X S ) = 0. Complementary feasible solutions therefore are optimal with zero duality gap.
De nition 2.4 An optimal solution (X ; S ) is a maximally complementary solution if 2
X and S have maximal rank .
It is easy to show that all solutions in the relative interior of P (resp. D) are maximally complementary. Moreover, all primal maximally complementary solutions share the same column space, as do all dual maximally complementary solutions [7]. These two column spaces are orthogonal. We will denote by QP and QD any two orthogonal matrices obtained by taking orthonormal bases for the two spaces as columns. Any X 2 P and S 2 D can then be written as (3) X = QP UX QTP ; S = QD US QTD for suitable matrices UX 0; US 0 (not necessarily diagonal). If (X ; S ) are maximally complementary then UX ; US 0.
3 Features of the central path In this section we assume that (P ) and (D) are strictly feasible. The analysis of this section will then apply to the embedding problem presented in the next section, as the embedding problem will be self-dual and strictly feasible. If the optimality conditions (2) for (P ) and (D) are relaxed to Tr(AiX ) = bi; X 0; i = 1; : : :; m yiAi + S = C; S 0 XS = I; X
2 Results pertaining to bounds on the rank of optimal solutions may be found in [12, 13], and on nondegeneracy
and strict complementarity properties of optimal solutions in [2].
6
with some > 0, then this system has a unique solution [7], denoted by X (); S (); y(). This solution can be seen as the parametric representation of a smooth curve (the central path) in terms of the parameter . The central path converges to the analytic center of the optimal set as ! 0, and the analytic center is a maximally complementary solution. The convergence result was rst obtained by Goldfarb and Scheinberg [7]. Only the weaker result stating that all limit points of the central path are maximally complementary will be used here, which was already established in [5]. We include a simple proof of the convergence of the central path to the analytic center of the optimal set, under the assumption that a strictly complementary solution exists.
Theorem 3.1 (Maximal complementarity) Consider any sequence ftg ! 0 with t > 0, t = 1; . Any convergent subsequence of (X (t ); S (t)) on the central path converges to a maximally complementary solution as t ! 0. Proof: The existence of the limit points is easily proved by showing:
Lemma 3.1 Given > 0, the set f(X (); S ()) : 0 g
is bounded.
Proof: Let (X 0; S 0) be any strictly feasible primal-dual solution, and (X (); S ()) a central solution corresponding to some > 0. By orthogonality, Lemma 2.1, one has Tr (X () ? X 0)(S () ? S 0) = 0: (4) The centrality conditions imply Tr (X ()S ()) = n, which simpli es (4) to Tr(X ()S 0 ) + Tr(X 0S ()) = n + Tr(X 0S 0): (5) The left hand side terms of the last inequality are nonnegative by feasibility. For a given > 0 one therefore has Tr X ()S 0 n + Tr(X 0S 0) which implies Tr(X 0S 0) ; 8 Tr (X ()) n + (S 0)
min 0 where min(S ) denotes the smallest eigenvalue of S 0.
Now using the fact that any positive semide nite matrix X satis es kX k Tr(X ) for the Frobenius norm, one has 0 0 kX ()k n + Tr((SX0)S ) ; 8 min 7
A similar bound can be derived for kS ()k.
2
To prove Theorem 3.1 we need to show that limit points of the sequences X (t) and S (t) are also maximally complementary, i.e. if X = limt!0 X (t ) and S = limt!0 S (t) then X and S have the same rank as maximally complementary solutions X and S respectively. The proof given here is a straightforward adaptation of the proof given in [5] for SDP problems in symmetric form.
Lemma 3.2 The matrices X := limt!0 X (t) and S := limt!0 S (t) are maximally complementary.
Proof: We prove here that the rank of X is the same as the rank of X . The proof for S proceeds analogously. From the proof of Lemma 3.1 we have that Tr(X (t)S ) + Tr(X S (t)) ? Tr(X S ) = nt : Using Tr(X S ) = 0 and S (t)=t = X (t )?1 we obtain that Tr(X (t )?1X ) + Tr(S (t)?1S ) = n (6) for all t > 0, which implies Tr(X (t)?1X ) n; since both left hand side terms in (6) are nonnegative. Let the rank of X be r, and let QX be an (n r) orthonormal matrix with the eigenvectors of X corresponding to positive eigenvalues as columns. Then one has X = QX X QX T where X is a (r r) positive diagonal matrix containing the positive eigenvalues of X . It follows that
Tr(X (t )?1X ) = Tr X (t )?1QX X QX T = Tr QX T X (t)?1QX X n: (7) Using this bound we derive
= Tr QTX X (t)?1QX X X ?1 1 Tr QT X ( )?1Q ( t X X X min X ) n( ) : min X For further reference we introduce the notation K := minn(X ) . Since X (t ) is symmetric positive de nite, its spectral decomposition for each t can be given as X (t) = Q(t)X (t)Q(t)T ; Tr QX T X (t)?1QX
8
where X (t) is an n n positive diagonal matrix and Q(t)T = Q(t)?1 is an n n orthonormal matrix. Then X (t)?1 = Q(t)T X (t)?1Q(t) and further B (t) = Q(t)T QX is also an orthonormal matrix. Using this we have
Tr(QX T X (t )?1QX ) = Tr QTX Q(t)X (t)?1Q(t)T QX = Tr B (t)T X (t)?1B (t) = Tr X (t)?1B (t)B (t)T n B ( ) B ( )T t i t i = K; (8) ( ) X t i i=1 where B (t)i denotes the i-th row of the matrix B (t). Introducing the notation (t)i = B (t)iB (t)Ti we have that 0 (t)i 1 for all i and
X
n
X
i=1
(t)i = Tr B (t)T B (t) = Tr B (t)B (t)T = r:
These last two relations imply that at least r of the n (t)i's are larger than or equal to 1 n?r+1 . We can choose an appropriate subsequence (indicated again by subscript t for the sake of simplicity) where these coordinates are xed. Denote the set of these indices by I . Then we have (t)i n ? 1r + 1 ; i2I and by (8) i 2 I: X (t)i (n ? r1+ 1)K ;
Using the notation X = limt!0 X (t) we conclude that the diagonal matrix X has at least r nonzero diagonal elements. Since Q(t) is orthonormal, for an appropriate subsequence (still indicated by subscript t) one has Q = limt!0 Q(t) orthonormal, thus X = lim X (t) = lim Q(t)T X (t)Q(t) = QT X Q t !0 t !0
has at least rank r. By noting that X has maximal rank among the optimal solutions, one has rank(X ) rank(X ) and thus rank(X ) = r. This completes the proof of Theorem 3.1. 2 We now prove the convergence of the central path to the analytic center of the optimal set under the assumption that a strictly complementary solution exists. This result has been proved by Ye [22] for general self-scaled conic problems. It is nevertheless insightful to derive the proof for the semide nite case, which is analogous to the proof in the LP case. The assumption of strict complementarity simpli es things, but is not necessary { Goldfarb and Scheinberg [7] have proved the result in the general case where no strictly complementary solution is available. They show that any limit point of the central path satis es the KKT conditions of the optimization problem which de nes the analytic center. 9
Theorem 3.2 The central path converges to the analytic center of P D if the analytic center is strictly complementary.
Proof:
By the proof of Theorem 3.1 there exists a sequence ft g such that X (t ) ! X , S (t) ! S as t ! 0, and the eigenvector-eigenvalue decompositions X (t) = Q(t)X (t)Q(t)T and S (t) = Q(t)S (t)Q(t)T converge, say Q(t) ! Q, X (t) ! X , and S (t) ! S . We assume the ranks of X and S to be r and q respectively. We further assume that r + q = n, i.e. that a strictly complementary solution exists. Now construct the orthogonal eigenvector matrices QP (n r), (resp. QD (n q)) by only taking the columns in Q corresponding to positive eigenvalues in X (resp. Z ). One can therefore write Q = [QP ; QD]. Similarly de ne QP (t ) and QD(t ) so that Q(t) = [QP (t); QD (t)] and QP (t) ! QP , QD (t) ! QD . We can now split X (t ) into two eigenvalue matrices PX (t ) and DX (t) associated with the eigenvector matrices QP (t) and QD (t) respectively. Any optimal pair (X ; S ) can be written (by (3)) as (9) X = QP UX QTP ; and S = QDUS QTD for suitable matrices UX 0 and US 0 (not necessarily diagonal). In terms of this notation one has (10) UX = lim P (t): t !0 X Recall from (6) that
Tr X (t)?1X + Tr S S (t)?1 = n;
(11)
which implies Tr (X (t)?1X ) n and Tr (S S (t)?1) n. Using Q(t) = [QP (t); QD (t)] one has
Tr X (t )?1X = Tr Q(t)(t)?1 Q(t)T X ?1 ?1 = Tr QP (t) PX (t) QP (t)T X + Tr QD(t) DX (t) QD (t)T X
h
i
h
i
Tr QP (t) PX (t) ?1 QP (t)T X ;
h
i
where we have discarded a nonnegative term to obtain the inequality. Taking the limit as t ! 0 and using (9) and (10) yields
lim Tr X (t)?1 X Tr QP UX?1 QTP X = Tr UX?1UX : t !0 10
One can prove the relation
lim Tr S (t)?1S Tr US US?1 t !0
in exactly the same way. Thus we obtain the following relation from (11): Tr UX UX?1 + Tr US US?1 n: Using the arithmetic-geometric mean inequality applied to the eigenvalues of the matrices [UX UX?1 ] and [UX UX?1], yields 1 det UX UX?1 det US US?1 r+q r +1 q Tr UX UX?1 + Tr US US?1 r +n q = 1: (12) Inequality (12) holds for any optimal pair (X ; S ). Substituting S = S in (12) gives det UX det UX and by setting X = X : det US det US : The limit points are therefore maximizers of the following concave optimization problem which de nes the analytic center of P D (see also [7]): max log det (UX US ) U ;U ;y
h
h
i
i
X S
subject to
m
Tr AiQP UX QTP
= bi; i = 1; : : : ; m
yiAi + QDUS QTD = C i=1 UX 0; US 0: Thus a unique limit of the central path exists (the concave problem has a unique maximizer), given by the analytic center of P D. 2 X
4 The embedding strategy In what follows, we no longer make any assumptions about feasibility of (P ) and (D). Consider the following homogeneous embedding of (P ) and (D): Tr(AiX ) ? bi =0 ? mi=1 yiAi + C ?S =0 bT y ?Tr(CX ) ? = 0 y 2 IRm; X 0; 0; S 0; 0; P
11
8i
9 > > > > > > > > = > > > > > > > > ;
(13)
A feasible solution to this system with > 0 yields feasible solutions 1 X and 1 S to (P ) and (D) respectively (by dividing the rst two equations by ). The last equation guarantees optimality by requiring a nonpositive duality gap. For this reason there is no interior solution to (13). The formulation (13) was rst solved by Potra and Sheng [14] using an infeasible interior point method. In this paper we consider the extended self-dual embedding [5], in order to have a strictly feasible, self{dual SDP problem with a known starting point on the central path. The advantage is that any feasible start path-following algorithm can be applied to such a problem. This is an important consideration in SDP, where many possible search directions and algorithms are available, with no clear method of choice at this time. The strictly feasible embedding is obtained by extending the constraint set (13) and adding extra variables to obtain: min y;X; ;;S;; subject to Tr(AiX ) ? bi +bi ? mi=1 yiAi + C ?C ?S bT y ?Tr(CX ) + ? ) ? ?bT y +Tr(CX ? y 2 IRm; X 0; 0; 0; S 0; 0; 0 where bi := bi ? Tr(Ai) C := C ? I := 1 + Tr(C ) := n + 2: P
=0 =0 =0 = ?
8i
9 > > > > > > > > > > > = > > > > > > > > > > > ;
(14)
It is straightforward to verify that a feasible interior starting solution is given by y0 = 0, X 0 = S 0 = I , and 0 = 0 = 0 = 0 = 1. It is also easy to check that the embedding problem is self{dual via Lagrangean duality. This implies that the duality gap is equal to 2 and therefore = 0 at an optimal solution since the self{dual embedding problem satis es the Slater condition. It is readily veri ed that = Tr (XS ) + + : (15) This shows that an optimal solution (where = 0) satis es the complementarity conditions: XS = 0 12
= 0 = 0: We can now use a maximally complementary solution of the embedding problem (14) to obtain information about the original problem pair (P ) and (D). In particular, one can distinguish between the three possibilities as discussed in the Introduction, namely (I) A primal{dual optimal pair (X ; y) is obtained with zero duality gap Tr(CX ) ? bT y = 0; (II) A primal and/or dual ray is detected; (III) A certi cate is obtained that no optimal pair with zero duality gap exists, and that neither (P ) nor (D) has a ray. Given a maximally complementary solution of the embedding problem, these cases are distinguished as follows (for a proof, see [5]):
Theorem 4.1 Let (y; X ; ; ; S ; ; ) be a maximally complementary solution
to the self{dual embedding problem. Then:
(i) if > 0 then case (I) holds; (ii) if = 0 and > 0 then case (II) holds; (iii) if = = 0 then case (III) holds.
Three important questions now arise:
How is the embedding problem actually solved? How does one decide if > 0 and > 0 in a maximally complementary solution, if only an -optimal solution of the embedding problem is available? What additional information can be obtained if case (III) holds?
These three questions will be addressed in turn in the following three sections.
5 Solving the embedding problem The embedding problem can be solved by any path following primal-dual method. To this end, one can relax the complementarity optimality conditions of the embedding problem 13
to
XS = I = = ; ~ S~ as follows: If one de nes new `primal and dual variables' X; X S X~ = ; S~ = ; then the centrality condition becomes the usual X~ S~ = I . It follows from (15) that = (n + 2) along the central path. This observation will be important in Section 7. Furthermore, it is straightforward to verify that Tr X~ S~ = 0, i.e. the orthogonality principle holds for the new variables. These two observations make the application of primal-dual path following methods straightforward: the search direction at a given point ~ S~) can be computed from (X; 2
3
2
3
6 6 6 6 4
7 7 7 7 5
6 6 6 6 4
7 7 7 7 5
Tr(AiX ) ? bi +bi + C ?C ?S ? mi=1 yiAi bT y ?Tr(C X ) + ? ?bT y +Tr(C X ) ? ? P
=0 =0 =0 =0
8i
and T T L (XS + SX ) L?1 + L (XS + SX ) L?1 = 2I ? L (XS ) L?1 + L(XS )L?1 + = 2 ? + = 2 ? where the matrix L determines which linearization of the centrality condition is used (see e.g.p[24] and [19]). In this way the embedding problem can be solved to -optimality with O n + 2 log(1=) worst-case iteration complexity. Note that and can be viewed as eigenvalues of X~ and S~ respectively, corresponding to a xed, shared eigenvector. This interpretation will be important in the next section. h
i
h
i
6 Separating small and large variables A path following interior point method only yields an -optimal solution to the embedding problem. This solution may yield small values of and , and to distinguish between cases 14
(I) to (III) it is necessary to know if these values are zero in a maximally complementary solution. This is the most problematic aspect of the analysis at this time, and only partial solutions are given here. Two open problems are stated which would help resolve the current diculties. In what follows the set of feasible X~ for the embedding problems is denoted by P~ and the optimal set by P~ . The sets D~ and D~ are de ned similarly. Finally, the dimension of the embedding problem is n~ := n + 2. To separate `small' and `large' variables we need the following de nition:
De nition 6.1 The primal and dual condition numbers of the embedding are de ned as P := max min i (X~ ); D := max min i (S~); ~ ~ ~ ~ ~ ~ X 2P i:i (X )>0
S 2D i:i(S )>0
The condition number of the embedding is de ned as the minimum of these numbers := minfP ; D g.
Note that is well de ned and positive because the solution set of the strictly feasible, self-dual embedding problem is compact (see e.g. [7]). In linear programming a positive lower bound for can be given in terms of the problem data [17]. It is an open problem to give a similar bound in the semide nite case:
Open problem 6.1 Given strictly feasible SDP problems (P ) and (D) one can de ne similarly to De nition 6.1: Derive a lower bound for in terms of the problem data.
If we have a centered solution to the embedding problem with centering parameter then we can use any knowledge of to decide the following:
Lemma 6.1 For any positive one has:
() n~ and () n~ () n~ and () n~
if > 0 and = 0 if = 0 and > 0;
where the superscript indicates a maximally complementary solution.
Proof:
Assume that is positive in a maximally complementary solution. Let S~ 2 D~ be such that is as large as possible. By de nition one therefore has . Recall that Tr X~ ()S~ n~;
15
which implies that the eigenvalues of X~ ()S~ satisfy i X~ ()S~ n~ ; 8 i:
In particular
() n~ :
This shows that Since ()() = one also has
() n~ n~ :
() n~ : The case where > 0 and = 0 is proved in the same way.
2
The lemma shows that once the barrier parameter has been reduced to the point where 2 n~ , then it is known which of or is positive in a maximally complementary solution, provided that one is indeed positive. The case = = 0 cannot be detected using Lemma 6.1. It is an open problem to establish the convergence rate of and in this case.
Open problem 6.2 Consider the central path (X (); S ()) for strictly feasible problems (P ) and (D) where
i (X ())i (S ()) = ; i = 1; : : : ; n:
Let T f1; : : : ; ng be the index set where
i(X ()) ! 0 and i(S ()) ! 0; as ! 0 8i 2 T: Establish an upper bound for i (X ()) and i(S ()) for i 2 T in terms of . In a recent paper Stoer and Wechs [18] consider the analogous problem in thep case of horizontal sucient linear complementarity problems, and prove a bound of O( ). The proof of Lemma 6.1 can easily be extended to the case where the -optimal solution is only approximately centered, where approximate centrality is de ned by ~ S~) := min(X~ S~) ; (X; max(X~ S~) for some parameter > 1. Formally one has the result
16
~ S~) be a feasible solution of the embedding problem such that (X; ~ S~) Lemma 6.2 Let (X;
for some > 1. One has the relations: and () Tr(X~ S~) n ~ ~ Tr(X S ) and n
if > 0 and = 0 if = 0 and > 0
where the superscript indicates a maximally complementary solution.
7 Remaining duality and feasibility issues If = = 0 in a maximally complementary solution of the embedding problem (i.e. case (III) holds), then one of the following situations has occurred: 1) The problems (P ) and (D) are solvable but have a positive duality gap; 2) either (P ) or (D) (or both) are weakly infeasible; 3) both (P ) and (D) are feasible, but one or both are unsolvable, e.g. inf X 2P Tr(CX ) is nite but is not attained. The case 2) was illustrated in Example 2.1. The remaining two cases occur in the following examples:
Example 7.1 The following problem (adapted from [20]) which is in the form (D): nd sup y2 subject to
2
3
2
3
2
3
6 6 6 6 4
7 7 7 7 5
6 6 6 6 4
7 7 7 7 5
6 6 6 6 4
7 7 7 7 5
0 0 0 1 0 0 1 0 0 y1 0 1 0 + y2 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 is solvable with optimal value y2 = 0 but the corresponding primal problem has optimal value 1. 2
Example 7.2 Another diculty is illustrated by the following problem (adapted from
[20]): nd
sup y2 17
subject to
2
3
2
3
2
3
6 4
7 5
6 4
7 5
6 4
7 5
1 0 0 0 y1 + y2 0 1 0 0 0 1 1 1 which is not solvable but supy2D y2 = 1. The corresponding primal problem is solvable with optimal value 1. 2 The aim is therefore to see what further information can be obtained in the case = = 0. To this end, recall that along the central path of the embedding problem one has (t) (t) = t and (t) = n~ t (16) which shows that (t ) ! = 0 implies (t )= (t) ! 0 as t ! 0: (17) This shows (by (14)) that: Tr (1 ) AiX (t) ! bi; 8i (18) t and m y ( ) 1 S ( ) ! C: i t A (19) i+ ( ) ( ) t !
X
i=1
t
t
In other words if either or both of the sequences 1 S ( ) 1 X ( ) and (20) t (t) (t) t converge, the limit is feasible for (P ) or (D) respectively. On the other hand, if (18) (resp. (19)) holds but (P ) (resp. (D)) is infeasible, then (P ) (resp. (D)) is weakly infeasible. If one also has (t ) ! 0 as ! 0 (21) t (t) then it also follows from (14) that 1 bT y( ) ? 1 Tr (CX ( )) ! 0: t t (t) (t) If this happens, at least one of the sequences in (20) diverges (or else an optimal pair with zero duality gap exists). On the other hand, one always has (t)=(t ) ! 0 if (t) ! 0, from (16). If it also holds that (t) ! 0 as ! 0 (22) t (t ) (
)
(
18
)
then
1 bT y( ) ? 1 Tr (CX ( )) ! 1; t t (t) (t) Tr (1 ) AiX (t) ! 0; 8i t !
and
(23)
m
yi(t) A + 1 S ( ) ! 0: (24) i (t) t i=1 (t) An asymptotic ray (or weak ray) is thus detected for (P ) and/or (D). It is shown in [10] that an asymptotic ray in (P ) (resp. (D)) implies weak infeasibility in (D) resp. (P ). The problem is that none of these indicators gives a certi cate of the status of a given problem. For example, there is no guarantee that (22) will hold if one (or both) of (P ) and (D) have weak rays. Luo et al. [10] derive similar detectors and show that these detectors yield no information in some cases. We therefore need to go a step further, by replacing the embedding of (P ) and (D) with a di erent embedding problem where `stronger' duals are embedded. This is the subject of the next section. X
8 Embedding extended Lagrange-Slater duals Assume now that the aim is to solve a given problem (D) in the standard dual form, like the problems in the examples. In other words, we wish to nd the value d = sup bT y S;y2D
if it is nite or obtain certi cate that (D) is infeasible, or alternatively, a certi cate of unboundedness. For the example problems the embedding of (D) and its Lagrangean dual (P ) will be insucient for this purpose. The solution proposed here is to solve a second embedding problem, using so-called extended Lagrange-Slater duals. To this end, the so-called gapfree primal problem (Pgf ) of (D) may be formulated instead of using the standard primal problem (P ). The gapfree primal was rst formulated by Ramana [15], and takes the form: minTr (C (U0 + Wm)) subject to Tr (Ai(U0 + Wm)) = bi; i = 1; : : :; m Tr (C (Ui + Wi?1)) = 0; i = 1; : : : ; m Tr (Ai(Ui + Wi?1)) = 0; i = 1; : : : ; m W0 = 0 19
2 6 4
WiT
I Wi Ui
3 7 5
0; i = 1; : : : ; m
U0 0; where the variables are Ui 0 and Wi 2 IRnn ; i = 0; : : : ; m. Note that the gap-free primal problem is easily cast in the standard primal form. Moreover, its size is polynomial in the size of (D). Unlike the standard primal (P ), (Pgf ) has the following desirable features:
(Weak duality) If (y; S ) 2 D and (Ui ; Wi); i = 0; : : :; m is feasible for (Pgf ) then bT y Tr (C (U0 + Wm)). (Dual boundedness) If (D) is feasible, its optimal value is nite if and only if (Pgf )
is feasible. (Zero duality gap) The supremal value of (Pgf ) equals the in mum value of (D) if and only if both (Pgf ) and (D) are feasible. (Attainment) If the supremum value of (D) is nite, then it is attained by (Pgf ).
The standard (Lagrangean) dual problem associated with (Pgf ) is called the corrected dual (Dcor ). The surprising result is that the pair (Pgf ) and (Dcor ) are now `gap-free' [16], i.e. (1) is satis ed. Moreover, a feasible solution to (D) can be extracted from a feasible solution to (Dcor ). The only problem is that (Dcor ) does not necessarily attain its supremum, even if (D) does. A natural question is whether (Dcor ) is strongly infeasible if (D) is only weakly infeasible. This would simplify matters greatly as strong infeasibility can be detected more easily. Unfortunately this is not the case | it is readily veri ed that the weakly infeasible problem (D) in Example 2.1 has a weakly infeasible corrected problem (Dcor ). In what follows we solve the embedding problem using (Pgf ) and (Dcor ) for our problem (D). We assume therefore that the solution of the embedding of (D) and its Lagrangean dual (P ) has yielded = = 0. We therefore already know that (D) is not strongly infeasible. Three possibilities remain: (i) the problem (D) is feasible and has a nite supremal value; (ii) the problem (D) is feasible and unbounded but does not have a ray; (iii) the problem (D) is weakly infeasible; 20
If (and only if) case (i) holds, then (Pgf ) and (Dcor ) will have the same ( nite) optimal values (zero duality gap). Problem (Pgf ) will certainly attain this common optimal value, but (Dcor ) may not. The possible duality relations are listed in Table 1. Status of (D) Status of (Pgf ) Status of (Dcor ) d < 1 Pgf = d dcor = d unbounded infeasible unbounded infeasible unbounded infeasible Table 1: Duality relations for a given problem (D), its gapfree dual (Pgf ) and its corrected problem (Dcor ).
In what follows the variables (y; X; ; ; S; ; ) refer to the embedding of (Pgf ) and (Dcor ). The feasible sets of (Pgf ) and (Dcor ) are denoted by Pgf and Dcor respectively. We will use the subscripts `gf' and `cor' for the variables of (Pgf ) and (Dcor ) respectively, but the problem data for (Pgf ) and (Dcor ) will be denoted by C; b; Ai for simplicity. We aim to identify or exclude the general situation where (Pgf ) and (Dcor ) are such that where sup bT ycor = X min Tr(CXgf ); (25) 2P ycor ;Scor 2Dcor
gf
gf
and the optimal value supycor ;Scor 2Dcor bT ycor may or may not be attained. If the optimal value is attained, the embedding yields a solution with > 0 and we are done. Similarly, if = 0 and > 0, a ray is detected and the status of (D) follows from Table 1. We therefore need only consider the case where the embedding of (Pgf ) and (Dcor ) has = = 0 in a maximally complementary solution. To proceed, we rst show that (21) must hold if d is nite.
Lemma 8.1 Assume that a given problem (D) has nite optimal value d. Then (21)
holds for the embedding of (Pgf ) and (Dcor ).
Proof:
Let t := (t)= (t ) and (Xgf ; ycor ; Scor ) 2 Pgf Dcor . Note that t ! 0 as t ! 1 by (17). For ease of notation we further de ne Xt := (1 ) X (t); St := (1 ) S (t): t t In terms of this notation one has from (14): Tr (AiXt) + tbi = bi; 21
m
(yt)iAi + St + tC = C:
X
i=1
Using the feasibility of Xgf and Scor it is easy to show that gf ) + tbT ycor + Tr(CXt) ? bT yt : Tr (XtScor + StXgf ) = Tr(Xgf Scor ) ? tTr(CX Substitution of bi = bi?Tr(Ai) and C = C ?I , and using Tr(Scor ) = Tr (C ? mi=1 (ycor )iAi) yields Tr (Xt Scor + StXgf ) = (1+ t)Tr(Xgf Scor ) ? tTr(Xgf + Scor ) ? tTr(C )+ Tr(CXt) ? bT yt (26) If (21) does not hold, then there exists an > 0 such that Tr(CXt) ? bT yt < ? (27) for some t which can be chosen arbitrarily large. Since Xgf and Scor were arbitrary we can assume that Tr(Xgf Scor ) < =2. Choose t such that (27) holds and tTr(Xgf Scor ) ? tTr(Xgf + Scor ) ? tTr(C ) < =2: The left hand side of (26) is always nonnegative, while the right hand side is negative for the above choice of t. This contradiction shows that if a pair (Xgf ; Scor ) exists with arbitrarily small duality gap, then (21) must hold. This completes the proof, since (Pgf ) and (Dcor ) are feasible with zero gap if and only if (D) is feasible with nite optimal value. 2 i
h
P
h
h
i
i
The next question is how to obtain the value d if it is nite. The following lemma shows that this value can be obtained from a sequence of centered iterates of the embedding as a limit value.
Lemma 8.2 Assume the optimal value of (D) to be nite, i.e. d < 1, and let Xgf be
an optimal solution of Pgf . One now has ! CX ( 1 t) T b y(t) = lim Tr d = Tr CXgf = lim t !0 t !0 (t ) (t) :
Proof: Let Xgf be any optimal solution of Pgf . (Recall that Pgf is always solvable and its optimal value equals the optimal value of (D)). Using the `subscript t' notation from the previous lemma, and the statement of the self{dual problem in (14), one can easily show that gf Tr Xgf St = Tr (CX ) ? bT yt ? tTr CX
22
or
gf Tr CXgf ? bT yt = Tr Xgf St ? tTr CX gf : Tr Xgf St + tTr CX
The second right hand side term converges to zero as t ! 0. The rst right hand side term can be made arbitrarily small, as can easily be seen from (26). This completes the proof. 2 We now show how to detect infeasibility or unboundedness of (D). Recall that if lim (t) = 0 (28) t !0 (t ) then an asymptotic ray is detected for (Dcor ) or (Pgf ). This implies weak infeasibility of either (Dcor ) or (Pgf ), and thus the status of (D) is known from Table 1. The possible combinations are listed in Table 2. It is therefore only necessary to consider the cases
() Status of (D) lim!0 Tr CX () unbounded 0 or 1 infeasible < 0 or ?1
lim!0 bT(y() ) < 0 or ?1 0 or 1
Table 2: Indicators of the status of problem (D) via the embedding of (Dcor ) and (Pgf ), for the case where lim!0 (()) = 0. In this case d cannot be nite.
where (28) does not hold. This is done in the following lemma.
Lemma 8.3 Assume that limt !0 ((tt)) = k, where 0 k < 1 in the embedding of (Pgf )
and (Dcor ). The status of (D) is now decided as follows:
() = lim Tr lim Tr CX !0 !0 () CX () = lim Tr lim Tr !0 !0 () !
!
bT y() () T b y() ()
!
!
= 1 if (D) is unbounded; = ?1 if (D) is infeasible.
Proof: Recall from Table 1 that (D) is infeasible if and only if(Pgf ) is unbounded. Let us assume that (Pgf ) is unbounded, and let K > 0 be given. By the assumption, there exists a Xgf 2 Pgf such that Tr(CXgf ) ?K . It is straightforward to derive the following 23
relation from the statement of the self{dual problem (14): 1 Tr(CX ( )) = Tr(CX ) + (t) ? (t) ? (t ) Tr(CX gf ) ? 1 Tr(S (t)Xgf ) t gf (t) (t) (t) (t) (t) gf ); ?K + ((t)) ? ((t )) ? ((t)) Tr(CX t
t
t
where we have discarded the last term (which is nonpositive) to obtain the inequality. Taking the limit as t ! 0 yields lim 1 Tr(CX (t)) ?k ? K: t !0 (t ) Since K > 0 was arbitrary, the second result follows. The case where (D) is unbounded is proved in a similar way. 2 In Table 3 the results of the lemma are summarized. The only question that cannot be
() Status of (D) lim!0 (()) lim!0 Tr CX () d < 1 0 d unbounded [0; 1) 1 infeasible [0; 1) ?1
lim!0 bT (y() ) d
1 ?1
Table 3: Indicators of the status of problem (D) via the embedding of (Dcor ) and (Pgf ), for the case where lim!0 (()) < 1.
answered by this analysis is whether or not (D) actually attains its optimal value, if it is nite.
9 Conclusion The embedding strategy yields p an -optimal solution of a given semide nite program and its Lagrangean dual in O n + 2 log(1=) iterations, provided a complementary solution pair exists. If no complementary pair exists, strong infeasibility and unboundedness are detected instead, if either occurs. The underlying assumption is that enough information concerning a maximally complementary solution of the embedding problem can be obtained from an -optimal solution. This issue is not yet satisfactorily resolved. If neither strong infeasibility nor a complementary solution pair is found, a second embedding problem can be solved using extended Lagrange-Slater dual problems instead of
24
standard (Lagrangean) duals. This embedding is used to generate sequences in terms of which weak infeasibility or a ( nite) optimal value of a given problem can be characterized. In this way infeasibility and unboundedness can be detected or the optimal value can be obtained. It is again assumed that some information concerning a maximally complementary solution of the second embedding problem can be obtained from an -optimal solution.
References [1] F. Alizadeh. Combinatorial optimization with interior point methods and semi{ de nite matrices. PhD thesis, University of Minnesota, Minneapolis, USA, 1991. [2] F. Alizadeh, J.-P.A. Haeberley, and M.L. Overton. Complementarity and nondegeneracy in semide nite programming. Working Paper, RUTCOR, Rutgers University, New Brunswick, NJ, 1995. [3] E.D. Andersen, J. Gondzio, C. Mezaros, and X. Xu. Implementation of interior-point methods for large scale linear programs. In T. Terlaky, editor, Interior point methods of mathematical programming, pages 189{252. Kluwer, Dordrecht, The Netherlands, 1996. [4] E.D. Andersen and Y. Ye. On a homogeneous algorithm for the monotone complementarity problem. Technical Report, Department of Management Sciences, University of Iowa, Iowa City, USA, 1995. [5] E. de Klerk, C. Roos, and T. Terlaky. Initialization in semide nite programming via a self{dual, skew{symmetric embedding. Technical Report 96{10, Faculty of Technical Mathematics and Computer Science, Delft University of Technology, Delft, The Netherlands, 1996. (To appear in OR Letters). [6] R.M. Freund and S. Mizuno. Interior point methods: current status and future directions. Manuscript, MIT Sloan School of Management, Cambridge, MA, USA, 1996. (To appear in OPTIMA, 19, 1996). [7] D. Goldfarb and K. Scheinberg. Interior point trajectories in semide nite programming. Working Paper, Dept. of IEOR, Columbia University, New York, NY, 1996. [8] A.J. Goldman and A.W. Tucker. Theory of linear programming. In H.W. Kuhn and A.W. Tucker, editors, Linear inequalities and related systems, Annals of Mathematical Studies, No. 38, pages 53{97. Princeton University Press, Princeton, New Jersey, 1956. [9] B. Jansen, C. Roos, and T. Terlaky. The theory of linear programming : Skew symmetric self{dual problems and the central path. Optimization, 29:225{233, 1994. 25
[10] Z.-Q. Luo, J.F. Sturm, and S. Zhang. Duality and self-duality for conic convex programming. Technical Report 9620/A, Tinbergen Institute, Erasmus University Rotterdam, 1996. [11] Y. Nesterov and A.S. Nemirovskii. Interior point polynomial algorithms in convex programming. SIAM Studies in Applied Mathematics, Vol. 13. SIAM, Philadelphia, USA, 1994. [12] G. Pataki. On the rank of extreme matrices in semide nite programming and the multiplicity of optimal eigenvalues. Technical Report MSRR-604, Carnegie Mellon University, Pitsburgh, USA, 1994. Revised Aug. 1995. [13] G. Pataki. On the facial structure of cone-lp's and semi-de nite programs. Technical Report MSRR-595, Carnegie Mellon University, Pitsburgh, USA, 1995. [14] F.A. Potra and R. Sheng. A superlinearly convergent primal{dual infeasible{interior{ point algorithm for semide nite programming. Reports on Computational Mathematics 78, Dept. of Mathematics, The University of Iowa, Iowa City, USA, 1995. [15] M. Ramana. An exact duality theory for semide nite programming and its complexity implications. DIMACS Technical Report 95{02, RUTCOR, Rutgers University, New Brunswick, New Jersey, USA, 1995. [16] M.V. Ramana and P.M. Pardalos. Semide nite programming. In T. Terlaky, editor, Interior point methods of mathematical programming, pages 369{398. Kluwer, Dordrecht, The Netherlands, 1996. [17] C. Roos, T. Terlaky, and J.-Ph. Vial. Theory and Algorithms for Linear Optimization: An interior point approach. John Wiley & Sons, New York, To appear December 1996. [18] J. Stoer and M. Wechs. Infeasible-interior-point paths for sucient linear complementarity problems and their analyticity. Manuscript, Institut fur Angewandte Mathematik und Statistik, Universitat Wurzburg, Wurzburg, Germany, 1996. [19] M.J. Todd, K.C. Toh, and R.H. Tutuncu. On the Nesterov-Todd direction in semide nite programming. Working paper, School of OR and Industrial Engineering, Cornell University, Ithaca, New York 14853{3801, 1996. [20] L. Vanderberghe and S. Boyd. Semide nite programming. SIAM Review, 38:49{95, 1996. [21] X. Xu, P.-F. Hung, and Y. Ye. A simpli ed homogeneous and self{dual linear programming algorithm and its implementation. Technical Report, Department of Mathematics, University of Iowa, Iowa City, Iowa, USA, 1994. [22] Y. Ye. Convergence behavior of central paths for convex homogeneous self-dual cones. Technical note, Dept. of Management Science, University of Iowa, Iowa City, USA, 1996. Avalable at http://www.mcs.anl.gov/home/InteriorPoint/archive.html. 26
p
[23] Y. Ye, M.J. Todd, and S. Mizuno. An O( nL){iteration homogeneous and self{dual linear programming algorithm. Mathematics of Operations Research, 19:53{67, 1994. [24] Y. Zhang. On extending primal-dual interior point algorithms from linear programming to semide nite programming. Technical Report, Department of Mathematics and Statistics, University of Maryland at Baltimore County, Baltimore, USA, 1995.
27
