iqml-like algorithm and inverse iteration algorithm

0 downloads 0 Views 302KB Size Report
ing the DTLS nonlinear singular value problem by simu- ... It is well known that the Total Least Squares (TLS) method is applied when observation ... b(z. −1)uk. (1) where yk is the system output at time instant k, uk is the system input, z. −1.
IQML-LIKE ALGORITHM AND INVERSE ITERATION ALGORITHM IN DYNAMIC SYSTEM IDENTIFICATION Hong. Yao, Masato. Ikenoue, Shunshoku. Kanae, Zi-Jiang. Yang and Kiyoshi. Wada Department of Information Science and Electrical Engineering Kyushu University 6-10-1, Hakozaki, Higashi-ku, Fukuoka 812-8581 Japan email: yao, ike, jin, yoh, [email protected] ABSTRACT The Structured Total Least Squares (STLS) problem is an extension of the Total Least Squares (TLS) problem for solving an overdetermined system of equations Ax ≈ b1) . The Dynamic Total Least Squares (DTLS) problem is a special case of STLS problem. In this paper, we illustrate the TLS singular value problem by defining some vectors from a new viewpoint. And we also derive the DTLS nonlinear singular value problem by this new viewpoint. Futhermore, we introduce a heuristic Iterative Quadratic Maximum Likelihood (IQML)-like algorithm for solving DTLS nonlinear problem. The main objective of this paper is to compare the difference between this variety of IQMLlike algorithm and the Inverse Iteration algorithm for solving the DTLS nonlinear singular value problem by simulation experiments with some conditions. The simulation results show that the convergence rate of IQML-like algorithm is faster than that of Inverse Iteration algorithm under the same conditions and the computational efficiency of the IQML-like algorithm is also higher compared with the Inverse Iteration algorithm.

In recent years, many different formulations have been proposed for the STLS problem: the Constrained Total Least Squares (CTLS) approach 2),3), the Structured Total Least Norm (STLN) approach 4),5), the Riemannian Singular Value Decomposition (RiSVD) approach 6),7) and the Bootstrapped Total Least Squares (BTLS) approach 8) and so on. All these approaches start more or less from a similar formulation, but the final formulation for which an algorithm is developed might be quite different. The Iterative Quadratic Maximum Likelihood (IQML)-like algorithm for solving STLS problem can be considered originating from the CTLS, RiSVD and BTLS framework. What we introduce this variety of IQML-like algorithm is basically from the IQML algorithm. The IQML algorithm was initially9) designed for estimating the parameters of superimposed complex damped exponentials in noise. The STLS problem finally leads to a nonlinear generalized singular value decomposition, which can be solved via an algorithm that is inspired by inverse iteration. Therefore the Inverse Iteration algorithm is usually considered the classical algorithm for solving the STLS problem. In this paper, we will illustrate the TLS singular value problem and the DTLS nonlinear singular value problem from a new viewpoint. We also will introduce a heuristic IQML-like algorithm, which is based on the Least Squares (LS) method for solving the DTLS nonlinear singular value problem. And we will discuss the Inverse Iteration algorithm and this variety of IQML-like algorithm for solving the DTLS nonlinear singular value problem under the same conditions. Moreover, we will compare the properties of this variety of IQML-like algorithm and Inverse Iteration algorithm by simulation experiments with some conditions.

KEY WORDS system identification, parameter estimation, total least squares, structured total least squares, dynamic total least squares

1 Introduction It is well known that the Total Least Squares (TLS) method is applied when observation noises exists in the input and the output at the same time, and the TLS approach is a popular method in data fitting problems where we have to solve an overdetermined system of linear equations Ax ≈ b. But in many cases the extended data matrix [A b] has a special structure(Hankel,Toeplitz,...), in this case, the TLS problem is what we called the Structured Total Least Squares (STLS) problem. The STLS problem is an extension of the ordinary TLS problem, that is the problem of approximating affinely structured matrices by similarly structured rank-deficient ones, while minimizing an L2 -error criterion. The Dynamic Total Least Squares (DTLS) problem is a special case of STLS problem.

479-092

2 Statement of the problem Consider the problem of estimating the parameters of system described by a(z −1 )yk

=

b(z −1 )uk

(1)

where yk is the system output at time instant k, uk is the system input, z −1 is the backward shift operator, i.e. z −1 uk = uk−1 , and a(z −1 ) and b(z −1 ) are polynomials in

204

z −1 of the form

5. (N + 1) × (n + 1) the matrix Y and U :

a(z −1 ) = b(z

−1

a0 + a1 z −1 + . . . + an z −n

) =

b0 + b1 z

−1

+ . . . + bn z

−n

.

(2)

= yk + ek = uk + dk

Ya

a(z −1 )ek − b(z −1 )dk

= [w0 , w1 , · · · , wN ]T = [z0 , z1 , · · · , zN ]T

(4)

Y2 a U2 b

(6) (7)

= [y0 , y1 , · · · , yN ]

(8)

u

= [u0 , u1 , · · · , uN ]T

(9)

(15)

b(S)u.

(16)

= =

A2 y B2 u.

(17) (18)

Similarly, we furthermore partition Z and W ⎤ ⎡ }n + 1 Z1 = [z Sz · · · S n z] (19) Z = ⎣ ... ⎦ Z2 }N − n ⎡ ⎤ W1 }n + 1 W = ⎣ ... ⎦ = [w Sw · · · S n w](20) W2 }N − n

(5)

y

= a0 y + a1 Sy + · · · + an S n y

we can obtain

we also can get the following results.

2. The input vector u and output vector y : T

(14)

⎤ ⎤ ⎡ }n + 1 }n + 1 B1 A1 , b(S) = ⎣ . . . ⎦ a(S) = ⎣ . . . ⎦ A2 B2 }N − n }N − n

To clarify the problem setting, some vectors are defined here. 1.The observed input vector w and observed output vector z : w z

[u Su · · · S u]



where vk is a composite noise term defined by =

=

Therefore, if we partition Y , U and a(S), b(S) into two parts, ⎡ ⎤ ⎤ ⎡ Y1 }n + 1 }n + 1 U1 Y = ⎣ ... ⎦ , U = ⎣ ... ⎦ Y2 U2 }N − n }N − n

where δij is Kronecker delta. It is also assumed that uk is a zero mean stationary random process with finite variance, and is independent of the measurement noise ek and dk . Our goal is to identify the system parameters from the noisy input and output. Substituting (3) into (1) and rearranging the result give the relationship between the observed input wk and the observed output zk

vk

U

Ub =

E[ei ej ] = σe2 δij , E[di dj ] = σd2 δij E[ei dj ] = 0

b(z −1 )wk + vk

(13)

n

Again, from (11), (14), we also can obtain

E[ek ] = 0, E[dk ] = 0

=

[y Sy · · · S n y]

= a(S)y

(3)

where it is assumed that ek and dk are both measurement noise with statistics

a(z −1 )zk

=

Here, N is the data length and n is the system order. From (10), (13), we have

Let zk and wk be the noise-corrupted measurements of yk and uk , respectively, i.e zk wk

Y

Z2 a = A2 z

(21)

W2 b = B2 w

(22)

In the next section, we will illustrate the generalized singular value problem of TLS method with the restriction.

3. The shift matrix S : ⎤ ⎡ 0 ⎥ ⎢ 1 0 ⎥ ⎢ S=⎢ ⎥ ; (N + 1) × (N + 1) . . .. .. ⎦ ⎣ 1 0

3 TLS problem Here, a new viewpoint is adopted to solve the TLS problem by (15), (16), (17), (18), (21), (22). At this time the TLS problem can be considered as a minimization problem of     −Z2 : W2 − −Y2 : U2 2 (23) F

4. The system parameters : a = b =

[a0 , a1 , · · · , an ]T [b0 , b1 , · · · , bn ]T

(10) (11)

ϕ =

[aT , bT ]T

(12)

with the constraint  −Y2

205

: U2



ϕ =

0.

(24)

Where  · F denotes the Frobenius norm. When the restriction of ϕT ϕ = 1 is decided to be assigned to ϕ, then the TLS criterion function is  2 1  J T LS =  −Z2 : W2 − −Y2 : U2 F 2  λ T −Y2 : U2 ϕ + (1 − ϕT ϕ). (25) + l 2

4 DTLS problem DTLS estimation is a method for estimating the system parameters by minimizing the following criterion function 1 1 (z − y)T (z − y) + (w − u)T (w − u) 2 2  λ + lT −Y2 : U2 ϕ + (1 − ϕT ϕ). (40) 2

J DT LS =

Where l and λ are Lagrangean multipliers. Setting all derivatives to zero results in the set of equations   −Z2 : W2 − −Y2 : U2 = lϕT (26)  = λϕT (27) lT −Y2 : U2  −Y2 : U2 ϕ = 0 (28) ϕT ϕ = If we postmultiply (27) by ϕ, we have  lT −Y2 : U2 ϕ = λϕT ϕ = λ From (28), we obtain λ = 0. Therefore  lT −Y2 : U2 =

0.

1.

Setting all derivatives equal to zero results in the set of equations



z−y −AT2 l (41) = B2T w−u

−Y2T l = λϕ (42) U2T  −Y2 : U2 ϕ = 0 (43)

(29)

ϕT ϕ =

(30)

From (42), (27), we have  lT −Y2 : U2 ϕ =

(31)

Similarly, if we postmultiply (26) by ϕ, from (31), we obtain the following equation

  −Z2 : W2 − −Y2 : U2 ϕ  T (32) = −Z2 : W2 ϕ = lϕ ϕ = l.

W2



=

σ 2 xT xϕ.

λϕT ϕ = λ.

(45)

(46)

From (17), (18), (21), (22), we can get the following equations, (Z2 − Y2 )a = (W2 − U2 )b =

A2 (z − y) B2 (w − u),

Using (41) and (43), then we find  −Z2 : W2 ϕ 

 −Z2 : W2 − −Y2 = = −(Z2 − Y2 )a + (W2 − U2 )b

(34) (35)

Here, if we define x = l/l, l = σ, (34) can be written as  −Z2 : W2 ϕ = σx. (36) (35) can be written as  σxT −Z2 :

(44)

From (43), we can obtain λ = 0. Therefore

−Y2 l = 0. U2

If we premultiply (26) by lT , from (31), we obtain the following equation

  −Z2 : W2 − −Y2 : U2 lT  (33) = lT −Z2 : W2 = lT lϕ. Finally, we have the following equations  −Z2 : W2 ϕ = l  lT −Z2 : W2 = lT lϕ.

1.

: U2

= −A2 (z − y) + B2 (w − u) = A2 AT2 l + B2 B2T l.

(47) (48)



ϕ

(49)

If we denote Da = A2 AT2 , Db = B2 B2T , we can obtain  −Z2 : W2 ϕ = (Da + Db )l. (50)

(37)

If we define the matrix L,

0 0 0 T T T n = , S L , · · · , (S ) l l l (51)

If we arrange them in order, we can get the following equations  −Z2 : W2 ϕ = σx, ϕT ϕ = 1 (38)

T −Z2 x = σϕ, xT x = 1. (39) W2T

we can get the set of equations

Obviously, ϕ is the singular vector, to  which corresponds the singular value σ of the matrix −Z2 : W2 . From these, we can see the TLS singular varlue problem derivation is straightforward. In the next section, we will illustrate the DTLS nonlinear singular value problem.

T

L a

=

T

a(S)

LT b =

206

b(S)T

0 l 0 l



= AT2 l

(52)

= B2T l.

(53)

From (41), (52) and (53), we can obtain z−y = w−u =

−AT2 l = −LT a B2T l = LT b.

where, Q1 ∈ Rp×q , Q2 ∈ Rp×(p−q) , R ∈ Rq×q . Second, we decompose u as u = Q1 z + Q2 w for certain vectors z ∈ Rq and w ∈ Rp−q . From (61), (62), we easily find that ⎤⎡ ⎤ ⎡ ⎤ ⎡ z Du v 0 0 RT ⎣ QT2 Dv Q1 QT2 Dv Q2 0 ⎦⎣ w ⎦ = ⎣ 0 ⎦. T T v 0 Q1 Dv Q1 Q1 Dv Q2 −R (64)

(54) (55)

Moreover, if we give the following equations Z −Y W −U

=

[(z − y) S(z − y) · · · S n (z − y)]

=

(56) [(w − u) S(w − u) · · · S (w − u)] . n

The solution of the system linear equation (z, w, and v) is easy because of the block triangular structure of the system matrix. u is computed from u = Q1 z +Q2w, which is then used to update Du and Dv . The Inverse Iteration algorithm is summarized as follows : Initialization : Choose u[0] , v [0] , τ [0] . Construct [0] [0] Du , Dv and normalize such that

(57) we can write  T T 0 l (Z − Y ) = (z − y)T LT  T T l 0 (W − U ) = (w − u)T LT .

(58) (59)

Hence, using (58), (59), we obtain  T T  l 0 −(Z − Y ) : W − U  −(z − y)T LT : (w − u)T LT =  T a LLT : bT LLT =

LLT 0 T (60) = ϕ 0 LLT

(v[0] )T Du[0] v [0] = (u[0] )T Dv[0] u[0] = 1 For k = 1 till convergence : 1. 2. 3. 4. 5. 6. 7. 8. 9.

Here, we define ϕ = v/v, x = l/l, l = σ, x = u/u, σ = τ u·v, Dl = LLT , Dv = A2 AT2 +B2 B2T , Du = uuT , then (50), (60) can be written as  −Z2 : W2 v = τ Dv u, uT Dv u = 1 (61)



Du 0 −Z2T u = τ v, v T Du v = 1. 0 Du W2T (62)

z [k] w[k] u[k] v [k] v [k] γ [k] u[k] [k] Du τ [k]

= = = = = = = = =

[k−1]

R−T Du v [k−1] [k−1] [k−1] −(QT2 Dv Q2 )−1 (QT2 Dv Q1 )z [k] [k] [k] Q1 z + Q2 w [k−1] [k] R−1 QT1 Dv u [k] [k] v /v  [k−1] [k] 1/4 ((u[k] )T Dv u ) [k] [k] u /γ ; v [k] = v [k] /γ [k] [k] [k] [k] Du /(γ [k] )2 ; Dv = Dv /(γ [k] )2 (u[k] )T Av[k]

 −Y2 : U2 and its Convergence test: Calculate [k] [k] largest and smallest singular value β1 and βq . If [k] [k] βq /β1 < m , go to 1, else stop, where m is the machine precision7). This algorithm as presented above is far from efficient. For instance, if p is much larger than q, the inversion of the (p − q) × (p − q) matrix (Qt2 Dv Q2 ) requires a lot of work 7) . The convergence of this algorithm has no formal been proved, we will demonstrate it by simulation experiments.

For (61), (62), it can be considered as the DTLS nonlinear singular value problem. Next, we will show two algorithms for solving the DTLS nonlinear singular value problem.

5 Inverse Iteration algorithm Here, we assume that Du and Dv would be constant matrices independent of u and v, then the minimal eigenvalue could be computed with inverse iteration. Next, we will do : For given Du and Dv , we perform one step of inverse iteration, using the QR-decomposition of the matrix A, in order to obtain new estimates of u and v. These are then used in updating Du and Dv , etc. In what follows, the iteration number will be indexed between square brackets. For the results upon convergence, we will use the index is used for  ∞. First, the QR-decomposition the matrix −Z2 : W2 .

 R  −Z2 : W2 = Q1 Q2 A = O (63)

6 IQML-like algorithm In this section we introduce a heuristic IQML-like algorithm for solving (61), (62). The principle of this variety of IQML-like algorithm is based on the Least Squares (LS) method. Next, we discuss it. Frist, we define two vectors : z2 = ϕ = Here, ϕˆ = θa0 .

207

[zn+1 , zn+2 , · · · , zN ]T ˆT [a0 , ϕ]

(65) (66)

Then, we denote  −Z2 : W2 =



−z2

: Ω



Initialization : Choose θˆ[0] , ϕ[0] . .

From (61), (62), we can write  −Z2 : W2 ϕ = (A2 AT2 + B2 B2T )l = Dv l



−Z2T LLT 0 l = ϕ = Dl ϕ. W2T 0 LLT From (68), Dv−1



−Z2



: W2

ϕ =

l

(67)

(69)

(70)

1. θˆ[k] [k]T 2. ϕ

ε ε

= =

G(z) =

T

(74)

Here, we use the Cholesky decomposition of Dv−1 , Dv−1 = GGT .

(75)

Similarly, from (66), (67), (75), we minimize (72), we have

  −z2T a0 T a0 ϕˆ −z Ω εT ε = GG 2 ΩT ϕˆ = (−a0 z2T + ϕˆT ΩT )GGT (−a0 z2 + Ωϕ) ˆ = [(z2T − θT ΩT )Ga0 ][a0 GT (z2 − Ωθ)]

(76)

therefore, we also can obtain θˆ =

(ΩT GGT Ω)−1 (ΩT GGT z2 )

=

(ΩT Dv−1 Ω)−1 (ΩT Dv−1 z2 ).

(77)

Next, we will use (77) to discuss this variety of IQML-like algorithm. To establish initial value, value  we use the singular decomposition of the matrix −Z2 : W2 . Therefore, we obtain  [U S V ] = SV D( −Z2 : W2 ) v

0.169901z −1 + 0.143831z −2 . (78) 1 − 1.575157z −1 + 0.606531z −2

The true input uk is white noise with zero mean and variance 1. The variances σe2 , σd2 of the measurement noise ek ,dk are from 0.01 to 0.05 respectively. We take the data length N = 100, and perform over 5 simulation experiments with LS method, TLS method, IQML-like algorithm and Inverse Iteration algorithm. The simulation results are shown in table 1. Table 1 shows the estimator accuracy of four estimation methods with ϕ−ϕ/ϕ(where ˆ ϕ stands for the parameters true value, ϕˆ stands for the parameters estimated value). And it also shows the iteration times (k) of this variety of IQML-like algorithm and the Inverse Iteration algorithm till convergence. From table 1, we can see the estimator accuracy of the Inverse Iteration algorithm and this variety of IQMLlike algorithm outperform by far LS estimator and TLS estimator. Again, the TLS estimator accuracy is better than LS estimator. From table 1, we also can confirm that the convergence rate of this variety of IQML-like algorithm is faster than one of Inverse Iteration algorithm under the same conditions.

Therefore, from the normal equation of LS estimation method, we can obtain θˆ = (ΩT XX T Ω)−1 (ΩT XX T z2 )

z2 )

Consider the following 2nd order system:

(X z2 − X Ωθ) (X z2 − X Ωθ) (73) (z2 − Ωθ)T XX T (z2 − Ωθ). T

[k−1] −1

7 Simulation

Where, ε is prediction error, e and d are both measurement noise. If, we denote ε = X T (z2 − Ωθ), then we have T

[k−1] −1

(ΩT Dv Ω)−1 (ΩT Dv [k]T [1, θˆ ]/[1, θˆ[k]T ]

= =

Convergence test: If θˆ[k] − θˆ[k−1]  < m , go to 1, else stop. Here, we can see that this variety of IQML-like algorithms is very easy in calculation. Next, we will compare the properties of two algorithms by simulation experiments.

= ϕT Dl ϕ = lT Dv l = εT ε = eeT + ddT . (72)

T

=

v(2 : 2(n + 1))/v(1) [1; θˆ[0] ]/[1; θˆ[0] ]

Construct Dˆv : [0] [0] [0]T [0] [0]T Dˆv = Aˆ2 Aˆ2 + Bˆ2 Bˆ2 This variety of IQML-like algorithm is summarized as follows : For k = 1 till convergence :

(68)

Then, we premultiply (71) by ϕT , we obtain

 −Z2T T ϕ Dv−1 −Z2 : W2 ϕ W2T

T

=

[0]

ϕ

Substituting (70) into (69) and rearranging the result yield



 −Z2T −Z2T l = Dv−1 −Z2 : W2 ϕ T T W2 W2 = Dl ϕ. (71)

T

θˆ[0]

8 Conclusion In this paper, we have illustrated the TLS singular value problem and the DTLS nonlinear singular value problem from a new viewpoint. And we have shown that the Inverse Iteration algorithm and this variety of IQML-like algorithm for solving the DTLS nonlinear singular value problem. Through the simulation results, we confirm the

= V (:, 2(n + 1)).

208

Table 1. The comparison of LS method, TLS method, Inverse Iteration algorithm and IQML-like algorithm N oise variance(σ2 )

Least Squares ϕ ˆ − ϕ/ϕ

T otal Least Squares ϕ ˆ − ϕ/ϕ

1 σd2 = 0.01, σe2 = 0.05

0.1884

0.0595

2 σd2 = 0.02, σe2 = 0.05

0.1889

0.0146

3 σd2 = 0.03, σe2 = 0.05

0.3183

0.0594

4 σd2 = 0.04, σe2 = 0.05

0.3026

0.0549

5 σd2 = 0.05, σe2 = 0.05

0.4092

0.1590

6 σd2 = 0.01, σe2 = 0.03

0.1648

0.0241

7 σd2 = 0.02, σe2 = 0.03

0.1864

0.0047

8 σd2 = 0.03, σe2 = 0.03

0.2910

0.0264

9 σd2 = 0.01, σe2 = 0.01

0.0893

0.0533

10 σd2 = 0.01, σe2 = 0.02

0.1632

0.0110

Inverse Iteration ϕ ˆ − ϕ/ϕ iteration times 0.0290 k = 22 0.0464 k = 25 0.0483 k = 35 0.0399 k = 61 0.0397 k = 50 0.0219 k = 23 0.0251 k = 23 0.0240 k = 40 0.0158 k = 21 0.0036 k = 20

IQM L − like ϕ ˆ − ϕ/ϕ iteration times 0.0309 k = 10 0.0379 k = 12 0.0313 k=9 0.0363 k = 12 0.0528 k = 12 0.0380 k=9 0.0119 k = 13 0.0217 k=9 0.0288 k=6 0.0069 k=8

convergence rate of this variety of IQML-like algorithm is faster than one of Inverse Iteration algorithm under the same conditions.

[8] H. Van hamme : Identification of linear system from time-or frequency-domain measurements ; PhD thesis, Vrije Universiteit Brussel, Dept, Electrical Engineering, ELEC, 1992

References

[9] Y. Bresler and A. Macovski : Exact maximum likelihood paramaeter estimation of superimosed exponential signals in noise ; IEEE Transactions on Signal Processing, 34(5):1081-1089, October 1986

[1] Lemmerling P., Vanhanmme L., V.Huffel S., De Moor B.: IQML-like algorithms for solving Structured Total Least Squares problem ; Internal Report 00-20, ESAT-SISTA, K.U.Leuven, Leuven, Belgium, 2000 [2] T. J. Abatzoglou and J. M. Mendel : Constrained total least squares; In Proceedings of IEEE International Conf. on Acoustics, Speech Signal Processing, pages 1485-1488, Dallas, 1987 [3] T. J. Abatzoglou and J. M. Mendel, and G. A. Harada : The constrained total least squares technique and its application to harmonic superresoultion; IEEE Transactions on Signal Processing , 39:1070-1086, 1991 [4] J. B. Rosen, H. Park, and J. Glick: Total least norm formulation and solution for structured problems; SIAM Journal on Matrix Anal, Appl, 17(1): 110-128, 1996 [5] S. Van Huffel, H. Park, and J. B. Rosen : Formulation and solution of structured total least norm problems for parameter estimation ;IEEE Trans on Signal Processing,44(10):2464-2474, 1996 [6] B.D.Moor: Structured Total Least Squares and L2 Approximaton Problem ; Linear Algebra and its Applications, Vol.188-189, pp.163-207, July 1993 [7] B.D.Moor:Total Least Squares for Affinely Stuctured Matrices and Noisy Realization Problem ;IEEE Trans. Signal Processing, Vol.42, 3104/3113, (1994)

209