ON THE IMPACT OF WEIGHTING MATRICES IN

ON THE IMPACT OF WEIGHTING MATRICES SUBSPACE ALGORITHMS

IN

D. Bauer, M. DeiEtler and W. Scher€r

Institut fb ökononet e, Operatians Researchttr.d TechnischeUntuersitätW;en,Aryentinierstr.8, Sgstemtheofie, A-1040 Wien, Austria WoAsons.Scherrcr@ tuu ien.ac.dt

Abstraci: In ihis paper someresultsconceuing the a€ymptoiicdistribution of the estimatesof th€ transfer function obiained by using subspacemethods a{€ eiv€n. The discussioncenierson the esiimationof the transferfunction rather than on ihe estimationof the systemmat c€s,sincethe iransferfunciion is a systeminvariant. Therefore the accuracyof the transfer function estimatecan be directly üsed to evaluatethe relative effici€ncyof difiereni meihods. The present paper uses the t€chniques and the a-symptotic expressions derivedin (Bauer€t ol., 1999).Inparticular the €tr€cisof certain weishtinsmatricesusedin the algorithmsare investieated. Keywords:subspacemethods,asymptoiicproperties,linear systems,state sp&e

1. INTRODUCTION Subspaceidentificationmethodsar€ med for ihe estimaiionol linear, time invariant,ffnite dimen sional state spacesystems.Many dinerent algorithms havebeenproposedin the pßt, in pariicular üoEsP(Verhegenj 1994),$asID (Va.nOverscheeand DeMoor, 1994) and ccA (Larinore, 1983).The aymptotic properti€sof subspaceestimat€s have been analyzedrecently in a couple ofpapeLs:(Baueret al., 1999;Baueränd Jansson, 2000;Deistleref ol., 1995;Jansson, 199?;Jansson and Wahlberg,1998;Petemelle, al., 1996;Viberc e, dl., i993). In (Jansson,1997)it is shown,thai the asympiotica.curacyof the lloEsP-esöinates of the pol€softhe systemdoesnot dependon a par ticular weightingmatrix used in the algorithms, on the oth€r hand, m is shown in (Bauer and t Support by the Aushian Fond. zur Förderung der wissem.haftlichenFoß.hung', Projekt P-11213-MAT i6 sraiefully acknowl€ds€d.Someof this worl hE b€€n done, while D. Bauer w6 hölding a pösldoc position at Nh€ Departnent lof Automatic cöntrol, Linköping Univeßity, Sseden,fund€d by the EU TMR proje.t'SI',

Jansson,2000),this is not true for the estimates of the transferfunction. In this paper we heavily draw on the r€sultsobtained in (Bauer, i998; Bauer et dl., 1999).Here the aim is to analyze the influ€nceof cedain weighiingmatriceson the asymptoticvarianceof ccAlN4sID iype estima[es.In particular it will be shown, that m far as the transf€r funciion esiimatesare concerned,the choiceof the 5ast, weightingmatrix does not influenceihe a"symptotic distribution. One consequence of this result is, that the numericälcomputationof the asymp iotic variarcesar€ simplified. The paper is organizedas follows: In the next sectionihe modelseiis definedand the algorithms are describedin somedetail. Also the asymptotic propertiesare stated.Seciion3 then pr€s€Dtsthe main result ol this paper, i.€. the analysisof the asymptotic expr€ssions. In section 4 the r€sults obiained axe discussed.In order to simplify the exposition,we will resti.t oürseivesio the c6e of systemswith no observedinputs. Also we will mainlydeal wiLhccA/N4sTD tlpe algofilhm,.;.e.

with subspacealgorithms, where in a ffr6t step ihe states are esiimaied, and th€n ihese state esiimatesäre usedlor the €siimationof th€ sysiem

2. MODEL SET AND ALGORITHMS In this paper finite dim€nsional,discrcte time, linea statespacesystemswithoutobservedinputs !t+r = Aq+K€.

(1J

are considered.Here (yr;t € Z) denotesthe sdimensionalobservedoutput processand (€i;l € Z) denoiess dimensionalergodic white noise of meanzeroand posiiivedeffnitevarianceO. (r,;t € Z) d€notesthe n-dimensionalstate sequence.It will be assumedihrotrghoutlhat the systemis stable, i.e. thaü l.L(.,l)l < I holdsfor all eisenvalues .\i of,,l, and that the systemis strictly minimumphäse,i.e. .L(i)l < I holdsfor all eigenvalues of ,4 = (A - IiC). Theseassumptionsin particular imply that the syst€mis in innovationform and that the innovation varianceis nonsingular.In order to guäranieeasymptotic normality of th€ parameter€stimales,we also imposesone addi-

E{e,lf,-r} = o EIEt€|\F,!j =o =tt/, E{et, *er,Jh - I = u',a," "e

n{"i.}"i':, + it, wh€re U" denot€sthe mairix, whosecolumnsare the left singular veciors corresponding ro the dominatingp singular values,the latter are the^diagonalentries in the diagonal matrix t". % corresponds^ to the respectiveright singular vecl,ors. R ac.ounl,stor l,he temaiDitrgsiogular va.lues.An estimateof rce is definedas I,I4;. re TV;lw; f '1'. Thc weighrings rv, ano rnF marflx , Arecxnarnearn mo-e detail below. For ihe molnent ii suffrcesio staie, that ? is related io basischangeand thus is requiredto be nonsingular. (3) Comprr. an pstimatFot lhe starFäs i, = frvr," ana then use the system equations to obtain estimatesof the various system matrices: Estimate d by tegression of gr onto 4'. Let d denote chis estimate, then i, u - Cir is arr estimat" of lltc ooi." Finally[/. K] is "srimatedby resressins i,+r on [tj,6jl, and the varianceo is estimat€d by the samplecovaLiance of 6r. Someimportart noiesand remarkson thesealgo rithms are siven in the following: . Th€ approximationin step 2 usestryo poe ilive definiLe weishr.ingmatrices t4lr an.l W;. It is easy to s€e, that the estimate /ro and thus the estimates(,i,ri,C,O) do not d€pend on the particular square roots (wfft'?, (w;lt'oI the6ematrices,where -- .-6 . \t' i. /t + \t1 l 2\ t" iJ/t+ \ t l z-- . . tJi t +

K, is obtained as a right factor for th€

i Ti"lli,i;,:;i;:.,:",'1T ilii ;T: this factor unique. In the ma]ysis below it qill be conrenient to use a noroaliza iion such that the first n columns of rp are equal to the identity mairix. Thus w€ 1i,s"l 1, wheres" = choosei = li/:(w;) l/", (0"x("'-")rlr. In many cas€sa difiereni normalizaiion is chosen,e.s. ? is chos€nsuch that rce =

where !i;1w;y,r, r"um"r,i;rc; 1",y,:.

at a ce ain rate dependineon the sample siz€.The inteser I > n will be chosento be fixed and finit€ for notationalsimpliciiy.Simila.rresulis would be possiblefor procedurcs allowing/ to increde to infinity asa-function ol, hc samplFsize.sFF(Baucr.1998). The following result follows from the respective resürtsin (Bauer €r dr., 1999): Theorcm1. Let f > n he a ffxed integer.Further assumeihatp is chosenI afunction ofthe sample s r / e I b u c h L h a Lp

It denotesthe sample varianceof It is easy to see from the regr€ssioneqüa tions defining,thJ estimat€s,that the estimaies (,4,Ä, C,^0)_us-ingihis normalizatio_n are relatedto (/,I{, C, O), via ,4 = 7.47-r, k = TK, ö = c" 1 and ö = ö wlere = i? 1. This meanstha.t boih €siimates "representthe samesystem. w€ will us€ the samenormalizationfor the true ma.tricesOr a.ndKp, i.€. we assumethat KpS" = J", Doing this, w€ have to a€sum€ in addition that the firct n columnsof the matrixB: Oj,Cehavefull rank n. Then by a bmis change,we can axhievethat rces" = 1" holds. In (Bauer et ol., i999), it is assumedih-ai the diagonalentriesofthe a.s.limit t" oft" a|F disi,inrr.Using'he abovenormal;?a'io rcps" = KpS" = 1r, we can replaceihis assumptionby the somewhatsimpler assump tion, ihat the ffrst n columnsof Ko havefull rank. It is easy to see [hai ihis assümption l,oldson ä seneric,et of all sysL€ms (l) oi order n. (For a proof of this statementargumenis compleielyparalellingthe onesgiv€n in (Bau€I €' or., leee) can be used). In this paperonly th€ followingchoicesofthe weighiingmairicesa.reconsidered:

real valueand po = mar{.\r(,i)l}, and suchihat p/(log")' + 0 for somed > 0. Then siven all the a.ssumpiions on ihe ergodicwhite noise,bhetrue systemand the integ€r/ mentionedabove:

(li-:);,.;=r,...r' where 1i = lEzrzi r denotes the covadancesof a process(4) wiih a rational spectra.ldensity,which nas no sp€crrarzeros. ' w;=f;landw; = I@). Here i/ denoiesthe samplevariana of Y,,+, . In a.Ucasesthe weighiingsare €ither d€t€rministic or functions of ihe sample covariancesof the observedprocess(e,). rn the following ,/J+, IIlt denotea.s. limits of the weighiings,i.e. the sample covariäncesare replacedby their population countepaxts. NoLe that ccA co-rresponds to the choice

Tbc tcsuliolTh.orm I s obtainedby lin"arizirg ihe mappingattehitrg the estimates(/,I(r Cj O) to th€ samplecovariances of the observedprocess, and by using th€ fact ihat tliese smple covarr arcesa{e a-sympiotically normal.In ihis paperthe linearizaiionof the SVD-siepis analyzedin detail, which will l€ad to someresultson ihe iduence of the w€ightingmatriceson the asymptoticdistdbution of ihe estimätes.

wi = ri , w; = ry alldN4SIDusestYr+= t61,w; =i;.

denotea generalizedleft inverseof OJ, iher

wi = ii, wf = rrn, andrilr+=

Note also,that for comktencyor asymptotic normalityth€ integerp hs to tend to inffniiy

\/rfe

lA- A.K r(.c c.a a) \ z

whereZ is ä muliiväriate Caussianrandom tar: abl€ with mean zero and vaLiancey and 4 denotesconverg€nce in distdbution. Not€, that the iheorem prescribesrates for the increaseofp, which dependson th€ true system. It has been proposedin (Petemell et dl., 1996) to usep = 2rÄrc, wherei/rc is the estimaieof the order in a lons autoresression for modelingei using AIc. This leadsto a consistenteiimate in the sense,that the restriction on p is fulfilled a.s. for ? lärge enough. Also note that similaxresults cä.nbe derivedfor the caseof additional observedinputs. For the correspondingmeihods and preliminary results see(Bauer,1998)inthe caseo{ ccalN4sID type o{ proceduresor (Bauer and Jansson,2000)for the HoESPtype of procedlles respectively.

3. THE MAIN RESULT

Lemma2. (Lieatization of weightedSVD). Lei _ '| l t. A v JI_ \ v l \ t| A

+\

]

tA

v]]

l ttrt+\ | \ rA v!\i|J ]

t k " K - \= o t t i - ß \ u - s - K - \ + o B_ B tW; _.w;|)+ \3) o ( B- B | " j

PROOF. F}om the properti$ of ihe SVD, it followsrhar rhe followilgorrlogonalirJrelations

t 16 20

v!\tt

16 1a 30.49 24.99 24_34 2r.63 t7_77 t7.t5

14.44 t7.69 15.30

J l

1a

I

h o l d . w h e r e 0 1 i s d e f i o F ds u c h r h a ' U " i , ü ;

=

lwl)-L/'Örrc,W;)tn. usinsthefactthat.p = o./,rphasrankn andthat rcpg"= ßeS"= h,l3) can be derivedhom skaightforwardcal.ulal,ions. Since th€ weightingmatricescän only influence the estimaiesof the ü.ansferfunctior via th€ esrrmäres ol Ine slates fr = ^pIr,p we may . Th€ linearizationdoesnot dependon (t7; wp ) an.t \wt - rrl r. rnrs rmpDesrhäl ih€ asymptoticdistribuiion of ih€ estimates is ih€ same,if ihe stochasiicweightingsare replacedby their asymptoticlimiis. r The lirearizarion aoe. nor d"p.nd on Wo , This impliesthat ihe asympioiicdistribution doesnot dependon ihe past w€ightingmairix. Th€seobservationslead directly to the main re-

TÄeoen ,9. Giv€nthe assumptionsofTheorem 1, then the asymptoticdistribution of the transfer fun.rion Btimal,Fsis unchangedit Wj and W; ärF renlarcd by tbeir a.s. limir.s (tor frxed / and p). Also the ma.trix w; does not influenceth€ asymptoticdistribution and thus can be set arbi t.arily. The ä€ympioticdistribution of the trarsfer tunclion esiimatedoesnoi dependon ihe dishibution of the noise. PnOOF. The firct staiem€rt follows direcily fiom equation(3). The secondstatementfollows from the fach,ihat the only sisnificantsourceof uncertaintyin equation(3) h due to B - B. This doesnot dependon ih€ distibution of the noise, but only on th€ secondmomentsfor p = p(") r co (for a proof seee.s. Lewis and Reinsel,1985). rurrhermoreler.(a,,ü,) = i';+rdtö;. rhen e.g.

e - C = {!, - i,, t,lli,, i,l I = ((6,,i,) + (,r t,,t'))(ti,tr) I ., l, = ((r'.r')

I r, ^, -e - ^ -; o ) \ ^\ .. p- . t t )- ), \\ ,! -, . 4 1 - , - , '

whichdoesnot dependor thenoisedistribution, if rcp-rcphasthesameproperty.SimilarstraishuoF sardcal.ula,ions shon'rhFsam.resulrfor ,4,Ii.

l6 t? 1a

1.92 l.ao

t.a2 1.74

t.67 1,70

Table t. In this table l,hepercentageof FLOPS a counted by üATLABneeded for rhe.alcula'ion ol rhe asymptotic rrianee using r.hefixed wcighringsas comparcdro rhe numbprof.alculationg nccded{or rh" erimared weighrings is shown.Tbe uppcrpart "hoqss . l, 'h. lowerpdt s = 2. In both casesn = 2. 4. DISCUSSIONOF THE RESULTS The main result contaiüsa numb€r oI int€Lesiing implications:Fißt, th€ result shows, that e6iimates of certain covariances, which are used for th€ weightingmatricesmighi,be e;rchangedwith th€ir r€spectiveexpectationswiihoui changing the asymptoticdistribuiion. This meansihal lhe (numerical)compütationof the dymptotic va.L ancecan be simplifiedconsiderably.In particular th€r€ is no need to compute the linearizal,ionof the Choleskyfaüorization of li./l or ti,- which is computationallyvery burdenso;e. As a sma.llexampleconsidertwo systemsof output dim€rsion one and two respectivelywith tNo states.The &tüäl valuesin the matric€sare of no imporLarcefor our discussion.In (Bauer el dJ.,1999)an algorithm for the calculationof rhe äsymptotic!arian..sis ourlinFd.The approximation of ih€ arymptotic varia.ncerequiresp to be chos€nlarge.Whenpis large,al6othe derivativeof th€ Choleskydecompositionofa large matrix has to be calculaied.In our implementationof the af gorithmsfor esiimatingihe asympioticvariarcesl usingfixed weiehtingmai.ices (and thus omiiiing the cälculationof the derivativeof the Cholesky factor) leads to the reduction in calculaiionsas givenin Table l. As can be seenfrom the results, th€ difl€rence 8€ts wors€ for increasins / and p. For s = I and tor J = p = 20 the number of flops requiredtor the calculationassumingthe weightingsto be estirheamounlwhenßun,irg ma[edisaboul,6l,imes them to be fix€d, whil€ the calculat€dasymptotic variancesdiffer by lessthar 1.7e-5. For s - 2 lhe resultsäre even more impressing:Lessthan two percenrof Lhp.alcula'ionsarF nccded.showing ihai the main part of the calculationtime is used for the dedvativ€ofthe Choleskyfador. We wani to ltLessagain,that iheseresultsonly referio our

implementalion,howeverqua.litativelythe same results will be obta.inedregardlessof the etüal implemeniation.Sincethe a.curacyofthe approx imation i6 influencedh€avily by the choiceof p, the resultsgiven in this paper 1€adeither to more a..urar" calculations ol rhe aeymproti.värian.e with [he same amount of computation time or to less computa.tionsholding the a.pproximation Second-, the resultsshowthat the weightingofthe past, % doesnot influencethe dymptotic distdbution of the estima.tes, ifihe order is specified corLectly.Therefore,in order to obiain optimal estimates, one only has to choosean opiima.l fir trte weighting'vr-. l his migbcbe of rteoretical inierest, since no analyiica] results on the optimal choiceof this mafrix exht, to the best of ih€ authors knowledge.Theorem 3 restricts the possibledesignparametersto the choiceof I and rne wergnung marnx wJ , However,the authoß want to emphasize,that the opiimization of the asympiotic accuracyof ihe transfer lunction estimatesis not th€ only meäsurefor ihe optim..lity of a. procedure.Also the questionof ihe approximationprope*ies lor misspecificatione.g. of the order might be oI irterest. In this -caseone cannot simply neglect the influenc€of Iy;, se€e.s. (Bauer e, oi., i998). Ii shouldbe remarked,thai ihe aboveresultsmay be generalized in mä.ny r€spects. lf a ditrerent normalizationfor the fetorization of ti/+\-rl2it \. t !

; r7tt i/- \-Il2

is_chosen,_ then the asymptoiic distribution of (,4,I1,C,o) rnisht dependon t4lt. Howeverthe asymptoticdistdbution of ary system invadant (e.9.the pol€sof th€ system,or the transferfunc iion) compuied from these estimaies,does_noi dependo^nthe^pästweighting,since (A,I{,C,A) and (,4,-K,C, o) descdb€the samesyst€m. The fact, that the a.symptoticdistribütion does noi dependon the distdbution of the noise sequence,a.ddsva]ue to the previously published plots, e.g. in (Bauer, 1998; Ba.uer et ol., 1998; Deistl€r€, al., 1995).Ii also is an importart ro Finally we want io note that the cäseof additional observ€dinpüts can be handledin a.similar fashion. For the ccAlN4SIDtype of procedures the re-sultsar€ analogously, i.e. th€ weishtingmatrix l{/t does not iEflu€ncethe asymptoticdistribution and ihe asymptoticdistdbution is not chansed,if weighiingsdtimated from data are repläcedby the^ireJ.pectations. For th€ fioEsPtype olalgorirhm.OJ i, us€oLoesrimäterhc marri."s n änd a. Thus by reversingth" rol"sofOl md ,ro in Lemmä3,onecancorclud€tha.tthe asymptotic

distribution of Oj and thts of the estimateol,4 ana C (if0J is normalizedsuitably) do noi depend on the weightineIii. Carryins theseresults overto ihe systempoles(i.e. otthe eisenvalues of A) we obtain ihe resuli siven in (Jansson,1997). How€verin this caseit is shown in (Bauer and Jansson,2000),that ihe äsymptoticdist bution ofthe transferfunction estimatemight dependon both weiehtinss.

5. CONCLUSIONS In this paper ihe eflectof th€ choiceofth€ weiehiing matriceson ihe asymptoticdistributionofthe trarsfer function estimatesobtained by subspace procedureshasb€eninvestigated.The main result states, that the asymptotic distribuiion is not chansed,$'hen weightings,which are esiimated from ihe data, are replacedby ihe corresponditrg €xpectations.As a. byproduct we also obtain the r€suli, thai the aßymptotic distribution of ihe €stimatedtransfer tunciion does not dependon the distdbuiion of ihe nois€.The implication of the main result is that ihe calculation of ihe asymptoticvarianceca.nbe significanilyspeeded up. lt iB a.lsoshown, that one of the weighiing matricesin the ccAlN4sIDtype ofproceduredoes noi influenceth€ asymptoticdistributionand thüs can be set arbitraxilyrwhile the oth€r weighting matdx influencesthe a€ymptotic a.ccuracy. The opiimal choiceol this matrix is still an openques tior. Fina.llywe want io emphäsize, that the whole discussionreferredto the c6e, wh€rethe order is specifiedcorrectly.Inthe cme,wherea lowerorder is specifi€dboth weighiingmatricesinflüenc€th€ asymptoticbias and thus have to be considered, when searchingfor an optimal approximation.

6. RDFERENCES Bauer, D. (1998).Some Asymptoiic Theory for hheEsiimationof LinearSystemsUsingMax, imum LikelihoodMethodsor SubspaceAleorithms. PhD thesis.TU Wi€r. Bauer, D. and M. Jansson(2000).Analysisof ihe asymptotic properiiesof the moesp type of subspacea.lgorithms.To appearin Attomotrcd,Apdl 2000. Bauer, D., M. Deistler and w. Schener (1998). User choicesin subspacealsorithms.In: Prr ceedinssol the CDC'98 Confercnce.'Iampa., Florida.pp.731-736. Ba.u€r,D., M. Dehiler and w. Schener (1999). Consistencyand ä"symptoticnormality of somesübspacea.lgorithmsfor systemswithout observedlnpnts. Autonottca 35, 7243 t254.

Deistler,M., K. Peternelland W. Scherrer(1995). Consistency and relative efficiency of suF spac€methods.,4utornattcd31, 1865-1875. Jansson,M. (199?).On Subspace Methodsin Sys t€m Identifica.tion and Sensor Array Signal Proc€ssing. PhD thesis.KTH, Stockholm. Jänsson,M. and B. wahlberg (1998).On consrstency of sübspacemethods for system identrfi.ation. Automotico34(12), 1507 1519. LatimoF.w. E. (1983).SysL"mideni.ificatio!. kducedorderffließ ad modelingviacanonical variate ajra.lysis.ln. Proc, 1983 Amer, Cantrcl Conlerence2. (H. S. R.aoand P. Dorato, Eds.).Phcataway,NJ. pp.445 451, IEEE ServiceCenter. Lewis, R. and G. R€iNel (1985). Prediction of multivariat€ time series by autoregressrve model fitting. Joutnal ol Mtitiüatidte AnaIvits 16,393-411. Peiemell,K., W. Scherrerand M. Deistler(1996). Statistical analysis of novel subspaceidentiffcation methods. Sisnal Proces'ins 52, 161t77. van Overschee, P. and B. DeMoor (1994).N4sid: Subspace algorithms for the identification of combineddeterministic-stochastic systems. Artomotico 30,75 93. Verhaegen,M. (199a). Identifica.iior of the det€rministic pari of mimo stä.tespacemodels given in innovations form from input oütput dara..Aüomatica sn\I), 61 74. Viberg, M., B. Otiersten, B. Wahlberg and L. Ljuns (1993). Performarce of subspace bas€dstate spacesystem identificaiion methods. In: Proc. of the 12th IFAC Wotld Cangras.Vol. 7. Sydney,Australia.pp. 369 372.