Natural Gradient Descent for Training Multi-Layer Perceptrons Howard Hua Yang and Shun-ichi Amari Lab. for Information Representation, FRP, RIKEN Hirosawa 2-1, Wako-shi, Saitama 351-01, JAPAN FAX: +81 48462 4633 E-mails:
[email protected],
[email protected] Submitted to IEEE Tr. on Neural Networks May 16, 1996
Abstract
The main diculty in implementing the natural gradient learning rule is to compute the inverse of the Fisher information matrix when the input dimension is large. We have found a new scheme to represent the Fisher information matrix. Based on this scheme, we have designed an algorithm to compute the inverse of the Fisher information matrix. When the input dimension n is much larger than the number of hidden neurons, the complexity of this algorithm is of order O(n2 ) while the complexity of conventional algorithms for the same purpose is of order O(n3 ). The simulation has con rmed the ecience and robustness of the natural gradient learning rule.
1 Introduction Inversion of the Fisher information matrix is required to obtain the Cramer-Rao lower bound which is fundamental for analyzing the performance of an unbiased estimator. It is also needed in the natural gradient learning framework [2, 3] to design statistically ecient learning algorithms for parameter estimation in general and for training neural networks in particular. In this paper, we assume a stochastic model for multi-layer perceptrons. Considering the parameter space as a Riemannian space with the Fisher information matrix as its metric, we apply the natural gradient learning rule to train a multi-layer perceptron. The main diculty encountered is to compute the inverse of the Fisher information matrix of a large size when the input dimension is large. By exploring the structure of the Fisher information matrix and its inverse, we shall design a fast algorithm with lower complexity to implement the natural gradient learning algorithm. The outline of the paper is as follows. The natural gradient descent method and related problems for training a multi-layer perceptron are described in Section 2. The Fisher information matrix and the scheme to represent this matrix are given in Section 3. The main focus of this paper is to present a fast algorithm to compute the inverse of the Fisher information matrix. Several issues related to this are discussed in Section 4: a group structure for the blocks in the Fisher information matrix is found in Section 4.1. Based on this nding the inverse of the Fisher information matrix is computed in Section 4.2-4.3; the complexity of our algorithm in computing the inverse of the Fisher information matrix is analyzed in Section 4.4; by the 1
result on the group structure of the Fisher information matrix, the Cramer-Rao lower bound for the committee machine is explicitly given in Section 4.5. The natural gradient vector eld is compared with the ordinary vector eld in Section 5 via an example to illustrate the e ects of the natural gradient learning in training multi-layer perceptrons. The e ectiveness of the natural gradient is further demonstrated by some simulation results in Section 6.
2 Natural Gradient Descent Method and Its Eciency 2.1 Model of a stochastic multi-layer perceptron
Assume the following model of a stochastic multi-layer perceptron:
z=
m X i=1
ai '(wTi x + bi ) +
(1)
where ()T denotes the transpose, N (0; 2 ) is a Gaussian random variable, and '(x) is a di erentiable output function of hidden neurons such as hyperbolic tangent. Assume that the multi-layer network has a n-dimensional input, m hidden neurons, a one dimensional output, and m n. Denote a = (a1 ; ; am )T the weight vector of the output neuron, wi = (w1i ; ; wni )T the weight vector of the i-th hidden neuron, and b = (b1 ; ; bm )T is the vector of thresholds for the hidden neurons. Let W = [w1 ; ; wm ] be a matrix formed by column weight vectors wi , then (1) can be rewritten as
z = aT '(W T x + b) + :
Here, the scalar function ' operates on each component of the vector W T x + b and this convention will be used throughout this paper. By this convention, the column vector ['(wT1 x + b1 ); ; '(wTm x + bm )]T
is written as '(W T x + b) and the row vector
['(wT1 x + b1 ); ; '(w Tm x + bm )]
is written as '(xT W + bT ). In this paper, we consider the realizable learning problem. Let
= (wT ; ; wTm; aT ; bT )T 1
denote the parameter vector for the student network and
= (wT ; ; wmT ; aT ; bT )T 1
for the teacher network. The problem is to estimate based on training examples
DT = f(xt ; zt ); t = 1; ; T g:
The training examples as well as the future examples for evaluating the generalization error are generated by two steps. First, xt is randomly generated subject to an unknown probability 2
density function p(x). Second, the corresponding output is generated by the teacher network as zt = aT '(W T xt + b ) + t where t N (0; 2 ). Note the variance of the additive noise t is the same as the variance of the additive noise in the stochastic perceptron model. The input-output pairs f(xt ; zt ); t > T g are the future examples. De ne the generalization error Eg (; ) = E [(z ? aT '(W T x + b))2 ] where the expectation E [] is taken with respect to the joint probability density function p(x; z; ). The purpose of learning is to minimize the generalization error but it cannot be easily evaluated in practice. Instead of the generalization error, we use the training error
Etr (; DT ) = T1
T X t=1
fzt ? aT '(W T xt + b)g
2
to derive a learning algorithm. Minimizing the training error is not equivalent to minimizing the generalization error in general (see, for example, Amari and Murata, 1993)[4]. Some auxiliary methods such as early stopping and regularization are necessary to avoid overtraining or over tting (see, for example, Ripley, 1996 [10]; Amari et al, 1997 [5]). The overtraining problem is important but it will not be discussed here. We shall only focus the learning algorithm for minimizing the training error.
2.2 Fisher information matrix
The joint probability density function (pdf) of the input and the output is
p(x; z; W ; a; b) = p(zjx; W ; a; b)p(x) where p(z jx; W ; a; b) is the conditional pdf of z when x is input. The loss function is de ned as the negative log-likelihood function
L(x; z; ) = ? log p(x; z; ) = l(zjx; ) ? log p(x) where
l(zjx; ) = ? log p(zjx; ) = 21 2 (z ? aT '(W T x + b))2 is also a loss function equivalent to L(x; z ; ) since p(x) does not depend on . Hence, given the training set DT , minimizing the loss function L(x; z ; ) is equivalent to minimizing the training error. Since @@L = @@l , the Fisher information matrix is de ned by
@L )T ] = E [ @l ( @l )T ] ( (2) G = G() = E [ @L @ @ @ @ where E [] denotes the expectation with respect to p(x; z ; ). The inverse of the Fisher information matrix is often used in performance analysis, because the Cramer-Rao states that an unbiased estimator b of the true parameter satis es E [kb ? k ] Tr(G? ( )) 2
1
3
where Tr() denotes the trace of a matrix. For the on-line estimator b t based on the independent examples f(xs ; zs ); s = 1; ; tg, the negative log-likelihood function is
L1;t = L(x1 ; ; xt ; z1 ; ; zt ; ) = ? log p(x1 ; ; xt ; z1 ; ; zt ; ) = Since
t X s=1
L(xs; zs; ):
@L (x ; z ; ))T ] = G() ( x ; z ; )( E [ @L s s s; @ @
due to the independence of the examples,
1;t @L1;t T Gt () = E [ @L @ ( @ ) ] = tG():
So when the examples f(xs ; zs ); s = 1; 2 ; g are independently drawn according to the probability law p(x; z ; ), the Cramer-Rao inequality for the on-line estimator is E [kb ? k2 ] 1 Tr(G?1 ( )) (3)
t t 2.3 Natural gradient learning Let = fg be the parameter space. The divergence between two points and , or two distributions, p(x; z ; ) and p(x; z ; ), is given by the Kullback-Leibler divergence D( ; ) = KL[p(x; z; )kp(x; z; )]: 1
1
2
2
1
2
1
2
When the two points are in nitesimally close, we have the quadratic form (4) D(; + d) = 12 dT G()d where G( ) is the Fisher information matrix. This is regarded as the square of the length of d . Since G() depends on , the parameter space is regarded as a Riemannian space in which the local distance is de ned by (4). Here, the Fisher information matrix G( ) plays the role of the Riemannian metric tensor. See (Amari, 1985)[1] and (Murray and Rice, 1993)[9] for information geometry. The natural gradient descent method proposed by Amari [3, 2] makes use of the Riemannian metric given by the Fisher information matrix to optimize the learning dynamics such that the Cramer-Rao lower bound is achieved asymptotically. The idea is to convert the covariant gradient @@l into contravariant form G?1 @@l . It is shown by Amari [2] that the steepest descent direction of a function C () in the Riemannian space is ?re C () = ?G?1()rC (): The on-line learning algorithms corresponding to @@l and G?1 () @@l are, respectively, the ordinary gradient descent algorithm:
t = t ? t @@l (zt jxt; t) +1
(5)
and the natural gradient descent algorithm: 0
t = t ? t G? (t ) @@l (zt jxt ; t ) +1
1
4
(6)
where and 0 are learning rates. It is proved in [2] that the natural gradient learning is Fisher ecient. When the negative log-likelihood function is taken as the loss function, the natural gradient descent algorithm (6) gives an ecient on-line estimator, i.e., the asymptotic variance of t driven by (6) satis es E [( ? )( ? )T ] 1 G?1 ( ) (7)
t
t
t
which gives the mean square error
E [kt ? k2 ] 1t Tr(G?1 ( )):
(8)
However, the algorithm (5) does not attain this limit. Remark 1 The equilibrium points are the same for the averaged versions of the dynamics (5) and (6), but the inverse of the Fisher information matrix optimizes the learning dynamics. The local minima problem still exists in the natural gradient learning. Remark 2 The natural gradient learning algorithm is ecient, but its complexity is generally high due to the computation of the matrix inverse. In later sections, we shall explore the structure of the Fisher information matrix for a multi-layer perceptron and design a low complexity algorithm to compute G?1 (). The dimension of G( ) is (nm + 2m) (nm + 2m), the time complexity of computing G?1 () by generally accepted algorithms is O(n3 ) which is too high for the on-line natural gradient learning algorithm. Our method reduces the time complexity to O(n2 ).
3 Fisher Information Matrix of A Multi-Layer Perceptron
3.1 Likelihood function of the stochastic multi-layer perceptron
In previous sections, we did not give any speci c assumption about the input. To compute an explicit expression of the Fisher information matrix, we now assume that the input x is subject to the standard Gaussian distribution N (0; I ), where I is the identity matrix. Pm 1 From l(z jx; ) = 22 (z ? i=1 ai '(wTi x + bi ))2 , we obtain m @l = ? 1 (z ? X a '(wT x + b ))a '0(wT x + b )x (9) i k k i k i @wij 2 k=1 m @l = ? 1 (z ? X ak '(wTk x + bk ))'(w Ti x + bi ) @ai 2 k=1 X @l = ? 1 (z ? m a '(wT x + b ))a '0 (wT x + b ) i k k i k i @bi 2 k=1
j
(10) (11)
When (x; z ) is the input-output pair of the stochastic multi-layer perceptron, the equations (9)-(11) become
@l = ? a '0 (wT x + b )x i j i @wij 2 i @l T @ai = ? 2 '(wi x + bi) @l 0 T @b = ? 2 ai' (wi x + bi); i
5
(12) (13) (14)
and in vector forms
@l = ? a '0 (wT x + b )x; i = 1; ; m i i @ wi 2 i @l = ? '(W T x + b) @a 2 @l = ? fa '0(W T x + b)g @b 2
(15) (16) (17)
where denotes the Hadmard product which is the componentwise product of vectors or matrices. The above three vector forms can be written as one: 2 0 (W T x + b)) x 3 ( a ' @l = ? 6 '(W T x + b) 7 (18) 5 4 2 @ a '0 (W T x + b) where denotes the Kronecker product. For two matrices A = (aij ) and B = (bij ), A B is de ned as a larger matrix consisting of the block matrices [aij B ]. For two column vectors a and b, a b gives the higher-dimensional column vector [a1 bT ; ; an bT ]T .
3.2 Blocks in the Fisher information matrix The parameter consists of m + 2 blocks = [wT ; ; wTm; aT ; bT ]T where the rst m blocks wi 's are n-dimensional and the last two blocks a and b are mdimensional. From (18), the Fisher information matrix G() = A() is of (n +2)m (n +2)m dimensions. Here, the matrix A() does not depend on . It is partitioned into (m+2)(m+2) blocks of sub-matrices corresponding to the partition of . We denote each sub-matrix by Aij . 1
1
2
2
The Fisher information matrix is written as G() = 12 [Aij ](m+2)(m+2) : The partition diagram for the matrix A() is given below: 8 > > > > > > > >
> > > > > > > :
where
wTm A ;m
wT w A; 1
.. .
1
.. .
11
wm Am; a Am b Am
1
.. .
; ;
+1 1
+2 1
.. .
1
(19)
aT A ;m .. .
1
bT A ;m
+1
Am;m Am;m Am ;m Am ;m Am ;m Am ;m
+1
+1
+1
+1
+2
+2
+1
.. .
1
+2
Am;m Am ;m Am ;m
+2
+1
+2
+2
+2
Aij = aiaj E ['0 (wTi x + bi )'0 (wTj x + bj )xxT ], for 1 i; j m, Ai;m = aiE ['0 (wTi x + bi)x'(xT W + b)] and Am ;i = ATi;m for 1 i m, Am ;m = E ['(W T x + b)'(xT W + b)], Ai;m = E [ai'0 (wTi x + bi)x(aT '0 (xT W + bT ))] and Am ;i = ATi;m for 1 i m, Am ;m = ATm ;m = E ['(W T x + b)(aT '0 (xT W + bT ))], Am ;m = (aaT ) E ['0 (W T x + b)'0 (xT W + bT )]. +1
+1
+1
+1
+1
+2
+2
+2
+1
+2
+2
+2
+2
+1
6
(20)
Here, E [] denotes the expectation with respect to p(x). From the above expressions, we know that A( ) = [Aij ] does not depend on 2 . The explicit forms of these matrix blocks in the Fisher information matrix are characterized by the three lemmas in the next section.
Remark 3 Since the loss function l(zjx; ) contains the unknown parameter , we should 2
use the following modi ed loss function in practice: l1 (zjx; ) = 21 (z ? aT '(W T x + b))2 : Modifying the learning rule (5), we have the following on-line algorithm:
t = t ? t @l@ (zt jxt ; t ):
(21)
1
+1
Since G?1 () @@l = A?1 ( ) @l@ 1 , the learning rule (6) is exactly the same as the following learning rule: 0 t+1 = t ? t A?1(t) @l@ 1 (zt jxt; t): (22)
3.3 Representation of blocks in the Fisher information matrix Lemma 1 For i = 1; ; m, the diagonal blocks are given by Aii = ai [d (wi; bi )I + fd (wi ; bi) ? d (wi ; bi )guiuTi ] 2
1
2
1
(23)
where
wi = kwi k denoting the Euclidean norm of wi ; ui = wi =wi ; Z1 x2 1 ('0 (wx + b))2 e? 2 dx > 0; d1 (w; b) = p 2 Z?1 1 0 x2 1 d2 (w; b) = p (' (wx + b))2 x2 e? 2 dx > 0: 2 ?1
When ai 6= 0, the inverse of the matrix Aii is given by A?ii 1 = a12 [ d1 (w1i ; bi) I + f d2 (w1i ; bi ) ? d1 (w1i; bi ) guiuTi ]; i = 1; ; m: i
(24) (25) (26)
The proof of Lemma 1 is given in Appendix 1. In particular, for a single-layer stochastic perceptron
z = '(wT x + b) + ; the expressions (23) and (26) give the Fisher information matrix: G(w) = 1 [d (w; b)I + fd (w; b) ? d (w; b)gu uT ]
2
and its inverse formula
1
2
i i
1
1 I + f 1 ? 1 gu uT ]: G? (w) = [ d (w; b) d (w; b) d (w; b) i i 1
2
1
2
When b = 0, the above formula is found by Amari in [3]. 7
1
(27) (28)
To obtain the explicit forms of the other blocks Aij in G, we need to introduce two bases in 0 is a learning rate. If the input is not Gaussian, we need a non-linear function to transform the input xt to a Gaussian process. Let Fx () be the cumulative distribution function (cdf) of xt and FN () be the cdf of n-dimensional Gaussian r.v., then we obtain a standard Gaussian process ut = FN?1(Fx (xt )) N (0; I ): If Fx () is unknown, it can be approximated by the empirical distribution based on the data set. This method is applicable to the input with an arbitrary distribution but it is not applicable to on-line processing since the empirical distribution is computed by a batch algorithm. If the probability density function (pdf) of the input can be approximated by some expansions such as the Gram-Charlier expansion and the Edgeworth expansion [12], Fx () can be approximated online by using an adaptive algorithm to compute the moments or cumulants in these expansions. After preprocessing, we use the data set De T = f(ut ; zt ); t = 1; ; T g instead of DT to train the multi-layer perceptron.
4 Inverse of Fisher Information Matrix Now we calculate the inverse of the Fisher information matrix which is needed in the natural gradient descent method. We have already shown the explicit form (26) of the inverse of Aii for 1 i m. To calculate the entire inverse of G, we need to compute the inverse of a matrix with the similar representation as Aij for 1 i 6= j m. To this end, let us de ne Gl(m; is a \converge phase". The learning rate functions i(t) do not have the search phase but they start learning with a weaker converge phase when i are small. When t is large, each learning rate function i (t) decreases as cti . In this example, we choose 0 = 1:25, 1 = 0:05, c0 = 8:75, and c1 = 1. These parameters
are selected by trial and error to optimize the performance of the GD and the natural GD methods at the noise level = 0:2. The learning rate functions 0 (t) and 1 (t) are compared in Figure 3 with their asymptotic curves cti . 1
10
: t
8 75
0
10
0 (t) 1
t
1 (t)
−1
10
PSfrag replacements10
−2
−3
10
0
50
100
150
200
250
300
350
400
450
500
Figure 3: 0 (t) vs. 1 (t) when 0 = 1:25, 1 = 0:05, c0 = 8:75 and c1 = 1. A 7-dimensional vector is randomly chosen as w for the teacher network. In this simulation, we choose w = [?1:1043; 0:4302; 1:1978; 1:5317; ?2:2946; ?0:7866; 0:4428]T : The training examples are the Gaussian random inputs to the teacher network and its outputs in Gaussian noise. Let wt and we t be the weight vectors driven by the equations (56) and (57) respectively. The error functions for the GD and the natural GD algorithms are kwt ? w k and kwe t ? w k: Let w = kw k. From the equation (58), we obtain the Cramer-Rao Lower Bound (CRLB) for the deviation at the true weight vector w : s (61) CRLB(t) = p dn (?w1) + d (1w ) : t 1 2 19
The error functions are compared with the CRLB in Figure 4.
= 0:2
error
6 natural GD GD CRLB
4 2 0 0
50
100
150
200
error
6
350
400
450
500
= 0:4
2
50
100
150
200
15 error
300
natural GD GD CRLB
4
0 0
PSfrag replacements
250 iteration
250 iteration
300
350
450
500
=1
natural GD GD CRLB
10
400
5 0 0
50
100
150
200
250 iteration
300
350
400
450
500
Figure 4: Performance of the GD and the natural GD at di erent noise levels = 0:2; 0:4; 1. It is shown in Figure 4 that the natural GD algorithm reaches CRLB at di erent noise levels while the GD algorithm reaches the CRLB only at the noise level = 0:2. The robustness of the natural gradient descent against the additive noise in the training examples is clearly shown by this example.
6.2 Multi-layer perceptron
Let us reconsider the simple committee machine with 2-dimensional input and 2-hidden neurons given in Section 5. The problem is to train the following multi-layer perceptron:
y = '(wT1 x) + '(wT2 x)
(62)
based on the examples f(xt ; zt ); t = 1; ; T g generated by the following stochastic committee machine: zt = '(w1T xt ) + '(w2T xt ) + t (63) with Gaussian noise t N (0; 2 ) and Gaussian input xt N (0; I ). Assume kwi k = 1. We can reparameterize the weight vector to decrease the dimension of the parameter space from 4 to 2: # " # " ) cos( ) cos( i i wi = sin( ) ; wi = sin( ) ; i = 1; 2: i
i
20
The parameter space is f = ( 1 ; 2 )g. Assume the true parameters are 1 = 0 and 2 = 34 . Due to the symmetry, both 1 = (0; 34 ) and 2 = ( 34 ; 0) are true parameters. Let t and 0t be generated by the GD algorithm and the natural GD algorithm respectively. The errors are measured by
"t = minfk t ? 1 k; k t ? 2kg; "0t = minfk 0t ? 1 k; k 0t ? 2kg:
In this simulation, we rst start the GD algorithm with the initial vector 0 = (0:1; 0:2) for 80 iterations, then we start the natural GD algorithm using the estimates of the parameters obtained by the GD algorithm at the 80-th iteration as the initial for the natural GD algorithm. The noise level is = 0:05. N independent runs are conducted to obtain "t (j ) and "0t (j ) for j = 1; ; N: Averaging these errors, we obtain v u X N u "t = t 1 ("t (j ))2 ;
N j=1
and
v u X N u 0 " t = t 1 ("0 (j ))2 :
N j=1
t
The averaging errors "t and "0 t for the two algorithms are compared with the CRLB in Figure 5 for N = 10. The search-then-converge learning schedule (60) is used in the GD algorithm while the learning rate for the natural GD algorithm is simply the annealing rate k1 . The CRLB is given in Section 4.5 by Theorem 2. 1
10
GD
0
10
natural GD
error
CRLB −1
10
−2
10
−3
10
0
100
200
300 iteration
400
500
600
Figure 5: The GD vs. the natural GD
7 Conclusions The natural gradient descent learning rule is statistically ecient. It can be used to train any adaptive system. But the complexity of this learning rule depends on the architecture of 21
the learning machine. The main diculty in implementing this learning rule is to compute the inverse of the Fisher information matrix of a large size. For a multi-layer perceptron, we have shown an ecient scheme to represent the Fisher information matrix based on which the space for storing this large matrix is reduced to O(n) from O(n2 ). We have also shown an algorithm to compute the inverse of the Fisher information matrix. Its time complexity is of order O(n2 ) when the input dimension n is much larger than the number of hidden neurons while the complexity of conventional algorithms for the same purpose is of order O(n3 ). An intermediate result is an explicit CRLB formula for a committee machine. The simulation results have con rmed the fast convergence and statistical eciency of the natural gradient descent learning rule. It is also veri ed that this learning rule is robust against the additive noise in the training examples.
8 Appendix
Appendix 1 Proof of Lemma 1: Let u1 = wi =wi and extend it to fu1 ; ; ung an orthogonal basis of
