Stochastic and PWM coding for an efficient implementation of Cellular Neural Networks F.Colodro, A.Torralba, R.G.Carvajal and L.G.Franquelo ´ Dpto. de Ingenier´ıa Electronica, Escuela Superior de Ingenieros Avda. Reina Mercedes, s/n, SEVILLA–41012 (Spain) Tlf: 34–5–4556849, Fax: 34–5–4629205 e–mail:
[email protected]
Abstract— This paper present the application of Pulse Stream Techniques (PSTs) to the hardware implementation of a Cellular Neural Network. The time differential equation of this networks suggests that the dynamic of one neuron status can be emulated by adding discretized packets of charge to a capacitor. This task can be carried out by driving a current source with a pulse stream signal.
I. INTRODUCTION
sysnapses product of a Neural Network. Based on this principles, the basic architecture of one neuron is presented in this paper. In section II the architecture is used to build an efficient implementation of a Cellular Neural Network (CNN). A detailed description of the basic modules and the result obtained from simulations are presented in section III, and finally some conclusions are discussed in section IV.
Different strategies of design have been used for integrated electronic implementation of Neural Networks II. THE NEURON STATE OF A CNN (NNs). Based on digital or/and analog technologies, many arquitectures have been described in the literature. The advantages and drawbacks of analog and digital computing are well–known. To overcome these drawbacks, Pulse Stream Techniques (PSTs) have been proposed [1]. PSTs take the advantages of both, digiAND2 SGN{A(i,j;k,l} S1 tal and analog implementations, as the information is φ I off I Rx carried by digital signals and analog circuitry is used 1 CMT Io SA(i,j;k,l) AND1 to control them. Pv’kl
. . .. . φ
Io
sum line
I
+ -
.
P vyij
o 2 The most common PSTs are Stochastic Logic (SL) and S2 Pulse Width Modulation (PWM). In SL, a signal is rep- SGN{A(i,j;k,l} AND3 resented by a stochastic pulse stream which takes the value 1 with a probability proportional to the instantaneous value of the signal. The main advantage of Figure 1: Block diagram of a neuron SL relies on the fact that the product of two stochastic signals can be implemented by means of a simple AND gate. On the other hand, there is not a direct way to implement the summation in a digital way. In A Cellular Neural Network (CNN) is an analog dy[2], a digital circuit called F was proposed, which is namic processor array, where the processing elements interact within a finite local neighborhood. The origable to make the summation of n stochastic pulses in inal CNN paradigm was first proposed by Chua and one clock cycle, at the cost of some hardware comYang in [3] and [4]. A concise tutorial on CNNs can plexity. A more efficient implementation can be made be found in [5]. A review of selected communications if current–mode analog summation is used, as shown in the field of CNN theory and applications, and a bellow. simulator with many examples, can be found in [6]. Integrated implementation of CNNs can be found in If one of the two stochastic signals in the input of a [7]–[15]. An implementation based on Pulse Stream AND gate is replaced by a Pulse Width Modulation current mode can be found in [16] (PWM) signal, the AND gate still carries out the multiplication operation. The product signal can be used The differential equations governing the (i; j )–th neuto switch on/off a constant current which feeds a capacitor. In this way it is possible to carry out the ron in a CNN is given by [3]:
C d dvxij t
=
, R1 vxij +
X
x
C (k;l) 2 Nr (i;j)
X C (k;l) 2 Nr (i;j)
B (i; j ; k; l) vukl
A(i; j ; k; l) vykl + +
Ioff
vyij = f (vxij )
(1)
IRx (2)
where vxij , vyij and vuij are the status, output and input of cell (i; j ), respectively. C and Rx are CNN parameters. Ioff is a bias term. Nr (i; j ) is a neighborhood of radius r around the (i; j )–th cell. A(i; j ; k; l) and B (i; j ; k; l) are the feedback and forward matrices associated to the cell (i; j ), respectively [3]. The piece-wise linear function f is a saturated activation function. The time discretized charging equations (1) and (2) of a CNN is implemented electronically by the circuit shown in figure 1. This architecture reproduces the behavioural of one unipolar neuron. (Note that CNNs with bipolar neurons are easily converted to unipolar neurons, with only a change in the bias term Ioff ).
I off
. . ... . . ...
sum line
Vref
Vref
CMT + -
.
P vyij
+
Vxij RAMP GEN. -
Figure 2: State of a CNN using a PST
The stochactic signal S A(i;j;k;l) and the PWM signal Pvykl are pulse streams which represent the values of A(i; j ; k; l) and vykl , respectively. So, the gate AND1 performs the product A(i; j ; k; l) vykl . The product pulse drives a bipolar current source and, depending on the sign of A(i; j ; k; l), charges o discharges a summation capacitor. In a space–invariant CNN, the templates A and B are shared by all the neurons, so that the stochastic pulses SA(i;j;k;l) and SB(i;j;k;l) are obtained only once in a common module and later propagated to the rest of the chip with a considerable saving in silicon area. Note that a broadcast propagation is possible in our implementation, because these pulse streams are digital signals.
III. BASIC MODULES OF THE NEURON STATE
PWM signals are associated to neuron outputs. PWM coding has been selected, because it can be implemented with a local ramp generator and a comparator (figures 1 and 2). The comparator and the local ramp generator are used to generate the PWM signal P vyij , which is fedback to drive the slave part of a high– swing cascade current mirror (Fig.- 2). In this way it is possible to take into account the effect of the resistor Rx (equation 1) using the current IRx . A dummy transistor was used to reduce the charge–injection problem; the capacitor can be implemented by means of the gate capacitance of a transistor in a digital CMOS technology. The circuit in figure 2 has been designed using a 1:0m
Figure 3: Continuous time dimamic of a circuit RC and the circuit in figure 2: a) Charging and b) Discharging.
.. .
VDD
V
ref1
Io
1
φ
Vref
2.6
φ
1
2.1
1
sum line
φ
2
φ
2
Voltage (V)
Io
. . . . . . ..
φ
Vref
1.6
Vref2
1.1
φ2
High−level language simulation result HSPICE result
0.6 −2.3
Figure 4: Current mirrors used for driving the cell capacitor. 3.9
17.7 Time (microsecond)
37.7
High−level simulation result HSPICE result
Voltage (V)
3.4
5V , digital CMOS technology. The current Ioff was adjusted to obtain a steady state value of 2:5V in the capacitor. The simulation results (obtained from HSPICE) of the circuit in Fig. 2 using two different initial values are depicted in figures 3.a and 3.b. The time evolution of a RC circuit for the same initial values is superimposed. The current sources and the switches, which are driven by the products A(i;j ; k; l) vykl and B (i;j ; k; l) Vukl , have been built using two high–swing cascode current mirrors as shown in figure 4. The pulse stream signals 1 and 2 represent the synaptic products (outputs of the gates AND2 and AND3 in Fig. 1, respectivelly) associated to the cell in figure 4.
2.4
1.9 −10.0
The comparation of the evolution of the summation capacitance voltage when the results are obtained from language C and HSPICE simulations are shown in figures 5.a and 5.b in two different cases. In both cases, the same stochastic signals associated to the synaptic weight were used in C and HSPICE. In this way we avoid that the simulation results were affected by different behaviour due to the random nature of the stochastic signals. The neighbor pixel values were chosen so that the final values of the cell state voltage was 0V in the first case (Fig. 5.a) and 5V in the second case (Fig. 5.b). Both figures verify that the pulse stream computational strategy can be implemented using a CMOS technology while preserving high accuracy.
40.0
90.0 140.0 Time (microsecond)
190.0
Figure 5: Comparation between the language C and HSPICE results when the final values of the cell state voltage is (a) 0V and (b) 5V
High–level language C simulations have shown that the computational strategy proposed in this paper is a good way for implementation Cellular Neural Networks. The simulated networks was able to successfully solve an edge extraction problem of a 21 21 picture.
2.9
IV. CONCLUSIONS
The application of Pulse Stream Techniques (PSTs) to the hardware implementation of Cellullar Neural Networks has been presented in this paper. The main features of the proposed architecture are:
The information is codified by digital signals which are very robust againts noise and interferences. The stochastic nature of some signals allow to implement the multiplication using a simple AND gate. The summation is performed in a current mode approach. Both features allow to perform the computation using a simple circuitry with high accuaracy. The feedforward and feedback operators can be stored in digital memory. In case of CNNs with invariant templates, the number of these weights is small and the efficiency in silicon area is preserved. Due to the stochastic codification of these weights, they can be propagated through the ASIC without corruption using a reduced number of physical lines.
High–level language simulations have been done in order to check the behaviour of a network when it is implemented using PSTs. Results of transistor–level simulations have been compared with the results of high–level simulations for the case of one neuron and they have been demonstrated negligible non–ideal effect. PSTs seem to be a promising way to build large arrays with programmable synapses. Presently we are designing the neuron layout to build a 21 21 CNN array using a 1 digital CMOS technology to experimentally demonstrate the advantages of the proposed architecture.
ACKNOWLEDGMENTS
This work has been supported by the Spanish Comisi´on Interministerial de Ciencia y Tecnolog´ıa (CICYT), under project TIC96–0860.
REFERENCES [1] A.Murray, D.del Corso and L.Tarassenlo. “Pulse– Stream VLSI neural networks mixing analog and digital techniques”. IEEE Trans. on Neural Networks, vol. 2, n. 2, pp. 193–204, Mar. 1991. [2] A.Torralba, F.Colodro. “Two digital circuits for a Fully Parallel Stochastic Neural Network”. IEEE. Trans. of Neural Network, vol. 6, no 5, 1264–1268, SEP. 1995. [3] L.O.Chua and L.Yang. “Cellular Neural Networks: Theory”. IEEE Trans. on CAS, vol. 35, n. 10, pp. 1257–1272. Oct. 1988. [4] L.O.Chua and L.Yang. “Cellular Neural Networks: Applications”. IEEE Trans. on CAS, vol. 35, n. 10, pp. 1273–1290. Oct. 1988. [5] L.O.Chua and T.Toska. “The CNN Paradigm”. IEEE Trans. on CAS–I, vol. 40, n. 3, pp. 147–156. March 1993. [6] Cellular Neural Networks. T.Roska and J.Vandewalle eds. John Wiley & Sons: Chichester 1993. [7] J.M.Cruz, L.O.Chua. “A CNN chip for connected component detection”. IEEE Trans. on Circuits and Systems, vol. 38, pp. 812–817, July 1991. [8] H.Harrer, J.A.Nossek and R.Stelzl. “An analog implementation of discrete–time cellular neural networks”. IEEE Trans. on Neural Networks, vol. 3, pp. 466–476, May 1992. [9] A.Rodr´ıguez V´aquez, S.Espejo, R.Dom´ınguez Castro, J.L.Huertas and E.S´anchez Sinencio. “Current–mode techniques for the implementation of continuous– and discrete–time cellular neural networks”, IEEE Trans. on Circuits and Systems–II, vol. 40, pp. 132–146, Mar. 1993.
[10] J.E.Varrientos, E.S´anchez Sinencio and J.Ram´ırez Angulo. “A current–mode cellular neural network implementation”. IEEE Trans. on Circuits and Systems–II, vol. 40, pp. 147–155, Mar. 1993. [11] I.A.Baktir, M.A.Tan. “Analog CMOS implementation of cellular neural networks”, IEEE Trans. on Circuits and Systems–II, vol. 40, pp. 200–206, Mar. 1993. [12] G.F.D.Betta, S.Graffi, Z.M.Kovacs, G.Massetti. “CMOS implementation of an analogically programmable cellular neural network”, IEEE Trans. on Circuits and Systems–II, vol. 40, pp. 206–215, Mar. 1993. [13] M.Anguita, F.J.Pelayo, A.Prieto and J.Ortega. “Analog CMOS implementation of a discrete– time CNN with programmable cloning templates”. IEEE Trans. on Circuits and Systems–II, vol. 40, pp. 215–218, Mar. 1993. [14] A.Paasio, A. Dawidziuk and V. Porra “VLSI Implementation of Cellular Neural Network Universal Machine”. IEEE Int. Symp. on Circ. and Syst. ISCAS’97, vol. 1, pp. 545–548, Jun. 1996. [15] M.Salerno, F.Sargeni and V.Bonaiuto “6x6DPCNN: a programable mixed analogue–digital chip for Cellular Neural Networks”. Fourth IEEE Int. Work. on CNN and Their Appl., pp. 451–456, 1996 [16] Ari Paasio, Adam Dawidziuk and Veikko Porra. “Pulse Stream Current Mode CMOS CNN chip”. Fourth IEEE Int. Work. on CNN and Their Appl., pp. 457–460, 1996.
