Biologically Realistic Neural Networks and Adaptive ... - CiteSeerX

3 downloads 622 Views 249KB Size Report
Knowledge Engineering and Discovery Research Institute, Auckland University of Technology, New. Zealand ... challenging aspects for neuroscientists is that current technologies can not keep track ... information processing with biological relevance, the biologically realistic neural .... ms (Delorme & Thorpe, 2001).
Biologically Realistic Neural Networks and Adaptive Visual Information Processing Simei Gomes Wysoski, Lubica Benuskova Knowledge Engineering and Discovery Research Institute, Auckland University of Technology, New Zealand {swysoski, lbenusko}@aut.ac.nz

ABSTRACT This work aims to review the basic concepts of biologically realistic neural networks when applied to visual pattern recognition. A new simple model of biologically realistic visual pattern recognition that adaptively learns by example through synaptic plasticity and changes in structure is also presented. This system uses a spiking neural network composed of integrate-and-fire neurons and Hebbian-based learning. The event driven approach used in the implementation optimizes processing speed in order to simulate networks with large number of neurons.

Keywords Biologically realistic neural networks, spiking neural networks, pattern recognition, adaptive systems.

1. Introduction It is an oversimplification to say that it is not possible to artificially emulate the human brain because of the limitations of current computational resources. In reality, the key issue for failing to properly emulate the human way of processing is the existence of many undeciphered details of the brain structure and behaviour. In respect to the current knowledge on the brain, briefly, so far it is possible to accurately: • • •

measure electrophysiological properties of a single cell with high accuracy (Kandel, Schwartz, & Jessel, 2000); measure the average activity of the neurons located in a certain area of the brain (electroencephalograms and fMRI); divide the human brain in compartments and perform a systematic analysis on how they are interconnected. Here we can map and build a flow chart of signal activity, define structural functions for each compartment, and deduce the relationship and dependencies among compartments.

However, little is known about connectivity among neurons, how the neurons behave as an ensemble, and how information is encoded. Attempts to reproduce connectivity of biologically realistic cells have been limited to consider ensembles of neurons placed homogeneously in a two dimensional array where the connectivity may follow a given criteria (all-to-all connections or neighbouring connections) or be chosen randomly. Some more careful investigations claim the brain is connected through many “small

world networks” (Gong & van Leeuwen, 2004). This theory, which is currently under experimental validation, basically suggests that neurons in the brain are strongly interconnected with other neighbouring neurons, while there are fewer connections among neurons located outside of the immediate neighbourhood. In an attempt to discover what is behind the reasoning of human minds, one of the most challenging aspects for neuroscientists is that current technologies can not keep track and measure all the signals used for inter-neural communication, even in a small portion of the brain. If this was possible, it could enable to accurately understand the emergence of intelligence from an ensemble of neurons. In an attempt to overcome this limitation, a common practice is to complement the experimental brain study with the development of computational models based on neurobiological data and to study their properties by theoretical and simulation means. In this direction there are the computational models of information processing with biological relevance, the biologically realistic neural networks. Of particular interest to this work is the modelling of visual information processing to perform visual pattern recognition. Thus, in this paper we will first review the basic concepts of biologically realistic neural networks and explore brain-like techniques for information processing. In addition, we will review how the visual system is formed and the emergence of knowledge through experience and structural development. A simple adaptive brain-like model of a visual system is exemplified and its correspondence with traditional computational models is highlighted.

2. Biologically realistic neural networks 2.1. Definition Biologically realistic neural networks or brain-like neural networks are networks that have a closer association with what is known about the way brains process information. In a simplified manner, these networks incorporate the following properties of the brain: a) neuron as a processing unit. The brain is basically composed of neurons and glial cells. Despite the number of glial cells being 10 to 50 times greater than the number of neurons, the role of information processing is given exclusively to the neurons. For this reason the glial cells are most of the time left aside in the computational models; b) neurons are constituted of 4 parts. Input, trigger, conduction and output (Kandel, Schwartz, & Jessel, 2000) are commonly represented in neuronal models. In Figure 1 we can see a representation of a biological neuron and its respective computational model; c) neurons interact using spikes. In biological brains, neurons transmit information through synaptic junctions in complex chemical reactions using neurotransmitters. This operation can be represented in electrical terms by electric pulses (short term high amplitude signals). To resemble this property, in biologically realistic neural networks the transfer of information is done through spikes. The use of spikes brings together the definition of time varying postsynaptic potential, PSP, firing threshold, ϑ, and spike latencies, Δ, as depicted in Figure 1;

d) neurons organize in ensembles with divergent and convergent connections. In a very simplified manner, the neurons connect to each other in two basic ways: through divergent and convergent connections. Divergent connection occurs when the output of a neuron is split and is connected to the input of many other neurons. Convergent connections are those where a certain neuron receives input from many neurons. It is with the organization of the neurons in ensembles that functional compartments emerge.

Figure 1 – Left: Representation of biological neuron. Right: Basic artificial spiking unit.

2.2 Information processing in traditional and biologically realistic neural networks Many times it is possible to directly associate traditional ways to perform information processing with the operations executed by spiking neural network models. The visual system, for instance, can be modelled using well established image processing techniques that already have a well described mathematical background (Fourier, Wavelets, etc). Spiking networks can perform similar tasks, however they are often more complex and difficult to evaluate, mainly because all the processing is done based on spike time. Nevertheless, it has been proven that spiking networks can, similarly to traditional networks, act as radial basis function, perform weighted sum of temporal code, and be used for universal approximation of continuous functions (Gerstner & Kistler, 2002). Besides, a single neuron can act as a coincidence detector, what can not be easily achieved with traditional neural networks. After a detailed comparison, Maass (1998) concluded that the traditional and biologically realistic neural networks are computationally equivalent. Further analysis showed that any sigmoidal neural networks can be successfully reproduced using almost the same number of spiking neurons, while to simulate spiking neurons many more sigmoidal units are necessary (Maass, 1998). As a final remark, in addition to the ability of the spiking neural networks to perform complex mathematical operations, being closer to known biology can help to discover principles and rules that govern the brain’s way of processing patterns that are not yet known.

2.3 Types of spiking neural networks (SNN) and their usage SNN have been traditionally used in computational neuroscience, usually in an attempt to model and evaluate the dynamics of neurons and how they interact in an ensemble. In terms of artificial neurons the Hodgkin-Huxley model (Hodgkin & Huxley, 1952) can be considered to be the pioneering approach describing the action potentials in terms of ion channels and current flow (Nelson & Rinzel, 1995). Further studies expanded this work and revealed the existence of a wide number of ion channels which vary from one neuron to another. Simulation with this type of neurons can be done using simulators like GENESIS and NEURON (Genesis; Neuron). The use of numerous compartments to describe a neuron has been shown to be very computationally expensive for simulations of networks with large number of units, in such a way that new models have been developed to produce similar behaviour (similar action potentials) at a lower computational cost. As an example of the simplified models, the integrate-and-fire neuron (Gerstner & Kistler, 2002) has ultimately the properties of a single capacitor and the Izhikevich model (Izhikevich, 2003) combines the Hodgkin-Huxley with the integrateand-fire model using a two-dimensional system of ordinary differential equations. Table 1 shows the different models of neuronal units according to their biological accuracy. For engineering applications, on the other hand, a neural network is normally comprised of rather simple linear/non-linear processing elements (Haykin, 1999; Bishop, 2000), where the sigmoidal function is widely used. SNN has often been considered too complex and cumbersome for this task. However, recent discoveries on the information processing capabilities of the brain and technical advances related to massive parallel processing, are bringing back the idea of using biologically realistic neural networks as a machine learning technique. A recent pioneering work has shown that the primate (including human) visual system can analyse complex natural scenes in about 100-150 ms (Delorme & Thorpe, 2001). Considering the average spiking rate of the neuronal cells in the brain, this result is very impressive. This theory suggest that neurons exchange only one or few spikes before the information processing is completed. As an output of this work, the authors proposed a multilayer feedforward network of integrateand-fire neurons that can successfully track and recognize faces in real time (Delorme, Gaustrais, van Rullen, & Thorpe, 1999). Table 1 - Classification of artificial neural network models according to the biological relevance.

Model

Biologically Motivated Hodgkin-Huxley Model

Usage

Tools for neuroscientists

Moderate biological relevance Spike Response Model Integrate-and-Fire Izhikevich SpikeNet Simulation of large networks Evaluate temporal properties and synchronicity of spiking neurons • Pattern recognition

• • • • • •

No biological relevance • McCulloch-Pitts • Adaline • Perceptron • •

Pattern recognition Engineering problems

2.4 Emergence of knowledge in biologically realistic neural networks Widely accepted to be the most similar to the brain’s way of learning is the Hebbian rule, that basically consists of strengthening the connections among neurons that are more active and decreasing the importance of less active connections. Hebbian learning rule is intrinsically unsupervised, as the connection weights are updated based only on the inner activity levels. The learning is not influenced by feedbacks or back propagation of errors, which typify supervised learning. Spike-Timing-Dependent Synaptic Plasticity (STDP) is a rule based on Hebbian learning that uses the temporal order of presynaptic and postsynaptic spikes to determine whether a synapse is potentiated or depressed (Song, Miller, & Abbott, 2000). Another variation of the Hebbian learning is BCM rule that modifies synaptic weights dynamically depending on the current and previous spiking rates (Bienenstock, Cooper, & Munro, 1982). All these works justify the biological relevance on different methods of unsupervised learning and self organization. Doya (1999) further hypothesizes that the cerebellum performs supervised learning, basal ganglia perform reinforcement learning and cerebral cortex unsupervised learning. In the same work, it is emphasized that all the three parts of the brain are able to learn rather than being genetically preprogrammed for static behaviours.

3. Biological visual information processing On the subject of biological approaches for processing incoming information, Hubel and Wiesel (1962) received many awards for their description of the human visual system. Through neurophysiological experiments, they were able to distinguish some types of cells that have different neurobiological responses according to the pattern of light stimulus. They identified the role that the retina has as a contrast filter as well as the existence of orientation selective cells in the primary visual cortex (see Figure 2). Their results have been widely implemented in biologically realistic image acquisition approaches. The idea of contrast filters and orientation selective cells can be considered a feature selection method that finds a close correspondence with traditional ways of image processing, such as wavelets and Gabor filters (Sonka, Hlavac, & Boyle, 1998).

Figure 2 – Contrast cells and direction selective cells.

3.1 Learning to search visual patterns – An example In this section we will explain an example of a biologically realistic system capable to perform visual pattern recognition. The SNN is composed of 3 layers of a modified version of integrate-and-fire neurons. The neurons have a latency of firing that depends upon the order of spikes received. Each neuron acts as a coincidence detection unit and the postsynaptic potential for neuron i at a time t is calculated as:

PSP (i , t ) = ∑ mod order ( j ) w j ,i

(1)

where mod ∈ [0,1] is the modulation factor, j is the index for the incoming connection and w is the corresponding synaptic weight. The system is implemented based on principles introduced in (Delorme & Thorpe, 2001; Delorme, Gaustrais, van Rullen, & Thorpe, 1999; Delorme, Perrinet, & Thorpe, 2001; Thorpe & Gaustrais, 1998). Each layer is composed of neurons that are grouped in two dimensional arrays forming neuronal maps. Connections between layers are purely feedforward and each neuron can spike at most once when an external source (image) excites the input neurons. The first layer represents cells of the retina, basically enhancing the high contrast parts of the input image. The output values of the first layer are encoded to spikes in the time domain. High amplitude values on the output of first layer are translated as a spike with a low time delay while low amplitude output values generate spikes with higher delays. This technique is called Rank Order Coding (Thorpe & Gaustrais, 1998) and basically prioritizes the pixels with high contrast that consequently are processed first. Second layer is composed of orientation maps, each one selective to a different direction (0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°). Each orientation can have several maps sensitive to different frequencies. It is important to notice that in the first two layers there is no learning, in such a way that the structure can be considered simply time domain encoders and passive filters. The third layer is where the learning takes place. Maps in the third layer represent the complex cells that are selective to more complex shapes and patterns. For learning, the weights of the third layer are adjusted in such a way that the relative order of firing will maximize the response of a given integrate-and-fire neuron using

Δ w j ,i = mod order ( j )

(2)

The network can, for instance, learn to discriminate objects, detect and recognize faces, facial expressions, hands posture, etc. The only requirement is the visual pattern needs to have well defined orientations and/or boundaries in order to properly excite the contrast and orientation maps.

During learning, the number of neuronal maps in each category changes according to the on-line learning algorithm proposed in Wysoski, Benuskova, & Kasabov, 2006. According to this algorithm, neuronal maps can be created or adaptively updated in an on-line manner. There are also inhibitory connections among neuronal maps in the third layer, so that when a neuron fires in a certain map, other maps receive inhibitory spikes in an area centred at the same spatial position. An input pattern belongs to a certain class if a neuron in the corresponding neuronal map spikes first. After learning is completed (weights trained), the network is configured to process test images. Here, each map in each layer is composed of the same number of neurons as the number of pixels in the input image. The spikes generated when an image is presented to the system are propagated to the second and third layer. A pattern is classified if neurons in one of the maps of the third layer generate output spikes. Further, if output spikes occur in layer 3, the spatial position of the neuron that generated the output spikes can be directly associated with the pattern’s position in the input image. Figure 3 shows the complete network architecture.

Figure 3 – Adaptive spiking neural network architecture for visual pattern recognition.

An interesting property of this system is the overall low activity of the neurons. It enables to simulate networks with large number of neurons where only few effectively take part in retrieving information. In this case, the computational performance of the simulation can be optimized using an event driven approach (Delorme, Gaustrais, van Rullen, & Thorpe, 1999; Mattia & del Giudice, 2000). In addition, many times the processing can be interrupted before the simulation is entirely completed. Once a single neuron of the

output layer (layer 3) reaches the threshold and generate an output spike, the simulation can be finished.

4. Discussion We have described that biologically realistic neural network models can be used for a wide range of purposes, from the pure modelling of brain signals to industrial applications. Biologically realistic neural networks have some interesting properties that make them unique and interesting to analyze. To mention only a few: 1. while spiking networks can perform all possible operations associated with traditional neural networks, new ways of connectivity and temporal coding yet to be discovered can bring new insights into creation of artificial systems with performance closer to human brains; 2. the event driven approach can speed up simulations due to the intrinsic low level of activity of the networks. The early interruption of the simulation can enable optimization of processing time, i.e., it is possible to optimize the network not only to maximize accuracy but also to minimize the information processing time; 3. suitability for hardware implementation. Since the operation of spiking neurons is inherently similar to operation character of elements of electronic circuits. A simple example of network architecture and behaviour is presented that can detect and recognize patterns. In the example, a high degree of sparseness is reached in the final layer, which ultimately can be considered to be maps of “grandmother” cells (Connor, 2005). On the pattern recognition view point it seems to be a waste of allocated memory since most of the neurons are rarely used (to each visual pattern to be recognized a neuronal map or a set of neuronal maps is/are allocated). However, despite the memory usage it does not affect processing speed due to the event driven approach applied to computation.

5. Acknowledgements The work has been supported by the Tertiary Education Commission – NZ (S.G.W.) and the NERF grant X0201 funded by FRST (L.B.). We also acknowledge Prof. Nikola Kasabov for the supervision and guidance.

6. References Kandel, E. R., Schwartz, J.H. & Jessell, T.M. (2000). Principles of neural science, ed. 4, McGraw-Hill, New York. Gong, P. & van Leeuwen, C. (2004). Evolution to a small-world network with chaotic units, Europhysics Letters, vol. 67, pp. 328-333. Gerstner, W. & Kistler, W. M. (2002). Spiking neuron models, Cambridge Univ. Press, Cambridge, MA. Maass, W. (1998). Computing with spiking neurons, In Maass, W. & Bishop, C.M. (eds), Pulsed neural networks, chap. 2, pp. 55-81, MIT Press, Cambridge, MA.

Hodgkin, A. L. & Huxley, A. F. (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve, J. Physiol, vol. 117, pp. 500544. Nelson, M. & Rinzel, J. (1995). The Hodgkin-Huxley model, In Bower, J.M. & Beeman (eds), The book of Genesis, chap. 4, pp. 27-51, Springer, New York. Genesis. http://www.genesis-sim.org/GENESIS/. Neuron. http://www.neuron.yale.edu/neuron/. Izhikevich, E. M. (2003). Simple model of spiking neurons. IEEE Trans. of Neural Networks, vol. 14, pp. 1569. Haykin, S. (1999). Neural networks - a comprehensive foundation, Prentice Hall, Engelwood Cliffs, NJ. Bishop, C. (2000). Neural networks for pattern recognition, Oxford University Press, Oxford. Delorme, A. & Thorpe, S. (2001). Face identification using one spike per neuron: resistance to image degradation, Neural Networks, vol. 14, pp. 795-803. Delorme, A., Gautrais, J., van Rullen, R. & Thorpe, S. (1999). SpikeNet: a simulator for modeling large networks of integrate and fire neurons, Neurocomputing, vol. 26-27, pp. 989-996. Hubel, D.H. & Wiesel, T.N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex, J. Physiol, vol. 160, pp. 106-154. Sonka, M., Hlavac, V. & Boyle, R. (1998). Image processing, analysis, and machine vision, ed. 2. Song, S., Miller, K. D. & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity, Nature Neurosci., vol. 3, pp. 919-926. Bienenstock, E. L., Cooper, L. N. & Munro, P. W. (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex, J. Neurosci., vol. 2, pp. 32-48. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex, Neural Networks, vol. 12, pp. 961-974. Delorme, A., Perrinet, L., & Thorpe, S.J. (2001). Network of integrate-and-fire neurons using Rank Order Coding B: spike timing dependant plasticity and emergence of orientation selectivity. Neurocomputing, vol. 38-40, pp. 539-545. Thorpe, S. & Gaustrais, J. (1998). Rank Order Coding, In Bower, J. (ed), Computational Neuroscience: Trends in Research, Plenum Press, New York. Mattia, M. & del Giudice, P. (2000). Efficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical Synapses, Neural Computation, vol. 12, pp. 23052329. Connor, C. E. (2005). Friends and grandmothers, Nature, vol. 435, pp. 1036-1037. Wysoski, S.G., Benuskova, L., Kasabov, N. (2006). On-line learning with structural adaptation in a network of spiking neurons for visual pattern recognition. In: S. Kollias et al (eds), Intl. Conf. Art. Neural Net. - ICANN 2006, Lecture Notes in Computer Science, vol. 4131, pp. 61-70, Springer, Berlin/Heidelberg.

Copyright Copyright © [2006] Simei Gomes Wysoski, Lubica Benuskova. The author(s) assign to NACCQ and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to NACCQ to publish this document in full on the World Wide Web (prime sites and mirrors) and in

printed form within the Bulletin of Applied Computing and Information Technology. Any other usage is prohibited without the express permission of the author(s).