FPGA Implementation of Artificial Neural Network - IJETAE

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013)

FPGA Implementation of Artificial Neural Network Hardik H. Makwana1, Dharmesh J. Shah2, Priyesh P. Gandhi3 1

P.G. Student, 2Principal, 3Assistant Professor, L.C.I.T. Bhandu If the result of summation exceeds the threshold than the neuron ‗fires‘ i.e. output of threshold function will be positive else, output of threshold function will be negative. In case when actual output is far from the desired output than weights and bias are set accordingly so that next time when same pattern will come, it will produce correct output.

Abstract—In this paper, a Single Layer Perceptron for AND, NAND OR and NOR functions have been Synthesized and Implemented on Spartan 3 FPGA. It has been trained with Perceptron Convergence Algorithm. The Implemented Perceptrons have been verified by using Modelsim by creating an exhaustive testbench. The device utilization summary illustrates that, the implemented perceptron utilizes a few of the slices on FPGA, which makes it suitable for large scale implementation. Keywords—Artificial Neural Network, FPGA Implementation, Perceptron Convergence Algorithm, Single Layer Perceptron, VHDL

I. INTRODUCTION With increasing demands of Machine learning and parallel computing, Artificial Neural Network plays dominant role in the field of Artificial Intelligence. When going for implementation of Artificial Neural Network (ANN), several approaches are there and a number of efforts have done to implement ANN by using Analog VLSI up till last decade. At that time gate arrays were not much computationally efficient as today and a couple of years before. To take advantages of FPGA like fast prototyping, run time re-configurability and hence to save cost of a single unit as compared to ASIC, FPGA based Implementation of ANN has been started a few years before and running with its full pace in today.

Fig.1. Block Diagram of an Artificial Neuron[10]

B. Types of Artificial Neural Networks[12] An artificial neural network can be classified into following types according to number of layers, number of neurons in a layer, orientation of neurons in a layer, type of feedback, etc.  Feedforward Neural Network – The feedforward neural network was the first and arguably most simple type of artificial neural network devised. In this network the information moves in only one direction — forwards: From the input nodes data goes through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network. This types of networks are also sometimes referred as Perceptron, whose primary task is to classify the input patterns into two or more distinct groups.

A. A brief overview of Artificial neural Networks Artificial Neural Network (ANN) is an information processing paradigm consists of interconnection of massively parallel processing elements called ―neurons‖. The ―Artificial Neuron‖ which is inspired directly from biological neuron consists of a number of input vectors, followed by multipliers which are often called weights followed by a summer and a threshold function as shown in fig.1.1. In Biological Neural Network, a cell or Neuron receives responses from other Neurons via dendrites. When cell body contain enough action potential from dendrites, it fires and sends activation signal via axon to the dendrites of other neuron. Same way, Artificial Neural Network has been conceptualized. It gets electrical signal from inputs and their summation is passed through a threshed function.

 Radial Basis Function (RBF) Neural Network – Radial basis functions are powerful techniques for interpolation in multidimensional space. A RBF is a function which has built into a distance criterion with respect to a center.

672

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) TABLE I APPLICATIONS OF ARTIFICIAL NEURAL NETWORK [9]

RBF neural networks have the advantage of not suffering from local minima in the same way as MultiLayer Perceptrons. RBF neural networks have the disadvantage of requiring good coverage of the input space by radial basis functions.

Name of Field Aerospace

 Kohonen Self-organizing Neural Network – The SelfOrganizing Map (SOM) performs a form of unsupervised learning. A set of artificial neurons learn to map points in an input space to coordinates in an output space. The input space can have different dimensions and topology from the output space, and the SOM will attempt to preserve these. Automotive

 Learning Vector Quantization Neural Network – Learning Vector Quantization (LVQ) can also be interpreted as a neural network architecture. In LVQ, prototypical representatives of the classes parameterize together with an appropriate distance measure, provides a distance-based classification scheme.

Banking Defense

 Recurrent Neural Networks – Recurrent neural networks (RNNs) are models with bi-directional data flow. Recurrent neural networks can be used as general sequence processors. Various types of Recurrent neural networks are Fully recurrent network (Hopfield network and Boltzmann machine), Simple recurrent networks, Echo state network, Long short term memory network, Bidirectional RNN, Hierarchical RNN, and Stochastic neural networks.

Electronics

 Spiking Neural Networks – Spiking neural networks (SNNs) are models which explicitly take into account the timing of inputs. The network input and output are usually represented as series of spikes (delta function or more complex shapes). SNNs have an advantage of being able to process information in the time domain (signals that vary over time.

Entertainme nt Financial

C. Applications of Artificial Neural Networks The following table summarizes applications of an Artificial Neural Network. [9]

Insurance Manufacturing

673

Application High performance aircraft autopilots, flight path simulations, aircraft control systems, autopilot enhancements, aircraft component simulations, aircraft component fault detectors Automobile automatic guidance systems, warranty activity analyzers Check and other document readers, credit application evaluators Weapon steering, target tracking, object discrimination, facial recognition, new kinds of sensors, sonar, radar and image signal processing including data compression, feature extraction and noise suppression, signal/image identification. Code sequence prediction, integrated circuit chip layout, process control, chip failure analysis, machine vision, voice synthesis, nonlinear modeling Animation, special effects, market forecasting Real estate appraisal, loan advisor, mortgage screening, corporate bond rating, credit line use analysis, portfolio trading program, corporate financial analysis, currency price prediction Policy application evaluation, product optimization Manufacturing process control, product design and analysis, process and machine diagnosis, real-time particle identification, visual quality inspection systems, beer testing, welding quality analysis, paper quality prediction, computer chip quality analysis, analysis of grinding operations, chemical product design analysis, machine maintenance analysis, project bidding, planning and management, dynamic modeling of chemical process systems.

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) Medical

Oil and Gas Robotics

Speech

Security Telecommunica tion

Transportation

A perceptron decides whether an input belongs to one of two classes (denoted by C1 and C2) is shown in the Fig. 2. The single node computes a weighted sum of the input elements, subtracts a threshold (H) and passes the result through a hard limiting nonlinearity such that the output y is either +1 or -1. The decision rule is to respond class A if the output is +1 and class B if the output is -1. Useful technique for analyzing the behavior of nets such as the Perceptron is to plot a map of the decision regions created in the multidimensional space spanned by the input variables.

Breast cancer cell analysis, EEG and ECG analysis, and prosthesis design, optimization of transplant times, hospital expense reduction, hospital quality improvement, emergency room test advisement Exploration Trajectory control, forklift robot, manipulator controllers, vision Systems Speech recognition, speech compression, vowel classification, text to speech synthesis Market analysis, automatic bond rating, stock trading advisory systems Image and data compression, automated information services, realtime translation of spoken language, customer payment processing systems Truck brake diagnosis systems, vehicle scheduling, routing systems

Fig. 2 Single Layer Perceptron and its Signal Flow Graph[10]

The number of neural network applications, the money that has been invested in neural network software and hardware, and the depth and breadth of interest in these devices have been growing rapidly. In today‘s era, as deice sizes, power consumptions, cost, etc. decreases, demand for implementation of ANN on VLSI increases. Although several efforts to implement ANN by using Analog VLSI had done in last decade[1,2,3], but most of them fails because of high cost and lack of re-configurability offered by Analog VLSI. There are a number of types of ANNs are there[10], but when going for implementation of ANN, especially on FPGA, up till now implementation of Perceptrons and up to certain extent implementation of Spiking Neural Networks on FPGA has been reported.[6,7] For instance, many researchers have focused on hardware implementation of a simple neuron [4, 5]. Although Perceptrons are most widely used neural networks and as other architectures of neural networks can be easily derived from Perceptrons, we have chosen Single Layer Perceptron to implement on FPGA.

(a)

II. SINGLE LAYER PERCEPTRON Definition : ―A perceptron is a type of Neural Network which computes a weighted sum of its inputs, and puts this sum through a special function, called the activation, to produce the output‖. The activation function can be linear or nonlinear.[11]

(b) Fig. 3 Single Layer Perceptron that classifies input vector into two classes (a) Linearly Separable Patterns (b) Linearly Non- Separable (Non- linearly separable) patterns[10]

674

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) These decision regions specify which input, values result in class C1 and which values result in class C2 response. The perceptron forms two decision regions separated by a hyper plane as shown in fig. 3. It also maps actual output with desired output. If actual output differs from the desired output, perceptron generates an error signal, based on that weights and bias of neuron are updated.[10]

III. IMPLEMENTATION OF SINGLE LAYER PERCEPTRON

Perceptron Convergence Algorithm[10] The updating of weights and bias process follows throughout a n algorithm which is known as Perceptron Convergence Algorithm. It is explained as below. Variables and Parameters: X(n)=(m+1)-by-1 input vector =[+1,x1(n), x2(n), ….. xm(n)]T W(n)= (m+1)-by-1 weight vector = [b(n), w1(n), w2(n), ….. wm(n)]T b(n) = bias y(n) = actual response (quantized) d(n) =desired response η = learning rate parameter, a positive constant less than unity

Fig.4 Single layer Single Neuron Perceptron for AND, OR, NAND, NOR function.

Fig. 4 shows block diagram of Single layer Single Neuron Perceptron which can be implemented for AND, OR, NAND and NOR function. Input of Perceptron used here for all functions is 2x4 matrix is: X= [1 1 -1 -1; 1 -1 1 -1]; Target matrix is different for all functions as all functions are different. But it is 1x4 always matrix for 2x4 input matrix. Target matrix for AND function: T=[1 -1 -1 -1]; Target matrix for NAND function: T=[-1 1 1 1]; Target matrix for OR function: T=[1 1 1 -1]; Target matrix for NOR function: T=[-1 -1 -1 1]; Perceptron gets input from input matrix and maps actual output (which is initially taken as zero) with target output. The goal is to minimize the generated the error signal which is nothing but the difference between actual output and target output.. In this paper, the error signal becomes zero within only two iterations for learning rate =1 and threshold=0. For varying learning rate and threshold, the Algorithm takes different number of epochs to converge as shown in fig.5. It has been inferred from the fig that for larger the learning rate, smaller number of epochs (iterations) required. To implement the Perceptron on Hardware, it is essential that all the values are in the fixed point format. This is because floating point numbers are never synthesizable.

STEP 1: Initialization. Set w(0)=0. Then perform the following computation for time step n=1,2,… STEP 2 : Activation. At time step n, activate the perceptron by applying continuous valued input vector x(n) and desired response d(n). STEP 3: Computation of Actual Response of Perceptron. y(n)= sgn[(wT(n)x(n)]. Where sgn(.) is the Signum function. STEP 4. Adaptation of Weight Vector. Update the weight vector of Perceptron: W(n+1) = w(n) + η.[d(n) – y(n)].x(n) Where, d(n)= +1, if x(n) belongs to class C1 = -1, if x(n) belongs to class C2 STEP 5: Continuation Increment time step n by one and go back to step 2. Single Layer Perceptron can learn or be trained by using this Perceptron Convergence algorithm.

675

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) The target output for pattern [1 1] is 1 and the actual output is 0, an error signal is generated which is equal to target output-actual output. Here it is 1. Now weights of the perceptron are updated according to step 4 of perceptron convergence algorithm. Now, the second pattern will come that is [1 -1], for this pattern, perceptron calculates output with updated values of weights based on step 3. Again it compares the actual output with target output, error signal is generated and weighs are again updated. This process continues for all the patterns until the error vector becomes zero i.e. there is no error for any of the patterns. After than it stops updating the weights because error signal is zero which is often called convergence of perceptron. Presentation of all the patterns once to the perceptron is called an epoch. Perceptron takes as many number of epochs as it requires to get into conversion state. As the number of epochs are less, total time required to converge is less. In this case, the Perceptron has been converged in only two epochs.

Fig.5 Simulation Results for Perceptron for AND function with learning rate varying from 30 to 200 and threshold values varying from 0 to 4000.

Synthesis Results: Single Layer Perceptron for AND function has been implemented on Xilinx xc3s1000-4fg320 (Spartan 3) in VHDL. The fixed point arithmetic used for implementation. Device Utilization Summary for it is given below.

Fig. 5 illustrates that for larger the learning rate, smaller the value of total number of epochs required. Moreover the flat lines in the steps illustrate quick changes in weights for same total number of epochs. A. Perceptron for AND function The following table explains AND function. Which is when two inputs are high, than only the output becomes high otherwise it remains low.

TABLE III DEVICE UTILIZATION SUMMARY FOR PERCEPTRON FOR AND FUNCTION

TABLE II AND FUNCTION

data1 1 1 -1 -1

data2 1 -1 1 -1

Logic Utilization y1 1 -1 -1 -1

Here, data1 and data 2 are the input data and t1 is the target output corresponding to the input data. Initially y1 is 0. When input vectors i.e. [data 1, data2] given as input to the perceptron of fig. 4, it calculates output based on Step 3 of perceptron convergence algorithm, which is zero due to weights and biases are zero.

Used Available Utilization

Number of occupied Slices

38

7,680

1%

Number of 4 input LUTs

71

15,360

1%

Number of bonded IOBs

37

221

16%

These results are for learning rate=1 and threshold=0. Simulation Results in Modelsim: In order to verify the implemented perceptron for AND function, an exhaustive testbench has been created. The simulation results are explained in fig. 6.

676

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) TABLE III DEVICE UTILIZATION SUMMARY FOR PERCEPTRON FOR NAND FUNCTION

Logic Utilization Number of Slices

Used Available Utilization 40

7680

0%


71

15360

0%


37

221

16%

Simulation Results in Modelsim:

Fig.6 Simulation Results of Perceptron for AND function

Here, ‗data1‘ and ‗data2‘ are the inputs, ‗t1‘ is target output of perceptron, ‗y1‘ is the actual output, which is initially zero, and after one epoch, it becomes equals to target output. Here, ‗e1‘ is the error signal. ‗Alpha‘ is learning rate which is set to 1 and ‗theta‘ is threshold which is set to zero. Weights are ‗w1‘ and ‗w2‘ as shown in fig. 4 and ‗b‘ is the bias (fixed input). ‗W1new‘, ‗w2new‘ and ‗bnew‘ are the updated values of weights and bias respectively. Same way, we have implemented the Single :ayer Perceptron for NAND, NOR and OR functions. Their Synthesis and simulation results are shown below.

Fig.7 Simulation Results of Perceptron for NAND function

B. Perceptron for NAND function Single Layer Perceptron for AND function has been implemented on Xilinx xc3s1000-4fg320 (Spartan 3) in VHDL. Single Layer Perceptron for NAND function can be implemented same way as implemented for AND function. The only change is the target output. Based on that, weight changes in each epoch are different and device utilization for the entire architecture is different than previous case. Device utilization summary is shown in Table III

The variables are same as explained after fig. 6. Their values are different. C. Perceptron for OR function Single Layer Perceptron for OR function has been implemented on Xilinx xc3s1000-4fg320 (Spartan 3) in VHDL. Device utilization summary is shown in Table IV. TABLE IV DEVICE UTILIZATION SUMMARY FOR PERCEPTRON FOR OR FUNCTION

Logic Utilization Number of Slices

677

Used Available Utilization 10

7680

0%


17

15360

0%


21

221

9%

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) Simulation Results in Modelsim:

Simulation Results in Modelsim:

Fig.8 Simulation Results of Perceptron for OR function

Fig.9 Simulation Results of Perceptron for NOR function

As shown in fig.7, all the values of weights are either zero or positive integers, so only one bit is used for representation of weights. Other variables are same as explained after fig.6. their values are different.

The variables are same as explained after fig. 6. Their values are different.

D. Perceptron for NOR function Single Layer Perceptron for NOR function has been implemented on Xilinx xc3s1000-4fg320 (Spartan 3) in VHDL. Device utilization summary is shown in Table V.

The Single Layer Perceptron for AND, NAND, NOR and OR functions are implemented on Xilinx xc3s1000 4fg320 (Spartan 3). Simulation results revels that, after one epoch, error vector becomes zero and hence the perceptron converges at the end of one epoch. Simulation for second epoch is done for testing purpose. In second epoch, error signal remains zero and hence, no weight update takes place. Device Utilization Summary illustrates that the implemented Perceptron utilizes a few number of slices which is ideally 0%. It means that it is suitable for Large scale implementation as well as for implementing Multi Layer Perceptrons on FPGA where device utilization is a crucial factor.

Logic Utilization

IV. CONCLUSION

Used Available Utilization

Number of Slices

22

7680

0%


39

15360

0%


29

221

13%

678

International Journal of Emerging Technology and Advanced Engineering Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 3, Issue 1, January 2013) [6]

Rezvani, R., Katiraee, M., Jamalian, A. H., Mehrabi, S. H., & Vezvaei, A. (2012, August). A new method for hardware design of Multi-Layer Perceptron neural networks with online training. In Cognitive Informatics & Cognitive Computing (ICCI* CC), 2012 IEEE 11th International Conference on (pp. 527-534). IEEE. [7] Upegui, A., Peña-Reyes, C. A., & Sanchez, E. (2005). An FPGA platform for on-line topology exploration of spiking neural networks. Microprocessors and Microsystems, 29(5), 211-223. [8] Lippmann, R. (1987). An introduction to computing with neural nets. ASSP Magazine, IEEE, 4(2), 4-22. [9] Beale, Mark, and Howard Demuth. "Neural network toolbox." For Use with MATLAB, User's Guide, The Math Works, Natick (1998). pp 1-6. [10] Simon Haykin ―Neural Networks A comprehensive Foundation‖ second edition, Prentice Hall, 2007, pp. 158-161. [11] Mu-Song Chen, Ph.D. Thesis ―Analysis And Design Of The MultiLayer Perceptron Using Polynomial Basis Functions‖ The University Of Texas At Arlington December 1991. [12] http://electronicsbus.com/artificial-neural-networks-ann-typesartificial-intelligence/

REFERENCES [1]

[2]

[3]

[4]

[5]

Bapuray,D. Vadiraj,.R. Rakesh, N and Vinayak, U. 2011 Design And Analog VLSI Implementation of Artificial Neural Network. International Journal of Artificial Intelligence and Applications (July 2011) 96-102 Koosh, V. F., & Goodman, R. M. (2002). Analog VLSI neural network with digital preturbative learning. Circuits and Systems II: Analog and Digital Signal Processing, IEEE Transactions on, 49(5), 359-368. Carvajal, G., Figueroa, M., Sbarbaro, D., & Valenzuela, W. (2011). Analysis and Compensation of the Effects of Analog VLSI Arithmetic on the LMS Algorithm. Neural Networks, IEEE Transactions on, 22(7), 1046-1060. Noory B , Groza V , "A reconfigurable approach to hardware implementation of neural networks ", Conference on Electrical and Computer Engineering , Vol 3 , PP: 1861 - 1864 , May 2003. Chujo N., Kuroyanagi S., Doki S., Okuma S., ―An iterative calculation method of the neuron model for hardware implementation‖, IEEE IECON, Vol. 1, pp. 664-671, 2000.

679