License plate recognition system is used to identify vehicles by their number ... artificial intelligence (AI), fuzzy logic (FL), neural network (NN) and genetic ...
Chapter 1 Introduction
Transport facilities have been increased abruptly in the last decade and traffic load has also been considerably increased. It becomes essential to monitor the road traffic which is dominant among all the means of transport. The vehicles are generally identified by their unique license plate numbers which are commonly referred as vehicle numbers in Indian context. The recognition or identification of vehicles can be achieved by recognizing the characters available in number plates of various vehicles. Closed circuit cameras or special purpose cameras are generally used to capture the images of number plates that can be used in identification process. Automatic license plate recognition systems would greatly help in automatic vehicle monitoring and surveillance applications; automatic toll systems, automatic parking systems etc. 1. 1. Automatic License Plate Recognition License plate recognition system is used to identify vehicles by their number plates using some image processing theory and soft computing tools. The systems can be of great help in several applications such as traffic monitoring systems, highway toll collection and LPR systems for an Intelligent Transport System. In 1976, the Automatic Number Plate Recognition (ANPR) system was introduced at the Police Scientific Development Branch in the UK (Roberts et al., 2012). This paper presented an LPR system as an application of computer vision that utilized the theory for building artificial system. Numerous algorithms could be found in literatures for automatic license plate recognition systems which are used in the process of license plate recognition for different countries. There are different systems for various countries because varying climatic conditions e.g. winter, type of plate, alphabet etc. for different countries. Kranti et al. (2011) presented an LPR system addressing the problem of an LPR system for license plates of typical developing country India. Large demand of transport monitoring systems require the information to be processed digitally with the help of suitable information technology (IT) techniques and soft computing tools such as artificial intelligence (AI), fuzzy logic (FL), neural network (NN) and genetic algorithm 1
(GA). Various methods available for LPR are implemented with the help of these techniques and tools and the performance of existing systems are being improved continuously. The recognition techniques have been developed and number plate recognition systems are today used in various traffic and security applications, such as parking, access and border control, and tracking of stolen vehicles. The license plates are monitored in certain duration of time while it is parked in parking system. When a vehicle enters the gate of parking area the number of license plate or number plate is automatically acquired, captured, recognized and stored in database. At the time of exit, number plate is recognized again and compared against available in the database of number plates. ANPR systems installed at various places can detect and monitor border crossings, access control and surveillance application. Most of the number plate localization algorithms require a number of procedures that causes increased computational time or considerable execution time which can be reduced by applying less and simpler algorithms. The results of LPR systems depend heavily on quality of the images of number plates and hence image enhancement plays an important role in LPR systems. If the quality of image is not good then the identification results would not be proper. The poor image quality may be due to several factors such as noisy pictures, noisy source of image acquisition system, noisy channel etc. The image capturing method should be properly implemented so as to avoid the possibilities of noise signals in images. Mathematical algorithms were implemented by Ondrej (2007) in his thesis work. Size consideration is another very important part of LPR systems. Adjustment of size of images becomes difficult in the case of fast moving vehicle since blur or other kind of distortion is added if the speed if high. Basically, LPR systems consist of the following main stages:
Image acquisition
Image Preprocessing
Plate localization and character detection
Feature extraction
Classification
Recognition.
2
There are several research contributions in existing literatures on LPR systems and their implementation. However, all of the methods are mainly divided into some major categories, as under:
Principal components analysis (PCA) based system
Independent component analysis method
Self-organizing maps system
Hidden Markov models for LPR
Neural network based LPR systems
1. 2 Basics of Digital Image Processing Image carries visual representation of an object or scene or a person, which is generated on the basis of absorption and reflection theory. When a light is incident on an object some amount is absorbed and remaining is reflected; amount of reflection causes the visual representation of an image. A moving image or a scene is a series of images projected rapidly and eye has capability to see it as integrated information. An image is formed by amount of light reflected or refracted. The reflected light is received or collected on a screen or a photographic plate; or it may be received by human eye (Gonzalez et al., 2009, Sonka et al., 2008). Following are different types of images based on the image acquisition:
Graphical images
Optical images
Perceptual images
Mental images.
1.2.1 Graphical Images If an image is created using lines or certain shapes with the help of commands then this is called graphical image. The commands may be mathematical statements written using a computer and any suitable programming language. Computer graphics studies concepts of graphical images which also includes characters and other attributes of images. Standard tools such as rectangle, circle, triangle etc. are used to create graphical images.
3
This can be done by using simple application of MS-paint also. Figure 1.1 shows an example of graphical image.
Fig.1.1: A graphical image.
1.2.2 Optical Images Optical images are produced with the help of lens or mirror system collecting the reflected, refracted, or diffracted light waves and object is formed. There are two main types of optical images namely real and virtual. In case of real image, the light rays are brought to a focus at the image position actually and the image is made visible on a screen or a sheet of paper. The real images are represented by a camera lens on film or a projection lens on a moving screen. Virtual images are made by rays that do not actually come from where the image seems to be (Pratt, 2007; Castleman , 2007; Gonzalez et al., 2009). An optical image can be seen in Fig. 1.2 that has been captured through an optical camera.
Fig.1.2: An optical image of license plate or number plate of vehicle.
4
1.2.3 Perceptual Images Images perceived by human being are called as perceptual images. These images may be true images or result of illustrations; and may be assumption of some images at far. An old woman is perceived as a young woman then only one image can be seen by the viewer. Fig.1.3 shows a perceptual image.
Fig.1.3: A perceptual image.
This type of figure or image is perceived by the brain and interpreted accordingly. The perception does not change until attention is made to different region or objects. The meaningful perception remains as static image.
1.2.4 Digital Images The captured or scanned images are stored in computer as digital images. An image can be represented as two dimensional data which consists of pixels. A pixel is short form of picture element or picture cell which carries some gray scale value also called as intensity value of the pixel. A picture or image consists of a number of pixels having some gray scale values. Mathematically, an image is a two dimensional (2 D) function, f(x, y), where x and y are the coordinate values in spatial domain or plane; and the magnitude of f(x, y) is the intensity value of pixel at (x, y).An example of real time image can be seen in Fig. 1.4 along with the pixel representation of the image. If x, y and the magnitude of f(x, y) in an image are discrete quantities then the image is said to be digital image. Image may be represented as two dimensional matrices whose elements are intensities of pixels present in image. Almost all image processing related operations operate on these pixels either in
5
spatial domain or in frequency domain or transform domain (Gonzalez et al., 2009; Pratt, 2010). The function f(x, y) can be expressed as:
( , )=
(0,0) ⋮ ( − 1,0)
⋯ ⋱ ⋯
(0, (
− 1)
⋮ − 1,
(1.1) − 1)
Each digital image has certain finite number of elements characterized by some coordinate values and intensity value. The coordinate indicates the position of pixel in an − 1,
image. In Equation (1) the image elements
− 1) represent the maximum
number of resolution starting from f (0,0).
(a)
(b) Fig.1.4: Digital Image. (a) An original image (b) Pixel coordinates in an image.
6
First element is specified by coordinates (0, 0) is the top leftmost element of an image as shown in Fig. 1.4. The coordinate begins from (0,0) and end as (0, N) as last element of that row then next row starts with (1,0) and ends by (1,N).
1.2.5 Color Image Normally, images are captured as color images or RGB images. The color images consist of main three color components namely red (R), green (g), and blue (B). If in monochrome or gray scale image, the intensity value is expressed in 8-bits for a pixel then in color image each pixel requires 24 bits including three components. Therefore, added dimension increases storage requirement as well as processing time since more operations need to be performed. An image of size (m x n) will be expressed as (m x n x3) because of three color components of the RGB image. RGB image can be very easily converted into gray scale image using a simple command of MATLAB as: >>rgbtogray (color image); The actual color of a pixel is determined by the combination of three components. For example a pixel having color components (0, 0, 0) is a black pixel and similarly pixel whose color components are (1, 1, 1) will be a white pixel. We have to remember here that (0, 0, 0) is not representing the pixel coordinate but the intensity values. In a color image (m, n, 3), there will be three components which are written as: a. Red component (m, n, 1) b. Green component (m, n, 2) c. Blue component (m, n, 3) An example of color image can be seen in Fig. 1.5 with the red, green and blue components. 1.2.6 Gray scale Image A gray scale image can be seen in Fig. 1.6 in which the brightness or intensity can be represented as only one component having some value. The pixel intensity of 8 bit images will be in the range of 0 to 255, total of 256 levels. In most of operations related to digital image processing, gray scale images are used for simpler and faster computations (Gonzalez et al., 2009 ; Jain, 1994).
7
Fig.1.5: A color image.
Fig. 1.6: A gray scale equivalent image of color image of Fig. 1.5.
1.3 Image Resolution and Aspect Ratio A 2D (two dimensional) image is specified by two common characteristics: image resolution and its aspect ratio. The brief description of these characteristics is given as:
Resolution: This represents pixel count in an image. In M x N image M is the number of pixels in horizontal direction and N is the number of pixels in vertical direction. The image resolution is defined as total number of pixels present in an image. Higher the image resolution more is the image content or details. In 256 x 256 size image there will be 65536 pixels and hence its resolution is measured in terms of 65536 pixels. The resolution is also expressed in dpi (dots per inch) i.e.
8
how many pixels are there per inch area. Resolution of an image may be computed by:
= ×
Aspect ratio: The aspect ratio relates width of an image to its height. This is usually expressed as P: Q, which means that, the ratio of number of pixels in width of image to that of number of pixels in height is P: Q.
1.4 Components of Digital Image Processing Image processing is set of tools or techniques which involve converting an analog or continuous image into digital form and then performing some operations upon it so that the image quality may be enhanced or some useful information could be extracted. Following are various purposes of using digital image processing for different research and development; and related activities:
Visualization: The images in remote sensing and satellite application which are not properly visible and analysis is difficult then the human visual appearance and perception as well as machine perception can be increased.
Image Enhancement and Restoration: If the quality of images is poor or we need to analyze the type and amount of noise in an image then enhancement and restorations processes respectively help to large extent.
Content Based Image Retrieval: This is achieved for getting images and related information of interest.
Measurement of pattern: The various patterns in image are measured and analyzed. This is useful particularly in fingerprint matching or forensic applications.
Pattern Recognition: Various types of objects or patterns can be recognized using image processing operations. This is very much important in biometrics as well as automated diagnosis in medical imaging.
9
Information Security and Access Control: Image processing used in biometrics greatly helps in access control in restricted areas and information security is also ensured.
Image processing is generally applied in two different ways namely analog image processing and digital image processing. Analog images are natural images which are originally captured through an acquisition system. Analog image processing also referred as visual image processing can be used for the hard copies like printouts and photographs. Various visualization techniques are developed by researchers and analysts and the images are interpreted accordingly.
Digital image processing and related
techniques help in manipulation of the digital images with the help of computers. There are numerous research papers available in literatures that address various applications of digital image processing. An extensive literature survey suggests that the robust approach of any application is yet to be achieved and designed. A generalized theory is very difficult to be established. However, with increasing use of the modern computing tools and advent of computers, this is expected to improve responses for the image processing results. Following are main components of a general purpose image processing:
Image Acquisition: Deals with capturing the images or samples.
Image Enhancement: Deals with the improvement of quality of images.
Image Representation: Deals with different ways in which image can be represented mathematically, graphically and statistically.
Image Transformation: Used to transform the input image from one domain into another, e.g., an image in spatial domain can be converted into frequency domain by using Fourier transform.
Image Restoration: Deals with analysis and modeling of the different types of noise mixed in images.
Color Image Processing: Various color spaces and formats are covered.
Image Compression: Used to reduce the size of image or reduce redundancy without any significant change in the inherent content of the image.
10
Morphological Image Processing: Used to represent or covert into suitable forms so that edges can be easily recovered. These operations are generally used with image segmentation.
Image segmentation, Representation and Description: Selected region of interests can be extracted and various boundaries, edges and other similar information could be obtained.
Object Recognition: Deal with pattern recognition and matching.
Fig. 1.7 illustrates an overview of image processing beginning with capturing of image to the final output as a consequence of any of the digital image processing operations. An image is captured by using a camera or sensor and sampling and quantization processes are used for storing the image in digital form in computers. Then we can process the image by using any of the components of image processing such as image enhancement, image segmentation and object recognition (Gonzalez et al., 2009; Sonka et al., 2008).
1.5 Sampling and Quantization A digital image is obtained by converting it from continuous analog image and stored in a digital form. The process of conversion from analog image into digital one is known as sampling and quantization. The continuous function of image is represented by a finite set of discrete values or observations using digitization process. For a continuous image, f(x, y) the process of digitizing the coordinate value is called sampling and digitizing the amplitude or intensity values is called quantization. A digital image is defined as set of discrete values representing intensity values of various coordinate points representing pixels. The analog signal, if expressed as X(t) is continuous in both time and amplitude. The sampling operation results in a signal that is still continuous in amplitude but discrete in time. A digital signal is formed from a sampled data or signal by encoding the sampled values into a finite set of values. The process of sampling and quantization can be seen in Fig.1.8.
11
Object
Image acquisition
Sampling and quantization
Digital storage system
Image transformation Image enhancement Image restoration Image segmentation Image compression Object recognition
Digital computer
Post -processing
Display
Fig.1.7: Basic architecture of digital image processing.
1.6 Statement of the Problem Automatic vehicle license plate detection and recognition is a key technique in most of traffic related applications. Numerous methods, techniques and algorithms have been developed for license plate detection and recognitions. Very few research contributions addressed the problem in context with Indian license plate recognition. Plate position affects the performance of license plate recognition (LPR) systems which has not been discussed and considered by important research papers. Some characters appear as similar characters in presence of noise signals such as ‘T’ and ‘I’; ‘E’ and ‘F’; ‘O’ and ‘Q’, etc. These major problems have been solved in present work.
12
Analog signal X(t)
Sampling
Sampled signalX(k)
Quantization
Discrete
Encoding
Digital
Fig. 1.8: Sampling, quantizing and encoding.
1.7 Organization of Thesis The present chapter introduced basic concepts of digital image processing, discussed an overview of license plate recognition systems. The statement of the problem is also presented. Chapter 2 highlighted the existing research contributions in the license plate recognition on various methods and their implementation. Challenges, findings and research scope were identified in this chapter. The concept of neural network and its training; and fuzzy logic was reported in Chapter 3. Chapter 4 discusses theoretical background of license plate extraction, character segmentation and detection; and character recognition. The proposed methodology and implementation algorithms have been discussed in Chapter 5. Results and discussion are reported in Chapter 6. Chapter 7 presents conclusions and future scope of the work done.
13
Chapter 2 Related Research
An extensive literature review has been made in the field of license plate recognition of various types of number plates of different countries such as China, Japan, Greek, USA France etc. The research contributions have suggested several methods of number plate recognition. Some of the major contributions are reported in this chapter along with their findings and limitations. Based on the limitations reported, problem has to be identified that would be addressed in the present work. 2. 1. Related Research As we could see the introduction of license plate authentication that a vital part in transportation is played by vehicles. Since the usage of vehicles has been increasing on account of population growth and human needs and therefore control of vehicles is becoming a real challenge to address. License Plate Recognition (LPR) that is used to identify vehicles by their license plates is a kind of automatic vehicle identification. Image acquisition, license plate detection, character segmentation and character recognition are four major phases of License Plate Recognition. Several areas including traffic volume control, unsupervised park monitoring, traffic law enforcement and auto toll collections
on
highways
extensively
use
license
plate
recognition
applications. Now, a report of already implemented research work is presented here. Adorni et al. (2002) developed an efficient low-level vision program design using Sub-machine-code genetic programming.
The work aimed at exploiting the intrinsic
parallelism of sequential CPUs for real-time applications as a variant of genetic programming (GP) for exploiting the intrinsic parallelism of sequential CPUs. Licenseplate recognition system is taken into consideration as a case study to show the potential of the approach. Very low-resolution binary patterns were classified in such an application along with preliminary results obtained using the design for license-plate extraction algorithm. However, the work still is very preliminary and requires a deeper analysis for pattern classifiers to generate programs that are both accurate and efficient. The problem remains to improve the results using suitable design algorithm which can 14
better exploit information provided by the low-level license-plate segmentation achieved by the GP-generated program. An even more interesting alternative might be developing a more general segmentation algorithm that could detect characters present anywhere in an image. Akoum et al. (2010) reported comparative results by designing
two neural
networks namely Hopfield and multi layer perceptron “MLP” based system for license number plate recognition. The system was tested over 400 images of license plates where the experimental result shows the ability of Hopfield Network to recognize correctly characters on license plate more than MLP architecture. The ability of Hopfield Network was tested to recognize correctly characters on license plate more than MLP architecture which has a weaker performance. A biggest problem in Hopfield design is the processing time for pictures of 42x24 pixels (90 seconds average, versus only 3 seconds in the case of pictures of 21x12 pixels). It could be improved by decreasing the number of neurons in the hidden layer but again the performance might degrade. Tahir et al. (2010) suggested an intelligent algorithm for recognizing the Pakistani license plate numbers in which images were captured and for detecting license plates it uses vertical edge matching based technique and after detecting plate character and numbers were extracted by horizontal and vertical scanning. The effectiveness of the algorithm is investigated over huge dataset captured in various illumination conditions. Recognition rate of about 92 % was found. However, the processing time for real time applications could be reduced and segmentation process could be improved to boost the recognition rate. A new reliable approach for Persian license plate detection was proposed in Ashtari et al. (2011). It was applied on color images using vertical search method for detecting plates and after detecting plates, plate characters are recognized by a support vector machine (SVM) with a homogeneous polynomial kernel of degree five. The result was based on images from speed control cameras on high ways that demonstrated the performance, exactness, speed and reliability of the method. Finally, the proposed method provided 96 % performance detection rate. High speed ability to setup and run on microprocessors and process color images was most important point without any resizing and converting. But the system suffered with large processing time because it avoided resizing of image.
15
Anagnostopoulos et al. (2006) proposed an intelligent traffic management through MPEG-7 vehicle flow surveillance where the algorithm extracts the number plate and character by using novel adaptive image segmentation technique, sliding concentric windows (SCW) and connected component analysis. The algorithm was tested with 1334 natural scene gray level vehicle images of different backgrounds and ambient illumination. The license plates properly segmented were 1287 over 1334 input images (96.5%). The overall rate of success for the proposed LPR algorithm is 86.0%. However, the success ratio is increased when the LPR software is accompanied with auxiliary infrared illumination units. Lorita et al. (2011) developed a multiple vehicles’ license plate tracking and recognition via i9sotropic dilation to classify moving vehicles with different backgrounds and varying angles by using binary large object (blob) and connected component analysis (CCA) techniques. Isotropic dilation describes any two region of interest (ROIs) that is discontinuous are typically treated as separate blobs and finally a multi-layer feed-forward back-propagation Neural Network is used to train the segmented and refined characters. It provides good results in different environmental factors such as the shadow effect and insufficient of light level. The system still needs to be enhanced in the future to overcome the mentioned restrictions. Clemens et al. (2007) proposed a LPR on an embedded DSP-platform and AdaBoost approach is used for localization purpose which was embedded in DSP platform for processing a video stream in real-time. Detected license plates are segmented into individual characters by using a region-based approach. Character classification is performed with support vector classification. The major advantage the system is real-time capability and that it does not require any additional sensor input. It focuses mainly on an optimization to the detection algorithm implementation and not on region segmentation methods. Babu et al. (2008) presented an efficient geometric feature based License Plate Localization and recognition for Indian License plates. It uses geometry-based method and a thin window scanning method for segmentation of characters and an artificial Neural Network (ANN) is designed to recognize characters of letter and characters of numbers. This work suggests a robust time algorithms of license plate location, segmentation and reorganization of the characters present in the located plate. The major limitations of the proposed technique include constraints on camera location and 16
positioning; and limited success in less-than-ideal conditions. FPGA implementation of a license plate recognition system on chip (SoC) was seen in Bellas et al. (2006) using automatically generated streaming accelerators. FPGA platforms provide the hardware and software infrastructure for building a bus-based SoC that meet the applications requirements. Hardware accelerators that provide application-specific extensions to the computational capabilities of a system are an efficient mechanism to enhance the performance and reduce the power dissipation. FPGA-based LPR system may be used in law enforcement. An overall LPR application speed up was achieved enabling real-time functionality under realistic road scenes. The method focuses on kernels that can be parallelized while leaving the sequential code to be executed by the scalar processor. This facilitates the field upgrade of such a system with new algorithms and new accelerators without costly re-designs of the system. Besides the LPR, the system can be used in a multitude of applications that can naturally be expressed as a series of streaming filters. Broumandnia et al. (2005) implemented a Farsi LPR system as an automatic inspection of transport systems that found its potential applications to areas such as automatic toll collection, traffic law enforcement and security control of restricted areas. The work suggests an automatic license plate recognition system for Persian license plates. This system worked under variable illumination, variable size of plate and dynamic backgrounds. This system was implemented with help of Tehran control traffic Company and performance of the LPR system has been tested on 400 vehicles images which captured under various sizes of plate and variable illumination conditions. The rate of success recognition is 95%. The average speed of this system is 2 sec. However, increasing speed up remained a big challenge for an improved LPR system. Cano et al. (2003) suggested an LPR for plate segmentation designed to work in a wide range of acquisition conditions, including unrestricted scene environments and light; perspective and camera-to-car distance. Although this novel text-region segmentation technique has been applied to a very specific problem, it is extensible to more general contexts, like difficult text segmentation tasks dealing with natural images. According to visual inspection of the whole set of 131 test images, the segmentation system has correctly located all the plates but two. Due to the unrestricted nature of the test set this can be considered a very promising result. The computational resource 17
demand of this segmentation technique is currently the main drawback, taking an average of 34 seconds the processing of a single 640x480 image. Chang et al. (2004) proposed an automatic LPR which plays an important role in numerous applications and a number of techniques. Most of the techniques worked under restricted conditions, such as fixed illumination, limited vehicle speed, designated routes, and stationary backgrounds. The proposed LPR technique consists of two main modules: a license plate locating module and a license number identification module. The former characterized by fuzzy disciplines attempts to extract license plates from an input image, while the latter conceptualized in terms of neural subjects aims to identify the number present in a license plate. Identifying license number over 1065 images, license plates have been successfully located and 47 images have been failed to identify the numbers of the license plates located in the images and hence the identification rate of success is 95.6%. Since color and edge are two fundamental features of license plates, the color edge detector introduced in the locating module is readily adapted to other color schemes by replacing the color parameters embedded in the detector. The locating and identification modules both perform in somewhat of a hybrid top-down and bottom-up manner. However, in order to make our techniques applicable to real-time applications in less restrictive working conditions, the replacing firmware components with hard-wired ones and using parallel machines should be used. Characters of license plates from video sequences were extracted in Cui et al. (1998) using a new approach to extract characters on a license plate of a moving vehicle, given a sequence of perspective-distortion-corrected license plate images. The extraction of characters as a Markov random field (MRF) is modeled where the randomness is used to describe the uncertainty in pixel label assignment. Now, the extraction of characters is formulated as the problem of maximizing a posteriori probability based on a given prior knowledge and observations. A genetic algorithm with local greedy mutation operator is employed to optimize the objective function. This approach provides better performance than other single frame methods. The prior knowledge which promotes the consistency between adjacent pixels can be represented in terms of the clique functions. The experimental results have shown better performance than other single-frame methods. By using the greedy mutation operator, computation cost can be reduced but it is still far 18
from the real-time requirement. Deb et al. (2008) proposed parallelogram and histogram based vehicle license plate detection to analyze road images which often contain vehicles and extract license plate (LP) from natural properties by finding vertical and horizontal edges from vehicle region. The method consists of three main modules: segmentation technique named as sliding concentric windows (SCW) on the basis of a novel adaptive image for detecting candidate region; refining by using HSI color model on the basis of using hue and intensity in HSI color model verifying green and yellow LP and white LP, respectively; and verify and detect VLP region which contains predetermined LP alphanumeric character by using position histogram. The success detection rate of license plate is up to 82.5%. The proposed method is sensitive to the angle of view and environment conditions, e.g. the plate of car license is defiled etc. These problems can be solved by using image processing operation as like as histogram equalization, high dynamic range imaging (HDR) etc. Dlagnekov et al. (2005) suggested the use of AdaBoost algorithm for LPR and LP detection. This is well explored problem with many successful solutions. The main goal is to evaluate how well object detection methods used in text extraction and face detection apply to the problem of LPR. A strong classifier is trained by the AdaBoost algorithm to classify parts of an image within a search window as either license plate or non-license plate. The algorithm has shown itself to be promising for the task of license plate detection. A simple and easy way to improve the ROC curve would be to use even more classifiers. The features currently used are also not as independent as they should be. The detected images of license plates being 45 × 15 are not very large and performing OCR on them will most likely not be very effective without first applying some sort of super-resolution technique that uses several frames from the video data. A neural network based artificial vision system for license plate recognition was conceptualized in Draghici et al. (1997) to analyze the image of a car given by a camera, locate the registration plate and recognize the registration number of the car. This paper described various practical problems encountered in implementing this particular application and the solutions used to solve them. The main features of the system presented are: controlled stabilityplasticity behavior, controlled reliability threshold, both off-line and on-line learning, self assessment of the output reliability and high reliability based on high level multiple 19
feedback. The system was designed using a modular approach which allows easy upgrading and/or substituting of various sub-modules thus making it potentially suitable in a large range of vision applications. The performances of the system make it a valid choice among its competitors especially in those situations when the cost of the application has to be maintained at reasonable levels. Duan et al. (2005) built a vehicle LPR (VLPR) system to meet the requirement due to a huge number of vehicles. An automatic VLPR system was used to read Vietnamese VLPs’ registration numbers at traffic tolls. The system consists of three main modules: VLP detection, plate number segmentation, and plate number recognition. In VLP detection module, an efficient boundary line-based method combines the Hough transform and contour algorithm. This method optimizes speed and accuracy in processing images taken from various positions. Now, horizontal and vertical projection is used to separate plate numbers in VLP segmentation module. Finally, each plate number is recognized by OCR module implemented by hidden Markov Model. The system can also be applied to some other types of VLPs with minor changes. The system performs well on various types of Vietnamese VLP images, even on scratched, scaled plate images. A number of texture-based approaches can be combined with machine learning methods to evaluate plate candidates to improve the accuracy and the speed of the algorithm. Ebrahimi et al. (2007) suggested LPR based on multi agent system using a new algorithm for vehicle license plate location. The algorithm was tested with 400 natural-scene gray level vehicle images of different backgrounds and ambient illumination. The camera focused in the plate, while the angle of view and the distance from the vehicle varied according to the experimental setup. The algorithm was tested in natural-scene gray-level vehicle images of different backgrounds and ambient illumination. The camera focused in the plate, while the angle of view and the distance from the vehicle varied according to the experimental setup. The performance increased. Fang et al. (2011) proposed LPR using sub-image fast independent component analysis to solve the problem that current license plate recognition methods, such as template matching and neural network computing that need a large number of samples and large amount of computation. This sub-image fast independent component analysis method helps to obtain the local feature of the image with a small amount of 20
computation. In order to obtain better recognition results, in the stage of character segmentation, segmentation is carried out. The relative coordinate dichotomy (RCD) was proposed for character segmentation and achieved good segmentation results. Experiments have shown that this method could extract local feature of characters very effective and an excellent recognition rate has been reached for character recognition. This work could also be tested for Chinese character recognition as an effective recognition method for the similar characters. Faqheri et al. (2009) designed real time Malaysian automatic LPR (M-ALPR) using hybrid fuzzy techniques for car plates that include the font and size of characters that must be followed by car owners. This paper suggests a new methodology to segment and recognize Malaysian car license plates automatically to solve the problem of segmenting different length licenses such as license with different number of character and number. There are two main objectives for this paper: first is to develop fuzzy rules to recognize the segmented characters and numbers from the same input-sets, which is the same size without overlapping between the characters and numbers sets. The work emphasizes to recognize non-standard plates by template matching theorem. The results yield 90.4% recognition accuracy and fuzzy based required 1.7 seconds and template matching based took 0.75 seconds to perform the recognition. The adaptability factor of the hybrid method is also discussed. The hybrid method has achieved good performance for real time recognition system. This factor is often ignored in performance evaluation or comparison, which is the appropriate criterion for an algorithmic assessment. A hybrid system would be tested on the standard database. Franc et al. (2005) achieved LP segmentation using hidden Markov chains for segmentation of a line of characters in a noisy low resolution image of a car license plate. The Hidden Markov chains are used to model a stochastic relation between an input image and corresponding character segmentation. The segmentation problem is expressed as the maximum a posteriori estimation from a set of admissible segmentations. The efficient algorithm for estimation based on dynamic programming is derived. The proposed method was successfully tested on data from a real life license plate recognition system. The proposed method is able to segment characters correctly even in images of very poor quality. The method achieved an error rate 3.3% estimated on data captured by a real LPR system. 21
Sun et al. (2006) proposed a new recognition method of vehicle license plate based on genetic neural network using Back Propagation (BP) neural network taking local minimum in the training process. The training of the network was finished in two steps. The GA was firstly used to make a thorough searching in the global space for the weights and thresholds of the neural network, which can ensure they fall into the neighborhood of global optimal solution. Now, in order to improve the convergence precision, the gradient method was used to finely train the network and find the global optimum or second-best solution with good performance. The method can save the time of training network and achieve a highly recognition rate. Its convergence is very slow and can easily plunge into local extremum. The selection of the initial weights and thresholds are random. Experimental results show that both the precision and robustness are obviously improved. Using the eigenvector while considering both structure feature and statistic feature, it can greatly improve the recognition rate. Hu et al. (2003) presented a novel approach for LPR using subspace projection and probabilistic neural network which combines subspace projection with probabilistic neural network to improve the recognition rate; and to identify low-dimension test samples which are obtained from actual license plate images by subspace projection. The recognition rate is obviously better than that of the conventional template matching method. Future work of this work aims at extending to other fields of image recognition. Kahraman1et al. (2003) implemented license plate character segmentation based on the Gabor transform and vector quantization. A novel algorithm was used by using the Gabor transform in detection and local vector quantization in segmentation. The method is more practical and efficient to analyze the image in certain directions and scales utilizing the Gabor transform. The filter response gives a rough estimate of the plate boundary. Accuracy of LPR detection was found as 98% whereas LPS (License plate segmentation) produced 94.2% accuracy. But this method is computationally expensive. Further scope of work aimed at integration of this method with an optical character recognition system and its performance analysis. Extraction of license plate region in automatic license plate recognition was performed by
Kannan et al. (2010) using image processing to identify a vehicle by reading its
license plate. The camera captures the image of pre-defined resolution and passes it to the 22
software module. This module forms the heart of the entire system and analyzes the input image, identifies the location of the license plate, segments the characters on it and recognizes the characters. The plate region is extracted with the help of connected components in the image. The characters in the license plate were segmented using labeling method. The plate extraction was successful for 91% of the test images. Successful design and implementation was achieved as the key element of the system for the actual recognition of the license plate region. Kasaei et al. (2011) developed license plate detection and recognition system for Persian license plates. An image-processing technique is used to identify a vehicle by its license plate. License plate location acts as an important stage in vehicle license plate recognition for automated transport system. Main stage of the work is the isolation of the license plate from the digital image of the car obtained by a digital camera under different circumstances such as illumination, distance and angle. The method begins with preprocessing and signal conditioning then license plate is localized using morphological operators. Template matching is used to recognize the digits and characters within the plate. This system was implemented with help of Isfahan Control Traffic organization and the performance was found 98.2% of correct plates identification and localization and 92% of correct recognized characters. High accuracy and robustness were reported and the method could also be applicable for other applications in the transport information systems, where automatic recognition of registration plates and signs is often necessary. This system is customized for the identification of Iranian license plates and tested over a large number of images. The system can be redesigned and tested for multinational car license plates. Deb et al. (2009) implemented vehicle license plate detection method using sliding concentric windows and histogram. A new method is adopted to analyze the images which often contain vehicles and extract LP from natural properties by finding vertical and horizontal edges from vehicle region. The proposed method consists of three main stages a novel adaptive image segmentation technique, color verification for candidate region by using HSI color model and decomposing candidate region which contains predetermined LP alphanumeric character by using position histogram to verify and detect vehicle license plate (VLP) region. This method is very effective in coping with different conditions such as poor illumination, varied distances from the vehicle and 23
varied weather. This is sensitive to the angle of view, physical appearance and environment conditions. This type of problem can be solved by using image processing operation as like as histogram equalization and high dynamic range imaging (HDR). The detection could be improved for progressive performance of vehicle license plate detection. Kim et al. (2002) presented color texture-based object detection as an application to license plate localization. LP localization system is developed and a support vector machine (SVM) is used to analyze the color textural properties of LPs. The plate regions are identified by applying a continuously adaptive mean shift algorithm. The method encountered problems when the image is extremely blurred or quite complex in color. The object detection problem can be addressed with fast and effective ROI selection process and delicate boundary location process. Kim et al. (2002) implemented a robust license-plate extraction method under complex image conditions. The car license plate was extracted from images with complex background and relatively poor quality. The approach focuses on dealing with images taken under weak lighting condition. This method was implemented using two steps: searching of candidate areas from the input image using gradient information and determination of the plate area among the candidates and adjusting the boundary of the area by introducing a plate template. It was found that 90% of the images were correctly segmented. As compared to the conventional edge-based approach, the processing speed is little lower but the approach is more robust as compared to other methods. Krol et al. (2009) achieved localization of vehicle license plates in images using fuzzy logic and morphological operations. The method obtains localization of license plates in a traffic scene image. This can be incorporated in a system for traffic monitoring or parking supervision. The algorithm used enables the localization of license plates in an image. License plates may contain black characters on white background or white characters on black background in old type plates. The license plates may be skewed horizontally. A fuzzy feature extractor neural network and its application in license plate recognition were tested in Rouhani et al. (2006). This paper presents a fuzzy neural network model to extract and classify selected features in sub-regions of a 2D images applied in Iranian automobiles license plate recognition (LPR). A plate locator subsystem locates candidate 24
regions and the candidate regions are presented to FNN (fuzzy neural network) to determine whether it has license plate or not. Input images have approximately 600 × 400 pixels so that license plate candidates are 180 × 60 pixel regions. First layer has 8 × 3 fuzzy neurons, each corresponding to a 100 × 40 region of input layer. The accuracy of proposed FNN license plate type recognizer was 100% for main patterns and over 98% for shifted patterns. The performance of FNN has been evaluated to be excellent in recognition of Iranian license plate types. Lee et al. (2003) proposed a real-time automatic vehicle management system using vehicle tracking and car plate number identification using two cameras: one for tracking vehicles and another for capturing LP (License Plate). The vehicles were tracked by applying the Condensation algorithm over the vehicle’s movement image captured from the first camera. The probabilistic dynamic model such as HMM (Hidden Markov Model) is taken to reflect the temporal change in shape of various vehicles. The segmented characters are recognized using the SVM (Support Vector Machine). General vehicle management system uses the hardware such as sensors for vehicle detection. However, such hardware devices are expensive, and can be activated by all moving objects. Proposed vehicle management system detects only vehicle, can be possible to recognize the LP insensitively to the position of CCD camera. Li et al.(2011) implemented the license plate recognition system based on fuzzy theory and back propagation neural network. In different conditions such as light and complex backgrounds, some car images are captured. This paper presents a method which applies fuzzy theory to enhance several features of for target. This method can improve the accuracy and efficiency of car license recognition, and enhance the system robustness. Using the fuzzy theory and improved BP neural network applications in vehicle license plate recognition system, solves the system's major problems. Fukumi et al. (2005) suggested neural network based threshold determination method for Malaysian license plate character recognition. This method recognizes characters of vehicle license plate in Malaysia by using a neural network based threshold method are presented. Vehicle license plate recognition is one of important techniques that can be used for the identification of vehicles all over the world. There are many applications such as entrance admission, security, parking control, airport or harbor cargo control, 25
road traffic control, speed control, toll gate automation and so on. For separation of characters and background, a threshold of digitalization is important and is determined using a three-layered neural network. In order to extract character portions and recognize the characters in license plates, the technique of determining a threshold value using a neural network was proposed. When character domain could be obtained, we can perform character recognition in high accuracy by a neural network. For better segmentation of license plate characters, we have to evaluate a method by two thresholds (upper and lower bounds) determined by neural networks or genetic algorithms. However, recognition accuracy can even be further improved in license plate character recognition. Mahini et al. (2006) presented an efficient features–based license plate localization method which solves multi-object problem in different image capturing conditions. The proposed algorithm is robust against illumination, shadow, scale, rotation, and weather condition. It extracts license plate candidates using edge statistics and morphological operations and removes the incorrect candidates according to the determined features of license plates. The proposed algorithm successfully detected the accurate location of the license plates in 96.5% cases, which outperforms the other available approaches in the literature. The license plate candidates are used based on vertical edges, morphological operation, and color analysis of the images; and by eliminating the incorrect candidate regions based on images features the correct license plate regions are obtained. Mai et al. (2011) achieved an improved method for Vietnam license plate location, segmentation and recognition. Automatic license plate recognition (ALPR) is very important in the Intelligent Transportation System (ITS). ALPR algorithm for Vietnam license plates (LP) consists of three main modules: license plate location (LPL), character segmentation, character recognition. In the location module, improved algorithm is used based on edge detection, image subtraction, mathematic morphology to locate LP region removing noise. It was implemented over 600 images taken from actual scenes, different background such as light conditions (night and day), angles, illumination, size and type, colors, reflected light, dynamic conditions. The efficiency of the proposed approach is improved and average rate of accuracy of the one-row LP is 96.93%, two-row LP is 95.82%.
26
Mecocci et al. (2006) proposed generative models for license plate recognition by using a limited number of training samples. Increased mobility and internationalization are the challenges to develop effective traffic monitoring and control systems. This is true for automatic license plate recognition architectures that handle plates from different countries with different character sets and syntax. This paper reports a new algorithm for License Plate recognition, developed under a joint research funded by the main Italian highways company. The research aimed at achieving improved recognition rates when dealing with vehicles coming from different European and nearby states. The overall rate of correct classification is 98.1%. When new types of license plates start to flow under the portals, the new recognizer can be almost immediately upgraded, because very few character images need to be collected. Thus the proposed algorithm is well suited to rapidly fine tune License Plate recognition systems in an international context, because it is possible to accommodate new types of license plates as they become available under the monitoring portals. Moghassemi et al. (2011), designed an Iranian License Plate Recognition using connected component and clustering techniques in which a LPR system can be divided into the detection and recognition stages. At the first phase regions of around plate is clip out by help of vertical and horizontal projections. Next accurate location of plate is recognizing by connected component analysis and clustering techniques. Due to the positioning of vehicle towards the camera, the rectangular of license plate can be rotated and skewed in many ways. So skew detection and correction is requiring after plate detection. In this study an efficient method is proposed to skew detection and recognition. Zernike and wavelet moments features with rotation and scale invariant property are used to recognition of license plate characters. The algorithms are robust to the different lighting condition, view angle, the position, size and color of the license plates when running in complicated environment. The overall performance of success for the license plate achieves 93.54% when the system is used to the license plate recognition in various conditions. The complete testing image database consists of 1000 digital license plate images from four sets. The majority of the images represent Iranian license plates from the natural scenes obtained in various illumination conditions. The overall performance of success for the license plate achieves 93.54% when the system is used to the license plate recognition in various conditions. Naito et al.(2000) proposed a 27
robust license-plate recognition method for passing vehicles under outside environment. The developed sensing system can expand the dynamic range of the image by combining a pair of images taken under different exposure conditions. In order to avert blurring of images against fast passing vehicles, a prism beam splitter installed a multilayered filter, and two charge-coupled devices are utilized to capture those images simultaneously. The performance of recognizing registration numbers on license plates has been investigated on real images of about 1000 vehicles captured under various illumination conditions. But it remains to improve the recognition algorithm to make it more robust for dirty and damaged plates. Nguyen et al. (2008) proposed a real-time and robust license plate localization method for traffic control applications. The edge content of gray-scale image is approximated using line segments features by means of a local connective Hough transform. A new, scale and rotation invariant, texture descriptor which describes the regularity, similarity, directionality and alignment is proposed for grouping lines segments into potential license plates. Evaluation is done over mage databases which were taken from real scene under various configurations and variability. The result shows that the method is realtime, robust to illumination condition and viewpoint changes. This approach includes a fast descriptor extraction based on an improvement of connective Hough transform. Future work aims to decrease false alarms and for this the empirical verification may be replaced by a learning verification using geometrical features, statistical features and character-based Haar features. Nomura et al. (2005) implemented an adaptive morphological approach for degraded character image segmentation. An algorithm based on the histogram automatically detects fragments and merges these fragments before segmenting the fragmented characters. A morphological thickening algorithm automatically locates reference lines for separating the overlapped characters. This method can be used to detect fragmented, overlapped, or connected character. Seriously degraded images as license plate images taken from real world are used in the experiments to evaluate the robustness, the flexibility and the effectiveness of our approach. This approach has successfully performed the automatic segmentation and extraction of isolated, fragmented, overlapped or connected characters from the set of 1005 real plate images processed during the 28
experiments. A pattern recognition experiment based on artificial neural networks techniques showed that high recognition rate is reached by using the feature vectors extracted by the proposed approach. Ozbay et al. (2005) achieved automatic vehicle identification (AVI) by plate recognition which has many applications in traffic systems (highway electronic toll collection, red light violation enforcement, border and customs checkpoints, etc.). The algorithm consists of three major parts: extraction of plate region, segmentation of characters and recognition of plate characters. For extracting the plate region, edge detection algorithms and smearing algorithms are used. Smearing algorithms, filtering and some morphological algorithms are used in segmentation stage. Finally statistical based template matching is used for recognition of plate characters. The performance of the algorithm has been tested on real images. This system is designed for the identification Turkish license plates and the system is tested over a large number of images. Finally it is proved to be %97.6 for the extraction of plate region, %96 for the segmentation of the characters and %98.8 for the recognition unit accurate, giving the overall system performance %92.57 recognition rates. This system can be redesigned for multinational car license plates in future studies. Xiang et al. (2004) suggested a hybrid method for robust car plate character recognition image based car plate recognition. A two-stage hybrid method for car plate character recognition is proposed which includes distinguishing similar characters by local structural features and developing a system architecture combining statistical and structural recognition methods. This method was tested with huge number of plate images captured in different environments from real applications, and proven to be successfully in commercial car plate recognition. How to recognize the characters is a challenging research problem. Petrovi´c et al. (2008) analyzed features for rigid structure vehicle type recognition. An investigation is obtained in terms of feature representations for rigid structure recognition framework for recognition of objects with a multitude of classes. The intended application is automatic recognition of vehicle type for secure access and traffic monitoring applications, a problem not considered at such a level of accuracy. The final system is capable of recognition rates of over 93% and verification equal error rates of fewer than 5.6% when tested on over 1000 images containing 77 different classes. The system is shown to be robust for a wide range of weather and lighting conditions. 29
Certain gradient
representations, such as the square mapped gradients, are capable of accurate and reliable recognition of vehicles from frontal views under a variety of conditions. Qin et al. (2010) suggested license plate recognition based on improved Bp neural network to identify the license plate. The approach can identify license plates more effectively and establish a very good technical base for the future license plate recognition. The results proved that the advantages of vehicle license plate positioning method. It can achieve good positioning and identification effects in many 174 conditions. Rahman et al. (2003) suggested a smart and simple algorithm for vehicle’s license plate recognition system. Based on pattern matching, this algorithm can be applied for real time detection of license plates for collecting data for surveying or for some application specific purposes. The prototype system will be integrated to the intersection surveillance video system for traffic surveying or for some application specific purposes. Rajaram et al. (2006) employed a novel learning-based framework for zooming and recognizing images of digits obtained from vehicle registration plates that were blurred using an unknown kernel. The image is modeled as an undirected graphical model over image patches in which the compatibility functions is represented as nonparametric kernel densities. The crucial feature of this work is an iterative loop that alternates between super-resolution and restoration stages. A machine-learning-based framework has been used for restoration which also models spatial zooming. Image segmentation is done by a column-variance estimation-based “dissection” algorithm. The main contribution of the work is the framework where the confidence scores of recognition are fed back to the restoration/super-resolution algorithm. The confidence scores are used to generate samples from the training data set based on which the potentials of the field are learned. Another important feature of this work is the integration of an LPR system, which is fully automated, robust, and self boosting. Another challenging field where this framework can be extended is time super-resolution. Inter-frame relationships can be learnt for applications like frame rate enhancement. This learning-based framework can easily be extended to handle different kinds and levels of blurs. Devi et al.(2011) suggested license plate recognition (LPR) technology that enables computer systems to read automatically the registration number (license number) of vehicles from digital pictures. The LPR system consists of four steps plate localization, preprocessing, 30
segmentation and normalization and optical character recognition (OCR). Morphological operator is applied to the image to identify the plate location. Then the plate region is then preprocessed by applying the histogram equalization technique. The smearing and morphological algorithms are used to segment the characters and the segmented result is normalized and fed to the OCR part. The characters are then recognized using the template matching algorithm. Roy et al.( 2011) presented number plate recognition for using in different countries and improved segmentation which identifies the characters directly from the image of the license plate. The non-adherence of the system to any particular country-specific standard & fonts effectively means that this system can be used in many different countries – a feature which can be especially useful for transborder traffic e.g. use in country borders etc. This can improve the system performance and make the system more efficient by taking relevant samples. The system was tested on 150 different number plates from various countries and an accuracy of 91.59% has been reached. The major sources of error were the skewness of the number plate and extreme variation in illumination conditions, which can be aptly removed by enhancing the approach further. The entire system was designed on MATLAB platform but for real time implementation this needs to be developed in C or any similar IDE (integrated development environment) specific to the hardware used. This could also lower the cost of the system as the C compiler is cheaper than MATLAB. Rastegar et al. (2008) proposed an intelligent control system using an efficient license plate location and recognition approach. After adjusting the image intensity values, an optimal adaptive threshold is found to detect car edges and then the algorithm uses morphological operators to make candidate regions. Features of each region are to be extracted in order to correctly differentiate the license plate regions from other candidates. The algorithm can efficiently determine and adjust the plate rotation in skewed images. An intelligent vehicle-access control system is proposed on an efficient license plate location and recognition approach. The multilayer perceptron neural network (MLPNN) is selected as a powerful tool to perform the recognition process. A study of the different parameters of the training and recognition phases showed that the proposed system reaches promising results in most cases and can achieve high success rates. A great effect of our method in license plate location has been confirmed by the 31
experiments. Shapiro et al. (2004) proposed multinational license plate recognition system which provides an inexpensive automatic solution for remote vehicle identification. Localization stage yields a gray-scale plate clip with printed characters. The method is independent on character size, thickness, illumination and is capable of handling plates from various countries. The method uses extensively the gray-scale information and is robust to breaks in character connectivity. It is tolerant to character deformations, such as shear and skew. It allows a very reliable verification of a plate candidate generated at the phase of localization. Future directions of this work lay in applying approaches known in a context of conventional OCR/ICR systems as “multiexpert” combination, or “voting”. Using RGB cameras with known in advance plate background/foreground colors would allow higher precision in character isolation. Abdullah et al. (2006) implemented license plate recognition using multi-cluster and multilayer neural networks. Due to the different types of license plates being used, the requirement of an automatic license plate recognition system is rather different for each country. An automatic license plate recognition system is proposed for Malaysian vehicles with standard license plates based on image processing, feature extraction and neural networks. Multi- cluster approach is applied to locate the license plate at the right position while Kirsch edge feature extraction technique is used to extract features from the license plates characters which are then used as inputs to the neural network classifier. The classification has significantly raised more problems compared to segmentation. Major adjustment must be made to reduce recognition errors. These errors may origin to insufficient segmentation algorithm or inefficient feature extraction method (Kirsch edge detector). Suresh et al. (2007) presented a novel method to enhance license plate numbers of moving vehicles in real traffic videos is proposed. A high-resolution image of the number plate is obtained by fusing the information derived from multiple, sub-pixel shifted, and noisy low-resolution observations. The image to be super-resolved is modeled as a Markov random field and is estimated from the observations by a graduated nonconvexity optimization procedure. This method is computationally efficient as all operations can be implemented locally in the image domain. A robust super-resolution algorithm is proposed to enhance the license plate text of moving vehicles, which uses 32
the information available from multiple observations of a vehicle to obtain a high-quality license plate image. Tsai et al. (2009) presented a robust system to recognize vehicle license plate by multi-frames learning. The position of a license plate is located adopting a morphology-based method to extract important contrast features as filters to find all possible license plate candidates after calculating motion energy from video frames. The contrast feature is robust to lighting changes and invariant to different transformations like image scaling, translation, and skewing. Due to noise, many impossible license regions may be extracted. Experimental results show that the proposed method is robust for the recognition of license plate. The results show that it is very robust and high accuracy to recognize the characters. However, huge data and poor speed are major challenges. Moreover, the current system can only recognize license plates, by combining the database and hardware it is possible to build a complete toll collection system for example, for real application. Villegas et al. (2009) implemented a LPR using a novel fuzzy multilayer neural network to solve the problem of license plate recognition using a three layer fuzzy neural network. The plate is detected inside the digital image using rectangular perimeter detection and the finding of a pattern by pattern matching, after that, the characters are extracted from the plate by means of horizontal and vertical projections. The characters detected are resized in order to obtain always the same size of a character. The problem of recognition was solved using a fuzzy three layer neural network. A filter would be required to avoid the problem of character similarity. Ho et al. (2010) presented a Macao license plate recognition system based on edge and projection analysis to recognize Macao license plates. Edge analysis and vertical / horizontal projection methods are used in the algorithm. The algorithm is able to resolve general problems such as complex background and rotation, as well as country specific challenges such as multiple types of plates and disunity of character fonts. This work achieves a high accuracy rate of about 92%. Vertical edges are detected and combined into rectangle regions which are further verified and optimized by the feature that Macao license plates contain the character “M” as the first character. Wang et al. (2008) implemented fuzzy-based algorithm for color recognition of license plates. Color recognition of license plates plays an important role in license plate recognition (LPR) system but it can be a challenging task as the appearances of license plates are affected by 33
various factors such as illumination, camera characteristics, etc. The HSV (hue, saturation and value) color space is employed to perform color feature extraction. Three components of the HSV space are firstly mapped to fuzzy sets according to different membership functions. The fuzzy classification function for color recognition is described by the fusion of three weighted membership degrees. A learning algorithm is used to obtain the correlative parameters. Experimental results show that the proposed algorithm achieves higher classification accuracy and better adaptability. The recognition accuracy and execution time can meet the requirements of the practical engineering applications. Future works will focus on improving precision of the localization algorithm and developing a better camera control algorithm. Meanwhile, further optimization of the program to improve its efficiency is also a concern. Wang et al. (2007) suggested a cascade framework for a real-time statistical plate recognition system which is designed to meet the requirements of performance, computational speed, and adaptation for vehicle surveillance applications, such as stolen car detection systems. These requirements are satisfied by adopting a cascade framework, utilizing plate characteristics, and developing fast one-pass algorithms. A peak-valley analysis algorithm is proposed to rapidly detect all promising candidates of character regions. The system can recognize plates over 38 frames per second with a resolution of 640 480 pixels on a 3-GHz Intel Pentium 4 personal computer. The framework is used to develop a plate recognition system that operates only with information presented in a single graylevel image and at a rate of more than 30 f/s. By eliminating the need to compute a multistage image pyramid, the proposed algorithm significantly reduces the initial image processing effort required for plate detection. This paper presents the co-design of plate detection and character segmentation using the same summed area tables to avoid recalculating the procedures. The test dataset includes plates under a very wide range of conditions, such as illumination, scale, location, or camera variation. A robust license plate recognition based on dynamic projection warping was implemented in Wang et al. (2004). This paper proposes a novel license plate recognition system combined with dynamic projection warping technique, which makes the processed image robust to shifting, paring, and scaling problems. The dynamic projection warping method has been succeeded in coping with the problems resulted from shifting, paring, and scaling in 34
character recognition. If the character analysis is further processed, the recognition rate is higher than 98%. Zhuo et al. (2011) implemented a Pulse Coupled Neural Network (PCNN)-based method for vehicle license localization. This algorithm uses morphological operations on a preprocessed, edge image yielded by PCNN. Then connected component analysis follows to get some candidate regions which probably contain license plate. Finally horizontal and vertical projections are used to work on each candidate region for an accurate plate extraction. The algorithm presented in this paper has been tested on a database of RGB natural scene vehicle images consisting of 100 samples in which 10 images with noise are contained. This algorithm is implemented on MATLAB 7.9.0(R2009b). The method based on PCNN and edge statistics has shown to be promising for the task of license plate localization. More work still needs to be done. When the background is complicated, our algorithm suffers from errors that are caused by the disturbing edges in the background. Thus the further research will focus on how to get rid of the unwanted edges in the background. Wijetunge et al. (2011) implemented a real-time recognition of license plates of moving vehicles in Sri Lanka for many applications such as detecting traffic light violations, access controlling, calculating parking fee and so on. However, detection and recognition of license plates can be seen as a complex problem. This paper presents an algorithm which can be used in Sri Lanka, for detecting and recognizing license plates automatically using image processing and neural networks techniques. Hough transformation and the affine transformation techniques are used to handle the skewed license plates. After extracting the license plate characters, a neural network is used to recognize those characters. The proposed system can deal with the images which contain the front view of a vehicle as well as the rear view of a vehicle. The algorithm is capable of achieving faster recognition in complex situations where most of the existing methods find it hard to detect and recognize LPs. In an event of real time implementation, undetected plates should be sent for manual inspection. Wu et al. (2011) proposed license plate recognition system with high accuracy at night. The system, based on regular PC, catches video frames which include a visible car license plate and processed them. Once a license plate is detected, its digits are 35
recognized, and then checked against a database. The template matching method was used and neural network method was also combined. The result showed that the accuracy is higher at night. From the data statistics of the application of this system, the system accuracy is over 97% at daytime and over 96% at night, compare with the accuracy before, over 98% at daytime and over 96% at night. The new system is more practical at night and in the indoor environment, as the low light intensity. The traffic is not heavy at night, but maybe there are some speeding cars. When the light intensity is low, this system works more efficiency and practically. The average recognition time is 0.8s. There are still many points that can be improved such as stability and the recognition of other country’s’ plates. Li et al. (1999) worked a dissertation on Neural Network Study System in Vehicle Identification using BP network to recognize the characters of the vehicle plates. However, BP network has inherent defects and, by improving it with network learning method, this dissertation proved the possibility of an increased network learning efficiency, effectively solving the low-speed and local minimum problems of the neural network constringency. Yan et al. (2001) suggested License Plate Recognition (LPR) system for potential applications to areas such as highway electronic toll collection, red-light violation enforcement, secure-access control at parking lots and so on. The system frame was introduced based on the web technique. Some problems of the technical emphases and realization were addressed and capabilities under practical conditions were found effective. A web-based user interface was provided in LPR system for the customers to get all of the related information or service which adopts the Web Browser/Web Server (B/S) mode. Further, an extensive research is required to make the system faster, more robust and more accurate. Guang et al. (2011) implemented License Plate Character Recognition Based on Wavelet Kernel LS-SVM in which LPR is a foundation component of modern transportation management systems and uses a set of computer image-processing technologies to identify vehicle by its license plate. The main problem is how to recognize every character of the license plate accurately and rapidly in case of noise, variation, blurs and other adverse conditions. LS-SVM is an evolution of classical SVM having higher speed. The method produced recognition rate of 98.3% and testing time of 0.13s. The inherent defects in neural network can also be overcome with higher 36
calculation speed than standard SVM by solving linear equations. This method performs good using multi-classifier model. The future work may address how to find the category rules to implement the optimal multi-classifier, method of experience risk analysis, cluster analysis would be took into account further. Zheng et al. (2005) proposed an efficient method of license plate location in vehicle license plate recognition for automated transport system. The license plate area contains rich edge and texture information which are extracted using image enhancement and Sobel operator and removed most of the background and noise edges by an effective algorithm. The method of license plate location makes use of the rich edge information in the plate area. The local areas in the original car image are enhanced to enhance the gradient image to intensify the texture of the plate region. If the vertical edges, the left diagonal edges and the right diagonal edges are all extracted, a better continuity in edge curves can be attained at the expense of more computation time. Zheng et al. (2010) presented accuracy enhancement for LPR which is useful for real time traffic management and surveillance. LPR contains two steps, namely license plate detection/localization and character recognition. Recognizing characters in a license plate is a very difficult task due to poor illumination conditions and rapid motion of vehicles. The method is based on the license plates detected using an AdaBoost algorithm. Then it follows the steps of character height estimation, character width estimation, segmentation and block identification. All unwanted areas are removed. The techniques used include image binarization, vertical edge detection, horizontal and vertical image projections and blob extraction. High accuracy of non character area removal and significantly higher recognition rate after character is segmented. License plate localization and deflection correction use Sobel horizontal edge detection and horizontal difference. The localization and correction is according to projection characteristic and statistic characteristic of license plate. Characters recognition extract valid character feature first and recognize the license plate characters by using BP neural network later. The method is limited to color characteristic of Chinese license plate. Zunino et al. (2000) presented a vector quantization for LPR and image coding. This paper presents a novel method based on vector quantization (VQ) to process vehicle images. This method makes it possible to perform superior picture compression for 37
archival purposes and to support effective location at the same time. In comparison with classical approaches, VQ encoding can give some hints about the contents of image regions; such additional information can be exploited to boost location performance. The main aim of using VQ for image representation is that a quad tree representation by the specific coding mechanism can give a system some hints about the contents of image regions. The reported results produced are promising, although a better comparison would involve a larger database. Many images must be processed to obtain representative performance parameters that require an industrial application in the field. Adomi et al. (2001) proposed an access control system with neuro-fuzzy supervision for access control to restricted areas. The method was based on vision, neural networks and a neuro-fuzzy system and used to detect the license-plate and to single out the characters it contains, while a neural network-based classifier is used to “read” the plate. Different plate layouts could be considered, since the system modularity makes it possible to introduce new plate-specific modules, without the need to modify any of the existing ones. Becerikli et al. (2007) presented neural network based license plate recognition system to detect license plate zone from image of vehicle which has standard license plate. Histogram equalization is executed on a gray leveled image, the succession at dark view license plates maybe increase. The process time may increase because of process count when we use any alternative approach. Caner et al. (2008) proposed an efficient embedded Neural-Network-based LPR methodology for a field-programmable gate array (FPGA)-based license plate recognition (LPR) system. Gabor filter, threshold, and connected component labeling (CCL) algorithms are used to obtain license plate region. This region is segmented into disjoint characters for the character recognition phase, where the self-organizing map (SOM) neural network is used to identify the characters. The system is portable and relatively faster than computer-based recognition systems. The robustness of the system has been tested with a large database acquired from parking lots and a highway. The memory requirements are uniquely designed to be extremely low, which enables usage of smaller FPGAs. The resulting hardware is suitable for applications where cost, compactness, and efficiency are system design constraints. The system is designed from scratch, including the video input/output units, as well as the FPGA core. Individual blocks in the system are pipelined for faster processing, proving 38
itself to be a fast approach with respect to other published methods. The memory requirements are uniquely designed to be extremely low, which enables usage of smaller, and thus cheaper, FPGAs. This low-cost dedicated hardware device can be used to forward only the extracted text information rather than bulky image data through wireless data links, thus reducing operational costs. Chen et al. (2009) discussed the application of a convolution neural network on face and License Plate Detection including two detectors, one for face and the other for license plates, are proposed, both based on a modified convolution neural network (CNN) verifier. Pyramid-based localization techniques were applied to fuse the candidates and to identify the regions of faces or license plates. Some geometrical filtering rules were applied to locate the license plate regions from images. Feng et al. (2010) suggested License Plate Recognition which is an important part of construction of intelligent transportation system and the license plate character is recognized by building BP artificial neural network. The recognition rate of number and alphabetic is relatively higher than Chinese from the recognition results of network. More characters are extracted for expressing adhesion, fuzzy and broken characters to improve accuracy of character recognition. Gesualdi et al. (2002) proposed character recognition in Car License Plates Based on Principal Components and Neural Processing. The method is evaluated on the extraction of principal components in character images of Brazilians’ license plates. For data compression, principal component methods, envisaging data reconstruction (PCA) or discrimination (PCD). The data compression approach allows reducing the complexity of the neural classifiers and possible online applications for an automatic car plate identification system can also benefit from the consequent speed increase in data processing. Frequency-domain image processing methodology has been successfully derived for an automatic extraction of the plate image from the original acquired photography, which is the first step required for the target automatic recognition system. Ge et al. (2006) implemented image characters location, segmentation and pattern recognition using LS-SVM to adopt gray-grads, shape and posture, vision model and so on. By comparing Least Squares Support Vector Machines (LS-SVM) with BP neural network in vehicle license plates pattern recognition and classification, the results showed that SVM is more convenient, fast and nicely of judgment, and will be the 39
mainstream of the future development in the application of finite sample data pattern recognition. Guo et al. (2008) presented License Plate Recognition System Based on Orthometric Hopfield Network on the binary scale picture platform. The entire process is divided into two parts: preprocessing and recognition. The novel Hopfield network can memorize all the images which need to be memorized. For enlarged pattern, time of recognition will increase. Koval et al. (2003) presented LPR system based on image processing using neural network which can be installed into a tollbooth for automated recognition of vehicle license plate information using a photograph of a vehicle. We can improve quality of the vehicle image using fusion technique, then extract the license plate and isolate characters contained on the plate, and finally identify the characters on the license plate using artificial neural network. The results have shown the ability of neural network to recognize correctly characters on license plate with probability of 95% in presence of noise with 50% density. The proposed method of license plate recognition can be implemented by police to detect speed violators, parking areas, highways, bridges or tunnels. Li et al. (2008) suggested an Algorithm for License Plate Recognition using Radial Basis Function Neural Network which is based on the sharing features of a variety of license plates (LP), the vertical edge was first detected by Sobel edge detector. Characters were segmented by means of prior knowledge and connected components analysis, and character recognition was conducted based on radial basis function (RBF) neural network. Lin et al. (2009) proposed character recognition method of license plate image based on multiple classifiers. For Chinese characters, features are extracted from gray-scale character images by Gabor filters which are specially designed from statistical information. In order to access a high recognition rate, 3 classifiers are used. The recognition rate has effectively improved compared to the method of using single classifier. License Plate Recognition is a real-time system in actual applications. However, our method is slower compared to the method of using single classifier. Future research aims to find a type of feature performs better with stroke distortions and to find an effective algorithm to improve the speed of recognition that could fulfill the needs of LPR system.
40
Mello et al. (2009) implemented a complete system for vehicle License Plate Recognition in which, three phases are involved in the process: image acquisition, plate recognition and post processing. The system first locates the plate in the image, and then it segments the characters of the plate and recognizes finally. To segment the characters, a threshold based algorithm based on fuzzy logic and a region growth algorithm are used. An automatic Brazilian vehicle license plate recognition system is proposed. The method was divided in three steps: plate localization, characters segmentation and characters recognition. Color independent algorithms are proposed here as Brazilian license plates have many possible colors and these colors are similar to vehicles ones. The plate localization and the character segmentation algorithms achieved 88.72% and 94.21% of correctness, respectively. These results show the strength of the new proposed algorithms and make them suitable for real world applications. Shan et al. (2010) implemented License Plate Character segmentation and recognition based on RBF Neural Network used to slit characters and extract the statistical features. Then the RBF neural network is used to recognize characters with the feature vector as input. The results show that this method can recognize characters precisely and improve the ability of license plate character recognition effectively. Based on the study of plate location, the method of vertical projection information with prior knowledge is proposed to segment character and extract the statistic feature, then use the RBF neural network to recognize with feature vectors as input. The recognition ability could be further improved effectively in future work Wang et al. (2009) used radial basis function for License Plate Recognition and a new license plate recognition approach is put forward based on the Radial Basis Function Neural Networks (RBFNN). License plate recognition is a both challenging and import recognition technique. Study the advanced features and optimization of the training algorithm remained as future scope of work. Wu et al. (2007) proposed LPR based on framelet which decomposes the plate characters by framelet and select the transform coefficients using wrapper method as the character features, then send them to BP neural network for recognition. Using the same experiment routine, the feature dimension is selected as 200, and the recognition rate reaches 96.65%. The work is only limited to Chinese characters of LPR. An in-depth research on license plate Chinese character on 41
selection of improved features is required. Xu et al. (2009) discussed an application of improved BP algorithm in vehicle LPR. It was proved that the improved BP was faster than the standard BP. Zhang et al. (2010) suggested a vehicle License Plate Recognition method based on Neural Network. Hopfield NN is a feedback network with association function which can figure out the weight of network according to some rules and update every nerve cell’s state constantly in curse of the network evolvement. It provides a new fast and effective method for the vehicle license identification. This method can effectively identify to be contaminated fonts. Zhang et al. (2008) proposed LPR based on Filled function method. It needs fast speed in finding an optimal solution and good optimization effect. Due to the complexity of Chinese character strokes, the recognition rate of Chinese character network is lower. Some numbers and letters are confusing. Future work includes how to correctly recognize the confusable numbers and letters.
2.2 Summary of Literature Survey and Problem Identification Several research papers were studied and the conclusions are briefly summarized here. The methods of plate extraction used in license plate extraction have following observations and challenges:
Methods of plate extraction of the plate region are mainly based on edge statistics and mathematical morphology;
Edge based methods alone can hardly be applied to complex images because of their being sensitive to unwanted edges exhibiting high edge magnitude or variance;
The brightness change in the license plate region is more remarkable and has difficulty in extracting license plate region when the boundary of the license plate is not clear or distorted;
If in the first scanning process the plate is not correctly extracted then the algorithm is repeated which reduces the threshold for counting edges. This is time consuming process;
The methods are generally computationally expensive and slow for images with large analysis;
42
The system could not detect plates of different size or images acquired from different view/distance without retraining;
The major advantage of wavelet transform is that it can locate multiple plates with different orientations in one image;
The methods are unreliable when the distance between the vehicle and the acquisition camera is either too far or too close; and
The generalized symmetry transform (GST) produces continuous features of symmetry between two points by combining locality constraint and reflection symmetry. This process is also time consuming because the number of possible symmetrical pixels in the image is huge.
The color based methods have following observations and challenges:
Color plays crucial role in color (or gray level)-based method;
High degree of accuracy in a natural scene as color is not stable when the lighting conditions change;
The color based methods fail at detecting various license plates with varying colors;
Color processing shows better performance even though it has difficulties in recognizing a car image having many similar parts of color values to a plate region; and
If the image was extremely blurred or quite complex in color then too much computation time would be required.
Character recognition in license plate recognition is most important part of LPR methods. The literatures on character recognition are summarized here:
A
morphological thickening algorithm is used to locate reference lines for
separating the overlapped characters and the segmentation cost calculation determines the baseline for segmenting the connected characters;
The approach can detect fragmented, overlapping, or connected characters and adaptively apply one of three algorithms without manual fine tuning;
43
Local greedy mutation operator is employed to optimize the objective function but the segmentation results were far from suitable for automatic character recognition;
The brightness distribution of various positions in a license plate image may vary because of the condition of the plate and the effect of lighting environment;
The recognition begins with a preprocessing and a parameterization of the region of interest (ROI) and the recognition result was reported to be 95.7% after a complex procedure of preprocessing and parameterization;
Multilayer perceptron neural networks were used for license plate character identification and the training method for this kind of network is the error back propagation (BP);
Kohonen’s self-organized feature maps (SOFMs) were implemented to tolerate noisy, deformed, broken, or incomplete characters acquired from license plates, which were bent and/or tilted with respect to the camera; and
The useful feature comes with the drawbacks of larger memory requirements and slightly slower execution speed compared to conventional neural networks.
44
Chapter 3 Basics of Neural Network and Fuzzy Logic
Soft computing technique or tools have made the task of automated processing very easy which include artificial intelligence (AI), neural network (NN), fuzzy logic (FL) and genetic algorithm (GA). Each of the tools has its own purpose of using in solving real time problems. For example, neural network is used in classification and feature extraction tasks; fuzzy logic is used in reducing amount of uncertainty; and genetic algorithm plays very important role in optimization related works. Among these soft computing tools, artificial neural network (ANN) and fuzzy logic have been used to great extent in the present work; and hence these two tools are discussed in this chapter. 3.1. Artificial Neural Network (ANN) For more than 50 years, there have been numerous applications of neural network. The neural network is analogous to human brain which consists of thousand of neurons like a human brain that consists of brain cells. If the network which consists of many neurons and structured in layers; is trained using computers and called as artificial neural network (ANN) because the network is not the natural brain but analogy of the same. The development of ANN played very important role in classifications tasks over several years and is basically gross simplifications of real (biological) networks of neurons. The paradigm of neural networks actually began during 1940s that promises to be a very important tool for analyzing various structures as a functional relationship of the human brain. As we know that the human brain is very much complex and understanding of biological neurons is not perfect and therefore several architectures of artificial neural networks have been discussed and proposed by various researchers and scientists. Generally, the ANN structures used for many applications consider the behavior of a single neuron as the basic computing unit for describing neural as information processing operations. Each computing unit is based on the concept of ideal neuron which responds optimally to the applied inputs. Experimental studies on neuro-physiology suggest that the response of a biological neuron is random and it is possible to obtain predictable
45
results after so many averaging or similar manipulations (Hertz et al., 1991; Bishop et al., 1995). ANNs can accommodate many inputs in parallel manner and encode the information in a distributed way. The information which comes through input is stored in a neural network and can be shared among its processing units. The ANNs are trained and corresponding weights of neurons are adjusted so as to get the desired result. Since the neurons are organized in terms of layers which means that there is one input layer; one output layer; and there may one or more intermediate layers also referred as hidden layers. Therefore, the input information is fed to neural network through input layer and the output or final result is obtained from output layer. If the result is not accurate then the weights of neurons of all the hidden layers are changed or modified till the result is desired. The aim of neural networks is to mimic the human ability to adapt to changing circumstances and the current environment. This depends heavily on being able to learn from events happened in the past and to be able to apply this to future situations. Artificial neural networks consist of many nodes as processing units which are analogous to neurons of human brain. Each node has a node function, associated with it which along with a set of local parameters determines the output of the node subjected to an input. Modifying the local parameters may alter the node function and ANN acts as an information-processing system. The signals are transmitted through connection links between nodes and the links possess an associated weight, which is multiplied along with the incoming signal (net input) for any typical network. The output signal is obtained by activating the net input signals. Figure 3.1 depicts a neural network as a single layer network whose simple structure is shown in the figure. In Fig. 3.1, a simple artificial neural network is shown with two input neurons (x1, x2) and one output neuron(y). The interconnected weights are given by w1 and w2. In a single layer network the interconnection is weighted. A typical multi layer ANN which is also sometimes written as MNN (multi layer neural network), comprises an input layer, output layer and hidden layer of neurons. MNNs are also called layered networks which can be used to implement arbitrary complex input/output mappings or decision surfaces separating different patterns. A 46
three-layer MNN is shown in Fig. 3.2 and its simplified block diagram representation in shown in Fig. 3.3. In a MNN, a layer of input units is connected to a layer of hidden units, which is connected to the layer of output units. The activity of neurons in the input layer represents the raw information that is fed into the network. The activity of neurons in the hidden layer is determined by the activities of the input neurons and the connecting weights between the input and hidden units. The behavior of the output units depends on the activity of the neurons in the hidden layer and the connecting weights between the hidden and the output layers (Haykin et al., 1994; Zurada et al., 1994).
i1
w1 Output Node
i2
y = i1w1 + i2 w2
w2
Fig. 3.1: A simple neural network having single node.
Fig. 3.2: Three layer neural network.
47
x
N1
N2
N3
y
Fig. 3.3: Three nodes of neural network. MNNs provide an increase in computational power over a single-layer neural network unless there is a nonlinear activation function between layers. Many capabilities of neural networks, such as nonlinear functional approximation, learning, generalization, etc are performed due to the nonlinear nature of activation function of each neuron. The market is flooded with new, increasingly technical software and hardware products and among the most popular hardware implementations are Hopfield, multilayer perceptron, self organizing feature map, vector quantization, radial basis function, and back propagation networks. ANNs perform the operations similar to that of the human brain and hence it is reasonable to expect a rapid increase in understanding of artificial neural networks leading to improved networks paradigms. 3.2 Use of Neural Networks Neural networks due to their excellent capability to derive meaning from complicated or complex or imprecise data can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A neural network if properly trained can be considered as an "expert" analyzer of information it has been given to, which can used to provide projections given new situations of interest and answer "what if" questions. There are many other advantages why we use neural networks in solving real time problems. Some of them are:
Adaptive learning: Neural networks have a capability of learning adaptively how to do tasks based on the data given for training or initial experience.
Self organization: An ANN can create its own organization or representation of the information it receives during learning time.
Real time operation: ANN computations can be performed in parallel and special hardware devices are being designed and manufactured exploiting this capability.
48
Fault tolerance: Partial destruction of a network leads to the corresponding degradation of performance.
An artificial neuron has two modes of operation; the training mode and the using mode. In the training mode, the neuron can be trained to fire (or not), for particular input patterns. In the using mode, when a trained input pattern is detected, its associated output becomes the current output. If the input pattern does not belong in the trained list of input patterns, the firing rule is used to determine whether to fire or not (Wilamowski et al. 2009).
i1
w1
w2
Neural
i2 Network
Output = f (i1w1 + i2 w2 + i3 w3 + bias)
w3 i3
bias Fig. 3.4: McCulloch and Pitts model (MCP).
Figure 3.4 shows a neural network model known as McCulloch and Pitt’s model, in which the output of a neuron is a function of the weighted sum of the inputs plus a bias. The role of the entire neural network is simply the computation of the outputs of all the neurons. ANNs are nonlinear information processing tools that are built from interconnected elementary processing devices called neurons. An ANN is also seen as information-processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information. ANNs learn like human being by examples and the experience gained thereof. An ANN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Learning in biological systems involves adjustments to the synaptic connections that exist between the neurons. The organizations and the weights of the connections determine the output. 49
A neural network is a massively parallel-distributed processor having a natural propensity for storing experimental knowledge and making it available for use. It resembles the brain in two respects: Knowledge is acquired by the network through a learning process, and Inter-neuron connection strengths known as synaptic weights are used to store the knowledge. Neural networks can be defined as parameterized computational non-linear algorithms for signal processing. These algorithms are either implemented on a generalpurpose computer or are built into a dedicated hardware. The signals are transmitted by means of connection links. The links possess and associate weight, which is multiplied along with the incoming signal (net input) for any typical neural net. The output signal is obtained by applying activations to the net input. An artificial neuron is characterized by: architecture which is connection between neurons; training or learning that helps in determining weights on the connections; and activation function. A simple ANN with two input neurons (x1, x2) and one output neuron (y) is shown in Fig. 3.4. The interconnected weights are given by w1 and w2. An artificial neuron is many input single output signal-processing element, which can be thought of as a simple model of a nonbranching biological neuron. Each of the inputs are multiplied by a connection weights and the weights are represented by w(n). Modern digital computers outperform humans in the domain of numeric computation and related symbol manipulation using powerful soft computing tools such as ANNs. Humans can effortlessly solve complex perceptual problems (like recognizing a man in a crowd from a mere glimpse of his face) at such a high speed and extent as to dwarf the world’s fastest computer. This can be very easily simulated using ANNs. 3.3 ANN Terminologies Some of important terminologies used in connection with artificial neural networks are discussed below: 3.3.1 Weights A neural network consists of a large number of simple processing elements called neurons. These neurons are connected to each other by directed communication links, 50
which are associated with weights. Weight is information used by the neural network to solve a problem. The weights that carry information are denoted by w1 and w2. They may be fixed, or some random values. Weights can be set to zero, or can be calculated by some methods. Initialization of weights is an important criterion in a neural network. The changes in neuron weight indicate the overall performance of the neural network. For example, x1= Activation of neuron 1 (input); x2 = Activation of neuron 2 (input); y = Output neuron; w1 = Weight connecting neuron 1 to output; w2 = Weight connecting neuron 2 to output, then based on all these parameters, the net input ‘Net’ is calculated which is summation of the products of the weights and the input signals, as: Net = x1w1 + x2w2. Generally, it can be written as: Net input = Net = ∑ xiwi. From the calculated net input, applying the activation functions, the output may be calculated. 3.3.2 Activation Functions The activation function is used to calculate the output response of a neuron. The sum of the weighted input signal is applied with a suitable activation to obtain the response. Same activation functions are used for neurons in same layer. There may be linear as well as non-linear activation functions. The non-linear activation functions are used in a multilayered net. A few linear and non-linear activation functions are discussed (Karlik et al., 2010).
Identity Function: This function may be expressed as f(x) = x for all x which can be seen in Fig. 3.5.
f(x)
x Fig. 3.5: Identity Function.
51
Binary Step Function: This function is written as: if f(x) >=Θ.
f(x) = 1;
if f(x) < Θ
= 0;
(3.1)
where ‘Θ’ is threshold value shown in Fig. 6. Mostly single layer networks use single binary step function to calculate the output. The binary step function is also called as threshold function or Heaviside function which is shown in Fig. 3.6.
1 f(x) 0 x
θ
Fig. 3.6: Binary Step Function.
Sigmoidal Function: These functions are usually S-shaped curves and hyperbolic and logistics functions are commonly used as sigmoidal functions. These are commonly used in multilayer networks such as back propagation network, radial basis function network etc. There are two main types of sigmoidal functions: binary and bipolar sigmoidal functions.
Binary Sigmoidal Functions: This function ranges from 0 to 1, expressed as, f(x)=
(
(3.2)
)
where σ is called the steepness parameter.
Bipolar Sigmoidal Function: The function ranges between +1 and -1. This function is related to the hyperbolic tangent function, expressed as: b(x) = 2 f(x) -1 = Figure 3.7 shows this function.
52
(
)
(
)
(3.3)
f(x)
1
X
-1
Fig. 3.7: Bipolar Sigmoidal Function. 3.3.3 Bias and Threshold If the weights are given as, W= (wij) in a matrix form. The net input to output unit yj is given as the dot product of the input vectors x= (x1, … , xi, … ,xn) and wj (jth column of the weight vector matrix), yinj = xiwj and yinj =∑
. A bias acts exactly as a
weight on a connection from a unit whose activation is always 1. The bias improves the performance of the neural network. Generally, bias should also be initialized either to 0, or to any specified value, based on the neural network used. If bias is present, then the net input is calculated as, Net= b + ∑ xi wi
(3.4)
where, Net is net input; b is bias; xi is input from neuron I; wi is weight of the ith neuron. Now, the activation function is obtained as:
53
+1; if net >=0 f(Net) =
-1; if net < 0
(3.5)
Threshold ‘Θ’ is a factor used in calculating the activations of the given network. Based on the value of the threshold the output may be calculated, i.e. the activation function is based on the value of ‘Θ’. For example, the activation functions may be, +1 if net >= Θ -1 if net < Θ
y= f (net) =
(3.6)
3.4 Training of Artificial Neural Network As a child is trained and the human brain learns a lot of things and gains experience and accordingly takes decisions in life based on learning experience. Similarly, neural network needs to be trained and learning therefore is very important process in ANNs. After designing a suitable ANN for a particular application, the network is ready to be trained. Initially, the weights are chosen randomly and then the training, or learning, begins. There are two main approaches of training (Lee et al., 1992) namely supervised and unsupervised learning. The majority of networks utilize supervised training. However, unsupervised learning is also used to perform some initial characterization on inputs (Amir et al., 1991; Karlik et al., 2010). 3.4.1 Supervised Learning In supervised learning or training of neural network, both the inputs and the outputs are provided; and the network processes the inputs and compares corresponding result against the desired output. Errors if any are then propagated back through the hidden layers of network and the weights are adjusted that controls network performance. This process repeats over and over as the weights are continually updated. The set of data 54
which enables the training is called the "training set." During the training of a network the same set of data is processed many times as the connection weights are ever modified or refined. The current commercial network development packages provide tools to monitor how well an artificial neural network is converging on the ability to predict the right answer. These tools allow the training process to go on for days, stopping only when the system reaches some statistically desired point, or accuracy. Sometimes networks do not adopt training because the input does not contain the specific information from which the desired output is derived. Networks also don't converge if there is not enough data to enable complete learning. Many layered networks with multiple nodes can memorize data and to monitor the network to determine if the system is simply memorizing its data in some non-significant way, supervised training needs to hold back a set of data to be used to test the system after it has undergone its training (Zhao et al., 2007). There are many algorithms used to implement the adaptive feedback required to adjust the weights during training process of network. The most commonly used technique is backward-error propagation that is back-propagation method. Supervised training provides a series of sample inputs and compares the output with the expected responses. The training continues until the network is able to provide the expected response. In a neural net, for a sequence of training input vectors there may exist target output vectors. The weights may then be adjusted according to a suitable learning algorithm. Supervised training is applied in pattern association tasks where a neural net is trained to associate a set of input vectors with a corresponding set of output vectors. Some of the supervised learning algorithms include Hebb learning algorithm, back propagation network, pattern association memory network, etc. 3.4.2 Unsupervised Learning Another learning method is unsupervised training which is also called as adaptive training. In unsupervised training, the network is provided with inputs but not with desired outputs. The system itself must then decide the features to be used to group the input data. Therefore, this is often referred to as self organization or adaptive method of learning. The adaption to the environment is the promise which would enable science 55
fiction types of robots to continually learn on their own as they come across with new situations and new environments. Life is filled with situations where exact training sets do not exist. Some of these situations involve military action where new combat techniques and new weapons might be encountered (Jennifer et al., 2004). One of the leading researchers worked on unsupervised learning is Tuevo Kohonen, an electrical engineer at the Helsinki University of Technology. A self-organizing network was developed which could learn without the benefit of knowing the right answer. It is an unusual looking network in that it contains one single layer with many connections. The weights for those connections have to be initialized and the inputs have to be normalized. The neurons are set up to compete in a winner-take-all fashion. The research continues that are structured differently than standard upon feed forward, back-propagation approaches. Kohonen's work deals with the grouping of neurons into fields. Neurons within a field are "topologically ordered." Topology is a branch of mathematics that studies how to map from one space to another without changing the geometric configuration. The three-dimensional groupings often found in mammalian brains are an example of topological ordering. In a neural net, if for the training input vectors, the target output is not known, the training method adopted is called as unsupervised training. The network modifies the weight so that the most similar input vector is assigned to the same output unit. The net is found to form a code book vector for each cluster formed. Unsupervised networks are far more complex and difficult to implement. It involves looping connections back into feedback layers and iterating through the process until some sort of stable recall can be achieved. The training process extracts the statistical properties of the training set and group similar vectors into classes. Some of the neural network training methods (supervised learning) are given below:
Feedback Networks: Boltzamann Machine (BM), Mean Field Annealing (MFT), Recurrent Cascade Correlation (RCC), Learning Vector Quantization (LVQ), Back propagation through time (BPTT) and Real-time recurrent learning (RTRL).
Feed forward Networks: Perceptron, Adaline, Madaline, Back propagation (BP), Cauchy Machine (CM), Artmap and Cascade Correlation (CasCor). 56
There are some methods under unsupervised learning:
Feedback Networks: Binary Adaptive Resonance Theory (ART1), Analog Adaptive Resonance Theory (ART2, ART2a), Discrete Hopfield (DH), Continuous Hopfield (CH), Discrete Bi-Directional Associative Memory (BAM), Temporal Associative Memory (TAM), Adaptive Bi-directional Associative Memory (ABAM), Kohonen Self-Organizing Map/Topology-Preserving map (SOM/TPM) and Competitive Learning.
Feed forward Networks: Learning Matrix (LM), Driver-Reinforcement Learning (DR), and Counter Propagation (CPN).
3.5 Basics of Fuzzy Logic Zadeh introduced the concept of fuzzy logic in “mathematics of fuzzy set theory” (1973). Fuzzy logic is a complex mathematical method that allows solving difficult simulated problems with many inputs and output variables. Fuzzy logic can reduce the amount of uncertainty present in real time situations and problems. This is unlike binary logic which has only two stable states. The fuzzy logic (FL) is set of multiple intermediate levels having their own probabilities are membership degrees. For example, a statement is given as this person is young or this person is old; similarly this object is hot or cold. In these statements, it is not clear that how old or young the person is; or how much hot or cold the object is. The degree of being hot or cold; or the age is not clear in definition using binary logic since it deals with only two logic levels either hot or cold. We can simplify this problem by defining a range of temperature for hot and cold. For example the temperature range from 00 to 200 is cold and the temperatures from 210 and higher is hot; and suppose an object whose temperature is 190, its degree can be clearly described that the degree of its being cold is 99% or so. Therefore, a membership function is introduced which can describe the degree. There may be numerous methods of defining membership function for decision-making processes. The membership functions are defined using certain rules for the fuzzy sets used to be tuned correctly. Fuzzy logic helps in decision making with estimated values under incomplete or uncertain information. 57
According to Zadeh (1973) and Klir (1995), most of the control systems are characterized by mathematical models that follow the laws of physics, stochastic models or models which have emerged from mathematical logic. A common difficulty of such constructed model is how to move from a given problem to a proper mathematical model. This complexity can be simplified by employing a tolerance margin for a reasonable amount of imprecision, vagueness and uncertainty during the modeling phase. Fuzzy logic allows lowering complexity by allowing the use of imperfect information in sensible manner and it can be implemented in hardware, software, or a combination of both. Generally, fuzzy logic control system is created using four major components as shown in Figure 3.8. The components are fuzzification interface, fuzzy inference engine, fuzzy rule matrix and de-fuzzification interface.
Rule matrix Input
Output Defuzzification
Fuzzification
Fuzzy inference system
Fig. 3.8: Fuzzy logic system.
According to classical set theory, a subset A of a set X can be defined by its characteristic function
as a mapping from the elements of X to the elements of the set
{0, 1} as: :
→ {0,1}
(3.7)
where the value zero represents non-membership and one represents membership. The statement is true if
( ) = 1, and false if it is 0. True or false can also be
defined in certain range of values as shown in Fig. 3.9. The characteristic function for this case may be expressed as: 58
( )= 1 0
≤ ≤ ℎ
(3.8)
Fig. 3.9: Values between a and b. 3.5.1 Membership Functions The membership function of a fuzzy set is a generalization of the indicator function in classical sets. It is a graphical representation of the magnitude of participation of each input. It associates a weighting with each of the inputs that are processed, define functional overlap between inputs, and ultimately determines an output response. The rules use the input membership values as weighting factors to determine their influence on the fuzzy output sets of the final output conclusion. A membership Function (MF) is a curve that defines how each input in the input space is mapped to a membership value between 0 and 1. It is defined as a membership degree function:→ [0, 1], and accordingly fuzzy set A is defined as: A = { (x, The membership function
A(x)
A(x))
| x X }.
quantifies the grade of membership of the elements x
to the fundamental set X. The value 0 means that the member is not included in the given set 1, describes a fully included member. The values between 0 and 1 characterize fuzzy members. Fuzzy membership function is used to define fuzzy constraint to single parameter which gets the values between 0 and 1. There are several types of membership functions used and some of them are discussed briefly.
59
Triangular Membership Function: This is a special case of linear function and can be represented by a triplet notation x
{a,b,c }. The membership function is
defined as:
0, for x < a
for a ≤ x ≤ b A(x)
=
(3.9) for b ≤ x ≤ c
0, for x > c
Linear relationship can be obtained between ‘a’ and ‘b’; and similarly between ‘b’ and ‘c’ and therefore the function is piecewise linear. This is very important when the function required is different at different intervals.
Trapezoidal Function: This is defined by using four parameters, x: { a,b,c,d }.
Normalized Gaussian Function:
This is normally used to represent vague,
linguistic terms, defined by using two parameters x : { , }: (
)
A(x) =
(3.10)
Sigmoidal Function: This is defined by two parameters x: { , }, as: A(x)
= [1 +
) -1
(
]
(3.11)
3.5.2 Fuzzification and Rule Matrix Fuzzification is the first phase of fuzzy logic processing that delivers input parameters for given fuzzy system based on which the output result will be calculated. The parameters are fuzzified using pre-defined input membership functions having different shapes. The most common used functions are:
60
Triangular shape.
Trapezoidal, sinusoidal and exponential can be also used.
Simpler functions do not make the implementation complex and do not overload the system. The degree of membership function is determined by placing a chosen input variable on the horizontal axis, while vertical axis shows quantification of grade of membership of the input variable. The membership functions associate a weighting factor with values of each input and the effective rules. The weighting factors determine the degree of influence or degree of membership (DOM) for each active rule. Rule matrix is used to characterize or specify fuzzy sets and fuzzy operators in form of conditional statements. A single fuzzy if-then rule can be written as: If x is A then y is Z, where A is a set of conditions that have to be satisfied and Z is a set of consequences that can be inferred. Fuzzy operators are used to combine more than one input using AND = min, OR = max and NOT = additive complement operators. The degree of membership for rule matrix output can take value of maximum, minimum of the degree of previous of the rule. Inference mechanism allows mapping given input to an output using fuzzy logic. It uses all pieces described in previous sections: membership functions, logical operations and if-then rules. The most common types of inference systems are Mamdani and Sugeno. 3.5.3 Defuzzification Mamdani (1977) also discussed about defuzzification that is normalization after the purpose of fuzzification is served. There will be need to “defuzzify” the fuzzy results generated through a fuzzy set analysis. As a result of applying the previous processes, one obtains a fuzzy set µA(x) from the reasoning process that describes, for each possible value x, how reasonable it is to use this particular value. In other words, for every possible value x, one gets a grade of membership that describes to what extent this value x is reasonable to use. The transformation from a fuzzy set to a crisp number is called defuzzification which is the process of producing a quantifiable result in fuzzy logic. Defuzzification is a reverse process of fuzzification and considered as a mapping from a 61
space of fuzzy set (logic) defined over an output universe of discourse into a space of non fuzzy (crisp) set. There are several methods for defuzzification. Some of them are:
Maximum Defuzzification Technique
Centroid Defuzzification Technique
Weighted Average Defuzzification Technique
Centre of Singleton Method
Mean-Max Membership
3.5.4 Feature Extraction Learner et al. (1996) suggested feature extraction using fuzzy logic and neural network approach that is the process of mapping the original features (measurements) into fewer features which include the main information of the input. A large number of feature extraction methods based on statistical pattern recognition or on artificial neural networks appears in the literature (Bourland et al., 1988; Chien et al., 1978; Fukunaga et al., 1990; Sanger et al., 1989; Kohonen et al., 1990; Devijver et al., 1982). In all of these methods, a mapping function f transforms a pattern ‘y’ of a d-dimensional feature space to a pattern ‘x, of an m-dimensional projected space, m
