Automatic Arabic Number Plate Recognition

Automatic Arabic Number Plate Recognition A. Abd El Rahman1, A. Hamdy2 and F.Zaki3 1

Sakhr Software, Egypt. Communication and Electronics Department, Faculty of Engineering, Helwan University. 3 Communication and Electronics Department, Faculty of Engineering, Helwan University 2

operations and strongly avoids float point calculations which adds speed enhancement.

Abstract---This paper presents a license plate recognition system for the Egyptian plates introduced in 2008.The proposed system is composed of three main stages; localization & skew correction stage, segmentation stage, and recognition stage. The localization stage uses the main feature of the plate where high contrast text-background is tagged with colored or gray area, to find the plate candidates in the image and to measure the skew angle. In segmentation stage, connected component analysis is applied to find objects belong to license number. The objects will be analyzed to attach diacritic and over segmented objects to each other to form a group of recognizable objects. The final objects will be split to digits and letter groups. In recognition stage, an adapted template match technique is introduced to recognize the digits and letter groups separately after normalizing them. The system is tested against a real video of two hours and the accuracy was 81% and average time per frame was 24 msec/frame.

In the segmentation stage, connected component analysis is applied to find objects in the localized plate. Objects widths summing is used instead of horizontal binary projection of the binarised plate to find the main line crossing all the digits and letters and to establish the upper and lower boundaries of the plate text. This technique yields only one strong maximum value, which in turn results in speed enhancement. The plate text boundaries are used to form recognizable objects that are composed of digits, letters, and their diacritic, or parts of over segmented digits and letters. The final recognizable object is split to digits and letter groups. In the recognition stage, template matching technique has been adapted to find the best match to the recognizable objects. A separate image for each digit and letter is extracted from the real scene images. Such image is normalized either to 30x60 (TA) or 40x50 (TB). Digits and letter objects are normalized to TA or TB based on their geometries. Each object is compared against its corresponding templates and the best match is chosen. The overall performance is measured using real outdoor video (two hours video containing about 203000 frames), from four different locations. To measure the algorithm accuracy, the plates noticed in the video are manually recorded. The algorithm applied to each frame of the video and each result is compared to the manually recorded plates to compute the error using nearest hamming distance. The system accuracy was 81% accuracy. The algorithm average time per frame was 24 msec/frame.

Keywords: Egyptian license plate recognition; connected component; template matching.

I. INTRODUCTION License Plate Recognition (LPR) is an image processing technology used to identify vehicles by their license plates [1]. LPR plays an important role in numerous applications such as unattended parking lots, security control of restricted areas, traffic law enforcement, congestion pricing, and automatic toll collection [2]. LPR is a very interesting but difficult one. It requires some complex tasks, detection, segmentation and recognition. These tasks become more sophisticated when dealing with plate images taken in various inclined angles or plate images with noise. Because this problem is usually used in real-time systems, it requires not only accuracy but also fast computation [3].

Fig. 1 shows typical scene images of the target plates. Fig. 2 shows typical plate dimensions.

Most previous works have in some way restricted their working conditions, such as limiting them to indoor scenes, stationary backgrounds, and fixed illumination [2]. Due to the different types of license plates being used, the requirement of an automatic license plate recognition system is rather different for each country [4]. This paper presents a license plate recognition system for the Egyptian plates introduced in 2008.The proposed system is composed of three main stages; localization& skew correction, segmentation, and recognition. In the localization stage, the main feature of plate of high contrast text-background, tagged with colored or gray area, was used to find the plate candidates in the image and to measure the skew angle. Such technique depends mainly on integer

c 978-1-4799-3160-6/13/$31.00 2013 IEEE

Figure 1. Sample of the Egyptian new plates

23

Figure 2. Typical plate dimension

II. THE PROPOSED ALGORITHM The proposed algorithm is composed of three stages; localization, segmentation, and recognition. A. Plate Localization The new Egyptian plate design is featuring black digits over white background area directly under colored or gray tag. This feature is used to find plate candidates in the input image. Before presenting the details of the proposed algorithm, three main concepts will be introduced, the local intensity change feature, the color feature and tag feature. B. Local intensity change (LIF) Consider the given image shown in Fig. 3 (a) , with dimensions W*H. The image divided, virtually by the red lines, to M by N strips. Also, consider the strip bounded by the green rectangle which coincides within the plate rectangle. The strip will have w*h, where w = W/M and h = H/N. If each vertical scan line pixels' intensity of such strip is added together, the intensity change across this strip will be as illustrated in Fig. 3 (b). Strip that has a constant color is illustrated in the graph in Fig. 3 (c). What differentiates between the two graphs is the high number of changes in the intensity due to the variation from text to background. Such feature will be referred to by LIF. If by design minimum target plate width is selected, and w is selected accordingly to be half of the plate width, one could expect strips located within plate rectangle to have at most six changes corresponding to three digits across one strip. Similarly, if by design maximum plate width is set to be equal W, minimum LIF would be 12/M. Such minimum level will be referred to by nCh. It should be noted that nCh will be an integer value.

(a)

(b)

(c)

Figure 3. (a) Typical image divided to 5 by 30 or M by N strips. (b) Histogram shows the intensity variations over the strip width for the strip framed by the green box. (c) Histogram shows the intensity variations over the strip width for strips located in the constant luminance areas.

24

C. Color feature (CLF) Strips located totally in colored area will have all pixels with high chroma values, whereas strips located in white or black, or generally in gray areas, will have all pixels with low chroma values. Some strips will have colored areas and others will have gray-level areas. It is important feature that strips within plate rectangle will have all pixels with low chroma value. Low or high chroma values depend on the exact colors used in the plate design and in the capture device used. So, if a suitable chroma threshold is selected (cThr), one could expect strips within the plate to have no colored pixels, or to be below specific value (nCo). Such feature will be referred to by CLF. For the purpose of this study, nCo will be 0.25 of the total pixel contained in one strip. D. Plate Tag feature (PTF) The plate will have upper colored/Gray tag on top of the number area. Such tag will form a feature that distinguishes plate candidates from other objects in the input image. E. Plate localization technique using LIF, CLF and PTF After introducing LIF, CLF, PTF, nCh, and cThr, the localization algorithm could be as follows: 1)

Given input image with W*H, it will be divided to M by N strips, each with width w and height h.

2)

nCh will be selected based on the maximum target plate width and cThr will be selected based on the capturing device. For the purpose of the study nCh was 4 and cThr 39.

3)

For each strip, calculate LIF and CLF.

4)

Ignore strips with LIFnCo.

5)

For each other strips, search vertically up to find the first colored strip, i.e CLF>nCo.

6)

If colored strip is not found, search up to find gray strip with intensity less than the current strip with specific value. Such value depends on gray level used in the plate design and on the capturing device. For the purpose of the study, this value is selected to be 30.

7)

If strips in steps 5 and 6 are not found the strip will be ignored.

8)

If colored strip is found, one will creep left and right as long as the pixel is colored (>cThr) to find the upper left and right corner of the plate rectangle. This step will end by reaching the plate edge of black pixel at left or right.

9)

If gray strip is found, one will creep left and right as long as the difference between gray level of pixels and the average strip gray level is 30, as explained in step 6, to find the upper left and right corner of the plate rectangle. This step will end by reaching the plate edge of black pixel at left or right.

2013 Second International Japan-Egypt Conference on Electronics, Communications and Computers (JEC-ECC)

10) Calculate the plate inclination angle using the corners points in steps 8 or 9.

The image in such rectangles will be normalized before passing it to the recognition stage.

11) Calculate the plate height based on the ratio of plate width to height which is 3.2.

A. Connected component analysis Connected component analysis (CCA) will be applied on the candidate plate which is prepared in the localization stage. The target of this analysis is to spot objects in the CP. In Fig. 6, typical CP is analyzed by CCA. Objects discovered are framed with pink (RO). Typical objects are digits, letters, top edge, right edge, middle edge, and bottom edge. Objects belonging to digits and letters (DLO) should be identified and others (NO) should be ignored. Since the exact design specification plate is not available, analysis is done to estimate the digits and letters dimensions. Around 30000 samples of the extracted digits and letters are analyzed. Width, height, and the relative ratio of height to DLO height (RH) are calculated. RH will be explained in section III.B. It was found that:

12) Verify that the current strip is located within the final plate rectangle boundary, otherwise ignore this strip. For images with 320x240, typical values for w, h, nCh are 32, 4,4, respectively, and nCo is 0.25*32*4=32. Fig.4 shows one image processed twice with different parameters. The white strips satisfy the initial criteria of nCh and nCo. For both of the two images in Fig. 4, the strip bounded with green box is the strip which is used to localize plate. The localized plate is bounded with violet box. Once the plate candidate found, all the strips within such candidate will be skipped in order to reduce the computations. This will enhance the overall performance.

1)

Digit (except ‘˾’ and ‘˺’ ) has average W/H of 0.45 and H/RH is approximately 1

2)

Letters could be divided into two groups relative to RH a)

For ‘΃’, ‘Ν’,‘ρ’,‘ω’,‘ϕ’,‘ϝ’, and ‘ϡ’, H/RH >0.82

b) For ‘Ώ’,‘Ω’,‘έ’,‘α’,‘ι’,‘ϑ’,‘ϥ’,‘ϫ’,‘ϭ’,and ‘ϱ’, 0.6 < H/RH < 0.75

Figure 4. A 320x240 image processed twice with different parameters (w, h, nCh, nCo, cThr). The left image is processed with (32, 4, 4,32,39) and the right image is processed with (64, 8, 8,128, 39). Shaded white area is the strips that meet criteria. Strip bounded with green box is the seed to localize the plate bounded with the violet box.

This information will be used in the recognition process. Objects that will not fall in these boundaries will be ignored. It should be noted that objects in the RO is hierarchically arranged from left to right and from top to bottom.

F. Plate skew correction The plate candidate skew angle is measured using the upper left and right corners points in the previous step. Before advancing to the next step, the localized plate will be deskewed and adaptive threshold will be applied. The final candidate will be passed to the segmentation stage. The final plate candidate will be referred to by CP. Figure 6. Typical localized plate after applying CCA. The object found is bounded with the pink.

B. Filtering RO The RO will be filtered based on three roles MOW, MOH, and MOS. Figure 5. The left image shows localized plate, bounded by yellow box. The inclination angle is 2.3 degrees and the final plate candidate after skew correction and thresholding is shown in the right.

III. PLATE SEGMENTATION In the segmentation stage, rectangles that confine the digits and the letters and its diacritics dot or dots will be constructed.

1) Maximum Object Width (MOW) Given an input image with width W, then the maximum object width that could fit in plate number is MOW = W/6. This is because in such image the maximum plate width would be W and it can contain maximum three digits and three letters only. It should be noted that plates with two letters are only distributed over 2/3 of half the plate. For W=320, MOW= (320/2)/3 = 53.


25

Appling MOW will remove the background objects and most of the large objects belong to edges. 2) Maximum Object Height (MOH) Given an input image with width W, then the maximum object height is MOH = 0.6 * (W/3.2). This because the maximum plate height would be (W/3.2) and digits and letters only consume 0.6 of such height. For W=320, MOH = 0.6 * (320/3.2) = 60. Appling MOH will remove some of the vertical edges. In some cases, localization algorithm will produce false large candidate plates. Appling MOH will also reject such candidates. 3) Maximum Objects widths Sum(MOS) Consider the top image in Fig. 7 with (x,y) coordinates centered on the top left corner. Starting from the top left corner, a horizontal line is drawn. The widths of all the objects intersecting with such line are added together. The sum of all widths is designated by OS(y). Plotting OS(y) against y will produce the segmented red curve. The maximum point of such curve is pointed to by MOS. The bottom image shows horizontal sum of black pixels, plotted in pink. By comparing the two plots in Fig. 7, the MOS will give strong and unique maximum value against multiple maximum values in the second method. Using MOS to search up and down for two points where OS decreases by 0.8 of MOS will give upper bound (UB) and the lower bound (LB) of the DLO. The distance between UB and LB will typically be used as RH. All the objects totally confined between UB&LP or partially confined with 50% inside will be included. Also, long objects that span UB and LB but less than 1.2RH, will be included. This filter will remove objects belonging to the upper and lower edges and Latin characters that may appear on some plate versions. Also, this role will remove vertical edges if they are longer than 1.2RH. The final object group after applying MOW, MOH, and MOS rules will be referred to by DLO.

scratches in the plate. So, the DLO contains other objects due to OSO. OSO should be attached to each other before passing it to the recognition stage. In most of the cases, both OSO and diacritic overlap horizontally. This feature is used to attach overlapped objects together. Fig. 8 top shows samples of the over segmented digits and letters. Also, Fig. 8 bottom-left shows typical over segmented letters and how they have been recovered as illustrated in Fig.8 bottom-right. The final rectangles of the DLO are referred to by RR .

Figure 8. The upper image shows the over segmented digits and letters. The lower left image shows typical plate candidate after filtering by applying MOW, MOH, and MOS. The remaining objects result in two objects for 'ϭ' and three objects for 'ϕ'. Attaching the related objects is shown in lower right image.

D. Vertical Edges removal Vertical edges object will still interface with DLO. The last role applied depends on the distribution of digits and letters through the plate. In the plates with four digits (4N_?L) plate configuration, left and middles edges are very close to the digits objects. Also, in the plates with three letters (?N_3L), middle and right edges are close to letters objects. This arrangement is used to remove vertical edges in such configurations.

Figure 9. The first image shows the left edge candidate that is removed by comparing it against RH. The right image shows middle edge that is removed by comparing its arrangement relative to other objects in the plate.

Figure 7. The first typical plate candidate with OS curve plotted. At the bottom the same image with the horizontal sum of black pixel is shown. The MOS will give strong unique maximum value against other method.

C. Over segmented Objects (OSO) It was expected that DLO will contain objects equal the number of digits and letters beside the diacritics. However, this is not correct. In many cases, digits and letters objects are found to be over segmented. That is due to the font used or

26

E. Splitting the digits and letters objects The design of the plate use nine digits and seventeen letters, i.e., the complete set will be 26. Without any assumption about the kind of object, any object in RR need to be compared against the entire set of 26. For 4N_3L, the total comparison is 7 * 26 = 182. If one could compare only to digits or letters, the total comparison would be 4*9+3*17=87. The gain in performance would be (182-87)/182 *100 = 53 %. However plates with mixed digits and letters in the right part of the plate, e.g., plates used with police vehicles, will be recognized as letters only. Such mixed plates are very rare; only one car is noticed during two hours video. By splitting the DLO to digits and letters groups, the objects will be ready to pass to the recognition process.


IV. PLATE RECOGNITION In the plate recognition stage a modified template match algorithm was used. The algorithm is used to find the best match of each digit in RR to one of nine digits image set or to find the best match of one letters in RR to one of seventeen letters image set. A. Template matching Template match is a mathematical technique to find the most similar part in large image (I) to a given smaller image (T). This is done by sliding T over each pixel in I and compare each pixel in T with pixel blow it in I. The result R of such comparison is a set of values each of corresponding similarity measure at specific coordinate in I. If dimensions of I is W*H and of T is w*h, R will contains (W-w+1)*(H-h+1) values. One could find the best match by searching for the maximum value in this set.

I

T

Figure 11. Images used in the recognition process. Upper images represent digits and letters which have W/H