this stage, the program recognizes the car's owner, generates a ticket form, and sends it ... manual procedures and all problems associated with them. Keywords ...
Lebanese License Plate Recognition for Driving Tickets Automation System Michel Owayjan, Jimmy Abboud, Boutros Costantine, Jean Saliba, Elie Abdo American University of Science and Technology (AUST) Departments of Computer and Communications Engineering and Computer Science {mowayjan, eabdo}@aust.edu.lb Abstract – Traffic laws violation is a major cause of deaths and injuries in Lebanon. Traffic law enforcement relies on issuing different types of driving tickets in order to deter violators. However, bribes, bureaucracy, and human errors are hindering the job of law enforcement officials. In this paper, a new fully automated driving tickets system is presented. After a violation occurs, a picture of the vehicle is taken, which is then analyzed to extract its license plate number using a novel image processing algorithm that recognizes both Arabic and Hindu numerals. At this stage, the program recognizes the car’s owner, generates a ticket form, and sends it to the post office to be mailed to the violator. This software automates all the work eliminating manual procedures and all problems associated with them. Keywords – License Plate Recognition, Character Segmentation, Arabic and Hindu Numerals, Driving Ticket Automation
I. INTRODUCTION Driving violations is one of the main concerns of the government, but as much as it tries to enforce the law, many problems stand in the way. These problems are more or less the direct result of unorganized methods in resolving this issue. Two means are being implemented in an effort to tackle the existing driving violations. The first is the manual ticketing procedure where the policemen’s presence is a necessity and the fines are prepared and given manually. The second is the semi-automated procedures where radars and cameras are used. However, since both procedures are not fully automated, they both have drawbacks and lack in efficiency. Moreover, they both lack coverage and are subject to bribe and mediation. Another obvious problem is the weakness in management control such as having several types of ticket forms, lack in regional coverage and difficulty in retrieving reports. II. EXISTING SOLUTIONS Several solutions and approaches exist to the previously mentioned problems. One of the approaches is related to an access control system, which allows a vehicle to pass by the checking test, whether it is fit or not. Other approach consists of saving all plate numbers in a certain database for security measures. However the most effective and important existing solution is the one that detects the driving violations that occur. A. Access Control The access control system is an approach that consists of checking the license plate of a certain vehicle, if the plate is
identified by the database, the car is authorized to pass through the gate, and otherwise the gate will not open. In the SeeLane system [1] the access control system is used to validate the plates and car’s number. When the vehicle approaches the secured area and starts the cycle by stepping over a magnetic loop detector (which is the most popular vehicle sensor), the loop detector senses the car and its presence is signaled to the LPR (License Plate Recognition) unit. The LPR unit then activates the illumination (invisible infra-red in most cases) and takes pictures of the front or rear plates from the LPR camera. The images of the vehicle include the plate and the pixel information is read by the LPR unit’s image processing hardware (the frame grabber). The LPR unit analyzes the image with different image processing software algorithms, enhances the image, detects the plate position, extracts the plate string, and identifies the fonts using special artificial intelligence methods (such as Neural Networks). Most LPR units are based on an application running on a PC operating under Windows. The LPR unit checks if the vehicle appears on a predefined list of authorized cars, if found, it signals to open the gate by activating its relay. The unit can also switch on a green “goahead” light or a red “stop” light and a Welcome Message with personalized data. The authorized vehicle enters into the secured area and the door is closed [1]. B. Parking Monitoring This system checks the license plate of all vehicles that pass through its loop detector. It is mainly concerned with saving all the license plates in a certain database which will be accessed to retrieve information (if needed) about vehicles that were detected in both private and public places. An example of such system is the InSignia monitoring system [2]. C. Driving Violations A violation is detected when the camera is triggered. A picture of the vehicle is taken and is associated with an infrared illumination. The picture is then analyzed and the plate number is retrieved. The system then checks in the database, searches for the retrieved number with its correspondent information, charges the specified fee amount, and issues the ticket automatically. The automated ticket can be delivered to the violator by mail. Those existing solutions and approaches provide certain considerable results, especially the License Plate Reader System (LPRS) [3].
III.
PROPOSED SOLUTION
The proposed solution, works just like the License Plate Reader System (LPRS) and includes the same features. In addition to the LPRS, the system allows Hindu numerals and Arabic characters recognition. Furthermore, it is applicable to Lebanese license plates. The system can also be connected to the main post office branch so the ticket can be delivered to the violator by mail. A. Hindu Numerals and Arabic Characters Recognition In addition to Latin characters and numbers recognition, the proposed system recognizes Hindu numerals and Arabic characters. The purpose of this feature is to provide the possibility for the Arab countries to implement such a system without changing the language on their plate numbers. A bi-language system also provides more precision in identifying plate numbers in case a number, in a certain language, was not clearly identified. B. Solution Applicable On Lebanese Plates The system is mainly used to identify Lebanese license plate numbers, by analyzing the plate according to the standard Lebanese plate shape, taking into consideration that the Lebanese plates have a unique mark, which is the horizontal bar on the left side of the plate with a white cedar. C. Hardware Design The hardware design of the system, shown in Figure 1, consists of several units. One or more cameras that take the images of the car (front or rear side) using Infra-Red (IR) illumination, which is a controlled light that bright up the plate, and allows day and night operation. A frame grabber which is an interface board between the cameras and the server allows the software to read the image information. A server, normally running Windows or Linux operating systems, runs the proposed application. The latter controls the system, reads the images, analyzes and identifies the plate, and interfaces with other applications and systems, charges the corresponding amount on the violator and generates reports. tab
ase
Da
Da
tab
A terminal, which is a PC, allows the police officer or the officials to manipulate the software, retrieve reports, and process and access all jobs dedicated for users. The database contains records of all cars and vehicles in order to search (manually or automatically) for the specific car by referring to its plate number. The post office database is the one that exists in the post office. It only contains the form of the ticket in order to print it and deliver it to the violator via mail. D. System Design Figure 2 shows the dataflow diagram of the system design. When a violation occurs, the camera is triggered and a picture is taken. This picture is forwarded to a centralized server where the image scanning procedure begins. The segmentation technique is launched afterwards in order to detect the plate using the character recognition technique; the plate’s number is then retrieved. After getting the plate number, a database search is initialized with a main purpose to locate the specified record for the retrieved plate number. Once the record is found, the fee is charged according to the type of violation, the violations are differentiated referring to a certain IP address given to each camera. Then the information about the car and the violator are extracted from the record and a ticket form is issued. Finally, through a connection that is established between the local server and the post office database server, the form is sent in order to be delivered to the violator by mail. E. Use Case Diagram The use-case diagram, shown in Figure 3, depicts how the main users interact within the system. The system administrator has access to the system control and is allowed to issue tickets and reports. The administrator has the authority to access all parts of the system. Then comes the user who exists in the department of circulate vehicles. This user has access to issuing the ticket form and retrieving reports, as well as the issued ticket. Finally, the post office agent is responsible for delivering the ticket to the violator. Camera
Captured Violation
Take a Picture
Forwarded Picture
Image Scanning
as e
Recognized Plate
st Po ce fi Of de Si
ver Ser
Create Ticket Form
Access Record and Charge Fees
e am Fr bber a Gr Te rm ina
Identified Number
Retrieved Specified Get Specified Record Record
Issued Form
Camera with IP
`
l
Figure 1. Hardware Design.
Sent Record
Send Form to Post Office
Stored Form
Post Office
Database
Figure 2.
Main Database
System Design.
Plate Number Recognition
Ticket Delivery
Department of Circulate Vehicles
Issue Ticket Form & Reports Post Office Agent
System Control Administrator
Figure 3. Use-Case Diagram.
F. ER Diagram An entity relationship diagram shows the entities and their attributes that should be included in the application, as for all the relationships between them. First of all the Driver table has a Car_License_Num attribute as primary key and First_Name, Middle_Name, Last_Name, Mother_Name, Date_Of_Birth, Address, Phone_No, Nationality, Driver_License_Type and Driving_License_Num as attributes. The Driver table is related to the Car table by a 0-to-many relationship, the Car table includes English_Plate_Num as primary key, Arabic_Plate_Num, Car_License_Num, Car_Type and Car_Color as attributes. The Car table is related to a third table Ticket with a 0-to-many relationship. The Ticket table contains Ticket_Num as primary key, English_Plate_Num, Arabic_Plate_Num, Town, Street, City, Issue_DateTime, Fee_Cost and Ticket_Type as attribures. A ticket may be issued to one and only one car; and a car is owned by one and only one driver.
Car
0
Has
Own
PK English_Plate_Num
0
Ticket
Arabic_Plate_Num Car_License_Num Car_Type Car_Color
PK Ticket_Num
Driver PK Car_License_Num
English_Plate_Num Arabic_Plate_Num Town Street City Issue_DateTime Fee_Cost Ticket_Type
First_Name Middle_Name Mother_Name Last_Name Date_Of_Birth Nationality Driver_License_Type Address Phine_No Driving_License_Num
G. License Plate Recognition Algorithm The proposed license plate recognition algorithm consists of six major phases that start after having the picture captured by the specified camera when a certain violation occurs. The first phase is image analysis, the second is license plate extraction, afterwards convert the extracted plate into black and white, then characters segmentation is applied to the black and white plate to obtain separated characters. After that last process, character recognition takes place, once the license’s plate number is retrieved, the car is identified and a ticket form is issued. 1) Image Analysis The software’s job mainly starts after receiving images taken from cameras when a violation occurs. The image’s size is reduced to 800 by 600 pixels. The program analyzes the image pixel by pixel. It keeps checking until it detects a blue pixel. At this point, two ranges of blue color are defined. The first range is set for an average contrast and brightness while the second one is set for brighter plates just by modifying the Blue part of the RGB color mode, since there are 256 different ranges of blue (from 0 to 255) [4]. 2) License Plate Extraction Once a blue pixel is found in the predefined range, its position relative to X and Y coordinates are saved, assuming that this could be the beginning of what the software is searching for. The program keeps scanning and reading blue pixels horizontally until it finds a white pixel. If the program did not find a white pixel, then it discards the first blue pixel and continues looking for a new one. If a white pixel is found, then X1(x1,y1) becomes the starting X from the first blue pixel and Xf(x2,y1) becomes the ending X from the last blue pixel reached before the white one. The program then goes back to X1(x1,y1), which is the same as Y1(x1,y1), and starts scanning vertically until it reaches a black pixel, which is the black border of the plate. If no black pixel is found, the program discards X1, Xf and Y1 and continues searching for a new starting blue pixel. If a black pixel is found, then Yf(x1,y2) becomes the last blue pixel right before the first blue pixel of the frame. As it is known, there are two standard forms for the Lebanese plates, one which stretches for 50 cm horizontally and another one which stretches 30 cm horizontally and is vertically larger than the first one (Figure 5) [5].
Figure 4. ER Diagram.
Figure 5. Lebanese License Plates.
The program differentiates between the two shapes by calculating (Yf-Y1) / (Xf-X1) because the license plate shape is proportional to this ratio. If the floor of this ratio is 2, then the detected plate is the 50 cm wide; therefore multiplying (Xf-X1) by 10 encloses all the plate. Otherwise, if the ratio is 3, then the detected plate is the 30 cm wide, and multiplying (Xf-X1) by 6 encloses all the plate. The image analysis and the license plate extraction process are described in details in the flowchart of Figure 6.
Start
Send Picture To Terminal
False Blue Analysis Range1
Picture
Pixel N Test Range1
True
N=1
N < 800*600
False
False N=N+1
N=N+1
Blue Analysis Range2
4) Character Segmentation Once the black and white conversion is completed, the segmentation technique starts. Referring to the points obtained earlier, Xf, Y1 and Yf, the program starts from points Xf+1 and Y1 and analyze the picture vertically, searching for a black pixel and discarding all white ones.
Pixel N Test Range2
True
True
Xn=Xn+1
Xn=Xn+1
True
The program keeps on analyzing the pixels of the plate until it reaches Yf. At this pixel, the program moves one pixel horizontally Xf+2 and restarts analyzing on the Y-axis. Once a black pixel is found, it starts by reading pixels vertically and scanning at the same time horizontally several pixels in order to form a white line that corresponds to 3/2 the size of the plate. This white line, when found, will be the base for the new Yf, and the end of the Latin characters.
True Pixel N Test Range1 Until XnYf
True Xf = Xn Xn = X1 Y1 = Yn
Yn=Yn+1
New Xf = (Xf-X1)*10 True
True
Xf = Xn Xn = X1 Y1 = Yn
The next step in character segmentation is to copy each black pixel found to a new picture box until a new white line is reached, but this time vertically. This indicates the end of the black shape, which could be either noise or the plate’s frame. For this reason, the program creates nine picture boxes that represent the maximum number of Latin characters. After the Latin character segmentation, the segmentation of the Arabic characters starts. X becomes the ending pixel which is the last pixel of the extracted plate, and the first Yf is now the starting Y.
Yn=Yn+1
False
True
α
Pixel N Test Range1 Until Yn 96
Xn = Xn+1
False Xn-X1 > α
False
Yn > New Y
True
True
Copy Picture: New X à Xn Y1 à New Y
Read Pixel N(Xn,Yn)
R=255 True G=255 B=255
Count = Count+1 New X = Xn Yn = Y1
Yn = Yn+1
R=0 G=0 B=0
True
True
H = Yn – New Y
New Y = Yn Yn = Y1 Xn = New X Count = 0
False
Access Lib Size H
Compare Shape With Nshape (XOR)
Select NShape
True
End
False
PMax > Range
False
False
False Next NShape Exist
True
Xn = Xn+1 Yn = Y1
End
False
Count PMax
True
PMax = % Point to NShape
6) Car Identification and Ticket Form Issue The last phase consists of searching for the plate number that was recognized in the database. All the needed information concerning the car and its owner are available in the database. A form is created using Crystal Report which is identical to the ticket being issued. In case of invalid license number, the picture is saved in a separate folder which is dedicated to manual checking. This could be faced in case of fake plates or misrecognized ones. IV.
TESTING AND RESULTS
In order to test and validate the proposed Lebanese license plate recognition system, a GUI implementation of the algorithm was applied using Microsoft Visual Studio and the VB.NET language. The database was created and accessed using SQL server, and the ticket reports were prepared using Crystal Report. Figure 10 shows the interface for the application developed to test the Lebanese license plate recognition system. The user can choose the type of adjustment from a combo box that contains the different choices (Extract Plate, Character Recognition For Latin Characters, Character Recognition For Arabic Characters, Extract Plate and Recognition for Latin Characters, Extract Plate and Recognition for Arabic Characters). The nine picture boxes on the left side are designed to show the segmented Latin characters (Arabic numerals). The pictures boxes on the right side show the segmented Arabic characters (Hindu numerals). Figure 11 shows the results after image processing.
Figure 11. Results of processing the license plate.
CONCLUSIONS AND FUTURE WORK The Lebanese License Plate Recognition for Driving Tickets Automation System presents a contribution in reducing driving violations by providing an easy and efficient way in recognizing the license’s plate number and therefore the car’s owner. This process is beneficial for the state since it eliminates the bribes and evasion from payment problems because the system is fully automated. In addition, it provides more coverage so no violation will be undetected and thereof more tickets will be issued. This system can also be used in parking monitoring systems for security reasons. For example in companies that only want their own employees to have access to the parking area. Future work includes adding a server to send SMSs to violators, as well as working on the security of the system. Moreover, special Lebanese license plates with different colors and similar standards should be recognized by the system. REFERENCES [1] [2] [3] [4]
[5]
Figure 10. The Application Interface.
[6]
Technology Highlights, Elements of Typical LPR Systems, [Online], Available at: http://www.licenseplaterecognition.com. A Real-time vehicle License Plate Recognition (LPR) System, [Online], Available at: http://visl.technion.ac.il. Zamir recognition Systems Ltd., LPR Technology, [Online], Available at: www.zamir.co.il. Nick Efford, Digital Image Processing, A Practical Introduction Using Java, Addison Wesley, Pearson Education – 2000 – ISBN 0-201-596237. Fernando Martin, Maite Garcia, Jose Luis Alba, New Methods for automatic Reading of VLP’s (Vehicle License Plates). Wikipedia, “Vehicle registration plates of Lebanon”, [Online], Available at: http://en.wikipedia.org/wiki/Vehicle_registration_plates_of_Lebanon.
