i PROJECT S TITE DESIGN AND DEVELOPMENT OF A LICENSE PLATE RECOGNITION SYSTEM USING CONVOLUTIONAL NEURAL NETWORK EH ZHENG YI This Report Is Submitted In Partial Fulfillment of Requirement For The Bachelor Degree of Electronic Engineering (Industrial Electronic) Fakulti Kejuruteraan Elektronik dan Kejuruteran Komputer Universiti Teknikal Malaysia Melaka JUNE 2016
ii DECLARATION UNIVERSTI TEKNIKAL MALAYSIA MELAKA FAKULTI KEJURUTERAAN ELEKTRONIK DAN KEJURUTERAAN KOMPUTER BORANG PENGESAHAN STATUS LAPORAN PROJEK SARJANA MUDA II Tajuk Projek : An Improved Standard Malaysian License Plate Recognition using Convolutional Neural Networks Sesi Pengajian : 1 5 / 1 6 Saya EH ZHENG YI. (HURUF BESAR) mengaku membenarkan Laporan Projek Sarjana Muda ini disimpan di Perpustakaan dengan syarat-syarat kegunaan seperti berikut: 1. Laporan adalah hakmilik Universiti Teknikal Malaysia Melaka. 2. Perpustakaan dibenarkan membuat salinan untuk tujuan pengajian sahaja. 3. Perpustakaan dibenarkan membuat salinan laporan ini sebagai bahan pertukaran antara institusi pengajian tinggi. 4. Sila tandakan ( ) : SULIT* *(Mengandungi maklumat yang berdarjah keselamatan atau kepentingan Malaysia seperti yang termaktub di dalam AKTA RAHSIA RASMI 1972) TERHAD** **(Mengandungi maklumat terhad yang telah ditentukan oleh organisasi/badan di mana penyelidikan dijalankan) TIDAK TERHAD Disahkan oleh: (TANDATANGAN PENULIS) (COP DAN TANDATANGAN PENYELIA) Tarikh:.. Tarikh:..
iii I hereby declare that the work in this project is my own except for summaries and quotations which have been duly acknowledge. Signature :... Author :... Date :...
iv DEDICATION I acknowledge that I have read this report and in my opinion this report is sufficient in term of scope and quality for the award of Bachelor of Electronic Engineering (Industrial Electronics) with Honours. Signature :... Supervisor s Name :... Date :...
To my beloved father and mother v
vi ACKNOWLEDGEMENT It is a genuine pleasure to express my deep sense to thanks and gratitude to my supervisor, Dr. Syafeeza binti Ahmad Radzi. Her dedication and keen interest above all her overwhelming attitude to help and mainly responsible on my final year project. Her advice and scientific approach have helped me to a very great extent to accomplish my final year project. I would also thank to Universiti Teknikal Malaysia Melaka and Fakulti Kejuruteraan Elektronik dan Kejuruteraan Komputer who provided a very good platform for student to learn and give endless support to accomplish my final year project. Lastly, I would like to thank to family and friends and others who are support either morally, financially and physically in this project. Your support, I am deeply appreciated.
vii ABSTRACT The number of vehicles on the road has increased drastically in this decade. Due to this matter, the implementation of license plate recognition system is needed that can help to control and surveillance on traffic flow. But, in previous research the license plate recognition system performed low accuracy on character recognition stage. So, this research is carried out to make improvement in recognition stage. This research is to develop a license plate recognition system that used to recognize the standard Malaysian license plate characters using Convolutional Neural Network. The license plate recognition system consist of pre-processing process that can localize license plate region in a sample and follow by segment license plate character for Convolutional Neural Network recognition. Besides that, the parameters used by four layered Convolutional Neural Network are analyzed and the best Convolutional Neural Network architecture for license plate character recognition is obtained. As a result, license plate pre-processing stage achieved 74.7% accuracy and Convolutional Neural Network recognition stage achieved 94.6% accuracy.
viii ABSTRAK Kenderaan di jalan raya telah meningkat secara mendadak dalam dekad ini. Oleh itu, pelaksanaan sistem pengecaman nombor plat kenderaan amat diperlukan untuk mengawal dan pengawasan terhadap aliran trafik. Tetapi sistem pengecaman nombor plat kenderaan yang dihasilkan dalam kajian sebelum ini, ketepatan dalam peringkat pegecaman nombor plat adalah rendah. Jadi, kajian ini dijalankan untuk penambahbaikan dalam peringkat pengecaman nombor plat. Kajian ini dijalankan adalah untuk menghasilkan satu sistem pengecaman nombor plat kenderaan yang berupaya mengecam standard nombor plat kenderaan Malaysia dengan menggunakan teknik rangkaian neural konvolusi. Sistem pengecaman nombor plat kenderaan terdiri daripada proses prapemprosesan yang boleh menyetempatkan nombor plat dalam sampel gambar dan diikuti dengan pengecaman nombor plat dengan menggunakan teknik rangkaian neural konvolusi. Selain itu, parameter yang digunakan dalam empat lapis rangkaian neural konvolusi telah dianalisis dan model rangkaian neural konvolusi yang terbaik untuk pengecaman nombor plat lesen telah diperolehi. Akhirnya, ketepatan nombor plat pra-pemprosesan mencapai 74.7% dan ketepatan pegecaman nombor plat dengan menggunakan teknik rangkaian neural konvolusi mencapai 94.6%.
ix TABLE OF CONTENT CHAPTER CONTENTS PAGE PROJECT S TITE... i DECLARATION... ii DEDICATION... iv ACKNOWLEDGEMENT... vi ABSTRACT... vii ABSTRAK... viii TABLE OF CONTENT... ix LIST OF TABLES... xii LIST OF FIGURES... xiii LIST OF ABBREVIATIONS... xv I INTRODUCTION... 1 INTRODUCTION... 1 PROBLEM STATEMENT... 3 OBJECTIVES OF THE STUDY... 4 SIGNIFICANCE OF THE STUDY... 4 SCOPE... 4 II LITERATURE REVIEW... 5 THEORY... 5 2.1.1 CNN... 5 2.1.2 JPEG... 7
x 2.1.3 Pixel... 7 2.1.4 Malaysia standard vehicle license plate... 8 2.1.4.1 License plate specification... 8 2.1.4.2 License plate measurement... 8 2.1.5 Grayscale image... 9 2.1.6 Binary image... 10 2.1.7 Sobel operator... 10 2.1.8 Morphology operation... 11 2.1.9 Connected component analysis... 13 2.1.10 Normalization... 13 LITERATURE REVIEW... 14 III METHODOLOGY... 26 FRAMEWORK FOR LPR SYSTEM DEVELOPMENT... 26 PRE-PROCESSING... 26 3.2.1 License plate localization... 26 3.2.1.1 Conversion from RGB to grayscale... 28 3.2.1.2 Sobel operator edge detection... 29 3.2.2 Character isolation from license plate area... 30 3.2.2.1 License plate image binarization using Otsu s approach... 30 3.2.2.2 License plate isolation using morphological operation.... 31 3.2.2.2.1 Median filtering... 31 3.2.2.2.2 Dilation... 31 3.2.2.2.3 Erosion... 32 3.2.2.2.4 Flood fill process... 32 3.2.3 Character segmentation... 32 3.2.3.1 License plate isolation using connected component analysis (CCA)... 32 3.2.3.2 Character resize and padding on segmented image... 33 3.2.3.3 Normalization... 34 CNN RECOGNITION... 34
xi 3.3.1 CNN architecture... 34 3.3.2 CNN system integrate into MATLAB... 35 LPR PROGRAM GUI... 37 IV RESULT AND ANALYSIS... 38 INTRODUCTION... 38 LPR SYSTEM PRE-PROCESSING OUTCOME... 38 4.2.1 Load image... 38 4.2.2 Grayscale... 39 4.2.3 Sobel operator edge detection and license plate localization... 40 4.2.4 Character isolation using Morphological operation... 42 4.2.5 Character segmentation using connected component analysis... 44 4.2.6 Character resize, padding and normalization... 44 ARCHITECTURE OF CNN... 45 4.3.1 Feature map... 46 4.3.2 Regulation parameter... 47 4.3.3 Pattern of connection... 48 4.3.4 Type of weight selection... 49 4.3.5 Y-constant... 50 LPR SYSTEM GUI... 50 PERFORMANCE OF THE DESIGNED LPR SYSTEM... 52 V CONCLUSION AND RECOMMENDATION... 53 CONCLUSION... 53 RECOMMENDATION... 53 REFERENCES... 55
xii LIST OF TABLES NO TITLE PAGE 1.1 Scope of this project... 4 2.1 The color depth that can represent in a pixel with different value of bpp... 7 2.2 The analysis of previous LPR system researches.... 14 2.3 Analysis of CNN researches... 25 4.1 Six type of connection... 48 4.2 Connection between C1 and C2 layers... 48 4.3 Type of weight versus accuracy... 49 4.4 Y-constant versus accuracy... 50 4.5 Accuracy of LPR system... 52
xiii LIST OF FIGURES NO TITLE PAGE 2.1 (a) convolution and subsampling LeNet-5 CNN architecture, and (b) fused convolution/subsampling in proposed CNN... 6 2.2 Malaysian standard vehicle license plate... 8 2.3 The standard measurement of the Malaysian vehicle license plate... 9 2.4 Grayscale image... 9 2.5 Binary image... 10 2.6 Image after applied by Sobel operator... 11 2.7 Morphology operation using structuring element... 12 2.8 Morphology dilation image... 12 2.9 Morphology erosion image... 12 2.10 Connected component in a binary image... 13 3.1 Framework of the LPR system development... 27 3.2 Flow chart of LPR system... 28 3.3 Conversion from RGB into grayscale image... 29 3.4 License plate localization using Sobel operator edge detection... 30 3.5 License plate area successful localized... 30 3.6 License plate binary image... 31 3.7 License plate isolation using Morphological operation... 32 3.8 Each of character is segmented into one image... 33 3.9 License plate image resized and padding into 22x12 pixel image... 34 3.10 Final architecture of CNN model... 35 3.11 mex function in C program... 36 3.12 C program execution algorithm in MATLAB... 36 3.13 Function mexgetvariableptr to retrieve number of characters need to be recognition.... 36
xiv 3.14 License plate character in Text form... 37 3.15 LPR system GUI... 37 4.1 Algorithm for load image file... 39 4.2 Jpeg format image is loaded into LPR system GUI... 39 4.3 Algorithm for grayscale process... 40 4.4 Grayscale image... 40 4.5 Sobel operator edge detection algorithm... 41 4.6 Sobel operator edge detection output... 41 4.7 Histogram of number of edge pixel in an image... 42 4.8 License plate localization... 42 4.9 Morphological operation... 43 4.10 License plate image after morphological operation... 43 4.11 Connected component character segmentation... 44 4.12 22x12 pixel character image... 45 4.13 Numeric data of single character... 45 4.14 Mean square error versus feature map graph... 46 4.15 Mean square error versus regulation parameter graph... 47 4.16 LPR system GUI... 51
xv LIST OF ABBREVIATIONS bpp - bits per pixel CNN - Convolutional Neural Network GUI - Graphic User Interface ITS - Intelligent Transport System LPR - License Plate Recognition
1 CHAPTER 1 INTRODUCTION 1 Introduction License plate recognition (LPR) system has become significantly important when it comes to security and traffic facilities. It applies optical character recognition method on images to read the vehicle registration plates. In terms of security, LPR has been used in traffic management to detect the owner of the car who has breach the traffic laws and to find stolen vehicles. LPR system is also used for access control to enter a building. The Automatic LPR system was introduced in 1979 at the Police Scientific Development Branch in United Kingdom for security purposes. Image processing is the main technique to be used in LPR recognition system. Therefore, developing a LPR system is challenging since the LPR image can appear to be dirty, motion blur, poor resolution, poor lighting, low contrast, etc. There are six primary algorithms for identifying a license plate. At the beginning, LPR system will use license plate localization technique to find and isolate the license plate on the input image. After that will follow by plate orientation and sizing which is responsible to compensate for the skew of the plate and to adjust the dimensions to the required size. Then, image normalization will perform to adjust the brightness and contrast of the image. The character segmentation will perform to segregate individual character from the license plate. Last step is apply optical character recognition technique to identify the individual characters. The recognition part of LPR system has almost a routine algorithm. It involves adaptive thresholding, component labeling, feature extraction and classification. Among
2 the five major parts, the character recognition process is the most challenging part. This is because, the recognition of the characters is highly dependent on the type of algorithms applied in the first four major parts. In fact, the segmented characters can appear in various looks. Therefore, a robust character recognition method is required and CNN is identified to cope with the mentioned challenges. CNN is well-known as a scale and rotation invariant in pattern recognition tasks. CNN accepts raw images that have been preprocessed with minimal preprocessing algorithm and train the input samples in supervised mode. It combines compression (dimensionality reduction), feature extraction and classification processes in a single architecture. Until now, CNN has been applied to various applications such as face detection [1-9], face recognition [10-14], gender recognition [15, 16], object recognition [17-19], character recognition [20-22], texture recognition [23], etc. Despite the listed advantages, CNN has limitations in terms of cost and speed. This is due to compute intensive image processing algorithm being incorporated in the design such as convolution and subsampling. The convolution process takes almost 90% of the processing time [24]. Therefore, in order to overcome the limitation, designing a small CNN size could aid in reducing the processing time. When applying CNN for LPR system, the step number (3) is not required as CNN is robust to illumination changes. For license plate recognition using CNN, several works have been reported in [24, 25]. In [25], they implemented LeNet-5 architecture with 7 layers by inserting the whole license plate as input and reported 98.25% accuracy. This work classifies between license plate and non-license plate and not recognizing the characters. They used 2400 license plates and 4000 non-license plate dataset and divided into train and test dataset. Since [25] is more on license plate detection, the accuracy rate is incomparable with this work. In [24], a special form of CNN called space displacement neural network (SDNN) is used to detect license plate image and the normal form of CNN is used to classify 36 characters (numbers 0-9 and letters A-Z) with a reported accuracy of 98.14%. They used 4800 training set comprising of individual plate characters that are manually segmented and labeled. The recognition architecture comprises of 4 layers with layer 1-4 uses 5, 14, 76 and 36 feature maps respectively. Levernberg Marquardt algorithm is used as the
3 learning algorithm. SDNN incorporates segmentation process inside CNN and has the ability to deal with large image by accepting the whole word into CNN. However, the algorithm is complex since several segmentation processes are commenced and the one with the highest scores will be selected for CNN training. This paper focuses on the character recognition part of the LPR system to improve the recognition rate. Since the network training time depends on the size of the CNN architecture, this paper discusses on the final CNN model that has the ability to produced 100% accuracy of identifying license plate characters. Problem statement The number of vehicles on the road has increased drastically in this decade. According to the Malaysia Automotive Association s statistic from year 2006 to 2015, there are more than 480,000 new vehicles registered every year. This huge number of vehicles on the road caused difficulties to control the traffic flow. The vehicle inspection operation becomes a challenging task. This causes implementation of the LPR system in intelligent transportation system (ITS) becomes significant. By this implementation, LPR system can help with surveillance and control the traffic flow. Due to this, the research of LPR system has become a popular topic by the researches. But, according to previous researches on LPR system, the system performed with low accuracy on character recognition stage. Those low accuracy character recognition techniques such as template matching and probabilistic neural network. Due to this matter, this project is proposed to overcome this problem with better character recognition technique.
4 Objectives of the study i. To develop a LPR system by using CNN method. ii. To propose an efficient and high accuracy algorithm for LPR system that can be used practically in the future. iii. To design a high accuracy CNN model that can recognize the license plate character efficiently. Significance of the study The importance of this research is to implement a high accuracy and efficient algorithm for a LPR system. The LPR system is developed to recognize the license plate number in a static image automatically. Furthermore, CNN is implementing to the LPR system for character recognition process in order to improve the recognition rate. Scope This research is to develop a LPR system. The project scopes in this research are listed in Table 1.1. Table 1.1: Scope of this project Project scopes Method or specification Tool to develop LPR system MATLAB software Coding involved MATLAB and C language Recognition method used CNN technique Type of license plate to be recognized Standard Malaysian vehicle license plate only Mode of LPR system Offline mode which only static images for input in this system. Resolution of the image 16 megapixels Developed system Software only without any hardware involve
5 CHAPTER 2 LITERATURE REVIEW 2 Theory 2.1.1 CNN CNN was proposed by LeCun in 1989 purposely for handwriting recognition. The CNN architecture was called LeNet-5 consisted of 7 layers. The architecture has an alternated convolution and subsampling layer twice as the feature extraction layer before connecting it to a three-layer multilayer perceptron (MLP) as the classifier. The vast number of layers has slowed down the training process since there are twice of convolution layer. In fact, subsampling layer has also contribute in slowing down the training process. In 2012, Mamalet and Garcia [26] have formulated a method of reducing the number of layers. The convolution and subsampling layer are fused resulting in simplifying the CNN architecture. Equation (1) indicates the fused convolution/subsampling process. l l l 1 l y pj f w ji x,, pi c u r v b j (1) u v l 1 l i M j u, v K c mod Sx = 0, r mod Sy = 0.
where K u, v N 0 u k ; 0 v x k y 2,, kx and ky are the width and the height of the convolution kernels wji (l) of layer(l) and bj (l) is the bias of feature map n in layer(l), c and r refer to the current pixel, and p refers to the particular training sample. The set Mj (l- 1) contains the feature maps in the preceding layer(l-1)that are connected to feature map n in layer(l). The notation f is the activation function of layer(l). The variables u and v describe the horizontal and vertical step size of the convolution kernel in layer(l).the final layer is a full connection following equation (3). 6 l 1 N l l l l y pj f x pi w ji b j i 0 1 (2) x pi (l-1) ( r, c) 1 step size Convolution y pj (l) Subsampling Mx (l) w ji (l) My (l) ky Sy kx (a) Sx x pi (l-1) ( r, c) Sx Mx (l-1) Mx (l) y pj (l) Sy My (l-1) ky w ji (l) My (l) kx Convolution + Subsampling 2 step size (b) Figure 2.1: (a) convolution and subsampling LeNet-5 CNN architecture, and (b) fused convolution/subsampling in proposed CNN where N (l-1) is the number of neurons in the preceding layer (l-1), wji (l) is the weight for connection from neuron i in layer (l-1) to neuron j in layer(l),bj (l) is the bias of neuron j in layer(l), and f represents the activation function of layer(l).
7 2.1.2 JPEG JPEG is a format for compressing image files. The word JPEG is the acronym for Joint Photographic Expert Group. JPEG is used lossy compression for an image. It is designed to compress the color and grayscale image. For lossy compression, the file size of the image is reduced by discarding the information of the image. But this change will not detect by human eye. JPEG format images able to support 16 million colors which achieved true color standard. The JPEG format is widely used in the digital camera s photograph and other photographic image capture devices. The maximum image size of can be supported is 65,536 65,536 pixels. 2.1.3 Pixel A pixel is the basic unit that used in digital image. It is a programmable color used by computer to identify and display the color on the screen. Resolution of an image is represented by using the unit of pixel. The higher the pixel used in an image, the closer the color between the image and the actual object. The number of different colors that can be differentiated in a pixel which is depends on the number of bits per pixel (bpp). A binary image using 1 bpp to represent the image. Each pixel can either on (white) or off (black). A 1 bpp image also can be known as monochrome. The Table 2.1 shows the number of bpp and number of colors that can be represent in a pixel. Table 2.1: The color depth that can represent in a pixel with different value of bpp Bits per pixel (bpp) Color depth per pixel 1 2 (monochrome) 2 4 3 8 8 256 16 65,536 (High color) 24 16,777,216 (True color)
8 A 16 bpp image can be known as high color graphic. The computer requires 2 bytes data space to store each pixel for the image information. One of the RGB component has 64 level intensity and other two components have 32 level intensity, so that total have 65,536 available colors using this high color graphic. True color support 24 bpp for three RGB colors. In each pixel it requires one byte needed to be stored for each channel. The true color has total 16,777,216 color variations. It usually used to display high quality photographic images or complex graphics. 2.1.4 Malaysia standard vehicle license plate 2.1.4.1 License plate specification Malaysian standard vehicle license plate rules derive from United Kingdom. The license plate, color, size, layout, shape and the size of the character are adhering to the guidelines provided by Road Transport Department Malaysia. The standard license plate is allowed to be used by Malaysian ordinary vehicle is license plate with white alphabets and numbers which embossed or glued on the black plate. Figure 2.2: Malaysian standard vehicle license plate 2.1.4.2 License plate measurement Arial bold is the standard character that can be used on the Malaysian license plate. Other than that, Road Transport Department Malaysia also standardizes the spacing of the license plate letters. The gap between the letters are 10mm. Spacing between alphabet group and number groups is 30mm. The height and width of the letters are 70mm and 40mm respectively, with a thickness of 10mm.
9 Figure 2.3: The standard measurement of the Malaysian vehicle license plate 2.1.5 Grayscale image Grayscale is a technique that processes a color digital image into a gray color image. Each of the pixels in the image will carry only intensity information. Where black pixel which carry weakest intensity value and white pixel carries strongest intensity value. The brightness level of a pixel is represented as a number from 0 to 255 or in binary from 00000000 to 11111111 which it only needs 1 byte to store for each pixel. Figure 2.4: Grayscale image