Real-Time License Plate Localisation on FPGA

Real-Time License Plate Localisation on FPGA X. Zhai, F. Bensaali and S. Ramalingam School of Engineering & Technology University of Hertfordshire Hatfield, UK {x.zhai, f.bensaali, s.ramalingam}@herts.ac.uk Abstract Automatic Number Plate Recognition (ANPR) systems have become an important tool to track stolen car, access control and monitor the traffic. The fundamental requirements of an ANPR system are image capture using an ANPR camera, and processing of the captured image. The image processing part, which is a computationally intensive task, includes two stages i.e. plate localisation and character recognition. This paper presents an improved license plate localisation () algorithm based on modified Sobel vertical edge detection operator and two morphological operations suitable for FPGA implementation. The algorithm has been successfully implemented on a Xilinx Virtex-4 FPGA and tested using a database of 1000 images that contains UK number plates. It consumes 28% of the available on-chip resources, runs with a maximum frequency of 114.20 MHz, has a detection rate of 99.1% and capable of processing one image (640 480) in 3.8ms. 1. Introduction Automatic Number Plate Recognition (ANPR) systems are rapidly becoming used for a vast number of applications. These include, e.g. automatic congestion charge systems, access control, tracing of stolen cars, or identification of dangerous drivers, and automatic Electronic Toll Collection (ETC) system [1, 2]. The main task in an ANPR is the processing of the captured image. This task is computationally intensive and includes two stages: (i) license plate localisation (); and (ii) optical character recognition (OCR). normally requires two major sub-tasks. The first one is to enhance the License Plate (LP) and eliminate Non-License Plate (Non-LP) areas and the second one is extracting the LP. Once the LP is detected, the symbols or characters on the LP will be segmented before starting the next sub-task (i.e. OCR). From the LP, only segmented symbols or characters will be kept to be used by an OCR algorithm where they will be converted into a text format. ANPR has to be performed under real-time constrains in order to work fast enough to accommodate relative speeds of more than 70 mph (110 km/h) on a busy highway. Therefore, the common hardware choice for its implementation is often high performance workstations and expensive supercomputers connected to an ANPR camera. However, the cost, compactness and power issues that come with these solutions motivate the search for other platforms. Recent improvements in the computing power of Field Programmable Gate Arrays (FPGAs) and Digital Signal Processors (DSPs) have motivated researchers to consider them as low cost solution for accelerating such computationally intensive task [3]. This paper presents an improved algorithm with a high detection rate based on a combination of Sobel operator and two morphological operations. The paper also presents a new speed and area-efficient architecture based on the proposed algorithm suitable for FPGA implementation. The proposed architecture is implemented and verified using the Mentor Graphics RC240 FPGA development board equipped with a 4M Gates Xilinx Virtex-4 LX40 and a database of 1000 UK car images with a resolution of 640 480. A Matlab implementation of the proposed algorithm was used as a proof of concept prior to the hardware implementation. The rest of this paper is organised as follows. Related work is reviewed in Section 2. Section 3 is concerned with the description of the proposed improved algorithm. The proposed architecture is described in Section 4. The FPGA implementation and experimental results are provided and discussed in Section 5. Section 6 concludes the paper. 2. Related Work Recent advances in technology have taken ANPR systems from fixed applications to mobile ones. Recent improvements in the computing power of FPGAs and DSPs have motivated researchers to consider them as an alternative solution to implement ANPR systems. These devices can be used as low-cost System-on-chip solution that allows the processing FPGA or DSP unit to be placed within an ANPR cameras housing to create intelligent cameras, which can record and process images for sending back to a server directly. A variety of algorithms have been used in FPGA and DSP-based ANPR systems. These include AdaBoost and SVM [3, 4], Gabor filter [5], 14

morphological operation [6], background modelling [3] and pixels classification [7]. However, these algorithms are either computationally expensive (i.e. high execution time) or have low detection rate which affect their practicability. The only work that involves the use of only one FPGA to implement an ANPR system or part of it is the work presented in [7]. Results show that it has the fastest processing speed to locate the LP with a relatively low detection rate compare to other existing work. Table 1 summarises the most recent existing FPGA and DSP-based ANPR systems. Table 1: Existing Hardware-Based ANPR Systems ANPR System Background Modelling based system [3] AdaBoost and SVM based system [4] Gabor Filter based system [5] Morphological Operation based system [6] Pixels Classification based system [7] System Part Rate (%) 96 Whole ANPR system Whole ANPR system Whole ANPR system 91.70 87 Hardware Platform TI C6414 DSP and Altera FPGA TI C64 DSP and PC FPGA Virtex IV FPGA Virtex II Pro and PC FPGA Virtex II pro 3. Algorithm Based on Sobel Vertical Edge Operator Speed (ms) 141.62 <50 9.25 The license plate region normally has high density of vertical edge information compared to other parts of the car [1]. One of the efficient methods to extract this vertical information is the use of Sobel vertical edge detection operator [1]. In order to extract the correct plate region, the use of two morphological operations has been proposed to enhance the vertical information. Figure 1 shows the flowchart of the overall proposed system. 3.1. Plate Feature Extraction The proposed algorithm employs a Sobel vertical edge detection operator to eliminate Non-LP area, where the most of unwanted information will be erased after utilising a specific threshold. Figure 2 shows the vertical Sobel operator. Figure 2: Sobel vertical operator. Once the vertical operator multiplies its corresponding 3 3 pixels and scans the entire image, a gradient value based image is obtained, where the range of these values reflects the difference between the two neighbour pixels at the horizontal direction. Because the plate region has very high variability of grey-level, the gradient values within the plate region are larger than the other gradient value within other Non-LP regions. Therefore, a proper threshold can easily separate LP regions and Non-LP regions. Figure 3 shows this process. Figure 3: Sobel vertical edge detection operator and thresholding processes. (a) Greyscale Image; (b) Vertical Gradient Image; (c) Binary Edge Image. Although the most unwanted information can be eliminated after the thresholding, some of the information remains on Non-LP region when under complex background environments are considered. Therefore, two morphological operations are introduced. A morphological close operation firstly is used for fusing pixels together, and then a morphological open is used for filtering out the rest of the unwanted information. The Structuring Elements (SEs) used for these morphological operations are shown in Figure 4. Figure 1: Block diagram of the Sobel operator based system. Figure 4: (a) The 3 22 rectangle shaped SE for Close Operation; (b) The 3 5 rectangle shaped SE for Open Operation. The close operation uses the 3 22 rectangle shaped SE to effectively fuse the binary edge information in the LP region. The close operation is followed by an open operation that uses a 3 5 rectangle shaped SE to remove some small Non-LP areas where the size of connected pixel groups is smaller than this SE. Figure 5 shows the result of this process. 15

Figure 5: The Figure 3 (c) after using morphological close (a) and open operations (b). 3.2. Selection of Candidate Region As it can be seen from Figure 5, the LP region in the result image can be easily extracted using Connected Component Analysis (CCA), where two selection conditions are used. In the proposed work, the CCA uses a 4-connectivity method, and labels them using different number. Once all groups of pixels have been determined, each pixel is labelled based on the group it belongs to. Therefore, a set of potential candidates can be selected from the image using the known geometrical conditions, which are mainly consisted by the length, width and ratio of plate region. Let P denote the extracted plate region with the size W L, the first criterion is the ratio R between the width and length of P (i.e. R = L/W ). The second criterion is the ranges of W and L. The third criterion is the area of P. The ranges of W, L and R were selected to be relatively large enough to cover most of the possible sizes of the plate region in the database. Figure 6 shows a flowchart that illustrates the selection process. Figure 6: Selection process flowchart. Basically, there are two selection conditions (Condition1 and Condition2) used for this purpose. For both conditions, the length, width, area and ratio of LP are considered. Condition1 is stricter than Condition2 where some of the candidates may not meet Condition1 but can meet Condition2. The maximum and minimum coordinates of the rectangular plate regions that pass one of the conditions are returned. Normally, the strictest condition (i.e. Condition1) is perfectly suited for selecting candidates from clear images; while Condition2 can be used for selecting candidates from bad quality images. The experiment results show that about 96% of the test images met Condition1 and 75% of the remaining images met Condition2, which means this extra condition can further increase the detection rate by around 3% with no significant increase in the execution time. The final license plate will be extracted from original greyscale image. Figure 7 shows the selected license plate. Figure 7: Selection of license plate. (a) Labelled Image; (b) License Plate. The proposed algorithm was first tested in Matlab environment using a database of 1,000 images containing UK number plates. The images are taken from car parks and motorway during the daytime and evening time (Infrared Images). The implementation results show a detection rate of 99% on this database. The algorithm proposed in [8] is also based on a combination of Sobel and Morphological Operations. One of the main advantages of the proposed algorithm compared to the one proposed in [8] is that it uses only one morphological open operation to eliminate Non-LP area, while the algorithm proposed in [8] uses three morphological open and one image subtraction operations to eliminate Non-LP area before applying CCA algorithm, which requires extra time for Non-LP area elimination and affects the performance of the entire ANPR system. In addition to the above, our proposed algorithm uses two geometrical conditions to select LP region, while the algorithm proposed in [8] uses single geometrical condition to perform the LP region selection. The two geometrical conditions increase the detection rate in the case of images taken in complex environments. 4. Proposed Architecture The proposed algorithm uses morphological operations which are multiplier/divider-free operations. This feature makes the proposed algorithm suitable for FPGA implementation as less hardware resources will be required. In addition to the above, the parallelism offered by FPGAs can be exploited to implement Sobel Operator. The building blocks of the proposed architecture are shown in Figure 8. They consist of a memory reader, a converter module, vertical Sobel operator, morphological and CCA modules. Figure 8: The building blocks of the proposed architecture. 16

4.1. Memory Reader and Converter Modules The memory reader module is used to read the RGB values for each pixel from the original RGB image which has a size of 640 480 and assign a position coordinate to it. For this architecture the converter module converts the RGB (24 bits) pixel streams into 16-bit greyscale. Because the 8-bit word length is not enough to cover all the range of pixel values after performing Sobel vertical edge operator, the word length of each pixel value is extended to 16 bits, which includes 1 sign bit. This module is also used for the greyscale to binary conversion using a fix threshold of 650 out of 1785, which means all values less than 650 will be treated as 0 and values larger or equal 650 will be treated as 1. 4.2. Sobel Vertical Edge Detection Operator The Sobel vertical edge detection operator module consists of two sub-modules, which are an N3 window extraction and convolution modules. The two sub-modules run in parallel. The N3 denotes a local neighbourhood of size 3 3 used to achieve pixel level operations. Figure 9 shows this process. Each clock cycle, the 16-bit pixel values will be passed into this module one by one. The same pixel value will be sent to Line Buffer 0 and Stage 0 at the same time. Once Line Buffer 0 is full, the pixel will be passed to Line Buffer 1; meantime, the values in Stage 0 and Stage 1 will be updated accordingly. After [(640 2)+2+3] clock cycles the first 3 3 matrix will be filed and after that the throughput rate of the matrices is one clock cycle (640 is the size of each row in the image, 2 is the number of buffers used and 3 is number of columns of the matrix). Figure 9: The block diagram of the N3 window extractor. The second sub-module is the Sobel vertical edge detection operator module, where the convolution of N3 and 3 3 mask is calculated. In order to simplify the setting of threshold, the Sobel vertical mask has been modified as shown in Figure 10. Figure 10: Sobel vertical edge detection mask for FPGA. The mask is designed for calculating the variance of edge information from horizontal direction. Firstly, this calculation produces the gradient value in horizontal orientation (Gx) and then the absolute magnitude of the horizontal gradient is calculated. The gradient magnitude actually can indicate where the vertical edges are, and using this modified mask larger gradient value can be obtained. This helps to set a single threshold for segmenting LP and Non-LP region. Figure 11 shows a block diagram of this convolution process. Figure 11: Block diagram of convolution process. In Figure 11, the group of pixels (P1, P2, P3), (P4, P5, P6) and (P7, P8, P9) are multiplied simultaneously by (-1, 0, +1), (-7, 0, +7) and (-1, 0, +1) respectively followed by an addition of the partial products. An additional addition is applied to the three results from the previous calculation. 4.3. Morphological and CCA Modules The morphological operations module consists of two sub-modules, which includes the morphological close and the morphological open sub-modules. The morphological close operation and the morphological open operation can be divided into morphological dilation and erosion operations. The binary dilation calculates the maximum pixel value in a specific SE. On the contrary, the binary erosion calculates the minimum value in a specific SE. The proposed algorithm uses a 3 22 rectangular shaped SE, however, for efficient hardware implementation where parallelism can be exploited, this rectangular shaped SE has been decomposed into two smaller rectangular SEs with sizes 1 22 and 3 1. Figure 12 shows the block diagram of the proposed pipelined dilation filter. Initially, the value of current input pixel (binary) is simultaneously passed into the internal Stage 0 and Line Buffer 0 then after every clock cycle it is passed to the next stage until it reaches Stage 21 and then the maximum pixel value of the current 22 pixels in the 22 stages is calculated. Meanwhile, the values of the pixels from two consecutive lines of the 17

binary image (i.e. 640 pixels per line) are stored into the two line buffers in order to calculate the maximum value from three consecutive pixels from the same column. The first origin of SE (1 22) is the twelfth pixel of the first line, so the first coordinate of output should be kept same as the coordinate of the twelfth pixel instead of the coordinate of the current input pixel. Figure 12: The block diagram of a pipelined dilation filter (3 22). The structure of the erode filter is similar to the dilate filter. The only difference is that the minimum value of the pixels is calculated instead of the maximum one. Figure 13 shows the block diagram of a pipelined erode filter. In the proposed architecture, there are two different SEs used for the two morphological operations (i.e. rectangle shaped SEs: 3 5 and 3 22) which are easily implemented using the block diagrams shown in Figures 12 and 13 by simply changing the number of stages (i.e. if the size of SE is 3 5, it requires five stages). Figure 13: The pipelined erode filter (3 22). The CCA module is used to mark and select candidate plate region from the entire binary image. Generally, the pixels of the input pixel stream are divided into several groups or blobs by the CCA module. The grouping is based on the pixels connectivity. Figure 14 demonstrates this procedure. Figure 14: The block diagram of CCA. The grouping is performed as follows. The binary stream is scanned from left to right starting from the top line. For instance, a comparison between the current pixel P1 from Figure 14, its upper pixel P1A and left pixel P1L, which have already been grouped, is performed. All pixels with value 0 will be assigned to one group with an index 0. If the value of P1 is 1 and the indexes of its neighbours are the same and not 0 then P1 will be assigned the same index as its neighbours. If the indexes of the two neighbours are different and not 0, then the indexes of this pixel and its upper neighbour P1A will be the same as its left neighbour (i.e. P1L ). If the indexes of the two neighbours are different and one of them is 0, then the index of this pixel will be the non-zero index of its neighbour. If the pixel value is 1 but the indexes of its neighbours are both 0, the index of a new group will be assigned to this pixel. Finally, the coordinates of each rectangular shaped group are recorded for the selection of candidates. Once the whole image is scanned, a selection of candidate region is performed using the selection process shown in Figure 6 which is mainly based on the geometrical relationship of LP region. 5. FPGA Implementation and Results The proposed architecture for has been successfully simulated in PAL Virtual Platform (PALSim) [9]. The total numbers of clock cycles for processing one image is 396567. After simulation, the architecture has been successfully implemented and verified using the Mentor Graphics RC240 FPGA development board equipped with a 4M Gates Xilinx Virtex-4 LX40 [10]. Handel-C and PixelStreams which is a library that can be used for rapid development of video image streaming applications have been used for the hardware description of the proposed architecture [11]. Table 2 shows the on-chip resource usage of the proposed architecture. Table 2: On-Chip Resources Usage Used Available Utilisation Occupied Slices 5,195 18,432 28% LUTs 7,168 36,864 19% Block Rams 17 96 17% The proposed architecture only requires about 28% on-chip resources, which means the 72% remaining on-chip 18

resources can be used for implementing the second part of ANPR (i.e. Segmentation of LP and OCR). The maximum running frequency is 114.2 MHz. The execution time for processing one frame can be roughly calculated using the following equation: c f Where T is execution time in ms; C is the number of clock cycles needed for one image; and f is the maximum running frequency. Based on Equation 1, the architecture can process one image and produce a result in 3.8 ms. This means that the proposed architecture satisfy the minimum requirement for real-time processing. The result achieved in terms of maximum running frequency and area used for implementing this important part of an ANPR system shows that there is enough room for implementing the whole ANPR system on one FPGA. The hardware implementation of the proposed algorithm has been compared with the best two DSP and FPGA hardware implementation approaches from Table 1. The overall performances of each system are shown in Table 4. Results achieved for the FPGA implementation of proposed architecture have shown significant improvements in term of rate and execution time under higher image resolution. The proposed design uses a database of 1000 images that contains UK number plates, which includes a variety of images taken at different times and conditions (e.g. car park, motorway and night time) which significantly affect the recognition rate. Algorithm Proposed algorithm Background Modelling [3] Pixels Classification [7] Table 4: Comparison with Existing Work Device FPGA Virtex-4 DSP C6414 and FPGA FPGA Virtex II Clock Speed (MHz) Image Size (pixels) Time (ms) (1) Rate (%) 114.2 640 480 3.8 99.1 600 (DSP) 6. Conclusions and Future work 352 288 141.6 96 72.06 256 256 9.25 87 Owing to the importance of in an ANPR system, an improved algorithm based on Sobel vertical operator and two morphological operations to efficiently detect LPs have been proposed in this paper. A new architecture based on the proposed algorithm suitable for FPGA implementation has also been proposed. Hardware implementation results have shown that FPGA can be used as low cost accelerator to implement an ANPR system under real-time environment. The architecture requires 28% of the available on-chip resources of a Virtex-4 FPGA device. Parallel building blocks have been used for the FPGA implementation and the whole system runs with a maximum frequency of 114.200 MHz and is capable of processing one image in 3.8 ms with a localisation rate of 99.1%. The 28% resources usage of the FPGA to implement the leaves 72% of the FPGA area free to be used for the remaining parts of an ANPR system (i.e. Segmentation of LP and OCR). This allows the entire ANPR system to be implemented on an FPGA that can be placed within an ANPR camera housing to create a standalone unit which will drastically improve energy efficiency and remove the installation and cabling costs of bulky PCs. References [1] C. N. E. Anagnostopoulos, I. E. Anagostopoulos, V. Loumos and E. Kayafas. A license plate-recognition algorithm for intelligent transportation system applications. IEEE Trans. Intell. Transp. 7(3): 277-291, 2006. [2] F. Lian, Y. Fan and Y. Zhang. Study of technology in electronic toll collection. Journal of Compute Engineering and Application, Vol.43, pp.204-207. [3] C. Arth, C. Leistner and H. Bischof. TRIcam: an embedded platform for remote traffic surveillance. The IEEE Conference on Computer Vision and Pattern Recognition, pp.125 125, 2006. [4] A. Clements, L. Florian and B. Horst. Real-time license plate recognition on an embedded DSP-platform. IEEE Conference on Computer Vision and Pattern Recognition, pp.1-8, 2007. [5] H. Cancer, H. S. Gecin and A. Z. Alkar. Efficient embedded neural-network based license plate recognition system. IEEE Transactions on Vehicular Technology, 57(5): 2675-2683, September 2008. [6] N. Bellas, S. M. Chai, M. Dwyer and D. Linzmeiser. FPGA implementation of a license plate recognition SOC using automatically generated streaming accelerators. The International Parallel & Distributed Processing Symposium, 2006. [7] T. Kanamori, H. Amano, M. Arai, D. Konno, T. Nanba and Y. Ajioka. Implementation and evaluation of a high speed license plate recognition system on an FPGA. The 7th IEEE international Conference on Computer and information Technology, pp.567-572, 2007. [8] P. V. Suryanarayana, S. K.Mitra, A. Banerijee and A. K. Roy. A Morphology Based Approach for Car license Plate Extraction. IEEE Indicon 2005 Conference, Chennai, India, pp.11-13, 2005. [9] PixelStreams User Manual. Graphics, Mentor. http://www.mentor.com/, Mentor Graphics Corporation, 2010, [Accessed 19 September 2010]. [10] PAL User Manual. Graphics, Mentor. http://www.mentor.com/, Mentor Graphics Corporation, 2010, [Accessed 15 September 2010]. [11] RC240 Datasheet. Graphics, Mentor. http://www.mentor.com/, Mentor Graphics Corporation, 2010, [Accessed 18 September 2010]. 19