Sustav prepoznavanja slikovnih uzoraka mobilnim uređajem. Image pattern recognition using mobile devices

Transcription

1 Sustav prepoznavanja slikovnih uzoraka mobilnim uređajem Image pattern recognition using mobile devices Alen Huskanović, David Ante Macan, Zoran Antolović, Boris Tomaš, Marko Mijač SAŽETAK Prepoznavanje slikovnih uzoraka se do sada obavljalo isključivo koristeći ili složena i skupa programska rješenja i uređaje (optičke čitače). Mobilni uređaji razvili su se do mjere u kojoj ih je moguće iskoristiti za takve obrade, ali algoritmi i sustavi koji to rade u realnom vremenu su rijetki. Ovaj rad opisuje razvoj algoritma u sklopu PEAS sustava, kojim je omogućeno prepoznavanje uzoraka na listi za odgovore (eng. scorecard) koristeći mobilne uređaje. Pri tome ćemo prikazati teorijske osnove prepoznavanja slikovnih uzoraka, implementaciju algoritma pomoću OpenCV i JavaCV biblioteka te praktična iskustva u takvom pristupu. ABSTRACT Image pattern recognition has been so far carried out almost exclusively using sophisticated and expensive software solutions and devices. Mobile devices have evolved to the extent that they can be used for such purpose, but the algorithms and systems that provide such functionalities in real time are rare. This paper describes the development of an algorithm within PEAS system, which enables the identification of answers in scorecard paper sheet using mobile devices. Thereby we will show the theoretical basis of the image pattern recognition, implementation of the algorithm using the OpenCV library and JavaCV, and the practical experience in such an approach. 1. INTRODUCTION Today, image processing has found use in many aspects of ICT technologies. But, until recently, this task was mainly reserved for desktop computers, workstations, server computers, due to its high hardware requirements. However hardware and software capabilities of today s mobile device are increasing rapidly, which among other things results in image processing becoming more and more available on those devices as well. Since the mobile devices are compact and highly portable, new possibilities are opening from many aspects, including the image processing one. Such possibilities include computer vision, virtual reality, augmented reality, pattern recognition etc., finding its application in entertainment industry, health, security, government, biometrics, tourism, education... One of the possibilities for utilizing image processing is its application in overall educational process. This paper shows the attempt to utilize mobile devices and image processing capabilities to improve evaluation process in education, as a part of PEAS ( Personal Exam Assistant ) project. PEAS tries to automate and speed up the process of exam evaluation in educational institutions, thereby realizing benefits for both teacher and students. A significant part of PEAS system is a mobile application whose role is to take a picture of scorecard sheet, identify the marked answers and send them to server system for evaluation. The central issue in this mobile part of PEAS system was the identification of marked answers on scorecard sheet in native mobile application. In order to achieve this functionality, we aimed to develop algorithm for distinctive marked rectangle detection from the mixed grid of marked and unmarked rectangles. Although used for the reason stated above, this algorithm can easily be modified for recognition of various different image pattern types. The algorithm itself is fragmented into several smaller partial problems which we managed to surpass successfully. Such partial problems are:

2 contour and edge detection, corner calculation, perspective correction, shadow removal, answer identification, answer marking on the picture itself... As previously mentioned, today s mobile devices have the capacity for such processing tasks, as well as the cameras of enough quality level, so that the images could be easily fetched from the devices camera and instantly prepared for further processing. In order to ease the implementation of algorithm we identified several image processing frameworks that could be used, such as: opencv, imagej, FastCV etc. For this algorithm, we used some of the functionalities provided by the opencv framework through its java wrapper (javacv). In chapter two the basics of image processing are covered. Also a brief comparison of available image processing frameworks is presented, and a more detailed description of opencv framework is given. Chapter three contains detailed description of recognition algorithm itself, along with its subsets and issues. In chapter four results and implemented algorithm is discussed and compared with other possible solutions. At last, final chapter (five) concludes the subject, states the advantages and disadvantages of the presented work, and suggests possible topics for further research. 2. DIGITAL IMAGE PROCESSING AND DATA EXTRACTION ON MOBILE PLATFORMS 2.1. Basic concepts Digital image processing is a term describing usage of computer algorithms on digital images in order to achieve certain goal, such as color correction, perspective modification, distortion etc., by modifying binary source data. Digital image is an image f(x,y) that has been discretized both in spatial coordinates and in brightness. It s represented by a 2D integer array, or a series of 2D arrays, one for each color band. The digitized brightness value is called grey level [1]. Each pixel of image represents part of an object, which has been illuminated during the image capturing process. Illuminating light has been partly absorbed and partly reflected, causing different color representation. Reflected light reaches camera sensors and is being recognized. Each digitalized pixel is holding information about each color channel (e.g. in RGB color system, each pixel holds information about red, green and blue channel, which combined result in final color of that pixel). Additionally, each pixel can carry information about fourth, alpha channel, which is used for defining image transparency settings. Image processing is still mostly used in industry, entertainment and marketing (e.g. [2], [3], [4]), image and video editing software (e.g. [5]), and, still in very small percentage, education [6]. It s important to emphasize usage of digital image processing in biometrics, especially in face recognition, age and gender estimation and other fields. Open source biometric Recognition project OpenBR is developing communal biometrics framework supporting the development of open algorithms and reproducible evaluations, on top of the OpenCV library [7]. Image processing is key element of modern video surveillance (e.g. [8]), real time traffic monitoring (e.g. [9]) and drowning detection systems (e.g. [10]), but also some of software that help people with disabilities (e.g. Camera Mouse by Boston College [11]). Data extracting from images also is field of utilization digital image processing, as it is process of recognizing patterns in images and creating digital data out of them such as vectors and/or arrays. For example, if bingo card is scanned and processed to discover marked numbers, data about this card is being extracted and digitalized using digital image processing. Similarly, in this paper, process of data extraction by using digital image processing of exam scorecards will be presented. Process will be broken in three parts: Cropping image to fit area of scorecard answers, perspective correction as preparation for evaluation algorithm and evaluation algorithm itself Image processing frameworks Image processing is available feature of many programming languages (Python, Java, MATLAB ), but powerful and fast processing is mostly done in. While one can choose to write program from scratch, many libraries and frameworks are available with precisely developed functions for different processing tasks. There are several popular libraries that offer functionalities for computer vision and image processing. A brief comparison of such libraries is given in Table 1.

3 Sinha [12] conducted performance comparison between 3 image processing frameworks: OpenCV, LTI and VXL. The results presented in Figure 1 show that OpenCV is faster than LTI and VXL, especially if used with Intel Performance Primitives. Comparison has been taken on Pentium M 1.7GHz and four tests were done: 2D DFT: Forward fourier transform of a image Library OpenCV [13] CImg [14] Ccv [15] FastCV [16] LTI-Lib [17] VXL [18] Aforge [19] Resize: to a image, bilinear interpolation. An 8-bit, 3 channeled image was used Optical flow: 520 points were tracked with a window and 4 pyramid levels Neural net: the mushroom benchmark of FANN Written in N/A N/A Language support Java (JavaCV),.NET (EmguCV), Python, MATLAB,, C Flash OS support Windows, Android, Maemo, FreeBSD, OpenBSD, ios, BlackBerry 10, Linux, OS X Linux, Solaris, OS X Linux, OS X, Windows Windows, OS X, Linux, Android, Windows Phone 8 Linux, Windows C# C# Windows, Linux, Solaris Windows, Windows Phone License BSD CeCILL license LGPL FastCV License Agreement GNU Lesser General Public VXL License LGPL v3 Table 1 Listing of available image processing libraries. Supported OS for mobile devices are bolded Figure 1 OpenCV vs VXL vs LTI: Performance comparison (less is better) [12] 2.3. OpenCV library One of the most popular and widespread libraries certainly is OpenCV (Open Source Computer Vision Library), an open source computer vision and machine learning software library, built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial projects [13]. Library itself has more than 2500 optimized algorithms, including comprehensive set of both classic and state-of-the-art computer vision and machine learning algorithms. These algorithms can be used to detect and recognize faces, identify objects, classify human actions in videos, track camera movements, track moving objects, extract 3D models of objects, produce 3D point clouds from stereo cameras, stitch images together to produce a high resolution image of an entire scene, find similar images from an image database, remove red eyes from images taken using flash, follow eye movements, recognize scenery and establish markers to overlay it with augmented reality, etc. OpenCV is written natively in, but it has several wrappers which expose OpenCV functions in different languages and different platforms. Some of them are EmguCV for.net C#, JavaCV for Java, OpenCV for Android SDK etc. OpenCV is very large open source project, with more than commits on GitHub, committed by more than 200 contributors. Except individual contributors, Corporate contributors such as Intel Inc. and Google Inc. are also involved in OpenCV development [20]. There are several reasons why we decided to use OpenCV: - Support for mobile platforms only OpenCV, FastCV and Aforge explicitly state

4 support for mobile operating systems, while only OpenCV and FastCV support Android. - Licensing OpenCV has more liberal licensing policy (BSD) than FastCV (FastCV License Agreement). - Community: OpenCV has significantly larger communit with more than community users and over downloads. - Contributors: OpenCV is actively maintained by more than 200 contributors, both individuals and companies. After we ve selected OpenCV, it was then a matter of choice wether we will use OpenCV with (Android NDK) or with Java (JavaCV wrapper). We have chosen JavaCV mainly because it was much easier to configure the whole project to work with JavaCV, and the syntax was more familiar to us, since Java is used for Android development. 3. ALGORITHM 3.1. Problem description In this section we will describe the algorithm developed as a part of the PEAS (Personal Exam Assistant) system. The implementation targets Android mobile platform, so we decided to utilize popular OpenCV image processing framework through Java wrapper (JavaCV). The main goal of this algorithm is to detect the handmarked answers on scorecard paper sheet. Scorecard sheet itself contains arbitrary number of initially unfilled square fields which represent possible answers to particular questions. User is required to fill (mark) by hand those square fields which according to him represent correct answers. Such completed scorecard is an input to our algorithm. In order to detect marked answers on scorecard, user takes a picture of it using its phone camera. Now, the first issue that appears here is the possibility the picture is not taken from proper undesirable perspective, so perspective must be corrected (Figure 2) and the image prepared. After that, the square fields detection process is started. The last step is to evaluate the color of detected squares, and to see if it can be considered as marked square. The following table lists and describes the most important OpenCV functions and functions developed by us: Name Description Source cvfindcontours getperspectivetrans form cvwarpperspective getlinesfromimage getcornerpoints getaveragecolor analyzearea int cvfindcontours(cvarr* image, CvMemStorage* storage, CvSeq** first_contour, int header_size, int mode, int method, CvPoint offset) The function retrieves contours from the binary image Mat getperspectivetransform(in putarray src, InputArray dst) Calculates a perspective transform from four pairs of the corresponding points Void cvwarpperspective(const CvArr* src, CvArr* dst, const CvMat* map_matrix, int flags, CvScalar fillval) Applies a perspective transformation to an image List<Line> getlinesfromimage(iplimag e src, int cannytreshold); Retrieves the list of lines detected on the image List<Point> getcornerpoints(iplimage src,list<line> lines, double mintreshold) Retrieves the list of four corner points from top left to bottom left one int getaveragecolor(iplimage img, int x, int y, int width, int height, int padding) Retrieves the RGB value of average color in forwarded region boolean analyzearea(cvrect rect, IplImage img, int thresholdtrue) Open CV Open CV Open CV Retrieves the true if the answer is marked or false if answer is not marked. Table 2 Main functions used to implement algorithm Our Ours Ours Ours

5 3.2. Perspective correction algorithm Since the answer detection algorithm needs answer square fields to be mapped in a grid, it may be necessary first to correct the image's perspective in order to put the grid back in normal position, as shown in Figure 2. Figure 3 Perspective correction process steps Figure 2 Perspective correction The whole perspective-correction process can be divided in few steps: 1. Contour detection 2. Frame recognition 3. Corner calculation 4. Corner position detection 5. Perspective transformation After the image was grayscaled, first we have used opencv's function 'cvfindcontours' to get all of the contours in the image. The function usually finds large number of contours, depending on the image size and complexity. But the biggest contour in the image is the frame which contains answers grid. Then it is simply a matter of getting the contour with biggest calculated area. After the frame has been found, we crop the image for easier handling (fewer pixels to worry about). And then we try to detect the lines on the frame. We also have to specify minimum length in pixels for object to be considered as a line. We use minimum of 200 pixels. After the lines get detected, in order to get the frame corners, it is necessary to calculate line intersections and sort them by their coordinates to know which corner belongs to which side. Furthermore, the only remaining thing is to insert the original image and its frame corners as parameters for the opencv functions 'getperspectivetransform' and 'cvwarpperspective', and we get the transformed image as a result Algorithm for detecting question squares The input for this part of the algorithm is the image of the cropped frame, reduced to a fixed width and height. Coordinate (0,0) representing the upper-left corner of the frame with the answers is known. Since the image is always the same size, we know exact distance from left and upper edge of the frame to first square field, i.e. we know the exact lengths of a and b. Also the width and height of the square are fixed, so the entire area of the first square can be calculated. Figure 4 Two more values are required: the distance from the upper-left corner of one square to upper-left corner of the first square to the right c, and the distance from the upper-corner of one square to upper-corner of the first square below d. Using aforementioned

6 values, the exact positions and areas of all square fields are calculated. Once all square fields are known we simply iterate through questions and answers and check whether the square representing an answer is marked or not. A simple pseudo-code representation of the algorithm is stated below: FOR each question averagecolor=getaveragecolor FOR each answer ismarked=analyzearea(area,averagecolor) if ismarked is true answer is marked else answer is not marked 3.4. Algorithm for recognizing answers Input to the algorithm is a rectangle object containing all square fields (answers) in one row (question). As mentioned before, square position is calculated relatively to the starting edge of base rectangle. At the beginning, average color of all squares is calculated, so it can easily be determined if square is marked (i.e. color is darker than average color of base rectangle) or not. Figure 5 visualizes conceptual steps of detecting marked answers. True problem is not to detect white and black squares, because there are no absolutely black or white ones. Trick is to determine darker and lighter squares, in comparison to average color of common area of all squares. We could use average color of entire scorecard, but shadows significantly affect detection, so localization of detection improves results. In step 1, safety padding is added, to ensure that average color will take enough white color in detection (if all squares are marked, average color will be dark, and detection won t be so efficient). In second step, average color is calculated, and compared with each square in step three. Figure 5 Steps for detecting marked answers Function getaveragecolor Input: Area to analyze, padding Include padding into area avgvalue = 0 FOR each pixel in area curvalue = RGB value of current pixel avgvalue = avgvalue + curvalue counter = counter + 1 avgvalue = avgvalue / count return avgvalue Function analyzearea Input: Area to analyze, threshold colorvalue=calculate average color if colorvalue < threshold return true else return false 3.5. Testing example For demonstration of our implementation, we will showcase few screens of our application PEAS which uses that implementation and whose source code can be found at BitBucket 1. Precondition for usage is obtaining the exam metadata by scanning QR code at the beginning of the exam. Information used in square detection algorithm is available in that metadata. Once exam is finished, user captures scorecard with PEAS application and photo (i.e. digital image) processing begins. 1

7 Figure 6 Capturing the scorecard Figure 7 Processing captured image them offer wrappers which ease the use of frameworks in mobile applications developed in other languages such as Java or C#. The wars of incompatible mobile platforms are taking their toll here as well. To our best knowledge, OpenCV came the farthest to support multiple platforms. But it is at least very questionable if the same image processing framework could be used to develop native mobile application for all major mobile platforms. Also, when choosing appropriate framework for our initial algorithm implementation on Android mobile platform, we witnessed a lack of proper API documentation in some frameworks, which really makes the use and configuration of framework difficult. After processing is done, recognized answers are displayed as overlay layer with green squares over the detected answers. User then checks marked answers and confirms that green marked square fields really are chosen answers. After validation, answer data is sent in HTTP request to web service which calculates the final score. Response contains data about exam result which is then shown in styled form of radial graph. Figure 8 Validation of marked answers 4. DISCUSION Figure 9 Result presentation The image processing and computer vision combined with mobile devices is still a new field. That said, there are certain technical and other issues that should be taken into consideration. For example, we have witnessed that only a part of image processing frameworks are suitable to be used on mobile platforms. While most of them are developed in programming language, few of In a past few years the processing capabilities of mobile devices have grown exponentially, however, image processing is still a demanding task. So in order to assure satisfactory performance, and in the same time not to overload mobile devices battery, an optimized algorithm for image processing is a top priority. The scorecard image that needs to be processed by our algorithm is taken by mobile phone camera. The quality of taken image can vary depending on quite a few factors. A lot of these factors that can significantly influence the image quality are not in our control. For example, on mobile market there is a large number of different mobile devices, with different properties and camera quality; the person that takes a picture can take blurred picture, or the image perspective can be shifted; improper lightning in the room, reflections or shades can also make the content of scorecard hard to detect; the marked answers could have to small contrast etc. Neutralizing the unwanted impact of aforementioned factors is not an easy task. We managed to mitigate the influence of taking picture in a wrong perspective by implementing algorithm for perspective correction, but dealing with reflection, shades, lightning, and contrast issues is something that we need and plan to improve. Although it certainly needs to be improved, our algorithm is implemented in fully functional mobile application, and even now gives satisfactory results in recognizing answers on scorecard sheet. Also, in our opinion, both the algorithm as a whole and the parts of the algorithm can be applied in different contexts. The perspective correction algorithm, the algorithm for detecting square fields, and the algorithm for detecting marked answers can be

8 easily adapted and used in many similar mobile applications. 5. CONCLUSION This paper summarizes our efforts in developing algorithm for detecting answered questions on scorecard paper sheet, using image processing on mobile devices. Image processing and computer vision is an exciting field in computer industry, which becomes even more exciting when combined with mobile devices. As we already stated, applications of these two technologies combined are numerous and possible in different areas of industry, education, government tourism etc. We believe this field is yet going to show its full potential. In algorithm implementation we used popular OpenCV framework through its JavaCV wrapper, since we found it to be fast, widespread, well documented and cross-platform framework. However, we also identified several issues and challenges when developing image processing in mobile applications. Also, we plan to improve our algorithm in several aspects: neutralizing shading, lighting, bad contrast, and reflection negative impacts; optimizing performance; delivering more responsive, real-time validation of marked answers. The developed algorithm is implemented in fully functional Android mobile application which we demonstrated in Testing example section. Also, we identified few interesting topics for future research on image processing mobile applications: feature and performance comparison of available frameworks; difference in performance between wrappers and C/ implementation; native mobile vs web applications; cross-platform development; optimization of image processing algorithms. REFERENCES [1] M. Petrou, Image processing: the fundamentals, 2nd ed. Chichester, U.K: Wiley, [2] Place IKEA furniture in your home with augmented reality - YouTube. [Online]. Available: [Accessed: 17-May-2014]. [3] Google Glass. [Online]. Available: [Accessed: 17-May-2014]. [4] Aurasma Reviews edshelf. [Online]. Available: [Accessed: 17-May- 2014]. [5] Creative Cloud now includes Creative Suite Master Collection and Design Premium features. [Online]. Available: [Accessed: 17-May-2014]. [6] How To Use Augmented Reality In Education - Edudemic. [Online]. Available: [Accessed: 17-May-2014]. [7] OpenBR Home. [Online]. Available: [Accessed: 17-May-2014]. [8] Eric Chu, Jim He, and Andrew Spencer, OpenCV and TYZX: Video Surveillance for Tracking. [Online]. Available: [Accessed: 17-May-2014]. [9] Kim-Sung Jie and Ming Liu, Computer Vision based Real-Time Traffic Monitoring System. [Online]. Available: [Accessed: 17-May-2014]. [10] Drowning detection, pool surveillance with Poseidon. [Online]. Available: [Accessed: 17-May-2014]. [11] Camera Mouse. [Online]. Available: [Accessed: 17-May- 2014]. [12] OpenCV vs VXL vs LTI: Performance Test AI Shack. [Online]. Available: [Accessed: 15-May-2014]. [13] OpenCV, OpenCV, [Online]. Available: [Accessed: 15-May-2014]. [14] The CImg Library - Template Image Processing Toolkit. [Online]. Available: cimg faq.html#ssf12. [Accessed: 16-May-2014]. [15] Community Core Vision. [Online]. Available: [Accessed: 16-May-2014]. [16] Computer Vision (FastCV) - Mobile Technologies - Qualcomm Developer Network. [Online]. Available: [Accessed: 16-May-2014].

9 [17] LTI-Lib. [Online]. Available: [Accessed: 16-May- 2014]. [18] VXL - Libraries for Computer Vision. [Online]. Available: [Accessed: 16- May-2014]. [19] aforge - AForge.NET Framework - Google Project Hosting. [Online]. Available: [Accessed: 17-May-2014]. [20] OpenCV - Contributors - OpenCV DevZone. [Online]. Available: [Accessed: 17-May-2014]. [21] Itseez/opencv GitHub. [Online]. Available: [Accessed: 17-May- 2014]. INFORMATION ON AUTHORS: Alen Huskanović ahuskano@foi.hr Faculty of Organization and Informatics Alen Huskanović is a regular student of the third year of undergraduate study Information Systems at the Faculty of Organization and Informatics in Varazdin. He has participated in various national and international competitions, among which stands out IEEEmadC international competition for the design of mobile applications where he won first prize. Actively working on the system PEAS with whom he came to Croatian finals of Microsoft Imagine Cup. He is primarily engaged in the development of applications for the Android operating system. David Ante Macan amacan@foi.hr Faculty of Organization and Informatics David Ante Macan is a third year undergraduate student of Information Systems at Faculty of Organization and Informatics. He has participated in various competitions, including the Croatian Microsoft Imagine Cup finals, won the best design award at IEEEmadC contest, won the third place at Infinum student Hackathon 2014, etc. His main area of interest is mobile development, especially Android. He is actively working on several projects, including the Personal Exam Assistant (PEAS) project at Faculty of Organization and Informatics. He is also a member of IEEE Institute and is one of the student coordinators and founders of the Laboratory for mobile technologies (MT Lab) at Faculty of Organization and Informatics. Zoran Antolović zantolov@foi.hr Faculty of Organization and Informatics Zoran Antolović is a third year undergraduate student of Information Systems at Faculty of Organization and Informatics, University of Zagreb. Primarily interested in web development and design, he participated in various projects, such as Microsoft Imagine Cup (finals 2014), Best Code Challenge 04 (top ), first place at Infinum Student Hackathon 2014, first place in Best design category at IEEEmadC international competition with PEAS mobile application. With other students, he has participated in Nestala djeca (missing childrens) project, developing entire web application and web services for mobile application.

10 Boris Tomaš Faculty of Organization and Informatics Pavlinska Varaždin tel: fax: Boris Tomaš, M.A. is a Junior researcher and Teaching Assistant at the Faculty of Organization and Informatics, University of Zagreb where he works at the Department of Information System Development. Boris' assignments at the Faculty are laboratory sessions delivered on the courses Geography Information Systems and Software Engineering. Fields of interest are: software development, mobile technologies, geoinformatics, bioinformatics, artificial intelligence, application marketing, Internet marketing. During his work career he worked on several projects in various positions. Boris is currently a doctoral student at the Faculty of Organization and Informatics, University of Zagreb. For his academic achievements he received Faculty Dean's award and University Rector's award. Marko Mijač mmijac@foi.hr Faculty of Organization and Informatics Pavlinska Varaždin tel: fax: Marko Mijač, as of July 2011 works at the Faculty of Organization and Informatics in Varaždin as a project KI Expert 2012, project MEDINFO associate, and as a teaching assistant on courses related to software engineering and information systems. Prior to his current employment he worked as a developer of intranet production planning system at Boxmark Leather d.o.o. He is interested in web, desktop and mobile applications development in various platforms and technologies.