Proceedings of Student-Faculty Research Day, CSIS, Pace University, May 3 rd, 2013

Transcription

1 Proceedings of Student-Faculty Research Day, CSIS, Pace University, May 3 rd, 2013 Comparative Analysis of Feature Extraction Capabilities between Machine and Human in Visual Pattern Recognition Tasks Utilizing a Pattern Classification Framework Amir Schur, Sung-Hyuk Cha, and Charles C. Tappert Pace University Seidenberg School of CSIS, White Plains, NY amirschur@aol.com; {scha, ctappert}@pace.edu Abstract There have been many recent advances in pattern recognition technologies, particularly those involving visual pattern recognition tasks. How do these machine capabilities compare to human capabilities in visual pattern recognition tasks? Which can perform better in the feature extraction processes, machine or human? This study compares machine and human in color and shape recognition tasks, as part of a visual pattern recognition system. A pattern classification framework will be used to provide a foundation for understanding where this work fits in context with other work. This is a work in progress for a dissertation by the first author. Keywords pattern recognition, feature extraction, human and machine comparisons I. INTRODUCTION What is the pattern classification framework? What are the current scientific advances in understanding human capabilities in this area? And what about machine capabilities? A typical pattern classification system is comprised of the following components: data capture, feature extraction, and classification [1]. Data capture involves obtaining raw data related to the object through sensors, and for visual objects the sensor is usually a camera. Feature extraction reduces the amount of raw information captured in the visual image of the object by measuring certain features. In face recognition, for example, this can be the distance between eye brows, length of the nose, etc. The last component is classification where the feature space is divided into decision regions. This is where the object is assigned a category. This division into three separate processes is also the typical biometric system architecture, in which the three separate technologies are signal acquisition technology, feature extraction technology, and matching technology [2]. Using the three pattern classification system components described above, machine and human capabilities will be explored in visual pattern recognition tasks. A. Data Capture In the visual recognition task performed by human, this activity is accomplished physically from the cornea through the aqueous humor, eye lens to the retina. The mechanism in the eye, initial portion of the human visual system, works much the same as a camera [3]. The shutter of camera can close or open depending upon the amount of light needed to expose the film in the back of the camera. The lens of a camera is able to focus on objects far away and up close with the help of mirrors and other mechanical devices. Both the human eye and a camera use something called a lens in fact, they both use the same type of lens, a converging lens. In the camera, the lens focuses the light onto a piece of film. The film has chemicals in it that basically trap the image on it, making it permanent. Instead of film, the eye uses the retina [4]. In a visual recognition task the process starts with either the human or the camera preprocessing an image. We either take a picture of an object with a camera or we see the object with our eye. There are definitely some differences between the human eye and the camera. For example the highest resolution digital camera currently available processes up to 80 megapixels, whereas our eye can process around 560 megapixels [5]. B. Feature extraction. Advancements in neuroscience have allowed us to capture what data actually gets transmitted from the retina to the brain. Berkeley researchers Frank Werblin and Botond Roska proved that there are between 10 and 12 output channels from the eye to the brain, each carrying a different, stripped-down representation of the visual world. One group of cells sends information only about edges (changes in contrast), another detects large areas of uniform colors, and another is sensitive only to the backgrounds behind the figure of interest [6]. This model is often used in the artificial intelligence field as sparse coding, which is a method of data reduction [7]. More data does not necessarily mean better performance. On the computer side, there have been tremendous advancements in visual feature extraction technologies. There are color based feature extractions, with can be grouped into: RGB, HSB, LAB, YCbCr, etc. There are texture based extraction technologies such as gabor filter and gist. Then there are shape/contour based feature extraction methods, which often are grouped into: distance vs angles, distance projection, min/max, area ratio, object count, etc. [8]. C. Classification/Matching The part of the brain responsible for human ability to deal with patterns of information is the neocortex [7]. The neocortex is also responsible for sensory perception, including visual perception. The primary visual cortex is the part of the neocortex that receives visual input from the retina. How does C4.1

2 the visual cortex process information? There is still much to learn, as we do not know enough about what happens here. In fact, science is yet to provide a full understanding of the brain, so it is not possible to propose accurate overall data models in the visual cortex [9]. Various methods of machine learning and application of advanced statistical methods for classifications can be considered equivalent to the human capabilities of visual object classification and decision making. II. RESEARCH FOCUS After extensive literature search in the three areas of pattern recognition, as elaborated in the introduction section above, it was decided to focus research in the area of feature extraction to perform a comparison between human and machine capability in visual recognition tasks. There is not enough concrete knowledge of the visual cortex or neocortex decision making process to perform any comparative analysis between our brain and any machine or computer system. There could be some comparison done to learn the similarities between our physical eye and visual processing technologies such as cameras. The fact that now we know quite a bit about how data is being transmitted from the retina to the brain, allows for more exploration in comparative analysis in this area. We will try to elaborate further on what is happening physically in the human visual recognition process from the retina to the brain. We will then elaborate on various technologies, focusing on computer systems, which are applicable in this focus area. A. Retina and visual data processing The retina contains two types of cells: rods which handle vision in low light, and cones which handle color vision and detail. When light contacts these two types of cells, a series of complex chemical reactions occur. The chemical that is formed (activated rhodopsin) creates electrical impulses in the optic nerve. The process is as follows [10]: The cell membrane (outer layer) of a rod cell has an electric charge. When light activates rhodopsin, it causes a reduction in cyclic guanosine monophosphate, which causes this electric charge to increase. This produces an electric current along the cell. When more light is detected, more rhodopsin is activated and more electric current is produced. This electric impulse eventually reaches a ganglion cell, and then the optic nerve. The nerves reach the optic chasm, where the nerve fibers from the inside half of each retina cross to the other side of the brain, but the nerve fibers from the outside half of the retina stay on the same side of the brain. These fibers eventually reach the back of the brain (occipital lobe). This is where vision is interpreted and is called the primary visual cortex. Cone pigments, which are color-responsive chemicals in the cones, are very similar to the chemicals in the rods. The retinal portion of the chemical is the same; however the scotopsin is replaced with photopsins. Therefore, the colorresponsive pigments are made of retinal and photopsins. There are three kinds of color-sensitive pigments: red- sensitive pigments, green-sensitive pigment, and blue- sensitive pigment. Each cone cell has one of these pigments so that it is sensitive to that color. The human eye can sense almost any gradation of color when red, green and blue are mixed. There are more chemical reactions that occur in the retina, but as elaborated in the color detection and low-light vision above, the activities are done physiologically utilizing bodyproduced chemical to perform the functions. This complex and automated system can be replicated outside the human body. What data does the brain actually get? Though we may think we capture all information, it turns out that we are just receiving hints, edges in space and time. Researcher Frank Werblin and Botond Roska showed that there are between 10 and 12 output channels from the eye to the brain, each carrying a different, stripped-down representation of the visual world. One group of cells sends information only about edges (changes in contrast); another group detects large areas of uniform colors, and the last group in sensitive only to the backgrounds behind figure of interest. Thus we can conclude there are three group of data transmitted between the retina and the brain: contrast, color and shape /contour. B. Visual Feature Extraction Technologies There are common implementations of visual feature extraction based on color, contour and texture. We will try to briefly describe what they are. These implementations are available even on basic computer systems as code libraries than can be utilized to build any software. Attention must be given to the concept and algorithm implementation. Any changes in scientific finding will definitely impact these software libraries. Also development in technology might make them obsolete or less used. 1) Color based methods There are many ways of extracting color information from an image, such as RGB (red green and blue), HSV (hue, saturation and value), LAB (L is for lightness and a and b for the color-opponent dimensions), YCbCr (Luma, blue difference, red difference). Just like the cone pigments in the retina, the RGB method color space comes from an additive model in which the three primitive colors red/green/blue are added together to reproduce the entire range of colors. This method is utilized in photography, television and computers. At theprograming level, for example, there is a color method in both AWT (Abstract Window Toolkit) and swing package (the two main packages in Java for Graphical User Interface). Color (int r, int g, int b). This method will create a color with the specified red, green, and blue integers, with values in the range between 0 and 255. The HSV color model was created by Alvy Ray Smith (one of the co-founders of Pixar Animation Studios), as a more user friendly alternative for designers. The hue parameter is circular, instead of ranging from 0 to 255 as RGB. C4.2

3 example in flower recognition counting the number of petals can help differentiate the contour of the flower. Fig. 1. HSV color model There are several variations of this model, such as HSL and HIS. HSL stands for hue, saturation, and lightness where as HIS stands for hue, saturation, and intensity. The LAB color space includes all perceivable colors which exceeds those of the RGB. An extension of this model is the CIELAB, which is the most complete color space as specified by the International Commission on Illumination. YCbCr is not an absolute color space; rather, it is a way of encoding RGB information. 2) Texture based methods Among various approaches to texture feature extraction, gabor filter has emerged as one of the most popular one. Gabor filter-based feature extractors can be interpreted as nonlinear functions that map images from original space to feature space, where each image is represented by its features [11]. Gist model provides high-level context information (a segment within a site) of a visual object using coarse features. Researchers find that scenes from differing segments contrast in a global manner, and this can be captured and utilized as a basis for recognition. The opposite spectrum is the salience model, where low level texture analysis is performed on a visual object [12]. These two models are often combined for object recognition. 3) Shape/contour based methods. The first task to complete in any contour based calculation is to separate the image from its background. Once the image is retrieved, there are various methods to represent the contour in a particular form. Various calculations are then performed to distinguish the object from other similar objects. There are various types of calculations that can be done to represent an object s contour. These calculations are typically not performed alone, but grouped together to try to capture various aspects of an object s contour. One well known method involves calculating distances versus angles. In this method, the distances from the contour point and the center of gravity are computed by going through the contour for all angles between 0 and 2PI for a given step size. Determining a starting point is necessary. Another method involves calculating the minimum and maximum distances from a contour point. The ratio of the minimum and maximum distance can also be calculated. Counting any distinct features of an object is another technique often employed. For III. PLANNED RESEARCH ACTIVITIES As elaborated above, the focus of this research is to perform a comparative analysis between human and machine capabilities in visual recognition tasks. Data from retina to the brain can be grouped into edges (changes in contrast), colors, and the backgrounds behind the figure of interest. Various technological advances in visual feature extraction are not too far away from those categories, and include color, texture and shape/contour. With these conditions, we can attempt to compare color extraction capabilities and shape recognition capabilities between human and machine. We will not perform contrast comparisons, since this seems to be impractical due to the lack of available use cases in this area. As the framework for analysis is the pattern classification framework, comparison will be done in terms of object identification, in particular in a feature extraction process. Accuracy of results is calculated by determining the number of the correct objects selected utilizing color recognition and contour recognition techniques. Another control mechanism is the time factor, the time taken to recognize an object which must be within acceptable parameters. The first hypothesis is that computer systems are better than humans in color recognition. The second hypothesis is that humans are most likely still better than computer systems in shape or contour recognition. A. CAVIAR Model Computer Assisted Visual InterActive Recognition (CAVIAR) is a model where the machine and human are interactive in a visual recognition task. In CAVIAR, human and machine interaction is continuous and can be performed in every step of the visual object recognition process. The human can even override the final result of classification. Utilizing the human-machine interaction model, experiments with CAVIAR proved that a higher accuracy level of visual pattern recognition was achieved interactively compared to that achieved by the machine alone or by the human alone [13]. The CAVIAR model has been ported to a handheld computing device application for flower recognition, and the application is called IVS (Interactive Visual System). IVS exploits the pattern recognition capabilities of humans and the computational power of a computer to identify flowers based on features that are interactively extracted from an image and submitted for comparison to a species database [14]. This flower identification software has six activities that are designed to be done automatically, but still can be overridden by human input. If human input is added on any activities, the AUTO button can complete the other activities automatically. The six activities are: 1. Determining the dominant color of the flower petal. 2. Determining the secondary, less dominant, color of the flower petal. C4.3

4 3. Determining the color of the stamen or center portion of the flower. 4. Counting the number of petals. 5. Getting the horizontal and vertical bounds of the flower, basically isolating the object from the background. 6. Getting the horizontal and vertical bounds of a flower petal to isolate a petal and measure its bounds. As stated above, the first three activities are associated with color recognition, while the latter three activities concern contour/shape recognition. In order to perform a balanced experimental design in these human-machine tasks, three separate experiments are described below. B. Experiment Design To compare machine and human in performing certain tasks, some control mechanisms need to be established. This will be done by performing the object recognition tasks in three ways: machine only, human only, and machine and human combined. The machine and human combination will be done in two separate ways. The first is to capture human input in color recognition tasks, while performing all other tasks automatically. The second will be utilizing human in contour recognition, while the other tasks are done automatically by the computer. 1) Machine only In this experiment, machine visual performance will be measured only in a particular visual recognition task. For the flower identification task this will be done using the AUTO feature within IVS. As computer systems are fast and able to store large amount of data, the machine time for task completion will likely be fast. The experimental design, however, must limit the acceptable completion time by humans. Therefore, any tasks that are completed beyond a reasonable threshold must be considered as a failure. 2) Human only In this experiment, human visual performance will be measured only in a particular visual recognition task. In this regard, a question comes to mind as to whether the machine is compared to an expert in the area of interest, or to an amateur. It is generally known that despite recent advances in the fields of computer vision and machine learning, well-trained expert humans are still generally more proficient than machines in recognizing most patterns [15]. Therefore, for this research we will focus only on human amateurs in the comparison. Prior knowledge or expertise is a variable factor that we do not want to introduce into the experiments. For flower recognition, the task is to get untrained participants to identify the type of flower. The participants will have access to a flower guide book and the flowers to be identified will be retrieved from that guide book. 3) Machine and human combined The intention of the combination is to measure separately the human input for the color recognition and for the shape recognition tasks for comparative purposes. Thus, for each sub-experiment one task will be done by the human participant, while the rest will be automated. The computer operations, of course, will be consistent as the algorithm in the software is set. Within the six available tasks within IVS, the human will perform the first three tasks manually for color identification, while all other tasks are performed automatically. And for shape recognition, the human will perform the last three tasks manually while the others are performed automatically. C. Data Collection In a data collection process 535 flower images were obtained and stored in IVS during a setup stage, and these images were collected not for comparative purposes but rather to observe the effect of human interaction [16]. Thus the manmachine data part cannot be used for this purpose. Another data collection must be performed for human-machine analysis by using the strategy described above. The writer has been fortunate to receive compiled code of IVS from Dr. Jie Zou [13], who advised that the code be reverse compiled, as they were never obfuscated. This will allow the writer to analyze exact methods utilized at the code level and map them to common feature extraction technologies as described above. This code-level access will assist in analyzing the experimental results and arriving at the final findings that is, in concluding whether the machine or human is better at the various tasks. IV. CONCLUSIONS The writer intends to perform comparative analysis between human and machine capabilities in visual pattern recognition tasks, particularly color and shape/contour recognition. The IVS (Interactive Visual System) tool will be utilized for data collection. Research will focus on comparing color recognition capabilities and shape recognition capabilities as part of a visual pattern recognition task. If the final finding shows that the machine is more accurate than human, then we can conclude the specific technology method used by IVS is better than human capability. If the human performs better than the machine, other techniques for color or texture recognition may need to be evaluated. Other tools may need to be utilized or modification of IVS to utilize a particular technique might be warranted. REFERENCES [1] Duda, Richard, O., Hart, P.E., and Stork, D.G., Pattern Classification (2nd ed) John Wiley & Sons, [2] R. Bolle, J. Connell, S. Pankanti, N. Ratha, and A. Senior, Guide to biometrics. New York: Springer, [3] University of Michigan Kellogg Eye Center, How the Eye Works, assessed from anatomy.html. C4.4

5 [4] Richards, Beth, Sept 26, 2010, Differences Between Human Eye and Camera, Assessed from /article/ differences-between-human -eye-and-camera/ [5] Notes on the Resolution and Other Details of the Human Eye, assessed from html, on March 20, [6] Roska, Botond, and Frank Werblin. "Vertical interactions across ten parallel, stacked representations in the mammalian retina." Nature (2001): [7] Kurtzweil, Ray. How to Create a Mind: The Secret of Human Though Revealed, Penguin Group, London, [8] Vuarnoz, Vincent. "Flower Recognition.", assessed from _flowerrecognition.pdf, July 30, [9] Vincent de Ladurantaye, Jean Rouat and Jacques Vanden- Abeele (2012). Models of Information Processing in the Visual Cortex, Visual Cortex - Current Status and Perspectives, edited by Stephane Molotchnikoff, ISBN: , InTech, DOI: /50616.Available from: visual-cortex-current-statusand-perspectives/models-of-information-processing-in-thevisual-cortex. [10] Bianco, Carl, MD, How Vision Works, accessed February 16, [11] Li, Weitao, et al. "Selection of Gabor filters for improved texture feature extraction." Image Processing (ICIP), th IEEE International Conference on. IEEE, [12] C. Siagian, L. Itti, Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 2, pp , Feb [13] Jie Zou, Computer Assisted Visual Interactive Recognition: Caviar. Ph.D. Dissertation. Rensselaer Polytechnic Institute, Troy, NY, USA. Advisor(s) George Nagy, [14] Interactive Visual System, Arthur Evans, John Sikorski, Patricia Thomas, Jie Zou, George Nagy, Sung-Hyuk Cha, Charles Tappert, [15] Coetzer, Johannes, Swanepoel, Jacques (Department of Mathematical Sciences, Stellenbosch University, Stellenbosch, South Africa) and Sabourin, Robert ( Ecole de Technologie Sup erieure, University of Queb ec, Montr eal, Canada), Efficient cost-sensitive human-machine collaboration for offline signature verification, accessed from: cations/ 2012/Coetzer_SPIE_DRR_2012.pdf. [16] Kathryn Durfee, Neville Kapoor, Matthew Muccioli, Richard Smart, David Wilkins, and Amir Schur, An Evaluation of the Effect of Human Interaction on the Accuracy of the Interactive Visual System, Proceedings of Student-Faculty Research Day, CSIS, Pace University, May 4th, C4.5