AUTOMATIC EYE DETECTION IN FACIAL IMAGES WITH UNCONSTRAINED BACKGROUNDS

Transcription

1 AUTOMATIC EYE DETECTION IN FACIAL IMAGES WITH UNCONSTRAINED BACKGROUNDS Dr John Cowell Dept. of Computer Science, De Montfort University, The Gateway, Leicester, LE1 9BH England, ABSTRACT Humans convey information on their emotional state by their facial expression. A system capable of determining the emotions of a person based on expression would have many useful applications including security, assessment of satisfaction of commercial products, and interactive systems which were able to respond in a more satisfactory manner taking into account the emotional state of the user. In this paper we describe a system which explores the use of two statistical metrics which use different characteristics of full face images to allow the position of the eyes to be calculated, in under one second, as the first stage in an automated system for measuring facial expressions. The test video images use head and shoulder images filmed on a plain background superimposed on a range of virtual backgrounds. KEY WORDS : facial expression, facial action encoding system, FACS, emotional classification. 1. INTRODUCTION Humans communicate through their facial expressions, speech, movement and hand gestures, but of these only one is always active: the expression on the face. Even a neutral display apparently showing no emotion conveys information to the viewer. Extensive research on the relative importance of these modes of communication indicate the importance of facial expression. Mehrabian [1] experiments indicated that the contribution of the spoken part of a message was only 7%, while the intonation accounted for 38% and the facial expression 55%. The ability to detect and classify the emotional state of humans from their facial expressions is an attractive one which has many application of a social, economic and political nature. While automatic classification systems are unlikely to perform better than a trained human observer, automatic systems are cheaper, do not grow tired, or lose interest and therefore can be used in a more extensive way. Identifiable applications include: Assessment of commercial products, particularly where the user interacts with the product over a long time period, such as a robot system, a vehicle or a movie. Interactive systems capable of identifying the users' degree of satisfaction could adapt to provide greater satisfaction to users. Security applications such as at airports where individuals exhibiting particular facial expressions identified with stress could be identified. Monitoring systems for people with severe disabilities with communication difficulties. 2. CATEGORIZATION OF EMOTION The Facial Expression Program of the 1980's has been the most influential work on the relationship between facial expressions and emotional states. This consists of a set of assumptions, theories and methods. Russell[4] captures the essence of the Program with a list of 14 items which contain essential information required for the development of an automated system. The first item in the list states that there are a small number of emotions (seven plus or

2 minus two). These emotions are happiness, surprise, fear, anger, contempt, disgust and sadness. This group of emotions has been used by most automatic emotion detection systems. The Facial Expression Program has broad but by no means universal acceptance, however it is clear that it is the most widely presented set of corollaries and is presented as the dominant paradigm in text books [5][6] and advisory documents for practicing psychologists[7]. While there is extensive research to indicate that for western literate people there are a common set of facial expressions which can be correctly interpreted with varying degrees of success (although better than chance), some expressions are interpreted more correctly than others. Notably happiness is the easiest to identify and disgust the hardest. There are strong suggestions from researchers such as Russell[8], Fernández-Dols[9] and Fridlund[10] that the earlier work of researchers such as Ekman[11][13] which pointed to a universality of expression is flawed. 2.1 The Facial Action Coding System The most widely accepted technique for an objective measurement of facial expressions is a system produced by Ekman[12] the Facial Action Coding System FACS. FACS breaks each facial movement into 46 action units (AU). Each AU is assigned a five point intensity value. FACS is the dominant encoding system in use, however a major problem it takes about 100 hours to become proficient in using FACS and to process one minute of video takes at least an hour. If this process could be automated it would be of significant benefit. 2.2 Automated Identification of Facial Elements There is currently has no satisfactory automated system available, however Pantic [2] gives a review of the state of the art and identifies three key areas: Detection of the face in an image; extracting data on the expression of the face ; and classification of the extracted data. The first two areas are within the field of image analysis and pattern recognition and satisfactory systems can be developed if a constrained environment is used, however the classification and interpretation of the extracted data is the most complex area, with an extensive body of literature beginning with Darwin[3] in Kapoor[14,15] reports on a fully automatic system for tracking the facial features of the upper face. The system operates in real time, however it requires an infra red camera to detect the eyes, since the eyes are the brightest part of the image in infra red. The system only looks at the upper part of the face and does not work in the presence of strong infra red sources such as bright sunlight or with a subject wearing glasses. The test images had a plain background. Tian[17] developed a neural network based solution for recognising sixteen action units, however the user had first to identify the location of the facial components in the first frame, which were then tracked using an adaptive template system. Donato[18] used a Gabor Wavelet approach to recognise eight action units, however the system has to be manually initialised to identify the facial components and head movement must be restricted. The use of Gabor wavelets has been extended by Tian[19], however although he uses an automated system for approximate identification of facial features used by Rowley[20] manual adjustment in the first frame is still needed. Kruger et al [21,22] use Gabor wavelets for face tracking in real time, however a grey scale template of the face to be tracked is required. Hager and Belhumeur[23] describes an effective tracking system which requires a defined template of the face to be tracked. Kruger et al[24] and Raja et al[25] use a colour based approach for tracking faces. In conclusion, the real-time detection and tracking of faces in an unconstrained video images is still a problem which has not been completely solved.

3 3. THE AIMS OF THE PROJECT Although the overall aim of this project is to detect the face, identify the expression and then relate the expression to an emotional state, this paper reports on the first phase of the work : the detection of the eyes in unconstrained images. All the data was collected using an inexpensive web cam and stored in standard AVI files with a resolution of pixels with 24 bit RGB colour at 15 frames per second. The subject were initially filmed in from of a plain lime-green background, and then superimposed on a variety of complex backgrounds. This had a number of advantages over collecting data in real-world situations. It is time consuming and expensive to find and visit a suitable range of locations; subjects may be unwilling or unable to visit all of the sites. Examples of the same frame but with different backgrounds are shown in figure 1: Figure 1 Four backgrounds the same subject (Martin). Ten backgrounds and seven subjects were used for the experimental part of the work. 3.1 Two Metrics for Eye Identification High level structural analysis of frames for shapes which correspond to the face, or parts of the face such as the eyes are computationally expensive. Due to the inherent variation in the shape of eyes this is an unreliable approach to adopt if the background is complex. For these reasons, a statistical approach is used which looks at two different metrics each of which looks at different characteristics of the region which is being detected. The region which we are searching for is the left eye. When this is found, the other eye, the nose and mouth can be readily found by using an adaptive template system such as used by Tian [17]. This aspect of the work is not reported here. 3.2 Metric 1 - Activity The region around the eye has a different degree of activity compared to other parts of the face, that is, the variation in intensity between small groups of adjacent pixels is characteristic of that region. A simple and very fast way of measuring activity is as follows: Calculate the difference in RGB intensity between a pixel and the pixel on its right, and the pixel below it. Sum these differences for a small rectangular region of width w, and height h, which approximates to the expected eye region. To calculate the activity centred on that pixel. Repeat for each pixel in the image, to find the activity centred around every pixel in the image. Experimentation has found that the size of the region scanned for each is not critical to the performance of the algorithm, but should conform roughly to the size of the eyes. In figure 2, image (a) shows the original image. Image (b) shows the degree of activity. Image (c) shows the original image with regions which have activity values outside specified limits, A 0 and A 1,as black.

4 (a) (b) (c) Figure2 Displaying activity values (Mario) metric is not sufficient on its own to isolate the eye region. For all the images considered, there existed A 0 and A 1 values which would identify the eye region. However numerous other regions were also shown as these exhibited similar activity characteristics. This indicates that the activity 3.3 Metric 2 - symmetry Another characteristic of the human face when viewed from the front is that it exhibits a high degree of symmetry about a centre line drawn about a vertical line between the eyes and the centre of the nose and mouth. Pixel under consideration Figure3 Measuring A and S. This metric was calculated by comparing adjacent groups of pixels and checking for the difference between them. The two regions used for each pixel were the rectangular area used for calculating the activity value, and a region of the same size on the right of that region. To test the performance of this approach a system has been implemented which allows the user to clock on any point of the image and calculates the activity and symmetry metrics for that point. Part of the working system is shown in figure 3. In this case a point near the centre of the left eye was clicked on. The activity for that point is calculated using the surrounding pixels in the left rectangle. The symmetry of that point is calculated by comparing the pixels in the left rectangle with those in the right region. In addition to calculating symmetry at a specified point, the symmetry value between a minimum value S 0 and a maximum value S 1 may also be displayed as illustrated in figure 4. In same way as for the activity diagram (figure 2), areas outside of the range S 0 to S 1 are shown in black. 3.4 Combining the Metrics Figure 4 displaying similarity Areas of the original image which match both the activity and also the symmetry metric as shown in figure 5a. The size of these areas is calculated by finding a nonblack pixel and region growing from that seed pixel to identify the extent of that area. The process is repeated for every region in order that the largest can be found. This is the region most likely to be the region corresponding (a) (b) to the eye on the left of the image. Figure 5 eye detection Taking the centre of this image and transferring it to the original identifies the eye as shown in figure 5b. The left eye is identified by the intersection of the two lines. (a) (b)

5 4. RESULTS To evaluate the performance of the system, seven subjects were used, five male, two female. Each subject was tested with ten backgrounds. The backgrounds were a variety of outdoor scenes showing roads and buildings, and indoor scenes of complex office and shop environments. Subject A0 A1 Mario John Martin Jane Pam Aladdin Ralph Table 1 : Activity thresholds The range of symmetry values was kept a constant in all cases with S 0 =80% and S 1 =95%. The critical factor in the success of the algorithm was the choice of the A 0 and A 1 values. In each case this varies dependent on the lighting conditions and the appearance of the person. The ranges used are shown in table 1. These values identified the left eye for every subject and every background, however, the A 0 and A 1 values were not the same for all subjects. The last two subjects, shown in figure 6 needed a different range. Aladdin, was filmed in lower light conditions than the other images and has the darkest skin colour of the subjects. Ralph is not fully face on to the camera and occupies a smaller proportion of the picture than the other subjects. Under more controlled conditions of light and background constant values of A0 and A1 values could be chosen, however since an important aim of the project is to work in unconstrained environments, this is an unsatisfactory (a) Aladdin (b) Ralph Figure 6 Subjects with low activity values. compromise. At this stage of the project the A0 and A1 values have to be chosen manually, however sufficient work has been done to strongly suggest that for typical images S0, S1 and A0,A1 values exist that the eye can be identified. 5. CONCLUSIONS The research has shown that for the images considered in a wide variety of backgrounds there is a range of activity and symmetry values which correspond to the region of the eye. The identification of regions which match both of these criteria identifies the eye region. The algorithm executes fast, taking about a second in the present form to identify the eye region. Having found this region, alternatives techniques can be used to find the other components which are involved in making facial expressions, such as the eyebrows, nose and mouth. The S 0 and S 1 symmetry values were found to be remarkably consistent for the images considered, a value of between 80% and 95% successfully identifying the eye region while not identifying many other areas of the image in the set of backgrounds considered. The weakness in the algorithm is that the absolute activity values vary depending on the subject and also the lighting conditions. One key area for future research is to investigate techniques which can automatically identify the A 0 and A 1 values. 6. REFERENCES [1] A Mehrabian. Communication without words, Psychology Today, vol 2 no4. pp [2] Maja Pantic, Leon J Rothkrantz. Automatic Analysis of facial expressions: The State of the Art. IEEE transactions on Pattern Analysis and Machine Intelligence, vol. 22 no. 12, pp December 2000 [3] C Darwin. The expressions of the emotions in man and animals. Chicago: University of Chicago Press (Original work published 1872)

6 [4] James A Russell, J Fernández-Dols. The psychology of facial expressions. Cambridge University Press ISBN [5] J G Carlson, E Hatfield. Psychology of emotion. New York: Holt, Rinehart & Winston. [6] J Ingram. The burning house : Unlocking the mysteries of the brain. London: Penguin. [7] Behavioural Science Task Force of the national Advisory Mental Health Council Basic behavioural science research for mental health: A national investment: motivation and emotion. American psychologist 50 pp [8] James A Russell Is there a universal recognition of emotion from facial expression? Psychological Bulletin 115 pp , [9] J M Fernández-Dols M Ruiz-Belda. Are smiles a sign of happiness? Gold Medal winners at the Olympic games. Journal personality and social psychology 69. pp [10] A J Fridlund. Human facial expressions: An evolutionary view. NY Academic Press 1995 [11] P Ekman. Universals and cultural differences in facial expressions of emotions. Nebraska Symposium on Motivation vol. 19 pp Univ. of Nebraska Press. [12] P Ekman. Emotions in the human face. Cambridge University Press [13] P Ekman. Friesen W., Ellsworth P. Emotions in the human face. Pergamon Press [14] Ashish Kapoor, Automatic facial Action Analysis. MSc thesis, program in media Arts and Sciences, School of Architecture and Planning MIT, May [15] Ashish Kapoor, Rosalind Picard. Real-time, Fully automatic upper facial feature tracking. Proceedings 5 th International Conference on Automatic Face and Gesture recognition. [16] I Essa, A Pentland. Coding, analysis, interpretation and recognition of facial expressions. Pattern Analysis and Machine Intelligence, 7: , July 1997 [17] Y Tian, T Kanade, J F Cohn., Recognising action units for facial expression analysis. Pattern Analysis and Machine Intelligence, 23(2), February [18] G Donato, M Bartlett, J Hager, P Ekman, T Sejnowski. Classifying facial actions. IEEE Pattern Analysis and Machine Intelligence, 21(10): , October [19] Ying-li Tian, T Kanade, J F Cohn. Evaluation of Gabor-Wavelet based facial Action Unit recognition in Image sequences of increasing complexity. Proc. of 5 th IEEE international conference on Automatic face and Gesture recognition, May 2002, pp [20] H A Rowley. S Baluja, T Kanade. Neural network based face detection. IEEE Transactions on Pattern Analysis and machine Intelligence, 20(1):2-38, January [21] Volker Krueger, Sven Bruns, Gerald Sommer. Efficient Head pose estimation with Gabor Wavelet networks. British Machine Vision Conference, Bristol UK September [22] Volker Krueger, Alexander Happe, Gerald Sommer. Affine real-time face tracking using Gabor wavelets. International conference on pattern recognition, Sept [23] G Hager, P Belhumeur. Efficient region tracking with parametric models of geometry and illumination. IEEE Trans. Pattern Analysis and machine Intelligence, 20(10): , [24] Volker Kruger, R Herpers, K Daniilidis, G Sommer. Teleconferencing using an attentive camera system. International conference on audio and video-based biometric person authentication, pp [25] Y Raja. J McKenna, S Gong. Tracking and segmenting people in varying lighting conditions using colour. International conference on automatic face and gesture-recognition. pp , Nara, Japan, April 1998.