Segmentation Extracting image-region with face

Transcription

1 Facial Expression Recognition Using Thermal Image Processing and Neural Network Y. Yoshitomi 3,N.Miyawaki 3,S.Tomita 3 and S. Kimura 33 *:Department of Computer Science and Systems Engineering, Faculty of Engineering, Miyazaki University 1-1 Gakuen Kibanadai Nishi Miyazaki Japan yoshi@cs.miyazaki-u.ac.jp **: Mechanics Department, Miyazaki Prefectural Industrial Research Institute Tsunehisa Miyazaki 880 Japan Abstract A method with Thermal Image Processing and Neural Network is presented forrecognition of facial expression. The method isbased on 2-dimensional detection of temperature distribution of face, using Infrared Rays. The front-view face in input image is normalized in terms of the size and the location, followed by measuring the local temperature-dierence between the averaged neutral and the unknown expression faces. For measuring the local temperaturedierence, several blocks on face aredecided, based on the studies in the elds of psychology, especially FACS-AU, and computer vision with visible ray, while the blocks including cheekbone region are decided experimentally for improving the recognition accuracy. The local temperature-dierence caused bytherear- rangement of face muscle and the inner temperature change is used as input data for Neural Network. By using Back Propagation method, neutral, happy, surprise and sad expressions were recognized with 90% accuracy. 1 Introduction The present investigation is concerned with computer vision for detecting human feeling or inner mental situation through facial expression. For a future robot to work or live peacefully with human, it will be necessary to present capability for recognizing facial expression to a robot under the condition of our daily lives. This is because the facial expression has the most important role for nonverbal communication among persons. Although the procedure for recognizing facial expression has been received considerable attention in the course of computer vision research[1, 2, 3, 4, 5, 6, 7], the present stage is far from the goal of humanlike capability, especially from the point of robustness to wide-range lighting condition and capability for understanding human feeling or mind. One reason is that daily shadow, reection and darkness have great inuence on the accuracy of facial expression recognition through the inevitable change of gray level. The other reason for the diculty is that input image from ordinary camera originally has a slight dierence between neutral and smiling faces, for example, from the point of gray level distribution. It is not easy even to extract the region including face by image analysis if the scene has variety like our daily lives. Some researchers have tackled the issue of understanding human feeling through facial expression[1, 2, 3, 4, 5, 6, 7]. However, the trials seem to be tough jobs because the gray level distribution of input image from ordinary camera has a just slight difference among various facial expressions, in addition to low contrast at feature-boundaries on face. For this reason, we try to use an image describing thermal distribution of face, with Infrared Rays (IR), instead of ordinary visible ray. Although human can not detect IR image, it is possible for a robot to process the information around it with thermal image by IR. On the point, a robot as computer system can be superior to human. Actually, human face is easily extracted in the scene when the range of skin temperature is selected for producing the thermal image with IR, using the value of 1 as emissivity[8, 9]. In the previous studies, we reported that neutral and smiling faces were distinguished with excellent accuracy[8, 9]. In the eld of psychology,facs(facial Action Coding System)[10] to analyze facial expression was developed, resulting in that it became to be possible for computer graphic researcher to analyze facial expression as a subject. Some investigations for describing facial expression through feature deformation, mainly based on Action units (AUs:elements of expression in FACS) or other description, have been reported[3, 4, 6]. However, FACS-AU is not quantitative, resulting in having ambiguity for describing facial expression with it. Therefore, the quantitative measure for facial expression or feeling is demanded. The goal of our current study is to present com- 1

2 puter vision enabling detection of human feeling. For reaching the goal, it is necessary to collaborate with psychologists and construct data base for distinguishing one facial expression from others. However, since the investigation presented here is at the rst stage for understanding human feeling with IR computer vision, image variety is limited for simplicity. The facial expressions selected here are neutral, happy, surprise and sad. The number of image-sequences is not big. The generality of the present procedure is discussed with a small speculation. In the course of IR image analysis for recognizing human information, in the present study, the thermal image processing and Neural Network (NN) are used for recognizing more kinds of facial expressions, generally. 2 Image Acquisition and Analysis System In this study, thermal images of human faces were produced by Thermal Video System (Nippon Avionics Co.,Ltd,TVS-3500) with IR. The principle of thermal image generation comes from well-known law by Stefan and Boltzmann, which is expressed by the following equation. W = 1 1 T 4 (1), where W is radiant emittance (W/cm 2 ), is emissivity, is Stefan-Boltzmann constant (= W/cm 2 K 4 ), T is Temperature (K). For human skin, is estimated at 0.98 to 0.99 [11]. In this study, however, the approximate value of 1 was used as for human skin. The values of for almost all substances except human skin are lower than that for human skin [11]. Accordingly, human face is easily extracted in the scene when the range of skin temperature is selected for producing the thermal image, using the value of 1 for. Face-images were observed by a monitor through an IR thermal image system with 16 (4 bits) as thermal level and recorded in 8- mm-video and then digitized with 256 levels (8 bits) per pixel. These digitized images were stored on xed disks having a spatial resolution of pixel elements and processed by a personal computer. The input images were produced under the operating condition which presented lower-gray-level for colder part and higher-gray-level for hotter part. In principle, the temperature measurement by IR dose not depend on skin color, darkness and lighting condition, resulting in that face and its characteristics are easily extracted in the input image containing face and its surrounding. Moreover, we have veried experimentally that the thermal face-image made by our system is not inuenced by the lighting condition[9]. Figure 1 shows an inuence of lighting at night on face-image of one person with visible ray or IR. Figure 1(a) is a face-image with visible ray and lighting, while Figure 1(b) is a face-image with IR and lighting. Figure 1(a) and 1(b) are face-images taken simultaneously for the same person. On the other hand, Figure 1(c) is a face-image with visible ray and without lighting, while Figure 1(d) is a face-image with IR and without lighting. Figure 1(c) and 1(d) are face-images taken simultaneously for the same person. As you can see from Figure 1, normal image with visible ray is strongly inuenced by lighting condition, while the thermal image by our system is not inuenced by the lighting condition. Even at night, the facial expression recognition by IR image processing were performed successfully without lighting, in the same way as that with lighting. Because IR is radiated from face itself, the present method for facial expression recognition has excellent robustness to lighting condition. Needless to say, the thermal image is not inuenced by shadow and/or reection, unless the local temperature of face is changed by them. As far as our investigation was concerned, the change of local temperature of face by shadow and/or reection has not been observed. The shortcoming of visible ray image analysis that the accuracy of facial expression recognition is strongly inuenced by lighting condition including variation of shadow, reection and darkness is considered to be perfectly overcome by exploiting IR. (a) (c) (b) (d) Figure 1: Examples of face-image at night ; (a)visible ray with lighting, (b)ir with lighting, (c)visible ray without lighting, (d)ir without lighting. 3 Recognition Algorithm Front-view faces were used as the input images. Figure 2shows the owchart for recognizing facial expression. The algorithm for recognizing facial expression is as follows. 3.1 Normalization of Size and Location For recognizing facial expression, the observed person should act naturally. Accordingly, the face-image 2

3 Input of face-image Normalization of size and location for face Producing differential image and measuring characteristic parameter Facial expression recognition with Neural Network Figure 2: Flowchart for recognizing facial expression. is recorded under the natural condition without any special restriction for observed person. However, as explained afterward, the facial expression is recognized with the dierential image between the averaged neutral and the test face-images. For this reason, as shown in Figure 3, the input image is normalized in terms of the size and the center of gravity for the face, through performing segmentation with Otsu's method[13], measuring the horizontal Feret's diameter and the center of gravity for the binary image, and performing Ane transformation for the input image in order to normalize the horizontal Feret's diameter and the center of gravity for the binary image. 3.2 Producing the Dierential Image and Measuring Characteristic Parameter Figure 4 shows the owchart for producing the binary image to measure the characteristic parameter for recognizing facial expression. Beforehand, an averaged neutral face-image is produced as a reference image A. Then, with an unknown image B and an averaged neutral face-image A, the dierential images A-B, B-A are made under the condition that the negative value made by the dierential calculation is transformed to 0. Figure 5 shows the example of A, B, A- B, B-A. In this case, the test image B is happy face. Then, for both images A-B and B-A, a processing of (1) smoothing with median lter, and (2) segmentation with level 2and level 3 as threshold, is performed, where level 2and level 3 correspond to the double and triple values for thermal dierence unit in IR apparatus, respectively. Figure 6 shows the example of binary images produced by the processing, applied to the extracted images from A-B and B-A shown in Figure 5. Moreover, for image A-B, a processing of (1) contrast enhancement with gray-scale transformation, (2) smoothing 5 times with median lter, (3) segmentation with Otsu's method, and (4) erasing region with small area is performed. Figure 7 shows the example of binary images produced by the processing, applied to the extracted image from A-B shown in Figure 5. Input image Producing averaged neutral face-image A Producing differential images A-B, B-A (B:test face-image) Segmentation Extracting image-region with face Smoothing Contrast enhancement Segmentation with threshold level2,3 Smoothing Measuring horizontal Feret s diameter and center of gravity Horizontal Feret s diameter Binary image Segmentation Erasing region with small area Binary image Affine transformation for face-image to normalize horizontal Feret s diameter and center of gravity for face-region Figure 4: Flowchart for producing binary image (Left), (Right),to measure characteristic parameter for recognizing facial expression. Figure 3: Flowchart for normalizing size and location for face. Totally, ve binary images are produced with above procedure. Then, for the binary images made by the processing, the measured area fraction in each block demonstrated in Figure 8() is divided by the corresponding average value of learning data. Then, for the 3

4 A:Averaged neutral B:Test A-B B-A binary image made by the processing, the smaller area fraction in two area fractions of blocks demonstrated in Figure 8() is divided by the corresponding average value of learning data. The choice of the smaller area fraction is eective for reducing the in- uence of face direction on the accuracy of facial expression recognition. Eventually, the 17 normalized area fractions are used as the values of characteristic parameters for facial expression. Each block demonstrated in Figure 8() is decided, based on the studies in the elds of psychology, especially FACS-AU[10], and computer vision with visible ray[3, 5, 7], while each block demonstrated in Figure 8() is decided experimentally for improving the recognition accuracy. Figure 5: Example of averaged neutral face-image A, test face-image B, dierential images A-B and B-A. Based on FACS-AU and optical flow Experimental :Block for temperature change From A-B From B-A Figure 8: Blocks for measuring characteristic parameters.():left,():right. Level:2 3 Level:2 3 Figure 6: Example of binary images produced with processing. Figure 7: Example of binary images produced with processing. 3.3 Recognition with Neural Network The 16 values of normalized area fraction from the processing are transformed to 2bits per data. The value of normalized area fraction from the processing is also transformed to 2bits. The condition for making 0, 1 data for the characteristic parameters is decided experimentally for improving the recognition accuracy. The 34 bits data which are transformed from the 17 values of characteristic parameters are used as input data for NN with three layers demonstrated in Figure 9. With Back Propagation (BP) method [12], the facial expression is recognized. The unit number of input layer is 34. The unit number of hidden layer is decided experimentally for improving the recognition accuracy for facial expression. The unit number of output layer is the number of facial expressions which should be recognized. 4 Experiment and Discussion For evaluating the recognition accuracy of the present method, image sequences of neutral, happy, surprise and sad faces of one female were collected. We assembled 10 images per facial expression as learning data and 20 images per facial expression as test data. The total number of images was 40 for learning data, 80 for test data. Then, the averaged face- 4

5 Input 2bit data row( ) 1 for 1 unit, 0 for others Output Figure 9: Neural Network for facial expression recognition. image produced from 10 neutral faces was used as a reference image A. The room temperature in recording all learning-images and all test-images was about 302K. For extracting face-image successfully, the detected temperature range by IR was selected as 303 to 307.5K. In a processing, level 2and level 3 as threshold for segmentation applied to the dierential images are estimated at 0.6 and 0.9K, respectively. The facial expression was made by intentional action and the judgement as right answer for the facial expression was from the observation for the image by herself. The unit number of input, hidden and output layer of NN was selected as 34, 17 and 4, respectively. Figure 10 shows each image example for 4 facial expressions. The 16 values of normalized area fraction from the processing are transformed to 2bits per data under the condition that the value smaller than 0.5 is transformed to 00, the value from 0.5 to smaller than 1.0 is transformed to 01, the value from 1.0 to smaller than 2.0 is transformed to 10, the value bigger than 2.0 is transformed to 11. Then, the value of normalized area fraction from the processing is transformed to 2bits under the condition that the value smaller than 0.5 is transformed to 00, the value from 0.5 to smaller than 0.8 is transformed to 01, the value from 0.8 to smaller than 1.2is transformed to 10, the value bigger than 1.2is transformed to 11. Table 1 shows the recognition accuracy of the present experiment. For happy and surprise, we got the excellent accuracy of 95 to 100 %, while, for neutral and sad, we got the good accuracy of 80 to 85 %. The totally averaged accuracy was 90 %. Moreover, through consideration on the poorly recognized data, it was found that the poorly recognized face had dierent characteristics from the normal input face in terms of thermal distribution. Since the input image was made by intentional action, the poorly recognized data might be caused by imperfect action. In addition, since the facial expression judgement as right answer was made by herself, the poor recognized data might be caused by her imperfect judgement with some ambiguity. However, the average recognition accuracy of 90 % can show that the present method for recognizing facial expression has the potential almost equal to human. Input emotion Neutral Happy Surprise Sad Neutral Happy Recognized emotion Neutral Happy Surprise 80 % 95 % 100 % 15 % Sad 20 % 5 % 85 % Table 1: Recognition accuracy for facial expression. Surprise Sad Figure 10: Examples of facial expressions. The present result was for one person. However, since the face identication method with IR image analysis has been developed [14], the data base of characteristic parameter for facial expression which can be made for each person will be available. Namely, after identication of the person, the facial expression can be recognized with the data base. The facial expression recognition in the case without learning data for the person and/or the facial expression is also future target. 5

6 The present method based on the thermal dierence between the averaged neutral and the test faces is considered to be easily applied to various facial expression. Since the NN output value of 0 to 1, before transformed into 0 or 1, is considered to express the degree linking up with the facial expression class, the intensity of facial expression for the test image will be dened with NN output value for recognizing facial expression in detail. Since the thermal change of face is caused by (1) muscular movement and (2) inner thermal change connected to mental and/or physiological transition, to separate these two inuences is also attractive target. To integrate face information from both IR and visible ray is also interesting target for more precise facial-expression-recognition. The dynamic analysis of facial expression change will be powerful for tackling the future targets mentioned above. Thermal image processing is considered to have potential to become a key for detecting human feeling or mind in our daily lives. 5 Conclusion A method for recognition of facial expression has been developed with Thermal Image Processing Technique. The method is based on 2-dimensional detection of temperature distribution of face. The frontview face in input image is normalized in terms of the size and the location, followed by measuring the local temperature-dierence between the averaged neutral and the unknown expression faces. The local temperature-dierence caused by the rearrangement of face muscle and the inner temperature change is used as input data for NN. Neutral, happy, surprise and sad expressions were recognized with 90% accuracy. Acknowledgments The authors wish to thank Mr.E.Hira of Mechanics Department, Miyazaki Prefectural Industrial Research Institute for his valuable suggestions for the experiment using IR apparatus. References [1] K. Sakai, M. Nagao and T. Kanade,\Computer Analysis and Classication of Photographs of Human Faces", In First USA-JAPAN Computer Conference, volume session 2{7, [2] A. L. Yuille, D. S. Cohen and P. W. Hallinan,\Feature Extraction from Faces Using Deformable Templates", Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 104{109, [4] K. Mase,\An Application of Optical Flow - Extraction of Facial Expression", IAPR Workshop on Machine Vision and Application, pp. 195{198, [5] K. Mase,\Recognition of Facial Expression from Optical Flow", Trans.IEICE,Vol. E74, No. 10, pp. 3474{3483, [6] K. Matsuno, C. Lee and S. Tsuji,\Recognition of Facial Expressions Using Potential Net and KL expansion", Trans. IEICE, Vol. J77-D-II, No. 8, pp. 1591{1600, (in Japanese). [7] H. Kobayashi and F. Hara,\Analysis of Neural Network Recognition Characteristics of 6 Basic Facial Expressions", Proc. of IEEE Internat. Workshop on Robot and Human Communication, pp. 222{227, [8] Y. Yoshitomi, S. Kimura, E. Hira and S. Tomita,\Facial Expression Recognition Using Infrared Rays Image Processing", Proc. of the Annual Convention IPS Japan, No. 2, pp. 339{340, [9] Y. Yoshitomi, S. Kimura, E. Hira and S. Tomita,\Facial Expression Recognition Using Thermal Image Processing", IPSJ SIG Notes, CVIM103-3, pp. 17{24, [10] P. Ekman and W. V. Fries, The Facial Action Coding System, Consulting Psychologists Press, Inc., San Francisco, CA, [11] H. Kuno, Sekigaisen Kougaku, pp. 22, 45, IEICE, Tokyo, (in Japanese). [12] D. E. Rumelhart, G. E. Hinton and R. J. Williams,\Learning internal representations by error propagation", Parallel Distributed Processing:Explorations in the Microstructures of Cognition, Vol. 1, MIT Press, pp , [13] N. Otsu,\A Threshold Selection Method from Gray-Level Histograms", Trans. IEEE, Vol. SMC- 9, No. 1, pp. 62{66, [14] Y. Yoshitomi, N. Miyawaki, S. Tomita and S. Kimura,\Facial Expression Recognition Using Thermal Image Processing and Neural Network", Proc. of 6th IEEE Internat. Workshop on Robot and Human Communication, this volume, [3] H. Harashima, C. S. Choi and T. Takebe,\3-D Model-Based Synthesis of Facial Expressions and Shape Deformation", Human Interface, Vol. 4, pp , (in Japanese). 6