MULTIMODAL EMOTION RECOGNITION FOR ENHANCING HUMAN COMPUTER INTERACTION

Transcription

1 MULTIMODAL EMOTION RECOGNITION FOR ENHANCING HUMAN COMPUTER INTERACTION THE THESIS SUBMITTED TO SVKM S NMIMS (Deemed to be University) FOR THE DEGREE OF DOCTOR OF PHILOSOPHY IN COMPUTER ENGINEERING BY PREETI KHANNA Under the Guidance of Dr. Sasikumar M. Director (R & D), Centre for Development of Advanced Computing, Mumbai SVKM s NMIMS University MUKESH PATEL SCHOOL OF TECHNOLOGY MANAGEMENT & ENGINEERING, Mumbai-56 MAY 2012

2 MULTIMODAL EMOTION RECOGNITION FOR ENHANCING HUMAN COMPUTER INTERACTION Thesis submitted by Preeti Khanna for award of the degree of DOCTOR OF PHILOSOPHY in ENGINEERING Under the Guidance of Dr. Sasikumar M. Director (R & D), Centre for Development of Advanced Computing, Mumbai SVKM s NMIMS University MUKESH PATEL SCHOOL OF TECHNOLOGY MANAGEMENT & ENGINEERING VILE PARLE (W), MUMBAI MAY 2012

3 DECLARATION I hereby declare that the research work embodied in the present thesis entitled "Multimodal Emotion Recognition for Enhancing Human Computer Interaction is the outcome of my own efforts under the guidance and supervision of Dr. Sasikumar M., CDAC, Mumbai. I further declare that to the best of my knowledge the thesis does not contain any part of any work which has been submitted for the award of any degree either in this University or in any other University without proper citation. Place: Mumbai Date: (Signature of Candidate) Preeti Khanna Forwarded By Dr. D. J. Shah Dean, MPSTME, SVKM s NMIMS

4 CERTIFICATE This is to certify that the thesis titled Multimodal Emotion Recognition for Enhancing Human Computer Interaction, embodies research work done by Preeti Khanna under my guidance and supervision for the degree of Doctor of Philosophy in Engineering of MPSTME, SVKM s NMIMS, Mumbai, India. To the best of my knowledge and belief the thesis embodies the work of the candidate herself and fulfils the requirements of the ordinance relating to the Ph.D. degree of the university. Place: Mumbai (Signature of Guide) Date: Dr. Sasikumar M. CDAC, Mumbai Forwarded By Dr. D. J. Shah Dean, MPSTME, SVKM s NMIMS

5 THESIS APPROVAL CERTIFICATE This is to certify that the thesis entitled Multimodal Emotion Recognition for Enhancing Human Computer Interaction submitted to NMIMS (Deemed-to-be University), for Ph.D. in Engineering by Preeti Khanna, a bonafide student of Mukesh Patel School of Technology Management and Engineering, Vile Parle (W), Mumbai-56 has been evaluated and recommended for the award of Doctor of Philosophy Degree in Engineering. Signature Signature Name-: Guide Date-: Name-: Examiner Date-: Dr. S.Y.Mhaiskar Dean, MPSTME, SVKM s NMIMS

6 Dedicated to My Little Master Abhinav Kaushal, My Husband Manish Kaushal And My Parents for their affection, support and encouragement..

7 ACKNOWLEDGEMENT This thesis would not have seen the day of fruition without the direct and indirect support, motivation and cooperation of many individuals whom I would like to briefly acknowledge here. I would like to acknowledge each one of them for their invaluable contributions to my work. First of all, I would like to convey my sincere gratitude to my supervisor, mentor and guide Dr. Sasikumar M. for giving me the opportunity to work with him. I am really thankful to him for his relentless patience in supervising my work and for his unique capacity to generate plenty of research ideas and alternatives. It s with immense gratitude that I acknowledge Dr. D.J.Shah, Dean, MPSTME, SVKM s NMIMS for encouraging me to come up and complete my work. I am thankful to Dr. Gita A. Kumta for their never-fading support. I would like to take an opportunity to thanks to Mrs. Vashali Kulkarni for her valuable suggestions. Thanks to my colleagues, past and present, for their various contributions and for their friendship. I am gratefully indebted to our institution, SVKM s NMIMS and specially School of Business Management for imparting knowledge through my peer group and colleagues and adding value. Needless to say I am very grateful for the moral support provided by my family and my friends. They never lost faith in my capability to get through and I thank them for their patience with my rambling about finishing the dissertation. It is their strong believe in my aspirations that kept me going all throughout. Special thanks to Manish for listening to my daily technical talks and trying hard not only to understand but also provide ideas to solve some of my problems. My special appreciation to Ujwala, Vipin, Deepti and my dearest Abhinav for being wonderful friends and for providing continuous support throughout my journey. Last but not least, I would like to acknowledge the most important persons in my life, my parents and parents-in-laws. They have offered me true examples of perseverance, integrity and moderation and they gave me unconditioned support.

8 ABSTRACT Emotions are fundamental to human lives and play an important role in communication between people. This interactional phenomenon has been extensively studied by several research groups worldwide. Enabling computers to read emotions has been motivated by the potentially wide range of applications which involve human-computer interfaces. Such an affective computer system is an active area of research, which include recognizing emotions as well as generating appropriate responses. This dissertation addresses the emerging area of research on emotions and human computer interaction. We present approaches for recognition of emotions by considering static image of face, speech and keyboard stroke pattern modalities. Each of these modalities has been investigated separately as unimodal systems. Recognizing the inadequacy of unimodal approaches, the study also explores multimodal emotion recognition framework by combining these modalities. The work involves identifying and extracting relevant features from each of these modalities and using various classification algorithms for classifying them into emotions. The features extracted from the face include the position of eyes, eyebrows, mouth, and nose and various distances involving them. Features of interest in speech are pitch, formant frequencies, parameters related to voiced and unvoiced region and MFCC. The features extracted from keyboard interaction are typing speed, number of backspaces used, number of errors made, etc. Each of these features was analyzed individually to see its variation across emotions. We have used classification algorithms such as neural networks, bayesian classification etc. to classify the emotions. Various studies involving different subsets of features and different classification algorithms were carried out to identify relevant features and classification framework. The performances were also evaluated for gender dependency across emotions for different subset of features and classification algorithms. The thesis presents a hybrid approach for multimodal emotion recognition. The criteria for combining the three modalities were based on if-then rules. A set of rules

9 were formulated from these modalities by analyzing each feature, in detail, individually across the target emotions. With the help of feature analysis, the importance of individual features could be found and measured across the set of target emotions. Studies are done separately for males and females, and also in a gender independent way. The performance of the traditional feature based model for multimodal approach was compared with this rule based hybrid model. Further the thesis explores issues of appropriate behavioural changes based on emotional states.