Affective Information Processing

Transcription

1 Affective Information Processing

2 Jianhua Tao Tieniu Tan Editors Affective Information Processing ABC

3 Dr. Jianhua Tao, PhD Professor Tieniu Tan, BSc, MSc, PhD Institute of Automation, Chinese Academy of Sciences 95 Zhongguancun East Road Haidian, Beijing, , P.R. China ISBN: e-isbn: DOI: / British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Control Number: c Springer-Verlag London Limited 2009 Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Printed on acid-free paper Springer Science+Business Media springer.com

4 Preface Affective information processing involves studies that attempt to give computers the humanlike capabilities of observation, interpretation, and generation of affect features. It is an important topic in human computer interaction (HCI), because it helps improve the quality and effectiveness of human to computer communications. Words such as affect, affective, and affection are usually linked to humans and have rarely been used for lifeless machines. Affective information processing is quite new in HCI which considers affect features and investigates ways of capturing and processing these by computers. The computation of affective multimodal information is also known as affective computing and has become a very hot research topic. In 2005, we organized the first international conference on affective computing and intelligent interaction (ACII2005) in Beijing. There were 134 papers accepted for the conference in total, and about 140 people from 32 countries attended the conference. The conference proceedings were published as a volume in Springer LNCS. In 2007, the second ACII was held in Lisbon with an attendance of around 150. ACII is now an event of the HUMAINE Association. From that time on, ACII has become a central event for researchers studying and developing the role of emotion and other affective phenomena as these relate to human computer and human robot interaction, graphics, artificial intelligence, robotics, vision, speech, synthetic characters, games, educational software, and so on. The international enthusiasm at ACII encouraged us to edit a book on affective information processing by soliciting original theoretical and practical work offering new and broad views of the latest research in this new and exciting area. Most of the authors were selected and invited from among the attendees of ACII2005, and some chapters were written by other leading researchers outside ACII2005. This ensures that the book has a balanced coverage of the important topics. The book summarizes some key technologies that are organized into four parts: Cognitive Emotion Model, Affect in Speech, Affect in Face, and Affect in Multimodal Information, with 17 chapters. Unlike other books which often focus on psychological problems or cognitive models for emotions, this book gives more content on how to process affective information for the next generation of v

5 vi Preface HCI technologies. Thus the multimodalities of affective information in the form of language, speech, and face are emphasized in this book. We have no intention of addressing all problems in affective information processing within this book, but try to give a wide description and some detailed knowledge in this area. We hope that the book can be taken as a useful reference book and will help promote more and deeper studies in this exciting area. As a final word, we would like to thank all the authors for their quality contributions. Our thanks also go to Wayne Wheeler and Catherine Brett from Springer for their invaluable help and guidance throughout the project. This book is supported in part by the National Natural Science Foundation of China under Grant ( ) Beijing, China May 10, 2008 Jianhua Tao Tieniu Tan

6 Contents 1 Introduction... 1 Jianhua Tao and Tieniu Tan Part I Cognitive Emotion Models 2 Affect and Emotions in Intelligent Agents: Why and How? Andrew Ortony 3 Cognitive Emotion Modeling in Natural Language Communication Valeria Carofiglio, Fiorella de Rosis, and Nicole Novielli 4 A Linguistic Interpretation of the OCC Emotion Model for Affect Sensing from Text Mostafa Al Masum Shaikh, Helmut Prendinger, and Mitsuru Ishizuka 5 Affective Agents for Education Against Bullying Ruth Aylett, Ana Paiva, Joao Dias, Lynne Hall, and Sarah Woods Part II Affect in Speech 6 Emotion Perception and Recognition from Speech Chung-Hsien Wu, Jui-Feng Yeh, and Ze-Jing Chuang 7 Expressive Speech Synthesis: Past, Present, and Possible Futures Marc Schröder 8 Emotional Speech Generation by Using Statistic Prosody Conversion Methods Jianhua Tao and Aijun Li vii

7 viii Contents Part III Affect in Face 9 Why the Same Expression May Not Mean the Same When Shown on Different Faces or Seen by Different People Ursula Hess and Pascal Thibault 10 Automatic Facial Action Unit Recognition by Modeling Their Semantic And Dynamic Relationships Yan Tong, Wenhui Liao, and Qiang Ji 11 Face Animation Based on Large Audiovisual Database Jianhua Tao, Panrong Yin, and Le Xin Part IV Affect in Multimodal Interaction 12 Affect in Multimodal Information Anna Esposito 13 Nonverbal Feedback in Interactions Kristiina Jokinen 14 Emotion Recognition Based on Multimodal Information Zhihong Zeng, Maja Pantic, and Thomas S. Huang 15 A Multimodal Corpus Approach for the Study of Spontaneous Emotions Jean-Claude Martin and Laurence Devillers 16 Physiological Sensing for Affective Computing Christian Peter, Eric Ebert, and Helmut Beikirch 17 Evolutionary Expression of Emotions in Virtual Humans Using Lights and Pixels Celso M. de Melo and Ana Paiva Index...335

8 Contributors Aijun Li Institute of Linguistic, Chinese Academy of Social Science, Beijing, Ana Paiva IST Technical University of Lisbon and INESC-ID, Avenida Prof. Cavaco Silva Taguspark, Porto Salvo, Portugal, Ana Paiva INESC-ID, TagusPark, Av. Prof. Dr. Cavaco Silva, Porto Salvo, Portugal, Andrew Ortony Northwestern University, 2120 North Campus Drive, Annenberg Hall, Evanston IL , USA Anna Esposito Dipartimento di Psicologia, Seconda Università di Napoli Via Vivaldi 43, Caserta, Italy IIASS, Via Pellegrino 19, 84019, Vietri sul Mare, SA, Italy, CelsoM.deMelo USC, University of Southern California, Christian Peter Fraunhofer Institute for Computer Graphics Rostock, Joachim Jungius Str. 11, Rostock, Germany Chung-Hsien Wu Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan, Eric Ebert arivis Multiple Image Tools GmbH, Schwaansche Straße 1, Rostock, Germany Fiorella de Rosis Department of Informatics, Via Orabona 4, Bari, Italy Helmut Beikirch University of Rostock, Faculty of Computer Science and Electrical Engineering, Albert-Einstein-Str. 2, Rostock, Germany Helmut Prendinger Digital Contents and Media Sciences Research Division, National Institute of Informatics, Hitotsubashi, Chiyoda-ku, Tokyo, Japan, ix

9 x Contributors Jean-Claude Martin LIMSI-CNRS, France Laurence Devillers LIMSI-CNRS, France Jianhua Tao National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, Joao Dias NESC-ID, TagusPark, Av. Prof. Dr. Cavaco Silva, Porto Salvo, Portugal, Jui-Feng Yeh National Chia-Yi University, Chia Yi Kristiina Jokinen University of Helsinki and Tampere, Finland, Le Xin National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing Lynne.Hall School of Computing and Technology, University of Sunderland, Sunderland, SR6 0DD, UK, Mostafa Al Masum Shaikh Department of Information and Communication Engineering, University of Tokyo, Hongo, Bunkyo-ku, Tokyo, Japan, Mitsuru Ishizuka Department of Information and Communication Engineering, University of Tokyo, Hongo, Bunkyo-ku, Tokyo, Japan, Maja Pantic Imperial College London University of Twente, Netherlands, Marc Schröder DFKI GmbH, Saarbrücken, Germany, Nicole Novielli Department of Informatics, Via Orabona 4, Bari, Italy Panrong Yin National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing Pascal Thibault Department of Psychology, McGill University, Montreal, Canada, Qiang Ji Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY, Ruth Aylett Mathematics and Computer Science, Heriot-Watt University, Edinburgh, EH14 4AS, UK, Sarah Woods Psychology Department, University of Hertfordshire, Hatfield, Herts, AL10 9AB, UK Tieniu Tan National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing

10 Contributors xi Thomas S. Huang University of Illinois at Urbana-Champaign Imperial College London, Ursula Hess Department of Psychology, University of Quebec at Montreal, PO Box 8888, station A, Montreal Qc. H3C3P8, Valeria Carofiglio Department of Informatics, Via Orabona 4, Bari, Italy Wenhui Liao The Thomson Corporation, Eagan, MN, Yan Tong General Electric Global Research Center, Schenectady, NY, Ze-Jing Chuang Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan Zhihong Zeng University of Illinois at Urbana-Champaign Imperial College London,