Representing People in Virtual Environments. Will Steptoe 11 th December 2008

Transcription

1 Representing People in Virtual Environments Will Steptoe 11 th December 2008

2 What s in this lecture? Part 1: An overview of Virtual Characters Uncanny Valley, Behavioural and Representational Fidelity. Part 2: Virtual Characters in VEs Games, Online Social Worlds, Immersive VR, Avatars, Agents. Part 3: 3D Studio Max Demo Part 4: Technical Aspects of Virtual Characters Graphics, Animation, Behaviour

3 What is in this lecture? Virtual characters and realism Graphics, Animation and Behaviour Virtual characters in Virtual Environments Avatars, agents, interaction and AI Character rigging and Animation 3DS Max Demo

4 Highly realistic characters

5 Highly realistic characters Highly realistic characters can cause more problems Realistic characters existing in stills (beginning to in games and VEs) but less so in movies. Not just a computing power issue There are a lot of complex issues to deal with when you have more realistic characters

6 Uncanny Valley As the behaviour and representation of robots (and other facsimiles) of humans approaches that of actual humans, it causes a response of revulsion among human observers. Theory from 70s by roboticist Masahiro Mori Controversial, its not very rigorous or scientific, many people don t believe it There are problems but it maybe captures something

7 The Uncanny Valley

8 The Uncanny Valley Dreamworks reduced realism of Princes Fiona (Shrek): she was beginning to look too real, and the effect was getting distinctly unpleasant Final Fantasy it begins to get grotesque. You start to feel like you're puppeteering a corpse

9

10 Uncanny Valley At low levels of realism, the more realistic a character the more people like it (even this is dubious) But when you get almost real then characters start to get disturbing This is very strong, the uncanny means very disturbing, corpses are used a lot as metaphors Interestingly, there are 2 graphs, movement and appearance, movement is more important

11 Different Levels of Realism Graphical Realism What it looks like (pictures, games, VE, film) Animation Realism How it moves, animation (film, games, VE) Behavioural Realism How it responds and interacts (games/ve)

12 Mismatch in Realism Maybe the problem is that levels of movement and behavioural realism do not match graphical realism This mismatch disturbs us, something that looks human but does not act like a human Consistency

13 Appearance vs. Behaviour Vinayagamoorthy, V., Garau, M., Steed, A., and Slater, M. (2004b). An eye gaze model for dyadic interaction in an immersive virtual environment: Practice and experience. Computer Graphics Forum, 23(1):1 11.

14 Realistic responses in VE? Individuals' self-rated performance was positively correlated with the perceived good mood of the agents Evidence of a negative response especially strong with the negatively inclined audience Sweating and stammering Vocal protests at the agent behaviours Virtual humans with minimal behavioural-visual fidelity can elicit significant user responses Holy grail: Virtual humans with high visual fidelity that mimic real-life context-appropriate behaviours

15 Appearance vs. Behaviour Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., and Sasse, A. M. (2003). The impact of avatar realism and eye gaze control on the perceived quality of communication in a shared immersive virtual environment. In Proceedings of SIGCHI, pages Beh. App. Random gaze Inferred * gaze Cartoon Form 3 pairs 3 pairs 3 pairs 3 pairs Higher Fidelity 3 pairs 3 pairs 3 pairs 3 pairs

16 Appearance vs. Behaviour Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., and Sasse, A. M. (2003). The impact of avatar realism and eye gaze control on the perceived quality of communication in a shared immersive virtual environment. In Proceedings of SIGCHI, pages App. Cartoon Form Higher Fidelity App. Cartoon Form Higher Fidelity Beh. Beh. Random gaze 3 pairs 3 pairs 3 pairs 3 pairs Random gaze High Low Inferred * gaze 3 pairs 3 pairs 3 pairs 3 pairs Inferred * gaze Low High

17 Appearance vs. Behaviour In each of the responses, the higher fidelity avatar had a higher response with the inferredgaze model And a low response with the random-gaze model Important to note that the differences between both the gaze models were very subtle Saccadic velocity and inter-saccadic intervals (means) Analysis demonstrated a very strong interaction effect between the type of avatar and the fidelity of the gaze model The higher-fidelity avatar did not outperform the cartoon-form avatar Similar hypothesis in the fields of robotics

18 Appearance vs. Behaviour Sparse environment abandoned building Minimise visual distraction One genderless cartoon form character Two gender-matched higher fidelity characters Behaviour Common limb animations and condition-dependent gaze animations Individuals listening in a conversation look at their conversational partner for longer periods of time and more often than when they are talking Negotiation task to avoid a scandal - 10 minutes

19 What responses do you get? Pertaub, D.-P., Slater, M., and Barker, C. (2002). An experiment on public speaking anxiety in response to three different types of virtual audience. Presence: Teleoperators and Virtual Environments, 11(1): David Not very comfortable with public speaking Asked to speak about his favourite subject: cables Behaviours triggered at appropriate intervals Look at the virtual humans

20 The Fear of public speaking The user was asked to give a presentation thrice Positive, Negative and Mixed Positive - agents smiled, leaned forward, faced the user, maintained gaze, clapped hands, etc. Negative - agents yawned, slumped forward, put feet on the table, avoided eye contact, and finally walked out Mixed - agents started off with largely negative responses and gradually turned positive

21 Realism vs Believability The lesson is that we need to be careful with realism for virtual humans Often we prefer to use the term Believability Not how much a character is objectively like a human How much we feel it is/respond to it as if it is Bugs Bunny is very Belivable Photorealism is only one element of believability But don t turn into an anti-realism zealot!

22 Part 2: Characters in Virtual Environments So far we haven t talked about characters in virtual environments Characters are often key to an environment, the primary content We are interested in people so populated environments are interesting

23 Games

24 Games Some characters are just there to be shot at

25 Games

26 Games Some games like the Sims have more complex interactions Supposedly, they have important social interactions (To me it seems more about going to the toilet)

27 Games

28 Games Mario is interesting, he is your representation in the game An avatar Can be compared to a cursor, marking your position, but is clearly something more A strong personality, you take on some of his personality, rather than him just representing you

29 Multi-user worlds

30 Multi-user worlds Avatars become much more important in multiuser worlds (the most important feature?) They also represent you to other people They affect how people perceive you

31 Immersive VR

32 Immersive VR In immersive systems you can interact with life size, real time character This changes the experience again

33 Avatars and Agents Characters in virtual environments fulfill many roles but there are two primary types Avatars Representations of you, or other people User controlled (tracked) Agents Others, that you interact with Computer Controlled Hybrid Part tracked, part simulated

34 Interactive Behaviour Key to both roles is the interaction with a character Composed of two elements, UI and AI User interface In what ways do we interact with a character? Artificial (Augmented) Intelligence How does the character respond? How is it controlled?

35 Agents Many different style of interaction for agents Cannon fodder, Non-player Characters, Crowds, Complex conversational agents Many interactions, shooting, moving, conversation (from dialogue trees to spoken interaction)

36 Agents - Game NPC UI: AI: Moving, shooting Simple conversation Finite state machines Scripts Path Planning

37 Agents - Embodied Conversational Agents UI: AI: Speech conversation Gestures etc. Tracking data Complex conversational AI methods

38 Agents - Embodied Conversational Agents

39 Virtual Humans - Agents Agents are entirely program controlled rather than representing an on-line human. These are examples from virtual fashion shows.

40 Avatars Your representation in the VE A vital part of multi-user worlds A very complex relationship A separate identity that to take on (e.g. Mario) A new identity that you create for yourself (e.g. Second Life)

41 Avatars Outline Why Virtual Human Representations? Avatars and Agents Representing a Person in a VE Designing Virtual Humans Emotion, Personality and Social Intelligence Believable behaviour Conclusions

42 Virtual Human Representations Useful and interesting applications are with other people Simulation of real events Training Entertainment Shared VEs The Others are entirely virtual The Others are entirely real As in shared (networked VEs)

43 Networked VEs Need some representation of the other people in the shared VE Typically called avatars Avatars represent the real tracked person Spatial representation Where they are, what they are looking at Behavioural representation What they are doing

44 Avatars in Shared VE

45 Avatar Representations Spatial Behavioural

46 Avatars and Identity Most users of virtual worlds use avatars as a means of identity creation Customization is vital Appearance, clothes, hair, sometimes animation The relationship to real identity is complex Have a different appearance, personality, gender Explore hidden sides of yourself Some people feel their avatar is More Me than their physical self

47 Avatars as social tools Ideally avatars is social VEs should support social interaction Display the bodily functions of communication (body language) However, most avatars in most virtual worlds don t The body movements often exist, but most users use them unrealistically or often not at all Primarily a problem of control

48 Avatar Mediated Communication

49 Controlling avatars Typed Text, Emoticons, Traditional GUI, Speech, Full body tracking

50 Problems with Controlling Avatars Two modes of control: at any moment the user must choose between either selecting a gesture from a menu or typing in a piece of text for the character to say. This means the subtle connections and synchronisations between speech and gestures are lost. Explicit control of behaviour: the user must consciously choose which gesture to perform at a given moment. As much of our expressive behaviour is subconscious the user will simply not know what the appropriate behaviour to perform at a give time is [BodyChat, Vilhjalmsson, H. and Cassell, J., 1998]

51 Problems with Controlling Avatars Emotional displays: current systems mostly concentrate on displays of emotion whereas Thórisson and Cassell (1998) have shown that envelope displays subtle gestures and actions that regulate the flow of a dialog and establish mutual focus and attention are more important in conversation. User tracking: direct tracking of a user s face or body does not help as the user resides in a different space from that of the avatar and so features such as direction of gaze will not map over appropriately. [BodyChat, Vilhjalmsson, H. and Cassell, J., 1998]

52 Solutions Always ensure that any control is done through a single interface (e.g. through text chat) BUT. The body language of an avatar should be largely autonomous, and indirectly controlled by users Minimize the level of control needed [BodyChat, Vilhjalmsson, H. and Cassell, J., 1998]

53 Solutions: Spark Text Chat based environment Parse users text input for interactional information Use this information to generate behaviour

54 Solutions: Spark

55 Solutions: PIAVCA Operator Speech Generation Script Database Speech Multi-model utterances User Interaction Movements Concurrent Behaviours Motion Queue Proxemics Posture Shifts Gaze Final Animation

56 Part 3: 3DSMax Demo

57 Part 4: Technical Aspects of Virtual Characters Graphics Polygon meshes, rendering Animation Skeletal animation, mesh morphing, physical simulation Behaviour

58 Graphics Techniques: Meshes, texture mapping, standard graphics stuff Hand modelling: can be cartoony or highly realistic 3D Scanning/phototextures: can have very high realism Rendering Opacity: Subsurface scattering

59 Modelling Scanned body results in huge mesh which can be rendered at different resolutions (numbers of polygons)

60 Body Animation Can Hand animate the skeleton Often use motion capture Real data = Realism (?)

61 Skeletal Animation The fundamental aspect of human body motion is the motion of the skeleton The motion of rigid bones linked by rotational joints (first approximation) I will discuss other elements of body motion such as muscle and fat briefly later

62 Typical Skeleton Circles are rotational joints lines are rigid links (bones) The red circle is the root (position and rotation offset from the origin) The character is animated by rotating joints and moving and rotating the root

63 Forward Kinematics (FK) The position of a link is calculated by concatenating rotations and offsets R 0 R 1 P 2 O 0 O 1 O 2

64 Forward Kinematics (FK) First you choose a position on a link (the end point) This position is rotated by the rotation of the joint above the link Translate by the length (offset) of the parent link and then rotate by its joint. Go up it its parent and iterate until you get to the root Rotate and translate by the root position

65 Forward Kinematics (FK) Simple and efficient Come for free in a scene graph architecture Difficult to animate with, often we want to specify the positions of a characters hands not the rotations of its joints The Inverse Kinematics problem: Calculating the required rotations of joints needed to put a hand (or other body part) in a given position.

66 Inverse Kinematics An number of ways of doing it Matrix methods (hard) Cyclic Coordinate Descent (CCD) A geometric method (secretly matrices underneath) R 0 P t R 1 O 1 O 2

67 Inverse Kinematics Start with the final link

68 Inverse Kinematics Rotate it towards the target

69 Inverse Kinematics Then go to the next link up

70 Inverse Kinematics Rotate it so that the end effector points towards the target

71 Inverse Kinematics And the next

72 Inverse Kinematics And the next

73 Inverse Kinematics And iterate until you reach the target

74 Inverse Kinematics And iterate until you reach the target

75 Inverse Kinematics And iterate until you reach the target

76 Inverse Kinematics And iterate until you reach the target

77 Inverse Kinematics And iterate until you reach the target

78 Inverse Kinematics IK is a very powerful tool However, it s computationally intensive IK is generally used in animation tools and for applying specific constraints FK is used for the majority of real time animation systems

79 Minimal Tracking for IK in VR Badler et al showed a minimal configuration for IK representing the movements of a human in VR ~hollick/presence/presence.html It was shown that 4 sensors are sufficient to reasonably reconstruct the approximate body configuration in real-time.

80 Representation Layered representation Skeleton structure forms a scene graph Scene graph embodies a set of joints A mesh overlays the scene graph As the skeletal structure moves the mesh must deform appropriately (otherwise there are holes) MPEG4 example

81 Facial Animation Don t have a common underlying structure like a skeleton Faces are generally animated as meshes of vertices Animate by moving individual vertices

82 Morph Targets Have a number of facial expressions, each represented by a separate mesh Each of these meshes must have the same number of vertices as the original mesh but with different positions Build new facial expressions out of these base expressions (called Morph Targets)

83 Morph Targets

84 Morph Targets Smoothly blend between targets Give each target a weight between 0 and 1 Do a weighted sum of the vertices in all the targets to get the output mesh

85 Using Morph Targets Morph targets are a good low level animation technique Also need ways of choosing morph targets Could let the animator choose (nothing wrong with that) But there are also more principled ways

86 Summary Virtual human avatars are necessary to represent people to themselves and in shared VEs. Virtual human agents are necessary to represent social situations. VHs are represented typically as skinned skeletal scene graphs, representing sets of joints. Forward kinematics determines overall configuration given joint angles and Inverse kinematics determines joint angles from requirements for end-effectors Representations typically need to be a mixture based on tracking data and inferred state. Morph targets are a method of mesh deformation often used for facial animation Later will go on to consider more sophisticated models of behaviour determination, and also social intelligence.

87 Believable Behaviour For agents the behaviour is completely programmed. For avatars the behaviour is ideally completely determined by the behaviour of the real tracked human. In practice the human cannot be fully tracked typically in VR only head and one hand movements are tracked!

88 Controlling/Inferring Behaviour Avatar Tracking In practice some elements of avatar behaviour are programmed not tracked E.g., breathing and eye blinking at the least Ideally can use information about mood to determine aspects of avatar behaviour. Impossible to track every aspect of the human s behaviour so much must be inferred and programmed. Real avatars are mixed. Mixture of both Agent Programming

89 Behaviour Autonomously deciding what action to take at a given time Not necessary for film but vital for real time interaction At the interface between Graphics, AI and A-Life The subject of the rest of this talk

90 Behaviour Outline An overview of early (land mark) behavioural simulation techniques An overview of social behaviour simulation taking in Control algorithms Psychological theories How social behaviour is expressed in animation

91 Social Intelligence These techniques work well as far as they go but do not model the complexities of human social interaction Vital if we are to have interesting interactions with autonomous characters Also useful for making our interactions via avatars closer to real interactions

92 Designing virtual humans GOAL: Represent the Person in VE consistently With perceived realism, believability Induce responses to the virtual human Inducing realistic/lifelike responses Enhancing collaborative experience Facilitate social communication and interpersonal relationships

93 Designing behaviour Creating apparent social intelligence is challenging Have to present behavioural cues to depict a perceived (and plausible) psychological state Or the near-truth internal state of the Person being represented Human behaviour is a very intricate phenomenon Dependent on many factors Extremely difficult to replicate especially if the design process is approached in an ad-hoc manner For instance: In social interactions within VE, the more visually realistic the virtual human, the more naturalistic users expect it to act

94 Inferring Behaviour: Animation imitating life Lasseter, J. (1987). Principles of traditional animation applied to 3d computer animation. ACM SIGGRAPH Computer Graphics, 21(4): Emotional models Controllers of behaviour in accordance to internal states Personality models Creating unique identities Conversation-feedback models Controlling behaviour Social models Interpersonal relationships and attitudes???

95 Categories of behavioural cues Argyle, M. (1998). Bodily Communication. Methuen & Co Ltd, second edition. Vocal properties Tone, Pitch, Loudness Facial expressions The most studied behavioural cue due to it s role in communication Gaze behaviour Probably the most intense social signallers Kinesics: Posture and Motion Numerous gestures depending on culture for instance Proxemics Culture and gender dependent

96 Facial expression In reality, facial expressions exist Normally animated by blending Morph Targets Different granularities of facial expression Facial action parameters (most basic units) Basic emotions Phonemes (mouth shapes for lip-sync) Principal component analysis

97 Gesture Normally animated by choosing from a library of gestures Very closely associated with speech Also back channel gestures by listeners (e.g. head nod) Different types of gesture E.g. beat, iconic Again see Cassell s work referenced earlier

98 Posture Coulson, M. (2004). Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28(2): Over 1000 stable postures have been observed Normally animated by choosing from (or blending between) a library of gestures Associated with attitude and emotion Associated also with interpersonal attitude

99 Measuring Success So the careful design of behaviour is important but there are caveats Success of a VE is measured in terms of the extent to which sensory data projected within a virtual environment replaces the sensory data from the physical world quantified by rating the individuals sense of presence during the experience For Virtual Humans: Success is taken as the extent to which participants act and respond to the agents as if they were real Subjective: Questionnaires, Interviews Objective: Physiological, Behavioural

100 Subjective means Traditional methods: Questionnaires and interviews Various questionnaires exist Criticised due to its various dependencies the individual s accurate post-hoc recall, processing and rationalisations of their experience in the VE and Varying interpretations of the word presence

101 Objective: Responses to stimuli Numerous possible objective measures Subconscious responses Threat-related facial cues provokes individuals to use different viewing strategies Neural responses Different areas of the brain are activated during +ve, -ve and neutral situations Psychological responses Stress and Anxiety in response to threat Physiological responses Galvanic Skin Responses, Heart Rate Variability, electrocardiograms, electromyography, Respiratory activity Behavioural responses Flight or Fight (based on cognitive appraisal) Vary based on cognitive factors, personality, emotional state, gender etc. How do we interpret the data and results?

102 Conclusion Virtual human agents are necessary to represent social situations Social intelligence is rather difficult to capture Emotional, personality and interpersonal models The design of behaviours should be implemented with consideration to many other factors Current Research focus on quantifying the successful creation of Virtual Humans using objective measures