Getting in Touch with a Cognitive Character

Transcription

1 Getting in Touch with a Cognitive Character Torsten Bierz * Peter Dannenmann Kai Hergenröther * Martin Bertram * Henning Barthel Gerik Scheuermann Hans Hagen * (*) University of Kaiserslautern, Germany ( ) German Research Center for Artificial Intelligence, Germany ( ) University of Leipzig, Germany bierz@informatik.uni-kl.de, Peter.Dannenmann@dfki.de, kai@hergenrother.de, bertram@informatik.uni-kl.de, Henning.Barthel@dfki.de, scheuermann@informatik.uni-leipzig.de, hagen@informatik.uni-kl.de Abstract We provide computer-animated characters with haptic interaction, allowing human users to interfere with cognitive characters. Our work presents an interface between control and animation of virtual characters (CONTACT) and HapTEK, a framework for haptic interaction of a human hand with virtual reality (VR). This way, humancomputer interaction becomes truly bidirectional, i.e. an animated character directly provides feedback to a human user, and vice versa, introducing a new quality with respect to the behavior of both. Our system for the first time offers the opportunity for human and artificial intelligence to get in touch. 1. Introduction Cognitive characters are introduced in computeranimated virtual environments as autonomous actors. Their behavior is driven by artificial intelligence, such that only a minimum of control is required. For example, a human animator can specify a certain task, like moving an item from point A to point B, and the cognitive character automatically decides which way to go and how to accomplish this task. This includes proper reaction on obstacles introduced by the user or by other cognitive characters. The objective is to make the character as self-dependent as possible, providing only abstract input. The simulation and animation of human characters in virtual environments has recently emerged in numerous areas of applications. At first only gaming, movie and advertising industries were interested in virtual characters. Nowadays, these are also integrated into development and manufacturing processes. In a virtual environment without haptics, interaction capabilities between human users and cognitive characters are quite limited. In a typical scenario, a user will specify a list of tasks and will observe the character s actions. In order to make a cognitive character even more selfdependent, we add the ability to provide force-feedback to a human user based on a haptic device. Haptic is referred to the tactile sense, which represents the perception of mechanical stimuli. In order to support the visualization of virtual environments, in our system mechanical stimuli representing shapes and surfaces are given to the user. To simulate the haptic properties of different shapes and materials, efficient collision detection and visualization methods are required. Although today s commercially available animation packages provide advanced tools for the creation of computer animations of virtual characters (e.g. inverse kinematics, key-frame editors), high quality animations are still costly and depend on skilled animators and programmers. To provide a higher degree of automation, artificial intelligence is introduced, supporting the animation process. Particularly, for the requirements of changing roles, different behavioral models need to be used. This issue is addressed by a research project named Control and Animation of Cognitive Characters in Virtual Environments (CONTACT, [3, 6]). CONTACT is the prototype of an animation system capable of generating motion sequences of virtual characters as well as permitting autonomous characters to be directed on multiple levels of abstraction. The system allows to define animations on a high-level basis mainly by specifying a task for a virtual character. This character will automatically work out an appropriate action plan, based on a list of possible actions and his environment. Actions implying movement of a character are created automatically by

2 combining reference motions from a database. The user is allowed to overrule the character s decisions and to force it to fall back on some predefined behavior. In the present work, we introduce a haptic interface for CONTACT, providing a new quality of interaction between a human user and a cognitive character. Therefore, we build an interface between our haptic controller HapTEK [10] and the CONTACT system. These are the requirements for our application: Collisions of the user s hand and the cognitive character of the scene are detected and proper force feedback is provided to the hand. The cognitive character will notice the human hand as an obstacle and will adjust its action plan to avoid a collision. On collision, the character may be displaced. Rendering and force feedback must be provided at interactive frame rates. In the next section we give a brief review of the current state of the art on haptics and on cognitive characters. Section 3 summarizes the basic principles of the CONTACT system. In section 4, the developed haptic system Hap- TEK with its basic functions is presented shortly, including underlying objectives. Afterwards, the combination of the two systems is described, including the goals and the difficulties we had encountered during implementation. We conclude our work in section 6. 2 A survey on haptic systems and animated characters Interactive force feedback offers a more realistic impression to the user, complementing the visual impression provided by an immersive VR environment. Haptic devices are being used in many applications, including surgical and medical simulation, painting, sculpting, visualization, and assisting technology for the blind and visually impaired [14]. There are also several approaches for using haptic devices as tele-operational systems [8, 13, 9] based on data exchange over a network. Other approaches offer haptic devices for feeling certain material properties [12]. Geometric models were among the first attempts to create computer animations. Forward and inverse kinematics are now widely used for key-frame interpolation. Physical principles are often introduced to reduce animation parameters to a minimum. Corresponding simulation tools are part of many animation packages, like 3D Studio. Max R, Alias WaveFront R, or Maya R. In recent years several researchers have extended these models towards behavioral animation in order to further automate the process of generating animations (see figure Figure 1: Models in Computer Animation [7] 1). Nevertheless, the modelling of a virtual character s behavior and consequently its animation is very difficult. Trying to focus on this issue in literature, there cannot be found a unified solution on how to perform this task. In fact, the difficulty is the the diversity of natural behavior within humans (see e.g. [15], [18]). One way to model a character s behavior is to employ scripting languages. The Character Markup Language CML, [2] describes a sequence of actions the character can perform in the virtual world. CML only serves as a means for representing the action sequences. It is also possible to generate such CML sequences by software tools that build up such scripts based on a set of action models, e.g., according to a set of current stimuli and an existing rule set that maps such stimuli to actions. In [2] such a piece of software is mentioned that generates a CML script based on inputs from a planning component taking into account the character s state and personality as well as his domain knowledge. However, (in contrast to the CONTACT framework) this approach only handles static environments where the characters actions are computed in advance. The control of characters on a high (or cognitive) level as described by Funge in [7], Chen et al. in [5] or in an earlier version by Tu and Terzopoulos [19] marks the next important step towards giving the characters a more intelligent behavior. In contrast to the reactive behaviors introduced by Reynolds and Anderson et al. they have got more sophisticated action selection techniques and are able to plan their actions further into the future. A step towards pursuing goals within a so-called Rulebased Behavior System is described by Raupp-Musse and Thalmann in [16]. They describe different levels of control for their actors depending on their role within the group. Another approach for directing the characters of a scene on several levels of abstraction has been introduced by Blumberg and Galyean in [4]. Within the CONTACT project we adopt Funge s view [7] (see figure 1) of explicitly using a cognitive model on top of a behavioral model in order to automate the process 2

3 Specification of behavior on the cognitive level. This task covers the development of methods for specifying all possible actions of each character on that level as logical descriptions of the action s preconditions and effect axioms in the Cognitive Modelling Language (CML, see [7]) as well as methods for specifying the character s goals in that description language. Based on this logical description of actions and goals a mechanism is provided to automatically generate in advance an appropriate sequence of actions to fulfill the given goals under consideration of the character s knowledge and perception of his environment. Figure 2: Architecture of the CONTACT Animation System of generating animations. This method of controlling virtual characters on a high (or cognitive) level marks the next important step in giving the characters a more intelligent behavior. A different approach, which describes the human interaction with virtual prototype products using haptic feedback was presented by Hiroo Iwata in [11]. Concerning the prototypes, the user is able to grab and feel their surface as force feedback. In [11] Iwata also suggested using haptic devices to manipulate character animations. A sccording to our knowledge, connecting a haptic system, permitting humans to closely interact with a computer to a cognitive character animation system is a new approach and has not been described before. 3 Basics of the CONTACT Project 3.1 Controlling and Animating Cognitive Characters Funge [7] raised the level of abstraction in animation generation by introducing autonomous characters. This idea was adopted by the CONTACT project, with the objective of developing a scalable and extensible working platform for autonomous characters in dynamic virtual environments. By this means, the user defines high-level goals within the cognitive layer, and virtual characters determine appropriate action sequences before starting the animation in order to obtain the given goal. These decisions of the cognitive character in the generated animation can be rejected or abandoned by the user by forcing the model to execute different actions. In order to reach the defined goals of the CONTACT project above the following research objectives were discussed: Specification of behavior on the behavioral level. For the implementation of behavioral patterns below the cognitive level methods for specifying such patterns and assigning them to actors are described by Reynolds in [17] and by Anderson et al. in [1]. Such methods are integrated into our animation system, too. Provision of a motion library. The animation of characters is generated by combining a number of motion sequences corresponding to the possible actions the characters can perform. These motion sequences are provided in the form of a library that can be accessed for retrieving the appropriate sequences according to the characters actions. The sequences have been established using motion capture techniques or by generating keyframed sequences. In order to use the motion sequences with different characters, a set of different sequences will be provided according to the characters differing anthropometries. Generation of the final animation. Based on the provided plans and the available reference motion sequences the final animation is generated. Supervision and interaction feasibilities. If the animator is not satisfied with a character s behavior on cognitive level, the system will allow the animator to force the character to fall back from the cognitive level to the behavioral level. In addition, the system allows the animator to force any actor to select and immediately execute one specific action of the available set of actions. Implementation of dynamic environments. For providing a realistic interaction of the virtual characters with their environment, dynamic environment objects are also integrated into the system. This permits the characters,for example, to pick up and 3

4 move objects, open or close doors and drawers, etc. In addition, due to changing dynamic environments, virtual characters must also be able to acquire environmental information to avoid collisions. 3.2 System Architecture The core concept of CONTACT is an object-oriented component system, such that new components can be integrated with little effort. The system provides its own communication infrastructure, implementing interfaces between the different modules. Figure 2 shows the main elements of the CONTACT animation framework for one cognitive character: The character s planning component has got access to a database containing the description of the character s possible actions as well as to a logical description of the character s current situation. This information, combined with the information about the character s (dynamic) environment is used by the planning component to generate an action plan. This action plan serves as a basis to generate the corresponding animation from a set of reference motions that are linked to the respective actions of the plan. reality engine HapTEK (see figure 3), only being dependent on the device drivers themselves. This offered the opportunity to implement all required functionalities within a single framework. System Architecture The HapTEK framework supporting the CyberGrasp R, CyberGlove R and CyberForce R device has been developed as part of a master thesis [10] at the University of Kaiserslautern. The development environment is based on Linux using non commercial software like the gnu C++ compiler and QT R 1. Additionally the Mico 2 CORBA 3 Interface has been integrated into the basic system. In addition, it contains a feature to exchange the virtual scenes. The system is built up modularly facilitating simple development of further extensions. An own virtual reality engine has been implemented using a scene graph, where every knot has its own homogeneous transformation matrix. OpenGL R 4 is used to traverse the graph and to generate and draw the current primitives. This framework provides stereoscopic viewing capabilities based on shutter glasses. This VR environment allows to estimate distances and positions of current objects more easily. Figure 3: Example of the HapTEK System. The user can grab, displace and throw geometric objects providing force feedback to the hand and to the five fingers. 4 The Haptic Framework When using Common-of-the-Shelf (COTS) haptic systems for applications of his own, the user encounters some difficulties since every haptic device has got its own device drivers and specifications. According to these drivers every vendor developed its own applications and extensions, e.g., for collision detection. In order to be independent of the predesigned software and to include additional functionalities required, we decided to develop our own virtual Figure 4: The operating range of the virtual hand One enormous problem is the navigation in the virtual world without dealing with additional devices. Classical input devices, like mouse and keyboard, are based on the assumption that the user can use both hands. However, the user is connected with one hand to the exoskeletal device of the CyberGrasp R and CyberForce R system. Navigating with the other hand may be uncomfortable. Therefore, we Common Object Request Broker Architecture 4 4

5 decided to integrate the navigation into the virtual hand of the haptic device (see figure 4). When the user leaves a neutral zone towards the border of the display, a movement is performed, e.g. if the hand leaves the zone to the left, the scene is rotated clockwise around the z-axis. By leaving the box at the bottom the scene rotates around the x-axis. If the user leaves the front or back border, the action performed is moving backward or forward. This allows the user to move and to grab objects simply by using only the haptic device. Within this solution it is also feasible to transport one object from one place in the scene to a different one without any usage of any further devices, e.g., the keyboard or the mouse. This approach offers more flexibility and simplicity than dealing with multiple devices at the same time. In our framework (see figure 5) we integrated two different kinds of objects: one that can be influenced by the haptic device and the others which cannot. Each of these influenceable objects definitely returns a force feedback to the current user in case it is grabbed, moved, thrown, shifted or touched. For interaction with those objects the integrated attributes are described now more in detail: Grabbing of objects: The interaction of grabbing is realized by touching an object with the thumb and at least one other finger, e.g. the middle or the index finger. Moving of objects: When a surface or object is grabbed, it is possible to transport or move it to a different point in the virtual scene. Throwing of objects: This interaction is performed when the user releases an object while the haptic device is still moving towards a specific direction. Shifting of objects: In order to be able to move an object without grabbing a shifting operation is integrated. The user can manipulate the object e.g. by pushing it with any finger. Now, the object moves according to the direction and the strength of the accomplished shifting. Modelling every surface or object in the application or source code is very time-consuming and counterproductive. By trying to simplify this integration, we integrated an interface for loading WaveFront R, VRML and MD3 R objects. Thus, it is possible to simply load and render the predefined objects in OpenGL. After the scene is loaded, the user is able to assign any of the aforementioned attributes, in order to adapt the object to his own perception. Figure 5: Combination of the HapTEK and CONTACT Framework 5 Combining the Haptic and the CONTACT Framework 5.1 The Integration During the integration of the haptic and the CONTACT framework a number of tasks had to be addressed. They can be classified to the following subjects: Merging the two different systems. The two systems have been developed using two different programming languages. In the CONTACT project Java was used and the software of the Hap- TEK system was realized using C++. However, both systems are designed to communicate via a network by using a CORBA client-server architecture (see figure 6) which can handle these issues. In this context the HapTEK system works as a server, providing its services to arbitrary clients. For permitting these clients to connect, we included a name service which dynamically returns the currently opened port to them. On this basis, the CORBA client integrated into the CONTACT framework can connect to the HapTEK CORBA server using the services provided by this system. Depending on the issues of merging the two different systems, we had to define an interface, which is based on IDL 5. The IDL is necessary to define the interface for calling procedures on remote machines in middleware systems. The remote procedures are: Set the force on the single fingers of the haptic device Get the actual force of the haptic device 5 Interface Definition Language 5

6 Get the position of the fingers of the haptic device Set the position of the fingers on the CORBA Server Get the position of the cognitive character Set the position of the cognitive character on the CORBA Server The values transmitted via the CORBA server are the actual position of the virtual hand s center and the position of the single fingers, that are provided by the HapTEK system. This system in turn receives the current position of the virtual character and its movement as well as the force acting on the hand if a collision has been detected. Accordingly, the CONTACT system emits these forces for the haptic device and the current position of the cognitive character. On the other hand, it receives the positions provided by the haptic device. Dealing with a firewall. In some computing environments it may be possible that the HapTEK device and the CONTACT animation framework are running in different networks protected by a firewall. In our case the Hap- TEK project has been developed at the University of Kaiserslautern whereas the CONTACT project is under development at the German Research Center for Artificial Intelligence (DFKI). Each institution handles its own network and, what is more important, has got its own subnet and firewall, which filters traffic quite effectively. So, one major issue in connecting the two components was the tunnelling of these firewalls, in order to realize the communication and data exchange. Handling these issues we decided to tunnel the firewalls by using the secure shell (SSH) protocol, which provides the possibility of making available a port from a remote computer as a virtual local port on another machine. Doing so, we were able to realize the data exchange via the internet. Synchronization. The two systems had to be synchronized in order to avoid displacement or invalid positions of the virtual hand and the character. This synchronization was integrated in the application. 5.2 Example and evaluation The combined system now works in the following manner: 1. Loading a scene. At the beginning a virtual scenario has to be loaded, which geometrically describes the environment. The Figure 6: Inter network connection. file could be in any of the data formats mentioned in section Defining the initial state. The cognitive character and the haptic device, represented as a virtual hand in the scene, are initialized and rendered corresponding to the user defined location. 3. Defining the character s goals. The user defines the character s high-level goals using the Cognitive Modelling Language (CML). These goals may include topics like (in an informal description) Take item box to position camera. These high-level goals together with a (CML-) description for the character s possible atomic actions serve as the basis for calculating the character s action plan. 4. Calculating the action plan. During the computation of an action plan for pursuing the above described goals, the CONTACT system determines a sequence of actions, typically including a plan for getting from one position in the scene to another one, avoiding collisions and using the heuristically best and fastest path. The different plans are displayed to the user, enabling him to possibly overrule the selected heuristically best plan and force the character to execute one of the other computed action plans. Having selected an appropriate action plan, the animated character starts moving. 5. Haptic interactive events. During the execution of the action plan, the haptic device offers the capability to dynamically influence the virtual character s calculated plan by simply holding the virtual hand as an obstacle in the virtual character s path. When a collision between the character and the hand is detected, the currently performed plan will be aborted. According to the current direction and speed of the cognitive character, forces for the haptic device are generated. Hereafter, the recalculation of the action plan for fulfilling the still uncompleted 6

7 Figure 7: Sequence of screenshots showing the cognitive character walking along a planned path and its interaction with the haptic fingers, which are represented by five spheres. tasks begins, avoiding interference with the hand as a newly emerged obstacle which cannot be passed. Thereupon the cognitive character continues its path in order to reach its objective and the forces are released. Of course, the user is certainly able to interfere with the character in this way not only once but multiple times. An example representing such an interaction is presented in figure 7. To simplify geometry for interactive rendering (using the CONTACT renderer on the remote side), the fingertips of the virtual hand are represented by spheres. In the first of the three screenshots the character approaches the virtual hand. In the second one a collision is detected and the character stops its planned path intermediately, in order to calculate a new path bypassing the hand. After having found a new path, the character turns around and walks a different way in order to reach its target. The remaining question is why using a haptic device for interfering with a cognitive character. One of the simplest reasons is related to human beings. If you want to prevent a person from walking a way, you can grab his shoulder or arm and he probably stops his movement. Another example is a police man controlling the traffic on a non traffic lighted junction. He rises his hand in order to stop the traffic. This basic gesture is commonly known in nearly every country in the world. It might also be feasible to use the keyboard, a joystick or a space mouse in order to realize this interaction with the cognitive character. For untrained persons, however, the handling of these devices and the corresponding navigation tasks are often very difficult. This disadvantage is repealed by using a haptic device. Its handling is based on natural movement of humans and thus easier to learn. 6 Conclusion and Future Work In this work we described our approach for merging a haptic with a cognitive character animation system. We employed a way of interacting with the cognitive character and influencing its planned path through the virtual scene. The corresponding interaction with the human user, which is the collision with the representation of the virtual hand, can be felt with the haptic device. After this collision the user may choose to withdraw the obstacle again. Then, the character replans its path and continues to pursue its task. In the current system, the character plans its paths from a starting position to target positions corresponding to the different goals in the logical (CML) description of its actions and objectives. A further step in refining the presented system could be selecting the character s different target points using the haptic device. By touching an object it could be added to the current plan, making the interaction and navigation much simpler. 7 Acknowledgement The HapTEK project was developed with funding of the Support Program of the Foundation for Innovation of Rheinland-Pfalz. The CONTACT system is funded by the German Federal Ministry for Education and Research under the contract number 01 IW A03. 7

8 References [1] Matt Anderson, Eric McDaniel, and Stephen Chenney. Constrained animation of flocks. Proc. Eurographics / SIGGRAPH Symposium on Computer Animation, [2] Yasmine Arafa, Kaveh Kamyab, Ebrahim Mamdani, Sumedha Kshirsagar, Nadia Magnenat-Thalmann, Anthony Guye-Vuilleme, and Daniel Thalmann. Two approaches to scripting character animation. Proc. Workshop Embodied Conversational Agents - let s specify and evaluate them! at AAMAS, [3] H. Barthel and P. Dannenmann. Animating virtual characters controlled on the cognitive level. IASTED International Conference on Visualization, Imaging and Image Processing (VIIP 2004), [4] Bruce M. Blumberg and Tinsley A. Galyean. Multilevel direction of autonomous creatures for real-time environments. Proceedings of SIGGRAPH 95, August [5] L. Chen, K. Bechkoum, and G. Clapworthy. A logical approach to high-level agent control. ACM Proceedings of the Fifth International Conference on Autonomous Agents, [6] P. Dannenmann and H. Barthel. Controlling the behavior of virtual characters on cognitive level. IASTED International Conference on Artificial Intelligence and Soft Computing (ASC 2004), [7] John Funge, Xiaoyuan Tu, and Demetri Terzopoulos. Cognitive modeling: Knowledge, reasoning and planning for intelligent characters. Proc. ACM SIG- GRAPH99, [8] Mario Gutierrez, Renaud Ott, Daniel Thalmann, and Frederic Vexo. Mediators: Virtual haptic interfaces for tele-operated robots. 13th IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN 2004), [11] Hiroo Iwata. Artificial reality with force-feedback: development of desktop virtual space with compact master manipulator. Proceedings of the 17th annual conference on Computer graphics and interactive techniques, [12] Hiroo Iwata, Hiroaki Yano, Fumitaka Nakaizumi, and Ryo Kawamura. Feelex: adding haptic surface to graphics. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, [13] P. Lemoine, M. Gutierrez, F. Vexo, and D. Thalmann. Mediators: Virtual interfaces with haptic feedback. Proceedings of EuroHaptics 2004, [14] Margaret L. McLaughlin, Joao P. Hespanha, and Gaurav S. Sukhatme. Touch In Virtual Environments - Haptics and the design of Interactive Systems. Prentice-Hall Inc., first edition, [15] Christoph Niederberger and Markus H. Gross. Towards a game agent. CS Technical Report 377, ETH Zürich, Switzerland, August [16] Soraia Raupp-Musse and Dahiel Thalmann. Hierarchical model for real-time simulation of virtual human crowds. IEEE Transactions on Visualization and Computer Graphics, 7(2): , [17] Craig W. Reynolds. Flocks, herds, and schools: A distributed behavioral model. Computer Graphics, 21(4):25 34, Proc. SIGGRAPH 87. [18] Daniel Thalmann and Jean-Sébastien Monzani. Behavioural animation of virtual humans: What kind of law and rules? Proc. Computer Animation IEEE Computer Society Press, [19] Xiaoyuan Tu and Demetri Terzopoulos. Artificial fishes: Physics, locomotion, perception, behavior. Proc. ACM SIGGRAPH 94, July [9] Mario Gutiérrez, Renaud Ott, Daniel Thalmann, and Frédéric Vexo. Mediators: Virtual haptic interfaces for tele-operated robots. 13th IEEE International Workshop on Robot and Human Interactive Communication, [10] Kai Hergenröther. Design and implementation of a haptics-enabled vr framework supporting immersion cyberglove / cybergrasp / cyberforce haptics hardware. Master s thesis, Technical University of Kaiserslautern,