Effect of Animated Characters and Adjustable Tags in AR Storytelling

Transcription

1 Effect of Animated Characters and Adjustable Tags in AR Storytelling R.P.C. Janaka Rajapakse* Yoshimasa Tokuyama** Tainan National University of the Arts*, Tokyo Polytechnic University** Abstract Mobile computers, smart phones, tablets, and computer software applications have become ubiquitous in academia and daily life. Education institutes and teachers are expanding their teaching materials with computer vision based applications. Augmented reality (AR) applications are very famous and educators use it to produce interactive teaching materials, specially focused on digital storytelling projects. Unfortunately, current AR-driven applications and their contents are not sufficient for the development of storytelling skills. The most common criticism is that the 3D contents are less animated and controlled with predefined and fixed tags which consume considerable amount of time for changing tags as well as not help to translate their correct story ideas. The main objective of this work is to investigate the effect of animated characters and adjustable tags in AR storytelling. This paper presents a simple framework for making manually adjustable AR-tags to represent the different character poses and different states of characters which would help to develop the storytelling skills. 1. Introduction Augmented Reality (AR) technologies and tools have been widely studied in numerous fields, including the game industry, advertising, fashion, interior design, education and learning, medical and surgical operations, product design and manufacturing, archeological restoration and museums, and countless other areas [1, 2]. This technology allows computer created CG contents to be superimposed over real-life live video feeds. While most of AR applications have been oriented to fields such as education or entertainment, mobile computing and high-quality mobile display technologies have gained enormous progress. Nowadays, mobile computers, smart phones, tablets, smart books, and pads are often used in academia and daily life. There are large numbers of mobile applications and services designed and optimized for edutainment. Within the academia mobile AR applications and computer vision based applications stand out and attract considerable attention [3, 4]. Moreover, commercial and open-source AR application development libraries have been introduced that enable a continuous increase of functionality and portability [5, 6, 7, 8, 9, 10]. Marker and tag based AR applications have been used to produce interactive teaching materials, specially focused on digital storytelling projects. The traditional workflow of digital storytelling framework focused on pre-production, production and post-production [11]. In such environment, it is difficult to increase the interactivity. AR can aid learning and make the overall storytelling process much more interesting and interactive [12, 13, 14]. Although AR is still a very 35 promising technology for digital storytelling, currently only a few applications exploit the potential of developing storytelling skills. The most of AR storytelling applications can be immersed a user, but it is not capable for the development of storytelling skills. The most common criticism is that the 3D contents in the virtual worlds are less animated and controlled with predefined and fixed tags which consume considerable amount of time for changing tags as well as not help to translate their correct story ideas. Moreover, existing applications change the role of storyteller to interactive presenter. The pedagogy of storytelling is very different from interactive presentation [15, 16]. In order to develop an AR-driven interactive pedagogical storytelling application it is important to investigate the weaknesses of the current AR applications. The main contribution of this pilot work is to examine the effect of animated characters and adjustable tags in AR storytelling. This paper focuses on simple framework for making manually adjustable AR-tags to represent the different character poses and different states of characters which would help to develop the storytelling tools. 2. Background 2.1 AR Tags and 3D Contents The most common AR applications published so far allow to the user to interact 3D contents by using several different techniques such as AR desktop tag (marker)-based, AR desktop tag-less, AR freehand tag-less, AR by mobile, AR by projection etc. Mainly, all technologies

2 use tracking methods for live video feeds and control superimposed 3D contents by common geometric transformations such as rotation, scaling, and translation, either by tag-based techniques or by tag-less techniques. The best known tag-based AR applications for storytelling are based upon the space trend tracking of a graphical tag, usually a geometric black&white symbol called confidence on which we can hook in real time a two-dimensional or three-dimensional input. Some AR applications can be controlled without a geometric marker through a process that recognize graphically a generic image or a portion of it, after which the tracking system will be able to recognize identity and orientation. The freehand controlled tag-less AR applications that trace position, orientation and direction of the user look, using tracking systems and various movement sensors, associated to natural interactive systems such as Microsoft s Kinect, Wavi Xtion by PrimeSense with Asus3D, Nintendo s Wii-mote and the Sony s PlayStation Move [18]. Recently RFID/NFC and QR codes have been associated in edutainment systems that allowing users to retrieve more information using their mobile phones [19]. Therefore, interactive digital edutainment tools and projects are rich with opportunities for assessment [20]. However, Taiwan elementary school has been proven to be a traditional and still conservative stage so that new technologies arrival created somewhat resistance to the regular use of these interactive edutainment tools. Adopting new technology and finding budgets are challenges that inevitably will cause a change in the management of both classrooms and schools. Naturally the experience says that any big change requires an adjustment phase but it is always very welcome cheap and easy tools for classroom use. Figure 1: Kids were interacting AR contents in the survey sessions. To understand the role of the 3D contents in an AR-driven digital storytelling application, it makes sense to examine the kid s feedback about the existing applications and their contents. We visited few elementary schools and did the pilot survey for acquisition of kid s feedback on AR-contents (see Figure 1). In those surveys, while the elementary school students were using few AR applications [12], and we carefully investigated that how these tools had shaped student s own ideas. Students engaged in these sessions not only had an entertaining experience, but they also asked very important questions. After analyzing their questions, we noticed that existing AR contents deal a lot in expressing What?, Who?, Where?, and When? while students were trying to express their ideas in How? and Why? variety. The problem is that how existing AR contents control the curiosity issues. Considering curiosity issues and handling rich AR contents, and turning them into an AR application is not an easy process. What does it take to create a successful AR storytelling application? There are several variables to consider: realistic 3D contents, animating contents, adjustable tags; rich interactivity, platform; time, available resources, and target ordinance. Jonassen et al. defined that the meaningful learning has to be active, intentional, cooperative, authentic, and constructive [20]. A successful AR storytelling application clearly provides meaningful learning environment as defined in [20]. 2.1 Storytelling and Interactive Media How do computer games tell stories, and what does this mean for current digital storytelling? Games are things that are designed systematically, thoughtfully, artistically for the purpose of being fun. And the game designers work is not just a matter of pure engineering. There's a lot of engineering involved, such as a lot of algorithms and interactive techniques. But there's also an artistic experiential side of game design that involves thinking about problems in a certain way. Therefore, game design has grown within a rich bed of interacting trends and traditions in interaction design and games, and that there are already a number of potentially competing, and overlapping concepts. Most recently introduced academic term Gamification focuses the use of game design elements in non-game contexts [21]. Our focus is storytelling, and expects to investigate on the various degrees to which different media and platforms support stories. Our point is not to settle the ontological status of games, but to focus on their storytelling functions not to decide what proportion of gaming is storytelling, but to identify how games tell stories. However, looking for storytelling in gaming makes too much of too little. But there are many gaming discussions on narratology vs. ludology. Our background review also experienced that the games primarily as mechanisms for narrative delivery and those emphasizing gaming s mechanical operations beyond stories. The storytelling application is not only an educational media as fancy content delivery mechanisms, but also a tool that helps kids design and develops their own ideas and improving storytelling skills. Creative learning tools such as Scratch [22], NetLogo [23], and LEGO [24], empower kids as self-motivated learners to program, build, animate, and design their own creations. LEGO, for example, would be a perfect tool for constructing an active brain. The purpose of this research is to develop such tools for storificatoin, sharing them, and expanding the realm of digital storytelling. Initial step of this research will proceed by developing a series of AR-driven storytelling applications. Each of these developments will guide to invent a story console and simple character control devices (watch-like) for interactive pedagogical storytelling. 36

3 3. Effect of Animation The first series of experiments was investigated the effect of animation on AR-contents. The two groups of AR stimuli were experimented. The first group of AR stimuli was consisting non-animated 3D contents and characters, and the second group of stimuli had animated contents. A total of fifty six elementary school kids (31 female and 25 male) between 6 to 13 years of age participated in the experiments. In the experiments, the participant s task was to interact the provided (randomly) AR stimuli from each group (see the Figure 1) and chose the interesting ones. Over 95% of the subjects judged that the realistic animated AR-contents were interesting than the still contents. 3.1 Realistic characters The based on the kid s evaluation on AR contents, this framework has focused upon the creation of realistic animated AR contents for the proposed character console. The character modeling and texture composition process were conducted using Autodesk Maya, Mudbox and Adobe Photoshop CS 5 software packages. In order to enhancing realism and visual impact of the virtual characters, UV-mapping techniques and material editing were concerned with real photographs and advanced material models. across the surface of a model. For example, in lion character, mane is hair-like and longer than other furs. To control exactly where the different fur grows, separated models were used (see Figure. 6). The several attributes maps were used to control fur parameters such as colors, length, baldness, direction, etc. Figure 3: Created 3D model of dog character (left), while painting skin weights (middle) and while rigging with FK and IK (right). In the modeling of human hair, dynamic hair curves in Maya were used to create dynamic motion. To modify and customize created stock of characters for breeding many versions with different costumes and body types, Autodesk Pinocchio tool was used. Figure 4 depicts few customized characters with different face textures. Figure 4: Customized characters with different facial features. The process of complex motion editing, blending and scaling were carried out by using Autodesk s MotionBuilder software package. To create more complex body language for biped characters, the captured motion data (from Vicon bonita-10, eight camera MOCAP system) were used and mapped to the different characters as shown in Figure 5. Figure 2: Creating 3D model & skeletons for skin binding (top), UV-mapping and textured version of character (bottom). Techniques of key-framing and Kinematics (IK/ FK) were used to make animation for peoples and animal characters. To make animated characters look real, anatomical structure guided joint hierarchies known as a skeletons, were created and placed inside the 3D models by using Maya software (Figure 2). Inverse kinematics rigs, technique of smooth binding of skin geometry and Maya muscle were used in order to create realistic deformation for character rigs. Different quadrupeds have varying models of walking and galloping as shown in Figure 3. To adding fur to the animals, polygonal models with properly mapped UV coordinates that lie within a 0 to 1 range in texture space were considered. For some animals, fur is not uniformly Figure 5: Motion data (left) and mapped characters (middle & right). 37

4 3.2 Multipose and Multistate How do multipose and multistate characters help to storytelling? Naturally, multipose has been part of storytelling since early ages of handmade puppet shows. Expensive game controllers and gesture based interactive applications can control multi-camera views and character poses [18], but less effort has been spent on multipose and multistate storification. However, typical AR applications don t facilitate such storification functions. The pose of the character, age, and body proportions are very important character attributes for digital storytelling (Figure 6). Our framework has considered providing adjustable AR tags for controlling multiposes and multistate of the characters. All AR stimuli were standalone applications and running on notebook PCs equipped with common USB webcams or intergrated webcams that provide images at a resolution of 640*480 pixels, with a frame rate of 30 fps. Experimental results were indicated that the over 90% of participants preferred to use MLT controllers than traditional blac&white makers. MLT control papers provided enough visual information to easily select the characters and their poses. Over 82% of subjects were interested in using MLT-based watch like wearable craft controller. However, it was difficult to control with a low-resolution commodity webcams. Our framework proposed more suitable AR controllers for storytelling. As depicted in the Figure 8, multi-mlt tag cylinder is for representing character s multiposes and multistate, and cube is for different characters. After testing in kid s workshops, character cube has improved with manually adjustable folded faces for multiposes and multistate as shown in Figure 9. Figure 8: Multipose & multistate cylinder (left), Character cube (right) Figure 6: Different poses obtained from skeleton and IK handling (top and middle), different body propositions in different ages (bottom). 4. Effects of Adjustable AR Tags Our second series of experiments addressed the effect of adjustable AR tags. In the experiments, participant s task was to judge the most interesting and suitable AR content controlling method for storytelling. Each AR stimuli were controlled by tree methods as shown in Figure 7, the first by traditional marker in black square(figure 7-a), the second was advanced markerless tracking(mlt) by printed images (b), and the third was also based on MLT, but watch like a paper-made craft (c). (a) (b) (c) Figure 7: Experimented AR-content controlling methods. Figure 9: Character cube with folded faces for multipose & multistate. 4.1 Implementation This research project is envisaged not only to just investigation, but also motivated to comprise the development of several AR storytelling applications for PC and mobile media. The first phase of the proposed framework was implemented with a D Fusion AR authoring tools [25]. After modeling and confirming all scenes and animations in the Maya software, the D Fusion exporter has used. It is working as a Maya plug-in which translate Maya data into Ogre3D data [26]: geometry (.mesh), lights, bones (.skeleton), materials (.material), and scenes (.scene). When using the longer MOCAP data in the Maya, timeline has managed according to the character poses and split it pose wise without overlapping (for example clip1 [10-50] and clip2 [51-100]) before exporting for D Fusion. To enable the MLT functionalities, AR scenarios has configured using D Fusion studio computer vision tool elements such as tracking scenario, camera calibration, camera capture. Basically, this process has composed with three steps: recognition, 38

5 initialization, and tracking as shown in the Figure 10. Registering imported scene files with corresponding captured images were implemented with Lua scripting [27] and maintained the scene data base. Figure 10: Recognition (left), initializing and tracking (right) To tracking multiple objects, used few planer tracking at the same time using the same video capture. The standard 3D transformation parameters (position, orientation, scale) were not suitable for all multiple 3D objects. For restoring initial parameters according to the multiple object proportions, D Fusion object editor was used. Player was used to deploying standalone applications. 5. Conclusion This paper has presented the initial phases of ongoing research project to develop a framework for interactive pedagogical storytelling. The results were helpful to identify the problems and practical challenges in AR storytelling. The development of this pilot prototype as well as initial field surveys has taken an important approach to introduce advanced media technologies for effective digital storytelling. This paper investigated AR storification and introduced a simple method for making manually adjustable AR-tags to represent the different character poses and different states of the character. This may guides and motivates kids to make their own story crafts. The current work is implementing the mobile versions of all developed AR story applications. Future work would develop a character console and simple sensor-driven AR content controller for pedagogical storytelling. References [1] R. T. Azuma, A survey of Augmented Reality, In Presence: Teleoperators and Virtual Environments, Vol. 6, No. 4, pp , [2] D. J. Haniff, C. Baber, W. Edmondson, Categorizing Augmented Reality Systems, International Journal of Three Dimensional Images, Vol. 14, No. 4, pp , [3] T. Höllerer, S. Feiner, Mobile Augmented Reality. In Karimi, H. & Hammad, A. (eds.) Telegeoinformatics: Location-based Computing and Services. London: Taylor & Francis Books Ltd., [4] D. Wagner, History of Mobile Augmented Reality, Communications, pp. 1-21, 2009 [5] W. Wagner, D. Schmalstieg, History and Future of Tracking for Mobile Phone Augmented Reality, Proc. of International Symposium on Ubiquitous Virtual Reality(ISUVR 09), pp. 7-10, [6] ARToolKit, A Software Library for Building AR Applications, [7] FLARToolKit, ActionScript version of ARToolKit, [8] Layar, Layer Specializes in Mobile Augmented Reality, [9] Hoppala, Mobile Augmented Reality Platform, [10] Vuforia, AR-based Application Development Environment, [11] Digital Storytelling in the Classroom, [12] Zooburst, Digital Storytelling Tool, [13] ARART, New Art Experience with Augmented Reality, [14] Toontastic, [15] J. C. Kuyvenhoven, In the Presence of Each Other: a Pedagogy of Storytelling, PhD thesis, University of British Columbia, [16] G. M. Deniston-Trochta, The Meaning of Storytelling as Pedagogy, Visual Arts Research, Vol. 29, No. 57, [17] K. Matsuo, M. Hagiwara, Entertainment AR Aquarium, Journal of Art and Science, Vol. 10, No. 4, pp , [18] B. Lange, S. Rizzo, C-Y. Chang, E. A. Suma, and M. Bola, Markerless Full Body Tracking: Depth-Sensing Technology within Virtual Environments, Proc. of Interservice/Industry Training, Simulation, and Education conference, paper no , pp. 1-8, [19] O. Haberman, A. Damal, R. Pellerin, U. Haberman, E. Gressier-Soudan, Exploring Contemporary Painting through Spatial Annotations Using RFID Tags. Proc. of the 11th Int. Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage, France, [20] D. H. Jonassen, D. H., Howland, J., Moor, J., Marra, R. M. (2003). Learning to Solve Problems with Technology: A Constructivist Perspective (2nd Edition), Prentice Hall, USA. [21] S. Deterding, D. Dixon, R. Khaled, L. Nacke, From Game Design Elements to Gamefulness: defining Gamification, Proc. of the 15 th International Academic MindTrek Conference: Envisioning Future Media Environments, pp. 9-15, [22] Scratch, [23] NetLogo, [24] Lego, [25] D Fusion Studio, from Total Immersion, [26]Ogre3D. [27] Lua scripting language, 39