, pp. 271-278 http://dx.doi.org/10.14257/ijseia.2016.10.1.26 Real-time AR Edutainment System Using Sensor Based Motion Recognition Sungdae Hong 1, Hyunyi Jung 2 and Sanghyun Seo 3,* 1 Dept. of Film and Digital Media, Seokyeong University, 2 GSAIM, ChungAng University 3 Electronics and Telecommunications Research Institute 1 sungdaehong@naver.com, 2 hyunning1202@gmail.com, 3 shseo@etri.re.kr Abstract Recently the Natural User Interface (NUI) technology which is capable of appreciating the whole human body has come to the fore with the development of digital technology. And this new interface has settled as the competitive contents in the growing experiential learning contents market, by arousing participants interests and maximizing their learning effect, through a gesture recognition-based noncontact type interactive education. This study suggests various interpretations on the basis of the education contents production which enables participant to experience the human body in real time. Also it has developed a gesture interface by utilizing the Kinect sensor that can recognize a participant s skeleton and behavior, and its education contents design has produced the end product by using Unity 3D authoring tool which can combine the real-time 3D model and animation together. Consequently, participant will be able to learn various inner organs such as brain and heart by different gestures, and the developed real-time human body exploring augmented reality system is expected to be widely used in many educational institutions Keywords: Augmented Reality, Kinect, Interactive Human, NUI, Physical Interface 1. Introduction Use of ICT (Information & Communication Technology) such as new media is rapidly increasing also in education field and the smart education market is expanding every day all over the world. However, it is actually rare to find a field application case of the latest ICT and it is only utilized in application of textbook like e-book. Therefore, a development of tangible type edutainment contents design which promotes the sense of reality and immersion was tried as one of the alternatives which meet the requirements for the high quality educational contents. Especially it is necessary to produce a tangible type edutainment design which can maximize the learning effect through an experiential factor. So, this study suggests a human body recognition real-time interactive tangible type experiential-learning contents design which recognizes a movement of the user and reacts to it. In Chapter 2, an interaction technology for virtual reality-based human body learning educational experience is discussed. Next to that, the applicable range for the contents production and development is sought by exploring other research cases for the real-time educational experience contents. In Chapter 3, a process of the real-time human body experiencing edutainment design production is materialized. Also it abstracts the human skeleton recognition and image processing of the Kinect sensor interface for the participant s behavior recognition in real time, and materializes a design which applies the augmented reality-based real time 3D contents. * Corresponding Author ISSN: 1738-9984 IJSEIA Copyright c 2016 SERSC
Real-time game engine Unity 3D was used as a way of demonstrating the final contents, and lastly, the conclusion and direction of future research for the real-time human body experiencing edutainment design is described. 2. Previous Work In digital contents field, the physical interactive games such as Xbox 360 and Nintendo Wii, specific functional games in various fields like education, entertainment, military, management, or medical treatment field, and virtual reality-based interactive experiencing contents have been leading the market. And recently, the virtual reality-based participatory contents is expanding to the learning service that a user can directly participate in the virtual contents, interact with and experience the learning contents [1,2]. 2.1. Digital Participatory Interface Technology Japanese Nintendo developed a game service named Wii, which adopted Wiimote, a 3D acceleration sensor-equipped game controller shown in Figure 1. This technology provides an interaction function which is required for playing the game, by recognizing the user s controller operation, and it demonstrates a superior function in recognizing even minor motions which is needed for game environment. But because of inconvenience in using the device and limited service-able environment, it is not appropriate for the learning purpose. Figure 1. Nintendo Wii Figure 2. KINECT by Microsoft Kinect is a motion sensing input device officially launched by Microsoft in October, 2010 for game console, XBOX 360. Edutainment content for children unveiled by Microsoft in 2012 is characterized by a use of gesture recognition technology, by enhancing voice recognition on the Kinect which recognizes a user s motion through 3D camera (Figure 2). However, it is not suitable for the education or broadcasting field because of its low accuracy in interaction performance, and even it requires an additional training for the user to operate it [3, 4]. 2.2. Case of Digital Experience Contents D strict shown in Figure 3 developed the contents which can freely control commercial menus with the infrared ray-based electronic glove. The augmented reality commercials solution developed by this company is a technology that enables a user to find the commercial images floated in the air by moving them around, through the hands motion with the infrared ray and marker- equipped electronic glove. 272 Copyright c 2016 SERSC
Figure 3. Hyper Presentation by D strict Figure 4. GestureTek s 3D Avatar Figure 5. 4D Technology Showcase by D strict Also GestureTek developed a full body 3D avatar control technology through a user extraction. Figure 4 shows the 3D avatar control by GestureTek. This technology performs a virtual reality navigation function by extracting human skeleton data in the user image. But unfortunately, it cannot provide an augmented reality experience which combines user image and virtual image partially. D strict also released a 4D system which obtains a view transform in the virtual space by recognizing the user s posture and gesture, through the Kinect sensor. Based on this, it supports a space movement and turning and can appreciate a concrete motion such as foot stamping or stretching-out a hand. Figure 5 shows the 4D technology showcase by D strict. 3. AR Based Human Body Experience System An augmented reality is a linkage system of virtual object. According to a definition by Paul Milgram, a mixed reality can be divided into Augmented Reality characterized by a virtual object overlapped on the basis of the real world, and Augmented Virtuality characterized by a reality superimposition on the basis of the virtual world [5]. It can also coordinate the virtual images in the real world and carry out a real-time interaction and especially it is a technology which provides a user with an improved sense of immersion and reality, as it is a seamlessly demonstrated technology within a 3D space [6]. Figure 6 shows the human-body learning H/W system. This system is a learning service technology that participants can take part in direct 3D stereoscopic virtual contents and experience the human-body learning contents through mutual interactions [7]. It can increase an immersion degree in learning as well as a sense of bond between learners, by a mutual interaction through user s motion in the real-time 3D virtual space. It is a device allowing close observation on user s skeleton, organ, frame, and even muscles by the Kinect sensor for user s motion, and the Kinect sensor which detects Copyright c 2016 SERSC 273
user s motion suggests the contents shown in Figure 7, which enables participant to learn the human body in detail, by showing organs, bones, and muscles within the human body in the image-based augmented reality space. Figure 6. Real-Time Human-Body Learning H/W System Figure 7. The Concept of Real-Time Human-Body Learning As the 3D Depth Sensors are equipped in Kinect which is used as a main device in a materialization of this system, it can recognize 1.2m to 3.5m depth information. Materialization is achieved by Unity 3D game engine, so that it can segue into a 3D virtual space area that can recognize a participant s motion, through a natural dissolve from 2D webcam to 3D camera. Figure 8 and Figure 9 shows the workflow of original AR workflow and the simplified AR workflow respectively. We employed the simplified AR flow in our system. Motion detection algorithm is applied in the augmented reality and the tracking technology is materialized in order that it can recognize a participant s motion in real-time and accurately harmonize synchronization of images displayed in the 3D character and UHD screen. Figure 10 explains the technical plan of our exhibition system. Participant is recognized by 2D camera as a skeleton according to its distance and a user can directly select the desired screen with a motion gesture. Divided into three layers - Skeleton and Organs, Skeleton, and Organ, screen interface is designed for user to see the whole body by touching each layer icon. In case of Organs, four organ icons - Brain, Heart, Lung, Liver, Stomach, and Intestine - are shown at the right section (Figure 11, 12). Touching a hand over each icon, a user can see the contacted organ and in that case, other parts become translucent Figure 13). With a touch on the organ icon, the selected 274 Copyright c 2016 SERSC
organ is shown largely at the screen with a rotation and each function is displayed in text, and then it is back to its place after 8 to 10 seconds later. Figure 8. General AR Workflow Figure 9. Simplified AR Workflow Figure 10. Technical Plan of Distance Sensor System A user s figure shown through a webcam is changed into 3D state if a user enters into the area of 3D by a dissolve effect, and UI design enables a user to be recognized by webcam as a skeleton frame according to the distance and also to easily select the desired screen. Figure 14 shows the operation of our system. A complete system of the developed real-time interactive human body experiencing contents is shown in the figure, and a conversational space where a real time interaction is possible is materialized by an expansion of Unity 3D engine. To be more specific, equipped with the photo-realistic 2D camera and depth sensor (Z-cam), it recognizes a user s motion by figuring out the depth-map obtained from Z-cam and user s skeleton structure. Also in order to enhance a realistic effect for a participant, it is built by UHD display system. Copyright c 2016 SERSC 275
Figure 11. Screen Transition of 3D Virtual Environment Figure 12. 3D Objects of Organs Figure 13. Interaction Image of 3D Organs Figure 14. The Result of our System 4. Conclusions It is said that NUI type experience contents which appreciates the whole human body is an effort to bridge a gap between a user and an avatar by practically projecting a user s motion onto an avatar and also that it induces a direct illusory experience of the case commonly through a camouflage of the process, behavior or action [8, 9]. Therefore, this study provides a strong experience effect of game behavior that is possible for a user to experience directly, and introduces a tangible type edutainment which improves learning effect by utilizing the human-body experiencing educational contents. This study is the contents suggesting a new direction which leads to a NUI-based interactive learning education. An event that lets a user experience a human body through a response to various motion gestures in real time has been applied and it materializes a system which switches a whole body into a single interface. It has also attempted to improve an effect of experiential learning as much as possible, by reproducing the role and state of real organs into an interactive simulation format. This user-based interactive education contents are expected to suggest a new direction to a participation method and learning education in the growing edutainment market. Also it will not leave its role just within a boundary of interaction, but also expand the education field as well. In future research, it is expected to study and develop the contents, by implementing a system which can achieve a learning effect on the basis of multiple participants and getting something out of just an individual participation. Acknowledgments This research was supported by SeoKyung University in 2016. 276 Copyright c 2016 SERSC
References [1] H.-C. Yoon and J.-S. Park, Avatar Animation Using Skeleton Tracking With Kinect Sensor, International Journal of Advancements in Computing Technology (IJACT), vol. 5, no. 12, (2013), pp. 339-344. [2] L. Meng, H. Xinyuan, Guoxin, The Research and Experiment about Interactivity and Immersion of Virtual Reality, International Journal of Digital Content Technology and its Applications (JDCTA), vol.7, no. 13, (2013), pp.132-141. [3] P. Horejsi1 and T. Gorner, Using Kinect Technology Equipment for Ergonomics, MM Science Journal, (2013), pp. 388-340. [4] C. Sinthanayothin, N. Wongwaen and W. Bholsithi, Skeleton Tracking using Kinect Sensor & Displaying in 3D Virtual Scene, International Journal of Advancements in Computing Technology (IJACT), vol. 4, no. 11, (2012), pp. 213-223. [5] P. Milgram, Augmented Reality, A class of displays on the Reality-Virtuality Continuum, SPIE vol. 2351, (1994), pp. 282-292. [6] J. S. Lee, J. A. Noh, S. H. Lim and J. S. Lee, An Activity Contents Technology Trend Based on Virtual Reality, ETRI, Electronics and Telecommunications Trends, (2012), pp. 73-82. [7] S.-M. Hwang, H.-G. Yeom, Computer Interface Construction for Recognition of Motion and Voice using KINECT, International Journal of Advancements in Computing Technology (IJACT), vol. 5, no. 12, (2013), pp. 235-240. [8] J.-Y. Kim and J.-H. Sung, The Formation of New Game Generation in Game-Extended Space: Focused on the Experience Game, Korea Game Society, vol. 10, no. 5, (2010), pp. 1-11. [9] H. Y. Jung, Characteristics of Interactive Performance Focused on Body Medium, ChungAng University, (2013). Authors Sungdae Hong is currently a full-time faculty member of the Dept. of Film and Digital Media at Seokyeong University in South Korea. He received his BS degree in Computer Design Major and his MS and Ph.D. from Chung-Ang University (Seoul) with an Art and Technology major in 2002~2008. With unique educational background from CG to Media Arts, he is interested in Interactive Media Art and Image Processing. He has presented his artworks in various digital art shows including SIGGRAPH 2006 G-Studio and ASCI Digital07, Graphite 2006~2007, Digital Playground in Island 2010~2013 (Director). His research is focused on Digital Image Processing. Hyunyi Jung received her B.A. degree in Dance from Rutgers, the state University of New Jersey New Brunswick, NJ, USA. She received M.S. Degree and is pursuing Ph.D Degree in Art & Technology at Graduate School of Advanced Imaging Science, Multimedia & Film in Chung-Ang University. She engages in the interface between movement and technology, and carries out various activities in order to apply new possibilities rising from the integration of different media to exhibition and area of creation using such media. Her research focuses on media performance and interactive art. Copyright c 2016 SERSC 277
Sanghyun Seo received his B.S. Degree in Computer Science and Engineering from Chung-Ang University in Seoul, Rep. of Korea, in 1998 and M.S. and Ph.D. degrees in GSAIM Dep. at Chung-Ang University in 2000 and 2010 respectively. He was a senior researcher at G-Inno System from 2002 to 2005. He was the postdoctoral researcher at Chung-Ang University, in 2010 and the post-doctoral researcher at LIRIS Lab, Lyon 1 University from Feb. 2011 to Feb 2013. Now, he is working at the ETRI. His research interests are in the area of computer graphics and non-photorealistic rendering, 3D GIS system and game technology. 278 Copyright c 2016 SERSC