Adaptive Projection Displays: an interactive art piece to explore a low cost system for public interactivity. A Thesis. Submitted to the Faculty

Transcription

1 Adaptive Projection Displays: an interactive art piece to explore a low cost system for public interactivity A Thesis Submitted to the Faculty of Drexel University by Jenna Dundas in partial fulfillment of the requirements for the degree of Master of Science in Digital Media May 2014

2 c Copyright 2014 Jenna Dundas. All Rights Reserved.

3 ii Dedications This thesis is dedicated to my mother and father, who constantly fought for me, and never stopped believing in me, even when everyone was telling you too. Without your constant support, I would not be where I am today. Thank you for teaching me that I am not defined by dyslexia and that I could accomplish anything I set my mind too. I would additionally like to dedicate this to the instrumental teachers in my life that allowed me to reach for the stars especially Beth Tormey, Mrs. Barchey, and Lauren Sterenczak. As teachers, you gave me the tools to succeed, and the confidence to take on any life throws at me.

4 iii Acknowledgments I would like to thank my advisor Michael Wagner, and committee Ted Artz and Youngmoo Kim for all your support and guidance throughout the project. Additionally, I would like to Jeremy Fernsler, Jervis Thompson, John Beton, and Stefan Rank. Thank you for all your help when I was stubborn and pinheaded. Additionally, I would like to thank Jason Kirk, Justin Patterson, Chester Cunanan, and Dan Newman for being an excellent sounding board for ideas and answering dumb questions about life. A large thank you, to the rest of the graduate students in the Digital Media program. Without you all, I would have lost my mind. The last thank you, is to my friends and family for their constant support and guidance throughout the years. Thank you for not slapping me when I continuously complained about school for hours on end.

5 iv Table of Contents LIST OF FIGURES... v Abstract... vi 1. Introduction Background Public Displays Interactive Large Scale Public Displays Interactive Kiosks Related Work Digital Displays that Utilize Tracking Technology Design of the Interactive Art Piece Goals of the Interactive Piece The System Requirements for the Interactive Art Piece Tracking Hardware Graphics Engine Scripts Display Unit System Communication User Experience Applications Retail Advertisements Digital Reconstructions Comparison to Existing Solutions Limitation Results Artist s Statement Future Work Multiple People Evaluation of Effectiveness Additional Interactive Elements Three-Dimensional Projection Mapping Accuracy of the Tracking System Conclusion Bibliography... 43

6 v List of Figures 2.1 SMSlingshot input device and display out in the public People interacting with the researchers display FRAGWRAP system of projecting visuals on fragrance bubbles stookie statue and system structure SnowGlobe fishtank VR Johnny Lee s Setup for 3D head tracking using the Wii Remote and sensor bar Components of the Art Piece Web camera test Unity Scene visuals placed throughout for depth PointMan within Unity3D Space, shows the position of the viewers joints Custom script on the Unity camera Communication of the interactive art piece design Sample storefront display Sample advertisement display Sample situation display that explored depth within a window... 31

7 vi Abstract Adaptive Projection Displays: a low cost system for public interactivity Jenna Dundas Advisor: Michael Wagner, Ph.D Committee: Ted Artz, and Youngmoo Kim, Ph.D Interactive digital public displays that track viewer s position are currently inaccessible to the average consumer. Many tracking systems available on the market are prohibitively expensive and are out of scope for small business owners to purchase. This research tests various consumer level tracking technologies to ascertain whether a system can be developed in a low cost and accessible manner. Microsoft s Kinect in tandem with Unity3D offers a system that is straightforward to use and allows for ease of implementation. The resulting technique can be quickly carried out to create an interactive digital public display.

8

9 1 1. Introduction The world is growing increasingly interactive, and displays that were once analog are becoming digital. Billboards are transitioning from paper to large-scale digital displays, which allows for multiple advertisements on these billboards. Within these new digital displays, the ability to transform the content displayed, is as easy as a press of a button [1]. As these surfaces transition from analog to digital, the prospect for interactive public displays becomes abundantly possible. These digital public displays, have the capability to develop beyond just images and words to create a lasting impression on the consumer. The development of content on these public display systems is evolving, however, developers have yet to exploit today s technology [2]. Digital public displays can become more than just digital billboards on the side of the road. These displays can be signs in storefronts, advertisements in transportation centers, and museum displays. Although public displays are making the transition from analog to digital displays, advertisers have not moved beyond using these display as more than digital posters [1]. The full potential of these digital public displays, is still an open area to explore. The ability to create new experiences and change the framework of existing public displays can be achieved with the growing number of displays developing within outdoor and indoor spaces [3]. While the interactivity of digital public display systems can be explored further, there is a social norm for interactive systems. For example after the viewer notices that the public display is interactive, the display should react instantly to the user, to express usability [4]. With the increasing number of digital public displays growing, a low cost system can be created to track the viewer and create the illusion of depth and space. Interactive three-dimensional worlds can go unexplored because of the complexity of the

10 2 interaction that takes place [5]. Although Cursan was talking about the interaction between digital three-dimensional object and a user, this problem manifests itself within the public display system. Previously, the ability to use three-dimensional cameras was only for companies that had a large budget. However with Microsoft s introduction of the Kinect, the ability for someone to implement three-dimensional cameras grew higher than ever before. As a result of the availability of three-dimensional cameras a low cost system can be built, that allows small business and consumers to create interactive threedimensional public displays. The system uses a combination of off the shelf technology and a low cost graphics engine to create an easily implemented interactive system. The system, will allow digital producers to create displays that utilize threedimensional graphics, light, and shadow to create the illusion of depth and space. Through this system, the three-dimensional camera will track the viewer s position and update the visuals depending on where the viewer is in relation to the display.

11 3 2. Background 2.1 Public Displays Public displays have been transforming and informing consumers for many years. These displays take the form of banners in storefronts, advertisements on public transportation, billboards, and even the side of buildings. The content on these displays can range from artistic pursuits, advertisements, or informational content. These public displays, traditionally have the content printed on them through various techniques. However, the ability to easily change these displays is minimal. To alter the content on these printed displays one must print another display and replace the old content with new, this can take time depending on a number of different factors. Large-scale digital public displays are growing more popular within the public environment [6]. With these digital displays becoming more popular form for content, than the traditional method, public displays are undergoing a transition content can be delivered on the displays. As a result of displays transitioning from analog displays to digital displays, the experience to transform the content on the display allows for a number of possibilities [1]. Additionally, with current technology the ability to transform static display systems to more interactive multifunctional displays can be forged [7]. Although digital displays allow for a completely new framework for public displays, they have been relying on the old structure. These digital public displays can range from large-scale media facades to kiosks, often the interaction between these two can vary [8]. So with the wide range of situations digital public displays can encompass, the higher need for new user experience for the viewer. This older analog structure borrows the concept of posters but transitions it to digital. This analog design of

12 4 public displays explores the ability to transition different content on the displays easily. However, these displays only display content in the form of digital poster displays. The digital nature of these displays allows for more creative frameworks because they have the ability to present information in an engaging experience. These digital displays are growing in numbers and becoming apart of consumer s daily lives, especially in urban environments [9]. Additionally, the future holds a world in which any surface will become a digital display, which can transform the world, and bring worlds out from homes and offices into the public life [10] With the growing number of these displays, interactive concepts for these displays can be constructed. Not only are these displays transitioning from analog to digital platforms, technology is becoming increasingly less expensive and available to the public and small businesses. Things like computers, monitors, digital cameras, and tracking technology are available to the masses, rather than to a select few. With the price of these technological devices increasingly dropping and the increasing number of digital public displays in today s environmental landscape, one can increase interactivity of the displays. Researchers are bringing these visual and interactive displays outside the labs, and into the public, ranging from museums, libraries, plazas, and building facades [8]. Although this transition is happening, the exploration of a low cost interactive system, which exploits consumer-based technology to explore depth and dimension, needs be explored further. The interactive system can utilize different technologies and allow for multiple interactions by utilizing existing software and hardware. Although researchers have explored this topic extensively, an easily accessible framework for a low cost consumer solution for interactive public displays, which exploits threedimensional spaces, has yet to be explored.

13 Interactive Large Scale Public Displays With the nature of public displays growing in numbers, researchers have begun to look into the creation of interactive displays, and their role in the environments that they occupy. From large displays that cover buildings, to small interactive screens that give information to the viewers. For example, within the urban environment the onset of these digital displays has brought to light the unique potential for a different user experience for these interactive or informative displays [11]. Although these interactive displays have been researched, and methods have been created for the use within the public, the ability for anyone to create these experiences are limited by their budget and ability to hire various engineers to create a system for them. Although these displays help show how our environments can become more interactive, and allow for new connections to spaces and people through these displays, the ability to use one within a store or environment is limited by budget and experience. Interactive public displays that are large scale can transform buildings from standard to magical, and allow for viewers to alter, interact, and experience something new. However, not all building facades, which have been transformed through media are interactive, some might transform a building, a space, or a wall in for the form of an advertisement [12]. These transformations could be large-scale projection mapping projects, banners, or posters. Although these displays tend to not be interactive, researchers created a system to explore large-scale media interaction on the facades of buildings. This project was called SMSlingshot, researchers projected digital images onto the side of the building s facade. The projected visuals, transformed the standard building, into a large interactive display. With this system, people would sling messages onto the side of the building with a custom input slingshot device, which would allow the viewers to influence the visuals that occupied the building [12]. The display researchers built reacted to the custom input device, which allowed viewers to

14 6 Figure 2.1: SMSlingshot input device and display out in the public take the input device and add messages to the buildings by shooting them onto the media facade (Figure 2.1 shows the setup, with custom input device). Through this public display project, the researchers found that the users enjoyed having authority over the public display and being able to contribute to the visuals [12]. Conveying interactivity of the system can be one of the more troubled aspects of interactive public displays. Researchers found that an interactive system that utilized the visualization of the mirror image of the observer, was registered by the observer as interactive display faster than a call to action accomplished [4]. The researchers developed a system that used Microsoft s Kinect to track the viewers, which then created the shadow of the viewer within the displays visuals. With this system, the researchers found that if an interactive system conveys interactivity the system must instantly communicate with the viewer, or the system will fail [4]. Public interactive systems have the potential to be engaging and fun. However, these systems need to take into account how they are calling the viewers to communicate with the system. As a result, systems that communicate as soon as someone comes into contact with him or her, are the most effective and attention grabbing. The focus of this research was mainly on digital representations of the viewers, and not on creating three-dimensional worlds that are affected by the viewer and how the viewer

15 7 might perceive this. To explore the ever-increasing low-cost technology within the technology market, researchers focused on creating an interactive system that allowed many participants to interact with the system. The researchers also wanted to create a system that would use off the shelf technology, to allow anyone to build their end product. Through their exploration, the researchers used three easily accessible pieces of technology, which anyone can find or purchase. The researchers wanted the set up to be simple so that someone could easily replicate, and not engineer a complex gaming console, which resulted in their decision to use a camera, a projector, and computer [13]. With the technology, the researchers used multiple-human tracking algorithms to create an experience that predicts the future steps of various people, within the given camera area [13]. Through their system, the ability to create a low-cost system for public interaction can be built. However, their system only details the creation of a system that handles two-dimensional images, mainly the projection of future footsteps Interactive Kiosks Digital public displays can go beyond transforming buildings with visuals and interactive elements. These digital public displays can range from large-scale media facades to kiosks, often the interaction between these two can vary [8]. These visual or informative displays can take interactivity elements and transform, into multiple directions to further mediate the world. For example, kiosks can allow for interactive aspects within their specific system, these elements can help inform how to purchase a product or give you information about an event. These kiosks can be touch screen based displays, which give information in different methods to the consumers. These interactive devices can be used for ticket sales or displaying informational content. However, these devices take money and research and development and are only used

16 8 Figure 2.2: People interacting with the researchers display by large companies that have the resources to build them and incorporate them into the public. Additionally their interactive elements tend to be minimal, and only allow for a select set of tasks for the users, which have been pre-designed for specific tasks. These specific tasks can be, for example, buying train tickets or getting information about a flight. They do not allow for someone to purchase one and apply his or her own business or personal needs to his or her structure. To explore how an interactive display handles within the real world environment, researchers deployed a number of interactive displays throughout different cities. Through these interactive displays, the researchers created an interactive system that would react to the user when a camera detected when the user was near the system [7]. Through this feature, the system display would go from static to interactive to communicate to the user the ability to use the system to gain information from (see Figure 2.2). The reasoning behind this research, was to explore how the user would understand that the system was interactive and how they interacted with the display. 2.2 Related Work There lies a number of different creative and technical projects that explore technology that has built exclusively for the researchers needs, while others utilize con-

17 9 Figure 2.3: FRAGWRAP system of projecting visuals on fragrance bubbles sumer products. Through the exploitation of these existing projects, aspects were borrowed to make up the resulting interactive art piece for this project Digital Displays that Utilize Tracking Technology To explore the idea of mapping scent to objects, researchers built a system called FRAGWRAP. The system they devised required that they add scent to a real object and project visuals that related to the scent within the object. The system needed to utilize both visuals and scent to further push the sense of smell within the human senses, because the use of images help aid the recognition of a particular sent (see Figure 2.3) [14].Through their exploration the researchers devised the concept of scent mapping, which maps scent and visual information to further push the recognition of scent within the human senses [14]. The researchers developed a method of tracking the position of the bubble and updating the projection mapped image to accompany the bubbles movement. By using the Kinect from Microsoft, they were able to utilize the RGB and depth camera to project images on the bubble and modify them in real-

18 10 Figure 2.4: stookie statue and system structure time [14], and Unity3D their process for the generation of the real-time graphics. In addition to the Kinect system, they used Processing to implement form and location control [14]. Through the FRAGWRAP system, one can see the ability to track objects and update visual information on the objects. Stookie is an interactive animated artwork, that used three-dimensional projection mapping techniques and tracking technology to create an interactive art experience [15]. Researchers on this project produced a head statue, which was built in a neutral facial position so that it does not convey any emotions to the viewers experiencing it. The neutral position of the head statues gave the freedom to the researchers to exploit three-dimensional projection mapping techniques to transform the statue, which altered facial expressions and eye movement of the statues [15]. They updated the statue s eyes depending on the location of an audience member through the exploitation of Microsoft s Kinect. The researchers used Unity3D for their graphics engine to update the visuals. Through use of the Microsoft s Kinect and Unity3D, the researchers were able to create an interactive experience that also employed both viewer

19 11 Figure 2.5: SnowGlobe fishtank VR tracking and three-dimensional mapped techniques (Figure 2.4).The only interactive aspects of this project were the movement of the eyes, which followed the movement of one audience member. Additionally custom-built animation were used to express facial emotions and update the statue, which was accomplished through preplanned situations outlined by the researchers. SnowGlobe is a spherical display, that focused on creating an interaction with three-dimensional objects [16]. Through this display researchers successfully tracked the head of the viewer using a Vicon motion capture device, this data was used to maintain motion parallax of the three-dimensional object in the display [16]. To track the head of the viewer researchers created custom glasses, which had three reflective infrared markers to track the location of the head [16]. Through the projection system, they were able to exploit the tracked data and use it to create the parallax in the three-dimensional image. Additionally within this system the user had the ability to interact with the three-dimensional model that was the display on the spherical surfaces (see Figure 2.5). Although this system did use three-dimensional projectors

20 12 researchers focused on creating a digital object that provided the ability for one to completely walk around it. The projection display was a spherical object that reflected a projected data off the hemispherical mirror, this allowed for a seamless curvilinear display surface [16]. Through this experiment in tracking within the spherical display, it shows tracking technology can be incorporated within a public interactive system, which utilizes light and shadow to create the illusion of depth and space. The idea of tracking a viewer and updating the computer-generated visuals has been explored using a number of different products. However, most of these products have been exploring this tracking technology within a game setting, usually using visually similar scenes. This visual scene tends to be a box with a grid pattern on it and targets placed at various points throughout the scene. One can find these videos easily throughout YouTube or any other search engine. However as awesome as these projects can be, they lack elements to make them applicable throughout one s daily routine. An example of a project that explored tracking technology and computer generated visuals, is researcher in human-computer interaction, Johnny Lee. Using the Wii Remote and sensor bar he created a system that tracked the viewer s head position to create the appropriate visual parallax [17]. The system that he created reverses the Wii Remote setup, by placing the remote by the display screen and placing the Wii sensor bar or infrared lights on a hat that is placed on the viewer s head [17]. The reverse set up allows the Wii Remote to send out infrared light, that the Wii sensor finds (system setup is Figure 2.6). Lee s tracking method finds the position of the viewer depending on their location within space [17]. The system, however, requires viewers to wear a hat or something on their head that would make this ideal for gaming situations, however, it could never be used within a public venue. The process is invasive because the viewer

21 13 Figure 2.6: Johnny Lee s Setup for 3D head tracking using the Wii Remote and sensor bar would have to place something on their head to experience it, rather than walking by and experiencing the interactivity instantly. Johnny Lee s system does not allow for instant interactivity, and a viewer cannot walk up to the system and instantly communicate with the system, rather they have to decide to communicate with it. The exploration of existing interactive displays and viewer tracking projects yields the possibilities of low-cost interactive system that explores three-dimensional graphics, light and shadow, and the viewers position to achieve depth and dimension. Through these previous projects that explored different technologies like Wii Remotes, Microsoft s Kinect, Untiy3D, and webcams, a design for an interactive art piece that explores tracking technology can be actualized. Through this interactive art piece the design of the system will foster a low budget. The low budget will allow for anyone to create and build anything he or she sees fit for his or her own digital public display needs. With digital public displays growing in numbers, all around the public environment, a low cost method that allows designers to create displays will allow for new interactive possibilities.

22 14 3. Design of the Interactive Art Piece Currently, there is software and hardware that tracks people, based on a number of different factors. Through the exploitation of existing software and hardware, the system can be devised that will update the projected artwork and create the illusion of depth and space to the viewer. By utilizing off the shelf technology, the formation of an interactive art piece that explores depth and dimension can be brought to fruition. The concluding interactive art piece can become a low cost responsive interactive public display system. The system will be built so that the visuals adjust depending on the viewer s position to the display. The user s experience to the piece will change depending on their position in relation to the display. Through the use of digital images, the art piece can update imagery depending on the viewer s location to simulate the illusion of visual depth for the viewer and space. The art piece will give the illusion that the digital imagery projected is behind the surface structure, or lying above the surface. By utilizing off the shelf technology, the visuals within this art piece can transform the displays into new worlds that the viewers can experience. 3.1 Goals of the Interactive Piece Through this interactive art piece, the goal was to explore the idea of depth and space within public displays, and how low cost system could be devised to allow for the masses to utilize tracking technology and digital displays. The process outlined allows for a higher degree of freedom for digital designers and allows them to create interactive digital public displays with depth, space, light, and shadow without having to create a way to render the graphics in real time. Unity3D gives the ability to the artist to change and alter images and the reactivity of the system without having to

23 15 explore the development of a graphics environment. 3.2 The System Requirements for the Interactive Art Piece The proposed interactive art piece can be operational, utilizing four different components. Each of the four components of the system is easily accessible to the average consumer, and fosters a low budget. The four components are tracking hardware, a graphics engine, scripts, and a display unit (Figure 3.1). To create an interactive art piece with a low budget in mind, the proposed system respectively used Microsoft s Kinect, Unity3D, Microsoft s software development kit, free script package called Kinect with MS-SDK, a few custom scripts, and a projector. Microsoft s Kinect has dropped the price of the original device and can be purchased for under a hundred US dollars. Unity3D offers a free version of their software, which can be downloaded through their website. Although the software company does offer a free version of their software; there are a number of limited functions to the engine. Features like light mapping, depth of field, fog, and other image effects are not offered in the free version. However the free version does still function for the end designer to utilize, and used to create an interactive art piece with its own set of qualities. The final cost to gather these elements for the interactive art piece can vary. Things that factor into the overall price are the type and quality projection display system is employed. However, the cost can slide between 400 dollars to 1000 dollars Tracking Hardware To form the interactive art piece, research went into a number of different consumer tracking products. Once a few options were selected, tests and evaluation of the technology yielded the end product. These products were web cameras, Microsoft s

24 16 Figure 3.1: Components of the Art Piece Kinect, and Vicon s Motion Capture cameras. The three devices were evaluated on accessibility, volume of tracking area, and portability. Through the exploration of these products, the leading product in the three categories was implemented into the interactive art piece. Web Cameras First on the list were web cameras. They have been growing in numbers throughout the last few years. It is hard to purchase a laptop or computer that does not have a camera built inside of it. As a result of web cameras omnipresence in today s modern world, the technology was a worthy option, and needed to be explored further. The next step was testing to see how web cameras held up for the tracking needs of the interactive art piece, after being deemed webcamera accessible and portable. The results of the tracking capabilities of web cameras were better than initially expected. The exploration of this product used FaceAPI and Unity to track the viewer and to

25 17 Figure 3.2: Web camera test update the visuals in relation to the viewer (Figure 3.2). With the web camera, the results were surprisingly satisfactory in a few aspects. However one problem this technology encountered was the limited tracking area. The viewer of the piece would have to be in a narrow space less than a foot away from the web camera. Stepping out of this range resulted in the system s graphics becoming jumpy and jittery. Although featuring a minimal tracking area in comparison to other tracking hardware, the web camera is cheap and portable. Ultimately the short tracking area of the webcam is its downfall. This downfall is the very small tracking area for a web camera, which raises one question. This question is does employing a higher quality web camera, achieve better results?

26 18 Vicon Motion Capture Cameras Vicon s motion capture cameras are one of the best three-dimensional cameras currently on the market today. However, this product is expensive and not an accessible product to the public, or the average consumer, or small business. As a result of the price and accessibility the Vicon s camera, their product was not an option for a low-cost interactive system. Microsoft s Kinect Within the last few year s one consumer product has revolutionized gaming, interactivity, and interfaces around the world. Microsoft s Kinect is one of the first consumer level three-dimensional cameras that use infrared light and sensors to gain the location of the viewer and several other positional data from them. This camera is not only affordable to consumers and small businesses; it is easily accessible and reasonable in price. The ability for one, to purchase one of these products, is a straightforward process. Microsoft s Kinect can hang, place, or move only and fast in comparison to other three-dimensional cameras. They are lightweight, small, and compact. As a result of this, the Kinect can easily be packed and transported to different locations. Additionally this camera gives a larger tracking volume than the area a webcam could achieve. As a result of these factors the Microsoft s Kinect was the choice product to use within this system. The end art piece s tracks the user using Microsoft s Kinect infrared emitter, and sensor. The ability to track a viewer without placing an object on their person can be achieved through the Kinect Technology. What this knowledge means is that the viewer does not have to decide to interact with the piece, by placing an object on their head, markers on their body, or any object they have to choose to wear to experience the piece. With this art piece the desire was to have instantaneous reactions to the

27 19 movement of the viewer, this way the design does not require anything from the viewer to experience the piece. This instant reaction is vital to the communication to the viewer of the systems interactivity, without this the system fails to communicate properly to the viewer. Microsoft s Kinect, uses the concept of structured light by using a speckled light pattern of infrared light. Through this process, the light bounces off the viewer to locate the position of their body in relation to the Kinect. Once the position of the viewer is found through the Kinect. The skeleton data is brought into Unity3D using Microsoft s software development kit and a script package that bridges the software development kit and Unity3D. Once this data is brought into the game engine, a custom script is applied to the camera. This script takes the place of the viewer s center chest, and updates whenever the viewer moves in relationship to the visuals. The viewer can move forward, backward, right, and left and the visuals will update with appropriate parallax depending on their position. Additionally another addition is the resetting of the scene once the viewer has left the volume of the Kinect tracking capabilities Graphics Engine Through this interactive art piece, the goal was to create a design that could be easily replicated for anyone to incorporate within his or her own particular projects. As a result of this the project focused on finding an existing method for graphics, rather than trying to write a custom graphics engine. With these aspects in mind, one option stood out. This graphics engine is called Unity3D, which is commonly used as a gaming engine. However, the program is robust and allows for the convergence of different projects and ideas that go beyond a gaming environment. Through this product, the ability to create real-time graphics became straightforward, and simple

28 20 Figure 3.3: Unity Scene visuals placed throughout for depth to be incorporated into an interactive art piece (Figure 3.3). The exploitation of this existing product allows for a designer to utilize existing aspects of the program. For example, the ability to light the scene and use different rendering techniques have been created already, and can be easily incorporated to create a higher quality image than one might be able to do when working on their own person graphics engine product. One of the downfalls of this graphics engine is that some results are only available within the professional version of the product. However this aspect can also be one of the great things about this product is that there is a completely free version for anyone to download and explore the technology. Essentially the free version of the engine is the same as the professional version, just with a handful of features not included. Included in the professional version are features like fog, rendering effects, shadows, and image results are at the designer hand. However within the free version the designer can only use hard shadows within

29 21 a directional light, which gives the designer some freedom. However, this freedom does not compare to the flexibility to create effects within the professional version. Through the use of Unity3D as the graphics engine, a designer can easily save out scenes within Maya as FBX s and import the finished graphics into a Unity3D scene. Additionally the designer can then explore in the features of Unity for rendering and add in animations in a very quick period. The development process is quick and simplified by using this graphic engine, rather than using a custom built solution, because there are pre built features in the program Scripts There is a small setup one needs to do to start working with Microsoft s Kinect within unity3d. Within this interactive art piece, the project relied on a few different components. These components are Microsoft software development kit (SDK), the script that bridges communication between the SDK and Unity3D, and a custom script. Microsoft s Kinect Software Development Kit Through Microsoft s website, they offer a software development kit also know as an SDK for free download on their website. This SDK is only for windows based machines, and uses C++, C#, or visual basic to create Kinect based applications. Through this SDK, the data from the Kinect is brought to the computer. Additionally through this SDK the drivers for the device are installed, this way the Kinect can communicate to the computer.

30 22 Figure 3.4: PointMan within Unity3D Space, shows the position of the viewers joints Kinect with MS-SDK Through the Unity asset store, there is a free package that bridges the Microsoft SDK to Unity3D. This way someone can use the skeletal data, depth data, gesture, and other information the Kinect offers to explore. This method was the best way to bring this data into Unity in comparison to other methods, and other various scripts available on the Internet. Through this process, there are a number of scenes that explore different aspects of data found from the Kinect. Through this interactive art piece, the use of the PointMan script was the main source of information for the position of the viewer (Figure 3.4 shows the PointMan within the Unity3D scene). The PointMan script is moving objects depending on the viewer s general place of their joints. These joints include head, shoulder center, shoulder left, shoulder right, elbow left, elbow right, wrist left, wrist right, hand left, hand right, spine, the hip center, hip left, hip right, knee left, knee right, ankle left, ankle right, foot left, foot

31 23 right. The interactive art piece uses this script by placing objects that have their render attribute turned off into the joints specified by the script. By doing this we can get the general position of the viewer within the Unity world. Once the PointMan is in the scene, one can then add a camera into the scene. The camera position should be slightly above where the center shoulder joint would be; this is the general head position of the viewer. Custom Script Although Microsoft s SDK and the Unity3D extension allows for skeletal data to be brought into the graphics engine, however there is still one last step. The last step is a small script that is applied to the camera within a given Unity3D scene, which updates the camera position depending on the viewer s position. What this script does is take the position of the placed PointMan center shoulder joint, and translates this position to the camera position. Through this information the camera gets x, y, and z values for its place within a given scene. By giving the camera x, y, and z positions this allows the viewer to move left, right, forward, backward, up, and down with the visuals updating accordingly. Additionally to the giving the camera movement through the body position of the viewer, the script can increase or decrease the amount of movement the camera does is response to the viewer s movement. This allows for a designer to decide how much or how little their designs will react to the viewers of their final designs. This script also gives the designer the option to put the distance the Kinect is in front or behind the display screen, and have their visuals be pushed back in space depending on the inches the give. To accomplish this, the designer must create their visual assets and then combine them into a unity prefab, and place it within their resources folder. Once the prefab has been created the designer can drag the prefab

32 24 Figure 3.5: Custom script on the Unity camera into the given spot within the unity script (see Figure 3.5). The script takes the inches from the screen supplied by the design and converts it to meters that are the starting unit of the graphics engine. After it converts the inches to meters it places the artwork at that given spot in the three-dimensional space Display Unit The display unit is the finish line of the whole interactive process. This part of an interactive art piece is where the visuals are displayed and all the work the design work artist has done can be visualized. Ideally, the process outlined through this interactive art piece would employ rear projection for the finished visuals that the designers created. Rear projects, allow for large displays and are a favorable way to mitigate occlusion from the viewer, on the display graphics. However, rear projection can be outside the projects goals. It can be space intensive, and requires extra setup time for the display. With the time and money in consideration, the end system is not dependent on the use to rear projection. The interactive display can be used any form of display, one would like to use or has access to. For example, a monitor, or television, or anything one can find to use. This way they can use smaller space or materials that they have access too, through their personal connects or business.

33 System Communication Outlined in the previous section are the components that make up the interactive art piece. These components loosely are Microsoft s Kinect, Unity3D, scripts, and a display unit. Through these components, there is a link to each to give the appropriate information to achieve the interactive art piece (see Figure 3.6 to see the flow of communication). The technology in the Microsoft s Kinect allows for the use of infrared technology to find the position of the viewer of the display s graphics. Through this technology, the position of the viewer is continuously sent to Unity3D the graphics engine, through the use of Microsoft SDK and a Unity package called Kinect with MS-SDK. Through these two components, the position of the viewer is brought into Unity3D. The unity package can be seen working through the use of the PointMan script in the Kinect Unity3D package. After this step, a custom script is placed on a camera that moves the camera with the position of the viewer. This information allows for x, y, and z movement. What this means for the viewer is that they can move left, right, forward, backward, up, and down within the Kinect range and the visuals will update depending on their position. After the location of the viewer is found within Unity, and the camera takes the position of the viewer s location, the graphics can be added into the Unity3D scene. Through these graphics, the design can utilize a combination of three-dimensional images and two-dimensional images to create their interactive art piece. Within the scene, the design can explore three-dimensional space and see how the movement between objects far and near from the camera can create depth and space. The more variation the designer adds into their scene with the scene depth, the more effective the illusion will appear in the end design. Additionally through the custom script, the design can increase the amount of camera movement or decrease it depending on

34 26 Figure 3.6: Communication of the interactive art piece design their own design goals of their interactive piece. Once the plan is finalized and the camera is set up within the scene, the design can be exported from Unity3D. Through this step, one will get the finalized interactive piece, which can be delivered to any computer with the Microsoft s Kinect SDK installed on it. By doing this, one can run the final graphics and output them to the display unit. The display unit can be a projector, a monitor, a television, or any type of display one has access to for their display. Through this the interactive art piece, a process for fast graphical development and low cost technology allows anyone to access it, and explore within their own needs. The graphics allow for the viewer to instantly interact with the system, and give them the authority over the graphics. This piece, can explore how the illusion of depth and space can be brought into public displays, to create a new method for displaying information. 3.4 User Experience The user experience is an essential aspect of this interactive art piece. Without thinking about how a user will experience the visual designs, the system would be unable to properly convey interactivity. Outlined through the background section

35 27 the importance of instant reactivity to the users and giving them authority over the display fosters use from passive observers. These two concepts were heavy motivators within the design of the art piece s user experience. The end interactive piece reacts instantly to a viewer entering into the tracking area. This reaction to the viewer means that they can be facing the Kinect, sideways to the Kinect, or their back can be to the Kinect. Through this detail, the system recognizes the viewer and takes their position and updates the camera accordingly. The system also does not need for the viewer to stand in front of the Kinect in a T or A pose to register them within the system. By having the system that reacts the viewer without asking them for anything the system will be more successful at conveying interactivity than it would be if it asked for the user to decide to communicate with it. This communication means that if the system were to ask the viewer to interact with it, then this would cause a delay, which would result in the viewer probably not interacting with the system. The system additionally provides the viewer authority over the displays and goes beyond the idea of solely reacting instantly to the viewer. This response means they are controlling the movement, and this gives them a feeling of authority over the display s visuals. With the interactive art piece as a public display, the way the system handles people is crucial. The system can only track and update the visuals based on the site of one viewer. However, through this limitation if a group of people is walking by the system, interactive art piece will lock onto one of the members of the group. Once this viewer leaves the tracking area of the Kinect the system will lock onto the next person it finds. The display will not get confused if there are multiple people within the system, because it finds one and locks on to them, rather than jumping around to everyone. Through this experience, the user can experience the visuals and

36 28 leave and allow the next person to experience the display. Additionally the ability for multiple people to be in the Kinect tracking section allows for people to interact with the display without having a specific area devoted just for that one specific person. 3.5 Applications With digital displays increasing in numbers throughout one s everyday life, the ability to create interactive displays is growing with the amount of displays transitioning from analog to digital. The process created for this interactive art piece can express how digital public displays can utilize existing technology to create the illusion of depth and space. The set-up of this art piece explores the use of light, motion, and space to create the illusion of depth where there is none in the actual display. Tracking the viewers location and updating visuals depending on their position within the space in relation to the display system create this illusion. The resulting interactive art piece can be utilized in three specific areas retail, advertisements, and historical reconstructions Retail The resulting idea can be applied to retail conditions in two different manners. The first manner in which this process of creating an interactive display can be used to create a look and feel for the store logo, within their storefront. This interactive art can take the form of a logo in front of visuals that match the store s design image. Through this design, the logo can be in front of the graphics and appear as though it is jumping off the front of the digital display setting (see Figure 3.7). Secondly retail can utilize this process by allowing people to visualize the product within various different conditions. For example, a consumer can have a picture of their living room and place a particular product they are thinking of purchasing

37 29 Figure 3.7: Sample storefront display within this three-dimensional world. Unlike other visualizations, the consumer can experience the product on a visual and spatial level. Through this system, the graphics can promote a product, an image, or a feeling that the designer is trying to convey to the users Advertisements Throughout one s day, you encounter a number of different advertisements from digital ads on one s computer to ads on the street or transportation system. Through utilizing this tracking framework for ads information can pop out to the consumer that the marketer would like to emphasize, rather than relying solely on visual structure of the graphic design to push their marketing message to the consumer. Through the use of the three-dimensional effect brand names and core information can be forward in the space emphasis importance to the viewer, where the less important information

38 30 Figure 3.8: Sample advertisement display can be pushed visually back in space (Figure 3.8) Digital Reconstructions Through the formation of this interactive art piece, the use of the process can be applied to historical digital reconstructions. An interactive piece can be utilized within the exhibit in diorama display manner to allow visitors to view worlds in which they would never be able to experience. The advantages to the digital version of this versus the recreation of the situation in real life is the ability to the museum to make updates and changes to situations that might change as new information arises about the history of the environment or artifact.

39 31 Figure 3.9: Sample situation display that explored depth within a window 3.6 Comparison to Existing Solutions Through this art piece, one can take the work done by computer scientists and apply them to real world situations, with an aesthetically appealing graphics. The concepts of design have been applied to the interactive art piece to push the visual boundaries of the graphics to push depth and dimension in the system (Figure 3.9). Design features like lighting and shadow and saturation have the ability to push the graphics and other scenes can be used beyond the classic target and box scene seen through various different channels on the Internet. Additionally through this process someone can create an interactive display that reacts to the viewer s position by updating the perspective of the graphics without having to build a graphics engine or place an object on the viewers person. Through the exploitation of these existing solutions and marrying them together, a low cost system can be born that allows someone with little technical background create the