TViews: An Extensible Architecture for Multiuser Digital Media Tables

Transcription

1 TViews: An Extensible Architecture for Multiuser Digital Media Tables Ali Mazalek Georgia Institute of Technology Matthew Reynolds ThingMagic Glorianna Davenport Massachusetts Institute of Technology In our increasingly digital age, the development of digital tabletop systems has the potential to extend traditional tabletop activities into the digital realm, shifting human computer interactions away from today s dominant desktop-based paradigm to a broader one of shared physical contexts in everyday spaces such as home living rooms, school classrooms, and workplace conference rooms, as well as public spaces such as cafés, shops, or museums. A media table is a general-purpose digital tabletop that can support a broad range of digital media applications, such as media asset management, story construction, multimedia learning, and digital game play, as well as many other applications that the application developer community will find. We are especially interested in the development of enabling technologies both hardware and software that will allow media tables to be economically viable, easy to use, and sufficiently general purpose for widespread user adoption. At its most basic level, a media table provides a horizontal interaction surface upon which the spatial configuration of tagged objects can be computationally interpreted and then subsequently augmented with coincident visual output. In existing digital tabletop prototypes, visual output is usually provided by rear or front projection, but the rapid development of large plasma and LCD displays at consumer price levels makes these display technologies increasingly attractive options for media tables. While some interactive table displays make use of touch as a means of input, rather than object tracking, the additional benefit provided by object tracking is the ability to identify and associate unique physical artifacts with different elements or functions within the application space. These physical artifacts can take on a form that is representative of their digital functionality, thus allowing an off-loading of some of the information within the interactive environment from a purely graphical form to a hybrid graphical and physical-world representation. For the TViews project, we have created a new, more extensible method and sensing framework for object tracking media tables that can manage large numbers of tracked objects while running many different media applications at once see the Previous Work sidebar for a discussion of other approaches. We have focused on the problem of tracking tangible interaction objects above a large LCD or plasma display, and have therefore set out to develop an interaction object tracking and identification solution that works with a rear-illuminated display, thus avoiding the occlusion problems common to front-illuminated displays. Although the TViews system does not directly sense user touch, we have attempted to preserve the spirit of the multiple touch and identity properties of Diamond- Touch while applying these properties to a tangible user interface system akin to Sensetable see the sidebar for information on these systems. TViews design goals The TViews table is a media interaction platform designed for shared living spaces within the home environment. This platform consists of three parts: a dynamic object tracking and identification system that works with an extensible set of tagged objects on the table s surface, a planar display that is integrated into the table s surface, and a set of application programming interfaces (APIs) that tie these elements together. TViews applications can therefore provide visual output that coincides with the tagged objects movements as well as the actions signaled by the human manipulation of those objects. Figure 1 (on page 49) illustrates the physical configuration of the TViews table. TViews is an extensible method and acoustic-based sensing framework for creating digital media tables that overcome many limitations of singlepurpose systems. The system provides a means for real-time multiobject tracking on the tabletop and for the management of large numbers of objects and applications across multiple platform instances. Published by the IEEE Computer Society /06/$ IEEE IEEE Computer Graphics and Applications 47

2 Previous Work Early work on interactive tables began with research prototypes such as the Digital Desk, built at Xerox EuroPARC, and the ActiveDesk work at the University of Toronto. The Digital Desk used an ordinary physical desk with a video camera mounted above it that could track the movements of an LED-tipped pen to determine where the user was pointing. 1 Later versions included overhead digital projection of electronic objects onto regular paper documents. The ActiveDesk allowed twohanded input through physical artifacts called bricks and was used with a drawing application called GraspDraw. 2 The next major steps in interactive tabletops took place at the Massachusetts Institute of Technology Media Laboratory, with the metadesk and I/O Bulb projects. 3,4 These systems differed from the Digital Desk and ActiveDesk systems in that they were designed for networked collaborative use. Applications included geographical visualization, holographic setup simulation, and urban planning. Both systems achieved multiple objects tracking through computer vision and accomplished the display with projected graphics from the rear (metadesk) or the front (I/O Bulb). Today, researchers are exploring interactive tables for different applications in a variety of physical contexts. Notable examples include the DiamondTouch table from Mitsubishi Electric Research Laboratories, 5 and the Sensetable tangible user interface system from the MIT Media Lab. 6 Another type of table was constructed from a commercially available interactive wall display, a DViT from Smart Technologies (see com), and has been used in tabletop interfaces such as those discussed elsewhere. 7,8 The DiamondTouch system is a capacitive touch sensing system that overcomes some of the major limitations of existing touch screen technologies. In particular, DiamondTouch offers two important improvements over existing touch screens: it detects multiple simultaneous touches from multiple users and unambiguously reports both the touch location and the identity of the user touching each point. However, the requirement that each DiamondTouch user must be coupled to a unique receiver through a chair or floor plate is awkward because these specially equipped chairs or floor plates must be tethered to the DiamondTouch display, which in turn limits the free placement of the display in a room. The DiamondTouch system also requires an antenna grid that is the same size as the display, with a number of rows and columns dependent on the display size and the touch coordinate resolution. As the display grows in size, the size and cost of the antenna grid and driving electronics grows, too. Finally, the opaque antenna grid limits DiamondTouch to frontprojection systems, although the DiamondTouch developers suggest that a transparent antenna array might one day be feasible with special materials. The Sensetable system is based on Wacom s tablet and pen technology. The system tracks interaction objects above a display surface. While a typical Wacom tablet can track only two pens at a time, the Sensetable developers modified the Wacom system by adding a duty cycling approach to allow tracking of a greater number of objects at once. This duty cycling approach reduces the effective tracking update rate by a factor proportional to the number of objects being tracked. Thus, as more interaction objects are added, the effective update rate for each object decreases. This severely constrains the number of interaction objects that can be employed on the Sensetable surface. Further, because of the opaque printed circuit board used for the grid of excitation coils and associated circuitry, the Sensetable system can only be used in a front projection mode. In contrast to the electromagnetic approaches used in DiamondTouch and Sensetable, the SmartBoard system uses a vision-based approach for tracking touch, pens, and erasers above a display surface. Because of the SmartBoard s vision-based approach, there is no antenna array opacity problem to constrain the display to front or rear illumination. Furthermore, the system does not depend on any other apparatus the user does not have to be tethered to the SmartBoard. Unfortunately, the technology does not scale to track many objects on the display surface because objects occlude each other from the view of the SmartBoard cameras. Furthermore, the SmartBoard does not provide a way to uniquely identify each different object with a large identity space object color could be used to distinguish between a few objects but it cannot tell one hand from another, nor distinguish between tens or hundreds of objects on the display surface. References 1. P. Wellner, Interacting with Paper on the Digital Desk, Comm. ACM, vol. 36, no. 7, 1993, pp G. Fitzmaurice, H. Ishii, and W. Buxton, Bricks: Laying the Foundations for Graspable User Interfaces, Proc. Conf. Human Factors in Computing Systems (CHI), ACM Press, 1995, pp B. Ullmer and H. Ishii, The metadesk: Models and Prototypes for Tangible User Interfaces, Proc. Symp. User Interface Software and Technology (UIST), ACM Press, 1997, pp J. Underkoffler and H. Ishii, Urp: A Luminous-Tangible Workbench for Urban Planning and Design, Proc. Conf. Human Factors in Computing Systems (CHI), ACM Press, 1999, pp P. Dietz and D. Leigh, DiamondTouch: A Multi-User Touch Technology, Proc. 14th Ann. Symp. User Interface Software and Technology (UIST), ACM Press, 2001, pp J. Patten et al., Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces, Proc. Conf. Human Factors in Computing Systems (CHI), ACM Press, 2001, pp S.D. Scott, S. Carpendale, and S. Habelski, Storage Bins: Mobile Storage for Collaborative Tabletop Displays, IEEE Computer Graphics and Applications, vol. 25, no. 4, 2005, pp U. Hinrichs et al., Interface Currents: Supporting Fluent Collaboration on Tabletop Displays, Proc. 5th Int l Symp. Smart Graphics, Springer, pp September/October 2006

3 In our exploration of interactive object use on a tabletop display as a platform for shared media and story interactions, 1 we identified a number of requirements necessary for the development of an extensible media table architecture: scalability of the display and interaction space in size, extensibility of the object namespace to accommodate a virtually unlimited number of unique objects, portability of interactive objects between platforms, and extensibility of the application space and management of objects and applications. TViews uses a novel combination of acoustic sensing and infrared communication technologies coupled with distributed processing both in the interaction objects themselves as well as in the table s central processor to achieve these objectives. Here, we provide a quick summary of the combination of technical features that we used to fulfill these requirements in the TViews system. To allow scalability of the interaction surface in size, the TViews object-positioning technology must function in a manner independent of the size and scale of the interactive surface. We can construct TViews surfaces in smaller or larger versions and at different scales and aspect ratios to accommodate different display types. We based the TViews object-tracking technology on the acoustic time of flight of positioning signals transmitted through the glass display surface itself, and passively monitored by the interaction objects. Since this acoustic positioning system does not require antenna grids or other materials covering the interactive surface, it s possible to provide a coincident display without suffering from occlusion or other problems inherent in other technologies, regardless of the display surface s size or aspect ratio. Furthermore, by broadcasting positioning signals from the display surface to all interaction objects simultaneously, there is no inherent limit to the number of interaction objects that users can use simultaneously on the same surface, and the position update rate is independent of the number of interaction objects. Our initial TViews prototypes have shown 100- Hz update rates and millimeter scale position measurement accuracy, essentially invariant to the number of tracked objects. To provide an extensible object namespace, TViews uses a globally unique digital identification number for each interactive object. The current 64-bit number allows an extremely large object namespace that the system uses in a hierarchical fashion to permit object types to be differentiated through fields in the ID namespace. We could extend this ID space to a larger numbering space if necessary. Supporting portability of interactive objects from one table to another requires two properties. First, a TViews table must identify any object from the entire namespace as soon as someone places it on the interaction surface. Second, any TViews table, regardless of its size or aspect ratio, must determine the position of any object placed on its interaction surface. To support the first required property, we developed Interactive tagged object (puck) Glass panel and flat-screen display 1 Physical configuration and setup of the TViews media table. an enumeration strategy for the singulation and identification of all objects currently on a given table even when the number of objects and their identities are not known a priori. Second, the computational system within the table performs the position solution based on time-of-flight measurements, performing this computation on the objects themselves. This way, the objects don t need to know anything about the table size, and only need to measure the arrival times of acoustic pulses and communicate them to the table. To support an extensible set of applications, TViews provides an API layer based on an event model that sends events to TViews applications when objects are added, moved, manipulated, or removed from the table. Thus, TViews applications interact with the table platform through the standard expedient of registering for input events from the table s control system. TViews also provides an application and object manager to support multiple applications on the table and to keep track of interaction objects associated with different applications. TViews hardware implementation As previously mentioned, TViews uses a combination of acoustic and infrared communication technologies to implement the feature set already outlined. Inside the table, a master control board connected to the interaction surface manages the communication and tracking of the large set of interaction objects (known as pucks) as users place and move them on the table surface. A rechargeable lithium polymer battery powers each puck; the prototype pucks last 8 to 12 hours on a single charge although further work on the puck circuitry should result in weeks or months of operation on a charge. TViews uses acoustic ranging pings to locate the pucks, while infrared transceivers bidirectionally communicate object identity, time of flight, and object manipulation information between the master control unit and each puck. To transmit acoustic ranging pings, we ve affixed piezoceramic transducers to the bottomside four corners of the display glass. These transducers launch 200-KHz ultrasonic acoustic waves into the glass display surface in a bulk longitudinal acoustic wave Table enclosure containing master control board and PC IEEE Computer Graphics and Applications 49

4 2 Top view of the sensor layout on the TViews table surface. Acoustic transmitters affixed to corners Glass panel Acoustic receivers in mobile objects Infrared transceivers around edge of display area Glass panel Infrared transceivers in mobile objects Display area Display area mode. Furthermore, we ve placed a frame consisting of eight infrared transceivers around the interaction surface edge for communication. An excess number of inexpensive infrared transceivers around the display surface edge ensures that hand or object occlusion doesn t block communication between interaction objects and the TViews table. Future versions of the TViews system will employ radio frequency communication to avoid this problem entirely. Each puck (interaction object) is equipped with an ultrasonic receiving sensor to pick up the acoustic waves as they travel underneath the glass display surface, as well as an infrared transceiver for bidirectional data communication with the table. Figure 2 shows the layout of the sensors on the glass. Our position sensing approach was inspired in part by the Tap Window project, a system for tracking taps and other impacts on large glass surfaces. 2 An important difference between the TViews sensing approach and previous work on acoustic tracking systems such as the Tap Window is that TViews simultaneously tracks many objects on the table rather than one object at a time. Another important distinction is that the Tap Window is a passive system, used to track impacts on glass from inactive interaction objects such as metal objects or human fists, while the TViews table tracks active objects containing sensing and computation at the interaction object level. Furthermore, the acoustic receivers in the TViews system are located in the interaction objects themselves, while acoustic transmitters are affixed to the corners of the glass. This reversal of the past approach endows the TViews display surface with a positioning utility, which enables the TViews system to scale to an essentially unlimited number of interaction objects on the table without a significant reduction in the refresh rate or accuracy of object positions. TViews application programming interface We have implemented the TViews platform API in Java, which runs on the PC housed inside the media table. It allows developers to create different types of media table applications that make use of the combination of real-time object tracking and embedded display. The TViews software underlying the API keeps track of the incoming messages from the master control board and parses them into three main types of events: puck added, puck removed, and puck metadata (for example, position, button state, and so on) updated. The API employs a Java event model to distribute events to applications that have registered themselves as listeners. These events can be extended to incorporate additional event notifications to subscribing applications about user actions that use any external I/O devices that might be attached to a puck, such as a button press. We will eventually extend the API to include support for bidirectional messaging, which would allow the application to send messages to the pucks or to control specific properties for each puck. For instance, a puck might flash to draw attention. Or if the puck is equipped with a small add-on display, the application might send a text, picture, or video message. Future pucks might even contain actuators that allow them to move, vibrate, or provide haptic input or output. Tabletop application design Over the past decade, digital media table researchers have created prototype applications that range across a variety of application domains and contexts, such as urban resource planning, 3 business management and supply chain visualization, 4 and digital storytelling. 1 Scott et al. provide a good overview of tabletop interface research and suggest interface design guidelines for effective colocated collaboration around tabletop displays. 5 Since tabletops naturally support shared activities, well-designed media table interfaces have the potential to support shared user interactions with digital media content in an intuitive and natural way. Past work on tabletop applications and creating our own prototypes have inspired us to formulate a taxonomy of design considerations for media table applications, shown in Table 1. This taxonomy highlights several issues specific to media table design in which tangible objects act as the primary means of interacting with the digital information. In addition to the design of the interactive objects, the approaches and metaphors used for interface and control and the strategies employed for connecting applications across multiple tables are also important for tabletop application design. Some examples of current work in interface metaphors include the use of touch or pen-based table displays that include storage bins for organizing media content on the tabletop 6 and the development of an interface current metaphor that supports shared access to content pieces by flowing them around the edge of the horizontal display surface. 7 Certain application types might point to specialized solutions that impose tight constraints on interface and interaction design, while other types might allow a 50 September/October 2006

5 Table 1. Taxonomy of interface design considerations. Considerations Approaches Less Constrained More Constrained Control and Objects as generic handles or control Objects with specific fixed meaning or Object mappings devices (puck, stylus) functionality design Extensions and Generic I/O add-ons (buttons, dials, Physical shape reflects object s customizations sliders, lights, displays) meaning in virtual space Shared interaction Simultaneous movement of Enforced turn-taking or simultaneous Interface and approach individual control points movement with coordinated control control Multiviewpoint Directionless interface with free Fixed interface elements or automated approach rotation of all virtual media objects reorientation of media objects Visual interface Tables provide independent views Tables coordinate their views into the Networked approach into the media space media space tables Remote object Results of remote actions and object All object movements and actions presence movements displayed displayed greater level of flexibility. In terms of tangible object design, interaction objects might act as generic controllers like a stylus that could grab onto any virtual media object or they might be given a fixed meaning designated by their color, shape, or other physical attributes. For example, the Urp system permanently associates the interactive objects with a specific digital meaning and functionality based on their physical form. 3 Because Urp s physical building models are attached to corresponding virtual building models with specific physical forms, they cannot be used to control a different kind of virtual media object without breaking the interface metaphor. At the visual interface level, media tables are a shared platform so we must consider how multiple users interacting through multiple points of control will coordinate their activities and how the design can enhance the experience of all participants. We can achieve this through simultaneous free movement of objects, or with more constrained approaches like enforced turn taking or coordinated control. For instance, user interaction with a picture sorting application might be best served by an interface that allows continuous and simultaneous movement of the interactive objects. In this way, multiple people could browse and organize a collection together, perhaps selecting images that interest them, and storing these images within their own particular interactive objects. In contrast, drawing a figure in a graphical drawing application might use coordinated control between two objects to anchor one corner and then stretch out the figure with a second object. GraspDraw a two-handed drawing application created on the ActiveDesk uses this interaction style. 8 In the case of a digital board game, the rules of the game might impose a turn-taking approach in the game play, which is common in many traditional tabletop games such as chess or checkers. However, new types of games might allow simultaneous collaborative engagement of players with multiple objects at once. Another important issue that arises in the visual design of tabletop applications is the question of image or interface orientation, because users are likely to view the display surface from different sides while seated around the table. For example, researchers at Mitsubishi Electric Research Laboratories have explored the automatic orientation of interface elements on tabletop displays using their DiamondSpin Java API, which provides arbitrary reorientation and resizing of media elements on the DiamondTouch tabletop display. 9 While the TViews API provides a means of managing the physical objects as users swap and move them around on the tabletop, the DiamondSpin API supports the management of the graphical objects and interface elements. As such, the two APIs would work well together to support both the tangible and graphical aspects of tabletop application development. Finally, tabletop designers need to think about how multiple networked media tables coordinate views and user interactions, which is in some ways parallel to the issues that arise when connecting desktop application environments. Multiple views into the media space can be tightly coordinated so that remote users will always see the same thing, while a less constrained approach might lead to each table acting as an independent window into a larger virtual environment. The latter approach is generally used in online role-playing games, where different players can be located in different parts of the virtual world at any given time, so each user sees a different view of the virtual world even though all users are interacting within the same virtual space. Beyond the coordination of the visual interface, tabletop application designers also need to consider how the presence of remote objects is displayed on connected tables. In a less constrained approach, connected tables might only see the results of actions that have been performed by remote objects, such as the placing of a game piece. Other applications might lend themselves to a tighter coupling of object presence. For example, a networked version of the Urp system displayed the footprints of remote buildings on a connected table so that remote users could see the way others were moving them in real time. 3 Sample TViews applications To demonstrate the TViews system s scalability and flexibility, we constructed two complete TViews media IEEE Computer Graphics and Applications 51

6 (a) (b) (Courtesy Samsung Electronics 2005) 3 TViews media tables constructed in two different furniture styles at (a) the MIT Media Lab and (b) Samsung Research. tables and evaluated them both separately and together as a networked system. One TViews table was located at the Massachusetts Institute of Technology Media Laboratory, and the other was located around the world at Samsung s research center in Suwon, Korea. Figure 3 shows the two tables. We built 18 pucks and 10 sample media applications over the course of the development and testing of the TViews table. All pucks and applications can run on both tables, and the pucks can be swapped between applications running on different tables. Several of the applications use the puck s expandable I/O feature through add-on buttons that provide customizable functionality within the application space. Using the expandable I/O feature, TViews object tracking technology could eventually be incorporated into existing devices such as digital cameras and cell phones to extend their capabilities into new areas. In addition to our own application development, our research collaborators at Samsung in Korea have designed applications for a media table that can connect to a user s home entertainment center and other media devices. They are exploring how a media table and personalized interactive objects might replace the remote control devices currently used to control televisions, DVD players, audio systems, and other media devices. They have also designed tabletop application interfaces that can log family history; assist with family scheduling for vacations and other events; support movie and text messaging to and from other portable devices; maintain personalized preferences for media devices; and assist in the creation of playlists for photos, music, and movies. For instance, a TViews media player application might allow existing portable mediaplayer devices equipped with TViews tags to be tracked on the table s surface, allowing users to download songs and movies, pull up related videos online, or find and display the commonalities or differences between playlists stored on separate devices. Photo applications As digital camera technologies continue to spread, people are capturing and storing increasing numbers of digital photographs. These photos can be shared via mobile messaging, downloaded to laptop and desktop PCs, uploaded to picture sharing Web sites, or simply printed out in regular paper form. There is clearly a need for powerful tools to manage these growing collections, and although many such tools exist for example, Apple s iphoto or Google s Picasa managing large image collections can be a tedious and repetitive task. The once leisurely group activity of sorting and browsing physical photographs on a table has been replaced with single-user point-and-click interfaces that don t facilitate face-to-face interaction and socialization around media collections. To bring back some of the shared fun into the process of organizing, sharing, and viewing digital photographs, we have developed several photo applications on the TViews table. In the TViews Picture Sorter, users can sort and organize their digital photographs in the same way they would organize a shoebox of physical photographs on a tabletop. New images appear in a pile at the center of the TViews table, and the pucks help sort them into smaller clusters. However, unlike physical photographs that can only be in one pile at a time, their digital nature allows users to copy and move the photographs into several clusters at once. While the application currently provides only basic sorting functionality, users would eventually be able to provide annotations for their photographs, and the system could incorporate intelligent sorting features based on these annotations or on other aspects of the images, such as preexisting metadata tags. Users could also save the image clusters as slideshows, post them to the Web for remote access, or view them within different browsing environments, such as the Map Browser. The TViews Map Browser takes advantage of GPS metadata to automatically organize images on a geographical map based on the time and location at which each picture was taken. A timeline on one side of the table is color coded by day and small colored dots appear at each photograph location on the map, indicating the days that photographs were taken there. Users attach the pucks to different days on the timeline and drag them around the map to reveal that day s images. The images appear clustered around the puck, and can be zoomed by pressing the button on top of the puck. When an image is zoomed, another puck can grab hold of it 52 September/October 2006

7 (a) (b) 4 In Map Browser the pucks help navigate a collection of images on a geographic map. In this case, the image collection is from a vacation on the west coast of Ireland. (a) A close-up view of the display surface and pucks and (b) two users jointly browsing the image collection with two separate pucks. and drag it to another part of the table for separate viewing. Figure 4 shows the Map Browser interface and users interacting with the application. Researchers have demonstrated applications for browsing media collections on a number of touch-based tabletops, such as the DiamondTouch and SmartBoard tables. 6,9 While touch-based interaction is well suited for manipulating pictures on the display surface itself, tangible objects provide the additional benefit of allowing media content to move off the tabletop display and into the physical objects themselves. In the case of media organization, people could thus bring to the table the collections they have stored within their own interactive devices, such as digital cameras or portable media players, allowing these objects to serve a dual purpose. By using the TViews expandable I/O feature and embedding TViews sensing technology into the base of existing portable media devices, there would be no need for users to plug these devices into the table for media content download, or to transfer their media files over a network connection. Instead, the personal storage devices themselves could become the handles with which users would move content to and manipulate it on the table s surface, allowing easy viewing and swapping of media collections from separate devices. The input and output elements on the tangible objects, such as small displays, buttons, and sliders could also help perform additional tasks, such as scrolling through the contents, copying collections to the table or other devices, and comparing media content collections between separate devices. Tabletop games With the advent of digital game technology, game designers have made extensive use of emerging digital technologies to enhance game play by immersing viewers into virtual worlds with stunning graphics and complicated artificial-intelligence-based rule engines. While many of these games can be played in a networked mode with thousands of players at a time, they no longer provide the face-to-face social interactions common to more traditional games, such as board games. We have implemented a simple Pente board game for the TViews table to demonstrate digital boardgame play on the TViews platform. The game can be played with two or three players, and the goal is to place five stones in a row on the grid or to capture five pairs of an opponent s stones. Each player receives a puck, allowing him or her to drop stones onto the grid. The game uses three different pucks: yellow, red, and blue. The yellow puck comes with a yellow removable plastic icon cap on top, and it can place only yellow stones on the table. This behavior demonstrates how the pucks can be physically customized for a particular application, and how the physical shape of the interactive objects can be permanently linked to different meanings or functionality within the application s virtual space. During gameplay, we noticed that users would identify with their own piece, often holding onto it even when they were not using it to make a move on the game board. This identification with physical playing pieces can also be seen with traditional board games. In Monopoly for instance, some players can be very selective when choosing their playing piece, preferring to be a car rather than a thimble, or an iron rather than a race horse. Moreover, if the red puck is moved to a different TViews table running the Pente application, it will retain its identity as a red puck in the game. Figure 5 (next page) shows the Pente game on the TViews table and users during gameplay. In addition to the basic game features, Pente can be played in a networked mode from two different tables at once. In this case, each player s moves show up on both tables at the same time. For instance, two players in the game might be located at one table, while the third player is located at a different physically remote table. The idea of networked tabletop gameplay could eventually be extended to include many existing networked games, such as online role-playing or simulation games. TViews user experiences In the spring of 2005, we conducted a preliminary trial of the TViews system in a real-world apartment. The table was situated in the home of a volunteer for several weeks, during which he hosted a series of small social gatherings at which guests were invited to inter- IEEE Computer Graphics and Applications 53

8 (a) (b) 5 This game of Pente on the TViews table supports two or three players at a time and can also be played in networked mode between multiple tables: (a) close-up view of the game interface and pucks and (b) two users playing a game. Similar to a traditional Pente board, the graphical interface is directionless, allowing users to play from any side of the table. act with the TViews table. It was particularly important for us to see how users would interact with the table in a home environment rather than in a laboratory setting because we wanted to evaluate how the technology and overall system design would hold up when put to realworld use. In addition to sharing media and playing games, our TViews users put the table through the usual everyday coffee table usage: they placed a wide assortment of objects on the table s surface, they placed beverages and ate meals on top of the display while playing games, and they adjusted and readjusted the table s placement in the living room during social events without any need for recalibration or adjustment of the object sensing technology. Overall, the system performed well, and participants were able to use the table and pucks without difficulty. The only significant technical issues that came up were position errors resulting from poor contact of the acoustic sensor to the surface of the glass and a lag in positioning if a puck was moved too quickly. We are continuing to further develop the TViews pucks to address these issues, and are testing possible noncontact or point-contact acoustic sensing solutions. Furthermore, the position estimation algorithm is currently implemented in Java, which is marginally too slow to keep up with the calculations required to estimate in real time the positions of all the pucks on the table as they are moved around. A low-level system driver optimized for real-time performance should instead perform these calculations. We intend the applications we have been developing and testing to support activities that users would normally do in a home setting, such as browsing pictures or playing games. For this reason we allowed users to contribute their own photo collections for sorting, so testing of the Map Browser was done using a collection of geographically tagged photos that the host user had taken on a recent trip to Ireland. With the photo-related applications, we found that users quickly adjusted to the use of multiple pucks to browse and sort the images, and we often observed two or more users viewing different collections at the same time. When this occurred, we noticed that users were able to fluidly and repeatedly switch their focus between their own interaction and that of others. This seamless switch in focus is common when people browse physical photographs on a table and demonstrates that this natural type of interaction transitions easily into the tangible tabletop space. We also observed many shared interactions, where one person would control a puck, and the other person would use a second puck to drag images from the first person s collection to take a closer look at them. In general, users found the picture management and browsing interactions to be both intuitive and engaging, and they particularly enjoyed the informal stories and reminiscences that were shared around the Map Browser. The Pente game proved to be popular, particularly with users who were already familiar with the rules. The puck interaction was natural and familiar for users since most traditional board games provide separate pieces for each player. While tabletop game interactions can also take place on touch-based displays, TViews provides players with physical objects that they can identify with during gameplay. The use of tangible objects combined with the TViews expandable I/O feature could also let players create customized gaming objects with special functionalities for certain games. Users were especially interested in these possibilities for roleplaying games. In online role-playing games, there already exists an abundance of graphical player-created content, and the customization of physical objects could build on this tradition even further. Conclusion Now that we have an extensible method for creating media tables and a straightforward API for tabletop development, we would like to grow the community of developers who can create applications for the platform. Many of these applications can be based on ideas that our colleagues have already described. For instance, the ability of TViews tables to be networked together opens up the application space for new kinds of multiperson gameplay that can bridge the worlds of preexisting physical and digital games. 54 September/October 2006

9 Together with the development of the application space, we would also like to push forward the design of different interaction objects, providing custom shapes for different applications, as well as additional I/O elements like small displays or sensors within certain interaction objects. We would also like to incorporate interaction object capabilities into existing digital media devices, such as digital audio players or digital cameras, providing new ways of sharing and exchanging the digital contents within them as they are placed on a TViews surface. If we continue to grow the media table application space, we hope this will feed back into increased investment into the development of the platform itself. Through this cycle of growth that pushes on both the platform and application development in turn, we hope that media tables will fulfill their promise and will one day be part of our everyday environment. Acknowledgments We would like to thank our research collaborators at Samsung, especially Jay Lee, Hongsuk Choi, Yanglim Choi, and Seung-Hun Jeon. We would also like to thank Joe Paradiso, Bill Mitchell, and the members of the Media Fabrics research group at the MIT Media Laboratory. References 1. A. Mazalek and G. Davenport, A Tangible Platform for Documenting Experiences and Sharing Multimedia Stories, Proc. ACM SIGMM Workshop Experiential Telepresence (ETP), ACM Press, 2003, pp J.A. Paradiso and C.-K. Leo, Tracking and Characterizing Knocks Atop Large Interactive Displays, Sensor Rev., vol. 25, no. 2, pp J. Underkoffler and H. Ishii, Urp: A Luminous-Tangible Workbench for Urban Planning and Design, Proc. Conf. Human Factors in Computing Systems (CHI), ACM Press, 1999, pp J. Patten et al., Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces, Proc. Conf. Human Factors in Computing Systems (CHI), ACM Press, 2001, pp S.D. Scott, K.D. Grant, and R.L. Mandryk, System Guidelines for Co-located, Collaborative Work on a Tabletop Display, Proc. European Conf. Computer-Supported Cooperative Work (ECSCW), Kluwer Academic Publishers, 2003, pp S.D. Scott, S. Carpendale, and S. Habelski, Storage Bins: Mobile Storage for Collaborative Tabletop Displays, IEEE Computer Graphics and Applications, vol. 25, no. 4, 2005, pp U. Hinrichs et al., Interface Currents: Supporting Fluent Collaboration on Tabletop Displays, Proc. 5th Int l Symp. Smart Graphics, Springer, pp G. Fitzmaurice, H. Ishii, and W. Buxton, Bricks: Laying the Foundations for Graspable User Interfaces, Proc. Conf. Human Factors in Computing Systems (CHI), ACM Press, 1995, pp C. Shen et al., DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction, Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI), ACM Press, 2004, pp Ali Mazalek is an assistant professor in digital media at the Georgia Institute of Technology. Her research interests include emerging physical sensing and tangible interaction technologies for media arts and entertainment. Mazalek has an MS and PhD from the MIT Media Laboratory s Tangible Media and Media Fabrics groups. She was a Samsung and MediaLabEurope Fellow. Contact her at mazalek@gatech.edu. Matthew Reynolds is cofounder and CTO of ThingMagic, an RFID systems company. His research interests include the physics of sensors and actuators, signal processing, and sensor networks. Reynolds has an SB and M.Eng. in electrical engineering and computer science from MIT and a PhD from the MIT Media Laboratory, where he was a Motorola Fellow. Contact him at matt@thingmagic.com. Glorianna Davenport directs the Media Fabrics group at the MIT Media Laboratory. Her research interests include collaborative coconstruction of digital media experiences, where narration is split among authors, consumers, and computer mediators. Davenport has a BA in English from Mount Holyoke, and a fine arts MA from Hunter College. Contact her at gid@media.mit.edu. For further information on this or any other computing topic, please visit our Digital Library at computer.org/publications/dlib. IEEE Computer Graphics and Applications 55