3-D Live: Real Time Interaction for Mixed Reality

Transcription

1 3-D Live: Real Time Inteaction fo Mixed Reality Simon Pince 1 Adian David Cheok 1 Fazam Fabiz 1 Todd Williamson 2 Nik Johnson 2 Mak Billinghust 3 Hiokazu Kato 4 1 National Univesity of Singapoe, {elesp,eleadc,eleff}@nus.edu.sg 2 Zaxel Systems, {toddw,nik}@zaxel.com 3 Univesity of Washington, gof@hitl.washington.edu 4 Hioshima State Univesity, kato@sys.im.hioshima-cu.ac.jp ABSTRACT We descibe a eal-time 3-D augmented eality videoconfeencing system. With this technology, an obseve sees the eal wold fom his viewpoint, but modified so that the image of a emote collaboato is endeed into the scene. We egiste the image of the collaboato with the wold by estimating the 3-D tansfomation between the camea and a fiducial make. We descibe a novel shape- fom-silhouette algoithm, which geneates the appopiate view of the collaboato and the associated depth map at 30 fps. When this view is supeimposed upon the eal wold, it gives the stong impession that the collaboato is a eal pat of the scene. We also demonstate inteaction in vitual envionments with a live fully 3-D collaboato. Finally, we conside inteaction between uses in the eal wold and collaboatos in a vitual space, using a tangible AR inteface. Keywods Video-Confeencing, Augmented Reality, Image Based Rendeing, Shape fom Silhouette, Inteaction INTRODUCTION Science fiction has pesaged many of the geat advances in computing and communication. In 2001: A Space Odyssey, D Floyd calls home using a videophone an ealy on-sceen appeaance of 2-D video-confeencing. This technology is now commonplace. Moe ecently, the Sta Was films depicted 3-D hologaphic communication. In this pape we apply compute gaphics to ceate what may be the fist eal-time holo-phone. Existing confeencing technologies have a numbe of limitations. Audio-only confeencing emoves visual cues vital fo convesational tun-taking. This leads to inceased inteuptions and ovelap [8], and difficulty in disambiguating between speakes and in detemining willingness to inteact [14]. Conventional 2-D videoconfeencing impoves mattes, but lage use movements and gestues cannot be captued [13], thee ae no spatial cues between paticipants [29] and paticipants cannot easily make eye contact [30]. Paticipants can only be viewed in font of a sceen and the numbe of paticipants is limited by Pemission to make digital o had copies of all o pat of this wok fo pesonal o classoom use is ganted without fee povided that copies ae not made o distibuted fo commecial advantage and that copies bea this notice and the full citation on the fist page. To copy othewise, o epublish, to post on seves o to edistibute to lists equies pio specific pemission and/o a fee. CSCW 02 Novembe 16-20, 2002, New Oleans, Louisiana, USA. Copyight 2002 ACM /02/0011 $5.00 Figue 1: Obseves view the wold via a head-mounted display (HMD) with a font-mounted camea. Ou system Figue 1: Obseves view the wold via a head-mounted display detects makes in the scene and supeimposes live video (HMD) with a font-mounted camea. Ou system detects makes content endeed fom the appopiate viewpoint in eal time. in the scene and supeimposes live video content endeed fom the appopiate viewpoint in eal time. monito esolution. These limitations disupt fidelity of communication [34] and tun taking [10], and incease inteuptions and ovelap [11]. Collaboative vitual envionments estoe spatial cues common in face-to-face convesation [4], but sepaate the use fom the eal wold. Moeove, non-vebal communication is had to convey using conventional avatas, esulting in educed pesence [29]. We define the pefect video avata as one whee the use cannot distinguish between a eal human pesent in the scene and a emote collaboato. Pehaps closest to this goal of pefect tele-pesence is the Office of the Futue wok [27], the Vitual Video Avata of Ogi et al.[25], and the wok of Mulligan and Daniilidis [23][24]. All sytems use multiple cameas to constuct a geometic model of the paticipant, and then use this model to geneate the appopiate view fo emote collaboatos. Although impessive, these systems cuently do not geneate the whole 3D model one cannot move 360 o aound the vitual avata. Moeove, since the output of these systems is mediated via pojection sceens the display is not potable. The goal of this pape is to pesent a solution to these poblems, by intoducing an augmented eality (AR) video-confeencing system. Augmented eality efes to the eal-time insetion of compute-geneated thee-dimensional content into a eal scene (see [2], [3] fo eviews). Typically, the obseve views the wold though a head mounted display (HMD) with a camea attached to the font. The video is captued, modified and elayed to the obseve in

2 eal time. Essentially we ceate a live vitual video avata and then use AR technology fo displaying that avata ove the eal wold (see Figue 1). In addition to ceating an extemely compelling sense of pesence, this facilities a wide ange of teleconfeencing and collaboative applications. In the fist pat of the pape, we eview pevious wok in AR confeencing. The enabling technology fo ou system is a novel method fo geneating abitay views of a collaboato at inteactive speeds. In the second section, we sketch this algoithm and demonstate a numbe of its advantages ove competing technologies fo eal-time communication applications. In the thid pat of the pape, we intoduce a numbe of othe applications of the technology to which ou system is paticulaly suited. These include visualisation of a collaboato in a vitual space, and a novel method fo uses in eal space to inteact with vitual collaboatos, using tangible use inteface techniques. Pio Wok Billinghust and Kato [6] fist exploed how AR could be used to suppot emote collaboation, and to povide gaze and non-vebal communication cues. Uses woe a lightweight HMD and wee able to see a single emote use appea attached to a eal cad as a life-sized, live vitual video window. The oveall effect was that the confeence collaboato appeaed pojected into the local use s eal wokspace (see Figue 2). Since the cads ae physical epesentations of emote paticipants, ou collaboative inteface can be viewed as a vaiant of the Ishii s tangible inteface metapho [Ishii 97]. Uses can aange the cads about them in space to ceate a vitual spatial confeencing space and the cads ae also small enough to be easily caied, ensuing potability. The use is no longe tied to the desktop and can potentially confeence fom any location, so the emote collaboatos become pat of any eal wold suoundings, potentially inceasing the sense of social pesence. Thee ae a numbe of othe significant diffeences between the AR confeencing inteface and taditional desktop video confeencing. The emote use can appea as a life-sized image and a potentially abitay numbe of emote uses can be viewed at once. Since the vitual video windows can be placed about the use in space, spatial cues can be estoed to the collaboation. Finally, the emote use s image is entiely vitual so a eal camea could be placed at the uses eye point allowing suppot fo natual gaze cues. In a use study that compaed AR confeencing to taditional audio and video confeencing subjects epoted a significantly highe sense of pesence fo the emote use in the AR confeencing condition and that it was easie to peceive non-vebal communication cues [6]. Indeed, the compelling natue of the AR confeence was amply shown by one use who leaned in close to the monito duing the video confeencing condition, and moved back duing the AR condition to give the vitual collaboato the same body space as in a face-to-face convesation. Moe ecent wok [7] has pesented an AR confeencing inteface the suppots multiple emote uses and applies alpha mapping techniques to extact video of the emote use fom the backgound and ceate a moe natual image (see Figue 2). In a use study with this inteface uses felt that the AR condition povided significantly moe co-pesence and impoved the undestanding of the convesational elationship between paticipants. 3-D LIVE AUGMENTED REALITY Oveview In this pape, we aim to inset a live image of a emote collaboato into the visual scene (see Figues 1 and 2). As the obseve moves his head, this view of the collaboato changes appopiately. This esults in the stable pecept that the collaboato is thee dimensional and pesent in the space with the obseve. In ode to achieve this goal, we equie that fo each fame: (i) the pose of the head-mounted camea elative to the scene is estimated, (ii) the appopiate view of the collaboato is geneated and (iii) this view is endeed into the scene, possibly taking account of occlusions. We conside each of these poblems in tun. HMD Camea Pose Estimation The scene was viewed though a Daeyang Cy-Viso DH-4400VP head mounted display (HMD), which pesented the same 640x480 pixel image to both eyes. A PemaCam SCM seies colo secuity camea was attached to the font of this HMD. This captues 25 images pe second at a esolution of 640x480. We employ the make tacking method of Kato and Billinhust [18]. We simplify the pose estimation poblem by inseting 2-D squae black and white fiducial makes into the scene. Vitual content is associated with each make. Figue 2: Pogession towads moe natual augmented eality video-confeencing. Initial wok by [6] associated 2-D images of single collaboatos with cad makes (left). Subsequent wok inceased the numbe of collaboatos and intoduced alpha-mapping to moe ealistically integate the 2-D video steam into the wold (cente). In this pape we intoduce fully live 3-D video confeencing (ight)

3 Since both the shape and patten of these makes is known, it is easy to both locate these makes and calculate thei position elative to the camea. In bief, the camea image is thesholded and contiguous dak aeas ae identified using a connected components algoithm. A contou seeking technique identifies the outline of these egions. Contous that do not contain exactly fou cones ae discaded. We estimate the cone positions by fitting staight lines to each edge and detemining the points of intesection. A pojective tansfomation is used to map the enclosed egion to a standad shape. This is then coss-coelated with stoed pattens to establish the identity and oientation of the make in the image Fo a calibated camea, the image positions of the make cones uniquely identify the thee-dimensional position and oientation of the make in the wold. This infomation is expessed as a Euclidean tansfomation matix elating the camea and make co-odinate systems, and is used to ende the appopiate view of the vitual content into the scene. Softwae fo augmented eality make tacking and calibation can be downloaded fom [35]. Vitual Viewpoint Geneation Backgound In ode to integate the vitual collaboato seamlessly into the eal wold, we need to geneate the appopiate view fo each video fame. To achieve this goal, we must geneate a model of the 3-D shape of the collaboato on each fame. A novel view can easily be constucted given the shape and seveal known viewpoints. One appoach is to gathe depth infomation using steeo-depth. Steeo econstuction can now be achieved at inteactive speeds [17][23][24]. Howeve, the esulting dense depth map is not obust, and no existing system places cameas all ound the subject. Related image-based endeing techniques [28][1] do not explicitly calculate depth, but still equie dense matches between images, and ae similaly pone to eo. Figue 3. Vitual viewpoint geneation by shape fom silhouette. Points which poject into the backgound in any camea ae ejected. The points fom A to C have aleady been pocessed and poject to backgound in both images, so ae maked as unoccupied (magenta). The points yet to be pocessed ae maked in yellow. Point D is in the backgound in the silhouette fom camea 2, so it will be maked as unoccupied and the seach will poceed outwad along the line. A moe attactive appoach to fast 3-D model constuction that has been exmployed by [21] and [22] is to use shape-fom-silhouette infomation. A numbe of cameas ae placed aound the subject. Each pixel in each camea is classified as eithe belonging to the subject (foegound) o the backgound. The esulting foegound mask is called a silhouette. Each pixel in each camea collects light ove a (vey naow) ectangula-based pyamid in 3D space, whee the vetex of the pyamid is at the focal point of the camea and the pyamid extends infinitely away fom this. Fo backgound pixels, this space can be assumed to be unoccupied. Shape-fom-silhouette algoithms wok by initially assuming that space is completely occupied, and using each backgound pixel fom each camea to cave away pieces of the space to leave a epesentation of the foegound object. Clealy, the econstucted model will impove with the addition of moe cameas. Howeve, the esulting depth econstuction may not captue the tue shape of the object, even with infinite cameas. The best possible econstucted shape is temed the visual hull [20]. Despite this limitation, shape-fom-silhouette has thee significant advantages ove competing technologies. Fist, it is moe obust than steeo-vision. Even if backgound pixels ae misclassified as pat of the object in one image, othe silhouettes ae likely to cave away the offending misclassified space. Second, it is significantly faste than eithe steeo, which equies vast computation to calculate coss-coelation, o lase ange scannes, which geneally have a slow update ate. Thid, the technology is inexpensive and equies no specialized hadwae. Fo these easons, the system descibed in this pape is based on shape-fom-silhouette infomation. We believe that this is the fist system that is capable of captuing 3D models and textues fom a lage numbe of cameas (15) and displaying them fom an abitay viewpoint at 25 fps (the captue camea fame ate). To the best of ou knowledge, the closest compaable system employs only 5 cameas and model quality suffes accodingly. Algoithm Oveview Given any standad 4x4 pojection matix epesenting the desied vitual camea, the cente of each pixel of the vitual image is associated with a ay in space that stats at the camea cente and extends outwad. Any given distance along this ay coesponds to a point in 3D space. In ode to detemine what colo to assign to a paticula vitual pixel, we need to know the fist (closest) potentially occupied point along this ay. This 3D point can be pojected back into each of the eal cameas to obtain samples of the colo at that location. These samples ae then combined to poduce the final vitual pixel colo. Thus the algoithm pefoms thee opeations at each vitual pixel: Detemine the depth of the vitual pixel as seen by the vitual camea. Find coesponding pixels in neaby eal images Detemine pixel colo based on all these measuements. Detemining Pixel Depth The depth of each vitual pixel is detemined by an explicit seach. The seach stats at the vitual camea pojection cente and poceeds outwad along the ay coesponding to

4 the pixel cente (see Figue 3). Each candidate 3D point along this ay is evaluated fo potential occupancy. A candidate point is unoccupied if its pojection into any of the silhouettes is maked as backgound. When a point is found fo which all of the silhouettes ae maked as foegound, the point is consideed potentially occupied, and the seach stops. To constain the seach fo each vitual pixel, the coesponding ay is intesected with the boundaies of each eal image. We poject the ay into each image to fom the coesponding epipola line. The points whee these epipola lines meet the image boundaies ae found and these bounday points ae pojected back onto the ay. The intesections of these egions on the ay define a educed seach space. If the seach eaches the futhest limit of this egion without finding any potentially occupied pixels, the vitual pixel is maked as backgound. Detemining Pixel Colo In geneal, we would pefe to establish the pixel colo based on the data fom cameas that ae closely aligned with the desied novel view. We ank the cameas based on thei poximity and choose the thee closest. We now compute whee the 3D point lies in each candidate camea s image. Unfotunately, the eal camea does not necessaily see this point in space - anothe object may lie between the eal camea and the point. If the eal pixel is occluded in this way, it cannot contibute its colo to the vitual pixel. We epeat the depth seach algoithm on a pixel fom the eal camea. If the ecoveed depth lies close enough in space to the 3D point computed fo the vitual camea pixel, we assume the eal camea pixel is not occluded - the colo of this eal pixel is allowed to contibute to the colo of the vitual pixel. In pactice, we incease system speed by immediately accepting points that ae geometically cetain not to be occluded. We take a weighted aveage of the pixels fom the non-occluded cameas, such that the closest camea is given the most weight. System Hadwae and Softwae Fouteen Sony DCX-390 video cameas wee equally spaced aound the subject, and one viewed him/he fom above (see Figue 4). Five Dual 1Ghz Pentium III video-captue machines eceived data fom thee cameas each. The video-captue machines pe-pocess the video fames by detemining the silhouettes and pass these to the endeing seve via gigabit Ethenet links. The endeing seve was based on a 1.7 GHz Pentium IV Xeon pocesso. The chaacteistics of ou algoithm allow us to geneate vey high quality models based on 15 cameas vey quickly. The figues in this pape wee geneated at 384x288 esolution at 25 fps with a latency of <100ms. Since each foegound object must be completely visible fom all cameas, the zoom level of each camea must be adjusted so that it can always see the subject, even as he/she moves aound. This means that the limited esolution of each camea must be spead ove the desied imaging aea. Hence, thee is an inevitable tade-off between image quality and the captued volume captued. Similaly, the accuacy of the depth estimate deceases as the cameas ae moved futhe fom the subject. Similaly, the physical space needed fo the system is detemined by the size of the desied captue aea and the field of view of the lenses used. We have expeimented with a 2.8 mm lens that povides appoximately a 90 degee field of view. With this lens, it is possible to captue a space that is 2.5m high and 3.3m in diamete with cameas that ae 1.25 metes away. Compaison with othe methods Ou system is simila in spiit the wok of Matusik et al. [22] who also pesent an image-based novel view geneation algoithm using sillhouette infomation. The pincipal diffeence is that Matusik et al. geneate the entie visual hull fom the cuent camea angle, wheeas we geneate only the visible pat. Lok [21] has poposed an altenative volume based appoach to econstuction. Both of the above systems scale linealy with the numbe of cameas added. Ou system scales much moe slowly in pactice, since the pixel colo estimation (which takes the bulk of the endeing time) only uses of a fixed numbe of camea images. 3-D INTERACTION FOR MIXED REALITY Ou full system combines the vitual viewpoint and augmented eality softwae (see Figue 5). Fo each fame, the augmented eality system identifies the tansfomation matix elating make and camea positions. This is passed to the vitual viewpoint seve, togethe with the estimated camea calibation matix. The seve esponds by etuning an RGBA image, and adepth estimate associated with each pixel. This simulated view of the emote collaboato is then supeimposed on the oiginal image and displayed to the use. In ode to incease the system speed, we intoduce a single fame delay into the pesentation of the augmented eality video. Hence, the augmented eality system stats pocessing the next fame while the vitual view seve geneates the view fo the pevious one. A swap then occus. The gaphics ae etuned to the augmented eality system fo display, and the new tansfomation matix is sent to the vitual view endee. The delay ensues that neithe machine wastes significant pocessing time waiting fo the othe and a high thoughput is maintained. In pactice, this means that thee is no noticeable delay fo uses when they move thei head, the view of the collaboato appeas to move simultaneously. One advantage of ou client-seve system is that the netwok equiements ae elatively low. Since fo each fame only one image is equied, the system is no moe VIDEO CAPTURE t x t y t z 1 Camea In MIXED REALITY CLIENT NOVEL VIEW GENERATION Display Out Figue 4. System oveview: Fifteen cameas feed video of the subject to five captue machines that calculate the silhouettes using and pass these to the endeing seve. The mixed eality client takes camea output fom the HMD and calculates the pose of the make. The associated tansfomation matix is then passed to the endeing seve which etuns the appopiate view of the subject. This is dawn into the subject s view with a total delay of <100ms

5 demanding than 2-D video confeencing in tems of bandwidth. In fact, it may be less demanding, since we know which pats of the image ae backgound, facilitating compession. Howeve, in contast to standad video confeencing, quality of sevice is extemely impotant to etain the inteactive qualities of the models. Augmented Reality Confeencing In the AR confeencing application, paticipant one (the collaboato) stands suounded by the vitual viewpoint cameas. Paticipant two (the obseve) sits elsewhee, weaing the HMD. We use the tems collaboato and obseve in the est of the pape to efe to these oles since the system is cuently uni-diecitonal Using ou system, a sequence of endeed views of the collaboato is sent to the obseve so that the collaboato appeas supeimposed upon a fiducial make in the eal wold. The paticula image of the collaboato geneated depends on the exact geomety between the HMD-mounted camea and the fiducial make. Hence, if the obseve moves his head, o manipulates the fiducial make, the image changes appopiately. This system ceates the peception of the collaboato being in the thee-dimensional space with the obseve. The audio steam geneated by the collaboato is also spatialized so that it appeas to emanate fom the vitual collaboato on the make. Fo ou application, we have chosen a elatively lage imaging space (appox 3x3x2m), which is descibed at a elatively low esolution. This allows the system to captue movement and non-vebal infomation fom gestues that could not possibly be captued with a single fixed camea. We pesent the example of an acto auditioning fo a play (see Figues 1,2c and 5a). The full ange of his pefomance, including pacing back and foth and kneeling can be captued by the system and elayed into the augmented space of the obseve. Subjects unifomly epot the feeling that the collaboato is a stable and eal pat of the wold and equied few instuctions. In fact, the lage space that is imaged is a design choice that was chosen to demonstate lage scale movements and gestues. A lage aea also gives us the ability to captue seveal people at once, and ou system poduces highly successful esults even they ae close togethe. The method of display on a small cad-make on a table top is especially suited fo viewing these movements as the lage aea is scaled to a smalle space and can be manipulated easily to view a collaboato fom the desied angle. We make this the pimay demonstation of ou technology as the ability to captue such movements is a novel aspect of ou system that cannot be achieved by competing methods. Figue 5. Video confeencing can take place in a table-top context allowing potability and easy manipulation of the position o we can geneate life-size avatas fo a moe natual inteaction. Howeve, one disadvantage of this aangement is that fo bi-diectional systems is that thee is an asymmety in the viewing positions. Conside two uses. It is cetainly possible to indicate to each the viewing diection of the othe so that they can oient themselves coectly. An example of this system with gaphical avatas is pesented in [5]. Howeve, the use cannot simultaneously oient himself to the collaboatos viewing diection (pesumably above his head) and to his view of the collaboato on his cad make (pesumably below his head). The obvious solution to these poblems is to ende the collaboato at life-size. Relative body postue and oientation of uses then emains exactly the same as in eal life (see Figue 5b). In such a situation, one might wish to to move the cameas much close and image a smalle volume at much highe esolution, simila to confeencing pesented in 23. It is still possible to tack the use s head movements in this configuation using fiducial makes on the floo o walls. Howeve, in pactice, subjects epot difficult keeping the make in the scene and viewing the collaboato simultaneously and featue tacking o a commecial vitual eality tacking solution ae supeio solutions. We have expeimented using an Intesense IS 900 tacke and the esult is extemely stable. In this case, communication is as close to eal-life as we can simulate. The collaboato appeas in full 3-D at eal scale in the use s space. This allows spatial elationships between paticipants to be maintained in addition to the vey natual visualisation that this technology pemits. Ou system is cuently unidiectional the use can see the collaboato but not vice vesa. A symmetical elationship intoduces a futhe complication: completely natual communication is disupted because mutual eye contact cannot be maintained when head-mounted displays ae won. One obvious solution is to mediate augmented eality confeencing with optical see though head-mounted displays in which the eyes can still be seen. Recent wok has also investigated the possibility of eplacing each collaboatos view of the othes head mounted display with a compute gaphic epesentation of the eyes [34]. Collaboation in Vitual Envionments Vitual envionments epesent an exciting new medium fo compute-mediated collaboation. Indeed, fo cetain tasks, they ae demonstably supeio to video-confeencing [31]. Howeve, it was not peviously possible to accuately visualize collaboatos within the envionment and a symbolic gaphical epesentation (avata) was used in thei place. Consideable eseach effot has been invested in identifying those non-vebal behavios that ae cucial fo collaboation [9] and elaboate intefaces have been developed to contol expession in avatas. In this section, we eplace the symbolic avata with a simulated view of the actual peson as they exploe the vitual space in eal time. We geneate the appopiate view of a collaboato in the vitual space, as seen fom ou cuent position and oientation. In ode to immese the use and collaboato in the vitual envionment, it is necessay to pecisely tack thei head oientation and position, so that the vitual scene can be endeed fom the coect viewpoint. These paametes wee estimated using the Intesense IS900 tacking system, which measues position to within 1.5mm and oientation to within

6 0.05 degee inside a 9x3m egion. Fo the obseve, the position and oientation infomation geneated by the Intesense system is also sent to the vitual view system to geneate the image of the collaboato and the associated depth map. This is then witten into the obseve s view of the scene. The depth map allows occlusion effects to be implemented using Z-buffe techniques. Figue 6 shows seveal fames fom a sequence in which the obseve exploes a vitual at galley with a collaboato, who is an at expet. The collaboato, who is in the vitual view system, is seen to move though the galley discussing the pictues with the use. The vitual viewpoint geneation captues the movement and gestues of the at expet allowing him to gestue to featues in the vitual envionment and communicate natually. Notice, that it is even clea fom the still pictues what is going on in the scene due to the ich gestual and body pose infomation conveyed by ou system. A citical featue of ou system is the ability to captue the paticipants in a lage aea and fom all diections so that they can move and visualise one anothe natually in the vitual space. We believe this is the fist demonstation of collaboation in a vitual envionment with a live, fully 3-D video avata. Tangible AR Inteaction One inteesting aspect of the video-confeencing application was that the vitual content was attached to physical eal-wold objects. Manipulation of such objects ceates a tangible use inteface with the compute (see Figue 1). In ou pevious application, this meely allowed the use to position the video-confeencing steam within his/he envionment. We can also apply these techniques to inteact with the use in a natual physical manne. Fo example, Kato et al. [19] demonstated a pototype inteio design application in which uses can pick up, put down, and push vitual funitue aound in a vitual oom. Futhe examples ae pesented in [26], [7] and[32]. We have exploed the use of tangible AR inteaction techniques in a collaboative entetainment application. The obseve views a miniatuized vesion of a collaboato exploing a vitual game envionment, supeimposed upon his desk in the eal wold (see Figue 7). The collaboato is associated with a physical paddle. The obseve can now move the collaboato aound the vitual envionment, o even pick him up and place him inside a new vitual envionment by manipulating the eal physical paddle. Afte [5], the paticula vitual envionment is chosen using a eal-wold book as the inteface. A diffeent fiducial make (o set theeof) is pinted on each page and associated with a diffeent envionment. The obseve simply tuns the pages of this book to choose suitable vitual wold. Tangible inteaction techniques ae also used to physically inteact with the collaboato. We pesent the example of a catoon style envionment (see Figue 7). We use the paddle to dop catoon objects such as anvils and bombs onto the collaboato, who attempts, in eal time, to jump out of the way. The ange map of the vitual view system allows us to calculate the mean position of the obseve and hence implement a collision detection outine. The obseve picks up the objects fom a epositoy by placing the paddle next to the object. He dops the object by tilting the paddle when it is above the obseve. Duing all of this, the collaboato does not view the use diectly, but sees a gaphical indicato of his viewing diection and sees the esults of his actions changes in vitual envionment o objects falling fom the sky. One possibility fo the futue is to integate a small camea into the paddle and use this to poject a lage view of the obseve into the backgound of the collaboatos view as if the obseve was a god-like figue in the sky. This type of collaboation between an obseve in the eal wold and a colleague in a vitual envionment is impotant and has not peviously been exploed. Although this application is lightheated, it is easy to think of eal wold applications. Conside the inteio design application of [19]. It would be possible fo a client to stand immesed in the vitual oom and communicate to a designe whee he wanted funitue placed. The system would allow the designe to visualise the client inside the oom and communicate with him, while still using the tangible techniques to make macoscopic changes to the oom contents. DISCUSSION AND FUTURE WORK We have pesented a shape-fom-silhouette algoithm, which is capable of geneating a novel view of a live subject in eal time, togethe with the depth map associated with that view. The efficiency of this algoithm pemits the development of a seies of live collaboative applications. The distinct featues of this system which impove collaboative wok ae (i) we captue a full 3-D model which is viewable fom all diections, (ii) we take measuements fom a lage aea allowing goss movements to be visualised, and (ii) we ende this model into the use s view of the eal wold. Using these featues, we have demonstate an augmented eality based video-confeencing system in which the image of the collaboato is supeimposed upon a thee-dimensional make in the eal wold. To the use the collaboato appeas to be pesent within the scene. This is the fist example of the pesentation of live, 3D content in augmented eality. Moeove, the AR system solves seveal poblems that have limited pevious video-confeencing Figue 6. Inteaction in vitual envionments. The vitual viewpoint geneation can be used to make live video avatas fo vitual envionments. We pesent the example of a guide in a vitual at galley. The subject can gestue to objects in the envionment and communicate infomation by non-vebal cues. Notice how easy it is to intepet the action in each of these fames, just by looking and body the body pose and oientation of the acto. The final fame shows how the depth estimates geneated by the endeing system can be used to geneate coect occlusion. Note that these fames wee geneated at 640x480 esolution

7 applications, such as the expession of non-vebal communication, potability, and the ability to aange collaboatos in a spatial configuation in the wold. We also use the vitual viewpoint system to geneate a live 3D avata fo collaboative wok in a vitual envionment. This is an example of augmented vituality in which eal content is intoduced into vitual envionments. As befoe, the obseve always sees the appopiate view of the collaboato but this time they ae both within a vitual space. The lage aea ove which the collaboato can be imaged allows movement within this vitual space and the use of gestues to efe to aspects of the wold. Lastly, we use tangible inteaction techniques to show how a use can inteact natually with a collaboato in a thee-dimensional wold. We pesent the example of a game wheeby the collaboato must dodge falling objects dopped by the use. This type of collaboative inteface is as a vaiant of Ishii s tangible use inteface metapho [16]. One obvious citicism of the system is that the cuent implementation is demanding on both space and computational esouces: we cuently use a lage studio which has been painted geen and base ou novel view geneation on 15 cameas whose images ae pocessed by 6 computes. This is a valid immediate concen, but one that is unlikely to pesist. The algoithm is natually amenable to paallel pocessing by specialised hadwae. Indeed a elated poject has aleady employed gaphics hadwae to acceleate shape fom-sillhouette algoithms [21]. Even in the absence of such hadwae impovements, Mooe s law suggests that in only fou yeas, a single compute will be able to implement this algoithm. Impoved segmentation techniques, wide angle lenses and moe automated calibation methods could see such systems integated into nomal ooms. The focus of ou futue eseach effot will be on exploing multiple use collaboations, whee each use stands in a vitual viewpoint system. Each should be able to see the othes fom the appopiate viewpoint, whethe they ae in a eal o a vitual space. We hope that this will allow effective compute mediated communication between lage goups of people than have been peviously possible. Fo futhe infomation please visit o REFERENCES 1. Avidan, S. and Shashua, A. Novel view synthesis by cascading tilinea tensos. IEEE Tans. Visualization and Compute Gaphics 4, 4 (Oct-Dec 1998) Azuma, R.T. A suvey of augmented eality. Pesence 6, 4 (Aug. 1997), Azuma, R.T., Baillot, Y., Behinge, R., Feine, S., Julie, S. and MacIntye, B. Recent advances in augmented eality. IEEE Compute Gaphics and Applications 21, 6 (Nov./Dec. 2001), Benfod, S. and Fahlen, L. A spatial model of inteaction in lage vitual envionments, in Poc. ECSCW 93 (Milano, Italy, Sept. 1993), Figue 7: Tangible inteaction sequence, demonstating inteaction between a use in AR and collaboato in AR. The sequence uns along each ow in tun. In the fist fame, the use sees the collaboato exploing a vitual envionment on his desktop. The collaboato is associated with a fiducial make paddle. This foms a tangible inteface that allows the use to take him out of the envionment. The use then changes the page in a book to eveal a new set of makes and VR envionment. This is a second example of tangible inteaction. He then moves the collaboato to the new vitual envionment, which can now be exploed. In the final ow, we pesent an inteactive game. The use selects a heavy ock fom a vitual asenal using the paddle. He then moves it ove the collaboato and attempts to dop it on him. The collaboato sees the ock ovehead and attempts to jump out of the way

8 5. Billinghust, M., Kato, H. and Poupyev, I. The MagicBook: An inteface that moves seamlessly between eality and vituality. IEEE Compute Gaphics and Applications 21, 3 (May/Jun 2001), Billinghust, M. and Kato, H. Real wold teleconfeencing. In Poc. CHI 99 (New Yok, 1999), Billinghust, M., Poupyev, I., Kato, H. and May, R. Mixing ealities in shaed space: an augmented eality inteface fo collaboative computing. In Poc. ICME 00, (New Yok, 2000), Boyle, E., Andeson, A. and Newlands, A. The effects of visibility on dialogue and pefomance in a co-opeative poblem solving task. Language and Speech 37, 1 (Jan./Ma. 1994), Cassell, J. and Thoisson, K.R. The powe of a nod and a glance: envelope vs. emotional feedback in animated convesational agents. Applied Atificial Intelligence 13, 4-5 (June 1999), O Conaill, B., Whittake, S. and Wilbu, S., Convesations ove video confeences: an evaluation of the spoken aspects of video-mediated communication. HCI 8, (1993), O Conaill, B. and Whittake, S., Chaacteizing, pedicting and measuing video-mediated communication: a convesational appoach. In K. Finn, A. Sellen, S. Wilbu (Eds.), Video Mediated Communication. LEA: NJ, 1997, Feine, S., MacIntye, B., Haupt, M. and Solomon, E. Windows on the wold: 2d windows fo 3d augmented eality. In Poc. UIST 93 (Atlanta, 1993), Heath, C. and Luff, P. Disembodied conduct: communication though video in a multimedia envionment. In Poc. CHI 91 (New Oleans, LA, 1991), Hindus, D., Ackeman, M., Mainwaing, S. and Sta, B., Thundewie: A field study of an audio-only media space. In Poc. CSCW 96, (Nov. 1996), Hoffman, H. Physically touching vitual objects using tactile augmentation enhances the ealism of vitual envionments. In Poc. VRAI 98, (Los Alamitos, CA, 1998), Ishii, H. and Ulme, B. Tangible bits: towads seamless intefaces between people, bits and atoms. In Poc. CHI 97 (Atlanta GA, 1997), Kanade, T., Kano, H. Kimua, S., Yoshida, A. and Kazuo, O. Development of a video-ate steeo machine. In Poc IRSC, (Pittsbugh PA, Aug. 1995), Kato, H. and Billinghust, M. Make tacking and HMD calibation fo a video based augmented eality confeencing system. In Poc. IWAR (San Fancisco CA, 1999), Kato, H., Billinghust, M., Poupyev, I., Imamoto, K. and Tachibana, K. Vitual object manipulation on a table-top AR envionment. In Poc. ISAR 00 (Munich, Gemany, 2000), Lauentini, A. The visual hull concept fo silhouette based image undestanding. IEEE PAMI 16, 2 (Feb. 1994), Lok, B. Online model econstuction fo inteactive vitual envionments. In Poc Symp. Inteactive 3D Gaphics 01, (Chapel Hill NC, Ma. 2001) Matusik, W., Buehle, C., Raska, R., Gotle, S.J. and McMillan, L. Image-based visual hulls. In Poc. SIGGRAPH 00, (New Oleans LA, 2000), Mulligan, J., Isle, V. and Daniilidis, K. Pefomance evaluation of steeo fo telepesence. In Poc. ICCV 01 (Vancouve, 2001) Mulligan, J. and Daniilidis, K. Real time tinocula steeo fo tele-immesion. In Poc. ICIP 01 (Thessaloniki, Geece 2001) Ogi, T., Yamada, T., Tamagawa, K., Kano, M. and Hiose, M. Immesive Telecommunication Using Steeo Video Avata. In Poc IEEE VR 01 (Yokohama, Japan, Ma. 2001) Poupyev, I., Tan, D., Billinghust, M., Kato, H. and Regenbecht, H. Tiles: A mixed eality authoing inteface. In Poc.Inteact 01 (Tokyo, Japan, 2001), Raska, R., Welch, G., Cutts, M., Lake, A., Stesin, L. and Fuchs, H. The office of the futue: A unified appoach to image based modeling and spatially immesive displays. In Poc. SIGGRAPH 98 (Olando FL, 1998), Seitz, S.M. and Dye, C.R. View mophing. In Poc. SIGGRAPH 96 (New Oleans LA, 1996), Sellen, A. and Buxton, B. Using spatial cues to impove videoconfeencing. In Poc. CHI 92, (Monteey CA, 1992) Sellen, A. Remote Convesations: The effects of mediating talk with technology. Human Compute Inteaction 10, 4 (1995), Slate, M., Howell, J., Steed, A., Petaub, D-P., Gaau, M. and Spingel, S. Acting in vitual eality. In Poc. CVE 00 (San Fancisco, CA 2000), Singe A., Hindus, D., Stifelman, L. and White, S. Tangible Pogess: less is moe in somewie audio spaces. In Poc. CHI 99, (Pittsbugh PA, 1999) Takemua, M. and Ohta, Y. Diminishing head-mounted display fo shaed mixed eality. In Poc. ISMAR 02 (Damstadt, Gemany, 2002). 34. S. Whittake and B. O Connaill. The ole of vision in face-to-face and mediated communication. Video Mediated Communication, (Finn, K., Sellen, A., Wilbu, editos) LEA, New Jesey,