Shared Imagination: Creative Collaboration in Mixed Reality. Charles Hughes Christopher Stapleton July 26, 2005

Size: px

Start display at page:

Download "Shared Imagination: Creative Collaboration in Mixed Reality. Charles Hughes Christopher Stapleton July 26, 2005"

Ralph Taylor
5 years ago
Views:

1 Shared Imagination: Creative Collaboration in Mixed Reality Charles Hughes Christopher Stapleton July 26, 2005

2 Examples Team performance training Emergency planning Collaborative design Experience modeling Scientific virtualization Guided rehabilitation Science Center groups Entertainment (design and experience) Creative Collaboration 2

3 What We Must Support Jointly visualized what if? scenarios Face-to to-face interactions Alternative POVs Personal and group creation Enablers Tangible and tactile components Constructive distractions (sandboxes) Shared display (1 st & 3 rd person views) Shared audio landscape Scalability, interoperability & portability Creative Collaboration 3

4 MR Contexts Physical Reality (PR) real world Virtual Reality (VR) purely synthetic Augmented Reality (AR) virtual assets registered in real world Augmented Virtuality (AV) real (people, props) layered in virtual space Creative Collaboration 4

5 Mixed Reality Continuum Creative Collaboration 5

6 Creative Process in MR PR is constrained by physical space VR limits person-to to-person expression and context of PR AR and AV support rich layering Collaboration often requires moving smoothly along the MR continuum Creative Collaboration 6

7 Milgram + Imagination Milgram s Reality-Virtuality Continuum Augmented Reality (AR) Augmented Virtuality (AV) Aristotle s- Media- Imagination Continuum P.T. Barnum Reality- Imagination Continuum Traditional Theme Park Physical Reality (Interface) Magic Show Compelling Mixed Reality (The Play) Imaginary Reality (Story) Virtual Reality (Media) Film Novel Creative Collaboration 7

8 Visual Rendering / Capture HMDs Dome Screens Demo Dome Optical Flat World MR Windows Video Light capture Creative Collaboration 8

9 MR Video See-Through

10 Audio Capture / Rendering Surround Hydrophones Designing Mixing Holophone Delivery in constrained settings Creative Collaboration 10

11 Special Effects - Colorkinetics SmartJack3 (USB to DMX) - Colorkinetics JuiceBox2 / icolor MR Lights - Gilderfluke MP3-50/40-4 Channel Dimmer Packs - Pneumatic / Smoke System - Sound Transducers ( Bass Shakers ) Creative Collaboration 11

12 Tracking Technologies Magnetic Optical Vision (often with markers/features) Acoustical Inertial Hybrid (hardware and soft/hardware) Calibration Creative Collaboration 12

13 Registration / Illumination Virtual and real must be properly placed relative to each other Inter-occlusion must be properly managed Mutual shadowing must occur, including shadows from real caused by virtual light The effects of ambient light (real and virtual) must be rendered Creative Collaboration 13

14 Story Virtual characters must have appropriate behaviors, reactive and proactive If appropriate, guidance should be provided to user(s) ) to attract them to overlooked activities / objects Replay must be provided for entertainment, review, etc. Creative Collaboration 14

15 Story/Play/Game Convergence Story Exposure Question Memorable Experience Play Share Game Expression Understanding Create Creative Collaboration 15

16 MR Software Reality B) Sensor Data C) Plug-Ins D) Scripted Behaviors A) MRSS Graphics Engine Audio Engine SFX Engine Story Engine After Action F) Experience Capture E) Multi-Sensory Augmentation Creative Collaboration 16

17 MR Software Suite Platform neutral Story Engine Agent-based Plug-in architecture Audio Engine Constraint-based adaptation (speaker placement) Graphics Engine DMX Engine Experience capture After Action Review Human Experience Modeler Creative Collaboration 17

18 An Innovative Interface: The Demo Dome

19 MS Demo Dome Placing Real Entities into Virtual Environments Creative Collaboration 19

20 Demo Dome Characteristics Lightweight Relatively Inexpensive Tolerant of lighting changes Distinct POVs E.g., consider a city planner and an architect Both need a common context (the cityscape) Each wants specialized information (codes vs physical / aesthetic characteristics) Creative Collaboration 20

21 Examples

22 Peer Collaboration Creative Collaboration 22

23 Along the Continuum Start in PR look at current plant Move to AR add new equipment and new windows Individuals jump to VR privately review designs Move to AV see each other while surrounded by new design Creative Collaboration 23

24 Trainer/Trainee Collaboration Creative Collaboration 24

25 Military Training Creative Collaboration 25

26 MR MINI-MOUT MOUT Creative Collaboration 26

27 VR Actors VR Sets Vehicular SFX Sniper Hostiles CGF HUD FFW Multi-Player MOUT VR Cues Merging Realities Creative Collaboration 27

28 Trainer View and Input Multiple POVs VR, MR, PR Auditory commands and cues Taps on shoulder (haptic( vest) Visual cues Visual and direct auditory interaction After action review Creative Collaboration 28

29 Free Choice Learning Creative Collaboration 29

30 Water s s Journey through Creative Collaboration 30

31 MR Sea OSC MR ROVER MR Content Creative Collaboration 31

32 Parent / Child Interaction Real Relevant Relational Creative Collaboration 32

33 Human Experience Modeler

34 Creative Collaboration 34

35 Rehabilitation: Blaine s s Breakfast Creative Collaboration 35

36 Rehab Specialist Input Auditory commands and cues Tap on shoulder (haptic( vest) Visual cues Visual and direct auditory interaction After action review Creative Collaboration 36

37 We wish to thank our partners

ucf.edu Home Page: http://www.cs.ucf.edu/~ceh/ Graphics Lab: http://graphics.

38 Charles E. Hughes School of Computer Science & Media Convergence Lab & School of Film and Digital Media & Text and Technology Program (English) Home Page: Graphics Lab: Media Convergence Lab: Creative Collaboration 38

39 Christopher B. Stapleton Media Convergence Lab & Institute for Simulation and Training & School of Film and Digital Media & Simiosys Home Page: Media Convergence Lab: Creative Collaboration 39

Authoring & Delivering MR Experiences

Authoring & Delivering MR Experiences Matthew O Connor 1,3 and Charles E. Hughes 1,2,3 1 School of Computer Science 2 School of Film and Digital Media 3 Media Convergence Laboratory, IST University of