Mixed Fantasy Delivering MR Experiences

Transcription

1 Mixed Fantasy Delivering MR Experiences Charles E. Hughes, CS + Film/DM + MCL, Chief Geek Christopher B. Stapleton, IST MCL + Digital Media, Chief Freak, creative lead at Islands of Adventure Paulius Micikevicius, CS, Post Doc, now at Armstrong Darin E. Hughes, IST MCL, Audio Producer Scott Malo, IST MCL, Modeler and Content Producer Matthew O Connor, IST MCL, Lead Programmer

2 The Ambiance Send us your best students, please Institute for Simulation and Training Visual System Laboratory (Moshell 1990) Media Convergence Laboratory (Stapleton 1999) Computer Science Graphics and Distributed Processing School of Optics ODALab (Rolland) Psychology MITMIT (Peter Hancock) Industrial Engineering Synergy (Kay Stanney) + Film & Digital Media, English (Text & Technology) Jan Mixed Fantasy 2

3 Mixed Fantasy Delivering MR Experiences MR AQUARIUM TIME PORTAL MS ISLE MR MOUT

4 Basic Concepts Story and multi-modality modality invite the user s s imagination along for the ride Allow multiple experience levels

5 Mixed Fantasy Continuum P.T. Barnum Reality- Imagination Continuum Milgram s Reality-Virtuality Continuum Augmented Virtuality (AV) Traditional Theme park Physical Reality Magic Show Compelling Mixed Reality (The Play) Imaginary Reality Augmented Reality (AR) Virtual Reality Film Novel Aristotle s- Media- Imagination Continuum Jan Mixed Fantasy 5

6 Mixed Reality VIRTUAL REALITY AUGMENTED REALITY AUGMENTED VIRTUALITY Jan Mixed Fantasy 6

7 Divers, Swimmers, Waders QuickTime and a TIFF (LZW) decompressor are needed to see this picture. Jan Mixed Fantasy 7

8 Production Stories

9 Time Portal Jan Mixed Fantasy 9

10 MS Isle Jan Mixed Fantasy 10

11 MS Isle Collaboration Jan Mixed Fantasy 11

12 Demo Dome Kiosk Portable & Collaborative Jan Mixed Fantasy 12

13 Geeks and Freaks Storyboard Animatic VR experience (linear, non-interactive) VR experience (interactive) MR experience Working prototypes at each stage Feedback at each stage Jan Mixed Fantasy 13

14 Human Factors Studies Soon, really It s s in the mail

15 MR MOUT Jan Mixed Fantasy 15

16 Physical Sets and Activation Lights and glass responsive to gun shots -- audio and FX Blue screen replaced by synthetic backdrops and used for clipping 3d models SFX like Smoke Effects appear to respond to virtual explosion. Jan Mixed Fantasy 16

17 Planned HF Testing ARI: Multi-modal cues Evaluate effectiveness of Multi-modal (Audio, Visual, A/V, Haptic) Cues Spatially, Directionally, Non-Spatial and non-directional. M&S: Mixed Reality Training Effectiveness Compare effectiveness of MR spectrum of training (Live, Desktop, VR, MR) with target acquisition under stress. MCL: MR Audio Perception Audio semantics, Perception of 3D audio display in Y-Axis, Y Audio/Visual deception in 3D audio display MIT 2: Leader Decision Making Preliminary (MRM 2.5): Time Pressured Decision-making with 3D Spatially registered Visual Cues (vs text-based) Secondary (MRM 3.0): Unit Leader Decision Making in MOUT training. Jan Mixed Fantasy 17

18 ARI Tests West Visual Cues Non Directional, Non Spatial Cue Fine Detail Spatial Cue Directional Cue Rough Detail Spatial cue Jan Mixed Fantasy 18

19 Science & Technology Integrating components Achieving interactive frame rates

20 Tools of the Trade Video See- Thru Display with Tracker Virtual Assets Desktop and Optical See- Thru Display with GPS Embedded Projection Display Jan Mixed Fantasy 20

21 The Pieces Graphics engine The workhorse must display and analyze Audio engines 3d engine and/or Multiple stereo engines Special effects (DMX controlled actuators) Sensor servers (trackers, light, sound, ) Story engine (SE) The only one who understands semantics Jan Mixed Fantasy 21

22 Delivery System Bringing it Together

23 Graphics Engine Capture real; augment with virtual Blend animations; manage path movement; Analyze scene and relation to interactions Communicate with SE Simple protocol SHARK_PATH MAKE PATH PATH1 SHARK_PATH LOOP SHARK MAKE MODEL JAWS SHARK ASSOCIATE SHARK_PATH SHARK SHOW SHARK LOOP SWIM 200 USER ASSOCIATE SHARK USER MOVETO Jan Mixed Fantasy 23

24 Delegated Rendering Show, Occlude, Hide, HUD Associate also is relevant Jan Mixed Fantasy 24

25 Challenge: Real-Time Mixing of Real / Virtual Placement of the virtual objects in the real scene Tracking / Registration Occlusion Blending the virtual objects into the real scene Rendering Lighting (even virtual lights on real objects) Shadows (all combinations of real and virtual) User interaction with the objects Selecting real/virtual objects Real done with impostors Jan Mixed Fantasy 25

26 Placement of Virtual Objects Occlusion of virtual by real and vice versa Approaches Markers (size estimates depth) Chroma-keying Occlusion models Computing depth by stereo vision for selected objects in scene Jan Mixed Fantasy 26

27 Chroma-Keying Allows only two layers behind/in front Appropriate for portals (doors, windows, etc.) Standard in film/tv studios Lighting is difficult to control in MR settings Jan Mixed Fantasy 27

28 Unidirectional Retro- Reflective Curtain and Personal Lighting Jan Mixed Fantasy 28

29 Depth from Stereo left right CPU: 10.9 ms GPU: 1.49 ms CPU + GPU: 1.3 ms CPU: ms Jan Mixed Fantasy 29

30 GPU-Based Algorithms Recently: 32-bit IEEE-standard floating point Performance trends (John Poulton) CPU: 1.7X in a year (14( months to double) GPU: 2.4X in a year (10( months to double) Likely to remain true in the next few years Parallelism 8 pipelines Each pipeline operates on 4-vectors Advanced functionality Single cycle: Sin and cos Linear interpolation Euclidean distance Jan Mixed Fantasy 30

31 Blending the Real and the Virtual Using color-transfer to simulate lighting Lighting the virtual objects Shadows Compute lights from the real scene Dynamic lighting subverts pre-computation Shading algorithms Quality vs. speed Cast by virtual objects Cast by real objects Jan Mixed Fantasy 31

32 Rendering and Illumination (real-time using GPU) 42 fps on Radeon 9800 Scene with 80,000 Triangles Jan Mixed Fantasy 32

33 Shadows with Virtual Light with shadow Jan without shadow Mixed Fantasy 33

34 Capturing Real Light Lady Bug Acquires Dynamic Lighting Jan Mixed Fantasy 34

35 Audio Engine(s) Communicate with SE using multicast Run forever, once started Provide audioscape Synthetic audio: Ambient sounds, SFX, voice commands Event-triggered triggered point source audio 3D Audio directional sound Surround sound Jan Mixed Fantasy 35

36 Challenges of Audio for Interactive Environments Must provide a means of merging both real and virtual sound sources Must be able to place audio in a truly 3D space Must have both pinpoint accuracy as well as diffused ambiance Must provide accurate acoustical simulation Jan Mixed Fantasy 36

37 DMX Engine(s) Communicate with SE using multicast UDP Can enter early or late Run forever, once started Talk to devices using DMX protocol Provide show effects Lights Smoke Moving objects (can that gets hit ) Even falling walls (Styrofoam) Jan Mixed Fantasy 37

38 Special Effects - Colorkinetics SmartJack3 (USB to DMX) - Colorkinetics JuiceBox2 / icolor MR Lights - Gilderfluke MP3-50/40-4 Channel Dimmer Packs - Pneumatic / Smoke System - Sound Transducers ( Bass Shakers ) Jan Mixed Fantasy 38

39 Story Engine Maintains ground truth Delegates behavior to various models Finite state Steering Anthropomorphic (ant, wolf, ) Constraint-based Rule-based Physics-based Supports after-action action review Jan Mixed Fantasy 39

40 Acknowledgements Naval Research Laboratory Canon Mixed Reality Laboratory Army RDECOM National Science Foundation Jan Mixed Fantasy 40

41 Computer Science Charles E Hughes -- ceh@cs.ucf.edu Paulius Micikevicius -- paulius@drake.armstrong.edu Institute for Simulation&Training Christopher B. Stapleton, + Digital Media -- chris@mcl.ucf.edu Darin E. Hughes -- darin@mcl.ucf.edu Scott Malo -- scott@mcl.ucf.edu Matthew O Connor -- matt@mcl.ucf.edu Jan Mixed Fantasy 41