Virtual Immersion Facility (VIF) A Training Perspective

Transcription

1 Virtual Immersion Facility (VIF) A Training Perspective 28 February 2008 Defence Research and Development Canada Recherche et développement pour la défense Canada Canada

2 Who we are System of Systems (SoS) Section in DRDC - Valcartier Simulation and Comprehension of Complex Situation (SCSC) Group Team composed of 3 defense scientists and 1 computer support Dr. Marielle Mokhtari François Bernier Éric Boivin Frédéric Drolet

3 Rational SCSC aims exploiting the Synthetic Environment (SE) for DND/CF (immersive environments, virtual reality, advanced visualization, simulation, serious gaming, haptic devices) We found out that there was no high-end general purpose immersive environment in the DND/CF Then we decided to create the Virtual Immersive Facility (VIF)

4 Virtual Immersive Facility Mission Explores the potential of advanced visualization, interactive simulation & immersive capabilities for the DND/CF Description Supports experimentations (or training), allows testing of applications, and provides computer infrastructures (140 m²) to many projects simultaneously The CAVE is the main component; the VIF also includes many other advanced devices Operational since August 2006 Investment of 1.5 M$

5 What is a CAVE? Developed in 1992 at the University of Illinois at Chicago Cave Automatic Virtual Environment (CAVE) Immersive virtual reality environment Room whose walls surround a viewer with projected images (Cruz-Neira et al., 1992) In the CAVE, you are no longer on the outside looking in but on the inside looking out (DeFanti)

6 Main advantages High degree of immersion by using advanced human machine interfaces (visual, haptic, tracking, odors, etc.) High degree of realism More natural interaction with the virtual environment High degree of presence (the sense of being in the virtual environment)

7 Visual Immersion - Full Field of View Synthetic Environment 1

8 Visual Immersion - Full Field of View Synthetic Environment 1 Synthetic Environment 2

9 Visual Immersion Full Field of View Synthetic Environment 1 Synthetic Environment 2

10 Visual Immersion - Continuous display Synthetic Environment 1 Synthetic Environment 2

11 Visual Immersion Perspective Matters Synthetic Environment 1 Synthetic Environment 1 Synthetic Environment 2

12 Visual Immersion Perspective Matters Synthetic Environment 1 Synthetic Environment 1 Synthetic Environment 2 Synthetic Environment 2

13 Visual Immersion - Continuous display Synthetic Environment 1 Synthetic Environment 1 Synthetic Environment 2 Synthetic Environment 2

14 Visual Immersion Perspective matters External perspective Immersed perspective

15 Interaction - Simulated Weapon (1)

16 Interaction Hands & Other Devices (2)

17 The presence Defined as the participant s sense of being there in the virtual environment (Bartfield et al., 1995) Participants feel they are part of the simulated world because they forget that their perceptions are mediated by technologies (perceptual illusion of non-mediation) The most important characteristic of immersive systems Facilitated by the immersion, the realism of the environment and natural interaction with the virtual environment It can be used to create stress, lost of equilibrium, fear, trigger reflex, increase concentration, motivation & interest, involvement in the situation, facilitate learning, better transmit the complexity of a situation, etc. Allows to integrate physical with cognitive aspects

18 Flex CAVE in Valcartier Reconfigurable "Holodeck" Immersive Mode Immersive Theatre Mode Flat Wall Mode Characteristics Virtual immersion system Flex by Fakespace: 3 walls and 1 floor of large size (2.4 m 3.3 m 3.3 m) High intensity (8000 Lumens) and high resolution ( pixels) stereoscopic projectors 3D realistic sound effects Real-time user tracking

19 Approach Focus on performance, flexibility & robustness Exploit high-end multi-screen visualization & immersion system Emphasize on technologies integration: visualization, simulation & interaction Exploit methodologies to improve application development Rely on VR innovations Exploit gaming

20 Applications Improve team-based activities by facilitating the collaboration & the development of a shared perspective (IMAGE project: team supported with interactive simulation & visualization) Provide more realistic & flexible training capability at lower cost (e.g.: stress inoculation, sensitive site exploitation, tactical combat casualty care, foreign culture ) Recreate the battlefield in a realistic way which can help in the treatment of post traumatic stress disorder treatment Facilitate extraction of meaningful patterns from (massive) datasets, help to detect the expected & discover the unexpected Improve situational awareness, mission planning / rehearsal & after action review Facilitate engineering review

21 Sensitive Site Exploitation Who SSE trainers Problem Existing training is mainly theoretical Training based on live simulation is very limited Exploration Since it is a highly spatial task, could the CAVE strengthen traditional Intelligence staff training by adding interactive experiments inside diversified, rich & realistic context-driven environments? Focused on the technical feasibility of anticipated required functionalities : CAVE, serious gaming, scenarios Demonstrated: interaction metaphors, fast scenario development, task realism, etc.

22 SSE Some examples

23 SSE Some examples Sound & scripts Object Move

24 SSE Some examples Sound & scripts Object Move

25 SSE Some examples Sound & scripts Object Move

26 Sound & scripts Object Move

27 Medic Training Who Centre Medical de Simulation des FC (CFB Valcartier) Problem Difficulty to train medics in an operational context Live simulation training is too constraining Exploration Could we exploit the CAVE to supplement the traditional medic (instrumented dummy) training with a combat situation? Tested with military and civilian scenarios

28 Medic Training

29 Immersive vs. non immersive simulations Who Capt. Mike Lakatos & John Benson form the tech staff Objective Assess the capabilities of immersive and non-immersive simulation platforms to assist training as outlined in the Training Needs Framework (TNF) Exploration Detachment level training (30 part.): 1 (CAVE) + 4 (desktop) Reinforcement to platoon operation & vehicle checkpoint operation scenarios 20 days of preparation + 4 days of training

30 Immersive vs. non immersive simulations Many Interesting lessons learned (observations) Familiarization faster than with desktop Gaming perception: lower than on desktop Improve motivation Trigger reflexes (e.g.: crouching when shoot) Increase fatigue level but also the switch off level Can easily create information overload Increase threat level perception Training combination of cognitive & physical levels

31 Knowledge Wall

32 Dr Marielle Mokhtari Defence Scientist Defence R&D Canada Valcartier (418) (4282) VIF Points of Contact François Bernier Defence Scientist Defence R&D Canada Valcartier (418) (4346) Frédéric Drolet Computer Support Defence R&D Canada Valcartier (418) (4351) Éric Boivin Defence Scientist Defence R&D Canada Valcartier (418) (4394)

33 Why not Head-Mounted Displays 1 (HMD)? Category Price FOV Resolution D equivalent Low-Cost <1k$ 25º 320x x600 Professional 1-10k$ 45º 800x x1024 Expert 10-50k$ 60º 1024x x High Performance k$ 112º 1024x x Laurendeau, D, Branzan-Albu, A., Boivin, E., Drouin, R., Martel, H., Ouellet, D., Schwartz, J.M. Survey of the State-of-the-Art on Synthetic Environments, Sensori-Motor Activities in Synthetic Environments, Simulation Frameworks and Real-World Abstraction Models, Contract W /QCA, Laval University, 2003.

34 CAVE Equivalent to HMD HMD <1k$ HMD 1-10k$ HMD 10-50k$ HMD k$ CAVE CAVE CAVE CAVE Ex: 25º FOV is equivalent to a user standing at 30 feet from the CAVE front screen

35 Comparison with HMD and CRT CAVE HMD Min HMD Max CAVE CRT CAVE DLP Resolution 640x x x x1050 Contrast 75:1 300:1 N/A 2000:1 Brightness 50 cd/m cd/m 2 ~10 cd/m 2 ~400 cd/m 2 FOV 24º 112º 270º 270º Latency ms ms 0 ms 0 ms

36 Virtual Immersive Facility - Drawing Server Area Development, Test & Control Area Experimentation Area

37 Connection Schema SGI Onyx 4 BOXX B BOXX A Video Card 1 Video Card 2 Video Card 3 Video Card 4 Sound Card Video Card 1 Video Card 2 Sound Card 5.1 Video Card 1 Video Card 2 Sound Card 5.1 Stereo sync in Stereo sync in LED Stereo Signal Switchbox Micro in Serial USB in 1 DVI out 2B DVI out 2A DVI out 1B DVI out 1A Stereo sync out Center + Subwoofer Micro in Rear L+R Front L+R Serial USB in 1 DVI out 2B DVI out 2A DVI out 1B DVI out 1A Stereo sync out Center + Subwoofer Micro in Rear L+R Front L+R Serial USB in 1 DVI out 2B DVI out 2A DVI out 1B DVI out 1A Stereo sync out Stereo IR signal generator Ethernet Center + Subwoofer Rear L+R Front L+R Optical DVI out 4B DVI out 4A DVI out 3B DVI out 3A Ethernet Ethernet RS-232 Stereo sync out Stereo sync in KVM 4x8 (UNIMUX-4X8-U) USB out 1 USB in 1A USB in 1B Center + Subwoofer Rear L+R Front L+R Microphone Center + Subwoofer Rear L+R Front L+R Microphone USB out 2 USB in 2A USB out 3 Analog Input Analog Input USB in 2B Home Theater Connect DTS-610 Home Theater Connect DTS-610 Digital Output Digital Output USB out 4 Optical S/PDIF Coaxial S/PDIF Optical S/PDIF Coaxial S/PDIF USB out 5 USB out 6 USB in 3A USB in 3B USB out 7 I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 LAN_A LAN_B A B C D E F AI1 AI2 AI3 AI4 A B Input Input CGIU Right CGIU Left USB mouse USB keyboard Video out PC Programming Port (RS232) C2ENET-2 C2ENET-2 Serial Serial KVM Digital audio S/PDIF Digital audio S/PDIF Digital audio Toslink Optical Digital audio Toslink Optical Digital audio RJ-45 CAT5 Analog audio RJ-45 CAT5 A.audio RCA (Front left) A.audio RCA (Front right) A.audio RCA (Rear left) A.audio RCA (Rear right) RS232/422/485 RS232/422/485 RS232/422/485 RS232/422/485 RS232/422/485 RS232/422/485 Rack-Mount for Immersion equipment Crestron Control System Serial Serial Video in #1 USB in #1 Video in #2 USB in #2 Input Ethernet PC Digital Audio Processor Output RS232 Lutron Graphic Eye Main Unit USB VGA Output MUX Link 1 Zone 1 Powerbar Power in A.audio RCA (Subwoofer) A.audio RCA (Center) A.audio RCA (Front left) A.audio RCA (Front right) A.audio RCA (Rear left) A.audio RCA (Rear right) Zone 2 MUX Link 2 Zone 3 MUX Link 3 Lutron GRX-232 MUX RS232 Link 1 MUX Link 2 RS232 MUX Link 3 AO6 AO5 AO1 AO2 AO3 AO4 Crestron Display Wall Control USB VGA Output Power FCU Ethernet HUB Ether#2 Ether#1 Ethernet 2.4 Ghz 10 mw Channel Audio Channel Audio Channel Audio Channel Audio Channel Audio Channel Audio Audio Channel RS-232 Emitter Only Receiver Only Grasp Serial Ethernet Input Audio Amplifier Output IS-900 Wand IS-900 Wireless Receiver Audio Channel Audio Channel Audio Channel Audio Channel Audio Channel Audio Channel Audio Channel Power RJ-48 RS-232 Ethernet Ethernet RS-232 Video Ethernet RS-232 Video Ethernet RS-232 Video Ethernet RS-232 Video 2.4 Ghz 10 mw Emitter Only Receiver Only IS-900 Controler IS-900 Wireless Receiver RJ-48 RJ-48 RJ-48 RJ-48 RJ-48 RJ-48 RJ-48 RJ-48 Floor Screen Projector Power RJ-48 Front right speaker Center speaker Front left speaker Middle Screen Right Screen Projector Projector Main Projectors Left Screen Projector Mobile station USB out 1 IS-900 Head-Tracker VGA in 1A VGA in 1B USB out 2 RJ-48 Rear right speaker Subwoofer Rear left speaker DVI in DVI in DVI in DVI in IS-900 Left-Hand Tracker Floor Screen Monitor RJ-48 Middle Screen Right Screen Monitor Monitor Control Monitors Left Screen Monitor Control station Ethernet USB out 1 USB out 2 Sonicstrip Sonicstrip IS-900 Right-Hand Tracker DVI out 1 DVI out 2 USB in 1 USB in 2 HD15 in 1B HD15 in 2B DVI in 1A DVI in 2A Development station #1 Ethernet USB out 1 USB out 2 DVI out 1 DVI out 2 USB in 1 USB in 2 HD15 in 1B HD15 in 2B DVI in 1A DVI in 2A Development station #3 Ethernet USB out 1 USB out 2 DVI out 1 DVI out 2 USB in 1 USB in 2 HD15 in 1B HD15 in 2B Development station #2 DVI in 1A DVI in 2A Ethernet USB out 1 USB out 2 DVI out 1 DVI out 2 USB in 1 USB in 2 HD15 in 1B HD15 in 2B DVI in 1A DVI in 2A Development station #4 Ethernet USB out 1 USB out 2 DVI out 1 DVI out 2 USB in 1 USB in 2 HD15 in 1B HD15 in 2B DVI in 1A DVI in 2A USB out 8 USB in 4A USB in 4B Input RS232 Video Matrix 16x16 RS422 Output O1 O2 O3 O4 O5 O6 O7 O8 O9 O11 O10 O13 O12 O15 O14 O16 Video Scaler VGA Video Scaler VGA Video Scaler VGA Floor Trap connectivity VGA out 1 VGA out 2 VGA out 3 VGA in 1 VGA in 1 VGA in 2 VGA in 3 VGA out 1 USB out 1 USB out 2 USB in 1 USB in 2 Stereo out Micro out Stereo in Micro in Phone Phone Ethernet 1 Ethernet 2 Ethernet 1 Ethernet 2 Power in 1 Power in 2 Power out 1 Power out 2

38 Future Battle Commanders with Advanced Decision Making Capabilities What if Combination if a commander could of of advanced virtually step visualization step into into an & interaction an actual operational techniques environment before and and simulation sending tools troops To in? To support in? What commander s if if he he could plan need plan his for for mission his entire mission by planning/rehearsal, by walking through each SA SA & step AARstep & AARand to manoeuvre, pointing to assist commanders out in out the in their the most their probable point of of retaliation? By By bringing resources such as as Google decision-making process. Earth & using the the latest in in gaming technology, it it is is possible to to develop the the future of of battle command planning...

39 Context Existing Limitations (1) (1) Applications can can generate easily 3D 3D models of of the the battlefield display on on 2D 2D monitors. (2) (2) Huge of of information to to understand, manipulate & structure display on on (small) 2D 2D monitors. (3) (3) No No easy way way to to interact with with a running simulation in in order to to change parameters value at at one one specific point to to rapidly see see the the impact Need of of an an advanced capability to to immerse commanders in in a SE SE supporting interaction visualization --simulation (& (& modeling). Idea Idea--Simultaneously increase the the overall comprehension of of situation information through simulation & intuitive visualization and and improve the the mechanisms for for user user perception interaction. Expected Outcome --Improve future commanders ability to to make (C2) (C2) decisions by by permitting interactive & dynamic virtual 3D 3D course of of action analysis, mission rehearsals, and and mission planning so so they they may may act act decisively in in the the face face of of quickly changing circumstances.

40 Adequacy (Large scale) immersive environments are are considered beneficial for for data data analysis & visualization. Increase the the volume of of data data available to to commander but butimprove perception of of these data. Immersive 3D 3D visualization stereoscopic technologies Improve the the commander s ability to to comprehend this this avalanche of of information and and act act upon it it quickly & decisively. Interactive simulation Human-in-the-loop Improve the the commander s ability to to control, to to act act upon the the simulation, then then to to control, to to act act upon scenario & models.

41 3D 3D virtual representations of of battle scenarios provide commanders with with a more reliable & accurate means of of viewing information on on the the battlefield (in (in RT). RT). Scenario simulation control adds a new new dimension in in the the understanding of of the the situation. Immersive view view of of a fictitious replica of of a specific site site enables commanders to to comprehend complex environment that that his his troops face face during operations (intuitive & efficient 3D 3D exploration of, of, navigation in, in, learning about). about). Using a combination of of technological innovations in in simulation, visualization & interaction, commanders can can quickly gain gain an an understanding of of evolving situations, explore their their options and and communicate their their plans.

42 U.S Activities CERDEC C2D CAVE Lab. The U.S. Army Communications Electronics Research Development & Engineering Command (CERDEC) Command & Control Directorate (C2D) has developed immersive C2 applications to support the future commander s ability to make decisions faster & more accurately. R&D in the application of immersive technologies & advanced visualization techniques To assist commanders in their decision-making process C2 Applications Basic functionalities Terrain models that provide the 3D simulation of the modeled area; User interface that permits selection of the operating mode: live or playback; Event Data Manager which receives, stores & forwards tactical messages; Presentation software which displays the scenes & permits the user to interact with the immersive environment

43 U.S Activities C4ISR On-The-Move 2006 Experiment Mission Planning & Execution - Soldiers response to technology Three leaders stated that the planning session was a great benefit to their planning. The Platoon Leader estimated that the technology improved his planning capabilities by 75%. The Platoon Sergeant stated that the technology is outstanding if it is accurate and up to date. He thought the colors & shapes of the buildings & the layout of the streets was very helpful. His estimate of how this improved their planning was 70%. The Robotics NCO estimated that the ROVR planning was 80% better than using a sand table. On the morning of the MOUT mission, the Platoon Sergeant was confident in the locations of the buildings of interest in the area & knew where his soldiers & unmanned assets were to be placed. This technology provided Defence R&D an Canada excellent R & tool D pour for la défense the Canada soldiers to become familiar with the layout of the MOUT site.

44 (Hypothesis) Mission Planning Session - MOUT Obtain a general view/orientation of the layout of the MOUT; Identify suspected enemy strongholds & safe houses; Determine the best locations to place UGS; Identify potential sniper positions; From an aerial view Identify routes leading into & out of the MOUT site; By freely navigating within the environment, use a bird-eye view of the MOUT street layout for improved orientation; Based on intelligence reports, predict where enemy forces would enter the site & able to determine approximate locations for UGS; Closer look at the terrain navigate down to a dismount or first person perspective street level for a closer inspection (buildings, streets, configuration of a square ) walk through the streets to determine best routes, identify specific buildings & identify any obstacles that may impede interrupt movements of troops walk around the suspected enemy safe houses and strongholds to determine entrance ways, windows & learn characteristics of each building able to identify a foothold, to learn the general Defence R&D layout Canada of this R & location D pour la défense and Canada to identify routes of ingress and egress to/from the building.

45 SANS CLASSIFICATION COTE DE SÉCURITÉ DE LA FORMULE (plus haut niveau du titre, du résumé ou des mots-clefs) FICHE DE CONTRÔLE DU DOCUMENT 1. PROVENANCE (le nom et l adresse) R&D pour le défense Canada - Valcartier 2. COTE DE SÉCURITÉ (y compris les notices d avertissement, s il y a lieu) Sans classification 3. TITRE (Indiquer la cote de sécurité au moyen de l abréviation (S, C, R ou U) mise entre parenthèses, immédiatement après le titre.) Virtual Immersive Facility - A Training Perspective 4. AUTEURS (Nom de famille, prénom et initiales. Indiquer les grades militaires, ex.: Bleau, Maj. Louis E.) BERNIER, F., Boivin, E., Mokhtari, M., 5. DATE DE PUBLICATION DU DOCUMENT (mois et année) a. NOMBRE DE PAGES 44 6b. NOMBRE DE REFERENCES 0 7. DESCRIPTION DU DOCUMENT (La catégorie du document, par exemple rapport, note technique ou mémorandum. Indiquer les dates lorsque le rapport couvre une période définie.) Présentation 8. PARRAIN (le nom et l adresse) 9a. NUMÉRO DU PROJET OU DE LA SUBVENTION (Spécifier si c est un projet ou une subvention) 9b. NUMÉRO DE CONTRAT 10a. NUMÉRO DU DOCUMENT DE L ORGANISME EXPÉDITEUR 10b. AUTRES NUMÉROS DU DOCUMENT N/A 11. ACCÈS AU DOCUMENT (Toutes les restrictions concernant une diffusion plus ample du document, autres que celles inhérentes à la cote de sécurité.) Diffusion illimitée Diffusion limitée aux entrepreneurs des pays suivants (spécifier) Diffusion limitée aux entrepreneurs canadiens (avec une justification) Diffusion limitée aux organismes gouvernementaux (avec une justification) Diffusion limitée aux ministères de la Défense Autres 12. ANNONCE DU DOCUMENT (Toutes les restrictions à l annonce bibliographique de ce document. Cela correspond, en principe, aux données d accès au document (11). Lorsqu une diffusion supplémentaire (à d autres organismes que ceux précisés à la case 11) est possible, on pourra élargir le cercle de diffusion de l annonce.) dcd03f ver.( ) SANS CLASSIFICATION COTE DE LA SÉCURITÉ DE LA FORMULE (plus haut niveau du titre, du résumé ou des mots-clefs)

46 SANS CLASSIFICATION COTE DE LA SÉCURITÉ DE LA FORMULE (plus haut niveau du titre, du résumé ou des mots-clefs) 13. SOMMAIRE (Un résumé clair et concis du document. Les renseignements peuvent aussi figurer ailleurs dans le document. Il est souhaitable que le sommaire des documents classifiés soit non classifié. Il faut inscrire au commencement de chaque paragraphe du sommaire la cote de sécurité applicable aux renseignements qui s y trouvent, à moins que le document lui-même soit non classifié. Se servir des lettres suivantes: (S), (C), (R) ou (U). Il n est pas nécessaire de fournir ici des sommaires dans les deux langues officielles à moins que le document soit bilingue.) This document presents the Virtual Immersive Facility in DRDC Valcartier, some of its technologies and how it can improve the training in the CF. Some exploratory projects are also presented. 14. MOTS-CLÉS, DESCRIPTEURS OU RENSEIGNEMENTS SPÉCIAUX (Expressions ou mots significatifs du point de vue technique, qui caractérisent un document et peuvent aider à le cataloguer. Il faut choisir des termes qui n exigent pas de cote de sécurité. Des renseignements tels que le modèle de l équipement, la marque de fabrique, le nom de code du projet militaire, la situation géographique, peuvent servir de mots-clés. Si possible, on doit choisir des mots-clés d un thésaurus, par exemple le Thesaurus of Engineering and Scientific Terms (TESTS). Nommer ce thésaurus. Si l on ne peut pas trouver de termes non classifiés, il faut indiquer la classification de chaque terme comme on le fait avec le titre.) Virtual Immersive Facility, Presence, Immersion,Training SANS CLASSIFICATION COTE DE SÉCURITÉ DE LA FORMULE (plus haut niveau du titre, du résumé ou des mots-clefs) dcd03f ver.( )