Perception in Immersive Virtual Reality Environments ROB ALLISON DEPT. OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE YORK UNIVERSITY, TORONTO

Overview Basic concepts and ideas of virtual environments Concentrate on applications of immersive displays Human factors issues in virtual reality and telepresence

Some Trends in Human Computer Interaction Trend towards more naturalistic interaction Increasingly humans are in the loop Exponential rise in computing power but displays and input devices are still limited as is human processing capability

Natural modes of interaction Speech, 3D manipulation, gesture Natural modes of display Interactive, dynamic, immersive 3D imagery Simulated environments Mapping of abstract data to virtual worlds Natural interaction for entertainment, art and simulation

Virtual Environments (VEs) Virtual Reality coined by Lanier late 1980s Many and varied definitions for virtual reality (VR): Computer-generated environment with and within which people can interact. The advantage of VR is that it can immerse people in an environment that would normally be unavailable due to cost, safety, or perception restrictions. A successful VR environment offers users immersion, navigation, and manipulation.* Many prefer term Virtual Environment * "virtual reality." The Columbia Electronic Encyclopedia, 6th ed., 2012, Columbia University Press..

Non-immersive VE Displays Gaming, Google Earth, 3D web visualizations Fish-tank VR a window into a VE with head tracking Online persistent virtual environments (Second Life etc) We will focus exclusively on immersive VR today.

Historical Overview Science fiction Heileg s sensorama 1956 Multisensory motorbike simulator Stereoscopic imagery, stereophonic sound, seat vibration, wind, and odour to enhance the film

Sutherland s Head Mounted Display Binocular display wire frame display Mechanical tracker ( Sword of Damocles ) Tested in Bell helicopter Went on to graphics fame as cofounder of Evans and Sutherland

Aircraft Simulators Early air powered link trainer 1929 (Ed Link) Most mature and successful application of immersive displays Advanced, high fidelity visuals, motion Modern CAE Ltd. simulators

VR Applications

http://www.cs.unc.edu/~walk/ F. Brooks, project Grope Virtual Vietnam, Virtually Better Inc. www.vrac.iastate.edu/research/ visualization/tornado/ Idaho National Laboratory Center for Advanced Energy Studies VR in Psychotherapy for Social Phobias (Slater)

City Planning/Heritage Virtual Beijing (Iowa State) ADHD Treatment Virtual Reality Medical Centre Virtual String Quartet University of Barcelona Dinner Party, Stoudt, Wexler, and Soto

VR as a tool to study perception Conflict between ecological validity and experimental control Ideally would like subjects to perceive and interact with a natural environment that is experimentally controlled A VE allows more complex and immersive stimuli while maintaining control Need to be vigilant about artefact

https://en.wikipedia.org/w/index.php?curid=789586

Virtual environments allows experimenter additional flexibility to violate the laws of physics Correlations between senses or modalities Coupling between action and sensory consequences (reafference)

Elements of a Virtual Reality Goal is to simulate a compelling synthetic environment using displays/effectors in response to user actions sensed by sensors Technology is important but principally serves to mediate the experience of the reality VE not tied to any particular technology

The sensation-perception-action cycle of the user is an integral part of a VR system. Normally when one acts in the world feedback from the senses confirms the expected result (re-afference) Current VR systems have serious limitations that limit the ability to create high fidelity realistic virtual worlds.

Perception-Action Cycle Human Senses State/changes in the environment Natural Environment Perception Cognition Motor System Effectors Actions in the environment

VE s -Closing the Sensory Feedback Loop Virtual reality and other advanced interactive displays simulate and maintain a model of the world to be created or augmented (processor and models) present or display the world to the user (displays and effectors) sense the actions of the user and environmental state to enable the simulation to react (sensors) user experience is intimately involved through action, perception, expectation, motivation

VR s -Closing the Sensory Feedback Loop Human Senses Displays Virtual Environment State/changes in the Perception Cognition environment Simulation Motor System Effectors Sensors Actions in the environment

Elements of a Virtual Reality: Simulation A reality is a place that exists and that we can experience A synthetic or virtual world is created (structure & behaviour) Allows for realities and virtual worlds unlike our own Laws of physics etc. must be programmed and can be changed Intelligent (humanistic) entities: agents and avatars (incl. self) AlloSphere UCSB

Disaster Training, Methodist University

Elements of a Virtual Reality: Simulation Intelligent (Humanistic) Entities in VEs Avatars Agents Virtual embodiment of self or another real person in the VE Allow for user to visualize own body and movements in the VE Synthetic intelligent entities Users often respond to these as if they were avatars

Elements of a Virtual Reality: Displays Present sensory stimuli to the user: visual most common but also auditory, touch, Two broad classes are typical 1. Body fixed or referenced Mounted and move with the user Head mounted display 2. Earth fixed or referenced User moves with respect to fixed display A desktop display or CAVE is typical

Head Mounted Diplays https://www.flickr.com/photos/bagogames/26729095226 Visually-coupled system display driven by movement of head; images for current viewpoint (Furness, 1969)

Projection Based Displays

Elements of a Virtual Reality: Sensors VE Interaction Passive experience of the real world is rare Typically we actively explore and interact with our world VE s should provide the same interactivity

Elements of a Virtual Reality: Sensors To generate the displays, need to know user s position and orientation Need to track user s head (hand, body ) in real time in order to respond to head (hand, body ) motion in real time Current tracking does not measure degrees of freedom possible in human motion commons.wikipedia.org

Elements of a Virtual Reality: Sensors Gloves/Motion capture One of the early VR input devices was the Dataglove Typically many degrees of freedom Additional tracking for position Animation/gesture recognition Immersion Cybergrasp Vicon mocap Leap motion YORK UNIVERSITY 29

Elements of a Virtual Reality: User Immersion: degree to which sensory input to all modalities is controlled by the SE interface (Draper, Kabur & Usher 1998) Our experience of the real world is limited to what we obtain through our senses. www.vrac.iastate.edu

NASA flight simulator

Elements of a Virtual Reality: User Presence: The subjective experience of being physically present in the VE rather than the real environment Being there Considered important in telecommunications, film, VEs Sensory deprivation, mental disorders, drugs can affect presence in the real world

Some Perceptual Problems in VEs Displays and rendering issues: update rate, resolution field of view rendering/modelling realism and fidelity scaling and registration, convergence, accommodation, stereo, parallax noise, distortions, alignment real time feedback to user must be timely

Perception-Action Cycle Human Senses Displays Virtual Environment State/changes in the Perception Cognition environment Simulation Motor System Effectors Sensors Actions in the environment

Sensory Discrepancy Consequences: fatigue, nausea (simulator sickness), poor performance poor fidelity and distortions of virtual space image slip, blur or instability Display lag is a major problem in head-slaved displays: when head moves or vibrates display is not updated immediately recent systems go to great lengths to reduce effective latency

Some Perceptual Problems in VR Sensors Issues: resolution, smoothness, degrees of freedom Natural locomotion typically highly restricted Various solutions have been tried

Virtuix Omni Cyberwalk (MPI)

Razzaque et al (2001) EUROGRAPHICS 2001 https://youtu.be/thk92rev1va

York University 40

VR as a Tool to Study Perception Ideally would like both a natural and experimentally controlled world VR allows more complex and immersive but controlled stimuli Example: Simulating spatial auditory attention in a gaze contingent display: The virtual cocktail party Breakfast at Tiffany s (1961 Blake Edwards)

Technique Gaze as an indicator of current conversational interest How should focus be switched based on gaze? Highlight attended source and deemphasize distractors? Is it sufficient to manipulate the volume?

Gaze-Contingent Audio Display Gaze location

Results GCD audio highlighting greatly improved accuracy Subjects found highlighted techniques easier They also found them natural Matched expectations of performance?

Making the Most of VR and AR State of the art VR systems limited by basic technical, perceptual and human factors issues Goal is to simulate a compelling synthetic environment using displays/effectors in response to user actions sensed by sensors Technology is important but principally serves to mediate the experience of the reality Can we capitalize on the unique immersive multimodal interaction possible in VR and AR to produce perceptual experiences that go beyond current display experiences?

The End