Graphics and Perception Carol O Sullivan Carol.OSullivan@cs.tcd.ie Trinity College Dublin
Outline Some basics Why perception is important For Modelling For Rendering For Animation Future research - multisensory perception and crowds
Why do we care? Rapid Developments in Graphics: Algorithmic: e.g., fluid, cloth, humans, etc Hardware: desktop PCs Low-end mobile devices, PDAs, cell phones More than just technical challenges: Fidelity? Plausibility? Presence? Perceptibility of errors? Evaluation? Metrics? Must consider human perception!
Modelling
Example Model simplification http://www.ixbt.com/
How to measure fidelity? Watson et al. 2001 Used experimental measures to evaluate simplification algorithms Naming times Ratings Forced-choice preferences Original (top), QSlim at 80% (middle) and Vclust 80% (bottom)
Perceptual simplification Users can guide the simplification process Pojar and Schmalstieg 2003 Kho and Garland 2003 Or salient features can be found automatically, or using an eye-tracker Howlett et al.2004 Lee et al.2005
We used an eye-tracker to determine the prominent features of models
Finding Salient Features We gathered information on where a participant was fixating while viewing a set of models.
We incorporated fixation data to produce a perceptual simplification metric Then asked people to name them, match them, and choose between them Evaluation
Rendering
Image Fidelity Fidelity to what? Stimulus Response Semantics How to assess fidelity? Metrics How to apply the notion to image production? Perceptually informed rendering To image reproduction? Tone mapping, contrast reduction, display device design How to evaluate perceptual methods?
Measuring Error Problem: Solution: Model the Human Visual System (HVS) VDP: Visible Differences VQEG/Modelfest SSIM: Structured Similarity
Visible Difference Predictors Probabilities of difference detection Two Images (e.g., frames of an animation) Daly/Myskowski
Example: Perceptual Rendering Can do user studies to find out what rendering components are perceptually important Then develop a metric to guide where to concentrate computationally expensive illumination components with respect to image quality. Ferwerda et al., 2004
High Dynamic Range image reproduction High Dynamic Range display device Seetzen et al. 2004 present two designs for a HDR display device, based on the idea of a modulated backlight 1) LEDs array and 2) Digital Mirror Device A standard LCD device provides for color and further intensity modulation Limitations of the eye s ability to perceive local contrast (ratio of 150 to 1) are taken into account to determine: The minimum number of LEDs necessary in the array Adequate resolution and blur for the DMD projector The optimal number of bits necessary to drive each backlight
High Dynamic Range image reproduction Spatial kernel f Input Influence g in the intensity domain for the central pixel Weights Images courtesy of Fredo Durand and Julie Dorsey Output
Visual Attention and Tasks Attention is largely controlled by task: Scene rendered at low, high and selective resolution. Task allocated Difference in quality largely not noticed by participants Cater et al. 2003 Selective Quality Image
Three varieties of realism Ferwerda 2003 Physical realism Same visual stimulation as scene Highly computationally expensive Photo-realism Same visual response as scene Takes observer s visual system into account Functional realism Same visual information as scene Information useful for completing a task
Human facial illustrations Gooch et al. 2004 Images courtesy of Bruce Gooch
Human facial illustrations Gooch et al. 2004 Presents a new technique for automatic NPR generation of faces from photos Photo illustration caricature Evaluates functional value of images Recognition task Learning task slower with photos Accuracy, speed
Duke et al. 2003 Radar display Triangles Circles Rendering and affect Which is the threat? Invariant of a sharp shape invokes connotations of threat
Experiments Assessment of danger and safety Radar, door, house & trees Assessment of strength and weakness Radar, strongest man, weakest man Goal-directed interaction Paths, object selection
Results Demonstration of how rendering style can convey meaning and influence judgement Illustrate how semantics, affect and other high-level invariants need to be taken into account when analysing rendering methods, not just perceptual adequacy and realism.
Distance and Scale in VEs What affects perception of distance in VEs? Triangulated walking task Thompson et al. 2003 Image Quality? Makes no difference Is it the HMD? Creem-Regehr et al. 2002 No artificially generating these restrictions in the real world did not produce the same problems No need to see own body (see also Lok et al. 2003) However, some cues are important Hu et al. 2002 Shadows and Interreflection affected performance in a placement task
Presence Slater et al. 2004 Presence = the sense of being there. In the past, questionnaires and interviews were predominant Physiological measurements proposed as a viable alternative Shock of entering room with precipice induces physiological response Meehan et al. 2002 Perhaps can also be used to measure and predict Breaks in Presence (BIP)
Animation
Measuring Error Reitsma and Pollard 03
Accuracy vs. Plausibility Chenney and Forsyth 00
Collision Handling
Collisions and Perception Evaluating the Visual Fidelity of Physically Based Animations. Collisions and Attention
Multisensory Perception: 1 The multisensory brain: Areas of brain not unisensory but active to other sensory information Growing body of evidence: Visual areas active during tactile perception Visual areas active during tactile object recognition Auditory areas of brain are active during lip reading (no sound)
Multisensory Perception: 2 The senses influence each other: You simply cannot predict perception by studying the senses in isolation
Perceptual metrics for crowds New metrics to evaluate human simulations, taking account of: Multisensory information: vision, motion and sound Crowd and scene scale Task based on psychophysical and neuroimaging results which will be used to: Devise new multisensory strategies for optimal LOD control Implement new error and comparison measurements for evaluation
State of the Art Dead Rising (Xbox 360), Capcom, 2006
State of the Art Madden NFL (Xbox 360), EA, 2006
State of the Art Project Gotham Racing (Xbox 360), Microsoft, 2005
State of the Art Assassin s Creed (Xbox 360), Ubisoft
Sound rendering for crowds Tsingos et al. - SIGGRAPH 2004 Wand and Straβer PBG 2005