Virtual Reality and Interaction. 1.Introduction to Virtual Reality (VR)

Transcription

1 Virtual Reality and Interaction 1.Introduction to Virtual Reality (VR) Shoichi Hasegawa I ll upload slides on

2 Media and Virtual Reality (VR) 2

3 Communication and Media Conversation A most natural and basic communication method. Painting Oral tradition Cave of Altamira and Paleolithic Cave Art of Northern Spain B.C cultural material and tradition transmitted orally from one generation to another. Theatre (or dramas, plays) are extended one. 3

4 Communication and Media Writing system Printing 4

5 Communication and Media Movable types Europe moves from Middle Ages to Modern Era For democracy, general public need to know many thing and newspapers and other mass media realize it. 5

6 Communication and Media Era of Mass media and technology Newspapers, Radio, Cinema, Television Personal media Letters, telegrams, phones 6

7 Era of Television Marshall McLuhan ( ) Understanding Media: the Extensions of Man, (McGraw-Hill, 1964). Media extends human s sense. He faces changing of children. His book is difficult to understand. So, followings based on RCA 630-TS, the first mass-produced electronic television set, which sold in A Schoolman s Guide to Marshall McLuhan by John M. Culkin or 平凡社 マクルーハン理論

8 Marshall McLuhan The medium is the message The medium is the thing you're missing. Everybody's hooked on content; pay attention to form, structure, framework, medium. The relation of the medium to the content. It is always content-in-form which is mediated. In this sense, the medium is co-message. The medium alters the perceptual habits of its users. In the process of delivering content the medium also works over the sensorium of the consumer. The medium is massage The relation of the medium to society. The media massage the society as well as the individual. 8

9 Marshall McLuhan 1) 1967 B.C. - All the senses get into the act. The reality which bombards him from all directions is picked up with the omni-directional antennae of sight, hearing, touch, smell, and taste. Eskimos travel without visual bearings in their white-on-white world and can sketch cartographically accurate maps of shifting shorelines. They have forty or fifty words for what we call snow. They live in a world without linearity, a world of acoustic space. 9

10 Marshall McLuhan 2) Art imitates life. ( 技術は人を模倣する ) Edward T. Hall offers the thesis that all art and technology is an extension of some physical or psychic element of man. Today man has developed extensions for practically everything he used to do with his body: stone axe for hand, wheel for foot, glasses for eyes, radio for voice and ears. Through the electronic media of telegraph, telephone, radio, and television, man has now equipped his world with a nervous system similar to the one within his own body. Now in the electric age, our extended faculties and senses constitute a single instantaneous and coexistent field of experience. It's all-at-once. It's sharedby-all. McLuhan calls the world a global village. 10

11 Marshall McLuhan 3) Life imitates art. ( 人は技術を模倣する ) We shape our tools and thereafter they shape us. Extensions of our senses begin to interact with our senses. The new change in the environment creates a new balance among the senses. The sense-ratio will differ for different media. The nature of the sensory effect will be determined by the medium used. McLuhan divides the media according to the quality or definition of their physical signal. hot media: hi-fidelity, cool media : low definition Film is hot; television is cool. Radio is hot; telephone is cool. The cool medium or person invites participation and involvement. It leaves room for the response of the consumer. 11

12 Marshall McLuhan 4) We shaped the alphabet and it shaped us. Structure of literary media=straight line : As an example, while lecturing to a large audience in a modern hotel in Chicago, a distinguished professor is bitten in the leg by a cobra. The whole experience takes three seconds. He is affected through the touch of the reptile, the gasp of the crowd, the swimming sights before his eyes. His memory, imagination, and emotions come into emergency action. A lot of things happen in three seconds. Two weeks later he is fully recovered and wants to write up the experience in a letter to a colleague. To communicate this experience through print means that it must first be broken down into parts and then mediate, eyedropper fashion, one thing at a time, in an abstract, linear, fragmented, sequential way. That is the essential structure of print. Once a culture uses such a medium for a few centuries, it begins to perceive the world in a one-thing-at-a-time, abstract, linear, fragmented, sequential way. And it shapes its organizations and schools according to the same premises. The form of print has become the form of thought. The electronic media have broken the monopoly of print; they have altered our sensory profiles by heightening our awareness of aural, tactile, and kinetic values. 12

13 Marshall McLuhan 5) 1967 A.D.-All the senses want to get into the act. Print repressed most sense-life in favor of the visual. The end of print's monopoly also marks the end of a visual monopoly. As the early warning system of art and popular culture indicates, all the senses want to get into the act. Some of the excesses in the current excursions into aural, oral, tactile, and kinetic experience may in fact be directly responsive to the sensory deprivation of the print culture. 13

14 Communication and Media WWW starts from literary and mass media Home pages of companies and universities Official announcement to audience Web 2.0 was bidirectional and literary media Blogs, Wikis, Facebook?, Twitter?, People send information to general public YouTube, Nico Video, Instagram Not a literary media 14

15 Communication and Media Video Games and Virtual Reality Has their world and story like cinema, novel User can interact with the world and story. They can provide direct experience. Ex: Safe master by Solidray Co.,Ltd. 15

16 Communication and Media AIP Cube (David Zeltzer 1992) Autonomy : the world is simulated in realtime. Interaction: Input, You can change the world. Presence: Display, as if you are there. VR Immersive simulation of universe VR Autonomy 自律性 Presence 臨場感 360 theater Immersive CAD Interaction 対話性 Word processor Ex: 360 theaters=(0, 0, 1) Video games = (0.5, 0.5, 0.5) 16

17 Video games and virtual worlds Video games tries to express virtual world Spacewar!, the first computer game. (1962) Pong (1972) 17

18 Video games Colossal Cave Adventure The player uses text input to control the game, The game state is relayed to the player via text output. 18

19 Video games 1980 Rogue: Exploring the Dungeons of Doom 19

20 Virtual Reality Game 20

21 Dream of Virtual Reality (VR)

22 Example of VR System Virtual Canoe I built it in

23 Example of VR System VibroSkate Skate board simulator My students build in 2015 for a student VR contents. 23

24 What is Virtual Reality Virtual Reality is an Experience of (a scene of) a world in a computer VR system is a system to provide the experience What we do when we experience the world Feel the world via our sensory system Change the world by moving our body Loop of sensory motor system is important Consciousness Sense Perception Mind Change Motion Body Environment 24

25 First Head Mounted Display Ivan Sutherland (1968) 1 st HMD was see through Augmented Reality will mention later Potential meter Graphics Reflecting head orientation 25

26 What is Virtual Reality Head Mounted Display (HMD) Motion Head rotation Sensor Vision The loop realize look around the world Consciousness Sense Perception Mind Change Motion Body Environment 26

27 The Ultimate Display (Sutherland 1965) Ivan Sutherland dreams virtual reality We live in a physical world whose properties we have come to know well through long familiarity. We sense an involvement with this physical world which gives us the ability to predict its properties well. For example, we can predict where objects will fall, how well known shapes look from other angles, and how much force is required to push objects against friction. We lack corresponding familiarity with the forces on charged particles, forces in nonuniform fields, the effects of nonprojective geometric transformations, and high-inertia, low friction motion. A display connected to a digital computer gives us a chance to gain familiarity with concepts not realizable in the physical world. It is a looking glass into a mathematical wonderland. The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked. 27

28 Dream of Virtual Reality Ultimate display = Ultimate VR = the mathematical wonderland into which Alice walked. Computer can control the existence of matter. A chair would be good enough to sit in Handcuffs would be confining a bullet would be fatal Gives us a chance to gain familiarity with concepts not realizable in the physical world. I captured by this dream when I was a university student in Through the Looking-Glass, and What Alice Found There 28

29 Virtual Reality System VR Interface Virtual World Autonomy 自律性 Presence 臨場感 Display Measurement Modeling Simulation Interaction 対話性 Interface Computer Sensors Display Motions Measure Input Virtual World Simulation 29

30 Human sensory system Five sense =Sight, Hearing, Taste, Smell, Touch. Traditional taxonomy of human sensory system Non-traditional senses Balance and acceleration (Vestibular sense) In inner ear there are three semicircular canals for angular velocity and otolithic organs for acceleration. Kinesthetic sense Temperature Pain Other internal senses 30

31 Aim to the reality of sensation A variety of equipment is developed CAVE Visual/Auditory/Olfactory Olfactory Display Nakamoto Lab, Tokyo tech Commercial HMD Oculus rift Computer Haptic Somatic Commercial locomotion device, Omni SPIDAR at Sato lab Tokyo Tech Torus Treadmill at Iwata lab, Univ. Tsukuba, Japan Some succeed as commercial devices and some do not.

32 Create Virtual World Modeling Input into computer = digitize is needed Property of objects such as mass, dimension, shape, color can be represented by digits This digitize or extraction of information is called modelling. Screen shot of shape modeling tool metasequoia Simulation Time lapse Data changes corresponding to user s inputs. The low of changing in real world = physics law 32

33 Physics laws to simulate Most virtual reality aims to create our daily world. Simulate laws which are dominant there. Kinetics Motion of Particle, Solid, Continuum mechanics Thermodynamics Electromagnetism Classical Dynamics, Relativity, Quantum Dynamics Chemical and Biological laws merely simulated. But will be important for Burning, Cooking, Rotting of food 33

34 Analysis and Synthesis of Experiences To reproduce an experience with VR Wearing displays for all senses are too bulky. Modeling/Simulation of whole property is difficult. Computer An efficient way: Extract the essence of the experience and reproduce it. What is the essence? It varies on experiences. VR = (Virtual=essential + Real) VR extracts essence from real experience and virtually reproduce it To do: Analyze and decompose the experience. Synthesize the experience virtually by displaying sensations which concern the essence. Next, I ll introduce VR system which I concerned and the essence to reproduce. 34

35 1997 年 Virtual Basket Free throw Feel the touch and inertia of the ball and throw it. Motion of the ball is simulated Visual, Tactile, Auditory displays 35

36 2000 年 Rubik's cube Both-hand dexterous rotation manipulation Force feedback for each finger Visual display Target object User s hand 36

37 2005 年 Virtual Canoe Row the canoe Force feedback for response of the paddle Visual and Auditory display Water resistance Waves Canoe motion Contact with rocks and shores 37

38 Virtual Canoe: Display paddle s force Human scale haptic interface SPIDAR 21

39 visualization of pressure and flow 24

40 Haptic (force and tactile) interaction with water 3-dimensional simulation of flow takes much computation Pre-computed 3D + 2D simulation Waves far from paddle and canoe flow is slow 2D wave realtime simulation Around paddle and canoe Turbulence, small area 3D flow database is precomputed. Use it as a texture mapping 22

41 Feedback force computation 3D flow simulation Pressure and flow velocity field Rigid body dynamics simulat with collision detection Paddle velocity Precomputed part Water velocity R Force feedback Realtime wave simulation Realtime part 23

42 Visual of virtual canoe:distant view 18

43 Visual of virtual canoe:riverbank and bed Visual Collision 19

44 Visual of virtual canoe:rock and trunk 20

45 2015 VibroSkate Skateboard simulator Head Mounted Display Vibration for road texture and velocity Treadmill for motion Velocity reflects Kick force 45

46 46

47 Autonomy of Virtual world I m especially interested in autonomy of VR Objects: Physics simulation Animal, human, society:??? 47

48 2004 Haptics + Dynamics

49 Haptics + Dynamics Dynamics: Differential equations are known ff = mmmm, ττ = IIωω Force is found from postural relationship of finger and the object. Simulation: Compute velocity and position of near future Position ff = kkkk Physics simulation Force Haptic interface 49

50 Haptics + Dynamics + Beetle The beetle just wale forward comes to the protagonist of the world. Presence of the insect was an impact to me. 50

51 Kobito Virtual Brownies by a students circle

52 52

53 Characters in VR world Synthesize people in VR other than the users. Virtual human/character Human Consciousness Sensor Perception Mind Motor Body Virtual world Change V Human Motor V Body V Human V Sensor V Mind V Motor V Bdoy Playback stored motion Behavior is programed. Can not handle unexpected user s action Generate motion from Virtual world, body, mind by simulation Can handle various situation Character AI (Artificial intelligence) 53

54 Virtual Creature 2004~Current Interaction with virtual creature Before human, Virtual animal should be made Human is a kind of animal Animal aspect of human should be made before making whole human. 54

55 Selective attention: A very simple model for the virtual mind Behavior decision is based on sensory systems Input information is limited. Selective attention A mechanism to decide target of sensory system. sensors Selective attention Selected information (e.g. position) motion Target of interest Decision s purpose Decision of behavior 55

56 Virtual Creature with attention Tactile model Vision model Physics simulator F Tactile attention Amount of attention position Visual attention amount of attention position v F v v Behavior generation rules Max Motor control Sensor Amount position Reaching target position gaze position etc Jv = ω ω = J # v 56

57 Sensory system and attention model Tactile model Vision model Physics simulator F Tactile attention information intensity position Position, area Visual information attention intensity Position, position type v F v v Top-down attention Intensity position Sensory information Target of attention Intensity Vision target: objects in the field of view position Tactility target: objects that make body contact Intensity of attention Max Visual object velocity in the retinal image tactile contact pressure 57

58 Behavior decision and motor control Rules for behavior & decision Hand target touching Eye target tracking Intensity of attention Decision of reaching the target The Position of focused target + the type of behavior decides the body part to reach and target position. Motion control Inverse kinematics and PD control of joints Physics simulator Rule of behavior decision Touch target Decision of reacing target Motor control Track target Target of attention Intensity Position Jv = ω ω = J # v 58

59 59