High Level Computer Vision. Introduction - April 16, Bernt Schiele & Mario Fritz MPI Informatics and Saarland University, Saarbrücken, Germany

Transcription

1 Perceptual and Sensory Augmented Computing High Level Computer Vision Introduction - April 16, 2014 MPI Informatics and Saarland University, Saarbrücken, Germany

2 Computer Vision Lecturer: Bernt Schiele Mario Fritz Assistants: Fabio Galasso Zeynep Akata Language: English Webpage: mailing list for announcements etc. use link on webpage to enroll in mailing list High Level Computer Vision - April 16, 2o14 2

3 Lecture & Exercise Officially: 2V (lecture) + 2Ü (exercise) Lecture: Wed: 14:15-16:00 (room 024) Exercise: Thu: 12:15-14:00 (room 024) typically 1 exercise sheet every 1-2 weeks part of the final grade pencil and paper, as well as matlab-based exercise, reading assignment (research papers, overview papers, etc.) & larger project at end of lecture we/you propose project, mentoring, final presentation Exam planned as oral exam after the SS - there will be proposed dates High Level Computer Vision - April 16, 2o14 3

4 Material For part of the lecture: available online High Level Computer Vision - April 16, 2o14 4

5 Why Study Computer Vision Science Foundations of perception. How do WE see? computer vision to explore computational model of human vision High Level Computer Vision - April 16, 2o14 5

6 Why Study Computer Vision Science Foundations of perception. How do WE see? computer vision to explore computational model of human vision Engineering How do we build systems that perceive the world computer vision to solve real-world problems: cars to detect pedestrians High Level Computer Vision - April 16, 2o14 6

7 Why Study Computer Vision Science Foundations of perception. How do WE see? computer vision to explore computational model of human vision Engineering How do we build systems that perceive the world computer vision to solve real-world problems: cars to detect pedestrians Applications medical imaging (computer vision to support medical diagnosis, visualization) surveillance (to follow/track people at the airport, train-station,...) entertainment (vision-based interfaces for games) graphics (image-based rendering, vision to support realistic graphics) car-industry (lane-keeping, pre-crash intervention, ) High Level Computer Vision - April 16, 2o14 7

8 Some Applications License Plate Recognition London Congestion Charge imagingandcameras.html London_congestion_charge Surveillance Face Recognition Airport Security (People Tracking) Medical Imaging (Semi-)automatic segmentation and measurements Robotics Driver assistance High Level Computer Vision - April 16, 2o14 8

9 More Applications Microsoft High Level Computer Vision - April 16, 2o14 9

10 Goals of today s lecture First intuitions about What is computer vision? What does it mean to see and how do we (as humans) do it? How can we make this computational? Applications & Appetizers 2 case studies: Recovery of 3D structure - slides taken from Michael Brown University / MPI Intelligent Systems Object Recognition - intuition from human vision... High Level Computer Vision - April 16, 2o14 10

11 Perceptual and Sensory Augmented Computing Applications & Appetizers... work from our group

12 Detection & Recognition of Visual Categories Challenges: multi-scale multi-view multi-class varying illumination occlusion cluttered background articulation high intraclass variance low interclass variance High Level Computer Vision - April 16, 2o14 12

13 Challenges of Visual Categorization low inter-class variation high intra-class variation High Level Computer Vision - April 16, 2o14 13

14 Sample Category: Motorbikes High Level Computer Vision - April 16, 2o14 14

15 Basic Idea global I know where the Eiffel Tower is local High Level Computer Vision - April 16, 2o14 15

16 Large Scale Object Class Recognition Learning Shape Models from 3D CAD Data 3D Computer Aided Design (CAD) Models for - computer graphics, game design - polygonal meshes + texture descriptions - semantic part annotations (may) exist Learning Object Class Model directly from 3D CAD-data: Michael Stark High Level Computer Vision - April 16, 2o14 16

17 Video... High Level Computer Vision - April 16, 2o14 18

18 Articulation Model Assume uniform position prior for the whole body Learn the conditional relation between part position and body center from data: 400 annotated training images High Level Computer Vision - April 16, 2o14 19

19 Modeling Body Dynamics Visualization of the hierarchical Gaussian process latent variable model (hgplvm) High Level Computer Vision - April 16, 2o14 21

20 High Level Computer Vision - April 16, 2o14 23

21 High Level Computer Vision - April 16, 2o14 24

22 Complete 3D Scene Modeling Goal: Infer consistent 3D world hypothesis from 2D image sequences with a moving monocular camera Tracking 3D Scene Model Integrate SoA object detectors, scene labeling Efficiently leverage domain knowledge [Wojek et.al.@eccv10] High Level Computer Vision - April 16, 2o14 29

23 [Wojek System sample video (pedestrians) ETH-Loewenplatz sequence: By courtesy of ETH Zürich [Ess et al., PAMI 09] High Level Computer Vision - April 16, 2o14 31

24 [Wojek System sample video (vehicles) High Level Computer Vision - April 16, 2o14 32

25 Sequential Model Update for Scene Labeling (Fritz,Levinkov) High Level Computer Vision - April 16, 2o14 33

26 Sequential Model Update for Scene Labeling (Fritz,Levinkov) High Level Computer Vision - April 16, 2o14 34

27 Perception for Manipulation High Level Computer Vision - April 16, 2o14 35

28 Perception for Manipulation High Level Computer Vision - April 16, 2o14 36

29 Multi-Class Video Co-Segmentation (Fritz, Chiu) High Level Computer Vision - April 16, 2o14 37

30 Multi-Class Video Co-Segmentation (Fritz, Chiu) High Level Computer Vision - April 16, 2o14 38

31 Efficient Object Detection with Shared Representations High Level Computer Vision

32 Perceptual and Sensory Augmented Computing Basic Concepts and Terminology Computer Vision vs. Computer Graphics

33 Pinhole Camera (Model) (simple) standard and abstract model today box with a small hole in it High Level Computer Vision - April 16, 2o14 50

34 Camera Obscura around 1519, Leonardo da Vinci ( ) when images of illuminated objects penetrate through a small hole into a very dark room you will see [on the opposite wall] these objects in their proper form and color, reduced in size in a reversed position owing to the intersection of the rays High Level Computer Vision - April 16, 2o14 51

35 Principle of pinhole......used by artists (e.g. Vermeer 17th century, dutch) and scientists High Level Computer Vision - April 16, 2o14 52

36 Digital Images Imaging Process: (pinhole) camera model digitizer to obtain digital image High Level Computer Vision - April 16, 2o14 53

37 (Grayscale) Image Goals of Computer Vision how can we recognize fruits from an array of (gray-scale) numbers? how can we perceive depth from an array of (gray-scale) numbers? Goals of Graphics how can we generate an array of (gray-scale) numbers that looks like fruits? how can we generate an array of (gray-scale) numbers so that the human observer perceives depth? computer vision = the problem of inverse graphics? High Level Computer Vision - April 16, 2o14 54

38 Perceptual and Sensory Augmented Computing Visual Cues for Image Analysis... in art and visual illusions

39 1. Case Study: Human & Art - Recovery of 3D Structure High Level Computer Vision - April 16, 2o14 57

40 1. Case Study: Human & Art - Recovery of 3D Structure High Level Computer Vision - April 16, 2o14 58

41 1. Case Study: Human & Art - Recovery of 3D Structure High Level Computer Vision - April 16, 2o14 59

42 1. Case Study: Human & Art - Recovery of 3D Structure High Level Computer Vision - April 16, 2o14 60

43 1. Case Study: Human & Art - Recovery of 3D Structure High Level Computer Vision - April 16, 2o14 61

44 1. Case Study: Human & Art - Recovery of 3D Structure High Level Computer Vision - April 16, 2o14 62

45 1. Case Study Computer Vision - Recovery of 3D Structure take all the cues of artists and turn them around exploit these cues to infer the structure of the world need mathematical and computational models of these cues sometimes called inverse graphics High Level Computer Vision - April 16, 2o14 63

46 A trompe l oeil depth-perception movement of ball stays the same location/trace of shadow changes High Level Computer Vision - April 16, 2o14 64

47 Another trompe l oeil illusory motion only shadows changes square is stationary High Level Computer Vision - April 16, 2o14 65

48 Color & Shading High Level Computer Vision - April 16, 2o14 66

49 Color & Shading High Level Computer Vision - April 16, 2o14 67

50 High Level Computer Vision - April 16, 2o14 68

51 High Level Computer Vision - April 16, 2o14 69

52 High Level Computer Vision - April 16, 2o14 70

53 High Level Computer Vision - April 16, 2o14 71

54 High Level Computer Vision - April 16, 2o14 72

55 Do you still think you see the world? High Level Computer Vision - April 16, 2o14 73

56 Do you still believe what you see? Experiment carefully point flash light into your eye from one corner don t hurt yourself! Observation you ll see your own blood vessels they are actually in front of the retina we ve adapted to their usual shadow High Level Computer Vision - April 16, 2o14 75

57 2. Case Study: Computer Vision & Object Recognition is it more than inverse graphics? how do you recognize the banana? the glass? the towel? how can we make computers to do this? ill posed problem: missing data ambiguities multiple possible explanations High Level Computer Vision - April 16, 2o14 76

58 Image Analysis vs. Synthesis from: Object Perception as Bayesian Inference Kersten 2003 High Level Computer Vision - April 16, 2o14 78

59 Complexity of Recognition High Level Computer Vision - April 16, 2o14 79

60 Complexity of Recognition High Level Computer Vision - April 16, 2o14 80

61 Complexity of Recognition High Level Computer Vision - April 16, 2o14 81

62 Complexity of Recognition High Level Computer Vision - April 16, 2o14 82

63 Complexity of Recognition High Level Computer Vision - April 16, 2o14 83

64 Recognition: the Role of Context Antonio Torralba High Level Computer Vision - April 16, 2o14 84

65 Recognition: the role of Prior Expectation Guiseppe Arcimboldo High Level Computer Vision - April 16, 2o14 85

66 Complexity of Recognition High Level Computer Vision - April 16, 2o14 86

67 Complexity of Recognition High Level Computer Vision - April 16, 2o14 87

68 One or Two Faces? High Level Computer Vision - April 16, 2o14 88

69 Class of Models: Pictorial Structure Fischler & Elschlager 1973 Model has two components parts (2D image fragments) structure (configuration of parts) High Level Computer Vision - April 16, 2o14 89

70 Deformations High Level Computer Vision - April 16, 2o14 90

71 Clutter High Level Computer Vision - April 16, 2o14 91

72 Example High Level Computer Vision - April 16, 2o14 92

73 Perceptual and Sensory Augmented Computing Recognition, Localization, and Segmentation a few terms let s briefly define what we mean by that

74 Object Recognition: First part of this Computer Vision class Different Types of Recognition Problems: Object Identification - recognize your pencil, your dog, your car Object Classification - recognize any pencil, any dog, any car - also called: generic object recognition, object categorization, Recognition and Segmentation: separate pixels belonging to the foreground (object) and the background Localization/Detection: position of the object in the scene, pose estimate (orientation, size/scale, 3D position) High Level Computer Vision - April 16, 2o14 94

75 Object Recognition: First part of this Computer Vision class Different Types of Recognition Problems: Object Identification - recognize your apple, your cup, your dog Object Classification - recognize any apple, any cup, any dog - also called: generic object recognition, object categorization, - typical definition: basic level category High Level Computer Vision - April 16, 2o14 95

76 Which Level is right for Object Classes? Basic-Level Categories the highest level at which category members have similar perceived shape the highest level at which a single mental image can reflect the entire category the highest level at which a person uses similar motor actions to interact with category members the level at which human subjects are usually fastest at identifying category members the first level named and understood by children (while the definition of basic-level categories depends on culture there exist a remarkable consistency across cultures...) Most recent work in object recognition has focused on this problem we will discuss several of the most successful methods in the lecture :-) High Level Computer Vision - April 16, 2o14 96

77 Object Recognition: First part of this Computer Vision class Recognition and Segmentation: separate pixels belonging to the foreground (object) and the background High Level Computer Vision - April 16, 2o14 97

78 Object Recognition: First part of this Computer Vision class Recognition and Localization: to position the object in the scene, estimate the object s pose (orientation, size/scale, 3D position) Example from David Lowe: High Level Computer Vision - April 16, 2o14 98

79 Localization: Example Video 1 High Level Computer Vision - April 16, 2o14 99

80 Localization: Example Video 2 High Level Computer Vision - April 16, 2o14 100

81 Object Recognition: First part of this Computer Vision class Different Types of Recognition Problems: Object Identification - recognize your pencil, your dog, your car Object Classification - recognize any pencil, any dog, any car - also called: generic object recognition, object categorization, Recognition and Segmentation: separate pixels belonging to the foreground (object) and the background Localization: position the object in the scene, estimate pose of the object (orientation, size/scale, 3D position) High Level Computer Vision - April 16, 2o14 101

82 Perceptual and Sensory Augmented Computing Basic Filtering

83 Computer Vision and Fundamental Components computer vision: reverse the imaging process 2D (2-dimensional) digital image processing pattern recognition / 3D image analysis image understanding High Level Computer Vision - April 16, 2o14 104

84 Digital Image Processing Some Basics (digital signal processing, FFT, ) Image Filtering - (taken from a class by Bill Image Filtering take some local image patch (e.g. 3x3 block) image filtering: apply some function to local image patch High Level Computer Vision - April 16, 2o14 105

85 Image Filtering Some Examples: what assumptions are you making to infer the center value? Goals of Image Filtering: reduce noise fill-in missing values/information extract image features (e.g.edges/corners)... 3 or 4 High Level Computer Vision - April 16, 2o14 106

86 Image Filtering simplest case: linear filtering: replace each pixel by a linear combination of its neighbors the prescription for the linear combination is called the convolution kernel High Level Computer Vision - April 16, 2o14 107

87 2D signals and convolution Components of convolution : Image: - continuous: I(x,y) - discrete: I[k,l] or I k,l filter kernel : g[k,l] filtered image: f[m,n] 2D convolution (discrete): special case: convolution (discrete) of a 2D-image with a 1D-filter High Level Computer Vision - April 16, 2o14 108

88 Linear Filtering (warm-up slide) High Level Computer Vision - April 16, 2o14 109

89 Linear Filtering (warm-up slide) High Level Computer Vision - April 16, 2o14 110

90 Try it out in GIMP You can try out linear filter kernels in the free image manipulation tool GIMP - availble at gimp.org open image from the menu pick: Filters - Generic Convolution Matrix... enter filter kernel in Matrix press ok to apply High Level Computer Vision - April 16, 2o14 111

91 Linear Filtering High Level Computer Vision - April 16, 2o14 112

92 Linear Filtering High Level Computer Vision - April 16, 2o14 113

93 Linear Filtering High Level Computer Vision - April 16, 2o14 114

94 Blurring High Level Computer Vision - April 16, 2o14 115

95 Blurring Examples High Level Computer Vision - April 16, 2o14 116

96 Linear Filtering (warm-up slide) High Level Computer Vision - April 16, 2o14 117

97 Linear Filtering (warm-up slide) High Level Computer Vision - April 16, 2o14 118

98 Linear Filtering High Level Computer Vision - April 16, 2o14 119

99 (remember blurring) High Level Computer Vision - April 16, 2o14 120

100 Sharpening High Level Computer Vision - April 16, 2o14 121

101 Sharpening Example High Level Computer Vision - April 16, 2o14 122

102 Sharpening High Level Computer Vision - April 16, 2o14 123