CSE 408 Multimedia Information System

Transcription

1 CSE 408 Multimedia Information System Intro to Images & Vision Yezhou Yang Lots of slides from Tamara Berg and L. Feifei

2 Intro to Computer Vision Source: L. Lazebnik

3 The goal of computer vision To perceive the story behind the picture What we see What a computer sees Source: S. Narasimhan

4 Source: C. Fowlkes

5 Source: C. Fowlkes

6 Example Applications Controlling processes (e.g. an industrial robot or an autonomous vehicle). Detecting events (e.g. for visual surveillance). Organizing information (e.g. for indexing and retrieval from collections of images and videos). Modeling objects or environments (e.g. industrial inspection, or medical image analysis). Interaction (e.g. as the input to a device for human computer interaction). Source: L. Lazebnik

7 The goal of computer vision To perceive the story behind the picture What exactly does this mean? Vision as a source of metric 3D information Vision as a source of semantic information Source: L. Lazebnik

8 Vision as measurement device Real-time stereo Structure from motion Multi-view stereo for community photo collections NASA Mars Rover Pollefeys et al. Goesele et al. Source: L. Lazebnik

9 Vision as a source of semantic information slide credit: Fei-Fei, Fergus & Torralba

10 Object categorization sky building flag banner bus face street lamp cars wall bus slide credit: Fei-Fei, Fergus & Torralba

11 Scene and context categorization outdoor city traffic slide credit: Fei-Fei, Fergus & Torralba

12 Qualitative spatial information slanted non-rigid moving object vertical rigid moving object horizontal rigid moving object slide credit: Fei-Fei, Fergus & Torralba

13 Why study computer vision? Vision is useful: Images and video are everywhere! Personal photo albums Surveillance and security Movies, news, sports Medical and scientific images Source: L. Lazebnik

14 Why study computer vision? Vision is useful Vision is interesting Vision is difficult Half of primate cerebral cortex is devoted to visual processing Achieving human-level visual perception is probably AIcomplete Source: L. Lazebnik

15 Why is computer vision difficult? Source: L. Lazebnik

16 Challenges: viewpoint variation Michelangelo slide credit: Fei-Fei, Fergus & Torralba

17 Challenges: illumination image credit: J. Koenderink

18 Challenges: scale slide credit: Fei-Fei, Fergus & Torralba

19 Challenges: deformation Xu, Beihong 1943 slide credit: Fei-Fei, Fergus & Torralba

20 Challenges: occlusion Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba

21 Challenges: background clutter Source: L. Lazebnik

22 Challenges: Motion Source: L. Lazebnik

23 Challenges: object intra-class variation slide credit: Fei-Fei, Fergus & Torralba

24 Challenges: local ambiguity slide credit: Fei-Fei, Fergus & Torralba

25 Challenges: local ambiguity slide credit: Fei-Fei, Fergus & Torralba

26 Challenges or opportunities? Images are confusing, but they also reveal the structure of the world through numerous cues Our job is to interpret the cues! Image source: J. Koenderink

27 Depth cues: Linear perspective Source: L. Lazebnik

28 Depth cues: Aerial perspective

29 Depth ordering cues: Occlusion Source: J. Koenderink

30 Shape cues: Texture gradient Source: L. Lazebnik

31 Shape and lighting cues: Shading Source: J. Koenderink

32 Position and lighting cues: Cast shadows Source: J. Koenderink

33 Grouping cues: Similarity (color, texture, proximity) Source: L. Lazebnik

34 Grouping cues: Common fate Image credit: Arthus-Bertrand (via F. Durand)

35 Bottom line Perception is an inherently ambiguous problem Many different 3D scenes could have given rise to a particular 2D picture Image source: F. Durand

36 Bottom line Perception is an inherently ambiguous problem Many different 3D scenes could have given rise to a particular 2D picture Possible solutions Bring in more constraints (more images) Use prior knowledge about the structure of the world Need a combination of different methods Image source: F. Durand

37 Connections to other disciplines Artificial Intelligence Robotics Machine Learning Computer Vision Psychology Neuroscience Computer Graphics Image Processing Source: L. Lazebnik

38 Origins of computer vision L. G. Roberts, Machine Perception of Three Dimensional Solids, Ph.D. thesis, MIT Department of Electrical Engineering, Source: L. Lazebnik

39 I. Low level vision Basic image formation and processing * = Linear filtering Edge detection Cameras and sensors Light and color Feature extraction: corner and blob detection source: Svetlana Lazebnik

40 How do we go from a 3d world to 2d image? 3D world Point of observation 2D image

41 How do we extract useful features?

42 How do we extract useful features?

43 II. Mid-level vision Segmentation and grouping source: Svetlana Lazebnik

44 The goals of segmentation Separate image into coherent objects image human segmentation Berkeley segmentation database: nch/ source: Svetlana Lazebnik

45 Terminology Segmentation, grouping, perceptual organization: gathering features that belong together Top-down segmentation: pixels belong together because they come from the same object Bottom-up segmentation: pixels belong together because they look similar source: Svetlana Lazebnik

46 The Gestalt school Grouping is key to visual perception Elements in a collection can have properties that result from relationships The whole is greater than the sum of its parts subjective contours occlusion familiar configuration source: Svetlana Lazebnik

47 Gestalt factors These factors make intuitive sense, but are very difficult to translate into algorithms source: Svetlana Lazebnik

48 Segmentation as clustering Source: K. Grauman

49 IV. High level vision - Recognition Finding correspondences Bag-of-features models Clustering and visual vocabularies Classification Sources: D. Lowe, L. Fei-Fei

50 IV. High level vision - Recognition sky building flag banne r face street lamp bus cars wall bus source: Fei-Fei, Fergus & Torral

51 How many object categories are there? source: Svetlana Lazebnik Biederman 1987

52 So what does object recognition involve? source: Svetlana Lazebnik

53 Verification: is that a lamp? source: Svetlana Lazebnik

54 Detection: where are the people? source: Svetlana Lazebnik

55 Identification: is that Potala Palace? source: Svetlana Lazebnik

56 Object categorization mountain tree building banner street lamp vendor people source: Svetlana Lazebnik

57 Scene and context categorization outdoor city source: Svetlana Lazebnik

58 Progress to date The next slides show some examples of what current vision systems can do Source: L. Lazebnik

59 Optical character recognition (OCR) Technology to convert scanned docs to text If you have a scanner, it probably came with OCR software Digit recognition, AT&T labs License plate readers Also used for zipcode reading by the USPS Source: S. Seitz

60 Face detection Many new digital cameras now detect faces Canon, Sony, Fuji, Source: S. Seitz

61 Face Detection for Privacy Face Blurring for Google Streetview

62 Face Detection for Privacy Face Blurring for Google Streetview

63 Smile detection? Sony Cyber-shot T70 Digital Still Camera Source: S. Seitz

64 Object recognition (in supermarkets) LaneHawk by EvolutionRobotics A smart camera is flush-mounted in the checkout lane, continuously watching for items. When an item is detected and recognized, the cashier verifies the quantity of items that were found under the basket, and continues to close the transaction. The item can remain under the basket, and with LaneHawk,you are assured to get paid for it Source: S. Seitz

65 Face recognition Who is she? Source: S. Seitz

66 Vision-based biometrics How the Afghan Girl was Identified by Her Iris Patterns Read the story Source: S. Seitz

67 Login without a password Fingerprint scanners on many new laptops, other devices Face recognition systems now beginning to appear more widely Source: S. Seitz

68 Object recognition (in mobile phones) This is becoming real: Microsoft Research Point & Find Google goggles Source: S. Seitz

69 iphone Apps: ( Source: L. Lazebnik

70 iphone Apps: ( Source: L. Lazebnik

71 Special effects: shape capture The Matrix movies, ESC Entertainment, XYZRGB, NRC Source: S. Seitz

72 Special effects: motion capture Pirates of the Carribean, Industrial Light and Magic Source: S. Seitz

73 Sports Sportvision first down line Nice explanation on Source: S. Seitz

74 Smart cars Slide content courtesy of Amnon Shashua Mobileye Vision systems currently in high-end BMW, GM, Volvo models By 2010: 70% of car manufacturers. Source: S. Seitz

75 Source: C. Fowlkes

76 Vision-based interaction (and games) Kinect projector+camera for depth Nintendo Wii has camera-based IR tracking built in. See Lee s work at CMU on clever tricks on using it to create a multi-touch display! Assistive technologies Source: S. Seitz

77 Vision in space NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of Vision systems (JPL) used for several tasks Panorama stitching 3D terrain modeling Obstacle detection, position tracking For more, read Computer Vision on Mars by Matthies et al. Source: S. Seitz

78 Robotics NASA s Mars Spirit Rover Source: S. Seitz

79 Source: C. Fowlkes

80 Earth viewers (3D modeling) Image from Microsoft s Virtual Earth (see also: Google Earth) Source: S. Seitz

81 Photo Tourism

82 Trafalgar Square

83 Photo Tourism overview Scene reconstruc tion Input photographs Photo Explorer Relative camera positions and orientations Point cloud Sparse correspondence System for interactive browsing and exploring large collections of photos of a scene. Computes viewpoint of each photo as well as a sparse 3d model of the scene.

84 The computer vision industry A list of companies here: Source: L. Lazebnik