Computational Illumination

Transcription

1 Computational Illumination Course WebPage : Ramesh Raskar Mitsubishi Electric Research Labs Ramesh Raskar, Computational Illumination

2 Computational Illumination Ramesh Raskar, Computational Illumination

3 Traditional film like Photography Detector Lens Pixels Image

4 Computational Photography: Optics, Sensors and Computations Generalized Sensor Ray Reconstruction Computations Upto 4D Ray Sampler Generalized Optics 4D Ray Bender Picture

5 Computational Photography Generalized Sensor Novel Cameras Processing Generalized Optics

6 Computational Photography Light Sources Novel Illumination Generalized Sensor Novel Cameras Processing Generalized Optics

7 Computational Photography Light Sources Novel Illumination Generalized Sensor Novel Cameras Processing Generalized Optics Scene: 8D Ray Modulator

8 Computational Photography Light Sources Novel Illumination Generalized Sensor Novel Cameras Processing Generalized Optics Display Recreate 4D Lightfield Scene: 8D Ray Modulator

9 Computational Photography Novel Illumination Light Sources Generalized Sensor Novel Cameras Modulators Generalized Optics Processing Ray Reconstruction Generalized Optics 4D Ray Bender Upto 4D Ray Sampler 4D Light Field 4D Incident Lighting Display Recreate 4D Lightfield Scene: 8D Ray Modulator

10 Computational Illumination Light Sources Generalized Sensor Novel Cameras Modulators Generalized Optics Processing Ray Reconstruction Generalized Optics 4D Ray Bender Upto 4D Ray Sampler Programmable 4D 4D Illumination field field + time time + wavelength 4D Light Field Display Recreate 4D Lightfield Scene: 8D Ray Modulator

11 Smarter Lighting Equipment What Parameters Can We Change?

12 Edgerton 1930 s

13 Edgerton 1930 s Stroboscope (Electronic Flash) Multi flash sequential photography Flash Camera Exposure Time

14 Computational Illumination: Programmable 4D Illumination Field + Time + Wavelength Presence or Absence, Duration, Brightness Flash/No-flash Light position Multi-flash for depth edges Programmable dome (image re-lighting and matting) Light color/wavelength Spatial Modulation Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG Exploiting (uncontrolled) natural lighting condition Day/Night Fusion Ramesh Raskar, Computational Illumination

15 Computational Illumination Presence or Absence, Duration, Brightness Flash/No-flash Light position Multi-flash for depth edges Programmable dome (image re-lighting and matting) Light color/wavelength Spatial Modulation Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG General lighting condition Day/Night Ramesh Raskar, Computational Illumination

16 Denoising Challenging Images Available light: + nice lighting - noise/blurriness - color No-flash

17 Flash: + details + color - flat/artificial Flash

18 Elmar Eisemann and Frédo Durand, Flash Photography Enhancement via Intrinsic Relighting Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard Szeliski, Michael Cohen, Kentaro Toyama. Digital Photography with Flash and No-Flash Image Pairs Denoise no-flash image using flash image No-flash Flash Result

19 Transfer detail from flash image to no-flash image No-flash + original lighting + details/sharpness + color Result

20 Cross-Bilateral Filter based Approach

21 Bilateral Cross Bilateral Cross Bilateral Filter When no-flash image is too noisy Borrow similarity from flash image edge stopping from flash image

22 Detail Layer / = Intensity Large-scale Detail Recombination: Large scale * Detail = Intensity

23 Need flash component! Ambient Flash

24 Build Exposure HDR image Multiple images with different exposure Debevec & Malik, Siggraph 97 Nayar & Mitsunaga, CVPR 00 Increasing Exposure

25 Build Flash HDR image Flash Intensity

26 Flash Exposure Sampling Flash Intensity Build Flash Exposure HDR image Agrawal, Raskar, Nayar, Li Siggraph05 Exposure

27 Capturing HDR Image Varying Exposure time Varying Flash brightness Varying both Ramesh Raskar, Computational Illumination

28 Flash and Ambient Images [ Agrawal, Raskar, Nayar, Li Siggraph05 ] Ambient Flash Result Reflection Layer

29 Intensity Gradient Vector Projection

30 Intensity Gradient Vectors in Flash and Ambient Images Same gradient vector direction Flash Gradient Vector Ambient Gradient Vector Ambient Flash No reflections

31 Different gradient vector direction Reflection Ambient Gradient Vector Flash Gradient Vector Ambient Flash With reflections

32 Intensity Gradient Vector Projection Reflection Ambient Gradient Vector Residual Gradient Vector Flash Gradient Vector Result Gradient Vector Ambient Flash Result Residual

33 Ambient Flash Projection = Result Residual =Reflection Layer Co-located Artifacts

34 Computational Illumination Presence or Absence, Duration, Brightness Flash/No-flash Light position Programmable dome (image re-lighting and matting) Multi-flash for depth edges Light color/wavelength Spatial Modulation Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG General lighting condition Day/Night Ramesh Raskar, Computational Illumination

35 Synthetic Lighting Paul Haeberli,, Jan 1992 Ramesh Raskar, Computational Illumination

36 Debevec et al. 2002: Light Stage 3 3 Ramesh Raskar, Computational Illumination

37 Image-Based Actual Re-lighting Debevec et al., SIGG2001 Light the actress in Los Angeles Film the background in Milan, Measure incoming light, Matched LA and Milan lighting. Matte the background Ramesh Raskar, Computational Illumination

38 Photomontage courtesy of A Agrawala courtesy of P. Debevec Ramesh Raskar, Computational Illumination

39 Ramesh Raskar, Computational Illumination

40 Table-top Computed Lighting for Practical Digital Photography Ankit Mohan, Jack Tumblin Northwestern University Bobby Bodenheimer Cindy Grimm, Reynold Bailey Vanderbilt University Washington University in St. Louis Ramesh Raskar, Computational Illumination

41 Make this darker Make this brighter Remove this specular highlight Soften this shadow Move shadow back Ramesh Raskar, Computational Illumination

42 Target Sketch Your Desires, Optimize Result Ramesh Raskar, Computational Illumination

43 Acquisition for Relighting Uniquely lit basis images Known light-positions object Ramesh Raskar, Computational Illumination

44 Aimed Spot : : low-risk movement Ramesh Raskar, Computational Illumination

45 From Jack Tumblin Ramesh Raskar, Computational Illumination

46 Overlapped Spots avoid aliasing Ramesh Raskar, Computational Illumination

47 Light Waving Tech Sketch (Winnemoller( Winnemoller,, Mohan, Tumblin, Gooch) Ramesh Raskar, Computational Illumination

48 Light Waving: Estimating Light Positions From Photographs Alone Holger Winnemöller ller,, Ankit Mohan, Jack Tumblin, Bruce Gooch Northwestern University Ramesh Raskar, Computational Illumination

49 Computational Illumination Quest for 4D Illumination Light Sources Generalized Sensor Novel Cameras Modulators Generalized Optics Processing Ray Reconstruction Generalized Optics 4D Ray Bender Upto 4D Ray Sampler Programmable 4D 4D Illumination field field + time time + wavelength 4D Light Field Display Scene: 8D Ray Modulator Ramesh Raskar, Computational Illumination

50 A 4-D 4 D Light Source [Debevec et al. 2000] [Masselus et al. 2002] [Matusik et al. 2002] [Debevec et al. 2002] [Masselus et al. 2003] [Malzbender et al. 2002] Ramesh Raskar, Computational Illumination

51 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Non-photorealistic Camera: Depth Edge Detection and Stylized Rendering using Multi-Flash Imaging Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi Yu, Matthew Turk Mitsubishi Electric Research Labs (MERL), Cambridge, MA U of California at Santa Barbara U of North Carolina at Chapel Hill

52 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Depth Edge Camera

53 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

54 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

55 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

56 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

57 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

58 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

59 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

60 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

61 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

62 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

63 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

64 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

65 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Depth Discontinuities Internal and external Shape boundaries, Occluding contour, Silhouettes

66 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Depth Edges

67 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Sigma = 9 Sigma = 5 Canny Intensity Edge Detection Sigma = 1 Our method captures shape edges

68 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Canny Our Method

69 Mitsubishi Electric Research Labs Photo MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Our Method

70 Mitsubishi Electric Research Labs Photo MultiFlash NPR Camera Result Raskar, Tan, Feris, Yu, Turk Our Method Canny Intensity Edge Detection

71 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

72 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk Shadows Clutter Many Colors Highlight Shape Edges Mark moving parts Basic colors

73 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk A New Problem Shadows Highlight Edges Clutter Mark moving parts Many Colors Basic colors

74 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

75 Mitsubishi Electric Research Labs MultiFlash NPR Camera Raskar, Tan, Feris, Yu, Turk

76 Computational Illumination Presence or Absence Flash/No-flash Light position Multi-flash for depth edges Programmable dome (image re-lighting and matting) Light color/wavelength Spatial Modulation (Intra-flash 2D Modulation) Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG General lighting condition Day/Night Ramesh Raskar, Computational Illumination

77 6-D Methods and beyond... Relighting with 4D Incident Light Fields Vincent Masselus, Pieter Peers, Philip Dutre and Yves D. Willems SIGG2003

78 Synthetic Aperture Illumination: Comparison with Long-range synthetic aperture photography width of aperture 6 number of cameras 45 spacing between cameras 5 camera s field of view Marc Levoy

79 2004 Marc Levoy The scene distance to occluder 110 distance to targets 125 field of view at target 10

80 2004 Marc Levoy Synthetic aperture photography using an array of mirrors 11-megapixel camera (4064 x 2047 pixels) 18 x 12 inch effective aperture, 9 feet to scene 22 mirrors, tilted inwards 22 views, each 750 x 500 pixels

81 2004 Marc Levoy Synthetic aperture illumination technologies array of projectors array of microprojectors single projector + array of mirrors

82 What does synthetic aperture illumination look like? 2004 Marc Levoy

83 What are good patterns? pattern one trial 16 trials

84

85 Underwater confocal imaging with and without SAP 2004 Marc Levoy

86 Computational Illumination Presence or Absence Flash/No-flash Light position Multi-flash for depth edges Programmable dome (image re-lighting and matting) Light color/wavelength Spatial Modulation Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG General lighting condition Day/Night Ramesh Raskar, Computational Illumination

87 Demodulating Cameras Simultaneously decode signals from blinking LEDs and get an image Sony ID Cam Phoci Motion Capture Cameras Visualeyez VZ4000 Tracking System PhaseSpace motion digitizer

88

89 Demodulating Cameras Decode signals from blinking LEDs + image Sony ID Cam Phoci Motion Capture Cameras

90 Mitsubishi Electric Research Labs R F I G R F I G Lamps Lamps : Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Interacting with a Self-describing World via Photosensing Wireless Tags and Projectors Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang, Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher Mitsubishi Electric Research Labs (MERL), Cambridge, MA

91 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Radio Frequency Identification Tags (RFID) No batteries, Small size, Cost few cents Antenna microchip

92 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Warehousing Routing Livestock tracking Library Baggage handling Currency

93 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Conventional Passive RFID Memory Micro Controller Memory Micro Controller Computer READER

94 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Tagged Books in a Library Id Easy to get list of books in RF range No Precise Location Data Difficult to find if the books in sorted order? Which book is upside down?

95 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Where are boxes with Products close to Expiry Date?

96 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Conventional RF tag Memory Micro Controller Conventional RFID RF Data READER Computer Photosensor Light Memory Micro Controller RF Data READER Computer Photo-sensing RF tag

97 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Memory Photosensor Micro Controller Light RF Data Projector READER Computer Photosensor? Compatible with RFID size and power needs Projector? Directional transfer, AR with Image overlay

98 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher a. Photosensing RFID tags are queried via RF b. Projector beams a time-varying pattern unique for each (x,y) pixel which is decoded by tags d. Multiple users can simultaneously work from a distance without RF collision c. Tags respond via RF, with date and precise (x,y) pixel location. Projector beams O or X at that location for visual feedback

99 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher

100 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher RFID (Radio Frequency Identification) RFIG (Radio Frequency Id and Geometry)

101 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Prototype Tag RF tag + photosensor

102 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Projected Sequential Frames Pattern MSB Pattern MSB-1 Pattern LSB Handheld Projector beams binary coded stripes Tags decode temporal code

103 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Projected Sequential Frames Pattern MSB Pattern MSB-1 Pattern LSB Handheld Projector beams binary coded stripes Tags decode temporal code

104 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Projected Sequential Frames Pattern MSB Pattern MSB-1 Pattern LSB Handheld Projector beams binary coded stripes Tags decode temporal code

105 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Projected Sequential Frames Pattern MSB Pattern MSB-1 Pattern LSB Handheld Projector beams binary coded stripes Tags decode temporal code

106 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Projected Sequential Frames Pattern MSB Pattern MSB-1 Pattern LSB Handheld Projector beams binary coded stripes Tags decode temporal code

107 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Pattern MSB Pattern MSB-1 Pattern LSB X=12 For each tag a. From light sequence, decode x and y coordinate b. Transmit back to RF reader (Id, x, y)

108 Mitsubishi Electric Research Labs R F I G Lamps Raskar, Beardsley, vanbaar, Wang, Dietz, Lee, Leigh, Willwacher Visual feedback of 2D position a. Receive via RF {(x 1,y 1 ), (x 2,y 2 ), } pixels b. Illuminate those positions

109 Computational Illumination Presence or Absence Flash/No-flash Light position Multi-flash for depth edges Programmable dome (image re-lighting and matting) Light color/wavelength Spatial Modulation Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG Natural lighting condition Day/Night Fusion Ramesh Raskar, Computational Illumination

110 A Night Time Scene: Objects are Difficult to Understand due to Lack of Context Dark Bldgs Reflections on bldgs Unknown shapes Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

111 Enhanced Context : All features from night scene are preserved, but background in clear Well-lit Bldgs Reflections in bldgs windows Tree, Street shapes Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

112 Night Image Background is captured from day-time scene using the same fixed camera Result: Enhanced Image Day Image Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

113 Mask is automatically computed from scene contrast Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

114 But, Simple Pixel Blending Creates Ugly Artifacts Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

115 Pixel Blending Our Method: Integration of blended Gradients Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

116 Nighttime image I 1 Gradient field x Y G 1 G 1 Importance image W Mixed gradient field x Y G G I 2 Daytime image x Y G 2 G 2 Gradient field Final result Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005

117 Smarter Lighting Equipment Programmable Parameters

118 Computational Illumination Light Sources Generalized Sensor Novel Cameras Modulators Generalized Optics Processing Ray Reconstruction Generalized Optics 4D Ray Bender Upto 4D Ray Sampler Programmable 4D 4D Illumination field field + Time Time + Wavelength 4D Light Field Display Recreate 4D Lightfield Scene: 8D Ray Modulator

119 Computational Illumination: Programmable 4D Illumination Field + Time + Wavelength Presence or Absence, Duration, Brightness Flash/No-flash Light position Multi-flash for depth edges Programmable dome (image re-lighting and matting) Light color/wavelength Spatial Modulation Synthetic Aperture Illumination Temporal Modulation TV remote, Motion Tracking, Sony ID-cam, RFIG Exploiting (uncontrolled) natural lighting condition Day/Night Fusion Course WebPage : people/ raskar/ photo/ Ramesh Raskar, Computational Illumination