Computational Photography

Transcription

1 Computational Photography Si Lu Spring onal_photography.htm 05/15/2018 With slides by S. Chenney, Y.Y. Chuang, F. Durand, and J. Sun.

2 Last Time o Image segmentation 2

3 Today o Matting Input user specified trimap matte foreground colors a new composite Reprint from Wang and Cohen

4 Problem of segmentation o Each pixel is assigned a binary label Foreground or Background o Cannot generate natural boundaries for semitransparent objects

5 Problem of segmentation Input Reprint from Sun et al Photoshop segmentation result

6 Segmentation and Matting o Segmentation Binary labeling, 0 or 1 o Matting A continuous value between [0, 1] 6

7 Background: Compositing o Compositing combines components from two or more images to make a new image Special effects are easier to control when done in isolation Even many all live-action sequences are more safely shot in different layers Credit: S. Chenney 7

8 Example: Perfect Storm Credit: S. Chenney 8

9 Mattes o o o o A matte is an image that shows which parts of another image are foreground objects Term dates from film editing and cartoon production How would I use a matte to insert an object into a background? How are mattes usually generated for television? Credit: S. Chenney 9

10 Alpha o Basic idea: Encode opacity information in the image o Add an extra channel, the alpha channel, to each image For each pixel, store R, G, B and Alpha alpha = 1 implies full opacity at a pixel alpha = 0 implies completely clear pixels alpha in (0,1) implies semi-transparency o There are many interpretations of alpha Is there anything in the image at that point (web graphics) Transparency (real-time OpenGL) o Images are now in RGBA format, and typically 32 bits per pixel (8 bits for alpha) Credit: S. Chenney 10

11 Working with Mattes o Compositing: insert an object into a background Call the image of the object the source Put the background into the destination For all the source pixels, if the matte is white, copy the pixel, otherwise leave it unchanged o Matting: generate mattes: Use smart selection tools in Photoshop or similar o They outline the object and convert the outline to a matte Blue Screen: Photograph/film the object in front of a blue background, then consider all the blue pixels in the image to be the background Advanced matting techniques Credit: S. Chenney 11

12 Over Compositing F B foreground color alpha matte background plate C B Credit: Y.Y. Chuang composite F =0 C C αf (1 α) B

13 Oscar Award, 1996 Smith Duff Catmull Porter Credit: Y.Y. Chuang

14 Matting problem o Inverse problem: Assume an image is the over composite of a foreground and a background o Given an image color C, find F, B and so that C= F+(1-)B B? C? F? Credit: F. Durand

15 Matting ambiguity o C= F+(1-)B o How many unknowns, how many equations? B? C? F? Credit: F. Durand

16 Matting ambiguity o C= F+(1-)B o 7 unknowns: and triplets for F and B o 3 equations, one per color channel C Credit: F. Durand

17 Matting ambiguity o C= F+(1-)B o 7 unknowns: and triplets for F and B o 3 equations, one per color channel o With known background (e.g. blue/green screen): 4 unknowns, 3 equations B C F Credit: F. Durand

18 Questions? Credit: F. Durand From Cinefex

19 Multiple backgrounds matting {, Fr, Fg, Fb } Credit: J. Sun

20 Traditional blue screen matting o Invented by Petro Vlahos (Technical Academy Award 1995) o Recently formalized by Smith & Blinn o Initially for film, then video, then digital Assume that the foreground has no blue From Cinefex Credit: F. Durand

21 Blue/Green screen matting issues o Color limitation Annoying for blue-eyed people o Blue/Green spilling The background illuminates the foreground, blue/green at silhouettes Modify blue/green channel, e.g. set to min (b, a 2 g) o Shadows How to extract shadows cast on background Credit: F. Durand

22 Blue/Green screen matting issues Credit: F. Durand From the Art & Science of Digital Compositing

23 Questions? Credit: F. Durand

24 Advanced matting techniques o Bayesian matting o Poisson matting Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung- Yeung Shum, SIGGRAPH 2004 o Robust matting o Soft Scissors o 24

25 Natural image matting o Solving complex, F, B given a single natural image and a user input trimap I F B r r F B g g F B b b Credit: J. Sun

26 Gradient Manipulation: Poisson Matting g div(g) g 2 2 x 2 2 y div(g) g x x g y y Credit: J. Sun

27 Poisson Equation Given the destination matte gradient v, the optimization problem becomes: min 2 * v with div(g) s. t. * Credit: J. Sun

28 Global Poisson Matting I F ( 1) B I ( F B) F (1 ) B I ( F B) I F B Credit: J. Sun 28

29 I F B I div( ) s.t. F B 1 0 x x F B Credit: J. Sun

30 Variational interpretation * I * arg min f s.t F B 2 I div( ) s.t. F B I F B Credit: J. Sun

31 Global Poisson matting result Credit: J. Sun

32 Next Time o Video stabilization 32