HDR videos acquisition

Transcription

1 HDR videos acquisition dr. Francesco Banterle

2 How to capture? Videos are challenging: We need to capture multiple frames at different exposure times and everything moves

3 How to capture? Different technologies based on exposure bracketing: beam-splitter; i.e. many sensors one lens stereo/multi-view HDR capturing varying exposure per pixel; i.e. bayer pattern varying shutter speed

4 Multi-sensors cameras Idea: to use more sensors to capture the same scene The light path is divided using beam splitters: careful alignment

5 Multi-sensors cameras

6 Multi-sensors cameras Debayering after HDR-merging why not before? It can corrupt colors in saturated regions It makes less visible sub-pixel misalignments of sensors

7 Multi-sensors cameras A Versatile HDR Video Production System Michael D. Tocci1,2 1 Chris Kiser1,2,3 Contrast Optical Design & Engineering, Inc. Nora Tocci1 2 University of New Mexico Pradeep Sen2,3 3 Advanced Graphics Lab Figure 1: HDR image acquired with our proposed system. On the left we show the final image acquired with our camera and merged with the proposed algorithm. The inset photos show the individual LDR images from the high, medium, and low-exposure sensors, respectively. A Versatile HDR Video Production System. Michael D. Tocci, Chris Kiser, Nora Tocci, Pradeep Sen. ACM SIGGRAPH 2011 Papers Our program. Abstract proposed system is simple, uses only off-the-shelf technology, Although High Dynamic Range (HDR) imaging has been the subject of significant research over the past fifteen years, the goal of acquiring cinema-quality HDR images of fast-moving scenes using available components has not yet been achieved. In this work, and is flexible in terms of the sensors that are used. Specifically, our HDR optical architecture: (1) captures optically-aligned, multipleexposure images simultaneously that do not need image manipulation to account for motion, (2) extends the dynamic range of avail-

8 Multi-sensors cameras Advantages: no ghosts no misalignments Disadvantages: high costs: sensors + calibration fixed dynamic range that can be captured reconstruction before debayering: complex reconstruction algorithms

9 Multi-cameras systems Idea: to use more cameras in a rig to capture the same scene: each camera has a different shutter-speed/iso A synchronization system is required

10 Multi-cameras systems Linear pattern Camera Square pattern

11 Multi-cameras systems: Geometric Calibration Geometric calibration of each camera: Intrinsic parameters: optical center, focale, pixel size in mm, field of view (angle), and aspect ratio. Extrinsic parameters; world position: position and rotation

12 Multi-cameras systems: Alignment There is the need to align other images onto a reference image (well-exposed one again!) How? Compute disparity map Warp images

13 Multi-cameras systems: Disparity Computation SSD(u, v, d) = nx k= n mx l= m 2 I 1 (u + k, v + l) I 2 (u + k + d, v + l) d o (u, v) = arg min d SSD(u, v, d) Note: typically n = m

14 Multi-cameras systems: Disparity Computation I1 I2

15 Multi-cameras systems: Disparity Computation I1 I2

16 Multi-cameras systems: Disparity Computation I1 I2

17 Multi-cameras systems: Disparity Computation I1 I2

18 Multi-cameras systems: Disparity Computation I1 I2

19 Multi-cameras systems: Disparity Computation I1 I2

20 Multi-cameras systems: Disparity Computation I1 I2

21 Multi-cameras systems: Disparity Computation I1 I2

22 Multi-cameras systems: disparity computation

23 Multi-cameras systems: disparity computation

24 Multi-cameras systems: warping

25 Multi-cameras systems: warping

26 Multi-cameras systems: warping

27 Multi-cameras systems: warping

28 Multi-cameras systems: warping

29 Multi-sensors cameras Advantages: no ghosts Disadvantages: misalignments + occlusions high costs: sensors + sync fixed dynamic range that can be captured

30 Varying exposure per pixel Idea: to apply bayer pattern not only for RGB colors but also to exposure Two possible solutions: varying gain a mask with varying neutral density filters: shutter time is not modified!

31 Varying exposure per pixel interleaved rows checkboard pattern

32 Varying exposure per pixel

33 Varying exposure per pixel

34 Varying exposure per pixel: reconstruction Ẑ Zo Z r

35 Varying exposure per pixel: reconstruction Ẑ Zo Z r

36 Varying exposure per pixel: reconstruction How can reconstruction be carried out? Linear interpolation can lead to artifacts Cubic interpolation; close to ideal sinc: Z r (x, y) = 3X 3X f(1.5 i, 1.5 j)z o (x 1.5+i, y 1.5+j) i=0 j=0 Reconstructed Kernel Signal

37 Varying exposure per pixel: reconstruction Let s see the matrix form: Z r = FZ o Ẑ = FZ o Z o = F Ẑ F = F T (FF T ) 1

38 Varying exposure per pixel

39 Varying exposure per pixel

40 Varying exposure per pixel

41 Varying exposure per pixel Advantages: low cost hardware: programmable videocameras; e.g. Canon DSLR with Magic Lantern no ghosts no misalignments Disadvantages: limited to 2-3 exposure images masks may be expensive to manufacture and difficult to align to an existing bayer pattern

42 Varying Shutter Speed Idea: to program the shutter speed or ISO; i.e. varying it at each frame Requirements: high frame rate videocamera programmable hardware

43 Varying Shutter Speed time 0 time 1 time 2 Courtesy of Jonas Unger

44 Varying Shutter Speed: reconstruction There is the need to align other images onto a reference image (well-exposed one again!) How? Compute Motion Estimation Warp images

45 Varying Shutter Speed: Motion Estimation apple u v frame t frame t+1 I t (i, i) =I t+1 (i + u, i + v)

46 Varying Shutter Speed: Motion Estimation SSD(i, j, u, v) = nx k= n mx l= m 2 I 1 (i + k, j + l) I 2 (i + k + u, j + l + v) OF o (i, j) = arg min u,v SSD(i, j, u, v) Note: this is a generalization of the disparity problem

47 Varying Shutter Speed: Motion Estimation Image courtesy of Jonas Unger

48 per block motion estimation

49 Varying Shutter Speed: Warp Image courtesy of Jonas Unger

50 Varying Shutter Speed: Warp Image courtesy of Jonas Unger

51 Varying Shutter Speed: Warp Image courtesy of Jonas Unger

52 Varying Shutter Speed: Warp Image courtesy of Jonas Unger

53 Varying Shutter Speed: Warp Image courtesy of Jonas Unger

54 Varying Shutter Speed: Advantages: Warp low cost hardware: high frame rate and programmable videocameras; e.g. Canon DSLR with Magic Lantern Disadvantages: limited to 2-3 exposure images moving camera and scene: camera alignment moving scene

55 Questions?