High Resolution Spectral Video Capture & Computational Photography Xun Cao ( 曹汛 )

Transcription

1 High Resolution Spectral Video Capture & Computational Photography Xun Cao ( 曹汛 ) School of Electronic Science & Engineering Nanjing University caoxun@nju.edu.cn Dec 30th, 2015

2 Computational Photography Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera. (Wikipedia) Computational photography is an emerging new field created by the convergence of computer graphics, computer vision and photography. Its role is to overcome the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world. (CMU Course Introduction) 2

3 CP for various imaging dimensions Depth & View (3D) Spectral(Color) Spatial Gigapixel UHD 3840*2160 HD 1920*1080 SD<720p Light Field Multiview Stereo 2D Gray scale 10Hz Multispectral RGB 30Hz Hyperspectral 60Hz 120Hz Ps.Fs Temporal Dynamic Range 8 bit 10/12 bit 24 bit

4 Computational Imaging Technology & Engineering The lab focuses on 3 kinds of computational cameras Spectral Camera High Resolution Spectral Video Camera: PMIS Multiview Stereo Camera Array High Accuracy 3D Reconstruction Super-Resolution Camera Nano-Scale Pixel Camera Gigapixel on Single Chip

5 Grayscale Imaging I( ) light source K K R( ) S( ) d S( ) scene R( ) sensor

6 Color Imaging I( ) light source ( R, G, B) R R( ) SR( ) d G R( ) SG ( ) d B R( ) SB( ) d S( ) scene R( ) sensor

7 Spectral Imaging light source R( ) I( )? scene R( )? Imaging system

8 Related Work (1) Filtered Camera based Spectrum Imaging [Kidono07] [Gat00][Yamaguchi06][Schechner02] Filter wheel Color Filter Array Programmable Filter Filter Scanning Key idea: Trade time for spectrum Prof. S.Nayar Columbia Spatially varying filter PAMI 02 Shortcomings: Incapable of capturing dynamic scenes Low spectrum resolution

9 Related Work (2) Coded Aperture Snapshot Spectral Imager (CASSI) [Brady 06] [Willett 07] [Gehm 07] [Wagadarikar 08] Key Idea: Coded Aperture 2D Imaging + reconstruction Spectrum resolution: 6 nm Spatial resolution: 256 x 248 Limited spatial resolution Limited accuracy Time-consuming reconstructing (20min / frame) Prof. D.Brady (Duke Univ.) CASSI: Applied Optics SPIE, JOSA 06-09

10 Related Work (3) Computed Tomography Imaging Spectrometer [Descour95] [Descour01] [Vandervlugt07] [Hagen08] Key Ideas: CT Projections + Reconstruction Shortcomings: Low Resolution Prof. E. Dereniak Arizona Computed Tomographic Imaging Spectrometer CTIS Difficult to Calibrate High Computational Cost different linear projections of the spectral data cube E. Dereniak Applied Optics SPIE, JOSA [JOSA 08]

11 Our Spectral Video Camera - PMIS 2008~2010: Prism-Mask Imaging Spectrometer (PMIS 1 ) Directly capture multispectral video High spectra-resolution Low cost Easy setup and calibration 2011~2014: Hybrid-Camera PMIS 2 Both high spectral and spatial resolution Real-time hyperspectral video capture 2014~now Scene-Adaptive PMIS 3 Space-time coded modulation Spectral video capture with improved accuracy and efficiency

12 A glance at PMIS 1 Pointgrey grayscale camera capturing system mask

13 System Principle Grayscale Camera Occlusion Mask Prism GIF source: Wiki Camera System Re-generated RGB Video

14 A Typical Camera lens sensor array camera

15 Camera & Prism prism lens camera sensor array Spectra Overlap!

16 Camera & Mask mask lens camera sensor array

17 Camera & Mask & Prism mask lens camera sensor array Spectra Overlap!

18 Spectral Resolution R spec W( S) a CCD cell size grayscale camera f s e WS ( ) mask prism aperture image plane sin ( ) sin ( ( )) n sin ( ( )) sin ( ( )) W( S) f (tan( ( ) a) tan( ( ) a)) e s

19 Spectral Resolution Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane W ( S) f (tan( ( ) a) tan( ( ) a)) e s

20 Spectral Resolution Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane R spec W( S) W ( S ) f (tan( ( ) a ) tan( ( ) a )) e s

21 Spectral Resolution Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane R spec W( S) W ( S ) f (tan( ( ) a ) tan( ( ) a )) e s

22 Spectral Resolution Tradeoff Spatial/Spectral Resolution f mask prism aperture image plane R spec W( S) W ( S ) f (tan( ( ) a ) tan( ( ) a )) e s

23 Spatial Resolution Small Mask Hole Distance Spectra Overlap! mask prism aperture image plane

24 Spatial Resolution Large Mask Hole Distance Unused Pixels mask prism aperture image plane

25 Spatial Resolution In practice, we can use a uniform mask Design Mask Hole Distance D d Perfect Alignment mask prism aperture image plane D d (tan( ( )) tan( ( ))) e s

26 Device Calibration

27 Calibration Overview Mapping Position to Wavelength Spectrum Calibration Geometry Calibration Radiance Calibration Non-constant CCDSensitivity Geometry Distortion caused by the prism (Smile Distortion)

28 Spectrum Calibration Spectrum Calibration Geometry Calibration Radiance Calibration

29 Spectrum Calibration captured spectra Ground truth fluorescent spectra

30 Spectrum Calibration target spectra Warp Ground truth fluorescent spectra

31 Spectrum Calibration Mapping Function : Wavelength <-> Position a f x mask prism aperture image plane sin x( ) f tan a arcsin( n sin( arcsin( ))) n Non linear, but smooth curve!

32 Geometry Calibration Spectrum Calibration Geometry Calibration Radiance Calibration

33 Geometry Calibration Predefined mask pattern captured image geometry calibrated image

34 Radiance Calibration Spectrum Calibration captured radiance genuine radiance Geometry Calibration Radiance Calibration

35 intensity sensitivity Radiance Calibration captured radiance genuine radiance z( ) b I( z) g( c( ) l( ) d ) z( ) a assuming c( ), l( ) locally constant light input wavelength l( ) g 1 ( I( z)) c( )( ) b a

36 Application 1: Human Skin Detection The W pattern in human skin reflectance [Angelopoulou01]

37 Application 1: Human Skin Detection

38 Application 2: Material Discrimination RGB Image IR Image Our measurement The differences in IR

39 PMIS 1 Conclusions Compared to Traditional Spectrometers Passive Multispectral Video Capture High spectral resolution Tradeoff spectral and spatial resolution Easy setup and calibration Applications Skin Detection Material Recognition Illumination Identification PMIS 1 : A Prism-Mask System for Multispectral Video Acquisition, IEEE Intl Conf. Computer Vision (ICCV), 2009, Oral IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 2011 High Resolution Multispectral Image Capture,US Patent

40 PMIS 1 : limitations Light throughput is limited by occlusion mask relatively small aperture Can NOT achieve both high spatial and spectral resolution Limited CCD resolution Spatial resolution (1000 pixels)

41 Scene or Object Gray Camera Low-Spatial High-Spectral Resolution Video Occlusion Mask Prism PMIS 2 : Hybrid Camera System RGB Camera High-Spatial Low-Spectral Resolution Video

42 PMIS 2 : System Pipeline Propagation

43 PMIS 2 : System Implementation

44 Propagation Algorithm Gray Camera RGB Camera High Low Spatial Resolution Resolution Multispectral RGB Video Video

45 Propagation Algorithm ms G ( d ) G ( d ) RGB xy c k r k s k k k ij RGB xy c R, G, B G ( d ) ( ) k r k G d s k ms e.g. R / R, k ij k for red channel

46 Propagation Algorithm Ground Truth Data Evaluation (11 datasets,.aix,.mat) - Spectral Database, University of Joensuu Color Group

47 Propagation Algorithm Temporal Enhancement Error 7.8% 4.3%

48 PMIS 2 : Results & Applications

49 Spatial Comparison PMIS 1 : Prism-Mask Multispectral-Video Imaging System (ICCV 09, PAMI 2011) PMIS 2 : Hybrid Camera Multispectral Video Imaging System (CVPR 2011, IJCV 2014)

50 Spatial Comparison PMIS 1 : Prism-Mask Multispectral-Video Imaging System (ICCV 09, PAMI 2011) PMIS 2 : Hybrid Camera Multispectral Video Imaging System (CVPR 2011, IJCV 2014)

51 Application 3: illumination recognition

52 Application 3: Automatic White Balance

53 Application 3: mixed illumination Fluorescent light Tungsten light

54 Application 3: mixed illumination Original Frame Fluorescent Light Tungsten Light Our Result

55 Spectral Comparison Poster vs. Water Color RGB Space Spectral Space

56 Application 4: Tracking

57 PMIS 2 Conclusions High Resolution Spectral Resolution - (1~6nm, adjustable) Spatial Resolution - ( ) Temporal Resolution - (15fps, Real-Time Capture) Additional Applications by PMIS 2 - Automatic White balance - Object Tracking PMIS 2 : Acquisition of High Spatial & Spectral Resolution Video with a Hybrid Camera System, IEEE Intl Conf. Computer Vision & Pattern Recognition. (CVPR), 2011 International Journal of Computer Vision (IJCV), 2014 A Computational Spectral Video Capture Device,China Patent. ZL X

58 PMIS 3 Basic Idea PMIS 1 & PMIS 2 Fixed-Pattern Mask Can we dynamically change the mask adaptive to the scene content?

59 PMIS 3 Prototype and Capturing Results Spatial Light Modulation Accuracy improvement PMIS 3 : Content-Adaptive High-resolution Spectral Video Acquisition Optics Letters, 39(15), pp , 2014 Optics Express, 22(16), pp , 2014 IEEE CVPR, pp , 2015 Targeted spectral acquisition by annotating regions of interest

60 Summary PMIS: High Resolution Spectral Video Camera vs Traditional Spectrometer: Snapshot Capability (Video) vs CTIS / CASSI: Real-Time Video Output Low Reconstruction Error Improved Resolution <-> Low Cost PMIS Hyperspectral Video Datasets Available : Ma C, Cao X, Dai, Q, et al. IJCV 2014 Spectra Viewing Software

61 Commercial Spectral Video Cameras Light Gene PMIS Camera Wavelength Range Spectral Resolution Temporal Resolution Spatial Resolution BaySpec nm 10 nm 8 fps 256*256 SoC nm 18 nm 60 fps 320*256 PMIS nm 6 nm 15 fps 1024*1024 (1M)

62 NSF China Acknowledgements Prof. Qionghai Dai, Dr. Steve Lin, Dr. Xin Dr. Yue Tao, Dr. Chenguang Ma Assistance in experimentation Moshe Ben-Ezra, Yanxiang Lan Helpful discussions on implementation issues

63 Computational Imaging Technology & Engineering Welcome to visit CITE Nanjing Univ.

64 The Optical Path

65 Spectra of Illuminations