CMOS BASED HYPERSPECTRAL IMAGING FOR COMPACT / LOW-COST / HIGH-VOLUME IMAGING SPECTROSCOPY. SPIE Baltimore - April 2016

Transcription

1 CMOS BASED HYPERSPECTRAL IMAGING FOR COMPACT / LOW-COST / HIGH-VOLUME IMAGING SPECTROSCOPY SPIE Baltimore - April 2016

2 WHY DO WE NEED HYPERSPECTRAL IMAGING? to improve vision and discrimination power n... by using spectral dimension of objects being imaged 2

3 SPECTRAL IMAGING OPEN ONE NEW DIMENSION POINT SPECTROSCOPY COLOR IMAGING HYPERSPECTRAL IMAGING x y λ B G R x y n Accurate spectral analysis of one spatial pixel only Seeing RGB colors of one image only spectral signature images revealing objects chemical composition 3

4 one wavelength dimension for each pixel... x y ʎ

5 CONVERGENCE OF IMAGING & SPECTROSCOPY High discrimination power SPECTROSCOPY Single point with highest spectral resolution IMEC approach: spectral filters on multiple pixels Mosaic pattern OR Tiled pattern OR Line pattern IMAGING SPECTROSCOPY RGB filters on multiple pixels High speed, high spatial resolution IMAGING 5

6 SPECTRAL FILTERS APPLIED TO IMAGE SENSOR WAFERS BY CMOS SEMICONDUCTOR PROCESS IMEC hyperspectral filter structures processed at wafer-level on top of commercial CMOS image sensor wafer (here on CMOSIS s CMV2000 & CMV4000 sensors) 6

7 we design & manufacture spectral filters directly onto image sensor pixels... 4x4 mosaic = 16 spectral bands ~ x16 cameras on-chip! 465 nm 546 nm 586 nm 630 nm 474 nm 534 nm 578 nm 624 nm 485 nm 522 nm 562 nm 608 nm 496 nm 510 nm 548 nm 600 nm 7... based on CMOS compatible semiconductor process

8 NANO-DEPOSITED FILTERS PIXEL LEVEL! 8 down to 2um pixels! 1 to 100+ colors

9 1B pilot-line factories on-site Advanced CMOS processes (22nn 7nm) R&D to production environment Class 1 clean processing 9

10 E.G. OF POSSIBLE APPROACH FABRY-PEROT SPECTRAL FILTERS θ L θ Wavelength selection depends on cavity length L k 2nLcos Tx (%) Narrow-band & high transmission efficiencies spectral filters Different cavity heights = different spectral wavelengths captured! FWHM ~ 5-20nm 10

11 CUSTOM HYPERSPECTRAL IMAGING SOLUTIONS Filter response Spectral filters can be integrated onto commercial image sensor wafers FWHM OD Optical filters & imager can be customized to match final application requirements 1 2 n Fill the sensor ONLY with selected bands of interest Continuous Line-scan Custom Line-scan Snapshot Tiled Snapshot Bayer Mosaic 11

12 OFF-THE-SHELF SPECTRAL IMAGING SENSORS RGB RGB + NIR Multi-spectral Hyperspectral R G B R G B NIR 450nm 1000nm 450nm 1000nm 70+ commercial vendors (in development) 4 bands Snapshot mosaic & tiled 16 / 25 / 32 / 40+ bands Linescan 100 / 150+ bands 12

13 LINESCAN 100+BANDS HSI SENSOR DESIGN (GEN1) Linescan GEN1 sensor with 100+bands (from CMOSIS CMV2000, without micro-lenses) Key specifications Spectral resolution: 100+ bands in nm FWHM: ~ 10-15nm Spatial resolution: 2048 pixels x length of scan Speed: up to 340 fps (full sensor frame) 13

14 QE (%) LINESCAN 150+BANDS HSI SENSOR DESIGN (GEN 2) Linescan GEN2 sensor with 150+bands (from CMOSIS CMV2000, without micro-lenses) Key specifications Wavelength (nm) Spectral resolution: 150+ bands in nm FWHM: ~ 10-15nm Spatial resolution: 2048 pixels x length of scan Speed: up to 340 fps (full sensor frame) 14

15 SNAPSHOT TILED 32 BANDS HSI SENSOR DESIGN Snapshot tiled sensor with 32bands (from CMOSIS CMV without microlenses) Key specifications Spectral resolution: 32 bands in nm FWHM: ~ 10-15nm Spatial resolution: 256x256 RAW per band Speed: up to 340 fps (full sensor frame) 15

16 SNAPSHOT MOSAIC 4X4 HSI SENSOR DESIGN Snapshot mosaic sensor with 4x4 =16bands (from CMOSIS CMV without microlenses) Key specifications Spectral resolution: 16 bands in nm FWHM: ~ 10-15nm Spatial resolution: 512x256 RAW per band Speed: up to 340 fps (full sensor frame) 16

17 QE (%) SNAPSHOT MOSAIC 5X5 HSI SENSOR Quantum DESIGN Efficiency of hyperspectral 5x5 mosaic sensor = 25 bands sensor #XXX band 1 band 2 band 3 band 4 band Snapshot band 8 band mosaic 9 band 10sensor band 11 with band 12 5x5 =25bands (from CMOSIS CMV without microlenses) band 15 band 16 band 17 band 18 band 19 band 20 band 21 band 22 band 23 band 24 band Wavelength (nm) Key specifications Spectral resolution: 25 bands in nm FWHM: ~ 10-15nm Spatial resolution: 409x216 RAW per band Speed: up to 340 fps (full sensor frame) 17

18 ROADMAP FOR OFF-THE-SHELF HSI SENSORS Snapshot MOSAIC VIS NIR VNIR VIS nm (4x4 =16bands) NIR nm (5x5 = 25bands) VIS+NIR nm (40+bands, dual-sensor) RGB+NIR Snapshot TILED NIR GEN nm (32bands) LINESCAN Wedge NIR VNIR SWIR GEN nm (100+bands) GEN nm (150+bands) GEN um (100+ bands)

19 Bringing hyperspectral imaging to machine vision!

20 ECOSYSTEM PARTNERS FOR HSI SENSOR CAMERA INTEGRATION High speed camera Cameralink or CoaXPress interface Compact optical design with ZIFF socket to swap different HSI sensors industrial, medical and global security Adimec Flexible camera platform Giga/Ethernet interface Focus on remote sensing, UAV VRMAGIC agriculture & machine vision Cubert Ximea Smart camera Giga/Ethernet interface Reprogrammable FPGA pre-processing hypercube data inside camera Focus on industrial machine vision Tattile Smart camera GigeVision and Genicam compliant A/B shaft encoder interface (RS-422 or HTL levels). FPGA pre-processing Agriculture, food sorting and industrial applications PhotonFocus USB 3.0 vision camera High-speed USB3.0 camera Ultra compact camera: 26.4 x 26.4 x 21.2 mm 27grams weight only! 3D-One Embedded vision module Giga/Ethernet interface Motorized optics and Multi-camera head option Embedded HSI data processing include storage Handheld mobile platform First proof-of-concept of IMEC s HSI sensors integration to mobile Android OS open platform USB2.0 interface 20 Bayspec

21 HSI DEMO-KITS FOR EVALUATION 21

22 compact pc-powered ready-to-go!

23 HSI SOFTWARE TO SUPPORT APPLICATION DEVELOPMENT License available to all demo-kit customers & partners Support including installation, training and Q&A (remotely and on-site) Source-code available for OEM integration for the following blocks: Cube registration Reflectance calculation Spectral corrections 23

24 2048 pixels LINESCAN 100+BANDS HSI SENSOR (GEN1) CUBE EXAMPLE HSI data-cube (after scan) x 100+ spectral band images! Sample spectra (x3 pixels extracted from leafs) Classified image (extracting leafs only) 1024 pixels 24

25 DRIVING APPLICATIONS FOR IMEC HSI Remote sensing UAVs & cube-sattelites for: Precision agriculture Environment monitoring Terrestrial / maritime earth observation Life-science instrumentation Imaging spectroscopy analyzers DNA sequencers / flow cytometers Water monitoring analyzers Blood / urine analyzers Machine vision / Optical sorting Food sorting / quality grading Pharmaceutical defect inspection Industrial process control Bank-note inspection Print quality inspection Security / Surveillance Industrial gas leaks monitoring Intrusion detection / authentication Forensics Anti-counterfeiting Medical imaging Surgery-guided imaging Endoscopy Ophthalmology / retina imaging Wounds imaging Fluorescence microscopy Automotive & Transport Night vision systems Fuel monitoring systems Driver drowsiness 25

26 ... integration path into UAVs for precision agriculture

27

28 Soil analysis Grows monitoring Disease and weeds alerts Nutrient deficiency diagnostics Yield prediction

29 LINESCAN SENSOR IN PRECISION AGRICULTURE 862 nm 2.5cm ground resolution! RAW image spectral image vegetation index map NDVI (vegetation index) management zone map 29

30 30 snapshot hyperspectral imaging in medical

31 REAL-TIME OXYGENATION MAPS Blood Spectral Signature Spectrometer As seem by 4x4 HSI camera Reconstructed Hb/HbO/HBT maps extracted from HSI snapshot mosaic 4x4 = 16bands HSI sensor on epileptic patient during brain surgery View live video here 31

32 ATM / BANK NOTE INSPECTION Hyperspectral (5x5 mosaic NIR) RGB Courtesy of Perception Park Studio 32

33 LINESCAN HSI SENSOR FOR RICE INSPECTION good samples Classification Training Set black good black dark Training Set of Hyperspectral signatures dark black_sample dark_sample background_dish good_wholegrain Object level classification + standard vision software processing good_wholegrain wavelength (nm) Principal Component Analysis black spot dark good dark_sample 1800 black_sample background background_dish

34 HIGH SPEED ANOMALIES DETECTION WITH 25+ BANDS SNAPSHOT MOSAIC SENSOR real leaf plastic leaf View live video here 34

35 FOOD FRESHNESS MONITORING FOR FOOD QUALITY GRADING APPLICATIONS 100% inspection via gill s blood spectrum Determine food quality freshness within few hours till 4 days 35

36 SAME TECHNO INDUSTRIAL TO CONSUMER? Remote sensing Consumer Life-science instrumentation Consumer connected health Machine vision / Optical sorting Consumer Automotive & Transport Consumer Medical imaging Consumer Security / Surveillance Consumer 36

37 HSI SOFTWARE IMAGE PROCESSING PIPELINE HSI cube Camera calibration, registration, synchronization, reflectance calculation...etc Spectral un-mixing, smoothening, PCA, SVM, k- means clustering,...etc Visualization, GUI... etc Pre-Processing Feature Extraction & Selection Feature Classification/ Recognition Post-Processing Camera/Sensor Specific Application Specific IMEC can support these SOFTWARE / Application layers Application Specific (to be developed for each platform) 37

38 IMEC COLLABORATION MODEL FOR DEVELOPING HSI SOLUTIONS, APPLICATIONS & MARKETS Spectral filter materials & Fab operations Sensor & Filter co- Design Camera & System Design Low level image proc. Firmware HSI image proc. SW Appli. SW Full system feasibility study Large data-sets & services 38

39