CMOS BASED HYPERSPECTRAL IMAGING FOR COMPACT / LOW-COST / HIGH-VOLUME IMAGING SPECTROSCOPY SPIE Baltimore - April 2016
WHY DO WE NEED HYPERSPECTRAL IMAGING? to improve vision and discrimination power... 1 2 n... by using spectral dimension of objects being imaged 2
SPECTRAL IMAGING OPEN ONE NEW DIMENSION POINT SPECTROSCOPY COLOR IMAGING HYPERSPECTRAL IMAGING x y λ B G R x y 1 2 3 1 n Accurate spectral analysis of one spatial pixel only Seeing RGB colors of one image only spectral signature images revealing objects chemical composition 3
one wavelength dimension for each pixel... x y ʎ
CONVERGENCE OF IMAGING & SPECTROSCOPY High discrimination power SPECTROSCOPY Single point with highest spectral resolution IMEC approach: spectral filters on multiple pixels Mosaic pattern OR Tiled pattern OR Line pattern IMAGING SPECTROSCOPY RGB filters on multiple pixels High speed, high spatial resolution IMAGING 5
SPECTRAL FILTERS APPLIED TO IMAGE SENSOR WAFERS BY CMOS SEMICONDUCTOR PROCESS IMEC hyperspectral filter structures processed at wafer-level on top of commercial CMOS image sensor wafer (here on CMOSIS s CMV2000 & CMV4000 sensors) 6
we design & manufacture spectral filters directly onto image sensor pixels... 4x4 mosaic = 16 spectral bands ~ x16 cameras on-chip! 465 nm 546 nm 586 nm 630 nm 474 nm 534 nm 578 nm 624 nm 485 nm 522 nm 562 nm 608 nm 496 nm 510 nm 548 nm 600 nm 7... based on CMOS compatible semiconductor process
NANO-DEPOSITED FILTERS PATTERNED @ PIXEL LEVEL! 8 down to 2um pixels! 1 to 100+ colors
1B pilot-line factories on-site Advanced CMOS processes (22nn 7nm) R&D to production environment Class 1 clean processing 9
E.G. OF POSSIBLE APPROACH FABRY-PEROT SPECTRAL FILTERS θ L θ Wavelength selection depends on cavity length L k 2nLcos Tx (%) Narrow-band & high transmission efficiencies spectral filters Different cavity heights = different spectral wavelengths captured! FWHM ~ 5-20nm 10
CUSTOM HYPERSPECTRAL IMAGING SOLUTIONS Filter response Spectral filters can be integrated onto commercial image sensor wafers FWHM OD Optical filters & imager can be customized to match final application requirements 1 2 n Fill the sensor ONLY with selected bands of interest Continuous Line-scan Custom Line-scan Snapshot Tiled Snapshot Bayer Mosaic 11
OFF-THE-SHELF SPECTRAL IMAGING SENSORS RGB RGB + NIR Multi-spectral Hyperspectral R G B R G B NIR 450nm 1000nm 450nm 1000nm 70+ commercial vendors (in development) 4 bands Snapshot mosaic & tiled 16 / 25 / 32 / 40+ bands Linescan 100 / 150+ bands 12
LINESCAN 100+BANDS HSI SENSOR DESIGN (GEN1) Linescan GEN1 sensor with 100+bands (from CMOSIS CMV2000, without micro-lenses) Key specifications Spectral resolution: 100+ bands in 600-1000nm FWHM: ~ 10-15nm Spatial resolution: 2048 pixels x length of scan Speed: up to 340 fps (full sensor frame) 13
QE (%) LINESCAN 150+BANDS HSI SENSOR DESIGN (GEN 2) Linescan GEN2 sensor with 150+bands (from CMOSIS CMV2000, without micro-lenses) Key specifications Wavelength (nm) Spectral resolution: 150+ bands in 470-925nm FWHM: ~ 10-15nm Spatial resolution: 2048 pixels x length of scan Speed: up to 340 fps (full sensor frame) 14
SNAPSHOT TILED 32 BANDS HSI SENSOR DESIGN Snapshot tiled sensor with 32bands (from CMOSIS CMV2000 - without microlenses) Key specifications Spectral resolution: 32 bands in 600-940nm FWHM: ~ 10-15nm Spatial resolution: 256x256 RAW per band Speed: up to 340 fps (full sensor frame) 15
SNAPSHOT MOSAIC 4X4 HSI SENSOR DESIGN Snapshot mosaic sensor with 4x4 =16bands (from CMOSIS CMV2000 - without microlenses) Key specifications Spectral resolution: 16 bands in 460-630nm FWHM: ~ 10-15nm Spatial resolution: 512x256 RAW per band Speed: up to 340 fps (full sensor frame) 16
QE (%) SNAPSHOT MOSAIC 5X5 HSI SENSOR Quantum DESIGN Efficiency 14 12 10 of hyperspectral 5x5 mosaic sensor = 25 bands sensor #XXX band 1 band 2 band 3 band 4 band 5 6 7 Snapshot band 8 band mosaic 9 band 10sensor band 11 with band 12 5x5 =25bands 13 14 (from CMOSIS CMV2000 - without microlenses) band 15 band 16 band 17 band 18 band 19 band 20 band 21 band 22 band 23 band 24 band 25 8 6 4 2 0 600 650 700 750 800 850 900 950 1000 Wavelength (nm) Key specifications Spectral resolution: 25 bands in 600-960nm FWHM: ~ 10-15nm Spatial resolution: 409x216 RAW per band Speed: up to 340 fps (full sensor frame) 17
ROADMAP FOR OFF-THE-SHELF HSI SENSORS Snapshot MOSAIC VIS NIR VNIR VIS 460-640nm (4x4 =16bands) NIR 600-975nm (5x5 = 25bands) VIS+NIR 450-875nm (40+bands, dual-sensor) RGB+NIR Snapshot TILED NIR GEN1 600-975nm (32bands) LINESCAN Wedge NIR VNIR SWIR GEN1 600-975nm (100+bands) GEN2 470-925nm (150+bands) GEN1 1-1.7um (100+ bands) 2013 2014 2015 2016 2017+ 18
Bringing hyperspectral imaging...... to machine vision!
ECOSYSTEM PARTNERS FOR HSI SENSOR CAMERA INTEGRATION High speed camera Cameralink or CoaXPress interface Compact optical design with ZIFF socket to swap different HSI sensors industrial, medical and global security Adimec Flexible camera platform Giga/Ethernet interface Focus on remote sensing, UAV VRMAGIC agriculture & machine vision Cubert Ximea Smart camera Giga/Ethernet interface Reprogrammable FPGA pre-processing hypercube data inside camera Focus on industrial machine vision Tattile Smart camera GigeVision and Genicam compliant A/B shaft encoder interface (RS-422 or HTL levels). FPGA pre-processing Agriculture, food sorting and industrial applications PhotonFocus USB 3.0 vision camera High-speed USB3.0 camera Ultra compact camera: 26.4 x 26.4 x 21.2 mm 27grams weight only! 3D-One Embedded vision module Giga/Ethernet interface Motorized optics and Multi-camera head option Embedded HSI data processing include storage Handheld mobile platform First proof-of-concept of IMEC s HSI sensors integration to mobile Android OS open platform USB2.0 interface 20 Bayspec
HSI DEMO-KITS FOR EVALUATION 21
compact pc-powered ready-to-go!
HSI SOFTWARE TO SUPPORT APPLICATION DEVELOPMENT License available to all demo-kit customers & partners Support including installation, training and Q&A (remotely and on-site) Source-code available for OEM integration for the following blocks: Cube registration Reflectance calculation Spectral corrections 23
2048 pixels LINESCAN 100+BANDS HSI SENSOR (GEN1) CUBE EXAMPLE HSI data-cube (after scan) x 100+ spectral band images! Sample spectra (x3 pixels extracted from leafs) Classified image (extracting leafs only) 1024 pixels 24
DRIVING APPLICATIONS FOR IMEC HSI Remote sensing UAVs & cube-sattelites for: Precision agriculture Environment monitoring Terrestrial / maritime earth observation Life-science instrumentation Imaging spectroscopy analyzers DNA sequencers / flow cytometers Water monitoring analyzers Blood / urine analyzers Machine vision / Optical sorting Food sorting / quality grading Pharmaceutical defect inspection Industrial process control Bank-note inspection Print quality inspection Security / Surveillance Industrial gas leaks monitoring Intrusion detection / authentication Forensics Anti-counterfeiting Medical imaging Surgery-guided imaging Endoscopy Ophthalmology / retina imaging Wounds imaging Fluorescence microscopy Automotive & Transport Night vision systems Fuel monitoring systems Driver drowsiness 25
... integration path into UAVs for precision agriculture
Soil analysis Grows monitoring Disease and weeds alerts Nutrient deficiency diagnostics Yield prediction
LINESCAN SENSOR IN PRECISION AGRICULTURE 862 nm 2.5cm ground resolution! RAW image spectral image vegetation index map NDVI (vegetation index) management zone map 29
30 snapshot hyperspectral imaging in medical
REAL-TIME OXYGENATION MAPS Blood Spectral Signature Spectrometer As seem by 4x4 HSI camera Reconstructed Hb/HbO/HBT maps extracted from HSI snapshot mosaic 4x4 = 16bands HSI sensor on epileptic patient during brain surgery View live video here https://vimeo.com/132097972 31
ATM / BANK NOTE INSPECTION Hyperspectral (5x5 mosaic NIR) RGB Courtesy of Perception Park Studio 32
LINESCAN HSI SENSOR FOR RICE INSPECTION good samples Classification Training Set black good black dark 500 450 400 350 300 Training Set of Hyperspectral signatures dark 250 200 150 100 50 black_sample dark_sample background_dish good_wholegrain Object level classification + standard vision software processing good_wholegrain 0 600 650 700 750 800 850 900 950 1000 wavelength (nm) Principal Component Analysis black spot dark good dark_sample 1800 black_sample 2000 2200 2400 background 1200 1400 1600 background_dish 2600 1000 2800 800 33 3000 600
HIGH SPEED ANOMALIES DETECTION WITH 25+ BANDS SNAPSHOT MOSAIC SENSOR real leaf plastic leaf View live video here https://vimeo.com/132098267 34
FOOD FRESHNESS MONITORING FOR FOOD QUALITY GRADING APPLICATIONS 100% inspection via gill s blood spectrum Determine food quality freshness within few hours till 4 days 35
SAME TECHNO INDUSTRIAL TO CONSUMER? Remote sensing Consumer Life-science instrumentation Consumer connected health Machine vision / Optical sorting Consumer Automotive & Transport Consumer Medical imaging Consumer Security / Surveillance Consumer 36
HSI SOFTWARE IMAGE PROCESSING PIPELINE HSI cube Camera calibration, registration, synchronization, reflectance calculation...etc Spectral un-mixing, smoothening, PCA, SVM, k- means clustering,...etc Visualization, GUI... etc Pre-Processing Feature Extraction & Selection Feature Classification/ Recognition Post-Processing Camera/Sensor Specific Application Specific IMEC can support these SOFTWARE / Application layers Application Specific (to be developed for each platform) 37
IMEC COLLABORATION MODEL FOR DEVELOPING HSI SOLUTIONS, APPLICATIONS & MARKETS Spectral filter materials & Fab operations Sensor & Filter co- Design Camera & System Design Low level image proc. Firmware HSI image proc. SW Appli. SW Full system feasibility study Large data-sets & services 38