Smart Image Sensing in CMOS Technology

Transcription

1 OMTA 2014 Conference Tianjin, China, Nov , 2014 Smart Image Sensing in CMOS Technology - optimization since information source Yang Ni, CTO/Prof/PhD New Imaging Technologies Bat. D 1F Impasse de la Noisette, Verrières le Buisson, F91370

2 General Architecture of Vision Machine Non-visual informations Optic Lens Visual information... Information Processing unit Generated actions Photodetector Array Feedback actions Optical Analog Analog/Digital Digital A global optimisation is necessary for high efficiency

3 Information Quality & Performance Primary analog information has paramount importance Quantization SNR is strongly dependant of signal amplitude Necessity of Pre-conditionning and regularization of analog signals Performance is strongly related to the quality and quantity of data Reversely proportionnal to the quantity of data Reversely proportionnal to the complexity of processing Data refining is prefered and necessary for high performance Increasing performance by data volume reduction Increasing performance by processing complexity reduction Optimisation at information source is necessary & also very efficient Getting stable and well-conditionned images Getting image data in more compacted and more salient form

4 ../.. s1 S/B=60db Conversion range over 10-bit S/B=50db S/B=60db s2 Conversion range over 10-bit S/B=10db If S2 can be pre-amplified x100, then much better SNR is possible.

5 ../.. 1 millions pixels per image 100 MIPS DSP 1 processed image per second 100 ins per pixel 10 K useful pixels per image 100 MIPS DSP 100 processed images per seconde 100 ins. par pixel Pre-processing generating more compact and salient images can boost considerably the final performance on the same hardware.

6 Implementation of Pre-Processing By using digital circuits Massively parallel processor array Dedicated ASIC, FPGA High perfromance DSP Impossibility to do primary analog signal conditionning and regularization! By using analog circuits Analog circuit at sensor s output Analog circuit under the columns of pixel array Analog circuit inside each pixel Resulting Smart Image Sensors Data processing by using physical laws Electronic Retina

7 Electronic Retina based Vision Machine Electronic Retina = smart pixel + massivelt parallel pre-processing Wide dynamic range with no saturation Autonomous Sensing Resulting Invariant and stable image Focusing processing power on useful data Reducing processing complexity Better performance of hardware & software Reduced data volume Reduced data bandwith Increased frame rate MPP DSP/ARM/uC PE Electronic Retina BENEFICES Higher opto-electrique performance Better processing efficiency with low power consumption Simpler, more stable and reliable hardware/software Shortened development cycle time

8 Smart Pixels Smart Pixel can & should Better conserver useful visual information Extract useful visual information Permit simplification of hardware/software Pixel Smartness depends on Targeted applications Global architecture Technology of realization Most basic smart pixel = MAGIC logarithmic pixel > 120dB intrinsic dynamic range (== in-pixel AGC) Very low Fixed Pattern Noise (<300uV with SNR=50dB ) Suited to any CMOS process

9 Classic WDR and Logarithmic WDR 1 Classic WDR techniques include Integration well adjustment (Melexis, Aptina, etc.) Multiple exposures (Pixim, SONY, etc ) Spatial variable exposures (OmniVision, SONY, etc.) Classic WDR techniques need Real-time imaging parameters adjustment Post image processing High system complexity and power consumption Logarithmic WDR No imaging parameter adjustment Directly usable WDR image Much simpler color processing

10 SNR (db) Classic WDR and Logarithmic WDR 2 50 WDR at pixel level = Native WDR linear Multiple-slope logarithmic ,001 0,01 0, illuminance (Lux) WDR from LDR pixels Logarithmic Pixel can give a seamless ultra wide dynamic range.

11 Previous developments S.G. Chamberlain, J. Lee, A Novel Wide Dynamic Range Silicon Photoreceptor and Linear Imaging Array, IEEE Journal of Solid-State Circuits, Vol. SC-19, No. 1, pp , Feb N. Ricquier, B. Dierickx, Pixel structure with logarithmic response for intelligent and flexible imager architectures, ESSDERC 92; published in Microelectronics Engineering vol.19, p.631 (1992). U. Seger, & al., Vision Assistance in Scenes with Extreme Contrast, IEEE MICRO, pp , T. Delbrück, and C.A. Mead, Analog VLSI Phototransduction by continuous time, adaptive, logarithmic photoreceptor circuits, California Institute of Technology, Computation and Neural Systems program, CNS Memorandum 30, CA 91125, K.Takada, S. Miyatake, Logarithmic-Converting CCD Line Sensor and Its Noise Characteristics, IISW 1997, p6-1/p6-4. S. Kavadias, B. Dierickx, G. Meynants, A self-calibrating logarithmic image sensor IEEE Workshop on CCD&AIS, Nagano Japan, June M. Loose, K. Meier, and J. Schemmel, A Self-Calibrating Single - Chip CMOS Camera with Logarithmic Response, IEEE Journal Solid - State Circuits, Vol.36, No.4, pp , 2001 High FPN Low Sensitivity Large image lag Loss of sensitivity with T!!

12 NIT s Solar-cell Log Pixel Design Y. Ni, F. Lavainne, F. Devos, "CMOS compatible photoreceptor for high-contrast car vision", Intelligent Vehicle Highway Systems, SPIE's International Symposium on Photonics for Industrial Applications, Oct.-Nov. 1994, Boston, pp Y. Ni, K. Matou, "A CMOS Log Image Sensor with on-chip FPN Compensation", ESSCIRC'01, Sept Villach, Austria, pp Physically exact dark reference on-chip FPN correction Improved sensitivity No image lag High Quality Image

13 Excellent High Temperature Behavior No sensitivity loss at high temperature (unique feature) Only linear-to-temperature offset (easy to compensate)

14 Extension to III-V InGaAs material Motivations WDR needed in SWIR band (0.9um - 1.7um) Higher dark current than in Silicon More uniformity in solar-cell mode design World unique InGaAs sensors > 120dB intra-scene DR No NUC or simple 1-point NUC No cooling needed Small and compact InGaAs cameras Raw Image > 120dB Intrascene DR Ultra-compact InGaAs camera

15 Sample videos High contrast scenes seen by Widy InGaAs sensor CMOS 768x um D1 logarithmic sensor Welding monitoring with NIT GS Log sensor

16 Analog Pre-Processings Objectives of Analog Pre-processing Data volume reduction Data regularization Spatial/Temporal filtering and Noise reduction Visual features detection Etc. Highly application oriented Part of global system level optimization Mostly hard wired Exemples Retina for 3D stereovision Retina for optical touch screen

17 3D Stereovision 1 Electronic Retinas Image sensing Image conditionning & regularization Spatial filters Visual features extraction Initial image DoG filtered image ASIC/FPGA/DSP Disparity Map Extracted MAX/MIN Programmable MAX/MIN width W Digital processing Visual features matching 3D map generation If W > Maximum Disparity, then direct error-free matching can be done between the extracted visual features (MAX/MIN). => Great simplification!!

18 3D Stereovision 2 AD Y 256x256 Pixel Array Analog Line Buffer Histogram based Regularization Ctrl Sig. Gaussian Filter Analog DoG Filter AD X Feature Extractor Decoder Extracted visual features for 3D stereo-matching

19 outcmp s p i x s r a m p e h i s t m e m pdrst outpixrst outpix o u t i n v 1 o u t i n v 2 o u t i n v 3 h i s t a l i m h i s t a m p o u t h i s t e q 3D Stereovision 3 pdalim dec_x inv1rst inv2rst dec_x Y AD Active Pixel Array 256x256 hv caisson ROW pixsuiv Vsspix pix pix rampe histsrci histsuiv Line buffer Histogram based Regularization Ctrl Sig. Gaussian Filter Differentiator & DoG diff cmp dec_x X AD MIN/MAX extractor Decoder shift cmplatch 256x256-pixel at 50 images/s with < 50mW power consumption (realised in CMOS 0.8um)

20 Electronic Retina for Optical Touch Screen 1 Retina A A Retina B B Against parasite ambiant lights Differential Image Sensing synchronized with Pulsed LED illumination Disalignment Compensation Smart Image Readout Single image line formed from an arbitary curve Reflective edges Electronic Retina Ambiant light suppression (stable image) Smart 2D to 1D conversion (volume reduction) Using standard CMOS process Pointer position is calculated by triangulation from the left and right shadows.

21 Electronic Retina for Optical Touch Screen 2 Highly efficient ambiant light suppression Suppression ration > 74dB 256x1600 pixels with 4.8um pitch CMOS 0.18um >12Mhz scanning speed Power consumption < 50mW

22 FPGA Electronic Retina for Optical Touch Screen 3 Smart Image Reading Window of Interest Line of Interest DRAM Useful data Complexe hardware, high cost and low performance Window of Interest Line of Interest NIT inside Only useful data NIT Retina Simpler, better and cheaper system!!

23 Smart Sensing with MAGIC pixels Highly reliable 3D vision for all conditions Smart log pixel detects changes everywhere! A lot to imagine!

24 Conclusions We have presented a new Vision Machine concept based on electronic retina This concept is inspired both from biology and the necessarity of global system level optimization of visual information processing. We have illustrated the importance of visual information pre-processing close to photodetector array and also the great benefices obtainable. We have demonstrated that simpler, better & cheaper systems can be realized by adopting this concept through some concrete realizations, both academic and industrial. Thank you for your attention!

25 Shortcoming and further directions Only 3T-pixel equivalent low light performance No possibility to compensate KTC noise No pinned photodiode and charge transfer mechanism Pixel pitch reduction 5um pixel pitch with 0.35um-like 0.18um process Further pixel compacting needed Further developments on the way Using more advanced process nodes Investigating noise reduction techniques both by design and by signal processing