Neuromorphic Event-Based Vision Sensors

Size: px

Start display at page:

Download "Neuromorphic Event-Based Vision Sensors"

Shon Stevens
5 years ago
Views:

Inst. of Neuroinformatics www.ini.uzh.ch Conventional cameras (aka Static vision sensors) deliver a stroboscopic sequence of frames Silicon Retina Technology Tobi Delbruck Inst.

com Sponsors: Swiss National Science Foundation NCCR Robotics project, EU projects SEEBETTER and VISUALISE, Samsung, DARPA, University of Zurich and ETH Zurich 1 Good Compatible with 50+years of MV

power trade-off 3 The Human Eye as a digital camera 100M photoreceptors 1M output fibers carrying max 100Hz spike rates >20 different eyes 180dB (10 9 ) operating range 3mW power consumption >GFLOPs

that mimics the optic nerve output of the eye Mahowald & Mead outer retina UPenn Magno-Parvo silicon retina CSEM Devise steerable-filter contrast vision sensor Yale Univ.

1 Inst. of Neuroinformatics Conventional cameras (aka Static vision sensors) deliver a stroboscopic sequence of frames Silicon Retina Technology Tobi Delbruck Inst. of Neuroinformatics, University of Zurich and ETH Zurich Shih-Chii Liu Sensors Group sensors.ini.uzh.ch inilabs GmbH inilabs.com Sponsors: Swiss National Science Foundation NCCR Robotics project, EU projects SEEBETTER and VISUALISE, Samsung, DARPA, University of Zurich and ETH Zurich 1 Good Compatible with 50+years of MV Allows small pixels (3.5um for MV) Golf-guides.blogspot.com Muybridge 1878 (150 years ago) Bad Redundant output Temporal Aliasing Limited dynamic range (60dB) Faces fundamental latency vs. power trade-off 3 The Human Eye as a digital camera 100M photoreceptors 1M output fibers carrying max 100Hz spike rates >20 different eyes 180dB (10 9 ) operating range 3mW power consumption >GFLOPs computing power Neuromorphic Event-Based Vision Sensors 4 5 Historical development of Event-Based Vision Sensors Dynamic Vision Sensor (DVS) delivers data as a sparse asynchronous stream of digital that mimics the optic nerve output of the eye Mahowald & Mead outer retina UPenn Magno-Parvo silicon retina CSEM Devise steerable-filter contrast vision sensor Yale Univ. Octopus imager JHU Temporal Change Threshold Detection Imager UZH-ETH Dynamic Vision Sensor (DVS) AIT Asynchronous Time-Based Image Sensor (ATIS) Seville Sensitive DVS UZH-ETH / inilabs Dynamic and Active Pixel Vision Sensor (DAVIS) UZH-ETH / inilabs CDAVIS, QVGA+VGA DAVIS, BSIDAVIS 6 Lichtsteiner, P., Posch, C., and Delbrück, T. (2008). A 128 x dB 15us Latency Asynchronous Temporal Contrast Vision Sensor. IEEE J. Solid-State Circuits 43,

DVS (Dynamic Vision Sensor) Pixel threshold The

Journal of Solid-State Circuits 2014 9 Uniform

1 lux) with high contrast text Photocurrent is

1 density chart Illumination ratio=135:1 High

Frame rate equivalent to 10 khz but 100x less

2 DVS (Dynamic Vision Sensor) Pixel threshold The DAVIS pixel I logi photoreceptor Event reset change amplifier (bipolar cells) ON OFF comparators (ganglion cells) Change Lichtsteiner et al., 2007 ±ΔlogI From Rodieck Brandli et al., IEEE Journal of Solid-State Circuits Uniform event threshold and wide dynamic range Low light performance ON 780 lux 5.8 lux Shot under moonlight (<0.1 lux) with high contrast text Photocurrent is <20% of dark current! 780 lux 5.8 lux Edmund 0.1 density chart Illumination ratio=135:1 High speed (low data rate) imaging Data rate <1MBps Frame rate equivalent to 10 khz but 100x less data Asynchronous Time-Based Image Sensor (ATIS) ATIS uses DVS to trigger local intensity measurements (10 khz image sensor x 16k pixels = 160 MBps) 14 2

ATIS Operation 15 Change detection (ON/OFF) Intensity (no background) Event-updated video (full background DAVIS (Dynamic and Active Pixel Vision Sensor) Pixel Intensity reset Intensity value I logi

, 2007 ±ΔlogI Courtesy Christoph Posch, ISCAS 2014 From Rodieck 1998 16 Because we used the full Tower CMOS image sensor (CIS) process Thick microlenses for large pixels RGB color filters

19mm Tower wafer run (SBRetFinal) taped out 20 July 2014 2 3 4 6 8 9 10 1 1 29mm Chips on the reticle set are: 1. 640x480 VGA DAVIS 2. 128x128 DAVIS for embedded apps 3. 64x64 exptl.

3 ATIS Operation 15 Change detection (ON/OFF) Intensity (no background) Event-updated video (full background DAVIS (Dynamic and Active Pixel Vision Sensor) Pixel Intensity reset Intensity value I logi photoreceptor Event reset change amplifier (bipolar cells) threshold ON OFF comparators (ganglion cells) Change Lichtsteiner et al., 2007 ±ΔlogI Courtesy Christoph Posch, ISCAS 2014 From Rodieck Because we used the full Tower CMOS image sensor (CIS) process Thick microlenses for large pixels RGB color filters Antireflection coating Buried and pinned photodiodes: 3 times less dark current Deep P implant: higher quantum efficiency, shields switches Photodiode substrate Cross section view of a pixel mm Tower wafer run (SBRetFinal) taped out 20 July mm Chips on the reticle set are: x480 VGA DAVIS x128 DAVIS for embedded apps 3. 64x64 exptl. Correlation Filter x260 SBRetFinal BSI DAVIS 5. Low power 0.5V binaural cochlea x260 Mono DAVIS 7. 4 ear AER-EAR cochlea x260 DAVIS with CFA 9. SBRet21 fail-safe DAVIS x192 sensitive Bernabe DVS/DAVIS x480 RGB DAVIS Oct finished wafers received from Towerjazz FSI RGB FSI Mono BSI Mono 3

Towerjazz FSI RGB full wafer Jan 26 2015 One FSI RGB wafer

Vision Sensor 25 2 streams vision sensors Heterogeneous

Event + Frame Dynamic and Active-pixel VIsion Sensor (DAVIS)

hi-res frame Dynamic and Color Active-pixel VIsion Sensor

An RGBW Color VGA Rolling and Global Shutter Dynamic and

4 Towerjazz FSI RGB full wafer Jan One FSI RGB wafer diced and packaged The CDAVIS Dynamic and Color Active Pixel Vision Sensor 25 2 streams vision sensors Heterogeneous CDAVIS pixel array Event Frame DVS APS (Fossum et al, 1993) Event + Frame Dynamic and Active-pixel VIsion Sensor (DAVIS) (Brandli et al, 2014) CFA (Bayer, 1975) Event + Color & hi-res frame Dynamic and Color Active-pixel VIsion Sensor (CDAVIS) C. Li et al. An RGBW Color VGA Rolling and Global Shutter Dynamic and Active-Pixel Vision Sensor, presented at the 2015 International Image Sensor Workshop (IISW 2015), Vaals, Netherlands,

Space-time view of the 2 streams of output CDAVIS Frame and event recording

18um CIS (Towerjazz) 640 x 480 frame Resolution 320 x 240 event 2x2 pixels:

15-32mW Power 4FPS frame + event: 106-120mW (w/on chip ADC) Back-side

potentially 100% fill-factor, compared with original 20% 31 BSI vs FSI DAVIS

approx. equivalent levels Back side illuminated: 0.

5 Space-time view of the 2 streams of output CDAVIS Frame and event recording of a lab scene Process 6M1P 0.18um CIS (Towerjazz) 640 x 480 frame Resolution 320 x 240 event 2x2 pixels: Pixel size 20x20um 2 14% (not Fill factor considering microlenses) DVS alone: 15-32mW Power 4FPS frame + event: mW (w/on chip ADC) Back-side illumination The BSIDAVIS Backside Illuminated silicon retina 30 BSI offers potentially 100% fill-factor, compared with original 20% 31 BSI vs FSI DAVIS sensitivity comparison same lens, same f/#, exposure duration adjusted for approx. equivalent levels Back side illuminated: 0.2ms exposure Front side illuminated: 3ms exposure Algorithms Conclusion: BSI is approx. 15X more sensitive. 3X from lack of CFA filter. Remaining 5X from BSI 34 5

Low level vision: Orientation features Example: Using synchrony to label with orientation

Record event time in spatial map 2. Find most synchronous orientation 3.

detect most coincident orientation and output corresponding orientation event Delbruck, 2007, U Tokyo Symp

(RectangularClusterTracker in jaer) For each packet of 1. For each event 1.

cluster 2. Prune starved clusters 3. Merge clusters, etc. Advantages 1. Low computational cost (e.g. <5% CPU) 2.

6 Low level vision: Orientation features Example: Using synchrony to label with orientation (SimpleOrientationFilter in jaer) DVS in Map of last event times Orientation For each event: 1. Record event time in spatial map 2. Find most synchronous orientation 3. Output orientation event encoding this orientation Using this picture of time, for each incoming event, detect most coincident orientation and output corresponding orientation event Delbruck, 2007, U Tokyo Symp 35 Delbruck, 2012, ECCV workshop, DvsOrientationFilter in jaer 36 High level vision: Tracking (RectangularClusterTracker in jaer) For each packet of 1. For each event 1. Find nearest cluster a) If event within a cluster, move cluster b)if event not within cluster, seed new cluster 2. Prune starved clusters 3. Merge clusters, etc. Advantages 1. Low computational cost (e.g. <5% CPU) 2. No frame memory (~100 bytes/object). 3. No frame correspondence problem 37/30 Robot Goalie 38 Achieves 550 FPS and 3 ms reaction time at 4% processor load with USB bus connections Working with event sensors

Demonstrations jaer/caer are open-source software projects (at http://jaerproject.net ) that allow you to do real time processing with event-based sensors on PCs.

7 Demonstrations jaer/caer are open-source software projects (at ) that allow you to do real time processing with event-based sensors on PCs. jaer/caer and the event-based sensors work together. Interaction: You can purchase a sensor at subsidized cost from inilabs.com, you can develop algorithms and join this community

Copyright T. Delbruck,

Copyright T. Delbruck, Spiking silicon retina for digital vision Inst. of Neuroinformatics, UNI-ETH Zurich Tobi Delbruck Inst. of Neuroinformatics UZH-ETH Zurich Switzerland Patrick Lichtsteiner PhD project Funding: UZH-ETH