A NOVEL VISION SYSTEM-ON-CHIP FOR EMBEDDED IMAGE ACQUISITION AND PROCESSING Neuartiges System-on-Chip für die eingebettete Bilderfassung und -verarbeitung Dr. Jens Döge, Head of Image Acquisition and Processing Group, Fraunhofer IIS / EAS
Overview Research Focus Application Challenges Software Programmable Vision-System-on-Chip Image Acquisition and Processing Building Kit Implementation Example Sheet of Light 3D Measurement System 2
Image Acquisition and Processing at Fraunhofer IIS/EAS High Performance Solutions Development of Dataflow Novel high-speed, high-dynamic-range Vision-Systems-on-Chip (VSoC) an image acquisition and processing (IAP) application building kit Estimation of alg. Complexity Image Source Camera Videofile Image Sequence Synthetic Data Filter 1 Filter 2 Filter n C++ C++ C++ Interface User Space Kernel Driver Phys. Network I/O-Hardware Environment Executable Specification (Golden Model) Image Source Cam Disk Data Processing pipeline CISModel SystemC Interface Mod 1 Mod 2 Mod n SystemC SystemC SystemC Firmware for Target MCU Phys. Network I/O-Hardware Environment methodology, algorithms and tools for application development Application Areas: Industrial metrology Integrated process control and FPGA Camera CMOS Image Sensor FPGA Embedded Soft-CPU Image Image Acquisition Processing (VHDL) (VHDL) Firmware (Linux) Camera control Postprocessing Interface Hardware Network GP-I/O Environment Final System Image Sensor System-on-Chip Micro Controler Image Pre- Sensor Focal Plane Processing Interface Sensor Interface (analog) (mixed sig.) (digital) (digital) Firmware (Linux) Camera control Postprocessing External Interface Environment (digital) Detection and tracking of moving objects 3
Application Challenge (1) Surface Inspection Measurement of power rails @vy =100 km/h Different surface characteristics Detection and classification of Contact area, Features (joints, dilations, cracks) and? Mounting parts (caps, holders) Sensor Resolution: Rz=0.1mm / Rx= Ry<1mm Min distance of recorded points: Dx=Dy <7mm Source: Fendrich, Lothar (Hrsg.): Handbuch Eisenbahninfrastruktur Images: Jens Döge High-Speed Minimum Data Volume Adaptive Sampling 4
Application Challenge (2) Activity and Presence Detection System 2D Pattern Based Privacy by Design Low Power Feature Output 5
Application Challenge (3) On-Chip Image Processing Task: Capture only frames with a certain sticker orientation! Fan rotating at 2000 rpm Processing speed 5 khz Frame rate 50 Hz On-Chip Image Analysis Feature-based Readout Trigger 6
Application Example (4) High-speed Image Analysis for Low-latency LASER Process Control Optical analysis of melt pool Temperature Total area Requirements Form (round, elongated) High Framerate Image processing Automatic ROI-determination Thresholding and binarization Morphological analysis Digital compression and readout Rule-based calculation of control parameters up to 10 khz Low-Latency down to 100µs On-Chip Image Analysis parameter deduction Laser power Scanning speed and trajectory 7
Application Example (5) Optical Sound (Vibration) Pick-up based on VSoC 8
Programmable Vision-System-on-Chip (VSoC) Concept (1) Classical Approach Camera & PC for observation and control parallel image acquisition within sensor's focal plane sequential readout, A/D-conversion and transfer of full images Transfer of full images results in a bottleneck high bandwidth requirements for image transfer latency cannot be arbitrarily reduced in closed loop operation high power requirements for input/output (IO) 9
Programmable Vision-System-on-Chip (VSoC) Concept (2) Our Approach - Early data reduction Execution of vision operations as early as possible Reduction of overall latency and bandwidth requirements total system power dissipation external hardware requirements Vision-Systems-on-Chip (VSoC) analog and digital processing at different levels of parallelism focus on the output of relevant features only autonomous operation 10
Programmable Vision-System-on-Chip (VSoC) Fundamentals Sensor matrix with 1024 x 1024 pixels @8.75µm Linear and /or logarithmic (HDR) characteristics 3 ASIP-processors for operation control Fully programmable Sensor readout (reset, acquisition, random access) Column-parallel analog and digital (SIMD) processing I/O operation with parallel, SPI and GP-I/O - I/F Software library with modules and skeletons Snapshot, rolling shutter and multiple RoI readout Column-parallel A/D-conversion with 1...10 Bit Column-parallel filtering and thresholding Linear and logarithmic data analysis Feature extraction (textures, corners) 11
Programmable Vision-System-on-Chip (VSoC) Special Features Trigger on image features Output data and RoI depending on results Activation / deactivation of data output Adaptation of output resolution (x / y / DN) Multiple RoI with different processing modes Feature analysis in RoI 1 HDR grey value output in RoI 2 Linear output with short exposure in RoI 3 Customizable binning of pixels in columns Suppression of speckle artefacts Adaptation of active area Compressed output of image features 12
Programmable Vision-System-on-Chip (VSoC) Charge Based Image Processing 1 3 t t Readout 1D convolution - normal or - windowing t - low pass filtering, - contrast enhancement t 2 4 t t Weighting Temporal filtering - amplification - attenuation - adaptation t - with or - without reset t 13
Programmable Vision-System-on-Chip (VSoC) Application-Specific Instruction Set Processor (1) Total Control of all sensor functions internal data flow local clock frequencies High speed parallel operation and multi-domain processing Low latency local control of execution and direct feedback of results into readout operation programmable GP-I/O 14
Programmable Vision-System-on-Chip (VSoC) Application-Specific Instruction Set Processor (2) Total Control of all sensor functions internal data flow local clock frequencies High speed parallel operation and multi-domain processing Low latency local control of execution and direct feedback of results into readout operation programmable GP-I/O 15
Programmable Vision-System-on-Chip (VSoC) Application-Specific Instruction Set Processor (3) Total Control of all sensor functions internal data flow local clock frequencies High speed parallel operation and multi-domain processing Low latency local control of execution and direct feedback of results into readout operation programmable GP-I/O 16
Image Acquisition and Processing (IAP) Building Kit Component Examples Python-GUI Python SDK 17
Image Acquisition and Processing (IAP) Building Kit Image Sensor Development System 18
Implementation Example Sheet of Light 3D Measurement System Dynamic Range in SoL mode Linear: >58dB feature Company A Company B Company C IIS/EAS IAP VSoC DR lin / HDR (db) 42 / - 57 / 90 53 / 120 58 / >120 + + Logarithmic: >120dB Log Algorithmic Flexibility Shutter rolling global global global / rolling Standard methods GP I/O 8/3 4/2 4/2 8/8 (SOL: max, trsh, cog) Additional new methods Table 1: sensor features RoI Company A Company B (global) (HDR / global) lines profiles / s profiles / s profiles / s 2x 8x 1024 (960) - 0.5 k 0.15 k (3.8k) (3.4k) 256 8k 2k 0.59 k 14.0 k 9.8 k 64 30 k 8k 2.3 k 42.3 k 18.4 k 8 30 k 39 k 14.8 k 103.5 k - (rolling) (filtering, log) Shutter and Synchronization Global and rolling shutter 8+8 software programmable GPIO Company C IIS/EAS IAP VSoC (HDR / global) profiles / s Table 2: maximum profile rate in high-speed mode with 1024 pixels per row 19
Summary Image acquisition + processing + Privacy by design High-speed @ low I/O bandwith Low-latency decisions Flexible adaptation to various needs Vision System-on-Chip 20
Contact Dr. Jens Döge Group Manager Image Acquisition and Processing Dirk Friebel Business Development Fraunhofer IIS/EAS Fraunhofer IIS/EAS Jens.Doege@eas.iis.fraunhofer.de Dirk.Friebel@eas.iis.fraunhofer.de +49 351 4640-831 +49 173 6560006 Fraunhofer Institute for Integrated Circuits IIS Engineering of Adaptive Systems EAS Zeunerstraße 38 01069 Dresden, Germany www.eas.iis.fraunhofer.de Visit us: hall 1, booth G42 21