A NOVEL VISION SYSTEM-ON-CHIP FOR EMBEDDED IMAGE ACQUISITION AND PROCESSING

Size: px

Start display at page:

Download "A NOVEL VISION SYSTEM-ON-CHIP FOR EMBEDDED IMAGE ACQUISITION AND PROCESSING"

Clare Thomas
5 years ago
Views:

1 A NOVEL VISION SYSTEM-ON-CHIP FOR EMBEDDED IMAGE ACQUISITION AND PROCESSING Neuartiges System-on-Chip für die eingebettete Bilderfassung und -verarbeitung Dr. Jens Döge, Head of Image Acquisition and Processing Group, Fraunhofer IIS / EAS

2 Overview Research Focus Application Challenges Software Programmable Vision-System-on-Chip Image Acquisition and Processing Building Kit Implementation Example Sheet of Light 3D Measurement System 2

Image Acquisition and Processing at Fraunhofer IIS/EAS High Performance Solutions Development of Dataflow Novel high-speed, high-dynamic-range Vision-Systems-on-Chip (VSoC) an image acquisition and

Complexity Image Source Camera Videofile Image Sequence Synthetic Data Filter 1 Filter 2 Filter n C++ C++ C++ Interface User Space Kernel Driver Phys.

3 Image Acquisition and Processing at Fraunhofer IIS/EAS High Performance Solutions Development of Dataflow Novel high-speed, high-dynamic-range Vision-Systems-on-Chip (VSoC) an image acquisition and processing (IAP) application building kit Estimation of alg. Complexity Image Source Camera Videofile Image Sequence Synthetic Data Filter 1 Filter 2 Filter n C++ C++ C++ Interface User Space Kernel Driver Phys. Network I/O-Hardware Environment Executable Specification (Golden Model) Image Source Cam Disk Data Processing pipeline CISModel SystemC Interface Mod 1 Mod 2 Mod n SystemC SystemC SystemC Firmware for Target MCU Phys. Network I/O-Hardware Environment methodology, algorithms and tools for application development Application Areas: Industrial metrology Integrated process control and FPGA Camera CMOS Image Sensor FPGA Embedded Soft-CPU Image Image Acquisition Processing (VHDL) (VHDL) Firmware (Linux) Camera control Postprocessing Interface Hardware Network GP-I/O Environment Final System Image Sensor System-on-Chip Micro Controler Image Pre- Sensor Focal Plane Processing Interface Sensor Interface (analog) (mixed sig.) (digital) (digital) Firmware (Linux) Camera control Postprocessing External Interface Environment (digital) Detection and tracking of moving objects 3

Application Challenge (1) Surface Inspection Measurement of power rails @vy =100 km/h Different surface

Mounting parts (caps, holders) Sensor Resolution: Rz=0.

4 Application Challenge (1) Surface Inspection Measurement of power =100 km/h Different surface characteristics Detection and classification of Contact area, Features (joints, dilations, cracks) and? Mounting parts (caps, holders) Sensor Resolution: Rz=0.1mm / Rx= Ry<1mm Min distance of recorded points: Dx=Dy <7mm Source: Fendrich, Lothar (Hrsg.): Handbuch Eisenbahninfrastruktur Images: Jens Döge High-Speed Minimum Data Volume Adaptive Sampling 4

5 Application Challenge (2) Activity and Presence Detection System 2D Pattern Based Privacy by Design Low Power Feature Output 5

Fan rotating at 2000 rpm Processing speed 5 khz Frame

6 Application Challenge (3) On-Chip Image Processing Task: Capture only frames with a certain sticker orientation! Fan rotating at 2000 rpm Processing speed 5 khz Frame rate 50 Hz On-Chip Image Analysis Feature-based Readout Trigger 6

7 Application Example (4) High-speed Image Analysis for Low-latency LASER Process Control Optical analysis of melt pool Temperature Total area Requirements Form (round, elongated) High Framerate Image processing Automatic ROI-determination Thresholding and binarization Morphological analysis Digital compression and readout Rule-based calculation of control parameters up to 10 khz Low-Latency down to 100µs On-Chip Image Analysis parameter deduction Laser power Scanning speed and trajectory 7

8 Application Example (5) Optical Sound (Vibration) Pick-up based on VSoC 8

transfer of full images Transfer of full images results in a bottleneck high bandwidth requirements for

9 Programmable Vision-System-on-Chip (VSoC) Concept (1) Classical Approach Camera & PC for observation and control parallel image acquisition within sensor's focal plane sequential readout, A/D-conversion and transfer of full images Transfer of full images results in a bottleneck high bandwidth requirements for image transfer latency cannot be arbitrarily reduced in closed loop operation high power requirements for input/output (IO) 9

system power dissipation external hardware requirements Vision-Systems-on-Chip (VSoC) analog and digital

10 Programmable Vision-System-on-Chip (VSoC) Concept (2) Our Approach - Early data reduction Execution of vision operations as early as possible Reduction of overall latency and bandwidth requirements total system power dissipation external hardware requirements Vision-Systems-on-Chip (VSoC) analog and digital processing at different levels of parallelism focus on the output of relevant features only autonomous operation 10

Programmable Vision-System-on-Chip (VSoC) Fundamentals Sensor matrix with 1024 x 1024 pixels @8.

11 Programmable Vision-System-on-Chip (VSoC) Fundamentals Sensor matrix with 1024 x 1024 Linear and /or logarithmic (HDR) characteristics 3 ASIP-processors for operation control Fully programmable Sensor readout (reset, acquisition, random access) Column-parallel analog and digital (SIMD) processing I/O operation with parallel, SPI and GP-I/O - I/F Software library with modules and skeletons Snapshot, rolling shutter and multiple RoI readout Column-parallel A/D-conversion with Bit Column-parallel filtering and thresholding Linear and logarithmic data analysis Feature extraction (textures, corners) 11

Programmable Vision-System-on-Chip (VSoC) Special Features Trigger on image features Output data and RoI depending on results Activation / deactivation of data output Adaptation of output resolution

12 Programmable Vision-System-on-Chip (VSoC) Special Features Trigger on image features Output data and RoI depending on results Activation / deactivation of data output Adaptation of output resolution (x / y / DN) Multiple RoI with different processing modes Feature analysis in RoI 1 HDR grey value output in RoI 2 Linear output with short exposure in RoI 3 Customizable binning of pixels in columns Suppression of speckle artefacts Adaptation of active area Compressed output of image features 12

13 Programmable Vision-System-on-Chip (VSoC) Charge Based Image Processing 1 3 t t Readout 1D convolution - normal or - windowing t - low pass filtering, - contrast enhancement t 2 4 t t Weighting Temporal filtering - amplification - attenuation - adaptation t - with or - without reset t 13

14 Programmable Vision-System-on-Chip (VSoC) Application-Specific Instruction Set Processor (1) Total Control of all sensor functions internal data flow local clock frequencies High speed parallel operation and multi-domain processing Low latency local control of execution and direct feedback of results into readout operation programmable GP-I/O 14

frequencies High speed parallel operation and multi-domain processing Low latency

15 Programmable Vision-System-on-Chip (VSoC) Application-Specific Instruction Set Processor (2) Total Control of all sensor functions internal data flow local clock frequencies High speed parallel operation and multi-domain processing Low latency local control of execution and direct feedback of results into readout operation programmable GP-I/O 15

16 Programmable Vision-System-on-Chip (VSoC) Application-Specific Instruction Set Processor (3) Total Control of all sensor functions internal data flow local clock frequencies High speed parallel operation and multi-domain processing Low latency local control of execution and direct feedback of results into readout operation programmable GP-I/O 16

17 Image Acquisition and Processing (IAP) Building Kit Component Examples Python-GUI Python SDK 17

18 Image Acquisition and Processing (IAP) Building Kit Image Sensor Development System 18

19 Implementation Example Sheet of Light 3D Measurement System Dynamic Range in SoL mode Linear: >58dB feature Company A Company B Company C IIS/EAS IAP VSoC DR lin / HDR (db) 42 / - 57 / / / > Logarithmic: >120dB Log Algorithmic Flexibility Shutter rolling global global global / rolling Standard methods GP I/O 8/3 4/2 4/2 8/8 (SOL: max, trsh, cog) Additional new methods Table 1: sensor features RoI Company A Company B (global) (HDR / global) lines profiles / s profiles / s profiles / s 2x 8x 1024 (960) k 0.15 k (3.8k) (3.4k) 256 8k 2k 0.59 k 14.0 k 9.8 k k 8k 2.3 k 42.3 k 18.4 k 8 30 k 39 k 14.8 k k - (rolling) (filtering, log) Shutter and Synchronization Global and rolling shutter 8+8 software programmable GPIO Company C IIS/EAS IAP VSoC (HDR / global) profiles / s Table 2: maximum profile rate in high-speed mode with 1024 pixels per row 19

20 Summary Image acquisition + processing + Privacy by design low I/O bandwith Low-latency decisions Flexible adaptation to various needs Vision System-on-Chip 20

Contact Dr. Jens Döge Group Manager Image Acquisition and Processing Dirk Friebel Business Development Fraunhofer IIS/EAS Fraunhofer IIS/EAS Jens.Doege@eas.iis.fraunhofer.

21 Contact Dr. Jens Döge Group Manager Image Acquisition and Processing Dirk Friebel Business Development Fraunhofer IIS/EAS Fraunhofer IIS/EAS Fraunhofer Institute for Integrated Circuits IIS Engineering of Adaptive Systems EAS Zeunerstraße Dresden, Germany Visit us: hall 1, booth G42 21

A NOVEL VISION-SYSTEM-ON-CHIP FOR EMBEDDED IMAGE ACQUISITION AND PROCESSING

A NOVEL VISION-SYSTEM-ON-CHIP FOR EMBEDDED IMAGE ACQUISITION AND PROCESSING Dr. Jens Döge, Head of Image Acquisition and Processing Group, Fraunhofer Institute for Integrated Circuits IIS, Division Engineering