Image Signal Processing for a Camera Monitor System with an ZYNQ FPGA

Transcription

1 Image Signal Prcessing fr a Camera Mnitr System with an ZYNQ FPGA M. Sc. Hchschule Ulm Prgrammable Prcessing fr the Future and Autnmus Car Frm classical FPGA t hetergeneus MPSC

2 Mtivatin Image Signal Prcessing fr a Camera Mnitr System with a ZYNQ FPGA Better aerdynamics Reductin f blind spt Data prcessing pssible Object tracking Blind spt detectin 2 Cncept car f the Daimler AG [Media.Daimler.cm]

3 Table f cntents Intrductin f the tpic Definitin f the cmpnents Camera Mnitr Nrmative Framewrk based n ISO and UN Regulatin N. 46 The image Signal Prcessing device Descriptin f the Signal Prcessing functins Cnclusin Data Prcessing cncept Image Signal Prcessing system Data Preprcessing Infrmatin visualizatin Resurce management 3

4 Intrductin f the tpic Autmated Driving 4 SAE Levels [Frbes.cm] Gal: Embedded flexible platfrm t develp and investigate signal prcessing tlchains and architectures RC-Car [HSU]

5 Intrductin f the tpic Camera Mnitr System (CMS) Image data Image data Camera ECU Status and Cntrl Status and Cntrl Mnitr Raw data Data preprcessing Data Data prcessing Infrmatin Infrmatin visualizatin Image Definitin f a Camera Mnitr system by [HSU & 1] 5

6 Definitin f the cmpnents The Camera Camera ECU Display Avnet FMC-Mdule HDMI input and utput LCED-Interface fr camera cnnectin Parallel FMC-Interface fr ZYNQ-Bard cnnectin ON Semicnductr VITA-200 Image Sensr Reslutin 1920 x 1080 Pixel Max. 92 fps LCED-Interface (LCD Caxial Embedded Display Interface) fr data transmissin LVDS and SPI-Interface fr cnfiguratin LCED-Interface Develped by TE [TE.cm] 2.7 Gbps/lane data rate 10.8 Gbps bandwidth Fur lanes Camera with FMC- Mdule [HSU] 6

7 Definitin f the cmpnents The Mnitr Camera ECU Display The used Mnitr is standard 24 Inch PC cmpnent Minimum 1024 x 768 Pixels Minimum 60 fps HDMI-Interface cnnectin Vehicle Mnitr fr Class III mirrrs 8 Inch Mnitr 800 x 480 Pixel 200 Mbit/s data rate Minimum 30 fps Mnitr has t be ISO and UN Reg. N. 46 cnfirm accrding t [2 & 3] Labratry Mnitr with test image [HSU] 7

8 Definitin f the cmpnents Nrmative Framewrk fr the parameter based n ISO 16505:2015 [2] and UN Regulatin N. 46 [3] 8 Standard and regulatin cvering min. requirements and test methds fr CMS Field f view shall cver at least the field f view f the cnventinal mirrr (hrizntal & vertical) Reslutin defines minimum distinguishable details Lcatin f the Mnitr inside f vehicle is defined Image quality is defined fr Artefacts / Smear Blming and lens flare Clr nise Sharpness Pixel faults Etc. Timing behavir als set, minimum frame rate f 30 Hz (15 Hz at lw light cnditin) Overlays

9 Definitin f the cmpnents The image Signal Prcessing device Camera ECU Display ZYNQ-7000 SC Evaluatin Bard (ZC702) XC7Z020 chip Definitin f the pictured cmpnents 1. Camera 2. Interface: LCEDI 3. FMC IMAGEON HDMI Mdule 4. HDMI/DVI Output 5. JTAG USB Prt 6. UART USB Prt 7. DC current cnnectr 9 Hardware setup f the mirrr replacement system [HSU]

10 Definitin f the cmpnents The image Signal Prcessing device Camera ECU Display The Xilinx ZYNQ ZC702 chip Part f the ZYNQ-7000 all prgrammable SC family Includes a Dual-Cre ARM Crtex-A9 Prcessr and an FPGA The FPGA: 85 K Lgic Cells 4.9 Mb Blck RAM 220 DSP Slices 200 I/O Pins 200 MHz Clck The ARM Crtex-A9: Dual-Cre Up t 2 GHz Clck 8 MB Cache 10 Xilinx ZYNQ-7000 SC [Xilinx.cm]

11 Descriptin f the Signal Prcessing functins Data Prcessing cncept Camera ECU Display One base f image signal prcessing is bject recgnitin Input t system is an image Image cnsists f pixel Each pixel carries little infrmatin Large number f pixels describe scene Preprcessing already implemented Parameters cnfigurable System architecture can be adjusted t different use cases Objects Features Pixels High Level Intermediate Level Lw Level Image Prcessing pyramid defined by [4] Recgnitin Classificatin Segmentatin Preprcessing 11

12 Descriptin f the Signal Prcessing functins Image Signal Prcessing system Camera ECU Display ECU Raw data Data preprcessing Data Data prcessing Infrmatin Infrmatin visualizatin Image 12 Image Signal Prcessing tlchain design by [HSU]

13 Descriptin f the Signal Prcessing functins Image Signal Prcessing system Camera ECU Display Implemented image Signal Prcessing system n the FPGA [HSU & 5] 13

14 Descriptin f the Signal Prcessing functins Data Preprcessing Data preprcessing is used t reduce size f image Prcessing Blck Descriptin Image Size Raw data input Raw data f camera in RGB frmat 1920 x 1080 Pixel, 24 Bit Pixel depth Clr space cnverter Transfrmatin RGB t YCbCr 1920 x 1080 Pixel, 24 Bit Pixel depth Chrma resampler Reduces clr saturatin (4:4:4 -> 4:2:2) 1920 x 1080 Pixel, 16 Bit Pixel depth ROI-generatr Cuts ut Regin Of Interest 1024 x 768 Pixel, 16 Bit Pixel depth Data preprcessing table [HSU & 6] Data reductin: Raw Data: 5.93 MB per image After cmpressin: 1.50 MB per image Factr 3.95 smaller Data Prcessing is much faster 14

15 Descriptin f the Signal Prcessing functins Infrmatin visualizatin - verlays ISO UN Regulatin N.46 Visualizatin Only transparent verlays allwed Any verlays (regardless f their transparency) are cnsidered as an bstructin Duratin Only temprary Only temprary Infrmatin Only driving related visual infrmatin Only rearward driving related visual infrmatin Size Size unlimited Maximal size f each verlay is 2.5% f the area which displays the minimum required field f visin. Overlays have t be included when calculating the bstructins within the required field f view (bstructins are limited t a maximum f 15% fr Class I mirrrs and t a maximum f 10% fr all ther classes) Cmparisn f the requirements fr verlays f ISO and UN Regulatin [1] 15

16 Descriptin f the Signal Prcessing functins Infrmatin visualizatin Class II Class I 16 Object detectin and tracking examples [1]

17 Descriptin f the Signal Prcessing functins Infrmatin visualizatin Each Overlay defines a new Layer Layer 0 is the riginal image Bypass image fr minimum delay All ther Layers are prcessing slutins Transparency defined Clr cnfigurable Picture f an example image fr the utput generated by the test plt generatr (TPG) [HSU] 17

18 Descriptin f the Signal Prcessing functins ZYNQ FPGA Resurce summary Currently implemented design uses: 7000 Flip Flps (7%) 6300 LUTs (12%) 181 Memry elements (1%) 10 DSPs (5%) The pwer cnsumptin is abut 2W (accrding t Xilinx Pwer estimatr) The current frame rate is 30 fps Implemented n Chip design, left the used resurces, right ruting [HSU] 18

19 Cnclusin HSU designs a develpment platfrm enabling investigatin f signal prcessing and architectures Embedded flexible platfrm based n Xilinx ZYNQ 7000 SC used fr a CMS research System design using Vivad, VHDL and IP Cres System parameter based n the nrmative framewrk f ISO 16505:2015 and UN Regulatin N. 46 Prcess separated int three parts Data preprcessing Data prcessing Infrmatin visualizatin First preprcessing peratins fr lw level tasks are already implemented 19

20 Bibligraphy 1. Handbk f Camera Mnitr Systems, Anestis Terzis eds., Springer Internatinal Publishing AG, Switzerland Rad vehicles Ergnmic and perfrmance aspects f Camera Mnitr Systems Requirements and test prcedures, Internatinal Standard, ISO 16505, May Addendum 45: Regulatin N. 46, United Natins, Design fr Embedded Image Prcessing n FPGAs, Dnald G. Bailey, Jhn Wiley & Sns, Singapre Schaltungs-Entwurf für ein Fahrerassistenzsystem, H.-M. Bühler, A. Maier, Studienarbeit, HS Ulm, Entwurf einer Bildverarbeitungs-Schaltung auf Basis eines SC-FPGA, M. Schmied, J. Schneider, Studienarbeit, HS Ulm, Entwurf und Bewertung eines Kamera-Mnitr-Systems als Fahrzeugspiegelersatz nach ISO 16505, M. Merlini, Bachelrarbeit, HS Ulm,

21 Image Signal Prcessing fr a Camera Mnitr System with an ZYNQ FPGA Thank yu fr yur attentin! M. Sc. Hchschule Ulm, Eberhard-Finckh-Str. 11, Ulm Jannik.maier@hs-ulm.de +49 (0731)