Hochperformante Inline-3D-Messung mittels Lichtfeld Dipl.-Ing. Dorothea Heiss Deputy Head of Business Unit High Performance Image Processing Digital Safety & Security Department AIT Austrian Institute of Technology
Bundesministerium für Verkehr, Innovation und Technologie 50,46% Industriellenvereinigung 49,54% 1300+ employees Budget: 120 Mio Future Networks and Services Intelligent Vision Systems Highly Reliable SW and Systems > 100 experts 2
Intelligent Vision Systems (IVS) for Process Automation & Inspection (Industry 4.0) Autonomous & Assistive Systems Surveillance & Protection Worldwide fastest vision sensor technology Robust and flexible 3D vision technology 3
Intelligent Vision Systems (IVS) for Process Automation & Inspection (Industry 4.0) Autonomous & Assistive Systems Surveillance & Protection 4
Process Automation & Inspection Example: Banknote Inspection World technology leader 100% inline inspection of newly printed banknotes KEY FACTS Speed 11 m/s (44 banknotes/s) Resolution 0,1 mm/pixel Color Multispectral inspection (color, infrared & UV) Multiview International 100% inspection of front-, backside and transmission Worldwide standard for banknote printing shops CURRENT RESEARCH FIELDS True color Hyperspectral imaging for true color inspection New security features Inspection of holograms and optical variable features 3D information for tactile elements Increasing optical resolution for micro text inspection 5
Projects and Solutions Electronics Infrastructure Print & Paper Metal Process Automation & Inspection world leading linescan PCB street surface & 3D PCB assembly failure classification wafer rail surface all in focus coin texture & 3D coin identification micro pores tube cracks banknotes packaging flex inspect tactile features OVI quality inspection 3D inspection classification photometric stereo light field high speed stereo line scan deep learning computational imaging 6
Computational Imaging Several images of an object under different conditions Redundancy and computation are used to enhance image quality (depth of field, resolution, contrast, sharpness, dynamic range, signal to noise ratio, ) and/or generate useful information, especially depth / 3D Examples: in-camera computation of digital panoramas, high-dynamicrange images, light field cameras, photometric stereo, Computing power is getting faster and cheaper far quicker than precise optical devices, stronger illumination etc. (Moore s law) 7
Photometric Stereo Object is acquired under different illumination angles Applications Surface inspection 3D reconstruction (surface normal, depth map) Material classification (Bidirectional reflectance distribution function) 8
Surface normals Depth maps Example: Coins (Photometric Stereo) 9
10
Photometric Stereo Advantages Sensitive to very fine details Information about glossiness / material classification Also suited for glossy surfaces No surface texture required Disadvantages Metrically not correct Globally not correct No measurement of steep edges 11
Light Field Object is acquired under different observer / camera angles Describes reflected light in each point of space (x, y) in each direction (u, v) State of the art CMOS chip with micro-lens array (e.g. Lytro) Multi-camera array Applications Computational refocusing / extended depth-of-field Changing viewpoint 3D reconstruction 12
Advantages of Light Field compared to Stereo Robustness: several acquisitions contain more information than two Less sensitive to specular reflections Less occlusion because of smaller stereo baseline Flexible number of views and viewing angles Additional methods for depth estimation possible, e.g. Multi-view block matching Depth from refocus Structure tensor in EPI domain Not only depth estimation, but also All-in-focus, Noise reduction, Deblurring, Super resolution, High-dynamic range, 13
Example: Coins (Light Field) 14
15
Light Field Advantages Metrically correct Globally correct Measurement of steep edges All in focus / enhanced depth of field Disadvantages Not sensitive to fine details Needs surface structure Not suited for glossy surfaces Depth map is incomplete 16
Light Field Photometric Stereo Advantages Metrically correct Globally correct Measurement of steep edges All in focus Disadvantages Not sensitive to fine details Needs surface structure Not suited for glossy surfaces Depth map is incomplete Advantages Sensitive to very fine details Information about glossiness / material classification Also suited for glossy surfaces No surface texture required Disadvantages Metrically not correct Globally not correct No measurement of steep edges Combination would be ideal 17
Requirements of Industrial Inspection Combination of Light Field and Photometric Stereo Good price/performance ratio Small installation space High speed Reproducibility Rough environment (dirt, heat) Robust against flutter, vibrations Minimal down-time Easy configuration Dome plus multi-camera array Costly Bulky Many components Area scan - not inline Many acquisitions Computationally expensive complicated Real-time fusion not solved Combination not ideal for industrial inline applications? 18
AIT Multi-line-scan Light-field Camera Camera 1 Line 1 Camera Line 2 2 Line 3 Camera 3 Light Field from Motion Fast CMOS area scan chip Multi-line scan acquisition 3D light field from motion Transport direction Pros Single sensor / single lens Flexible number of views Simple adjustment Easy data transfer Compact form factor Lower costs Inline Fast: 10.000 Hz 19
Light Field Acquisition Line Scan 20
Epipolar Planes Light field method: the more an epiolar plane is slanted, the higher the disparity, the nearer the object point to camera, the higher the object point 21
AIT Multi-line-scan Light-field Camera - Revisited Different observer angle light field AND different illumination angle photometric stereo in each object point (at a given point of time) Fusion is inherent in setup Only 1 camera, 1 optics, 1 illumination Angles interdepend! None of the known methods applicable 22
Epipolar Planes Overexposure Is it a bug or a feature New photometric method: finding line (=view) with the intensity peak Light field method: determining depth from slant New fused method: finding view with the intensity peak along the epipolar line Patents pending 23
Example: Euro Depth maps (derived from slant) light field alone Surface normals (derived from peak) photometric stereo alone Surface normals (derived from peak, but considering the slant) photometric stereo with light field 24
Example: Canadian Silver Dollar Depth maps (derived from slant) light field alone Surface normals (derived from peak) photometric stereo alone Surface normals (derived from peak, but considering the slant) photometric stereo with light field 25
Comparison: Photometric Stereo (Inline) and Light Field with Photometric Stereo (Inline) 26
Comparison: Photometric Stereo (Dome) and Light Field with Photometric Stereo (Inline) 27
xposure fastest line scan camera computational imaging High-Performance Vision Fusion of technologies and expertise deep learning High-Performance Vision Optimized Solutions for Real-time Embedded Vision Systems embedded parallel hardware 28
xposure - Worldwide Fastest Line Scan Sensor 600.000 lines per second monochrome, or 200.000 lines per second RGB color, or 50.000 light field acquisitions per second Digital output 1,2 Giga-Pixel / second (10 bit) Flexibility because of random access to 60 pixel lines Developed with Fraunhofer IMS 29
xposure - Inspiring Speed and Resolution Example m/s km/h Resolution @ 600kHz mm Earth observation satellite 7200 25920 12,0 Moon around earth / bullet 900 3240 1,5 Aircraft 300 1080 0,5 ICE train 90 324 0,15 Print inspection 10 36 0,017 Metal surface inspection 1,8 6,5 0,003 enabling a new dimension of high speed image processing applications 30
xposure Camera Launch at Vision fair 2016 in Stuttgart Focused on key requirements for quality inspection High speed High resolution High signal to noise ratio Small: 8x8x8 cm FPGA for control and preprocessing Sensor manufactured by Fraunhofer IMS Automotive certified Long term availability Made in Germany 31
AIT Multi-line Scan Light Field and Photometric Stereo Setup Advantages High speed acquisition and and fast algorithms (real-time) Small, few components, good price / performance ratio Inline (line-scan approach) Fine surface details with metrically and globally correct depth map Results robust to material properties and lighting conditions All in focus, enhanced image quality Flexible: accuracy vs. speed can be chosen dynamically Technology protected by several patents 32
AIT Austrian Institute of Technology your ingenious partner Dipl.-Ing. DOROTHEA HEISS Deputy Head of Business Unit High Performance Image Processing Digital Safety & Security Department AIT Austrian Institute of Technology GmbH 2444 Seibersdorf Austria T+43(0) 664 8251177 dorothea.heiss@ait.ac.at