Eyedentify MMR SDK. Technical sheet. Version Eyedea Recognition, s.r.o.

Transcription

1 Eyedentify MMR SDK Technical sheet Version ADVANCED COMPUTER VISION SOLUTIONS

2 Copyright 2017, Eyedea Recognition s.r.o. All rights reserved Eyedea Recognition s.r.o. is not responsible for any damages or losses caused by incorrect or inaccurate results or unauthorized use of the software Eyedentify. Gemalto, the Gemalto logo, are trademarks and service marks of Gemalto and are registered in certain countries. Safenet, Sentinel, Sentinel Local License Manager and Sentinel Hardware Key are registered trademarks of Safenet, Inc. NVIDIA, CUDA are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and/or other countries. Microsoft Windows, Windows XP, Windows Vista, Windows 7, Windows 8, Windows 8.1 and Windows 10 are registered trademarks of Microsoft Corporation. Contact: Address: Vyšehradská 320/ , Prague 2 Czech Republic web: info@eyedea.cz

3 Table of Contents 2 Table of Contents 1 What s New Product Description Technical Details System Workflow Distribution Contents Input Requirements Input Image Vehicle Position Scale in Pixels per Meter Image Rotation Hardware Requirements Minimal Requirements Recommended Requirements Supported Operating Systems Binary Models Description Supported Outputs Performance... 15

4 What s New 3 1 What s New Eyedentify MMR Release date: 2017/11/13 Completely new binary models for vehicle classification with actual data for 2017/Q3. Binary models for vehicle classification now support view recognition feature. View type (frontal, rear) is now the one of the vehicle classification outputs. Separate binary models for frontal and rear view are not the part of the distribution anymore, one binary model for both views can be used. Eyedentify shared library now supports explicit linking only. Example codes for explicit linking are the part of the release package. Static linking was removed. Added new structure ERImage for image manipulation which internally uses OpenCV library. ERImage API is defined in the er_image.h and er_explink.h header files. EdfImage structure was removed. Header file edf_utils.h removed. Contained functions for EdfCropParams structure manipulation (edfcropparamsallocate, edfcropparamswrap and edfcropparamsfree) moved to the edf.h header. Remaining functions from edf_utils.h were removed from the public API or replaced by the ERImage API functions. EdfCropImageConfig structure updated with antialiasing settings. Structure EdfClassifyConfig for classification configuration added. Function edfclassify can be configured using the EdfClassifyConfig instance on the input. New version of licensing software Sentinel LDK 7.6 used. Example image data moved to the package root folder data. Java Wrapper built with the Java version 1.8.0_151. Eyedentify MMR Release date: 2017/02/08

5 Product Description 4 2 Product Description Eyedentify MMR SDK is a cross-platform software library developed to provide vehicle recognition functionality. It defines an interface between the user's software and our state of the art recognition solution. Eyedentify MMR SDK allows the user to recognize a vehicle located in digital image. The recognition output contains the information about the vehicle s view, category, make, model and vehicle s color. Vehicle view can be frontal or rear. Vehicles are divided into 6 categories bus, car, heavy truck, light truck, motorbike and van. The manufacturers of the vehicles are defined in the make output parameter e.g. Toyota, Volkswagen, etc. The model parameter then distinguishes the bodywork of vehicles created by specific manufacturer Avensis, Passat, etc. 2.1 Technical Details Eyedentify MMR SDK consists of two libraries base Eyedentify library and the recognition module. Both are cross-platform x86/x64 libraries with C interface. The base Eyedentify library is the only entry point, the user never uses the recognition module directly. The recognition module is loaded and configured using the Eyedentify library. The configuration parameters are loaded from one the binary models, which are contained in the distribution. The Eyedentify library provides following APIs: C native API Java JNI API C# API (Windows only) Officially supported operating systems and platforms: Windows 7, 8, 8.1 and 10 o 32 and 64 bit (Visual Studio 2015) Ubuntu and higher o 32 and 64 bit User s Code C/C++/Java C native API Java JNI API Eyedentify Library Recognition module Recognition module Recognition module

6 Product Description System Workflow The workflow of the MMR system consists of: image acquisition, vehicle license plate detection, input image cropping, vehicle descriptor computation and vehicle classification. The image acquisition and the vehicle license plate detection are not the part of this SDK and must be solved separately. The process starts with the input image cropping with respect to the license plate detection. The image crop is done using the SDK, it crops and transforms the image in the way that only the vehicle of interest is contained in the cropped image. The crop is the input of the machine learning algorithm, which is contained in the SDK s recognition module. The output is the descriptor real number vector describing the input vehicle in the condensed form. The descriptor is then classified, where the output is the classification result human readable output of the vehicle recognition. 1) Image acquisition Not the part of the SDK 2) License plate detection Not the part of the SDK 3) Image cropping Input image cropping around the vehicle of interest. 4) Descriptor computation Vehicle descriptor computation from the vehicle image crop ) Classification Vehicle descriptor classification and getting the MMR output. View: Category: Make: Model: Color: frontal CAR VW Golf WHITE

7 Input Requirements 6 3 Distribution Contents The following list is an excerpt from the Eyedentify MMR SDK directory structure, highlighting the most important directories and files contained in the software distribution. A brief description of the items is given. [Eyedentify MMR SDK]/ distribution main folder o documentation documentation folder 3rdparty-licenses. licenses to used 3 rd party libraries Eyedentify MMR SDK - Developer's Guide.pdf SDK Developer s Guide Eyedentify MMR SDK - Technical sheet.pdf. SDK Technical Sheet o o o o o o examples.. examples source files edf-explink. explicit linking support sources example-edf-explink explicit linking example example-vcl-api basic MMR API example hasp. software protection binaries sdk. Eyedentify MMR SDK mandatory data include. header files lib... Eyedentify library modules.. recognition modules folder edfdeep-vcl.. MMR recognition module o lib.. recognition module shared library o model. recognition module binary models wrappers... API wrappers java Java API wrapper csharp.. C# API wrapper LICENSE.txt... SDK license README.txt.. SDK Readme file

8 Input Requirements 7 4 Input Requirements The input of the MMR system is divided into two parts: image data and scene specification. The first part, which is in the Input Image section, specifies the image capture criteria and the way how to represent the image data. The second part, which is covered in the Vehicle Position, Scale in Pixels per Meter and Image Rotation sections, describes how to set the input parameters of the MMR system. 4.1 Input Image The results of the vehicle classification are dependent on two facts: the way how the image was taken and the way how the image is stored. Image capture requirements are specified in the Scene criteria section Scene criteria In order to get the highest possible recognition accuracy several rules must be respected during the vehicle input image data collection. The criteria specify camera scene setting in case of Vehicle alignment, Top view and Image borders paragraphs. The camera capture quality criteria are specified in Blurred image and Aspect ratio paragraphs. Vehicle alignment The front/back of the vehicle in the input image must be aligned in the way the license plate is horizontally aligned. The alignment could be done on the user s side or using the Eyedentify MMR SDK with specifying the rotation compensation parameter (see Image Rotation). WRONG: The front of the car is not horizontally aligned. CORRECT: Image with compensated in-plane rotation. The front of the car is now horizontally aligned.

9 Input Requirements 8 Top view The vehicle must be captured from top frontal/back view. WRONG: The view is frontal but not top. CORRECT: The view is top frontal. Image borders The vehicle should be sufficiently distant from image borders, should not be cropped or occluded. The surrounding red rectangle in the illustration should not go out of the image and should be fully visible to achieve the best recognition accuracy. WRONG: The car is too close to the image edge. CORRECT: The car is sufficiently distant from image edges.

10 Input Requirements 9 Blurred image The vehicle in the input image cannot be blurred by motion or by wrong camera settings. All details on the vehicle s bodywork must be clearly visible for successful recognition. WRONG: Wrong camera settings and fast motion causes that the vehicle is blurred. CORRECT: Image of the vehicle is very sharp and all the bodywork details are clearly visible. Resolution Resolution of the input images should be 20 pixels per meter at least, i.e. the minimal EU license plate's width should be approximately 10 pixels. For more information about the pixels per meter unit see the chapter Scale in Pixels per Meter. 10 pixels at least Aspect ratio Pixel aspect ratio should be 1:1. Other aspect ratios are not supported. WRONG: Pixel aspect ratio is not 1:1. CORRECT: Pixel aspect ratio is 1:1.

11 Input Requirements Vehicle Position Position of the vehicle of interest is defined using the 2D point in the image coordinate system. To get the vehicle position from the image automatically the license plate detector software must be used. The input of the MMR system defining the vehicle position is then specified as a 2D point representing the center of the license plate in the input image coordinate system. 4.3 Scale in Pixels per Meter The license plate detection output, defined as four 2D points representing the corners, is used to compute the license plate center. Regarding the fact that the position of the vehicle is specified by one 2D point only for correct vehicle localization the scale must be defined. The scale is defined in the pixels per meter how many pixels in the image represents one meter in the real world Description Scale in pixels per meter is the unit used in Eyedentify SDK to define the resolution of the input image with respect to the dimensions of the observed object in real world. We will call this unit using shortened name scale_px_per_m which also appears as the name of the variable in the SDK. Known scale_px_per_m is used during the input photo preparation where the image data are transformed with respect to the location of the car. We need the scale unit defined for the front/rear vertical plane of the vehicle (front/rear Eyedentify MMR SDK module used). The vertical plane is illustrated in the Illustration 1. Illustration 1: Horizontal plane on the vehicle's rear where the scale_px_per_m should be defined. We use the fact that the license plate dimensions are well defined and also Eyedentify SDK is usually connected with the detector which is searching for the license plate or one of its part in the input image, so we compute the scale_px_per_m with respect to the license plate dimensions. When the license plate or its part is detected in the input image we have the dimensions measured in the image from the detection in pixels. The physical dimensions of the license plate and its parts are defined by law in each country of origin but in many cases the same format is used. The formula below defines the relation between the size of the object (license plate or its part) in the image in pixels and in real world where the result is the scale_px_per_m Formula scale_px_per_m = size of the object in the image [px] size of the object in real world [m]

12 Input Requirements 11 There are several important facts that must be take into account to get correct results: The scale_px_per_m is varying across the image due to perspective projection. The engine allows error max. +/- 20% in scale_px_per_m. To achieve this a stable scale_px_per_m estimate (i.e. license plate/vehicle detector) is needed. Estimate of scale_px_per_m on small objects has higher error than on large ones (especially on low resolution images). It is therefore recommended to estimate the scale_px_per_m from license plate width rather than from license plate s letter height Use cases License plate width Scale is computed from the measured width of the license plate detection. scale = license plate detection width [px] physical width of the license plate [m] [px/m] License plate s letter height Scale is computed from the height of the detected license plate s letter. scale = height of the detected license plate s letter [px] physical height of the license plate s letter [m] [px/m] License plate s OCR Scale is computed from the width of the license plates OCR area. scale = license plates OCR area width [px] physical width of the license plate s letter area [m] [px/m] Vehicle s width Scale is computed from the width of the vehicle detection. scale = width of the vehicle detection [px] physical width of the vehicle [m] [px/m] Static camera setting Scale could be precomputed for the specific static camera setting. In this case the scale must be defined for each possible vehicle position in the camera image. If the camera is set to observe the road lane(s) we can generate the scales for the camera view using the fact that the vehicles are always placed on the plane representing the visible part of the road (green plane in the image). Illustration 2: Road lane with green highlighted plane where cars are placed.

13 Input Requirements Examples License plate width used to get the scale in pixels per meter: License plate s letter height used to get the scale in pixels per meter: License plate s OCR used to get the scale in pixels per meter: Vehicle s width used to get the scale in pixels per meter: 4.4 Image Rotation Vehicle image, before it could be processed by the MMR SDK, must be vertically aligned as described in the Vehicle alignment chapter. The SDK supports rotation parameter setting during input image preparation. The rotation parameter specifies the clockwise rotation of the input image in degrees. The center of rotation is the specified vehicle position in the image (see Vehicle Position) center of the license plate.

14 Hardware Requirements 13 5 Hardware Requirements 5.1 Minimal Requirements Processor: RAM: Hard disk: 1.0 GHz, single core, x86 platform, embedded (i.e. Intel Atom) 2 GB 1 GB free space 5.2 Recommended Requirements Processor: 2.0 GHz, dual core, x86 platform (i.e. Intel i5) RAM: Hard disk: 4 GB 2 GB free space 5.3 Supported Operating Systems Windows Linux Microsoft Windows 7/8/8.1/10 - Win32 and x64 platform Ubuntu and higher - i686 and x86_64 platform Windows is registered trademark of Microsoft Corporation. Linux is registered trademark of Linus Torvalds.

15 14 6 Binary Models 6.1 Description Currently, modules work in daylight only. At night, binary models work only when using artificial lighting of vehicles. View Category Make Model Color Fast Frontal/Rear/Both Vehicle category (BUS, CAR, HVT, LGT, VAN and MTB) Manufacturer of the vehicle (e.g. VW, Ford, Fiat ) Vehicle instance defined by a bodywork (e.g. Golf, Mondeo, 500 ) Color of the vehicle bodywork Fast version of the binary model Binary model name View Category Make Model Color Fast CNN_MMR_VC_BGR_PRECISE_2017Q3.dat Both Yes No No No No CNN_MMR_VC_BGR_FAST_2017Q3.dat Both Yes No No No Yes CNN_MMR_VCM_BGR_PRECISE_2017Q3.dat Both Yes Yes No No No CNN_MMR_VCM_BGR_FAST_2017Q3.dat Both Yes Yes No No Yes CNN_MMR_VCMM_BGR_PRECISE_2017Q3.dat Both Yes Yes Yes No No CNN_MMR_VCMM_BGR_FAST_2017Q3.dat Both Yes Yes Yes No Yes CNN_MMR_COLOR_FRONT_BGR_2016Q3.dat Frontal No No No Yes No Input resolution Input channels Input color space Resolution of the input image after cropping. Number of color channels of the input image after cropping. Color space of the input image after cropping. Binary model name Input resolution Input channels Input color space CNN_MMR_VC_BGR_PRECISE_2017Q3.dat BGR CNN_MMR_VC_BGR_FAST_2017Q3.dat BGR CNN_MMR_VCM_BGR_PRECISE_2017Q3.dat BGR CNN_MMR_VCM_BGR_FAST_2017Q3.dat BGR CNN_MMR_VCMM_BGR_PRECISE_2017Q3.dat BGR CNN_MMR_VCMM_BGR_FAST_2017Q3.dat BGR CNN_MMR_COLOR_FRONT_BGR_2016Q3.dat BGR

16 Supported Outputs View Category Make Model Color Number of supported view outputs (frontal, rear) Number of supported category outputs (car, bus, van, ) Number of supported make outputs (Toyota, VW, ) Number of supported model outputs (Avensis, Passat, ) Number of supported color outputs (black, blue, gray, ) Binary model name View Category Make Model Color CNN_MMR_VC_BGR_PRECISE_2017Q3.dat CNN_MMR_VC_BGR_FAST_2017Q3.dat CNN_MMR_VCM_BGR_PRECISE_2017Q3.dat CNN_MMR_VCM_BGR_FAST_2017Q3.dat CNN_MMR_VCMM_BGR_PRECISE_2017Q3.dat CNN_MMR_VCMM_BGR_FAST_2017Q3.dat CNN_MMR_COLOR_FRONT_BGR_2016Q3.dat Performance Timings and Sizes Processing Time: A single vehicle classification (descriptor computation only). Following hardware used: CPU: Intel i7-3930k 3.20GHz GPU: NVIDIA GeForce GTX Titan X ARM: RASPBERRY Pi 2 Model B (900MHz quad-core ARM Cortex-A7) Binary model name Size [MB] CPU processing time [ms] GPU processing time [ms] ARM processing time [ms] CNN_MMR_VC_BGR_PRECISE_2017Q3 221 CNN_MMR_VCM_BGR_PRECISE_2017Q CNN_MMR_VCMM_BGR_PRECISE_2017Q3 236 CNN_MMR_VC_BGR_FAST_2017Q3 23 CNN_MMR_VCM_BGR_FAST_2017Q CNN_MMR_VCMM_BGR_FAST_2017Q3 28 CNN_MMR_COLOR_FRONT_BGR_2016Q

17 Accuracy Overview Accuracy is reported for several testing sets. The accuracy is the number of correctly classified vehicles divided by the number of classified vehicles. The accuracy is measured only on vehicles where the vehicle was recognized by trained operators ( Evaluated column in dataset statistics table) Dataset 1 Country: Czech Republic Year: 2015 Table 1. Dataset statistics on Dataset 1. Class Total vehicles Unknown vehicles Unlabeled vehicles Evaluated Category 33, / 0.57% 0 / 0.00% 32,991/ 99.43% Make 33, / 0.82% 0 / 0.00% 32,908 / 99.18% Model 33, / 0.10% 3,742 / 11.28% 29,404 / 88.62% Table 2. Recognition accuracy on Dataset 1 Module name Category Make Model CNN_MMR_VC_BGR_PRECISE_2017Q % N/A N/A CNN_MMR_VCM_BGR_PRECISE_2017Q % 99.11% N/A CNN_MMR_VCMM_BGR_PRECISE_2017Q % 99.11% 98.42% CNN_MMR_VC_BGR_FAST_2017Q % N/A N/A CNN_MMR_VCM_BGR_FAST_2017Q % 98.64% N/A CNN_MMR_VCMM_BGR_FAST_2017Q % 98.64% 97.95% Dataset 2 Country: Netherlands Year: 2015 Notes: Images are from a video. Table 3. Dataset statistics on Dataset 2. Class Total vehicles Unknown vehicles Unlabeled vehicles Evaluated Category 5,640 0 / 0.00% 783 / 13.88% 4,857 / 85.12% Make 5, / 0.39% 29 / 0.51% 5,589 / 99.10% Model 5,640 0 / 0.00% 847 / 15.02% 4,793 / 84.98% Table 4. Recognition accuracy on Dataset 2. Module name Category Make Model CNN_MMR_VC_BGR_PRECISE_2017Q % N/A N/A CNN_MMR_VCM_BGR_PRECISE_2017Q % 97.40% N/A CNN_MMR_VCMM_BGR_PRECISE_2017Q % 97.40% 93.94% CNN_MMR_VC_BGR_FAST_2017Q % N/A N/A CNN_MMR_VCM_BGR_FAST_2017Q % 96.04% N/A CNN_MMR_VCMM_BGR_FAST_2017Q % 96.04% 92.13%

18 Dataset 3 Country: Italy Year: 2015 Table 5. Dataset statistics on Dataset 3. Class Total vehicles Unknown vehicles Unlabeled vehicles Evaluated Category 2, / 1.55% 0 / 0.00% 2,158 / 98.45% Make 2, / 5.36% 0 / 0.00% 2,207 / 94.64% Model 2,192 0 / 0.00% 2,192 / % 0 / 0.00% Table 6. Recognition accuracy on Dataset 3. Module name Category Make Model CNN_MMR_VC_BGR_PRECISE_2017Q % N/A N/A CNN_MMR_VCM_BGR_PRECISE_2017Q % 98.58% N/A CNN_MMR_VCMM_BGR_PRECISE_2017Q % 98.58% N/A CNN_MMR_VC_BGR_FAST_2017Q % N/A N/A CNN_MMR_VCM_BGR_FAST_2017Q % 98.00% N/A CNN_MMR_VCMM_BGR_FAST_2017Q % 98.00% N/A Dataset 4 Country: Netherlands Year: 2014 Table 7. Dataset statistics on Dataset 4. Class Total vehicles Unknown vehicles Unlabeled vehicles Evaluated Category 11,856 0 / 0.00% 2,792 / 23.55% 9,064 / 76.45% Make 11, / 0.17% 2,275 / 19.19% 9,561 / 80.64% Model 11,856 0 / 0.00% 2,833 / 23.90% 9,023 / 76.10% Color 11, / 34.75% 0 / 0.00% 7736 / 65.25% Table 8. Recognition accuracy on Dataset 4. Module name Category Make Model Color CNN_MMR_VC_BGR_PRECISE_2017Q % N/A N/A N/A CNN_MMR_VCM_BGR_PRECISE_2017Q % 99.23% N/A N/A CNN_MMR_VCMM_BGR_PRECISE_2017Q % 99.23% 97.50% N/A CNN_MMR_VC_BGR_FAST_2017Q % N/A N/A N/A CNN_MMR_VCM_BGR_FAST_2017Q % 98.77% N/A N/A CNN_MMR_VCMM_BGR_FAST_2017Q % 98.77% 96.94% N/A CNN_MMR_COLOR_FRONT_BGR_2016Q3 N/A N/A N/A 87.90%

19 Dataset 5 Country: Poland Year: 2014 Table 9. Dataset statistics on Dataset 5. Class Total vehicles Unknown vehicles Unlabeled vehicles Evaluated Category 4, / 0.87% 1,013 / 23.24% 3,307 / 75.88% Make 4, / 4.96% 1,147 / 26.32% 2,995 / 68.72% Model 4,358 0 / 0.00% 4,358 / % 0 / 0.00% Table 10. Recognition accuracy on Dataset 5. Module name Category Make Model CNN_MMR_VC_BGR_PRECISE_2017Q % N/A N/A CNN_MMR_VCM_BGR_PRECISE_2017Q % 98.88% N/A CNN_MMR_VCMM_BGR_PRECISE_2017Q % 98.88% N/A CNN_MMR_VC_BGR_FAST_2017Q % N/A N/A CNN_MMR_VCM_BGR_FAST_2017Q % 98.25% N/A CNN_MMR_VCMM_BGR_FAST_2017Q % 98.25% N/A Dataset 6 Country: Italy Year: 2016 Table 11. Dataset statistics on Dataset 6. Class Total vehicles Unknown vehicles Unlabeled vehicles Evaluated Category 6,576 2 / 0.03% 106 / 1.61% 6,468 / 98.36% Make 6, / 13.81% 24 / 0.36% 5,644 / 85.83% Table 12. Recognition accuracy on Dataset 6. Module name Category Make Model CNN_MMR_VC_BGR_PRECISE_2017Q % N/A N/A CNN_MMR_VCM_BGR_PRECISE_2017Q % 92.65% N/A CNN_MMR_VCMM_BGR_PRECISE_2017Q % 92.65% N/A CNN_MMR_VC_BGR_FAST_2017Q % N/A N/A CNN_MMR_VCM_BGR_FAST_2017Q % 94.58% N/A CNN_MMR_VCMM_BGR_FAST_2017Q % 94.58% N/A