Software ISP Application Note

Size: px

Start display at page:

Download "Software ISP Application Note"

Eileen Gordon
6 years ago
Views:

1 NXP Semiconductors Document Number: AN12060 Application Notes Rev. 0, 10/2017 Software ISP Application Note 1. Introduction This document describes the software-based image signal processing application(sw-isp) and its implementation on the i.mx8 family device System-on- Chip [1] (SoC) processors. This application is used for image processing in i.mx8 family device. This pipelined image processing engine is optimized by onchip GPU. It includes functions of Bad Pixel Correction, White Balance, Histogram Equalization, High-Quality Demosaicing, High-Quality Noise Reduction. Its purpose is to use on-chip GPU to implement ISP function with a high speed processing. Contents 1. Introduction Software theory SW-ISP Pipeline Optimized by OpenCL SW-ISP Pipeline Optimized by OpenVX Performance measurement SW-ISP Pipeline Optimized by OpenCL/OpenVX Conclusion References System on Chip, Yocto Project, Bayer Image, YUV/RGB Convert, available at en.wikipedia.org Revision history NXP B.V.

Bad Pixel Correction White Balance Histogram Equalization High-quality Demosaicing High-Quality Noise Reduction (Optional) Software theory Raw Image RGBA Image SW-ISP Pipeline GPU Hardware 2.

The SD card image is created by using the Yocto Project [1]. SW-ISP pipeline optimized by OpenCL1.2 [2]. SW-ISP pipeline optimized by OpenVX1.1 [2].

2 Bad Pixel Correction White Balance Histogram Equalization High-quality Demosaicing High-Quality Noise Reduction (Optional) Software theory Raw Image RGBA Image SW-ISP Pipeline GPU Hardware 2. Software theory Figure 1. SW-ISP pipeline block diagram From the software point of view, the full SW-ISP application consists of these parts: Linux OS. The SD card image is created by using the Yocto Project [1]. SW-ISP pipeline optimized by OpenCL1.2 [2]. SW-ISP pipeline optimized by OpenVX1.1 [2]. SW-ISP pipeline includes five functions of image processing. Bad Pixel Correction algorithm [3], White Balance algorithm [4], Histogram Equalization algorithm [5] and High-Quality Demosaicing algorithm [6] are applied into Bayer image [1], while High-Quality Noise Reduction algorithm [7] is applied into Y- channel alone. The High-Quality Noise Reduction process consists of following steps: 1) The RGBA image is converted to YUV image [1]. 2) Apply bilateral filtering [7] to Y-Channel to reduce noise of image. 3) Filtered Y-Channel image is merged into UV-Channels to covert to RGBA image. In both OpenCL and OpenVX pipelines, input raw images captured by Bayer camera (Used color filter array over Bayer camera is shown in Figure 2.) are read from file. The raw values would be uploaded to a GPU as a grayscale texture. Every function in SW-ISP pipeline is independent, so they can be combined willfully. These functions are combined in various ways to show processed images of each function. The combination can be controlled by keys. For example, if press 1 when the application is running, the pipeline only implement 2 NXP Semiconductors

3 Software theory function of High-quality Demosiacing. The application block diagram is shown in Figure 3. Figure 2. Bayer color filter array NOTE The color filter array is blue-green-green-red (BG/GR). It is 50% green, 25% blue, and 25% red. Read Bayer Image Keys Control Press 1 Press 2 Default or press 3 Press 4 Press 5 High-quality Demosaicing Bad Pixel Correction Bad Pixel Correction Bad Pixel Correction Bad Pixel Correction White Balance Histogram Equalization White Balance High-quality Demosaicing High-quality Demosaicing High-quality Noise Reduction Display(RGBA) Software ISP Pipeline of OpenCL/OpenVX Figure 3. Application block diagram NXP Semiconductors 3

Software theory Figure 4 shows effects of each algorithm: Raw Bayer Image Demosiacing of Raw Image Raw Image Demosiacing Bad Pixel Correction White Balance Histogram Equalization Noise

To show results on screen directly, format of output image is RGBA and the width of the output image is 19

4 Software theory Figure 4 shows effects of each algorithm: Raw Bayer Image Demosiacing of Raw Image Raw Image Demosiacing Bad Pixel Correction White Balance Histogram Equalization Noise Reduction Figure 4. Comparison of algorithms applied to an image. NOTE Because the currently released software ISP works in the demo mode only, the input array of Bayer image is BG/GR mode. To show results on screen directly, format of output image is RGBA and the width of the output image is 1936 which is same with that of Frame Buffer SW-ISP Pipeline Optimized by OpenCL In OpenCL pipeline, combine different functions of SW-ISP by setting kernel arguments. The detailed directory structure of the SW-ISP of OpenCL application is shown in Figure 5: 4 NXP Semiconductors

5 Software theory Figure 5. Directory structure of the SW-ISP of OpenCL application The root directory is called SW-ISP-OpenCL. There are three subdirectories in the root directory: Build - contains executable file generated by the SW-ISP and the binary file called ISP_OCL_FB_Opt. Content - contains the prepared raw images ( garden04_bayerdata_300f.raw ) in the *.raw format and input txt file( 1920x1080.txt ). These raw images were captured by Bayer camera (resolution: 1920 x 1080, Bayer array: BG/GR). Name of raw images is put in 1920x1080.txt. The application can read raw data by 1920x1080.txt. Source - contains source code running on CPU ( isppipeline_ocl.cpp ) and GPU ( kernel.cl/ kernel.cl.bin ). kernel.cl is C code of kernels. kernel.cl.bin is compiled binary file of kernels. The application is compiled on the target, so all files of SW-ISP-OpenCL should be copied to i.mx8 board. Compile and run the application by the following commands: 1) Change directory to Source. $: cd SW-ISP-OpenCL/Source/ 2) Compile the source codes, and executable file will be generated in the Build. $: make 3) Change directory to Build. $: cd../build/ 4) Run the program (disable High-quality Noise Reduction). $:./ISP_OCL_FB_Opt -f../content/1920x1080.txt 5) Run the program (enable High-quality Noise Reduction). $:./ISP_OCL_FB_Opt -f../content/1920x1080.txt -d 2.2. SW-ISP Pipeline Optimized by OpenVX In OpenVX pipeline, combination of different functions are controlled by graphs in OpenVX. The detailed directory structure of the SW-ISP of OpenVX application is shown in Figure 6: NXP Semiconductors 5

6 Software theory Figure 6. Directory structure of the SW-ISP of OpenVX application The root directory is called SW-ISP-OpenVX. There are three subdirectories in the root directory: Build - contains header file used in compiling of OpenVX kernels. Executable binary file called ISP_OVX_Opt will be put here after compiling the project. Content - contains the prepared raw images ( garden04_bayerdata_300f.raw ) in the *.raw format and input txt file( 1920x1080.txt ). These raw images were captured by Bayer camera (resolution: 1920 x 1080, Bayer array: BG/GR). Name of raw images is put in 1920x1080.txt. The application can read raw data by 1920x1080.txt. Source - contains source code running on CPU ( pipeline_kernel.cpp, pipeline_kernel.hpp main.cpp and define.hpp ) and GPU ( kernels_vxc.hpp ). kernels_vxc.hpp is code of kernels. The application is compiled on the target, so all files of SW-ISP-OpenVX should be copied to i.mx8 board. Compile and run the application by the following commands: 1) Change directory to Source. $: cd SW-ISP-OpenVX/Source/ 2) Compile the source codes, and executable file will be generated in the Build. $: make 3) Change directory to Build. $: cd../build/ 4) Run the program (disable High-quality Noise Reduction). $: /ISP_OVX_Opt -f../content/1920x1080.txt 5) Run the program (enable High-quality Noise Reduction). $: /ISP_OVX_Opt -f../content/1920x1080.txt -d 6 NXP Semiconductors

7 Conclusion 3. Performance measurement This section describes the profiling of the SW-ISP (OpenVX, OpenCL) application. The main profiling information is frame rate of SW-ISP. Running time of kernels is also statistic. Table 1. Common SW-ISP profiling conditions Feature Description Input method Image (1920 x 1080, Bayer) Graphical backend fb Platform OpenCL,OpenVX HW i.mx8dv BSP GPU driver OpenVX 1.1 OpenCL 1.2FP Used HDMI resolution 1920 x 1080 (1080p) Doc Software ISP Application Release date Sept 23th 3.1. SW-ISP Pipeline Optimized by OpenCL/OpenVX Table 2 shows the statistic results of different functions. The default pipeline includes bad pixel correction node, white balance node and high-quality demosiacing node. Frame rate of the default OpenCL pipeline is ~28fps, and that of the default OpenVX pipeline is ~39fps. Table 2. Comparison of SW-ISP OpenCL and OpenVX on i.mx8dv SW-ISP Functions Consumed time of OpenCL (ms) Consumed time of OpenVX(ms) Bad Pixel Correction Node 12 4 White Balance Node 8 8 Histogram Equalization Node 6 7 Demosiacing Node 11 5 RGBA2YUV Node 7 4 High-quality Noise Reduction Node YUV2RGBA Node 12 4 Pipeline(default) Conclusion This application represents a software ISP solution. This solution is optimized by both OpenCL and OpenVX, running on GPU. The SW-ISP application has these main advantages: High application performance (OpenCL: ~28 fps, 1080p, OpenVX: ~39fps, 1080p). Algorithm of ISP optimized deeply by OpenCL/OpenVX. Support different combination of functions. SW-ISP optimized by OpenCL support most OpenCL platform. NXP Semiconductors 7

8 Revision history 5. References 1. System on Chip, Yocto Project, Bayer Image, YUV/RGB Convert, available at en.wikipedia.org. 2. OpenCL 1.2 and OpenVX 1.1 Reference Pages, available at 3. Bad pixel detection and correction algorithm is referred to 4. White balance algorithm is referred to 5. Histogram Equalization algorithm is referred to 6. High-quality demosaicing algorithm is referred to 7. High-quality noise reduction algorithm is referred to 6. Revision history Table 3. Revision history Revision number Date Substantive changes 0 10/2017 Initial release 8 NXP Semiconductors

10 How to Reach Us: Home Page: nxp.com Web Support: nxp.com/support Information in this document is provided solely to enable system and software implementers to use NXP products. There are no express or implied copyright licenses granted hereunder to design or fabricate any integrated circuits based on the information in this document. NXP reserves the right to make changes without further notice to any products herein. NXP makes no warranty, representation, or guarantee regarding the suitability of its products for any particular purpose, nor does NXP assume any liability arising out of the application or use of any product or circuit, and specifically disclaims any and all liability, including without limitation consequential or incidental damages. Typical parameters that may be provided in NXP data sheets and/or specifications can and do vary in different applications, and actual performance may vary over time. All operating parameters, including typicals, must be validated for each customer application by customer s technical experts. NXP does not convey any license under its patent rights nor the rights of others. NXP sells products pursuant to standard terms and conditions of sale, which can be found at the following address: nxp.com/salestermsandconditions. NXP, the NXP logo, NXP SECURE CONNECTIONS FOR A SMARTER WORLD, Freescale, and the Freescale logo are trademarks of NXP B.V. Windows is a registered trademark of Microsoft Corporation in the United States and/or other countries. Yocto Project is a registered trademark of The Linux Foundation. Linux is a registered trademark of Linus Torvalds in the U.S. and other countries. Arm, the Arm logo, and Cortex are registered trademarks of Arm Limited (or its subsidiaries) in the EU and/or elsewhere. All other product or service names are the property of their respective owners. All rights reserved NXP B.V. Document Number: AN12060 Rev. 0 10/2017

QWKS Ethernet Accessory Card, User's Guide

NXP Semiconductors Document Number: QWKSEACSG User's Guide Rev 0, April, 2017 QWKS Ethernet Accessory Card, User's Guide Contents Contents Chapter 1 Introduction...3 Chapter 2 QWKS Ethernet Accessory Card