2nd International Worksop on Materials Engineering and Computer Sciences (IWMECS 2015) Image Reconstruction Based On Bayer And Implementation On FPGA Sun Cen 1, a, Duan Xiaofeng 2, b and Wu Qijing 3, c 1 Xi'an Institute of Optics and Precision Mecanics, CAS,, Xian 710119, Cina; 2 Xi'an Institute of Optics and Precision Mecanics, CAS,, Xian 710119, Cina; 3 Xi'an Institute of Optics and Precision Mecanics, CAS,, Xian 710119, Cina. a suncen2009@126.com, b duanxiaofeng@opt.ac.cn, c wuqijing@126.com, Keywords: bayer image, gradient, color reconstruction, FPGA. Abstract. Tis paper mainly introduces color images interpolation algoritm wic based on bayer template. In order to estimate te missing two components in one pixel accurately, tis algoritm selects gradient correlation between pixels to calculate tem. Te bayer image, wose resolution ratio is 5120*3840, captured by CMOS, is output by DVI interface. Experimental results sows tat tis algoritm bot ensure te quality of reconstruction images and te real-time processing on FPGA. Introduction Wit te deelopment of information tecnology, quality of people s life as improed. More and more people use digital cameras replace film cameras. Te sensor of digital cameras is a ligt-sensitie Carge-Coupled Deice or Complementary Metal Oxide Semiconductor. Te image wic is captured by sensor is a gray scale image; digital cameras need a series of image processing to enance image quality. For example, te color image reconstruction based on bayer template, wite balance, automatic integration algoritm and so on. Te one important image processing is color image reconstruction based on bayer template, it is known as demosaicing. In order to capture a full-color image, teoretically require tree potosensitie arrays in digital cameras, eac of tese arrays to capture one of red, green and blue colors. Tis way of capturing a color image requires a large storage space, ig cost and ig difficulty deice deployment. Most digital cameras put a color filter array CFA on single potosensitie array to aciee primary separation, so tat te adjacent potosensitie array can capture different colors of ligt (red, green or blue). Eac pixel lost te oter two color components needs an image interpolation algoritm to rebuild full-color image. tis algoritm as been implemented on Xilinx FPGA spartan6 platform. Wole system can be diided into tree parts, te first part is to write data into SDRAM; te second part is to read data out from SDRAM and calculate tese data by interpolation algoritm and restored in SDRAM; te tird part is to read data out from SDRAM, output troug DVI interface and bus. R 1.1 G 1.2 R 1.3 G 1.4 R 1.5 G 2.1 B 2.2 G 2.3 B 2.4 G 2.5 R 3.1 G 3.2 R 3.3 G 3.4 R 3.5 G 4.1 B 4.2 G 4.3 B 4.4 G 4.5 R 5.1 G 5.2 R 5.3 G 5.4 R 5.5 Fig. 1 Bayer template Common demosaicing metods Bilinear interpolation. Bilinear interpolation algoritm is one of te widely used interpolation algoritm. Wen we calculate R and B at G alues in Figure 1 based on tis algoritm, We can estimate G and B at R alues and R and G at B alues by same metod. Bilinear algoritm as low complexity. It is easy to implement on FPGA and works well in smoot area. But at te edge area, pixel alues as direction, we cannot easily calculate pixel alues by te aerage of surrounding 2015. Te autors - Publised by Atlantis Press 849
pixels. So, tis algoritm will ae zigzag on te edge of te area, it is called zipper effect. Constant color difference. Tere are a lot of diisions in te formula of Aberration interpolation, if a pixel alue is 0, tis pixel will not be calculated by te aboe formula. Tis is a limitation of Aberration interpolation. In order to oercome tis limitation of Aberration interpolation, tere is a proposed algoritm based on Mondrian model, it is called Aberration interpolation, it use subtraction instead of diision. Gradient direction interpolation algoritms. Gradient information is an important factor affecting isuals; it is not only can transfer most of image information, but can also sketc out te basic outline of objects in an image. We can get useful structure information from edge detection so tat we can reduce te processed data and improe efficiency. Gradient direction interpolation algoritm is an adaptie interpolation algoritm; eac pixel can get a more appropriate interpolation direction. Tis interpolation as been widely used, you can cange te size of region to get a better interpolation results. 3*3 regions: Eery pixel as tree components, te relationsip between tree components are ery close in te researc of algoritm. Tis algoritm is first-order edge detection proposed by Hibbard. 5*5 regions: Second order differential interpolation algoritm is proposed by Laroce edge detected algoritm, in tis algoritm, edge detection is a linear sift inariant second order differential operator, it is called Laplace operator. For calculating interpolation direction at G based on Laroce edge detected algoritm, we remain comparing orizontal direction and ertical direction. Te difference is tat we calculate a pixel s gradient interpolation direction by itself instead of G gradient interpolation direction. Hig quality linear interpolation algoritm. Seeral interpolation algoritms described aboe do not take into account te correlation between te arious color components. Te experiments sow tat ariation of different color components are ery slowly in local region. If we add information of te oter two components, we will estimate te oter two components accurately. Luminance information is muc more important tan crominance information, so we will not discard red alues at R 3.3 wen we calculate G alues at R 3.3, R alues is useful information. Malar proposed a new type of ig quality interpolation algoritm on 2004. Te wole idea of tis algoritm is aerage and correction. Tis algoritm cooses a 5*5 filter for calculating and makes full use of correlation between tree cannels. Te filters are sown as following: Fig.2 Filters Tese linear filters make full use of correlation between tree cannels, tey come up wit appropriate alues of {α, β, γ}, is α = 1/2, β = 5/8, γ = 3/4, te quality of tis algoritm is muc iger tan bilinear. Waelet transform Interpolation algoritm. Te purpose of waelet transform is to distinguis ig frequency signal from low frequency signal. Te signal as been conerted into waelet domain by Scaling and translation transformation. For many problems wic cannot be soled by Fourier transform, Waelet transform works ery well and ery effectie. S.mallat reference Burt and Adelson pyramid image decomposition and reconstruction algoritms, lead te field of computer ision into waelet domain, put forward a fast algoritm for waelet transform called mallat. 850
Interpolation algoritm based on waelet transform as a good effect in color image reconstruction, but te ig complexity and a large amount of computation limit te algoritm application. It cannot be complement on FPGA. Proposed linear interpolation algoritm. Bilinear is easy to lead zipper effect in ig frequency region, color constant will ae a good interpolation results in smoot areas, but many error alues in sudden canges region. Gradient interpolation algoritm works well in ig frequency region, subtle canges will be processed as image edge, so tis algoritm keeps edge information as well as brings in random noises. Te color caracteristic is ery important information in image and te ig frequency region is also an important part of an image. Terefore, te design of proposed algoritm is based on gradient information. First order and second order edge detection algoritm is easier to implement on FPGA. In order to find a more appropriate real-time processing and ig quality image reconstruction, I put forward a proposed algoritm based on gradient algoritm. Tis algoritm ensures quality of image reconstruction; aciee real-time and low resource consumption. For te missing G at R 3.3 : a. We define ertical direction T, orizontal direction T, 45 direction T and d 135 T d direction, formulas are sown as following: T 2* R -R -R (1) 3.3 3.1 3.5 T 2* R -R -R (2) 3.3 1.3 5.3 T 2* R -R -R (3) d 3.3 1.5 5.1 T 2* R -R -R (4) d 3.3 1.1 5.5 b. Comparing T and T to determine interpolation direction, formulas are sown as following: ( G T T 3.2 g3.3 G3.4 )/ 2 * T ( G T T 2.3 g3.3 G4.3 )/ 2 * T (5) (6) g3.3 ( G G G 3.2 2.3 4.3 G )/ 4 *( T T ) T T (7) 3.4 c. Comparing T d and T d to determine interpolation direction, formulas are sown as following: ( B B T T 2.2 4.4 d d d 3.3 ( )/ 2 * 2.4 4.2 d T d d 3.3 ( B B 2.2 4.4 4.2 d d b3.3 )/ 2 * b B B T T (8) T (9) b B2.4 B )/ 4 *( T T )/ 2 T T (10) d d Implement on FPGA: Te wole system consists of tree parts: first, sensor captures Bayer image; second, demosaicing; tird, output module. Te wole system is sown as following: Fig 3. System 851
Video_buffer module: Tis module mainly consists of data storage module and storage control module. Te data captured by sensor are wrote in SDRAM, ten, FPGA write tese data in Video_buffer wic is SDRAM, and ten read tese data in Ddr3_ain order,ddr3_a consists of four blocks, control module transferred eigt pixels eery time in parallel to Imp_ci wic is demosaic module. Demosaic module: Demosaic module deide into two parts: interpolation processing and control module. Control module load data into interpolation module by address signal. Eac pixel in bayer image captured by sensor is 10bits, in order to meet te requirements of clock syncronization; we conert eac pixel into 16bits. Tis algoritm includes simple addition or subtraction and comparison operation. Tere are 5120*3840 pixels in one frame image. DVI output module: Tis module includes buffer part and control part, too. Control module conert te processed data into image standard data, and ten sae tese image data into Ddr3_b. before getting out image data, control module transferred tese data into data_out_buffer and ten displayed by DVI interface. Te frame can reac 25 frame per second. Performance: I coose different algoritms to process same image, te results sows as following: Details: Fig 4.Lena Fig 5. From left to rigt; original,,5*5regions gradient, ig-quality linear, proposed metod. Details: Fig 6. CCD prototype Fig 7. From left to rigt; original, 5*5regions gradient, ig-quality linear, proposed metod. 852
Table 1 PSNR, for te 4 images, A)Bilinear; B)5*5 regions gradient; C)ig-quality linear; D) proposed metod Numble c A B C D R 25.25 28.55 29.12 29.81 1 G 26.55 26.61 27.15 27.65 B 25.21 25.54 25.84 27.52 R 22.50 25.31 25.51 26.22 2 G 27.32 24.22 25.12 26.35 B 23.01 23.18 23.25 24.20 R 31.84 31.78 31.97 32.87 3 G 33.55 32.55 33.68 34.51 B 30.27 32.41 32.86 34.63 R 32.20 32.12 33.52 34.28 4 G 33.16 33.54 33.42 34.59 B 34.61 36.21 36.35 37.21 R 28.02 29.44 28.53 30.78 Aerage G 30.18 29.23 29.84 30.79 B 28.70 29.34 29.58 30.89 Summary Tis paper proposed an efficient image reconstruction algoritm, tis algoritm estimate alue of pixel based on gradient correlation between pixels. Tis metod is erified on FPGA, it can proide a ig-speed, ig-quality reconstruct image. Tis wole system as been applied to industrial successfully wit low-cost, ig-quality. References [1] HUANG W T, CHEN C H, TSAI Z H, et al. Hig-quality and ig-speed demosaicking algoritm design[j]. Taipei Uniersity Tecnol., 2007,40(2):13-25. [2] Nai-Xiang Lian, Lanlan Cang, Yap-Peng, et al. Adaptie Filtering for Color Filter Array Demosaicking. IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL.16,NO.10,OCTOBER 2007. [3]Zang Qigui, Zao Lijuan, Implementation of Bayer Image Interpolation based on FPGA. Taiyuan Uniersity of Tecnology, 2010 [4]Ning Yang. Processing system of ideo image based on FPGA. Dalian Maritime Uniersity.2008. [5]Malar H S, HE L W, Cutler R. Hig quality linear interpolation for demosaicing of bayerpatterned color images[c]// Processing of te IEEE International Conference on Speec, Acoustics, and Signal Processing, Montreal, Canada; IEEE,2004. [6] Jung, T.-Y., Yang, S., Lee, J., and Jeong, J., Enanced Directionally Weigted Demosaicing for Digital Cameras, IEEE International Conference on Signal Image Tecnology and Internet Based Systems 2008, 518-522 (2008). 853