Improving the Performance of Single Chip Image Capture Devices

Transcription

1 Digial Loyola Marymoun Universiy and Loyola Law School Elecrical Engineering & Compuer Science Faculy Works Elecrical Engineering & Compuer Science Improving he Performance of Single Chip Image Capure Devices Barbara E. Marino Loyola Marymoun Universiy, Rober L. Sevenson Universiy of Nore Dame Reposiory Ciaion Marino, Barbara E. and Sevenson, Rober L., "Improving he Performance of Single Chip Image Capure Devices" (2003). Elecrical Engineering & Compuer Science Faculy Works.. hp://digialcommons.lmu.edu/cs_fac/ Recommended Ciaion Marino BE, Sevenson RL; Improving he performance of single chip image capure devices. J. Elecron. Imaging. 000;2(2): This Aricle is brough o you for free and open access by he Elecrical Engineering & Compuer Science a Digial Loyola Marymoun Universiy and Loyola Law School. I has been acceped for inclusion in Elecrical Engineering & Compuer Science Faculy Works by an auhorized adminisraor of Digial Commons@Loyola Marymoun Universiy and Loyola Law School. For more informaion, please conac digialcommons@lmu.edu.

2 Journal of Elecronic Imaging 2(2), (April 2003). Improving he performance of single chip image capure devices Barbara E. Marino Loyola Marymoun Universiy Deparmen of Elecrical Engineering and Compuer Science Los Angeles, California Rober L. Sevenson Universiy of Nore Dame Laboraory for Image and Signal Analysis Deparmen of Elecrical Engineering Nore Dame, Indiana 6556 Absrac. Single chip charge-coupled devices (CCDs) coupled wih filers for isolaing red, green, and blue color conen are commonly used o capure color images. While his is more cos effecive han muliple chip sysems, bes resuls are obained when full RGB color informaion is obained for every poin in an image. The process of color subsampling in a single chip sysem degrades he resuling image daa by inroducing arifacs such as blurry edges and false coloring. We propose an algorihm for enhancing color image daa ha were capured wih a ypical single chip CCD array. The algorihm is based on sochasic regularizaion using a Markov random field model for he image daa. This resuls in a consrained opimizaion problem, which is solved using an ieraive consrained gradien descen compuaional algorihm. Resuls of he proposed algorihm show a marked improvemen over he original sampled image daa SPIE and IS&T. [DOI: 0.7/.56063] Inroducion The capure of color image informaion for digial processing requires he measuremen of a leas hree color specral bands a all poins of he image using a color scanner or camera. Typically, color is measured using a combinaion of filers and ligh-sensiive elemens, such as chargecoupled devices CCDs, o measure he red, green, and blue conen of he image. Opimally his is done by alernaely isolaing each color primary and measuring he inensiy of each color a each poin in he image. This is done using eiher a single CCD array wih muliple exposures hrough differen color filers or by using muliple CCD arrays wih differen color filers. Since muliple CCD sysems are expensive and muliple exposures are ofen no pracical, single chip CCD sysems have been developed. Individual elemens of he CCD array are each coupled wih a filer for measuring he color conen of one of he primaries. These individual filers are arranged in a mosaic Paper received Jun. 8, 999; revised manuscrip received Mar. 3, 2000 and Dec. 9, 200; acceped for publicaion Dec. 2, /2003/$ SPIE and IS&T. paern over he array, which effecively samples he image daa since each elemen in he CCD array can only measure one color. Since he differen specral bands are no longer being sampled a he same physical locaion, color arifacs are ofen inroduced. These arifacs are mos noiceable a edge locaions where he edges in each of he color primaries do no correlae wih each oher. This inroduces blurry edges and false coloring. The arifacs can be grealy improved by processing he image afer i has been capured. Researchers have proposed a wide variey of mehods o reduce hese arifacs. Ozawa and Takahashi, 2,3 and Sugiura, Asakawa, and Fujino examined his problem as i perains o digial video cameras. Omori and Ueda 5 deermined a correced, high-resoluion image using muliple images of he same objec, shifed in phase. Messing and Sezan 6 used muliple images capured by a camera operaing in burs mode o produce a single high resoluion image. This problem as i perains o digial sill cameras has been examined from many differen perspecives. Go, Sohn, and Lee 7 invesigaed an inerpolaion scheme based on neural neworks. Sakamoo, Nakanishi, and Hase, 8 and Toi 9 explored algorihms as suiable and opimal for specific plaforms. One prominan arifac produced from single chip image capure is he false coloring inroduced when he edges of objecs appear misaligned in differen specral bands. To compensae for his, researchers have proposed algorihms ha employ cross-channel correlaion. This research includes he work of Hur and Kang, 0 Kimmel, and Kuno and Sugiura. 2 These mehods are simple, nonieraive weighed inerpolaion schemes. We propose an ineraive algorihm for improving he resuls of single chip color image capure. The algorihm is based on sochasic regularizaion using a Gaussian image model wih a deerminisic line process o realign he edge informaion. The image model is used in a maximum a Journal of Elecronic Imaging / April 2003 / Vol. 2(2) / 209

3 Marino and Sevenson Fig. 2 Subsampling. Y i, j r g, Fig. Single chip CCD array. poseriori esimaion echnique, resuling in a consrained opimizaion problem. This is opimized using an ieraive consrained gradien descen compuaional algorihm. Secion 2 describes he forward process of capuring an image using a single chip CCD array. The esimaion echnique and he proposed algorihm used in he enhancemen process are inroduced in Sec. 3. Secion presens he resuls of esing he proposed algorihm, and Sec. 5 summarizes he resuls of his work. 2 Color Image Capure Using a Single Chip CCD Array As menioned in he previous secion, color image daa can be measured using a single chip CCD array. The layou of a ypical single chip CCD array is shown in Fig.. 3 The leers R, G, and B represen he filer associaed wih each paricular CCD elemen given is spaial locaion in he array. An elemen labeled R is coupled wih a filer ha enables he elemen o isolae and measure he red conen of he image a ha paricular locaion. Similarly, G elemens measure green informaion and B elemens measure blue informaion. Noice ha here are as many green elemens as red and blue combined. Single chip color CCDs are designed his way, since green appears brigher o he human visual sysem HVS han oher colors. Green herefore carries more visually imporan informaion han red or blue. During he image capure process, he values measured by he wo green elemens in he same 2 2 block are averaged o produce a single value of green. Measuring color informaion in his way effecively subsamples he color daa by a facor of 2 in each dimension, resuling in one value of each of he color primaries for each 2 2 block of pixels in he original image. A more radiional echnique for subsampling is accomplished by averaging he values of each color primary in he block. For he purpose of his discussion we refer o his radiional means of subsampling as proper subsampling. Le X represen a full resoluion M N color image in RGB space, where X V MN, V 0,. 3 Le Y represen he resuls of properly subsampling X by a facor of 2 in each direcion, Y V MN/. A color pixel a he i, j pixel locaion of he subsampled image Y is denoed by a vecor in 3-D space, Y i, j, where r, g, and b are he color elemens of ha vecor. Tha is, b i, j where r, g, b 0,. For noaional convenience a pixel in he full resoluion image X i, j is defined o conain he pixel values of he 2 2 sampling block of a CCD array, see Fig. 2. In his case he vecor X i, j conains four elemens for each of he primaries, X i, j R 0,0 R 0, R,0 R, G 0,0 G 0, G,0 G, B 0,0 B 0, B,0 B, i, j, 2 where he values of he elemens of his vecor are also conained in he inerval 0,. The forward process of obaining a properly subsampled image Y from a full resoluion image X is given by, Y i, j j SXi, j. Xi, A marix S, which subsamples he complee original image, Y S X, can be appropriaely formed from he emplae marices S. Improper subsampling achieved by capuring an image using a single chip CCD array is given by he equaion Y i, * j i, j S*X i, j X Again, a marix can be appropriaely formed o describe he process of improperly subsampling he complee image, Y* S *X, where Y* represens he resuls of improperly subsampling X / Journal of Elecronic Imaging / April 2003 / Vol. 2(2)

4 3 Bayesian Esimaion and he Enhancemen Process The enhancemen process involves esimaing Y from Y*. Le he esimae of he properly sampled color image daa be represened by Ŷ. This esimaion problem is ill posed, since here is no unique soluion. One mehod of esimaing a unique, and hus well-posed soluion is hrough Bayesian esimaion. To deermine he esimae Ŷ of he properly sampled color image daa Y, from he improperly sampled daa Y*, a maximum a poseriori MAP echnique is used. Employing his echnique, he esimae Ŷ can be wrien Improving he performance of single chip image capure devices, Ŷ S arg min X Z c C d c X 3 where Z is defined as he se of all possible images ha solve he forward problem, i.e., Z X V MN :Y* S *X. Le Xˆ arg min X Z c C d c X. 5 Ŷ arg max log Pr Y Y*. Y V MN/ 7 From Eq. 3 we see ha Xˆ can firs be compued and hen used o deermine Ŷ as follows, Using Bayes rule, Pr Y Y* Pr Y* Y Pr Y Pr Y* Pr Y* X Pr X Y Pr Y Pr Y* Pr Y* X Pr Y X Pr X, 8 Pr Y* which gives he esimaion he form, Ŷ arg min log Pr Y X log Pr Y* X Y V MN/ log Pr X. The full resoluion image X is inroduced ino he esimaion problem, since we have a good model for Pr(X). The condiional densiies are based on our knowledge of he subsampled image given he original image daa. Since he sampling process is known for boh he proper and improper cases, hese condiional densiies are known exacly, Pr Y X 0, Y S X, Y S X, 0 Pr Y* X 0, Y* S *X, Y* S *X. To model he image daa, a Markov random field MRF is assumed wih he Gibbs densiy funcion Pr X Z exp d c X, c C 9 2 where Z is a normalizing consan, is he regularizing parameer, c is a local group of pixels called cliques, C is he se of all cliques hroughou he image, d c is a coefficien vecor for clique c, and is a funcion of he cliques, which is furher defined laer. A MAP esimae for his esimaion problem can be found by solving he following minimizaion problem: Ŷ S Xˆ. 6 The derivaion of his minimizaion problem has been sparse. I is desired only o summarize well-known resuls. For furher informaion, he reader is referred o Refs. 5 and 6. The qualiy of he resuling esimae of Xˆ depends on he form of and d c. The funcion and he coefficiens in d c are se based on a priori assumpions abou he image daa. The a priori assumpion ha is incorporaed ino his work is ha image daa are basically smooh, however, edge informaion mus be mainained and realigned. For he firs par of he assumpion, he coefficiens in d c are se so ha d c X provides a measure of smoohness. This is done by using finie difference approximaions o a firsorder derivaive as he image daa smoohness measure. A pixel X i, j he four discree direcional derivaives approximae a roaionally symmeric operaor wihin a 3 3 pixel grid and are given as d i, j,0 X X i, j X i,j d i, j, X X i, j X i,j d i, j,2 X X i, j X i, j d i, j,3 X X i, j X i,j. 7 The second par of he assumpion can be aced on by exploiing he correlaion beween he color channels o deermine he rue edges in he image. A locaions where rue edges have been deermined, no smoohing is done o ensure he edges are mainained. These crieria can be me by using he convex quadraic correlaor funcion wih a line process, defined as a 0, a 2, if an edge is deeced oherwise. 8 When a correlaed edge is no deeced, he quadraic erm produces a leas squares fi o he daa, smoohing ou he false edges. When an edge is deeced, no cos is associaed wih his erm, aligning and preserving he rue Journal of Elecronic Imaging / April 2003 / Vol. 2(2) / 2

5 Marino and Sevenson Fig. Correlaed edges. edges. Noe ha he convexiy of ensures ha he minimizaion remains convex and he resuls sable. 6,7 A srong correlaion exiss beween he individual color planes of a full color image. An edge of an objec is ofen presen and aligned in more han one color plane. Images ha have been subsampled using a single chip CCD array lose his correlaion due o he physical separaion of he individual elemens in he array. Le a deeced edge be defined as a locaion where he direcional derivaive is greaer han some hreshold, d i, j,m X T, 9 where T is he value of ha hreshold. Le a poenial edge be defined as a locaion where a rue edge may have been originally locaed based on he presence of a deeced edge. The process of correlaed edge deecion is accomplished by firs deecing he edges presen in each of he color planes, deermining he locaion of poenial edges, and finally deermining he acual edges by maching up he correlaed poenial edges. Consider he problem of locaing edges in he direcion d i, j,0 Fig. 3 Deeced and poenial edges. X. Figure 3 illusraes how poenial edge locaions follow from he locaion of deeced edges. The RGB cubes represen he 2 2 sampling blocks of he CCD array wih he sampling locaions for each of he primary colors indicaed. The locaion of poenial edges follows naurally from he knowledge of he forward process and he locaion of he deeced edges. If an edge is deeced in he red or blue color planes, he acual edge could have been locaed a any poin, since ha paricular color plane was las sampled. In he green color plane, idenifying poenial edges is a lile more challenging. This is because he green value of a pixel is he average of wo sample poins. If an acual edge is locaed in he cener of a 2 2 sampling block, he edge is smoohed and replaced by a wo-sep edge. The final sep of correlaed edge deecion is o reconsruc he acual edge informaion given he poenial edges. Figure shows ha poenial edges from all he color planes line up a he locaion of he acual edge. I is no always he case, however, ha edges conain edge informaion in all color planes. I may be ha an acual edge only exiss in one color plane. Therefore, a mehod of deermining acual edge locaion is needed, which is independen of he number of color planes ha conain poenial edges. To include hese cases, he acual edge is said o exis a he locaion wih he highes number of poenial edges. In he case of a ie, he acual edge is said o exis a he locaion i was deeced, i, j. The previous developmen describes a mehod for idenifying correlaed edges in he direcion d i, j,0 X. The deec- ion of edges in he oher direcions follows a similar developmen. The necessary moivaion now exiss o inroduce he funcional form using a Markov random field model and he quadraic correlaor funcion. The proposed exponenial kernel of he modified Gaussian Markov random field GMRF image model is 3 X c C V c X i j m 0 d i, j,m X, 20 where X is a funcion of he fully specified image X. To find he MAP esimae Xˆ, he convex funcional X mus be minimized subjec o he consrain X Z. A seepes descen projecion echnique was seleced o minimize he funcional in Eq. 20. This consrained opimizaion mehod performs a seepes descen sep a each ieraion. Denoe he image esimae a he n-h ieraion as X (n). The 0 ieraion, X (0), can be iniialized using an expansion of he subsampled image daa Y* by replicaing he sample values hroughou he sampling neighborhood. To updae he image esimae X (n) in each ieraion, a sep in a paricular direcion v (n) is aken, where v (n) is of he same size as X (n). A common choice for his direcion is he negaive of he gradien direcion, where he gradien direcion can be found as, u n X n X n. 2 The direcion of descen mus hen be mapped ono he consrain space o confine he updae o he consrain of 22 / Journal of Elecronic Imaging / April 2003 / Vol. 2(2)

6 Improving he performance of single chip image capure devices Fig. 5 Enhancemen of color squares capured using single chip CCD arrays: (a) original image, (b) properly subsampled image, (c) improperly subsampled image, and (d) image enhanced using proposed algorihm. Journal of Elecronic Imaging / April 2003 / Vol. 2(2) / 23

7 Marino and Sevenson Fig. 6 Expansion of Fig. 5: (a) original image, (b) properly subsampled image, (c) improperly subsampled image, and (d) image enhanced using proposed algorihm. 2 / Journal of Elecronic Imaging / April 2003 / Vol. 2(2)

8 Improving he performance of single chip image capure devices Fig. 7 Enhancemen of color eggs capured using single chip CCD arrays: (a) original image, (b) properly subsampled image, (c) improperly subsampled image, and (d) image enhanced using proposed algorihm. Journal of Elecronic Imaging / April 2003 / Vol. 2(2) / 25

9 Marino and Sevenson Fig. 8 Expansion of Fig. 7: (a) original image, (b) properly subsampled image, (c) improperly subsampled image, and (d) image enhanced using proposed algorihm. 26 / Journal of Elecronic Imaging / April 2003 / Vol. 2(2)

10 Improving he performance of single chip image capure devices Eq.. Since he marix defining he forward process is known, his projecion marix can be found as 8 v i, j I S* S*S* S* u i, j, v i, j l mui, j. 23 Now ha he projecion operaor has been defined, he magniude of he direcion aken can be deermined. Le X n X n n v n, 2 where he scalar (n) represens he size of he sep aken in he direcion v (n). For fas convergence, a value for (n) should be seleced ha gives he opimal sep size oward he global minimum of he funcional. This can be done by approximaing he funcional wih a runcaed Taylor series and selecing he sep size ha minimizes his approximaion in he seleced direcion of descen. 9 The process of deermining descen direcion, projecing ono he consrain space, and calculaing he sep size is repeaed unil he problem converges on a soluion. Experimenal Resuls This secion demonsraes he value of he proposed algorihm hrough wo represenaive examples. Figure 5 shows he resuls of subsampling and enhancing he image of an array of colored squares. Four images are shown: he original image, he properly and improperly subsampled images, and he enhanced image. In addiion o he full size images, an expanded view of a single color block from each image is included o show greaer deail see Fig. 6. The original image was subsampled according o he processes of properly and improperly subsampling an image, discussed in Sec. 2. Noice he inroducion of false colors along he edges in he improperly subsampled image. The enhanced image obained from applying he algorihm proposed in his work shows an improvemen in he qualiy of he edges. The arifacs such as he false colors due o uncorrelaed edge informaion have been grealy reduced. The example given in Fig. 7 shows he resuls of subsampling and enhancing he image of a bowl of colored eggs. Again four images are shown. An expanded view of he cener of he image is shown in Fig. 8. Noice he inroducion of false colors paricularly along he op and boom edges of he bowl. Also, false colors are presen Table Signal o noise raio of he inerpolaed images as compared o he original image. along he boundaries of he blue eggs in he subsampled image. These arifacs have been grealy reduced in he enhanced image. Table quanifies he experimenal resuls. The SNR of he improperly subsampled image is compared o ha of he resuls achieved using he proposed enhancemen algorihm. In he case of boh es images, a gain of approximaely db is achieved. 5 Conclusion The physical separaion of he individual elemens in a single chip CCD array cause he edges in he RGB color planes o become misaligned. This inroduces arifacs such as false coloring along he edges of an image. Enhancing subsampled color image daa is a useful sep in producing a beer qualiy image for viewing. We propose an enhancemen algorihm based on sochasic regularizaion using a Gaussian image model wih a deerminisic line process o realign and mainain he edge informaion. The resuling compuaional algorihm involves an ieraive consrained gradien descen, which requires a correlaion operaor o realign he edge informaion. Resuls show ha he algorihm works well o reduce, and ofen eliminae, he visible effecs of his ype of image capure. References Image Colors Eggs Improperly subsampled 8.3 db 8.3 db Enhanced 9.6 db 9.2 db. B. E. Schmiz and R. L. Sevenson, The enhancemen of color image daa capured using single chip CCD arrays, Proc. Conf. Image and Video Process. IV, R. L. Sevenson and M. I. Sezan, Eds., Proc. SPIE 2666, N. Ozawa and K. Takahashi, A correlaive coefficien muliplying CCM mehod for chrominance moire reducion in single-chip color video cameras, IEEE Trans. Elecron Devices 38 5, N. Ozawa, Chrominance moire reducion using CCM signal inerpolaion single-chip color video camera, ITEJ 6 2, H. Sugiura, K. Asakawa, and J. Fujino, False color signal reducion mehod for single-chip color video cameras, IEEE Trans. Consumer Elecron. 0 2, S. Omori and K. Ueda, High-resoluion image using several sampling-phase shifed images, Proc. IEEE In. Conf. Consumer Elecron., pp June D. S. Messing and M. I. Sezan, Improved mul-image resoluion enhancemen for colour images capured by single-ccd cameras, Proc. IEEE In. Conf. Image Process., pp Sep J. Go, K. Sohn, and C. Lee, Inerpolaion using neural neworks for digial sill cameras, IEEE Trans. Consumer Elecron. 6 3, T. Sakamoo, C. Nakanishi, and T. Hase, Sofware pixel inerpolaion for digial sill cameras suiable for a 32-bi MCU, IEEE Trans. Consumer Elecron., T. Toi, Color signal processing echnique for single-chip CCD cameras ha employ CPUs wih SIMD insrucion ses, IEEE Trans. Consumer Elecron. 6 2, B. S. Hur and M. G. Kang, High definiion color inerpolaion scheme for progressive scan CCD image sensor, IEEE Trans. Consumer Elecron. 7, R. Kimmel, Demosaicing: Image reconsrucion from color CCD samples, IEEE Trans. Image Process. 8 9, T. Kuno and H. Sugiura, New inerpolaion mehod using discriminaed color correlaion for digial sill cameras, IEEE Trans. Consumer Elecron. 5, R. W. G. Hun, The Reproducion of Colour in Phoography, Prining Journal of Elecronic Imaging / April 2003 / Vol. 2(2) / 27

11 Marino and Sevenson & Television, Founain Press, Tolworh, England R. W. G. Hun, Measuring Color, Ellis Horwood, London, England B. E. Schmiz and R. L. Sevenson, Color palee resoraion, Compuer Vision, Graphics and Image Process. Graphical Models and Image Process. 57 5, R. L. Sevenson, B. E. Schmiz, and E. J. Delp, Disconinuiy preserving regularizaion of inverse visual problems, IEEE Trans. Sys. Man Cybern. 2 3, B. E. Schmiz, Curve reconsrucion: A balance beween smoohness and disconinuiy preservaion, Maser s Thesis, Universiy of Nore Dame Feb R. R. Schulz, Improved definiion image expansion, Maser s Thesis, Universiy of Nore Dame Jan B. D. Bunday, Basic Opimizaion Mehods, Edward Anmold, Balimore, MD 98. Barbara E. Marino received her BSEE degree in 989 from Marquee Universiy, and her MS and PhD degrees in elecrical engineering from he Universiy of Nore Dame in 993 and 996, respecively. In 996, she joined he faculy a Loyola Marymoun Universiy where she currenly serves as associae professor and deparmen head. Concurren o his academic appoinmen, Dr. Marino has been involoved in research wih he Je Propulsion Laboraory. Her ineress are in he area of image processing and elecronic imaging. She is a member of IEEE, SPIE, SWE, Tau Bea Pi, and Ea Kappa Nu. Rober L. Sevenson received his BEE degree (summa cum laude) from he Universiy of Delaware in 986, and his PhD in Elecrical Engineering from Purdue Universiy in 990. While a Purdue he was suppored by graduae fellowships from he Naional Science Foundaion, DuPon Corporaion, Phi Kappa Phi, and Purdue Universiy. He joined he faculy of he Deparmen of Elecrical Engineering a he Universiy of Nore Dame in 990, where he currenly holds he rank of professor. His research ineress include image/video processing, robus image/video communicaion sysems, mulimedia sysems, ill-posed problems in compuaional vision, and compuaionsl issues in image processing. Dr. Sevenson has consuled for several companies and governmen agencies in he areas of image/video compression, real-ime video processing, nigh vision sysems, and image processing. He has published and presened over 00 papers and has received research funding from NSF, NASA, The Deparmen of he Air Force, The Deparmen of Defense, Sun Microsysems, Apple Compuer, Inel, Microsof, Moorola, Lockheed Marin, and W. J. Schafer and Associaes. Dr. Sevenson is a member of IEEE, SPIE, Ea Kappa Nu, Tau Bea Pi, and Phi Kappa Phi. He is currenly serving as an associae edior for he IEEE Transacions on Image Processing and for he IEEE Transacions on Circuis and Sysems for Video Technology. He has previously served as an associae edior for he Journal of Elecronic Imaging. 28 / Journal of Elecronic Imaging / April 2003 / Vol. 2(2)