Coor magng usng a monochromatc dgta camera and a pup-doman dffuser Jonathan Hauser Mchae A. Goub Amr Averbuch Menachem Nathan Vaer A. Zheudev Omer Inbar and Sha Gurevtch Schoo of Eectrca Engneerng Facut of Engneerng Te Avv Unverst Ramat Avv Te Avv 69978 Israe Schoo of Computer Scence Facut of Eact Scences Te Avv Unverst Ramat Avv Te Avv 69978 Israe *Correspondng author: hausero@post.tau.ac. Receved XX Month XXXX; revsed XX Month XXXX; accepted XX Month XXXX; posted XX Month XXXX (Doc. ID XXXXX); pubshed XX Month XXXX Coor RGB magng wth hgh throughput was acheved b a monochromatc dgta camera wth a dspersve dffuser at ts pup. An acqured snapshot monochromatc mage was converted to coor coordnates through spectra b resortng to dgta processng wth a compressed sensng-based agorthm of spectra magng. Resuts of optca evauaton and cabraton of an optca sstem and coor magng eperments are reported. OCIS codes:. Introducton Estng red-green-bue (RGB) magng and photographc methods are based on an addtve coor mode of human vson wth three prmar coors. A straghtforward approach for the eectro-optca mpementaton of coor vson s based on tme-sequenta coor fterng that resorts ether to a set of mechanca echangeabe coor fters [-4] or to an eectronca tunabe coor fter e.g. a qud crsta fter [5] or Fabr-Perot nterferometer [6]. However tmesequenta coor fterng s not appcabe to snge snapshot photograph of dnamc fast changng obects. Furthermore each of the three RGB coor fters transmts about one thrd of the ncdent ght fu whe the other two-thrds of ght are absorbed causng substanta ght fu osses. The wde used Baer RGB coor fter arra (CFA) [7] ncudes a spata pattern of a perodca repeatng pattern of one red two green and one bue fters facng pes at the mage sensor as shown n Fg.. The Baer CFA enabes snapshot dgta coor magng but eads to resouton oss and nherts the substanta ght fu osses of tme-sequenta coor fters. Whe resouton oss n redundant optca mages s customar compensated b an nterpoaton process caed demosacng the ower than 4% ght throughput remans an ntrnsc dsadvantage of the RGB CFA. The whte-rgb (WRGB) CFA [8] n whch one of the green fters n the Baer pattern s repaced wth a whte (transparent or monochromatc ) one as shown n Fg. (b) ncreases ght throughput on up to 5%. Furthermore coor reconstructon n WRGB s senstve to nose. Some mprovement n ght throughput can be acheved b resortng to a mted number of RGB pes at a sparse set of ocatons on the mage sensor wth most whte pes [9] abet at the epense of coor renderng quat reducton between the sparse RGB pes. In an aternatve approach wth a Foveon arra [] each pe of the mage sensor conssts of 3 stacked aers of photodetectors where the top aer s senstve to bue ght the mdde aer s senstve to green ght and the bottom aer s senstve to red ght. Whe deverng the snapshot mode at fu spata resouton the Foveon arra st suffers from unacceptabe osses n ght fu at the aers of the mage sensor. To summarze a estng coor magng methods suffer from nherent osses n ght throughput. The quest for utmate ncrease n ght throughput ed us to the dea of removng a absorbng coor fters and re on a monochromatc mage sensor that senses the entre ght spectrum at each whte spata pe. We notced that coor components can be routne computed from the fu spectrum at each spata pe of the mage. Past studes report on spectra magng (SI) [-6] snapshot spectra magng (SSI) [7-] and compressve sensng (CS) [-7] for the nstantaneous (n a snge "shot") acquston of the spectra cube wth spata and spectra data of the obect. However the probem of coor magng b resortng to an ntermedate stage of SSI has not et been et addressed. Ths paper deas wth a method of computatona spectra-wse coor magng wth the ad of a reguar monochromatc mage sensor (wthout an coor fters) and wthout coor fters of an knd n the optca path between an maged obect and the mage sensor. Our method resorts to an ntermedate stage of snapshot spectra magng foowed b dgta computng of coor coordnates or tr-stmuus vaues from the spectra cube. Secton descrbes a desgn of the optca magng sstem that forms a dffused and dspersed (DD) mage
nstead of a sharp focused mage on the mage sensor to enabe we posed computatona reconstructon of the coor components through the fu spectrum at each spata pe. To acheve ths we nstaed a speca taored optca transparent dspersve phase-on statc dffuser (or smp "dffuser") at the entrance pup of the magng ens. Secton 3 descrbes a procedure for optca evauaton and cabraton of the spectra-based coor magng optca sstem to defne ts sensng matr for spata and spectra mage data. Secton 4 s dedcated to taored CS agorthms for dgta processng of the DD mage recorded on the monochromatc mage sensor to reconstruct the fu spectra cube and coor components of the mage. Secton 5 descrbes optca eperments for spectra-based coor magng based on our method. Secton 6 reports on epermenta nvestgaton for power osses n our method n comparson wth a reguar Baer RGB CFA camera. Secton 7 provdes dscusson of resuts concusons and mestones for future research n the fed.. Mode of the optca sstem wth a monochromatc sensor and dffuser A schematc aout of the spectra-based coor magng sstem wth a monochromatc mage sensor s shown n Fg.. The coor mage s obtaned through an ntermedate stage of snapshot spectra magng descrbed n [6]. In more deta the sstem n Fg. ncudes an magng ens a wde-bandpass spectra fter the transmssve phaseon dffuser at the pup doman.e. at the entrance pup or at the sstem aperture of the magng ens a monochromatc mage sensor and a dgta processor wth bocks for spectra cube reconstructon and converson to coor data. The stages of the spectra-based coor magng n our research are as foows: Eecute optca cabraton to fnd the sensng matr of transformaton of the obects spectra cube to the DD mage; Optca create a snge DD mage on the monochromatc mage sensor; Dgta reconstruct the spectra cube of the obect from the DD mage usng teratve CS-based agorthms [6] for SSI; Dgta convert the reconstructed spectra cube nto RGB components. Ths secton presents a set of mathematca equatons based on [6] and adapted to current notatons. Frst Eq. () descrbes the compe pup functon of the optca sstem wth the dffuser. Equatons ()-(4) descrbe the pont spread functon (PSF) and the contnuous convouton ntegra as the monochromatc response of the optca sstem comprsng the dffuser. Spata and spectra sampng are presented n Eqs. (5) (8). Formaton of a pochromatc DD mage on the monochromatc mage sensor s descrbed b Eq. (9). The atter enabes the matr formuaton of Eqs. () - () n terms of the sensng matr mode of the optca sstem. In the dffuser desgn we resorted to a combnaton of approaches that are customar n CS theor and cassca spectroscop. Specfca for successfu deconvouton modern CS theor requres a hgh randomzed sstem response wth a taored mage bur whereas cassca spectroscopc sstems comprse a dspersve optca eement ke a prsm or a dffracton gratng. Our snge taored phase dffuser provdes waveength-dependent ght dffuson for randomzaton and serves aso as an nherent dsperser. For characterzaton of the dffuser we use Cartesan coordnates u v at the et pup whch has aperture dmensons D D u v. Even though the dffuser s ocated at the entrance pup of the magng ens we resort here to the et pup coordnates u v because other pupdoman coordnates ma be brought to u v b scang wth the pup magnfcaton coeffcents. Specfca the dffuser was desgned as a set of phase dffractve grooves [6] to provde a D random phase functon ϕ ( u ; λdes ) at a desgn waveength λ des as shown n an nset n Fg.. At an arbtrar waveength the dffuser ehbts chromatc dsperson and provdes a waveength dependent pup functon [6] P ( u ; λ) = e p ϕ( u ; λ ) des λdes λ n( λ ) ( λ ) n des () wthn u.5* Du.5* Dv n λ s the refractve nde of a transparent matera of the dffuser grooves. We assumed that before ntroducton of the D dffuser optca resouton of the spata shft nvarant magng sstem was matched to the pe ptch of the mage sensor. Accordng the coherent PSF of entre magng sstem can be cacuated as the PSF of the dffuser.e. as the nverse Fourer transform of the pup functon Eq. () whch can be reduced n our case to the D verson.5* Du v where ( ) ( λ ) ( λ ; λ ) ( π ) h ; = P Rν ep ν dν ().5* Du u' u' u' where νu ' = u λr and R s the dstance from the center of the et pup to center of the mage sensor. The ntenst I ( ; λ ) contrbutng to the DD mage and cacuated separate at snge waveength λ at each coordnate of the obect ma be epressed b ust a D convouton ( λ) ( λ) ( λ) I ; = h -; I ; d (3) I of the dea ("non-dspersed") mage I( ; λ ) (.e. the obect n scae of the mage) wth the ncoherent PSF I ( ; λ ) ( ; λ ) h = h. (4) Mathematca formuatons for the dgta mage processng obvous requre a transfer from contnuous coordnates and waveengths to dscrete ndces of vectors and matrces. Fg. 3 shows the mathematca matr mode for the transformaton of the spectra cube to the DD mage. To defne spata sampng we consder the dscrete peated structure of the mage sensor whch s characterzed aong and aes b a D spata ptch δ δ a number N N of pes and a number N of bts per pe. The b magng zoom has to match the etenson of a bur that s caused b the PSF of the dffuser such that the entre DD mage stas wthn the aperture of the mage sensor.e. wthn N pes n each row and N pes n each coumn. The non-dffused-non-dspersed mage obtaned wthout a dffuser at same zoom s ess spread and occupes accordng on a smaer number N < N of pes ocated n the centra part of each mage sensor row wth margns c = ( N N) on both sdes. In the waveength drecton the spectra cube s aso samped to a fnte number L of spectra bands wth centra waveengths λ = L. The dscrete verson of the spectra cube voes ( + ; ) I = I λ can be epressed as NL N matr ( ) I = ( I = N = N = L) (5) c
wth concatenated spectra and - spata dmensons as shown n Fg. 3. I ; λ h -;λ are The dscrete versons of ( ) and PSF ( ) ( ; ' ) I = I λ δ (6) ( ) ( ) h = hi ' - + ; λ c = hi c δ- + ; λ c. (7) Now the contnuous D convouton n Eq. (3) can be appromated b a dscrete D aperodc convouton apped separate to each of N mage rows = N = N and snge waveength band N = I = h I. (8) The contrbuton of the entre set of the waveengths bands = L to pes of the DD mage and can be epressed as a weghted sum of the ntenstes of I apped through a the waveength bands L L N ' = κ = = = = A I I A I I = κ h. (9) Note that A provde eements of the sensng matr and n our mode do not depend on nde = N. Therefore rows of the obect are maged ndependent of each other n match wth Eqs. (3) and (8). Accordng each row of the DD mage s n one-to-one correspondence wth a respectve row of the spectra cube. For further data processng Eq. (9) can be epressed n matr form where I s the spectra cube matr AI = I () ( ) ' I = I = N = N () s the N N matr representng the DD mage and s N ( A N N L) A = = = = () NL bock Toeptz sensng matr as shown n Fg. 3. Note that A has fewer rows than coumns and operates on the spectra cube to form the DD mage. 3. Determnaton of the sensng matr b sstem evauaton and cabraton The sensng matr A can be drect accessed b sstem evauaton and cabraton procedures e.g. b drect PSF measurements. Ths secton descrbes the spata and spectra cabraton procedures that further deveop and mprove resuts of [6]. In the sstem shown n Fg. we used an Pad screen as an obect n some of the cabraton measurements and n addton a set of L =33 narrow bandpass spectra fters wth nm FWHM b Thorabs that covered the 4-7nm waveength range n equa gaps of nm. We aso used a commerca Ocean Optcs USB4-VIS-NIR cabrated spectrometer. To prepare for spectra cabraton measurements we consdered the nomna spectra transmttance of the narrow nm FWHM bandpass fters as shown n Fg. 4 and aso a measured one whch was accessed wth the spectrometer whose fber nput port was paced frst before and then after the bandpass fters. We aso averaged the spectra transmttance to an arra M = L wth the nm resouton as shown n Fg. 4 (b). The nomna and measured arras are n prett good match ecept for the 43nm and 5nm bands. At ths stage we negected spectra cross-taks that were caused b spectra overaps between bandpass fters. In order to characterze the reatve spectra senstvt κ of the optca sstem n each the waveength band = L we resorted to a spata unform whte patch obect on the Pad screen and measured mage spectrum n two was: frst wth the spectrometer whose fber nput port was paced after the bandpass fter and then b sensng the mage n the optca sstem of Fg. specfca wthout the dffuser but wth sequenta nstaed narrow bandpass fters. The pes wthn the mage of the whte patch were spata averaged for nose reducton. The data from the spectrometer was spectra averaged down to the nm resouton. Then the reatve spectra senstvt of the optca sstem κ that partcpates n Eq. () was cacuated b dvdng the data of the mage pes wth the data from the spectrometer. At 4nm 4nm 7nm and 7nm waveength bands we notced ow SNR due to ow spectra power of the Pad screen and used nomna vaues nstead of measured ones. In the cabraton measurements for the monochromatc D PSF wth a horzonta dffusng D dffuser we dspaed a narrow 4- pes thck whte vertca ne on the Pad. Then we maged t to the mage sensor n the dark through sequenta echanged narrow bandpass fters whch correspond to L waveength bands. We shoud note that the whte ne's wdth corresponded to a geometrca mage wdth of appromate pes wthout a dffuser. Thnner ne woud have been preferred however the chosen wdth was the mnma to obtan reasonabe SNR for the PSF measurements wth the dffuser. The measured D PSF at dfferent vertca ocatons of the fed of vew was provded n ths arrangement drect b rows of the mage sensor data. The ntegraton tme of the mage sensor n the PSF measurements vared from one to other waveength to match effects of chromatc dsperson varatons n the spectrum of the vertca whte ne the spectra transmttance of the narrow band-pass fters quantum effcenc of the mage sensor and the overa spectra senstvt of the optca sstem. Whe the chromatc dsperson quantum effcenc of the sensor and the spectra senstvt of the optca sstem shoud be emboded n the sensng matr the other factors shoud be compensated. Accordng we deveoped and used the foowng spectra correcton scheme n the PSF measurements for pes receved from the mage sensor:. Acqure monochromatc mage sensor data from the vertca ne obect through the th spectra fter.. Dvde the measured pes vaues b the ntegraton tme at the waveength band reatve to mnmum ntegraton tme. 3. Dvde the measured pe vaues b the correspondng spectrum of the whte ne obect as obtaned b spectrometer before the bandpass fters and downscaed to nm resouton. 4. Dvde the measured pe vaues at each waveength band b the correspondng bandpass fter's transmttance M Fna after the cabraton procedure we get the sensng matr eements A n accordance to Eq. (9). 4. Reconstructon of coor data from monochromatc DD mage For reconstructon of coor coordnates from the DD mage we frst reconstructed a spectra cube [6] and then converted from the
spectrum at each spata pe to the RGB coor coordnates at same pe b we-known equatons of the CIE 93 standard [8]. Ths secton presents a set of mathematca equatons as foows. Equatons (3) and (4) provde the tpca CS mert functon for the numerca quat of the spectra cube reconstructon whch ts teratve process s descrbed n Eq. (5). Fna Eqs. (6) and (7) descrbe converson from spectra cube to RGB data of the obect whereas Eqs. (8) - () descrbe the fgure of mert for the reconstructon quat. The mode epressed b Eq. () shows that the recorded DD mage I ncudes a near mture of spectra and spata data of the entre spectra cube I as descrbed b sensng matr A. The number N N of equatons for I n Eq. s ess than the number of unknown varabes N NL n I. The CS probem conssts of the reconstructon of matr I n such a wa that Eq. () wth a gven matr I becomes satsfed wth some accurac. Even though the reconstructon probem seems to be -posed and ma have an nfnte number of soutons the CS theor [ 9] resorts to the sparse nature of the D dgta mages to fnd a sensbe souton. The mathematca reaton between the spectra cube matr I and ts respectve sparse representaton d can be represented as a near transform d = DI or nverse transform I = Ψd wth a sparsfng matr D whch we mpemented as waveet-frameet transforms [3]. Sparst of an mage n the waveet-frame doman means that d has reatve sma number K of non-zero frame coeffcents. The CS theor [] states that the K -sparse sensbe souton d (and consequent I ) can be reconstructed for a speca cass of K -sparse matrces Θ= AΨthat satsf a restrcted sometr propert (RIP) of respectve order K [ 6]. In CS-based reconstructon of sparse representaton d of spectra cube I from DD mage I we ook for a souton of Eq. () that s mama sparse. Specfca we resorted to the mnmzaton of a functona µ χ d + A I I + d DI (3) where µ and χ are Lagrange weght coeffcents and the and norms are defned as d = d d = d. (4) The souton of the mnmzaton probem Eq. (9) was found b a cosed oop of non-adaptve near proectons n the foowng spt- Bregman teratve (SBI) process [6 9-3]: ( µ χ ) µ ( ) χd ( ) + + = shrnk ( D + χ ) + + + = + δb ( D ) m+ m m+ = c + δc ( A I ) m+ m m m I = AA+ E A c + d b d I b m m m b b I d m m m m c (5) m m where m s the number of the teraton b and c are ntermedate vectors n eecuton of the teratons A denotes a transposed matr A δ = δ = E s unt dagona matr and the shrnk b c functon shrnk ( γ) sgn ( ) ma ( γ ) = for a threshod γ s apped to each vector component. After competon of the teratons [6] we obtan the reconstructed spectra cube I wth voes I arranged as a set of vectors wth spectra data for ever spata pe. In order to transfer from the reconstructed spectra cube I of dmensons N N L the requred RGB representaton of sze N N 3 the foowng procedures [33] were apped. We cacuated the XYZ coor coordnates at each spata pe = N = N b resortng to the CIE 93 XYZ standard observer coor coordnates defnton [8] whch can be epressed n our notatons as L = I = L = = L = I = X Y I Z Here ( ) ( ) z( ) z ( λ ) ( λ ) ( λ ) (6) λ λ λ are the CIE's coor matchng functons that provde the numerca descrpton of the chromatc response of the standard observer versus waveength λ of the ght [8]. Then we scaed XYZ coordnates Eq. (6) to the nterva [] b cppng negatve vaues to and normazng b the mama XYZ coordnate vaue. Subsequent near converson from XYZ to standard RGB coor components C c denoted wth nde c= RG or B was mpemented b mutpcaton over the foowng constant matr [34]: C R 3.46 -.537 -.4986 X C G -.9689.8758.45 = Y C B.557 -.4.57 Z (7) wth subsequent scang of the Cc to the nterva [] b cppng ther negatve vaues to and vaues greater than vaues to. We aso ncreased the brghtness of the mages b mutpng the RGB vaues b a coeffcent s>. Fna the reconstructed RGB mage comprses N N 3 RGB components ( ) C = ( C c = N = N c= RG B). In order to evauate the quat and errors n the reconstructed RGB mage C we compared t to a reference RGB mage C of sze N N 3 whose spectra cube was measured drect wth the set of narrow bandpass fters for each of the L waveengths n the 4-7nm range. As the reconstructed and the reference RGB mages do not necessar share the same dnamc range we normazed each of the data arras C C to ts mama vaues Cma ma C c c = Cma = ma C c (8) c where denotes the spata pe and c= RG or B. Then we epressed the normazed root-mean-square errors (RMSE) of the fu RGB mage as ( )
.5 N N 3 ( ) ( ) Cc Cma Cc C ma (9) = = c= RMSE = 3 NN and cacuated the peak sgna-to-nose rato (PSNR) as PSNR = og ( RMSE). () For anass of the separate R G and B components we aso defned smar quanttes C mac C mac RMSE c and PSNR c where c= RG B b obvous modfcaton of Eqs. (8) (9) and () n removng mamzaton and averagng wth respect to nde c. For even more detaed anass for the spata dstrbuton of the RGB mage errors we defned further smar quanttes C C RMSE ma ma and PSNR per spata pe b obvous modfcaton of Eqs. (8) (9) and () n removng mamzaton and averagng wth respect to ndces. 5. Resuts of optca eperment for coor magng wth monochromatc sensor The concept of our CS-based coor magng camera was proven n an optca eperment based on optca arrangement n Fg.. The hardware ncuded obect generator as an Pad screen mounted on the optca bench at a fed dstance of 88cm n front of the magng ens a 5Mpe N =-bt monochromatc mage sensor wth Aptna Demo b kt [35] a DSL935 Sune magng ens and a wde bandpass spectra fter for the entre vsbe range. The mage sensor had a pe ptch δ = δ =.μm and 59 944 pes that was enough to ncude the spread ntroduced b the dffuser. Even though our generc hardware and software enabed arge dmensons for proof of the concept we actua reconstructed the spectra cube wth dmensons N N L = 56 56 33 pes. Front (near frst ens surface) poston of the 3.mm entrance pup of DSL935 Sune magng ens wth numerca aperture (NA) of.7 enabed proper pacng of the dffuser at the pup. The reference spectra measurements were eecuted wth the Ocean Optcs USB4 spectrometer. The D dffuser wth 3.mm cear aperture and 4 strpes of 8 µm wdth was fabrcated at the Te Avv Unverst Nano-Center factes wth a standard 6-depth eve bnar starcase technoog on a.5mm thck doube-sde poshed fused sca substrate. A speca aumnum hoder was used to attach dffusers to the ens housng and convenent echange them. For cabraton measurements of the PSF and the sensng matr foowng procedure of secton 3 we resorted to the fter whees wth the set of L =33 narrow bandpass Thorabs spectra fters. The mage sensor's ntegraton tme was adusted at each spectra band to keep the peak ntenst beow 9% of the saturaton eve. We then subtracted the dark mage from measurements and averaged the resuts for severa postons to ed the PSF at each spectra band as a pe row of the mage sensor. The normazed PSF was then centered padded wth zeroes and ntegrated nto the bock Toeptz sensng matr A as epaned n Fg. 5. Fgure 6 shows the optca measured sensng matr consstng of 33 bocks each matchng one waveength band. To have frm references for quat evauaton of the CS reconstructed RGB mage n our eperments we conducted drect measurements of the spectra cube for a set of test obects Mandr Coor checker and Peppers and converted spectra cubes to RGB mages. The spectra cube of each test obect was drect acqured b magng wthout the dffuser through a sequenta changed set of L narrow band-pass fters. At each waveength band and spata coordnate we apped the spectra correcton of the measured spectrum b dvdng ts pes over narrow bandpass fter's transmttance M and the spectra senstvt vaue κ of the optca sstem. For comparson of our spectra-based coor magng method wth reguar RGB magng we drect acqured RGB mages n PNG format b a conventona 5Mp RGB camera [35] wth Baer CFA. To keep consstenc we used the same Sune DSL935 magng ens and paced the wde bandpass fter for the vsbe range n front of t as was used for the DD mage. In optca bench eperments for snapshot magng of the coored obects wth the monochromatc mage sensor we acqured severa epermenta snapshots of DD mages of dfferent obects n the sstem of Fg.. The data of the snapshots was dgta pre-processed wth a nose reducton fter [4]. The dgta reconstructon of the spectra cube was performed usng the SBI process wth thorough optmzed parameters µχ n Eqs. (3) and (5). We have apped aso spata medan fterng on each of the monochromatc mages of the spectra cube after each SB teraton. Fgures 7-6 show epermenta resuts of rea optca magng and subsequent dgta processng for obtanng coor RGB mages from the monochromatc mage sensor. Fgures 7 and 3 for the Mandr" "Coor Checker" and Peppers" obects show reference RGB epermenta DD and reconstructed RGB mages. Fgures 7 and 3 show the orgna obects as dspaed on the Pad Fgs. 7(b) (b) and 3(b) show drect acqured RGB mages wth 3.ms.6ms and.ms ntegraton tmes. Fgures 7(c) (c) and 3(c) show reference RGB mages cacuated from spectra cube measurements wth the 33 bandpass fters wth a brghtness factor of s=.5. Fgures 7(d)-(f) (d)-(f) and 3(d)-(f) show the RGB mages that were CS-SSI reconstructed from the optca recorded epermenta data. The CS-SSI reconstructon was based on agorthms of secton 4 wth brghtness factors of s= s=.5 and s=.5 respectve. The burrng of the reconstructed B mage s ke caused b hazng on some of the bue-regon narrow bandpass fters whch were used for the correspondng PSF measurements. Note that some burrng s aso observabe n the reference B band mage whch was cacuated from measurements taken wth the same set of fters. Addtona cause s a ow SNR eve n the PSF measurements at weak spectra regons of the Pad's screen. The cacuatons for PSNR and RMSE between the reference mages of Fgs. 7(c) (c) 3(c) and the reconstructed mages of Fgs. 7(f) (f) 3(f) RGB mages eded 6.4 7. 9. and.5.4. respectve. Fgures 7(g) (g) and 3(g) show the DD mages acqured n raw PNG format b the monochromatc mage sensor wth 3.ms.6ms and.ms ntegraton tmes. We emphasze that the RGB mages n Fgs. 7(d)-(f) (d)-(f) 3(d)-(f) were obtaned wth a monochromatc mage sensor n a camera equpped wth a fu transparent phase-on dffuser. Fgures 8 and 4 show a comparson between reference and reconstructed RGB vaues at 8 samped spata coordnates that were marked n Fgs. 7(c) (c) and 3(c). Fgures 9 and 5 show the separated R G and B mages of the "Mandr" "Coor Checker" and Peppers obects. Fgures 9 and 5 show the reference mages and Fgs. 9(b) and (b) 5(b) shows the CS-SSI reconstructed RGB mages. The PSNRC (RMSEC) vaues for the three obects were foowng: for Fg. 9(b) 5 SBI runs eded 7.6 (.3) 5.6 (.7) and 4.7 (.8) for Fg. (b) 4 SBI runs eded 5.8 (.6).4 (.96) and 7. (.4) for Fg. 5(b) SBI runs eded 6.4 (.5).4 (.86) and. (.78) a vaues are provded separate for the R G and B mages respectve. Fgure 6 shows more detas of the reconstructon errors for the
"Mandr" "Coor Checker" and "Peppers" obects. Fgure 6 shows the PSNRC vaues of the R G and B mages of the 3 obects whe Fgures 6(b)-(d) show the spata dstrbuton PSNR for the same 3 obects. To summarze Fgs. 7-6 revea satsfactor vsua quat and PSNR eves of the RGB mages obtaned from the monochromatc mage sensor. 6. Lght throughput of the RGB camera wth monochromatc mage sensor Lght throughput s one of the most mportant and compettve features of the proposed here coor magng wth monochromatc mage sensor. In order to evauate the ght throughput of our method we have shot mages of whte red green and bue unform obect patches from the Pad screen and compared resuts of our method wth reguar RGB magng. Each patch was shot n three dfferent hardware confguratons: ) b the monochromatc camera wth a whte pes and wthout the dffuser when mage of each patch corresponded to 56 56 pes on the sensor pane.; ) b the monochromatc camera wth the dffuser; ) b the reguar 5Mp RGB camera wth Baer CFA. The same magng ens and wde bandpass spectra fter for the vsbe range were used n a of the measurement modes. The ntegraton tme n the monochromatc mage sensor was set to.6ms whch satsfed a peak ntenst of 9% of the saturaton ntenst eve n the case wthout the dffuser. The ntegraton tme was dentca n a of the measurement modes. For nose reducton each mage was obtaned b sequenta averagng of 5 frames. The mages were obtaned n raw PNG format to avod an correctons b the but-n graphc processor of the camera. We aso obtaned and subtracted correspondng dark mages from the measurements. Fgure 7 shows ght throughput percentage bars for each measurement mode and each of the coors of patch obect. At the Baer CFA camera the nonzero pes that do not match the R G and B obects characterze spectra "cross-taks" between the RGB prmares of the sensor and the Pad screen. The tota ght n arbtrar unts was cacuated b summng the camera pes.e. ght ntenstes vaues. For the monochromatc camera wthout the dffuser cacuaton of the fu was performed wthn 56 56 pes. For the monochromatc camera wth the dffuser fu cacuaton was etended to 56 X 59 pes of the DD mage. For the RGB camera fu cacuaton was apped separate for each of the R G and B pe groups wthn boundares of the mage of the patch obect.e. 8 8 pes for each of R and B mages and 8 8 pes for the G mage. The rato of ght fu n dfferent confguratons was referenced to the fu obtaned b the monochromatc camera wthout the dffuser. Fgure 7 shows that the ght throughput of our method for the whte patch obect s appromate 89% from that of the monochromatc camera wth a whte pes. Ths vaue s sgnfcant hgher than the 39% tota ght throughput of the RGB camera wth Baer CFA cacuated for a RG and B pes atogether. Fgure 7(b)-(d) show that the ght throughput of our method for the coor obects s between 87.5%-89.6% from that of the monochromatc camera wth a whte pes. Ths throughput s sgnfcant hgher than the correspondng 7.8% 36.9% and 8.7% vaues of the Baer CFA sensor. Therefore our method provdes more than doube gan n the ght throughput compared to an equvaent reguar RGB camera equpped wth a conventona Baer CFA. Such gans were epected because CFA n reguar RGB camera absorbs rough two-thrd of the ncdent ght n ether of the RG or B fters whe our suggested method makes use of hgh effcent transparent phase on dffuser. 7. Dscusson and concusons We epermenta proved the feasbt of coor magng wth a monochromatc mage sensor and a pup doman phase-on statc dffuser. In our approach the ght transmttance throughput s more than twce hgher than wth customar used ght absorbng RGB Baer coor fter arras. The ke eement of our optca sstem s a dffuser desgned to create a randomzed sensng matr that s evauated drect from cabraton measurements of the PSF. The dffuser can be produced b routne dffractve optcs technoog n requred quanttes ncudng those of consumer-market mobe devces. The dffuser ma be an embedded ntegra part of a dedcated monochromatc dgta camera of a mobe devce or aternatve serve as an eterna patch on the wndow of a dgta camera. The use of a monochromatc sensor nstead of a reguar mosac coor sensor ncreases the amount of the captured ght and therefore the senstvt of the camera whch devers sgnfcant advantages for ow ght magng appcatons. Our method rees substanta on spata and spectra mng of the D ght fed at the mage sensor and subsequent reconstructon of the spata and coor data wth CS-based agorthms. Such s acheved b proper use of the sparst propert natura attrbuted to photographc mages. We epermenta demonstrated the feasbt of reconstructng RGB mages through SSI and near teratve process of spt Bregman teratons. The deveoped spectra-wse based coor magng agorthms can be mpemented n the frmware of dgta cameras. Therefore the common used Baer coor fter arra at mage sensor can be substtuted wth a transparent phase-on dffuser at the pup thus mprovng the ght throughput of the optca sstem. Whe hgh ght throughput s aread acheved n our method spata resouton sgna to nose rato and coor renderng quat st to be mproved. For quat mprovement n further research we foresee optmzaton of the pup dffuser more precse cabraton of the sensng matr of the magng sstem and deveopment of effcent compressve sensng agorthms drect n space of coor coordnates. Ths research was parta supported b the Israe Mnstr of Scence and Technoog (Grants No. 3-46 and 3-36). References. E. G Nassmbene Coor vdeo record and paback sstem Sept. 5 97. US Patent 3598.. J. Sampse Whte ght enhanced coor fed sequenta proecton Aug. 3 993. US Patent 533385. 3. I. Šoc Chan b-refrngent fters Cechosovack fzcesk zurna vo. 9 no. pp. 37 49. 4. C.-K. Lu K.-T. Cheng and A. Yng-Gue Fuh Desgns of hgh coor purt RGB coor fter for qud crsta dspas appcatons usng Fabr-Perot etaons Dspa Technoog Journa of vo. 8 pp. 74 78 March. 5. P. J. Mer C. C. Hot "Mutspectra magng wth a qud crsta tunabe fter" SPIE DOI:.7/.98889 (995). 6. A. Raz and D. Mendovc Sequenta fterng for coor mage acquston Opt. Epress vo. pp. 6878 6883 Nov 4. 7. B. Baer Coor magng arra Ju 976. US Patent 39765. 8. H. Honda Y. Ida G. Itoh Y. Egawa and H. Sek A nove Baer-ke WRGB coor fter arra for CMOS mage sensors vo. 649 pp. 649J 649J 7. 9. Chakrabart Aan Wam T. Freeman and Todd Zcker. Rethnkng Coor Cameras. 4 IEEE Internatona Conference on Computatona Photograph (ICCP) (4).. Rchard F. Lon Rchard B. Merr (Foveon Inc.) Vertca coor fter sensor group arra that emuates a pattern of snge-aer sensors wth
effcent use of each sensor group's sensors Februar 4 6 US Patent 699866 B.. D. J. Brad Optca Imagng and Spectroscop We-Interscence (9).. Y. Garn I. T. Young and G. McNamara Spectra magng: prncpes and appcatons Ctometr Part A 69A 735 747 (6). 3. J. W. Uhr M. L. Huebschman E. P. Frenke N. L. Lane R. Ashfaq H. Lu D. R. Rana L. Cheng A. T. Ln G. A. Hughes X. J. Zhang and H. R. Garner Moecuar profng of ndvdua tumor ces b hperspectra mcroscopc magng Trans. Res. 59 366 375 (). 4. R. L. Long K. B. Wash and C. V. Greens Sugar magng of frut usng a ow cost charge-couped devce camera J. Near Infrared Spectrosc. 3 77 86 (5). 5. H. Lang Advances n mutspectra and hperspectra magng for archaeoog and art conservaton App. Phs. 6 39 33 (). 6. D. H. Foster K. Amano S. M. C. Nascmento and M. J. Foster Frequenc of metamersm n natura scenes J. Opt. Soc. Am. A 3 359 37 (6). 7. X. Yuan T. H. Tsa R. Zhu P. Lu D. J. Brad and L. Carn Compressve Hperspectra Imagng wth Sde Informaton IEEE J. Se. Top. Sgna.9 964-976 (5). 8. L. Gao R. Kester N. Hagen and T. Tkaczk "Snapshot Image Mappng Spectrometer (IMS) wth hgh sampng denst for hperspectra mcroscop" Opt. Epress 8 433-4344 (). 9. M. Descour and E. L. Derenak "Computed-tomograph magng spectrometer: epermenta cabraton and reconstructon resuts" App. Opt. 34 487-486 (995).. A. Gorman D. W. Fetcher-Homes and A. R. Harve Generazaton of the Lot fter and ts appcaton to snapshot spectra magng Opt. Epress 8 56 568 ().. A. Wagadarkar R. John R. Wett and D. Brad "Snge dsperser desgn for coded aperture snapshot spectra magng" App. Opt. 47 B44-B5 (8).. D. L. Donoho "Compressed sensng" IEEE Trans. Inform. Theor 5 89 36 (6). 3. A. Averbuch P. Nettaanmäk and V. Zheudev "Spne and spne waveet methods wth appcatons to sgna and mage processng" Vo. : Perodc spnes Sprnger (4). 4. E. J. Candès J. K. Romberg and T. Tao "Stabe sgna recover from ncompete and naccurate measurements" Communcatons on Pure and Apped Mathematcs 59 7 3 (6). 5. H. Argueo and G. R. Arce Restrcted sometr propert n coded aperture compressve spectra magng IEEE Statstca Sgna Processng Workshop (SSP) (). 6. M. A. Goub A. Averbuch M. Nathan V. A. Zheudev J. Hauser S. Gurevtch R. Mansk and A. Kagan "Compressed sensng snapshot spectra magng b a reguar dgta camera wth an added optca dffuser" Apped Optcs 55 (3) 43-443 (6). 7. A. Stern "Optca Compressve Sensng" CRC Press (6). 8. Smth Thomas; Gud John "The C.I.E. coormetrc standards and ther use". Transactons of the Optca Socet. 33 (3): 73 34. do.88/475-4878/33/3/3 Apr 93. 9. Z. Shen "Waveet frames and mage restoratons" n R. Bhata edtor Proceedngs of the Internatona Congress of Mathematcans Vo. IV 834 863 New Deh Hndustan Book Agenc (). 3. T. Godsten and S. Osher "The spt Bregman method for reguarzed probems" SIAM Journa on Imagng Scences 33-343 (9). 3. J. F. Ca S. Osher and Z. Shen "Spt Bregman methods and frame based mage restoraton" Mutscae Modeng and Smuaton: A SIAM Interdscpnar Journa 8 337-369 (9). 3. A. Averbuch and V. Zheudev Interpoator frames n sgna space " IEEE Trans. Sgn. Proc. 54 6-39 (6). 33. Davd H. Foster. ur: http://personapages.manchester.ac.uk/staff/d.h.foster/tutora_hsir GB/Tutora_HSIRGB.htm (Ju 7). 34. Internatona Eectrotechnca Commsson (IEC) 6966--:999 "Mutmeda sstems and equpment - Coour measurement and management - Part -: Coour management - Defaut RGB coour space srgb" October 999. 35. Aptna Imagng Demo kts. ur: https://www.onsem.com/powersoutons/product.do?d=mt9p3 (3 Ju 7).
Fgures. (b) (c) Fg.. Baer RGB pattern; (b) WRGB pattern; (c) Monochromatc sensor pattern wthout coor fters. Band PSF Band PSF : Band PSF : Band 33 PSF 33 Normazaton Centerng Zero Paddng Fg. 5: Integraton of the PSF data nto the sensng matr A. A Integraton to Sensng Matr A A = [A A A A 33 ] Entrance pup dffuser Obect u Bandpass fter v Imagng ens v' u' Sstem aperture Et pup Dffused and dspersed mage ' Image sensor R ' Dgta processor Spectra cube reconstructon Coor mage data Fg.. Coor magng sstem wth monochromatc mage sensor a pup-doman dffuser and a dgta processor for compressed sensng mage reconstructon. 5 5 5 3 4 5 6 7 8 = 3 4... 3 33 =N (-)+ ' N = = =N (-)+ L N pes =L = = L N pes DD mage I' c N N Fg. 6. Sensng matr A but on the base of the cabraton measurements. The matr eements are encoded n pseudo-coor for dspa. c N pes Sensng matr A N pes N pes =L N coumns Spectra cube I N coumns Fg. 3. Transformaton of the spectra cube to the DD mage. A s the sensng matr drect reated to the PSF..8.6 Transmttance.4..8 4 45 5 55 6 65 7 [ nm ].6 Transmttance.4. 4 4 4 43 44 45 46 47 48 49 5 5 5 53 54 55 56 57 58 59 6 6 6 63 64 65 66 67 68 69 7 7 7 Fg. 4. The spectra data for the narrow band-pass fters; nomna spectra transmttance as functon of the waveength for ever fter; (b) averaged wthn nm waveength bands transmttance the bue - nomna and the eow - measured. [ nm ] (b)
(b) (c) (b) (c) (d) (e) (f) (d) (e) (f) (g) Fg. 7. Obect "Mandr". as dspaed on the Pad; (b) drect acqured b the reguar RGB sensor; (c) reference cacuated from spectra cube measurements wth spectra fters; (d)-(f) CS-SSI reconstructed RGB mages; (g) DD mage. Fg.. Obect "Coor Checker". as dspaed on the Pad; (b) drect acqured b the reguar RGB sensor; (c) reference cacuated from spectra cube measurements wth spectra fters; (d)-(f) CS-SSI reconstructed RGB mages; (g) DD mage. (g).8.8.8.8 () () (3) (4).8.6 () () (3) (4).8.8.8.6.6.6.6.6.6.6.4.4.4.4.4.4.4.4.........8 (5) (6) (7) (8).8.8.8.8.6 (5) (6) (7) (8).8.8.8.6.6.6.6.6.6.6.4.4.4.4.4.4.4.4...... Fg. 8 The R G and B vaues for the Mandr at 8 samped spata ocatons of Fg. 7(c). Bue - reference red - CS-SSI reconstructed... Fg.. The R G and B vaues for the Coor Checker at 8 samped spata ocatons of Fg. (c). Bue-reference red - CS-SSI reconstructed. (b) (b) Fg. 9. Separated R G and B mages of the Mandr obect. reference cacuated from spectra cube measurements wth spectra fters; (b) CS-SSI reconstructed after 5 SBI apped to DD mage. Fg.. Separated R G and B mages of the Coor Checker obect. Reference cacuated from spectra cube measurements wth spectra fters; (b) CS-SSI reconstructed after 4 SBI apped to DD mage.
(b) (c) PSNR C 5 5 Mandr Coor Checker Peppers.4.4 7.6 5.8 6.4 5.6 7. 4.7. 5 (d) (e) (f) Fg. 3. Obect "Peppers". as dspaed on the Pad; (b) drect acqured b the reguar RGB sensor; (c) reference cacuated from spectra cube measurements wth spectra fters; (d)-(f) CS-SSI reconstructed RGB mages; (g) DD mage..8.6.4..8.6.4..8.6.4..6.4. Fg. 4. The R G and B vaues for the Peppers at 8 samped spata ocatons of Fg. 5(c). Bue-reference red - CS-SSI reconstructed. (g) () () (3) (4).8.6.4..8.6.4. (5) (6) (7) (8).8.8.8.6.4..6.4. Fg. 6. The RGB reconstructon errors for "Mandr" "Coor Checker" and "Peppers" obects. PSNRC vaues of the R G and B monochromatc mages of the "Mandr" "Coor Checker" and "Peppers" obects; (b)-(d) spata dstrbuton for PSNR for the same 3 obects. Lght Throughput [%] Lght Throughput [%] 8 6 4 8 6 4 89. 88.6 (b) (c) (d) Whte Pad Patch 7.4 3. Green Pad Patch 8.8 39.3 5. 5.7. (c) 36.9 47.7 Lght Throughput [%] Lght Throughput [%] 8 6 4 8 6 4 89.6 87.5 Red Pad Patch 7.8.8 (b) Bue Pad Patch (d).4 3. 8.7 3.7 3. Fg. 7. Lght throughput comparson resuts between our camera and reference CFA RGB camera based on raw PNG-format data. Pad obects were patches of dfferent coors whte (b) red (c) green and (d) bue. Insets show fragments of the raw mages from the RGB sensor. (b) Fg. 5. Separated R G and B mages of the Peppers obect. Reference cacuated from spectra cube measurements wth spectra fters; (b) CS-SSI reconstructed after SBI apped to DD mage.