Image analyss usng modulated lght sources Feng Xao a*, Jeffrey M. DCarlo b, Peter B. Catrysse b, Bran A. Wandell a a Dept. of Psychology, Stanford Unversty, CA 9435, USA b Dept. of Electrcal Engneerng, Stanford Unversty, CA 9435, USA ABSTRACT Wth the development of hgh-speed CMOS magers, t s possble to acqure and process multple mages wthn the mager, pror to output. We refer to an magng archtecture that acqures a collecton of mages and produces a sngle result as multple capture sngle mage (MCSI). In ths paper we descrbe some applcatons of the MCSI archtecture usng a monochrome sensor and modulatng lght sources. By usng actve (modulatng) lght sources, t s possble to measure object nformaton n a manner that s ndependent of the passve (ambent) llumnant. To study ths archtecture, we have mplemented a test system usng a monochrome CMOS sensor and several arrays of color LEDs whose temporal modulaton can be precsely controlled. Frst, we report on expermental measurements that evaluate how well the actve and passve llumnant can be separated as a functon of expermental varables, ncludng passve llumnant ntensty, temporal samplng rate and modulaton ampltude. Second, we descrbe two applcatons of ths technque: (a) creatng a color mage from a monochrome sensor, and (b) measurng the spatal dstrbuton of the passve llumnant. Keywords: Actve magng, dgtal camera, llumnant estmaton, color magng. INTRODUCTION In bologcal systems, percepton s often lnked to acton []. For example, the human perceptual system appears to contan sensory pathways that are coupled to specfc motor goals [2]. The actve aspects of human behavor nclude changes n head and eye poston, or adjustments of the senstvty of the pupl, lens, and vsual senstvty n response to changes n the ambent llumnaton. Bats emt sgnals that help them extend the capabltes of ther audtory system. We call an magng method that mproves mage understandng through the use of sensor moton or the emsson of specfc sgnals an actve magng method (AIM). In engneerng systems, many methods can serve as actve ads to sensory performance. It s possble, for example, to ntroduce a varety of dfferent sgnals (e.g., sonar and radar) nto the envronment to mprove sensory measurement. Actve magng methods that add lght to a scene can also assst object recognton or detecton [3, 4]. An nterestng example s the use of spatally modulated llumnaton to reduce false matches n stereo magng for depth recovery [5, 6]. Recently, Ando s group [7, 8] has descrbed experments wth a specalzed monochrome CMOS sensor that uses tme-doman correlaton coupled wth monochrome llumnant modulaton. In ths actve magng method, the sensor performs multple captures and then calculates a sngle mage that represents the scene as f t were llumnated only by the known modulatng source. We have decded to explore the applcaton of actve magng methods wth modulatng lght sources to color magng. Varous color magng applcatons can beneft from acqurng mage data under a known llumnaton source, and ths can be acheved through such actve magng methods. Here, we descrbe an expermental apparatus we have bult to nvestgate these applcatons. We also descrbe two examples that llustrate the applcablty of the method. In one applcaton we use several modulatng lght sources to derve a color mage from a monochrome sensor. In another applcaton we use modulatng lght sources to estmate the spatal dstrbuton of the ambent llumnaton. 2. ACTIVE IMAGING THEORY In ths secton we develop notaton to characterze actve magng method appled to color magng. The analyses are developed for the case of a temporally modulated llumnaton source that s ncdent on a small, unform surface patch. We show how AIM can be used to derve scene propertes from the tme-doman correlaton of multple mages. * For a color verson of ths paper please go to ftp://whte.stanford.edu/users/bran/pdc/spemodulate.pdf Correspondence: Emal: fxao@stanford.edu; Telephone: 65 725 255; Fax: 65 723 993
Consder a small surface patch located at poston x. Suppose the total llumnaton at locaton x s comprsed of two parts: an actve llumnant, A(x,λ,t), that s under our control, and a passve (ambent) llumnaton, P(x,λ,t), that s naturally present n the envronment. Further suppose that the actve llumnant may be comprsed of the sum of multple actve llumnants A (x,λ,t). Hence, the total llumnaton at a locaton x s gven by E(, x λ,) t = P(, x λ,) t + A(, x λ,) t = Px (, λ, t) + A( x, λ, t) N = = P( x, λ, t) + A ( x, λ) + A( x, λ, t) N = () The A term s the mean level of the sum of the actve llumnants; ths mean level s constant over tme. The A terms are modulatons of the actve llumnant about ths mean level. In most cases, the passve llumnaton does not change over tme, so that P(x,λ,t)=P(x,λ). Consequently, we may smplfy the equaton to be E( x, λ, t) ( P( x, λ) A ( x, λ)) A( x, λ, t) = + + (2) In most applcatons t s desrable to arrange the modulaton of the actve llumnant so that ts relatve spectral composton s unchanged as the ntensty modulates. Ths results n the further smplfcaton of the modulatng term, N = E( x, λ, t) ( P( x, λ) A ( x, λ)) A( x, λ) F( t) = + + (3) N = We call the temporal functons F (t) the modulaton control functons. The system desgner chooses these functons, and t smplfes subsequent calculatons when they are desgned to form an orthogonal set. Moreover, because we have separated out the mean, all of these functons have a zero mean: F ( t) F j ( t) dt =, j & F ( t) dt = δ (4) Next consder the response of an deal camera wth spectral senstvty R(λ) measurng lght scattered from a surface patch wth reflectance S(x,λ). The responses of the camera sensors over tme wll vary accordng to rxt (, ) = R( λ) Ex (, λ, tsx ) (, λ) dλ (5) We can demodulate the temporal sgnal of the camera measurements, r(x,t), to calculate a sgnal assocated wth each of the dfferent modulated lght sources. Namely, to calculate the th demodulated mage we calculate the nner product of the camera response and the th modulaton control functon. r( x) = r( x, t) F( t) dt = = R( λ) S( x, λ) E( x, λ, t) F( t) dt R( λ) S( x, λ) A( x, λ) dλ (6) Hence, the th -demodulated mage depends only on lght emtted by one of the actve llumnants; the mage s ndependent of the passve llumnaton and other actve llumnants.
3. EXPERIMENTS 3.. System setup The basc theory of usng AIM to modulate lght sources s straghtforward. But n practce, many mplementaton detals and nose may lmt ts accuracy. Thus, we have developed an expermental apparatus to explore the advantages and lmtatons of AIM for color magng. Several key elements of the laboratory are shown n Fgure. The dgtal mager s a customdesgned /3 CMOS monochrome dgtal pxel sensor [9] that s controlled by an FPGA (Actel). The scene s maged through a Computar TV zoom lens wth 8-6 mm focal length. The modulated lght sources are provded by three LED arrays. Each array s composed of 5x7 dentcal LEDs whose ntensty s controlled usng a Natonal Instruments AT-AO- DAQ board. Wth ths board, we can create any waveform that we want for the LED arrays. A LCD projector (Sony, Model VPL-X) provdes a passve llumnaton whose ntensty and spatal dstrbuton can be programmed. A Macbeth chart and other smple targets are used n the experments. All of the expermental devces are controlled by a hgh level nterface wrtten n Matlab. Fgure : Apparatus used to perform AIM experments. Images were captured usng a dgtal pxel sensor (DPS) wth a large aperture lens. The capture propertes of the mage sensor (ntegraton tme, samplng rate) were controlled by a FPGA usng a smple assembler language. LED arrays provded the actve llumnants. A calbrated LCD projector provded the passve llumnaton. Image acquston and llumnaton controls were managed from a computer that contaned a DAC, frame-grabber, and Matlab software. The spectral propertes of varous system elements are shown n Fgure 2. These spectral functons were measured usng narrowband lght sources provded an Orel monochromator and a PhotoResearch PR-65 spectral radometer. The spectral responsvty of the CMOS sensor array s shown n panel (a). The spectral power dstrbutons of the three types of LED arrays are shown n panel (b). 3.2. Independence of passve llumnant ntensty In the deal case, the response to the actve llumnant should not be affected by the presence of a steady passve lght source (Equaton 6). How well does ths property hold for a real camera system that ncludes nose, quantzaton, and other lmtatons? To answer ths queston, we estmated the ntensty of a fxed actve llumnant modulaton as we vared the passve llumnant ntensty. We used the blue LED array as the actve llumnant (though the results should be ndependent of the choce of LED array) and the LCD projector to vary the passve lght ntensty. We measured lght reflected from a whte
(a)..8 (b). red green blue Spectral responsvty.6.4 Relatve power.6.2.2 4 5 6 7 W avelength (n m ) 4 5 6 7 W avelength (n m ) Fgure 2: Characterzaton of the apparatus. (a) The spectral responsvty of the CMOS sensor. (b) The spectral power dstrbutons of the LEDs. Mean ampltude (dgtal value) 7 5 3 4 8 2 6 Passve lght ntensty (dgtal value) Fgure 3: Response to actve llumnant as the passve llumnant ntensty ncreases. The blue LED array modulated at Hz wth a 4 dgtal counts peak-to-peak swng. The horzontal axs measures the passve llumnant ntensty (dgtal counts from the lnear sensor). The vertcal axs measures the ampltude of response to actve llumnant. Dash lne shows the expected response and sold lne shows measured response. Error bar shows the standard error of measurement patch on the Macbeth chart (9x9 pxels) that was llumnated by both the passve and actve llumnants. At each passve ntensty level, the blue LED array modulated at Hz n snusod waveform; we acqured 8 pctures wth an exposure tme
( T) of 5 ms and an nter-sample nterval ( I) of 75 ms. The 8 sample measurements were demodulated to estmate the response to the blue LED array alone. Fgure 3 shows the ampltude of the estmated blue LED array modulaton at ten dfferent passve llumnant ntensty levels. Although the passve lght ntensty vared from 3 to 25 tmes the ntensty of actve llumnant, the estmated value remaned relatvely constant. The standard error of the ampltude estmate from the 8 pxels n the patch was.6 dgtal counts. 3.3. Effects of sample number Increasng the exposure tme or ncreasng the number of samples per cycle both mprove the accuracy of the estmate. Yet, wthn a fxed measurement perod these two parameters oppose one another: longer exposure tmes mply fewer samples per cycle. Hence, we measured how the tradeoff between sample number and exposure tme nfluences the accuracy of the estmated actve llumnant. The second experment analyzes ths tradeoff for our partcular measurement system. In ths experment, we agan modulated the blue LED array at Hz, but ths tme we kept a constant level of passve llumnaton. In our frst measurement, we ncreased the number of samples by fxng the nterval between exposures ( I) and decreasng the exposure tme ( T). Fgure 4a shows that ncreasng the number of samples ths way ncreased the estmaton error. Hence, for ths magng system t s more mportant to have long exposure duratons than multple samples. Fgure 4b shows that fxng the exposure tme but decreasng the nterval between exposures, mproves performance, as expected. (a) (b).5 Estmaton error (dgtal value).4.3.2. Estmaton error (dgtal value).4.3.2. 4 8 2 6 2 Sample number ( T gets sm aller) 5 5 2 Sample number ( I gets smaller) Fgure 4: The trade-off between number of samples and exposure duraton. The horzontal axs measures the number of samples measured. The vertcal axs represents the standard error of the estmated modulaton (n dgtal counts). In panel (a) more samples were obtaned by decreasng the exposure duraton. In panel (b) more samples were obtaned by varyng the tme nterval between the mage exposures. For ths magng system, mprovng the mage by extendng the exposure duraton was more valuable than ncreasng the number of ndependent samples of the modulatng actve llumnant. 3.4. Effect of the ntensty of modulated lght sources Fnally, we estmated the performance of the system n response to varatons of the actve llumnant ampltude. Usng the blue LED array (Hz), we vared the llumnant ampltude from a peak-to-peak near zero to a peak-to-peak of 5. Images were acqured from a 9x9 patch of the whte Macbeth surface. The ampltude and standard error were estmated usng all 8 pxels. The thrd experment nvestgates how the standard error vares as a functon of modulaton ampltude. The smooth curve n Fgure 5 shows the measured error; t s large for modulatons less than 5 dgtal counts and then remans roughly constant for modulatons above 2. The dashed curve n Fgure 5 shows the expected standard error based on smulatons that take nto account the quantzaton error of the mager.
5 Measurements Estmaton error (%) 5 Smulatons 2 3 4 5 Actve lght ntensty (peak to-peak; dgtal value) Fgure 5: Estmaton error for the modulated lght source. The horzontal axs measures the blue LED ntensty (dgtal counts from the lnear sensor). The vertcal axs represents the standard error of the estmated modulaton (n percentage). The smooth and dashed curves show expermental and smulated values. 3.5 Selectng the modulaton control functons All modulatng control functons are not equvalent. In the absence of nose and system consderatons, orthogonal control functons wth equal energy wll lead to the same system performance; but for real systems other factors may nfluence the choce of functons. These factors nclude the need to synchronze the actve llumnants and acquston devce, hardware capabltes, scene characterstcs, and system nose and quantzaton characterstcs. For example, some control functons requre the modulator and demodulator to be synchronzed. If the llumnant controller and camera are not closely coupled, ths synchronzaton may be dffcult to acheve. There are some control functons, however, that do not requre accurate synchronzaton. Snusodal functons, for example, can be demodulated wthout precse synchronzaton between the llumnant and the camera. Ths s not the case for other control functons, such as square wave functons (Haar bass). For these control functons the emtter and camera should be synchronzed to demodulate the sgnal properly by avodng acqustons durng the on/off transton. Scene characterstcs are another mportant factor to consder when selectng control functons. Scene ambent llumnants are usually ether steady or flckerng at 6 or 2 Hz, so t s mportant not to pck a bases set that contan energy n the same frequency range as the scene llumnants. By avodng the frequency range of typcal scene llumnants, the demodulator can acheve hgher performance because there s less nose n the frequency band that t s detectng and demodulatng. Lastly, the nose propertes of the entre system should be consdered when selectng modulatng control functons. Suppose that the man system nose s due to quantzaton, and all other sources of nose are neglgble. In ths case snusodal bases wll outperform square wave bases, and ths wll be especally true f many samples can be obtaned. The reason for ths advantage s that multple samples from the square wave bass set do not reduce the system quantzaton error. However when a snusodal bases set s used, the detected sgnal occupes many dfferent levels and the quantzaton nose s more evenly dstrbuted around zero. Ths error s added together and averaged away durng the demodulaton process; the negatve error values more evenly match the postve error values and these opposng errors tend to cancel. If other nose sources are the prncpal lmt, then other control functons may be preferred.
4. APPLICATIONS The ablty to measure an mage wth a controlled actve llumnant, even n the presence of a passve llumnant, can be used n a varety of novel mage acquston algorthms. We close ths paper by descrbng brefly two applcatons. 4. Color magng wth a monochrome sensor From the earlest days of electronc color magng, engneers have produced and acqured mages usng a feld sequental method. The frst televson dsplays used a rotatng color wheel to produce a sequence of red, green and blue mages. The vsual pathways ntegrate these mages to form the mpresson of a sngle color mage. Smlarly, t s possble to acqure a color mage usng a monochrome sensor by sequentally placng color flters n the lght path. Agan, the three mage sequences can be combned to produce a sngle color mage. Feld sequental color has some advantages and many dsadvantages compared to capture usng an mage sensor wth a mosac []. One of the most sgnfcant dsadvantages s that the method s only approprate for a statc scene; object moton between the capture tmes produces very strong color artfacts. Usng actve llumnants, t s possble to modfy the feld sequental approach to create color mages from a monochrome sensor. Suppose an actve llumnant comprses three lght sources, say red, green and blue sources. If the source control functons are orthogonal, then t s possble to demodulate the acqured tme seres nto three mages correspondng to the scene llumnated by the red, green, and blue sources. These three mages, acqured smultaneously, can be combned to form a sngle color mage. Fgure 6a shows the tme seres of an mage pxel measurng a whte surface whle the three llumnants were modulated. The red, green and blue llumnants were modulated at dfferent frequences and ther sgnatures can be seen n the ampltude spectrum of the tme seres shown n the panel on the rght of Fgure 6a. Ths spectrum has energy at three peaks frequences 2 7 5 (b) Fourer Transform 9 2 8 Tme (frame) 6 Ampltude Dgtal value (a) 6 2 2 6 Frequency R G B RGB Fgure 6. A color mage measured wth a monochrome mage sensor and three actve llumnants. Red, green and blue actve llumnants were modulated at three dfferent frequences. (a) The tme seres at a sngle pxel measured from a whte surface s shown on the left, and the ampltude spectrum of the tme seres s shown on the rght. Three frequences, correspondng to the modulatons of the three actve llumnants, contan sgnfcant energy. (b) The mages obtaned by demodulatng at the R, G and B llumnant frequences along wth the combned color mage from the three modulatons (RGB) are shown.
correspondng to the modulatons of the red, green and blue lght sources. Fgure 6b shows the three separate mages reconstructed by demodulatng the mage seres at every pxel as well as the fnal color mage assembled from the three color mages. It s possble to use ths feld smultaneous method wth many dfferent sets of modulaton control functons that are suted to specfc applcatons. 4.2 Estmatng the spatal dstrbuton of the passve llumnant In many applcatons, t s useful to dstngush between mage edges caused by surface markngs and those caused by llumnaton dscontnutes, such as shadows. In certan crcumstances, actve llumnants can be used to separate these two knds of mage edges. As an example, suppose that the sensor responds to a narrow wavelength band near λ. Further suppose that the actve llumnant s spatally constant across the mage regon. Then, we can express the spatal response of the camera mage to the actve llumnant, as r ( A x ) = A ( λ, x ) R ( λ ) S ( λ, x ) dλ A ( λ ) ( ) (, ) R λ S λ x (7) Smlarly, the spatal sensor response to the passve llumnant s r ( P x ) = P ( λ, x ) R ( λ ) S ( λ, x ) dλ P ( λ, ) ( ) (, ) x R λ S λ x (8) The rato of these two mages across space s proportonal to the spatal varaton of the passve llumnant, P( λ, x) rp ( x) (9) A( λ ) r ( x) A Hence, usng the measurements of the actve and passve llumnants we can estmate the spatal varaton of the passve llumnant. Image edges that are caused by spatal varatons n the llumnaton can be dstngushed from those due to a surface markng. The process s llustrated by the measurements n Fgure 7. Panel (a) shows an mage of a patterned surface that s llumnated by a passve source. The llumnaton s uneven, causng a strong shadow edge. The dashed lne drawn on top of the fgure shows the ntensty (dgtal value) along one of the scan lnes. Panel (b) shows an estmate of the spatal structure of the passve llumnaton. Dvdng the actve and passve llumnant mages as descrbed n Equaton 9 derves ths estmate. The dotted lne supermposed on the fgure shows the estmated passve llumnant ntensty. Panel (c) shows the measured spatal pattern of the llumnant obtaned usng a unform gray card nstead of the patterned surface. Panel (d) compares the scan lne n the orgnal mage wth the estmated and measured passve llumnant ntensty. In reflectve regons of the mage, the agreement s good. Systematc errors can be noted n regons when the surface s very dark. 5. DISCUSSION AND CONCLUSIONS We have mplemented and evaluated an actve magng method for color magng. We found that t s possble to measure mages from multple modulatng lght sources, and that these mages correspond to measurng the scene under one of the actve llumnants. The computaton of such mages s well suted to the MCSI archtecture n whch a collecton of mages s captured and processed wthn a sensor, but only a reduced number of mages are produced. Usng our expermental apparatus, we have measured how accurately the modulatng and ambent llumnaton can be separated as a functon of expermental varables, ncludng LED ntensty, temporal samplng rate and modulaton ampltude. Fnally, we have demonstrated how ths technque can be used to create a color mage from a monochrome sensor, and how the technque can be appled to dstngushng mage edges caused by surface markngs from those caused by llumnaton edges.
(a) Orgnal mage (b) Estmated llumnant (c) Measured llumnant (d) Measured Estmated Intensty estmate.6.2 Orgnal 2 3 Pxel poston Fgure 7: An actve magng method can be used to dscrmnate surface markngs and shadows. See text for detals. ACKNOWLEDGMENTS Ths work was supported by the Programmable Dgtal Camera project at Stanford Unversty (http://smartcamera.stanford.edu/pdc.html). Feng Xao acknowledges the support from Stanford Graduate Fellowshp program. We also thank K. N. Salama for helpng wth the control of LED arrays. REFERENCES. Ballard, D., Introducton to Natural Computaton. 997: MIT Press. 2. Goodale, M.A. and G.K. Humphrey, The objects of acton and percepton. Cognton, 998. 67(-2): p. 8-27. 3. Holmes, R.B., Applcatons of lasers to magng of dstant objects. SPIE Proceedngs, 993. 87: p. 36-35. 4. Mormoto, C.H. and M. Flckner. Real-Tme Multple Face Detecton Usng Actve Illumnant. n Fourth IEEE Internatonal Conference on Automatc Face and Gesture Recognton. 2. 5. McDonald, J.P., R.J. Fryer, and J.P. Sebert, Stereo scene codng usng SLM actve llumnant, n Proc. 26th Int. Symp. on Automotve Technology and Automaton Mechatroncs. 993: Aachen. p. 69-76. 6. Kang, S., et al. A Multbaselne Stereo System wth Actve Illumnant and Real-tme Image Acquston. n The Ffth Internatonal Conference on Computer Vson. 995. 7. Ando, S., K. Nakamura, and T. Sakaguch. Ultrafast Correlaton Image Sensor: Concept, Desgn, and Applcatons,. n Proc. IEEE CCD/AIS Workshop. 997. Bruges, Belgum: IEEE. 8. Ando, S. and A. Kmach. Tme-Doman Correlaton Image Sensor: Frst CMOS Realzaton of Demodulator Pxels Array. n Proc. '99 IEEE CCD/AIS Workshop. 999. Karuzawa. 9. Yang, D.X.D., et al. A 64*52 CMOS mage sensor wth ultra wde dynamc range floatng-pont pxellevel ADC. n 999 IEEE Internatonal Sold-State Crcuts Conference. 999. San Francsco, CA, USA.. Catrysse, P., A. El Gamal, and B.A. Wandell. Comparatve analyss of color archtectures for mage sensors. n Proc. of the SPIE. 999. San Jose.