A Smar Sensor wih Hyperspecral/Range Fovea and Panoramic Peripheral View Tao Wang,2, Zhigang Zhu,2 and Harvey Rhody 3 Deparmen of Compuer Science, The Ciy College of New York 38 h Sree and Conven Avenue, New York, NY 003 2 Deparmen of Compuer Science, The CUNY Graduae Cener 365 Fifh Avenue, New York, NY 006 Email: {zhu, wang}@cs.ccny.cuny.edu 3 Cener for Imaging Science, Rocheser Insiue of Technology 54 Lomb Memorial Drive, Rocheser, NY 4623-5604 Email: rhody@cis.ri.edu Absrac We propose an adapive and effecive mulimodal peripheral-fovea sensor design for real-ime arges racking. This design is inspired by he biological vision sysems for achieving real-ime arge deecion and recogniion wih a hyperspecral/range fovea and panoramic peripheral view. A realisic scene simulaion approach is used o evaluae our sensor design and he relaed daa exploiaion algorihms before a real sensor is made. The goal is o reduce developmen ime and sysem cos while achieving opimal resuls hrough an ieraive process ha incorporaes simulaion, sensing, processing and evaluaion. Imporan issues such as mulimodal sensory componen inegraion, region of ineres exracion, arge racking, hyperspecral image analysis and arge signaure idenificaion are discussed. Keywords Hyperspecral imaging, panoramic vision bio-inspired sensing, arge racking.. Inroducion Recenly, a grea deal of effor has been pu ino adapive and unable muli-specral or hyper-specral sensor designs wih goals o address he challenging problems of deecing, racking and idenifying arges in highly cluered, dynamic scenes. Represenaive large programs include: he DARPA s Adapive Focal Plane Array (AFPA) Program [], ARL s Advanced Sensor CTA [2], and NSF s Cener for Mid-Infrared Technologies for Healh and he Environmen [3]. Even commercial real-ime, specral unable, polaralizaion-enabled, hyperspecral imaging (HSI) sysems are available [4]. However, oday s HSI sysem are limied o scanning mode deecion, and are usually large, complex, power hungry and slow. The abiliy o perform HSI in a saring mode is criical o real-ime argeing mission. This brings up conflicing requiremens for real-ime search wih he abiliy o deec and idenify difficul and hidden arges using hyperspecral informaion, while saying wihin he processing ime and size available suied o small plaforms. In addiion, he limied field of view (FOV) of convenional sensor design does no saisfy he requiremens of large area search. We propose a novel single sensor design ha has boh a panoramic view and a hyperspecral/range fovea, and ha inegraes sensing and processing for real-ime arge deecion. I is an exension o he convenional foveaed-peripheral sysem: ) I has a panoramic view insead of jus a wide-angle view. 2) I is capable of 3D and hyperspecral daa acquisiion insead of jus high-resoluion color vision in he foveaed region. Therefore, his sensor design is inspired by he biological vision sysems of human eyes, bu i does no merely mimic human vision. I uses dual-panoramic scanners o deec moving arges and o aid acive conrol of he hyperspecral/range fovea o acquire needed signaures of arges a various locaions for achieving real-ime hyperspecral and range imaging. Anoher imporan feaure of he work is o use a realisic hyperspecral simulaion ool o es our sensor designs in various scenarios, which allows us o more effecively refine our sensor parameers. In addiion, i can furher help us o reduce he cos of fabricaion of a sensor design ha migh no ye mee he requiremens of real-world applicaions before he images capured by he sensor are processed for he applicaions. Using realisic scene simulaion and image generaion using he designed sensor parameers, various algorihms can be applied o he generaed images and hen he sensor design can be evaluaed and furher refined. In his paper, issues such as sensor designs, peripheral background modeling, and arge signaure acquisiion will be addressed in deail. This design and he relaed daa exploiaion algorihms - -
Fig.. The design concep of he DPSHRF. The dash lines indicae he single viewpoin of boh he foveal hyperspecral imager and he wo line scanners. will be simulaed and evaluaed in our general daa simulaion framework. The paper is organized as he following. Secion 2 shows he design of he bio-inspired adapive mulimodal sensor plaform - he dual panoramic scanners wih hyperspecral/range fovea (DPSHRF) for he ask of racking moving arges in real ime. Secion 3 describes he simulaion environmen for implemening our simulaion approach, and he parameer configuraion of he sensor plaform. Secion 4 presens he image exploiaion algorihms for deecing and racking moving arges, and specral classificaion in recognizing he moving objecs. Conclusions and discussions will be provided in Secion 5. 2. A Bio-Inspired Sensor Design The daa volumes in consideraion have wo spaial dimensions (X and Y), a specral dimension (S, from a few o several hundred), and a ime dimension (T). This four dimensional (4D) image in X-Y-S-T may be augmened by a 2D range image (in he XY space). Ideally, a sensor should have 360-degree full spherical coverage, wih high spaial and emporal resoluion, and a each pixel have full range of specral and range informaion. However, his ype of sensor is difficul o implemen because of he enormous amoun of daa ha mus be capured and ransmied, mos of which will evenually be discarded. Therefore, paricularly for real-ime applicaions, every collecion mus face fundamenal rade-offs such as spaial resoluion vs specral resoluion, collecion rae vs SNR, field-of-view vs coverage, o name a few examples. Undersanding he rade-offs and using algorihms ha can be adaped o changing requiremens can improve performance by enabling he collecion o be done wih maximum effeciveness for he curren ask. To break he dilemma beween FOV and spaial/specral resoluion, we invesigae a bio-inspired daa collecion sraegy, which can achieve real-ime imaging wih a hyperspecral/range fovea and panoramic peripheral view. This is an exension of he funcions of human eyes ha have high-resoluion color vision in he fovea and black-whie, low-resoluion arge deecion in he wide field-of-view peripheral vision. In our design, he fovea is enhanced by HSI and range informaion, and he peripheral vision is exended o panoramic FOV and has adapive specral response raher han jus black-whie. Our proposed sensor plaform, he dual-panoramic scanners wih a hyperspecral/range fovea (DPSHRF) (Fig. ), consiss of a dual-panoramic (omnidirecional) peripheral vision and a narrow FOV hyperspecral fovea wih a range finder. This inelligen sensor works as he follows. In he firs sep, wo panchromaic images wih 360-degree FOV are generaed by roaing wo line scanners around a common roaing axis, poining apar o wo slighly differen direcions. The angle difference beween he wo scanners can be adjused for deecing and racking moving arges wih differen velociies and disances. An iniial angle is used a he beginning. Then he deecing resuls from he wo scans can deermine wha he new angle difference should be - eiher decreased if a arge is moving oo fas, or increased if he arge is moving oo slow. There are wo advanages of using line scanners ha will be furher amplified. Firs, a line scanner can have a full 360-degree horizonal FOV. Second, resuled images are inherenly regisered. Moving arges can hen be easily and quickly deermined by he differences of he wo panoramic - 2 -
images generaed from wo scanners. The nex posiion and he ime of a moving arge can be esimaed from he difference of wo regions of ineres (ROIs) ha include he arge. In real-ime processing, he comparison is sared whenever he second scan reaches he posiion of he firs scan, herefore, only a small porion of panoramic images is used before full-view panoramas are generaed. The deail of he arge deecion processing algorihm will be discussed in Secion 4. Then, we can urn he hyperspecral/range fovea wih a specific focal lengh calculaed based on he size of he objec, and o he prediced region ha includes he moving arge. Thus, hyperspecral/range daa is recorded more efficienly for only he ROIs ha include possible moving arges. The wo line scanners and he hyperspecral/range imager are aligned so ha hey all share a single effecive viewpoin. The specral daa can be efficienly recorded wih a foveal hyperspecral imager (FHI) [6] which maps a 2D spaial image ino a spaial D image. This is implemened by using a micro mirror as a fovea ha inerceps he ligh ono a beam splier for generaing co-regisered range-hyperspecral images using a ranger finder and he FHI. The FHI consiss of a fiber opical reformaer (FOR) [7] forms a D array ono a dispersive hyperspecral imager (DHI) [8] which produces a 2D hyperspecral daa array wih one dimension as spaial and he oher as specral. The spaial resoluion of he FOR is deermined by he diameers of opical fibers which are conrolled during he opical design process. The blurring effec from cross-coupling of opical fibers is no significan magniude as shown in [9]. Finally, a co-regisered spaial-specral/range image is produced by combining wih he panchromaic images which are generaed by he dual-panoramic scanners. In summary, his sensor plaform improves or differs from previous designs [2, 6, 9] in lieraure in four aspecs: () A dual scanning sysem is designed o obain moving arges in a very effecive and efficien manner. A panoramic view is provided insead of a normal wide-angle view. (2) An inegraion of range and hyperspecral fovea componen is used for arge idenificaion. (3) The dual-panoramic scanners and he hyperspecral/range fovea are co-regisered. (4) Acive conrol of he hyperspecral sensor is added o faciliae signaure acquisiion of arges of various locaions ha can only be deermined in real-ime. 3. Scene Simulaion and Sensor Modeling The sensor design concep is esed hough he simulaion ool DIRSIG. Various broad-band, muli-specral and hyperspecral imagery are generaed hrough he inegraion of a suie of firs principles based radiaion propagaion sub-models [0]. Before performing scene simulaion and sensor modeling, we need o se up differen scenarios and configure he sensor parameers. One of he complex scenarios we consruced including 4 cars having exacly same shapes and 3 differen pains moving o differen direcions wih various speeds (Fig. 2). All four cars will pass hrough he cross secion a he boom corner of he main building in he scene a a cerain ime. Various behaviors of he moving vehicles such as simple moving, overaking, passing hrough, and ec., are moniored by our sensor plaform which is placed in fron of he main building. The scan speed of each line scanner can be se from 60 Hz o 00 Hz selecable, hus one enire 360 scan ake from 6.0 seconds down o 3.6 seconds. This ime consrain is no a problem for real-ime arge deecion since deecion and scanning are coninuous and simulaneous. The number of pixels per line in he verical direcion is se o 52 o mach he horizonal scanning resoluion. Few seleced specral bands are capured by dual line scanning. The focal lengh is fixed a 35mm for boh line scanners, and he angle beween he poining direcions of he wo scanners is 0 so ha he ime he second scan reaches he posiion of he firs scan is only abou 0.s. In heory, he ime difference beween wo scans should be much less han one second o avoid a lo of uncerainy of acion changes in moving vehicles. Two scanners are used so ha () he more accurae direcion and he focal lengh of he hyperspecral fovea can be esimaed; and (2) moving arge deecion can sill be performed when background subracion using a single scanner fails due o cluered background, muliple moving arges, and he ego-moion of he sensor plaform. The focal lengh of he hyperspecral imager is auomaically adjused according o he arge deecion resuls generaed from he wo line scanners. To simulae he hyperspecral imager, we use a frame array sensor wih small spaial resoluion a 70 x 70 for he hyperspecral daa, and he ground ruh range daa provided by DIRSIG are ransformed ino range images. The specral resoluion is 0.0 µm ranged from 0.4 µm o.0 µm. Differen porion of bandwidh can be seleced and deermined by analyzing he model specral profile. The simulaion will enable a close invesigaion of inelligen sensor designs and hyperspecral daa selecion and exploiaion for user designaed arges. The DIRSIG simulaion environmen allows us o use an ieraive approach o mulimodal sensor designs. Saring wih user and applicaion requiremens, various arges of ineres in differen, cluered background can be simulaed using he scene-arge simulaion ools in he DIRSIG. Then he adapive mulimodal sensor ha has been designed can be modeled using he sensor modeling ools wihin he DIRSIG, and mulimodal sensing daa (images) can be generaed. Targe deecion/idenificaion, background modeling and mulimodal fusion algorihms will be run on hese simulaed images o evaluae he overall - 3 -
4.. Deecion and racking in peripheral views Fig. 2. A simulaed urban scene image capured a laiude=43.0 and longiude=77.0, 000 meers above. The ellipses show he iniial sae of he four cars. The recangles show he sae where hose cars move afer 2s. The DPSHRF sensor (in blue do) is placed in fron of he main large building. The simulaed scene is capured a 8am in a ypical summer day. performance of he auomaed arge recogniion, and o invesigae he effeciveness of he iniial mulimodal sensor design. The evaluaions of he recogniion resuls agains he given ground-ruh daa (by simulaion) can provide furher indicaors for improving he iniial sensor design, for example, spaial resoluion, emporal sampling raes, specral band selecion, he role of range informaion and polarizaion, ec.. Finally, a refined sensor design can again be modeled wihin he DIRSIG o sar anoher ieraion of sensor and sysem evaluaion. 4. Daa Exploiaion and Adapive Sensing The basic procedure for acive arge deecion and racking is as follows. A few seleced specral bands are used o iniialize he deecion of arges eiher based on moion deecion or scene/arge properies in prior scenarios. Then, for he poenial ineresing arges, he fovea urns o each of hem o ge a high-resoluion, hyperspecral image wih range informaion. This can be done in real-ime so ha racking of one arge and swiching beween muliple candidaes is made possible. Finally, he signaures of he arges can be obained by auomaically analyzing he hyperspecral daa in he fovea and by selecing he mos relevan bands for such arges. This kind of funcion needs he acive conrol of he sensor o fuse he peripheral and fovea vision in an efficien manner. In he following, we elaborae he principle by using some commonly used algorihms in arge deecion, racking and idenificaion, using our bio-inspired mulimodal sensor. The firs sep is o find ROIs ha possibly conain moving arges (Fig 3). Simple background subracion beween a scanned image and a background image is no sufficien because he panoramic background (wih rees, building, ec.) may change due o illuminaion changes over a large span of ime. The advanage of using he wo consecuive scanners is he abiliy o quickly deec a moving arge in real ime using frame difference wihou producing oo much noise from he background. Furher, a morphological noise removal echnique [] is applied o remove small sparse noises wih he opening operaion and fill small holes wih he closing operaion. However, he resuls from frame difference canno provide accurae locaion and size informaion of he moving arges. Therefore, bounding boxes are defined from he frame difference resuls o mask off hose background regions for background subracion, which can provide more accurae locaion and size informaion of he moving arges. Fig. 3c shows some bounding boxes ha can be used as masks for performing he background subracion of each individual panoramic scan. The hreshold is se very low since we are ineresed in any changes in moion comparing o he relaive saic background. Of course, false alarms can also be generaed by evens such as he change of a large shadow, bu his can be verified once we capured he hyperspecral image. The background image is updaed for only hose pixels belonging o he background afer each 360-degree roaion, hus moving objec exracion is mainained over ime. A every roaion, each of he wo line scanners will generae a sequence of D image lines ha are combined o generae he panorama. Thus, regisraion problems can be avoided wih he sabilized line scanners. Real-ime arge deecion can be achieved since he scanning and deecion are performed simulaneously and coninuously. The nex sep is o esimae he region of he nex posiion ha may conain a arge once he wo ROIs of he same arge are found a wo differen imes resuling from wo differen scans (Fig. 3d). The locaion and size differences of he wo regions can deermine he relaive bearing angle of he hyperspecral/range fovea imager o zoom on he moving arge. The posiion of exraced region from he dual-scans indicaes which direcion he arge is moving o. Also, he size of wo regions can indicae wheher he arge is moving closer o he sensor or farher. Therefore, we can calculae he nex posiion where he arge will be. Then, he raio of he previous wo regions can be used o esimae he new focal lengh of he hyperspecral imager. The angle difference of wo scans for wo ROIs a differen imes i and, can be used o predic he posiion of he nex ROI having he moving arge a he ime, 2, when he hyperspecral/range imager can be in - 4 -
a) b) c) d) Fig. 3. All 360 panoramic images (52 x 3600) shown here are inegraed from verical scan lines capured by he dual-panoramic scanners. a). Panoramic image from he firs scanner, wih he moving arges indicaed inside red circles. b). Panoramic image from he second scanner, again he same moving arges indicaed inside green circles. c). frame difference beween a and b, group of ROIs are labeled. d). background subracion from wo scans inside boundaries defined by c. Red recangles showed ROIs from firs scan, blue recangles showed ROIs from second scan. (Close-up view of each labeled region can be seen clearly in Fig. 4). place. Therefore, given he ime 2, we can esimae he panning and ile angles of he hyperspecral/range imager. Noe ha only he angles relaive o he cener of a region are needed. The urning angles (i.e., panning and iling) of he hyperspecral/range imager should be: θ 2 = θ + 2 i ( θ θ where he superscrip x and y correspond o he panning angle (in he x-direcion) and he iling angle (in he y-direcion), respecively. The angle θ i corresponds o he angle posiion of a ROI a a ime i as shown in Fig. 3. The focal lengh of he hyperspecral/range fovea is inversely proporional o he desired FOV of he hyperspecral/range imager,α, in order o have he arge in he full view of he FOV. The FOV angle can be esimaed as R i 2 i ) () α = + l (2) P where R 2 is he prediced size of he arge region a l P is he number of scanning lines per radius. 2, and The relaionship beween R 2 and he previous wo regions of he same arge a differen imes can be expressed as R R i R 2 ( i ) = ( 2 ) R Then a hyperspecral foveal sho of a ROI from he calculaion can be aken. Thus, hyperspecral/range daa is recorded in a more efficien way, only for ROIs. I is possible for some regions o be idenified ha do no have rue moving arges inside. Then he hyperspecral classificaion in nex sep can verify his siuaion. 4.2. Targe classificaion using 3D and HSI fovea Targes can be classified based on hyperspecral measuremens, shape informaion, and he inegraion of boh. There has been a lo of work in recognizing objecs using 3D shape informaion [e.g., 2, 3]. Here we will only describe how o use a arge s deph informaion and (3) - 5 -
he informaion of is background o perform beer hyperspecral classificaion. Recognizing a arge needs o compare he arge s specrum associaed wih each pixel o is raining specrum. In our experimens, a specral library was pre-buil wih some exising models. Various vehicles wih differen colors and shapes can be impored and esed in he simulaion scene. In he paricular scenario in Fig. 2, four cars having he same shape bu differen pains are modeled. Two are red, one is brown and one is black. Iniial specral signaures of he four cars were capured from differen angles in he same background. The capuring angles and surroundings are imporan and need o be considered carefully because hose facors can significanly affec he effecive radiance reaching he sensor, L ( l, θ, φ, λ), where l is he slan range from sensor o arge, θ, φ and λ are he zenih angle, he azimuh angle and he wavelengh, respecively. The general expression for L is more complex and fully described in [4]. However, we can simplify L if we are only ineresed in he reflecive (visible) bands, he general equaion can be furher expressed as: ( L ( l, σ, λ), L ( θ, φ), L ( θ, φ, λ), L ( l, θ, )) L( l, θ, φ, λ) = f s ds bs us λ (4) where σ is he angle from he normal o he arge o he sun, L s is he solar radiance, L ds is he downwelled radiance from he sky due o he amospheric scaering, L bs is he specral radiance due o he reflecion from background objecs, and L us is he scaered amospheric pah radiance along he arge-sensor line of sie. In he raining sage, he background is known and fixed, hus L bs can be cancelled ou. The angles of he sun o he arge and he of arge o he sensor are known, hus we can keep his informaion and esimae a new specral profile of he model arge once we need o monior a new arge a a differen ime. L ds and L us can also affec he iniial specral profile if he weaher condiion changes significanly. In he curren experimens, we only use one amospheric daase which can also be replaced and changed in he simulaion in he fuure. Afer handling all reflecive varians, various endmembers ha represen he specral exremes ha bes characerize a maerial ype of a arge were seleced, and heir specral curves were sored in he specral library daabase. We used he sequenial maximum angle convex cone (SMACC) [5] o exrac specral endmembers and heir abundance for every model arge. In comparison o he convenional pixel puriy index (PPI) [6] and N-FINDER [7], SMACC is a much faser and more auomaed mehod for finding specral endmembers. Simply speaking, SMACC firs finds exreme poins or vecors ha canno be represened by a posiive linear combinaion of oher vecors in he daa as a convex cone, and hen a consrained oblique projecion is applied o he exising cone o derive he nex endmembers. The process is repeaed unil a olerance value is reached, for example, max number of endmembers. Each endmember specrum, defined as H, can be presened mahemaically as a combinaion of he produc of a convex 2D marix conains endmember specra as columns and a posiive coefficien marix: N H ( c, i) = R( c, k) A( k, j) (5) k where i is he pixel index, j and k are he endmember indices, and c is he specra channel index. Some endmembers migh have less specra differences in erm of redundancy. Those can be coalesced based on a hreshold so ha he mos exreme specra are idenified and used o represen he enire coalesced group of endmembers. In he esing sage, he same arge specra may be varied in differen condiions such as various surface orienaions and surroundings. However, he significan specral signaure of a arge can be esimaed and maybe furher correced wih he help of range informaion produced from a ranger finder. Knowing he angles of he sun and he sensor, he deph map (i.e. range daa) can indicae wheher he informaion of a background objec close o he arge should be couned when processing he arge specra. The resul specra will have similar shape bu he magniudes will be sill differen due o he variaions of illuminaion inensiies and direcions. A specral angle mapper (SAM) [8] algorihm is used o mach he arge specra o reference specra. The SAM is insensiive o illuminaion and albedo effecs. The algorihm deermines he specral similariy beween wo specra by calculaing he angle beween he specra and reaing hem as vecors in a space wih dimensionaliy equal o he number of bands [8]. Smaller angles represen closer mach. The deph informaion and he relaive locaion of he sun and he sensor can deermine wheher a arge specra should be adjused by he surrounding specra when performing classificaion. As a resul, each pixel is classified eiher o a known objec if he arge specrum is mached wih he library specrum of ha objec, or o an unknown objec, for insance, he background. To disinc muliple objecs from daabase, he resuls from differen group of endmembers of differen arges are compared. 4.3. Experimenal Resuls Fig. 4 shows he processed resuls for he following four cases: A) Muliple arges wih differen specral signaure. B) A arge is under a shadow cas by rees. C) There is no moving arge (hus a false alarm). D) Only one side of he arge specral signaure can be acquired and he oher side can no be deermined due o he insufficien reflecance of he sun ligh and he surroundings. Various soluions can be possible, for example: ) o place he sensor plaform a - 6 -
Index ROIs Fovea parameers Fovea Sho (in RGB) Sample Specral Profile Specral Curves Annoaions Deph Map SAM wihou deph SAM wih deph Resuls A Zenih: 89.0 Azimuh: 80.0 Focal Lengh: 245mm B Zenih: 88.5 Azimuh 9.0 Focal Lengh: 205mm Top Curve: Car on Righ Boom Curve: Car on Lef Top Curve: Car no in Shadow Boom Curve: Car in Shadow Top Curve: Maerial Boom Curve: Maerial 2 Top Curve: Fron body Boom Curve: Side body Red Car Brown Car Red Car C Zenih: 88.5 Azimuh 220.0 Focal Lengh = 225mm D Zenih: 88.0 Azimuh 330.0 Focal Lengh: 25mm False Targe Black Car Fig. 4. Processing Resul of he simulaed urban scene. Each index corresponds o each labeled region in Fig 2d. The column ROIs shows close-up view of resul indicaed in Fig 2d. Hyperspecral fovea shos demonsraed here wih only 3 RGB bands (which are also marked as verical lines in he sample specral profile column, in blue, green and red, respecively. Only he significan specral signaures of arges are shown here. Final mapping resuls are shown in binary only o indicae he arges and he background. The classificaion is based on he mach resul wih each model arge specral profile in daabase. anoher posiion; 2) o reconfigure sensor parameers such as adjus he heigh and he poining direcion; and 3) o implemen a beer classificaion algorihm. Therefore he experimenal resuls can quickly drive feedback o adjus and improve he sensor design and he algorihm implemenaions. Various scenarios and cases can be consruced and esed in he simulaion framework before a real sensor is even made. One of he useful advanages of he co-regisered hyperspecral and range imaging is o use he range informaion o improve he effeciveness of he hyperspecral measuremens. For example, in Fig. 4B, he shadowing of he vehicle (he red car) under he rees can be analyzed by he relaion among he locaion of he sun, he locaions of he rees from he panoramic background, and he surface orienaions of he vehicle. Considering he deph informaion, he SAM can be obained for surfaces of he vehicle under he influence of he ree shadows (herefore looks greenish). In Fig. 4D, he relaions beween he surfaces orienaions of he vehicle (he black car) and he locaion of he sun can also ell which surfaces are illuminaed. Therefore he well-illuminaed surfaces (i.e. he op of he car bod can be seleced based on he srucural informaion obained from he range daa. The analysis so far is very preliminary bu is very promising for fuure research. 5. Conclusions and Discussion We have described our bio-inspired mulimodal sensor design ha enables efficien hyperspecral daa collecion for racking moving arges in real-ime. The real-ime hyperspecral/range fovea imaging sysem furher exends he capabiliy of human fovea vision. We have also demonsraed a simulaion procedure o evaluae he new senor design, o es algorihms and o acively conrol he sensor for daa selecion. Consrucing a sophisicaed sensor like he one described in he paper is ime-consuming and cosly, and migh no even mee he applicaion requiremens afer being fabricaed. Therefore, we believe he simulaion procedure used in he paper is one soluion o horoughly evaluae a new sensor design in ha realisic images can be generaed in order o es various algorihms before he sensor has been made. By simulaion, various componens can be reconfigured or replaced for specific siuaions or asks. The image processing algorihms described in he paper are designed only o demonsrae he basic idea of effecively capuring hyperspecral daa in ROIs based on daa exploiaion. Needless o say, more sophisicaed algorihms need o be developed for more challenging asks. In addiion, we only described one specral classificaion mehod for recognizing he objec. More precise and efficien hyperspecral classificaion rouines may be applied. In he fuure, we will sudy wo aspecs of daa processing: range-specral inegraion and inelligen specral band selecion. Boh issues will be grealy faciliaed by our proposed approach wih advanced scene and sensor simulaion. Range-specral inegraion. There are many facors ha need o be considered in correcing he acquired hyperspecral daa o reveal he rue maerial reflecance, including source illuminaion, scene geomery, amospheric and sensor effecs, specral and space - 7 -
resoluion, and ec. In he low-aliude airborne or ground imaging cases, he scene geomery is probably he mos imporan facor. Therefore, he design of co-regisered hyperspecral and range fovea will provide boh specral and geomery measuremens of he 3D scene in a high resoluion, so ha a range-aided specral correcion can be performed. Using he DIRSIG ools, we have simulaed boh hyperspecral images and ranges images for several seleced arges wih known 3D models and specral properies, and he nex sep o derive algorihms o perform specral correcion by he more effecive 3D srucure informaion of he arges given by he range images and he background informaion given by he panoramic scanners. Opimal band selecion. Afer he analysis of he hyperspecral daa, he mos useful wavelenghs ha can capure he arge s signaures can be seleced via unable filering; and he ask of racking and arge recogniion will only need o use he few seleced bands or a few key feaures raher han all of he bands. This sudy will be carried ou in several scenarios involving differen arges in a challenging background or differen backgrounds. We will compare he hyperspecral profiles (i.e. 3D images wih wo spaial dimensions and a specral dimension) of various arges agains differen background maerials, and hen derive he opimal specral signaures o disinguish a arge from is background. We will also invesigae how he range informaion can be used in improving he effeciveness of signaure exracion and arge recogniion. The DIRSIG arge and scene simulaion ools could provide sufficien samples as raining examples for us o opimal hyperspecral band selecion. Acknowledgemens This work is suppored by AFOSR under he Discovery Challenge Truss (DCTs) Program, Award #FA9550-08--099, and in par by AFRL/SN under Award No. FA8650-05--853 and by NSF under Gran No. CNS-055598. References [3] K. Kincade, MIRTHE cener aims o ake mid-ir sensors o new heighs. (Opical Sensing), Laser Focus World, July, 2006. [4] POC, POC s Real-Time Hyperspecral Imaging, Available: hp://www.poc.com/, las visied July, 2008 [5] DIRSIG, hp://dirsig.cis.ri.edu/, las visied Ocober, 2008 [6] Flecher-Holmes, D. W. and A. R. Harvey, 2005. Real-ime imaging wih a hyperspecral fovea, J. Op. A: Pure Appl. Op. 7, 2005, pp. S298-S302 [7] Fibreopic Sysems Inc. 60 Moreland Rd, Uni A, Simi Valley, CA 93065, USA (hp://www.fibopsys.com/) [8] Headwall Phoonics, Inc., 60 River Sree, Fichburg, MA 0420, USA.(hp://www.headwallphoonics.com/) [9] A. R. Harvey, D W Flecher-Holmes, Imaging aparaus, GB Paen applicaion 025248.6, 2nd July 2002 [0] J. R. Scho, S. D. Brown, R. V. Raqueño, H. N. Gross, and G. Robinson, "An advanced synheic image generaion model and is applicaion o muli/hyperspecral algorihm developmen", Canadian Journal of Remoe Sensing, Vol. 25, No. 2, June 999 [] P. Soille, Morphological image analysis: principle and applicaions, Springer Press, 999 [2] A. Diplaros, T. Geves, I. Paras, Combining color and shape informaion for illuminaion-viewpoin invarian objec recogniion. Image Process, Vol., Issue, pp.-, Jan. 2006 [3] T. M. Sar and M. A. Fischler, Conex-based vision: recognizing objecs using informaion from boh 2D and 3D imagery, Paern Analysis and Machine Inelligence, IEEE Trans. Vol. 3, Issue 0, Oc 99 [4] J. R. Scho, Remoe Sensing: he image chain approach, 2nd ed. Oxford, NY: Oxford Universiy Press, 2007. [5] J. Gruninger, A. J. Rakowski and M. L. Hoke. The sequenial maximum angle convex cone (SMACC) endmember model". Proceedings SPIE, Algorihms for Mulispecral and Hyper-specral and Ulraspecral Imagery, Vol. 5425-, Orlando FL, April, 2004. [6] J. W. Boardman, F.A. Druse, and R.O. Green, Mapping arge signaures via parial unmixing of AVIRIS daa. Fifh Annual JPL Airborne Earh Science Workshop, Vol., AVIRIS Workshop, pp. 23-26, 995 [7] M. F. Winer, N-FINDR: an algorihm for fas auonomous specral end-member deerminaion in hyperspecral daa. Proceedings SPIE, Vol. 3753, pp. 266-275 [8] F. A. Kruse, e al. 993, "The specral image processing sysem (SIPS) - Ineracive visualizaion and analysis of imaging specromeer daa". Remoe Sensing of he Environmen, v. 44, p. 45-63. [] Su Horn, DARPA s Adapive Focal Plane Array (AFPA) Program, Available: hp://www.arpa.mil/mo/programs/afpa/index.hml [2] A.C. Goldberg, B. Sann, N. Gupa, 2003, Mulispecral, hyperspecral, and hree-dimensional imaging research a he U.S. Army research laboraory, Proceedings of he Sixh Inernaional Conference of Informaion Fusion, 2003. Vol., 2003, pp: 499-506 - 8 -