Knowledge Transfer in Semi-automatic Image Interpretation

Transcription

1 Knowledge Transfer in Semi-auomaic Image Inerpreaion Jun Zhou 1, Li Cheng 2, Terry Caelli 23, and Waler F. Bischof 1 1 Deparmen of Compuing Science, Universiy of Albera, Edmonon, Albera, Canada T6G 2E8 {jzhou, wfb}@cs.ualbera.ca 2 Canberra Laboraory, Naional ICT Ausralia, Locked Bag 8001, Canberra ACT 2601, Ausralia {li.cheng, erry.caelli}@nica.com.au 3 School of Informaion Science and Engineering, Ausralian Naional Universiy, Bldg.115, Canberra ACT 0200, Ausralia Absrac. Semi-auomaic image inerpreaion sysems uilize ineracions beween users and compuers o adap and updae inerpreaion algorihms. We have sudied he influence of human inpus on image inerpreaion by examining several knowledge ransfer models. Experimenal resuls show ha he qualiy of he sysem performance depended no only on he knowledge ransfer paerns bu also on he user inpu, indicaing how imporan i is o develop user-adaped image inerpreaion sysems. Keywords: knowledge ransfer, image inerpreaion, road racking, human influence, performance evaluaion. 1 Inroducion I is widely acceped ha semi-auomaic mehods are necessary for robus image inerpreaion [1]. For his reason, we are ineresed in modelling he influence of human inpu on he qualiy of image inerpreaion. Such modelling is imporan because users have differen working paerns ha may affec he behavior of compuaional algorihms [2]. This involves hree componens: firs, how o represen human inpus in a way ha compuers can undersand; second, how o process he inpus in compuaional algorihms; and hird, how o evaluae he qualiy of human inpus. In his paper, we propose a framework ha deals wih hese hree aspecs and focus on a real world applicaion of updaing road maps using aerial images.

2 2 Road Annoaion in Aerial Images Updaing of road daa is imporan in map revision and for ensuring ha spaial daa in GIS daabases remain up o dae. This requires normally an inerpreaion of maps where aerial images are used as he source of updae. In real-world map revision environmens, for example he sofware environmen used a he Unied Sae Geological Survey, manual road annoaion is mouse- or command-driven. A simple road drawing operaion can be implemened by eiher clicking a ool icon on he ool bar followed by clicking on maps using a mouse, or by enering a key-in command. The ool icons correspond o road classes and view-change operaions, and he mouse clicks correspond o he road axis poins, view change locaions, or a rese ha ends a road annoaion. These inpus represen wo sages of human image inerpreaion, he deecion of linear feaures and he digiizing of hese feaures. We have developed an inerface o rack such user inpus. A parser is used o segmen he human inpus ino acion sequences and o exrac he ime and locaions of road axis poin inpus. These ime-samped poins are used as inpu o a semiauomaic sysem for road racking. During he racking, he compuer ineracs wih he user, keeping he human a he cener of conrol. A summary of he sysem is described in he nex secion. 3 Semi-auomaic Road Tracking Sysem The purpose of semi-auomaic road racking is o relieve he user from some of he image inerpreaion asks. The compuer is rained o perform road feaure racking as consisen wih expers as possible. Road racking sars from an iniially human provided road segmen indicaing he road axis posiion. The compuer learns relevan road informaion, such as range of locaion, direcion, road profiles, and sep size for he segmen. On reques, he compuer coninues wih racking using a road axis predicor, such as a paricle filer or a novely deecor [3], [4]. Observaions are exraced a each racked locaion and are compared wih he knowledge learned from he human operaor. During racking, he compuer coninuously updaes road knowledge from observing human racking while, a he same ime, evaluaing he racking resuls. When i deecs a possible problem or a racking failure, i gives conrol back o human, who hen eners anoher road segmen o guide he racker. Human inpu affecs he racker in hree ways. Firs, he inpu affecs he parameers of he road racker. When he racker is implemened as a road axis predicor, he parameers define he iniial sae of he sysem ha corresponds o he locaion of road axis, he direcion of road and he curvaure change. Second, he inpu represens he user's inerpreaion of a road siuaion, including dynamic properies of he road such as radiomeric changes caused by differen road maerials, and changes in road appearance caused by background objecs such as cars, shadows, and rees. The accumulaion of hese inerpreaions in a daabase consiues a human-ocompuer knowledge ransfer. Third, human inpu keeps he human a he cener of he conrol. When he compuer fails racking, new inpu can be used o se he correc

3 he racking direcion. The new inpu also permis promp and reliable correcion of he racker's sae model. graylevel graylevel pixel pixel Horozonal vericle Fig. 1. Profiles of a road segmen. In he lef image, wo whie dos indicae he saring and ending poins of road segmen inpu by human. The righ graphs shows he road profiles perpendicular o (upper) and along (lower) he road direcion.. 4 Human Inpu Processing The represenaion and processing of human inpu deermines how he inpu is used and how i affecs he behavior of image inerpreer. 4.1 Knowledge Represenaion Typically, a road is long, smooh, homogenous, and i has parallel edges. However, he siuaion is far more complex and ambiguous in real images, and his is why compuer vision sysems ofen fail. In conras, humans have a superb abiliy o inerpre hese complexiies and ambiguiies. Human inpu o he sysem embeds such inerpreaion and knowledge on road dynamics. The road profile is one way o quanize such inerpreaion in he feaure exracion sep [5]. The profile is normally defined as a vecor ha characerizes he image greylevel in cerain direcions. For road racking applicaions, he road profile perpendicular o he road direcion is imporan: Image greylevel values change dramaically a he road edges and he disance beween hese edges is normally consan. Thus, he road axis can be calculaed as he mid-poins beween he road edges. The profile along he road is also useful because he greylevel value varies very lile along he road direcion, whereas his is no he case in off-road areas. Whenever we obain a road segmen enered by he user, he road profile is exraced a each road axis poin. The profile is exraced in boh direcions and combined ino a vecor (shown in Fig. 1). Boh he individual vecor a each road axis

4 poin and an average vecor for he whole inpu road segmen are calculaed and sored in a knowledge base. They characerize a road siuaion ha human has recognized. These vecors form he emplae profiles ha he compuer uses when observaion profile is exraced during road racking. 4.2 Knowledge Transfer Depending on wheher machine learning is involved in creaing a road axis poin predicor, here are wo mehods o implemen he human-o-compuer knowledge ransfer using he creaed knowledge base. The firs mehod is o selec a se of road profiles from he knowledge base so ha a road racker can compare o during he auomaic racking. An example is he Bayesian filering model for road racking [4]. A each prediced axis poin, he racker exracs an observaion vecor ha conains wo direcional profiles. This observaion is compared o emplae profiles in knowledge base for a maching. Successful maching means ha he predicion is correc, and racking coninues. Oherwise, he user ges involved and provides new inpu. The second mehod is o learn a road profile predicor from sored road profiles in he knowledge base, for example, o consruc profile predicors as one-class suppor vecor machines [6]. Each predicor is represened as a weighed combinaion of raining profiles obained from human inpus in he Reproducing Kernel Hilber space, where pas raining samples in he learning session are associaed wih differen weighs wih proper ime decay. Boh knowledge ransfer models are highly dependen on he knowledge obained from he human. Direc uilizing of human inpus is risky because low qualiy inpus lower he performance of he sysem. This is especially he case when profile selecion model wihou machine learning is used. We propose ha human inpus can be processed in wo ways. Firs, similar emplae profiles may be obained from differen human inpus. The knowledge base hen expands quickly wih redundan informaion, making profile maching inefficien. Thus, new inpus should be evaluaed before being added ino he knowledge base, and only profiles ha are quie differen should be acceped. Second, he human inpu may conain poins of occlusions, for example when a car is in a scene. This generaes noisy emplae profile. On he one hand, such profiles deviae from he dominan road siuaion. Oher he oher hand, hey expand he knowledge based wih barely useful profiles. To solve his problem, we remove hose poins whose profile has a low correlaion wih he average profile of he road segmen. 5 Human Inpu Analysis 5.1 Daa Collecion Eigh paricipans were required o annoae roads by mouse in an sofware environmen ha displays he aerial phoos on he screen. None of he users was

5 experienced in using he sofware and he road annoaion ask. The annoaion was performed by selecing road drawing ools, followed by mouse clicks on he perceived road axis poins in he image. Before performing he daa collecion, each user was given 20 o 30 minues o become familiar wih he sofware environmen and o learn he operaions for file inpu/oupu, road annoaion, viewing change, and error correcion. They did so by working on an aerial image for he Lake Jackson area in Florida. When hey fel confiden in using he ools, hey were assigned 28 asks o annoae roads for he Mariea area in Florida. The users were old ha road ploing should be as accurae as possible, i.e. he mouse clicks should be on he rue road axis poins. Thus, he user had o decide how close he image should be zoomed in o idenify he rue road axis. Furhermore, he road had o be smooh, i.e. abrup changes in direcions should be avoided and no zigzags should occur. The ploing asks included a variey of scenes in he aerial phoo of Mariea area, such as rans-naional highways, inra-sae highways and roads for local ransporaion. These asks conained differen road ypes such as sraigh roads, curves, ramps, crossings, and bridges. They also included various road condiions including occlusions by vehicles, rees, or shadows. 5.2 Daa Analysis We obained eigh daa ses, each conaining 28 sequences of road axis coordinaes racked by users. Such daa was used o iniialize he paricle filers, o regain conrol when road racker had failed, and o correc racking errors. I was also used o compare performance beween he road racker and manual annoaion. Table 1. Saisics on users and inpus. User1 User2 User3 User4 User5 User6 User7 User8 Gender F F M M F M M M Toal number of inpus Toal ime cos (in seconds) Average ime per inpu (in seconds) Table 1 shows some saisics on users and daa. The saisics include he oal number of inpus, he oal ime for road annoaion, and average ime per inpu. The number of inpus reflecs how close he user zoomed in he image. When he image is zoomed in, mouse clicks raverse he same disance on he screen bu correspond o shorer disances in he image. Thus, he user needed o inpu more road segmens. The average ime per inpu reflecs he ime ha users required o deec one road axis and annoae i. From he saisics, i is obvious ha he users had performed he asks in differen paerns, which influenced he qualiy of he inpu. For example, more inpus were recorded for user 4. This was because user 4 zoomed he image ino more deail han

6 he oher users. This made i possible o deec road axis locaions more accuraely in he deailed image. Anoher example is ha of user 3, who spen much less ime per inpu han he ohers. This was eiher because he was faser a deecion han he ohers, or because he performed he annoaion wih less care. Table 2. Performance of semi-auomaic road racker. The meaning of,, and is described in he ex.. n h (in seconds) (in seconds) c User1 User2 User3 User4 User5 User6 User7 User Time saving (%) n h c 6 Experimens and Evaluaions We implemened he semi-auomaic road racker using profile selecion and paricle filering. The road racker ineraced wih he recorded human daa and used he human daa as a virual user. We couned he number of imes ha he racker referred o he human daa for help, which is considered as he number of human inpus o he semi-auomaic sysem. In evaluaing he efficiency of he sysem, we compued he savings in human inpus and savings in annoaion ime. The number of human inpus and ploing ime are relaed and so reducing he number of human inpus also decreases ploing ime. Given an average ime for a human inpu, we obained an empirical funcion for calculaing he ime cos of he road racker: c = + λn (1) h where c is he oal ime cos, is he racking ime used by road racker, nh is he number of human inpus required during he racking, and λ is an user-specific variable, which is calculaed as he average ime for an inpu λ i = oal ime for user i oal number of inpus for user i (2) The performance of semi-auomaic sysem is shown in Table 2. We observe a large improvemen in efficiency compared o a human doing he asks manually. Furher analysis showed ha he majoriy of he oal ime cos came from he ime

7 used o simulae he human inpus. This suggess ha reducing he number of human inpu can furher improve he efficiency of he sysem. This can be achieved by improving he robusness of he road racker. The performance of he sysem also reflecs he qualiy of human inpu. Inpu qualiy deermines how well he emplae road profiles can be exraced. When an inpu road axis deviaes from he rue road axis, he corresponding emplae profile may include off-road conen perpendicular o he road direcion. Moreover, he profile along he road direcion may no more be consan. Thus, he road racker may no find a mach beween observaions and emplae profiles, which in urn requires more human inpus, reducing he sysem efficiency. Fig. 2 shows a comparison of sysem wih and wihou processing of human inpu during road emplae profile exracion. When human inpu processing is skipped, noisy emplae profiles ener he knowledge base. This increases he ime for profile maching during he observaion sep of he Bayesian filer, which, in urn, causes he sysem efficiency o drop dramaically. Fig. 2. Efficiency comparison of semi-auomaic road racking. 7 Conclusion Sudying he influence of human inpu o he semi-auomaic image inerpreaion sysem is imporan, no only because human inpu affecs he performance of he sysem, bu also because i is a necessary sep o develop user-adaped sysems. We have inroduced a way o model hese influences in an image annoaion applicaion. The user inpus were ransferred ino knowledge ha compuer vision algorihm can process and accumulae. Then hey were processed o opimize he road racker in

8 profile maching. We analyzed he human inpu paerns and poined ou how he qualiy of he human inpu affeced he efficiency of he sysem. References 1. Myers, B., Hudson, S., Pausch, R.: Pas, presen, and fuure of user inerface sofware ools. ACM Transacions on Compuer-Human Ineracion 7 (2000) Chin, D.: Empirical evaluaion of user models and user-adaped sysems. User Modeling and User-Adaped Ineracion 11 (2001) Isard, M., Blake, A.: CONDENSATION-condiional densiy propagaion for visual racking. Inernaional Journal of Compuer Vision 29 (1998) Zhou, J., Bischof, W., Caelli, T.: Road racking in aerial image based on human-compuer ineracion and bayesian flering. ISPRS Journal of Phoogrammery and Remoe Sensing 61 (2006) Baumgarner, A., Hinz, S., Wiedemann, C.: E±cien mehods and inerfaces for road racking. Inernaional Archives of Phoogrammery and Remoe Sensing 34 (2002) Zhou, J., Cheng, L., Bischof, W.: A novel learning approach for semi-auomaic road racking. In: Proceedings of he 4h Inernaional Workshop on Paern Recogniion in Remoe Sensing, Hongkong, China (2006) 61-64