Mutual Adaptation to Mind Mapping in Human-Agent Interaction

Transcription

1 Proceedings of te 2002 IEEE Int. Worksop on Robot nd Humn Interctive Communiction Berlin, Germny, Sept , /02/$ IEEE 105 Mutul Adpttion to Mind Mpping in Humn-Agent Interction Seiji YAMADA Ntionl Institute of Informtics Hitotsubsi, Ciyod, Tokyo , Jpn Tomoiro YAMAGUCHI Nr Ntionl College of Tecnology 22 Yt-co, Ymto-Koriym, Nr , Jpn Keywords Mutul dpttion between umn nd robot, mind-expression mpping, instnce-bsed lerning Abstrct Tis pper describes umn-gent interction frmework in wic user nd life-like gent mutully cquire te oter s mind mpping troug mutul mind reding gme. A lot of studies ve been done on life-like gent including umnoid robots. Troug development of vrious life-like gents, n internl stte (we cll mind) of n gent like emotion, processing lod s been recognized to ply n importnt role in mking tem believble to user. For estblising effective nd nturl communiction between n gent nd user, tey need to red te oter s mind from expressions nd we cll te mpping from expressions to mind sttes mind mpping. If n gent nd user don t obtin tese mind mppings, tey cn not utilize beviors wic significntly depend on te oter s mind. We formlize suc mutul mind reding nd propose frmework in wic user nd life-like gent mutully cquire mind mppings ec oter. In our frmework, user plys mutul mind reding gme wit n gent nd tey grdully lern to red te oter s mind. Eventully we implement our frmework nd mke experiments to investigte its effectiveness. 1 Introduction In tese severl yers, lot of studies ve been done on life-like gent like softwre gent[7][6] nd robot gent[11][9]. A typicl life-like gent ppers on Web sopping pge nd ssists user in inputting is/er order. Troug te development of vrious life-like gents, n gent s mind, n internl stte 1 representing emotion, processing lod s been recognized to ply very importnt role in mking tem believble to user[2]. Tus resercers re trying to implement mind model on n gent for mking it 1 Teory of Mind s been developed in psycology, nd our work is relted wit it. However we do not del wit model for describing umn mind, rter our term mind mens prt of computtionl internl sttes of n gent nd umn like sttes of processing lod, resoning, ttention nd so on. more believble[2][10]. However, even if mind genertion mecnism is fully implemented on life-like gent, tere is significnt problem tt mind reding between user nd n gent is difficult. For estblising effective communiction between life-like gent nd user, tey need to be ble to identify te oter s mind from n expression nd we cll tis tsk mind reding. If mind reding is impossible, tey cn not ct umn-like beviors wic significntly depend on te oter s internl stte. For exmple, life-like gent sould kindly nd crefully beve to depressed or busy user, nd intuitively communicte its processing lod to user troug (fcil) expression. Toug mind reding is lwys done - mong umn, it between life-like gent nd user becomes fr more difficult. Becuse design of gent s expressions depends on personl preference, socil culture nd so on. For exmple, Fig.1 sows vrious expressions nd corresponding minds of Microsoft R gents, wic is known s te most populr like-like softwre gent. We cn esily identify minds from some expressions (Surprised, Congrtulte in Fig.1), owever minds from some expressions (Confused, Decline, Process in Fig.1) my be rd to be identified. Consequently life-like gent nd user need to cquire reltion between n expression nd mind wen tey ctully encounter. We cll suc mpping from n expression to mind mind mpping. Since tis sitution occurs independently of weter n life-like gent is implemented on softwre or pysicl robot, umn-robot interction needs to cope wit tis problem of lerning mind mpping mutully. In tis pper, we propose umn-gent interction frmework in wic user nd life-like gent mutully cquire mind mppings ec oter. Tey ply mutul mind reding gme togeter nd grdully lern mind mppings ec oter. Instnce-bsed lerning is pplied to gent s lerning. We fully implemented our frmework on PC wit CCD cmer nd eventully we mke experiments for investigting mutul mind reding. Velásquez[10] proposed emotion model bsed on society of mind. His model is for generting umnlike emotions using multi-gent system rcitecture in wic ec gent corresponds to primitive emotion nd emotions re emerged s result of te in-

2 106 Confused Congrtulte Decline Plesed Process Suprised Tink Humn Agent s = e i T s = e j n : e x i x = M (e ) : x e e = M (x ) x x M (x ) = e : x e m s =e m n : e x x = M (e ) T s =e n Figure 1 Vrious expressions of Microsoft gents. Figure 2 A frmework for emotionl interctions between gent nd user. terctions. However te purpose of is reserc is to generte emotions nd moods like umn, nd not to build frmework for interction between n gent nd user. Vrious reserces of vtrs ve been done intensively on interction wit mn[3], nd tey found out interesting view points bout communictive gents. However teir purpose is to develop vtrs tt cn nturlly communicte wit umn nd our one is to design interction between umn nd n gent. A lot of reserces on fcil expression recognition [5] ve been done tus fr. We cn utilize tese tecniques to ctegorize sensed expressions. However our interest is concerned wit mutul lerning of mind mpping, nd our reserc objectives is quite different from fcil expression recognition. Humn robot interction ve been lso studied ctively. In prticulr, Ono nd Imi proposed cognitive model to describe ow umn reds robot mind nd investigted its vlidity experimentlly[9]. Toug teir work is excellent nd interesting, it s no mutul lerning of mind reding like our work. 2 Lerning of Mind Mpping In tis section, we formlize our frmework to del wit mutul mind reding between gent nd user. First te following primitives re introduced. Mind stte s,s : A vrible s nd s stnding for stte of mind for n gent nd user respectively. A primitive mind is substituted for tis vrible. Primitive mind E = {e 1, }, E = {e 1, }: E nd E re sets of m elements of gent s nd user s minds respectively. We cn define tese primitive minds depending on prticulr tsk. Primitive expression X = {x 1, }, X = {x 1, }: X nd X re sets of gent s nd user s primitive expressions. Mind mpping M:x e = {x i e j, }: Tis mens user s mny-to-one mpping from primitive expressions to primitive minds wic ws lerned by n gent. M:x e mens gent s mind mpping lerned by user. Expression mpping M:e x, M:e x: A user s (or n gent s) one-to-mny mpping from primitive minds to primitive expressions. Mind trnsition function T (c), T (c): Tis function determines te next mind of n gent/user depending context c. Tis context c my include its current mind, te oter s current mind, success rtes nd so on. Using te bove nottions, we describe frmework in wic life-like gent nd user interct troug expressions s sown in Fig Wt sould be lerned? Wit te frmework described in Fig.2, we define lerning of mind mppings nd mutul lerning of mind mppings in te following. Lerning of mind mpping: An gent(or user) cquires te oter s mind mpping M :x e (or M :x e ). Mutul lerning of mind mppings: An gent nd user mutully cquire te oter s mind mpping, M:x e nd M :x e. Since designer is ble to develop n gent by imself in prcticl situtions, we cn ssume tt te following prmeters wic re concerned wit n gent re given. Primitive minds of user my not be essentilly determined by designer. However we consider tt n gent (or its designer) sould determine primitive minds of user becuse ow n gent utilizes tem is significntly dependent on te gent s bility. Primitive minds of user nd n gent. Primitive expressions of n gent. A mind trnsition function of n gent. Except primitive minds of user, we give no constrin to user. A user freely lerns n gent s mind mpping. Given te bove prmeters, te mutul lerning of mind mppings is cieved by procedures described in te next subsection.

3 Lerninginngent Becuse user is ble to utonomously lern n gent s mind mpping in our frmework, we give no restriction to user witin is/er lerning. Tus we develop only lerning procedures of n gent. Since primitive minds of user re given, n gent does not need to cquire tem. Also user s primitive expressions re obtined by ctegorizing cptured imges wit CCD cmer. Hence if user s primitive mind e is estimted wen user s expression x is observed, n gent cquires n instnce of user s mind mpping x e. After n gent stores sufficient suc instnces troug interctions wit user, it becomes ble to estimte user s primitive mind from is/er observed primitive expression by instnce-bsed lerning[1] like NN(nerest neigbor) metod[4]. Agent Lerning procedure c C, c =(I,S): n instnce. I c: n ttribute vector. V : set of clsses v. S c: sequence of ltest n nswer pirs. S c =[s 1, s 2,, s n]=[(v 1, good), (v 2, nogood), ] 1. A new ttribute vector I new is given. 2. Investigte te most similr instnce c sim to I new by computing te distnce between te ttribute vectors. 3. Determine clss ˆv V usingtefollowing eqution. Rndom selection is done for tie-breking. ˆv rgmx g(v,s) v V s S csim were g(v,s) = 1ifs =(v, good), g(v,s) = 1 if s =(v, nogood), nd g(v,s) =0ifno(v, ). If no instnce in n initil period, determine ˆv t rndom. 4. Indicte ˆv to user, nd e/se nswers YES or NO to ˆv. 5. If te nswer is YES, dd (ˆv, good) intos of c sim, nd remove te oldest s from S if necessry. 6. If te nswer is NO, dd (ˆv, nogood) intos of c sim respectively, nd remove te oldest s from S if necessry. Also if te distnce between c sim nd I new is over tresold α,ddnewinstnce (I new, [(ˆv, nogood)]) to C. Wen n gent guess user s mind nd sows it to user in lter mutul mind reding gme, e/se nswers by Yes or No to te estimted mind. Tus n gent needs to utilize No nswer wic is not generlly employed for instnce-bsed lerning. Since te number of clsses (user s primitive minds) is usully over tree, we cn not determine wic clss te No nswer is positive instnce to. Tus we modified simple instnce-bsed lerning lgoritm IBL2[1] to bebletodelwit No nswer. Wen No nswer is given to n estimted primitive mind, n gent stores it s new instnce ving negtive evlution to te estimted clss. To del wit suc negtive e- vlution, n gent ssigns set of recent evlutions nd estimted minds to n instnce nd determines its clss by mjority vote. Detil procedures of gent lerning re sown in te following. In ll te lter experiments, we set prmeters s n =2,α = 900 empiriclly. 2.3 Success rte nd finis condition of lerning Te success rte r(e) for primitive mind e is computed by te following eqution. Tis success rte is lso utilized to evlute user s lerning. Te verge vlue R of ll r(e) is used to indicte te progress of lerning. r(e) = Te number of success nswer pirs in S c S c Finis condition for lerning of n gent nd user is described s R = 1. Tis mens recognitions of ll primitive minds become complete wen te condition is stisfied. 3 Mutul Mind Reding Gme A priml objective of mutul mind reding gme is to collect instnces for gent s instnce-bsed lerning efficiently nd brodly. An instnce is pir of n observed user s expression nd n estimted primitive mind. In tis pper, gme in wic plyer estimtes te oter s mind stte troug te (fcil) expression to compete for te ccurcy is clled mutul mind reding gme. A problem of tis gme is tt user s cognitive lod becomes ig. To solve tis, tis gme is designed so tt user my enjoy it to ply prt in collecting trining dt ctively, nd s results, te user s cognitive lod becomes low. Anoter objective of mutul mind reding gme is concerned wit trust nd motivtion[8][6]. We consider tt it is not good ide to give user n gent wic fully lerned user s mind mpping from te s- trt. On tis mtter, Scneidermn rgued tt suc sopisticted gent would give user feeling of loss of control nd understnding nd te user does not try to do modeling te gent[8]. Tus we believe tt user is effectively motivted troug mutul mind reding gme. Procedures of mutul mind reding gme re given in te following. Note tt n gent tells its correct mind to user wen is/er nswer is incorrect, but user does not do so to reduce is/er cognitive lod. 1. An expression of n gent is displyed to user in GUI. 2. A user guesses gent s mind from seeing te expression, nd tells te mind to n gent by clicking button. 3. An gent replies Yes (te guess is correct) or No (te guess is incorrect) wit te correct mind s judgment ginst te oter s guess.

4 108 Figure 3 Environment of umn-gent interction. 4. An gent sees n expression of user by CCD cmer. 5. An gent guesses user s mind from te cptured expression, nd sows te mind to user troug GUI. 6. A user replies Yes (te guess is correct) or No (te guess is incorrect) s judgment ginst te oter s guess. 7. Te bove procedures re repeted until finis condition of mutul lerning (described in 2.3) is stisfied. 4 Implementtion We fully implemented our frmework. A system consists of lptop computer (SONY VAIO-SR9G/K) nd CCD color cmer (Cretive Medi: Web- Cm Plus) wit USB. Te resolution of te cmer is (8bit color). We used VineLinux2.1, C nd GTK+. Also Video4Linux API ws employed for imge cpture progrmming. An experimentl environment is sown in Fig.3. In pse of gent s lerning, n gent sequentilly cptures imges of user expressions per 500ms, nd obtins stble expression. Tis stble expression mens continuous four imges wit distnce less tn tresold. We experimentlly set te tresold s 250. Wen stble expression is obtined, te ed imge is used s cptured imge. Tis mecnism llows user to control te timing to present is/er expression to n gent. Cptured imge is trnsformed into n imge wit wit 8bit grey scle for n instnce. Since computtionl cost depends on te size of n imge, we used suc smll grey imge. Tus n instnce is described by vector wit 256 vlues of 1200 dimensions. Te similrity between instnces is defined te Euclid distnce. Figure 4 Humn guesses gent s mind. We do not employ ny feture detection for describing n instnce. Becuse lrge computtionl cost mkes system response slow nd neiter te best fetures nor te best detection metod for ny (fcil) expression recognition s been developed. In sted, we consider tt user dptively forms is/er expressions so tt n gent cn recognize tem. Tis is user s dpttion to n gent, nd gent s lerning is gent s dpttion to user. Fig.4 sows snpsot of GUI wen user guesses gent s mind. Wen user clicks te Strt button, n gent sows its expression. Ten user guesses gent s mind, nd clicks one of Primitive Mind buttons. If user clicks te button, n gent tells te judgment wit te correct mind like messge in Fig.4. Also two progress brs re sown for indicting verge success rtes R (described in 2.3) of user nd n gent. A user cn understnd te degrees of lerning progresses by seeing te progress brs. A gme finises wen bot of two progress brs reces to te rigt edges. Fig.5 sows interfce were n gent guesses user s mind. Wen user clicks Strt gent s recognition button, n gent begins to cpture user s imges. After stble expression is cptured, te four imges re sown te window. Also stored instnces re indicted wit lbels nd te distnce between tem nd cptured imge. Using te most similr instnce, n gent guesses user s mind nd tells it to user like Fig.5. A user nswers to it by clicking Yes or No buttons.

5 109 Confused Plesed Suprised Tink Figure 6 Four expressions used for experiments. Figure 5 An gent guesses user s mind. 5 Experiments 5.1 Experimentl metods We mde experiments to verify mutul lerning between user nd n gent, nd to investigte its crcteristics. Troug ll te experiments, we employed eigt subjects consisting of five grdute students, tree stff mjoring Computer Science t Tokyo Institute of Tecnology. We used four primitive minds nd primitive expressions for n gent sown in Fig.6 nd tree primitive minds Ordinry, Tinking, Decline for user. As te primitive minds increse, mutul lerning becomes rder. We empiriclly consider te number of tese primitive minds is vlid for prcticl experiments. Before experiments, we briefly gve subjects te following instructions. However we did not explin detil procedures of gent s lerning, success rtes nd menings of cptured imges, instnce imges in Fig.5. Rules of mutul mind reding gme. Explntion on GUI: menings of two progress brs, buttons nd tbs. Advise to ffect user s expressions: it is effective to sligtly rotte, tilt ed nd touc fce. Due to gent s bility, fine expressions on fce is rd to be recognized. Tree primitive minds Ordinry, Tinking, Decline for user. Also we set n gent s mind trnsition function T (c) described in section 2 s te simplest one: rndom trnsition. Tis mens n gent s mind cnges into next mind rndomly independently of context. We will improve tis function lter to mke umn lerning more efficient. Under te bove conditions, ec of eigt subjects plyed mutul mind reding gme once wit n gent nd we investigted trnsitions of user s nd - gent s success rtes, success rtes for ec primitive mind, te number of interctions until lerning finised nd rel time tken for gme. We counted n interction by pir of gent s guess nd user s guess in gme. 5.2 Observing mutul lerning Fig.7 sows representtive results for success rtes of user nd n gent. Te two success rtes grdully increses s interctions progressed, finlly bot of tem converged to 1 nd te gme finised. Tus we re ble to observe mutul lerning of mind mppings (described in 2.1) in te experimentl results. Since user nd n gent sometimes filed to guess te oter s mind, increses of two success rtes re not monotonic. In ll te experiments, we observed suc mutul lerning of mind mppings between user nd n gent. A single gme took bout 5 15 minutes, nd most subjects seemed to enjoy experiments. Te stored instnces for tree subjects s-1, s-2, s-4 re sown in Fig.8. In contrst tt only tree instnces wic were minimum to ctegorize tree user s minds were stored for s-2, over five instnces were s- tored for s-1 nd s-4. For ll te subjects, te numbers of stored instnces ve significnt dispersion. Seeing te expressions in te instnces in Fig.8, most of tem were done by tilting ed or toucing fce. Instruction to ffect expressions migt excessively restrict user s expressions. Since our work is in n erly stge, tere re some limittions nd open problems. In tese experiments, te number of primitive minds were reltively smll. Tus we cn utilize simpler metod tt we directly sow user tble like Fig.1 to remember mind mpping. However user intends to feel loss of control nd understnding in suc sitution s Scneidermn clims. Tus we consider our pproc of gme my outperform suc simple pproc. Wile our metod is pplicble to lrge number of primitive minds, mutul lerning becomes very slow nd we need dditionl metods to improve it.

6 110 Success rte s-1 s-2 s-4 Figure 7 Figure 8 Interctions Humn Agent Representtive mutul lerning. Ordinry Tinking Decline Stored instnces of tree subjects. 6 Conclusion We proposed umn-gent interction frmework in wic user nd life-like gent mutully cquire teir mind mppings troug mutul mind reding gme. For describing mind interctions between life-like gent nd user, we defined elements of our frmework nd developed gent s lerning procedures by using n instnce-bsed lerning metod. Ten, to cquire te mind mpping ec oter, we developed mutul mind reding gme in wic user nd life-like gent try to recognize te oter s mind from te oter s expression. We implemented our frmework nd mde preliminry experiments by employing subjects. As results, we found out mutul lerning between user nd gent troug mutul mind reding gme nd some crcteristics of mutul lerning. A life-like gent in our frmework is not necessry to be softwre gent, my be pysicl robot gent like pet robot. Actully we re currently developing system in wic user cn interct wit SONY pet robot AIBO wit CCD cmer. In suc cse, n expression of n gent my be gesture rter tn fcil expression, nd we need imge processing like segmenttion of fce region. We consider te umnrobot interction gives us muc interesting penomen between umn nd robot gent, nd we try to modify our frmework to del wit tem effectively. References [1] D.W.A,D.Kibler,ndM.K.Albert. Instncebsed lerning lgoritms. Mcine Lerning, 6:37 66, [2] J. Btes. Te role of emotion in believble gents. Communictions of te ACM, 37(7): , [3] J. Cssell. Embodied converstionl gents: Representtion nd intelligence in user interfce. AI Mgzine, 22(4):67 83, [4] B. V. Dsrty. Nerest Neigbor (NN) Norms: NN Pttern Clssifiction Tecniques. IEEE Computer Society Press, [5] H. Kobysi nd F. Hr. Recognition of six bsic fcil expressions nd teir strengt by neurl network. In IEEE Interntionl Worksop on Robot nd Humn Communiction, pges , [6] P. Mes. Agents tt reduce work nd informtion overlod. Communictions of te ACM, 37(7):30 40, July [7] MS gent Web pge. ttp://msdn.microsoft.com/msgent/. [8] B. A. Myers, A. Cyper, D. Mulsby, D. C. Smit, nd B. Sneidermn. Demonstrtionl interfces. In Proceedings of 1991 Conference on Humn Fctors nd Computing Systems, pges , [9] T. Ono nd M. Imi. Reding robot s mind: A model of utternce understnding bsed on te teory of mind mecnism. In Proceedings of te Seventeent Ntionl Conference on Artificil Intelligence, pges , [10] J. D. Velásquez. Modeling emotions nd oter motivtions in syntetic gents. In Proceedings of te Fourteent Ntionl Conference on Artificil Intelligence, pges 10 15, [11] J. D. Velásquez. An emotion-bsed pproc to robotics. In Proceedings of te 1999 IEEE/RSJ Interntionl Conference on Intelligent Robots nd Systems, pges , 1999.