Fixation-Image Charts

Transcription

1 Fxton-Imge Chrts Kuno Kurzhls, Mrcel Hlwtsch, Mchel Burch, Dnel Weskopf Unversty of Stuttgrt P rc p nt1 2 1 P rc p nt2 3 P rc p nt3 P rc p nt4 c F x ons equenc e Col orl eg end c c Fgure 1: Fxton-mge chrts dsplyng the fxton sequences of four prtcpnts on tmelnes wth nformton out fxton durton (heght of the mges), sccde lengths (length of colored rs), nd sccde drectons (ccordng to the color legend). A susequence consstng of 4 fxtons (1 4) nd 3 sccdes ( c) s hghlghted n the closeup nd depcted n the stmulus on the rght. Astrct 1 We fcltte the comprtve vsul nlyss of eye trckng dt from multple prtcpnts wth vsulzton tht represents the temporl chnges of vewng ehvor. Common pproches to vsully nlyze eye trckng dt ether occlude or gnore the underlyng vsul stmulus, mprng the nterpretton of dsplyed mesures. We ntroduce fxton-mge chrts: new technque to dsply the temporl chnges of fxtons n the context of the stmulus wthout vsul overlp etween prtcpnts. Fxton durtons, the dstnce nd drecton of sccdes etween consecutve fxtons, s well s the stmulus context cn e nterpreted n one vsul representton. Our technque s not lmted to sttc stmul, ut cn e ppled to dynmc stmul s well. Usng fxton metrcs nd the vsul smlrty of stmulus regons, we complement our vsulzton technque wth n nterctve flter concept tht llows for the dentfcton of nterestng fxton sequences wthout the tme-consumng nnotton of res of nterest. We demonstrte how our technque cn e ppled to dfferent types of stmul to perform rnge of nlyss tsks; nd dscuss dvntges nd shortcomngs derved from prelmnry user study. Wth the ncresng populrty of eye trckng n vrous reserch felds [Duchowsk 2002], mny experments re conducted nd nlyzed wth estlshed sttstcl nd vsul nlyss technques. Typclly, sttstcl evluton s ppled for hypothess testng, potentlly showng sgnfcnt dfferences etween ndependent vrles. Vsulzton s employed to convey nd llustrte these results. Recently, vsul nlytcs pproches [Andrenko et l. 2012; Blscheck et l. 2016] hve ecome populr. They comne sttstcl methods nd nterctve vsulzton to enle the explorton of eye trckng dt (e.g., from plot study) nd support the understndng of the complex sptotemporl nture of the dt. Keywords: Eye trckng vsulzton, sttc nd dynmc stmul, scnpth, fxton Concepts: Humn-centered computng Vsul nlytcs; Vsulzton systems nd tools; E-ml: nme.surnme@vsus.un-stuttgrt.de Permsson to mke dgtl or hrd copes of ll or prt of ths work for per- Introducton The generl nlyss prolem we ddress s the nterpretton of scnpths n spce nd tme. A scnpth, consstng of fxton sequences nd sccdes etween the fxtons, cn e very complex, even for sngle prtcpnt. The comprson of scnsonl or clssroom use s grnted wthout fee provded tht copes re not mde or dstruted for proft or commercl dvntge nd tht copes er ths notce nd the full ctton on the frst pge. Copyrghts for components of ths work owned y others thn the uthor(s) must e honored. Astrctng wth credt s permtted. To copy otherwse, or repulsh, to post on servers or to redstrute to lsts, requres pror specfc permsson nd/or fee. Request permssons from permssons@cm.org. c 2016 Copyrght held y the owner/uthor(s). Pulcton rghts lcensed to ACM. ETRA 16,, Mrch 14-17, 2016, Chrlston, SC, USA ISBN: /16/03 DOI:

2 pths from multple prtcpnts s even more dffcult. To ddress ths ssue, mny pproches exst, some sed on eye trckng metrcs [Holmqvst et l. 2011], strng-comprson lgorthms [Duchowsk et l. 2010; West et l. 2006], nd vsulzton [Golderg nd Helfmn 2010]. However, mny estlshed metrcs, such s verge fxton durtons or verge scnpth lengths, ggregte the dt, neglectng ts sptotemporl structure. These pproches mpr nlyss tsks for whch t s mportnt to fnd out how vewng ehvor, represented y fxtons nd sccdes, chnged over tme. If the temporl dmenson of the underlyng eye trckng dt s ncluded n the nlyss, vsul representton of the mesured dt typclly supports ther nterpretton. A smple nd common pproch s plot of mesured vrle over tme (see Fgure 2). Ths vsulzton provdes vlule nsght nto how the vrle chnged over tme nd multple prtcpnts cn e compred y overlyng ther respectve plots. Even strng-sed lgorthms tht clculte the smlrty etween scnpths eneft from vsul representton of smlr susequences. However, gze ehvor s lrgely dependent on the vsul stmulus. Wthout ncludng the context of the stmulus, we wll not e le to understnd the resons why certn chnges n recorded mesures occur. To ddress ths ssue, reserchers usully hve to nvestgte the recorded dt s sttc nd nmted representton, overld on the stmulus. A typcl exmple of such vsulzton s the gze plot (Fgure 2). Here, gze ponts or fxtons re represented y connected crcles tht mntn the temporl order of the scnpth. The rd ndcte fxton durtons. These crcles re drwn on the stmulus, ndctng where prtcpnt ws lookng t whch pont n tme. Ths pproch suffers from occluson due to the mssve overdrw tht occurs when mny prtcpnts hve to e compred over long tmespns. Other pproches such s ttenton mps ggregte the dt nd re therefore not suted for comprtve sptotemporl nlyss of multple prtcpnts. If res of nterest re vlle, lterntve tmelne vsulztons such s scrf plots [Rchrdson nd Dle 2005; Kurzhls et l. 2014] cn provde semntc scnpth representton of the sequentl vsts n the nnotted res. Especlly for dynmc stmul, ths nnotton s often tme-consumng preprocessng step requrng the defnton of oundng shpes tht cn lso chnge ther poston, sze, nd vslty. Our mn contruton s vsul nlytcs pproch tht comnes descrptve sttstcs of fxton metrcs, nterctve flterng technques, nd vsul representton of fxton sequences. Wth our nterctve vsulzton technque (Fgure 1), we provde n lterntve pproch for scnpth comprson tht retns the dvntges of sttstcl nd gze plots. A detled nlyss of scnpths from multple prtcpnts over tme s possle wthout the tedous nnotton of res of nterest. For exmple, our pproch cn e ppled for effcent lelng of fxtons (Secton 4.1) nd for hypothess uldng y nvestgtng ptterns n the vsulzton (Secton 4.2). 2 Relted Work The numer of new vsulzton technques for eye trckng nlyss ncresed sgnfcntly over the lst yers. Blscheck et l. [2014] provde n overvew nd txonomy of stte-of-rt technques. The vsulzton methods cn e ctegorzed n two mjor clsses: some technques re pont-sed,.e., the underlyng dt re ether rw gze ponts or ggregted fxtons, nd the other technques re sed on ddtonl nformton out res of nterest (AOIs). Our technque cn e clssfed s pont-sed pproch ecuse AOIs re not requred. The stmulus content s pssve,.e., prtcpnts cnnot nfluence the presented stmul. We fxton durton [ms] sccde length [px] tme [ms] tme [ms] () Fgure 2: Typcl exmples of vsul nlyss of the sptotemporl spects of eye trckng dt: () sttstcl plots of fxton durton nd sccde length, () gze plot drwn on the stmulus. provde sptotemporl nformton out the dt from sttc nd dynmc stmul n sttc, yet nterctvely djustle vsulzton. Other technques wth smlr propertes ether ggregte fxtons [Duchowsk et l. 2012] or scnpth trjectores [Duchowsk nd McCormck 1998; Hurter et l. 2014]. We do not ggregte the dt over multple prtcpnts or over tme (except for the ggregton of rw gze dt to fxtons), llowng for n esy comprson etween prtcpnts. Furthermore, our vsulzton gurntees the stmulus context to e free from overlp, whch s n dvntge compred to gze plots. The concept of our vsulzton s sed on comnng eye trckng mesures wth the context of the stmulus y cretng thumnl mges of fxted res. A smlr mge-sed pproch to lel fxton dt s presented y Pontllo et l. [2010], who show mges of fxted regons to the nlyst for sem-utomtc clssfcton of fxted res. However, the uthors ppled ths pproch only to ssgn fxtons to lels, further nlyss wth sttstcl or vsulzton technques s stll requred for ths nnotted dt. Wth our pproch, the nlyst cn nterpret the dt drectly n the vsulzton. Kurzhls et l. [2016] dsply fxton mges for every mesured gze pont n the dt on tmelnes to compre eye movements of dfferent prtcpnts over tme. The uthors represent the complete sequence of dt smples, fxton nformton s only mplctly vsle y duplcted fxton mges on the tmelne. We dopt ther pproch to much more compct tmelne y showng only one mge per fxton. We compenste the resultng synchroncty y n ddtonl vsul component: the tme strems. Furthermore, n the work of Kurzhls et l., nformton out sccde drectons nd sccde lengths s not vlle. Wth our new pproch, we provde vsulzton tht s much more sclle n the temporl dmenson nd provdes ddtonl fxton nformton. Addtonlly, we ntroduce n nterctve flter concept tht ws not ncluded n tht prevous work. There re other concepts of our vsulzton tht re vsully smlr to other technques: the tme strems nd the flter dls. Burch et l. [2013] pply horzontl rver vsulzton on tmelne to dsply the temporl chnges of ttenton on dfferent AOIs,.e., the flow of ttenton etween AOIs. In contrst, we pply vertcl rvers to vsulze the temporl coherence of fxtons etween prtcpnts s tme strems,.e., we use smlr vsul desgn for very dfferent dt vsulzton prolem tht does not requre ny AOIs. Hlwtsch et l. [2015] pply flter wheels to fnd scnpth sequences sed on sccde drectons. Ther pproch s sed on the scnpth trjectory lone, no vsul connecton to stmulus context s provded. We lso llow the nlyst to flter y sccde drecton wth our flter dl, ut the flter results re nstntneously vsle n context of the stmulus. ()

3 Fgure 4: Tme strems strtng from fxed tmelne ntervls nd comprsng ll fxton mges tht le wthn the ntervl. () () Fgure 3: () Multple fxton metrcs cn e encoded n one fxton mge. () The sequences of fxton mges represent the scnpths of prtcpnts nd cn e compred wth ech other. 3 Vsulzton Desgn A vsulzton for the nlyss of sequentl vewng ehvor should convey the lner order nd durton of fxtons 1, dsply nformton out the poston of fxtons nd the drecton of sccdes 2, nd depct the ttended context of the stmulus 3. Addtonlly, ndvdul prtcpnts should e vsle for comprson 4. Our vsulzton s desgned to meet these requrements y ntroducng dfferent vsulzton components. 3.1 Vsulzton Components The vsul desgn of our pproch conssts of three components (Fgure 3) tht re comned on tmelnes for ndvdul prtcpnts seprtely. Nmely, fxton-mge chrts consst of fxton mges (top) to dsply the propertes of fxton 1 nd the context of the underlyng stmulus 3, dstnce rs (mddle) s n ndctor for sccde lengths 2 etween consecutve fxtons (ottom, rght), nd tme strems (Fgure 4) to mntn the temporl synchronzton etween prtcpnts Fxton Imges The fxton mges re generted sed on the poston of fxton on the stmulus. Assumng vsul ngle of 2 nd vewng dstnce of 64 cm n front of 24 WUXGA screen, the m- ge s cropped round the fxton poston (Fgure 1), cretng pxels mge tht pproxmtely contns the foveted re [Nyström nd Holmqvst 2010; Rjshekr et l. 2004]. So fr, the fxton mge provdes us nformton where the prtcpnt ws lookng. To encode the durton of fxton, the heght nd wdth of the mge re djusted ccordngly (Fgure 3). We normlze the fxton durton y the longest fxton of ll prtcpnts. Ths pproch hs the eneft tht longer fxtons lso receve more spce n the vsulzton. To emphsze the dfferences etween low vlues, we use the squre root of the vlues. The fxton sequence of prtcpnt s vsulzed y stckng the correspondng fxton mges next to ech other (Fgure 3). Hence, the consecutve fxton mge n the sequence s lwys rght to the current fxton. If sngle fxtons re selected wth our flterng methods (see Secton 3.2.2), we show compct representton of the results under the tmelnes. In ths cse, we dsply the consecutve fxton under the dstnce r, ndctng the trget poston of sccde (Fgure 3, ottom) Dstnce Brs Below ech fxton mge, dstnce r ndctes the dstnce etween consecutve fxtons (Fgure 3, mddle). Although not clculted explctly y lgorthm, ths dstnce s often ppled s n mplct ndctor for sccde length. The heght of the r descres the Euclden dstnce etween two consecutve fxtons, normlzed y the mxmum dstnce wthn ll prtcpnts fxtons. Ths representton s smlr to tme plot of ths metrc (Fgure 2, ottom) tht s common n eye trckng reserch nd s esy to nterpret y eye trckng experts. The dstnce rs re color-coded y sccde drecton to show even more nformton out the sptl relton etween two fxtons. The ngle etween the horzontl nd the connecton lne etween two fxtons s used to otn the color from the color-legend wheel (Fgure 3). By ths pproch, t s esy to detect rs of smlr length nd color, whch cn ndcte repettve ehvor (e.g., redng from left to rght). We decded gnst glyph representton for drectons (e.g., rrows) n fvor of the etter scllty of colors tht s requred n n overvew of the vsulzton. However, the drecton-to-color encodng requres some prctce for nterpretton. Therefore, we ncluded the color legend drectly to the correspondng flter dl (Secton 3.2.2) to help wth the nterpretton of drectons Tme Strems Snce the numer nd length of ndvdul fxtons vres for ech prtcpnt, there s vryng numer of fxton mges. For dsplyng the mges on tmelne, there re two optons: showng the mges on n solute tmescle or showng the mges stcked next to ech other. The frst pproch preserves the synchronzton

4 e c FlterHstogrm mn n mx d Fgure 5: Vsulzton overvew: () Fxton-mge chrts, () flter query nterfce, (c) query results, (d) lel edtor, (e) stmulus vew. etween prtcpnts, ut the resultng tmelne would crete gps etween fxtons nd the horzontl scllty would e mpred. Therefore, we decded for the second pproch, neglectng solute temporl poston of fxtons. Ths pproch cretes dense representton of ll fxtons n ther sequentl order, requrng less spce on the horzontl xs. However, t results n synchronous tmelnes due to the vryng numer of fxtons, mprng n effcent comprson etween prtcpnts. To compenste the synchroncty etween prtcpnts, we ncluded tme strems for equdstnt tme ntervls. For ech tme strem ntervl, the strem psses the fxton mges wth the respectve tme stmps. The resultng segments re fnlly comned to set of Bézer curves tht mrk the correspondng tmespns for ll prtcpnts. The tme strems re rendered n the ckground of the fxton mges wth n lterntng color scheme tht cn e djusted ndvdully. Fgure 4 shows n exmple of tme strems from four prtcpnts lookng t n nfogrphc (Fgure 6). In the exmple, the numer nd durton of fxtons for prtcpnt 5 nd prtcpnt 4 re smlr n the frst two seconds, thus the strem s less dstorted. After four seconds, prtcpnt 6 hs two longer fxtons ledng to stronger dstorton n the tme strems. Wth these tme strems, the synchroncty cn e compensted nd the tmelnes remn comprle. We llow the user to djust the selected tmespn ntervl. 3.2 Intercton The fxton mge chrt dsplys n overvew of consecutve fxtons wthout overlp. However, to mprove the vsul serch for mportnt events n the eye trckng dt, ddtonl ntercton technques re requred. Therefore, we provde lnked vew tht dsplys the dt wth estlshed scnpth vsulztons, flter optons to fde out fxton mges unmportnt for the current nlyss, nd lelng functon for selected fxtons Lnked Vew The ddtonl vew shows the complete stmulus n the stmulus vew (Fgure 5(e)). A gze reply of selected prtcpnts s 1 messge 2 () () Fgure 6: The two sttc stmul Areco messge 1 nd UAD nfogrphc 2 re nvestgted wth our vsulzton n ths pper. dsplyed durng the plyck of the recorded dt. The lnkng etween the stmulus vew nd the vsulzton s drectonl. Chngng the tme n the stmulus vew selects the respectve fxton mges n the vsulzton. Selectng fxton mge n the vsulzton sets the stmulus vew to the ccordng tmestmp. As t s common prctce, the fxton durton s encoded y the rdus of ndvdul fxtons. Addtonlly, lels re rendered nto the vsulzton durng plyck, llowng for the verfcton of the lelng process. Lookng t our exmple dt from the nfogrphc, we hve two fxtons from prtcpnt 0 nd 6 n the current tme step tht re leled s Green Sgn (Fgure 5(e)). By selectng consecutve fxtons of prtcpnt, the respectve prt of the scnpth s lso hghlghted n the stmulus vew Flterng The fxton-mge chrts provde good overvew of ll fxtons n dtset. However, we need support for selectng nd hghlghtng fxtons ccordng to dfferent propertes. In ths wy, t ecomes esy to nvestgte fxtons nd lel them. Therefore, we ntegrted n nterctve query nterfce tht llows us to flter ccordng to the followng ctegores:

5 Fgure 7: Vsul pttern of redng ehvor n our vsulzton. Mny short sccdes n redng drecton (red) pper together wth few () short nd () long regressons gnst the redng drecton (purple). F l t er1: m g es F l t er2:r e d ng F l t er3:r eg r es s ons Fgure 8: Three lels re defned for fxtons on the pcture (yellow), redng (red), nd regressons durng redng (purple). Wth three flter queres, out 80% of the fxtons cn e leled very effcently. The propertes fxton durton, fxton dstnce, nd sccde drecton cn e ppled s flter crter for selectng fxton mges. Hence, knowledge out certn eye movements cn e ppled to hghlght correspondng fxtons. We wll dscuss ths pproch n more detl n Secton The nlyst cn select reference mge from the vsulzton nd retreve smlr mges. To ths end, we provde two smlrty mesures. The frst s sed on the detecton nd mtchng of SIFT fetures [Lowe 1999] etween the reference nd the fxton mges. To normlze the smlrty mesure, we use the numer of mtchng fetures etween the reference mge wth tself. The second smlrty mesure s sed on hstogrm comprson wth the Bhttchryy dstnce [Bhttchryy 1946] etween the mges. We choose these two mesures ecuse of ther pplclty to dfferent nlyss tsks. Regons wth smlr structure cn e dentfed y SIFT fetures, wheres regons wth smlr color cn e dentfed y hstogrms. 4 Flter y fxton dt: Flter y mge smlrty: All flters cn e enled nd djusted ndvdully y seprte dls (see Fgure 5()). All enled flters re conctented y logcl AND connecton. The nlyst cn choose vlue nd djust the flter rnge wth the mouse to select the vlues ccepted y the flter. Fxton mges outsde of the selected rnges wll e fded out n the vsulzton, hghlghtng only the currently relevnt mges. In our exmple, reference mge from green sgn s selected nd the hstogrm smlrty s ctvted n the serch query. The query results show us the most smlr fxtons on green sgns n the pcture, sorted y ther temporl ppernce. 3.3 Lelng Supportng the dssemnton of nlyss results, we llow the nlyst to specfy lels tht cn e ssgned to fxtons (Fgure 5(d)). In contrst to ntersecton tests wth AOIs, lels cn descre more thn just regons or ojects, e.g., redng ehvor or tsk-specfc events (see Secton 4.1). Lels re vsle n the vsulzton through ther ssgned color n the ckground of the fxton mges nd n the stmulus vew through ther nme. Implementton Our vsulzton prototype s mplemented n C++ usng Qt 4.8. Clcultons on mge smlrtes were performed wth OpenCV The source code of our prototype cn e downloded3. Use Cses To demonstrte our vsulzton concept, we pply the technque to sttc (Fgure 6) nd dynmc stmul (Fgure 9). The dt ws recorded wth To T60XL nd fltered wth the To fxton flter wth stndrd settngs (velocty threshold = 35 pxels/smple; dstnce threshold = 35 pxels), s recommended y To4. Our current mplementton s desgned for sngle stmul wth dt from multple prtcpnts. However, our concept cn lso e extended to dt from multple prtcpnts nd multple stmul, s t would e the cse n experments tht re recorded wth mole eye trckng glsses. 4.1 Effcent Lelng of Fxtons For the pplcton to sttc stmul, we nvestgte test dtset recorded from 8 prtcpnts, lookng t Wkped pge out the Areco messge sent nto spce (see Fgure 6). A free-vewng tsk wth tme lmt of 90 s hd to e performed. In ths use cse, we wnt to show how our pproch cn e ppled to lel fxton dt very effcently. Leled fxtons re often necessry to evlute the effects of chnges n ndependent vrles on mesured dependent vrles sttstclly. Leled fxton dt could lso come from n utomtc lgorthm (e.g., for redng detecton [Cmpell nd Mglo 2001]) nd the nlyst mght e nterested n verfyng the results of the lgorthm. There re sclly two pproches to lel fxton dt: Defne res of nterest nd check for ech fxton f t les nsde one To Studo 3.2,

6 c e d f Fgure 9: Results for the Kte dtset [Kurzhls et l. 2014]. () Flterng for long fxtons does not led to cler result. Nevertheless, we cn see tht mny of the long fxtons occur when the prtcpnts looked t the kte. Furthermore, ther gzes moved n smlr drectons (vsle n the color of the rs) when they looked t the kte t the sme tme. () Flterng for long sccdes n upwrd drecton provdes clerer results. Most of these sccdes strt t the person controllng the kte. (c) To confrm ths gze ehvor, we use the mge smlrty flter to otn ll fxtons relted to ths person. We cn see tht ll susequent sccdes go n upwrd drecton. (d) We pck out one of these sequences to nlyze ths ehvor further. The fxton mges lredy show tht the gze moves etween the person nd the sky. (e) The respectve scnpth confrms ths. (f) Lookng t the respectve prt of the vdeo stmulus, we cn see tht the kte leves the cmer vew n upwrd drecton n ths vdeo sequence. of these res, or nspect ech fxton ndvdully nd ssgn n pproprte lel to t. Both pproches hve ther dvntges nd shortcomngs. The mn shortcomng for oth pproches s tht the defnton of res of nterest or the ssgnment of lels cn e very tme-consumng. We m for the second pproch, snce especlly for dynmc stmul, the nnotton of res of nterest s tedous preprocessng step. We tke dvntge of the flter query dscussed n Secton to mprove the nnotton step. Snce our vsulzton lredy provdes n overvew of ll fxtons of the dtset, we cn pply dfferent flter settngs to fnd smlr fxtons nd ssgn them to lel. We choose three dfferent lel ctegores: mge, redng, nd regressons. Imge descres fxtons tht le on the correspondng regon of the stmulus (Fgure 6, top-rght). Note tht n ths cse, the sme lelng could e cheved y defnng res of nterest on the mge regon. However, the dvntge of the ndvdul lelng comes wth the lels redng nd regressons. The men fxton durton durng redng s pproxmtely t 225 ms nd the men sccde length t 2, dependng on numer of fctors [Ryner 1998]. Regressons typclly occur durng redng nd re chrcterzed y sccdc jumps opposte to the redng drecton. Longer regressons typclly pper when the prtcpnts jump ck n the text, ether to red the next lne, or ecuse they dd not understnd the text. Fgure 7 shows typcl redng pttern n our vsulzton. The short fxtons under 500 ms on the text re followed y short sccdes under 100 px n redng drecton (red). Regressons (purple) pper durng the redng process ether to () re-nvestgte words (n ths cse the word Areco) or () red the next lne. In the overvew vsulzton (Fgure 8, top), we cn lredy see n the mges tht most of the 1793 fxtons le on regons wth text nd few fxtons on the pcture on the pge. Flter 1: By pckng one of the fxton mges wth pcture content, we otn the smlrtes of ll other fxton mges to ths reference. The mn dfference etween the text nd the mge regon les n ther hstogrms. Bsed on ths, we cn remove ll fxton mges under certn threshold nd mrk the remnng fxtons wth the mge lel (yellow). Flter 2: To dentfy redng fxtons, we strt wth hstogrm smlrty for fxton mge contnng mnly text. Wth these flter settngs for the mge smlrty, we cn now dd the nformton out the fxton durton, sccde length, nd sccde drecton to dentfy fxtons where redng fxton ppered. Therefore, we djust the flter dl for the drecton to spn of out 90 to the rght sde nd djust the other dls ccordngly. Ths leves us wth ll the fxtons for the lel redng (red). Flter 3: Fnlly, we keep the mge smlrty for text nd just chnge the sccde drecton flter to the left. By dectvtng the other flters, we select long nd short regressons. The resultng fxtons re leled s regressons (purple). After ths process, we hve leled out 80 percent of the fxtons wth only three flter queres n very short tmespn. For the remnng 20% of fxtons, we cn ether decde ndvdully whch lel they elong to, or perform new flter queres. The resultng leled dt could now, for exmple, e compred wth the results of clssfcton lgorthm for redng ehvor. 4.2 Explortve Dt Anlyss The use cse n Secton 4.1 requred the nlyst to pply prevous knowledge from the lterture out the propertes of fxtons for redng ehvor to perform the lelng tsk. In our second exm-

7 ple, we ssume tht the nlyst does not know wht specl ehvor to look for. Hence, n explortve dt nlyss cn help dentfy potentlly nterestng ptterns. In some cses, t s hrd to otn useful nformton y just lookng t the overvew vsulzton. In these cses, we cn pply dfferent flter methods to reduce the mount of vsle nformton. Our flter methods cover stndrd eye trckng metrcs lke fxton durton or sccde length. Fgure 9 demonstrtes the utlty of these flter methods. Flterng for long fxtons does not drectly led to cler pttern. However, t s vsle tht n mny cses, the prtcpnts looked t the kte when long fxton occurs. However, some of the long fxtons lso occur t the person controllng the kte. Whle the long fxtons on the kte re followed y sccdes n dfferent drectons, the long sccdes followng long fxtons on the person seem to go ll n upwrd drecton. Ths cn e verfed y flterng for long sccdes n upwrd drecton. We cn see tht lmost ll of them strt t the person. Ths cn e further confrmed wth the mge smlrty flter. To fnd n explnton for ths ehvor, we cn look t respectve sequence of sccdes. We cn see tht the gze seems to move etween the person nd the sky. Ths s lso confrmed y the scnpth vsulzton. However, the kte does not pper n the fxton mges. The respectve sequence of the vdeo stmulus provdes n explnton for ths. We cn see tht the kte left the cmer vew t the upper order nd tht the prtcpnts then looked t the person controllng the kte. From ths, t cn e ssumed tht the gzes prmrly followed the moton of the kte. However, when the kte leves the scene, the gzes move etween the person nd the sky. The prtcpnts seem to wt for the kte to repper. 5 Dscusson In n ntl qulttve user study, we presented our technque to three vsulzton experts from our nsttute wth experence n eye trckng nlyss. They were not nvolved n the development process of the vsulzton prototype. In seprte sessons, they tested the pplcton on three dtsets. Ech sesson took out 60 mnutes to nvestgte the dtsets. The frst dtset, Areco Messge, ws used s test dt, where the concept nd functonlty of the pplcton ws explned. For the second nd thrd dtset, UAD Infogrphc nd Kte, the experts were llowed to freely explore the dt nd serch for notcele events nd ptterns. Durng the explorton, the thnk loud method ws ppled to protocol the experts nlyss strteges. After the ntl explnton, the vsulzton ws nterpreted esly y the experts. The overvew of the vsulzton ws nvestgted shortly efore ll experts ppled the flter queres. Here, the strteges dffered etween the experts. We oserved tht the mn strteges for the frst serch queres were the serch for the longest fxton durtons, the longest sccdes, nd drect serch queres for mges smlr to fxton mges the experts consdered mportnt n the vsulzton. Although the vsulzton ws receved well y the experts, there re lso potentl shortcomngs. Bsed on the expert feedck nd generl consdertons, we dentfed the followng ssues. Long fxtons nd smooth pursut n vdeos: In contrst to sttc stmul, vdeo content cn chnge drstclly etween consecutve vdeo frmes. By ggregtng smpled gze dt nto fxtons, those chnges re lost. In our current mplementton we show the frst vdeo frme correspondng wth the tmespn of fxton. Therefore, ojects tht moved through the fxted re wthn one fxton tmespn cn only e dentfed n the vdeo re- Fgure 10: Comprson of gze strpes [Kurzhls et l. 2016] (top) nd fxton-mge chrts (ottom). The mrked tmespn n the gze strpes s covered y sngle mge n the fxton-mge chrt. ply. Smlr to ths prolem s the queston how smooth pursut cn e hndled y the vsulzton. Both scenros requre us to dsply more nformton out the underlyng vdeo frmes. For exmple, the ffected fxton mges could e splt nto mge sequences tht descre the stmulus context more detled. Those mge sequences could e stcked on the tmelne, or smll nmton loops could e ppled, preservng the dense lyout on the tmelne. Amguous lels: As wth mny utomtc lgorthms, n some cses the clculted smlrty etween mges mght e hgh, lthough the wtched content s dfferent. Ths mght hppen for exmple when two ojects re close to ech other nd lthough the prtcpnt looked t one oject, prts of the other oject cn e vsle n the cropped mge. In ths cse, the mge cn pper n oth flter queres, f we serch for the ojects seprtely. Therefore, we suggest tht the lelng process s performed y serchng for the mge smlrty of oth ojects nd fnlly, mguous fxtons wth more thn one lel cn e nvestgted ndvdully to solve the lelng prolem. The effcency of ths pproch could e mproved y n ddtonl vew tht lsts mguous fxtons. Addtonlly, new smlrty metrcs could e ntegrted nto the system, provdng etter ccurcy for specfc nlyss tsks. Comprson to gze strpes: Fgure 10 shows comprson of gze strpes [Kurzhls et l. 2016] nd our proposed technque for the sme prt of the Kte dtset, s gze strpes s the pproch tht s most smlr to ours. We cn see tht temporl synchroncty s reduced n fxton-mge chrts. Whle t s esy to see n gze strpes tht ll shown prtcpnts look t the sme prt of the stmulus, ths nformton s more dffcult to otn from fxton-mge chrts. The tme strems n the ckground llevte ths ssue to some degree. Furthermore, fxton-mge chrts requre more spce n vertcl drecton. However, the spce s used to provde ddtonl nformton. We cn see tht for the selected pont n tme (lck order), the gzes of ll shown prtcpnts proceed nto lower rght drecton (rown rs). Ths cnnot e drectly deduced from gze strpes. Fnlly, fxton-mge chrts exht much hgher scllty wth respect to the covered tmespn. The fgure shows tht lmost the full tme rnge covered y the gze strpes s represented wth sngle mge n the fxton-mge chrt.

8 6 Concluson We presented new vsulzton pproch to dsply nd compre fxton dt from multple prtcpnts. Our technque represents scnpths y mge sequences tht convey ddtonl nformton out fxton durtons, sccde lengths nd drectons, nd the locl context of the stmulus. Through the compct representton of fxtons, our pproch scles well wth respect to the durton of recorded eye trckng dt. We demonstrted tht our pproch llows n effcent lelng of fxtons. Furthermore, t supports the explortve nlyss of eye trckng dt nd enles the detecton of nterestng fxton ptterns. However, further evluton consderng the performnce nd cceptnce of the technque n comprson to estlshed methods wll e requred. So fr, we ppled our technque to sttc nd dynmc stmul wth pssve stmulus content. For future work, we pln to further evlute our technque for ctve stmulus content from nterctve pplctons nd mole eye trckng. For those scenros, fxton ptterns wll e hrder to nterpret, s the underlyng stmulus cn vry extremely etween prtcpnts due to chnges n the envronment (e.g., from dfferent lghtng). Acknowledgements Ths work ws funded y the Germn Reserch Foundton (DFG) s prt of the Cluster of Excellence n Smulton Technology (EXC 310/2) nd the Prorty Progrm SFB/Trnsrego 161. References ANDRIENKO, G. L., ANDRIENKO, N. V., BURCH, M., AND WEISKOPF, D Vsul nlytcs methodology for eye movement studes. IEEE Trnsctons on Vsulzton nd Computer Grphcs 18, 12, BHATTACHARYYA, A On mesure of dvergence etween two multnoml popultons. Snkhyā: The Indn Journl of Sttstcs 7, BLASCHECK, T., KURZHALS, K., RASCHKE, M., BURCH, M., WEISKOPF, D., AND ERTL, T Stte-of-the-Art of Vsulzton for Eye Trckng Dt. In EuroVs - STARs, BLASCHECK, T., JOHN, M., KURZHALS, K., KOCH, S., AND ERTL, T VA 2 : A vsul nlytcs pproch for evlutng vsul nlytcs pplctons. IEEE Trnsctons on Vsulzton nd Computer Grphcs 22, 1. BURCH, M., KULL, A., AND WEISKOPF, D AOI rvers for vsulzng dynmc eye gze frequences. Computer Grphcs Forum 32, 3, CAMPBELL, C. S., AND MAGLIO, P. P A roust lgorthm for redng detecton. In Proceedngs of the 2001 Workshop on Perceptve User Interfces, 1 7. DUCHOWSKI, A., AND MCCORMICK, B. H Gzecontngent vdeo resoluton degrdton. In Proceedngs of SPIE, vol. 3299, DUCHOWSKI, A. T., DRIVER, J., JOLAOSO, S., TAN, W., RAMEY, B. N., AND ROBBINS, A Scnpth comprson revsted. In Proceedngs of the Symposum on Eye Trckng Reserch & Applctons, DUCHOWSKI, A. T., PRICE, M. M., MEYER, M., AND ORERO, P Aggregte gze vsulzton wth rel-tme hetmps. In Proceedngs of the Symposum on Eye Trckng Reserch & Applctons, DUCHOWSKI, A A redth-frst survey of eye-trckng pplctons. Behvor Reserch Method, Instruments, & Computers 34, 4, GOLDBERG, J. H., AND HELFMAN, J. I Vsul scnpth representton. In Proceedngs of the Symposum on Eye Trckng Reserch & Applctons, HLAWATSCH, M., BURCH, M., AND WEISKOPF, D Vsul nlyss of eye movements y herrchcl flter wheels. In 19th Interntonl Conference on Informton Vsulston IV, HOLMQVIST, K., NYSTRÖM, M., ANDERSSON, R., DEWHURST, R., JARODZKA, H., AND VAN DE WEIJER, J Eye Trckng: A Comprehensve Gude to Methods nd Mesures. Oxford Unversty Press. HURTER, C., ERSOY, O., FABRIKANT, S., KLEIN, T., AND TELEA, A Bundled vsulzton of dynmc grph nd trl dt. IEEE Trnsctons on Vsulzton nd Computer Grphcs 20, 8, KURZHALS, K., BOPP, C. F., BÄSSLER, J., EBINGER, F., AND WEISKOPF, D Benchmrk dt for evlutng vsulzton nd nlyss technques for eye trckng for vdeo stmul. In Proceedngs of the Workshop Beyond Tme nd Errors: Novel Evluton Methods for Vsulzton, KURZHALS, K., HEIMERL, F., AND WEISKOPF, D ISeeCue: Vsul nlyss of gze dt for vdeo. In Proceedngs of the Symposum on Eye Trckng Reserch & Applctons, KURZHALS, K., HLAWATSCH, M., HEIMERL, F., BURCH, M., AND WEISKOPF, D Gze strpes: Imge-sed vsulzton of eye trckng dt. IEEE Trnsctons on Vsulzton nd Computer Grphcs 22, 1. LOWE, D. G Oject recognton from locl scle-nvrnt fetures. In The proceedngs of IEEE Interntonl Conference on Computer Vson, vol. 2, NYSTRÖM, M., AND HOLMQVIST, K Effect of compressed offlne foveted vdeo on vewng ehvor nd sujectve qulty. ACM Trnsctons on Multmed Computng, Communctons, nd Applctons 6, 1, 4:1 4:14. PONTILLO, D. F., KINSMAN, T. B., AND PELZ, J. B SemntCode: Usng content smlrty nd dtse-drven mtchng to code werle eyetrcker gze dt. In Proceedngs of the Symposum on Eye-Trckng Reserch & Applctons, RAJASHEKAR, U., CORMACK, L. K., AND BOVIK, A. C Pont-of-gze nlyss revels vsul serch strteges. In Proceedngs of SPIE, vol. 5292, RAYNER, K Eye movements n redng nd nformton processng: 20 yers of reserch. Psychologcl Bulletn 124, 3, RICHARDSON, D. C., AND DALE, R Lookng to understnd: The couplng etween spekers nd lsteners eye movements nd ts reltonshp to dscourse comprehenson. Cogntve Scence 29, 6, WEST, J., HAAKE, A., ROZANSKI, E. P., AND KARN, K eyeptterns: softwre for dentfyng ptterns nd smlrtes cross fxton sequences. In Proceedngs of the Symposum on Eye Trckng Reserch & Applctons,