Abstrct Better Relising Direct Mnipultion C.A.D H. Gough, R. Green, nd M. Billinghurst Hitlb NZ, University of Cnterbury Emil: Christin.gough@hitlbnz.org Direct Mnipultion is n pproch to designing user interfces, which forms the bsis of Grphicl User Interfces. Despite the importnce of the concept, no mthemticl model of Direct Mnipultion hs yet been developed. This pper proposes model of Direct Mnipultion, which reltes cognitive distnce with user fmilirity nd the novel concepts of tech bis, velocity nd inerti. Keywords: Direct Mnipultion, Usbility. 1 Introduction While the technology of minstrem computer-bsed systems hs chnged significntly over the lst few decdes, the techniques used to interct with them hve remined firly sttic. Tody s Grphicl User Interfces (GUIs) re, s were the originl implementtions, ttempts to relise the idels of Direct Mnipultion 1,2 (DM). Given the importnce of the concept of DM in the context of user interfce (UI) design nd reserch, it is surprising tht there is no forml mthemticl model of DM. uch model would be invluble for optimising nd evluting UIs nd in the development of techniques for bridging of the gulfs of execution nd evlution 2 further thn tht which is possible with trditionl GUIs; such s is intended in the reserch fields of Tngible User Interfces (TUIs), Perceptul User interfces (PUI), Augmented Relity (AR) nd Virtul Relity (VR). This pper presents model of DM tht reltes cognitive distnce with user fmilirity nd concepts of tech bis, velocity nd inerti. 2 Directness The senstion of incresed usbility nd interctivity provided by good DM user interfce is known s directness 1,2. The components of directness re the cognitive distnce between the user nd the computer (), nd certin user-relted fctors (U) which interct to provide senstion of enggement for the user. The gol of DM interfce is therefore to provide the optiml senstion of directness by minimising cognitive distnce nd mximising enggement. 2.1 Cognitive Distnce Cognitive Distnce 2 is mesure of the gulfs of execution nd evlution the conceptul gp between the user s ides nd intentions, nd the wy in which they re expressed to, or represented by, the system. A lrge distnce is representtive of lrge gulf of execution or evlution, signifying tht lot of cognitive lod is incurred in trnslting between the user s intentions nd the system s representtions, or vice vers. Tht is, lrge distnce of execution mens it is reltively difficult for the user to express their query or desires to the system, nd lrge distnce of evlution indictes lot of work for the user to interpret output from the system. Figure 1: An overview of the vrious components of cognitive distnce. A simple model of cognitive distnce would be the summtion of the semntic (x s ) nd rticultory (x ) components of both the gulf of execution (i) nd evlution (o). There is much reserch motivted by the improvement of the experience of directness by minimising cognitive distnce 3, 4, 5, 6, 7, 8, 9, 10. The field of Tngible User Interfces 3, 4, 5 seeks to bridge the gulfs of execution nd evlution by imbuing grspble objects with significnce to the system.
The MIT s tngible medi group hs produced mny excellent exmples of such work, such s in the cse of Illuminting Cly 10, where the user performs lndscpe nlysis nd design work by intercting with soft, putty-like substnces. The geometry of this cly is cptured in rel time using lser scnner, nd the resulting nlyses re displyed both on surrounding disply devices nd projected directly bck on to the cly itself. Figure 2: The MIT Tngible Medi Groups' Illuminting Cly TUI. The field of Augmented Relity lso seeks to bridge the gulfs of execution nd evlution further thn tht which is possible with trditionl GUIs, but does so by ugmenting the user s senses with technology. For instnce, there re mny exmples of medicl reserch 11,12,13 tht utilises AR technology to ssimilte informtion from technologies such s ultrsound scnning nd presents it to the surgeon in such mnner tht it ppers s though they cn see inside their ptient s they perform the opertion, even during procedures such s biopsies nd keyhole surgery where such visibility is not possible. Figure 3: AR-supported surgery. The surgeon is given the impression of "seeing inside" their ptient. However, despite the seemingly inevitble benefits, such reserch often hs difficulty minimising cognitive distnce s much s expected. In 2004 Cludi Nelles 8 conducted comprtive nlysis of two pproches to uthoring content for AR/VR pplictions. The first system, clled itar, ws fully tngible pproch. The second, Ctomir, resembled more trditionl content uthoring tool such s Mcromedi s flsh - but dded the option of viewing nd performing bsic interctions with scenes immersively if desired. In her conclusion, Cludi sttes tht [by] exmining the overll outcomes for efficiency, errors nd plesurbility [sic] observed with ll prticipnts, it would seem s if itar [the tngible pproch] is the more usble of the two tools. But, this conclusion is overly simplistic, nd it is necessry to go into more detil to come to meningful conclusions regrding the reltive usbility of both tools. The fct tht the tngible pproch should be the overll winner is to be expected being provided the senstion of physiclly mnipulting elements of the content being creted is rgubly more direct thn tht of engineering the content vi complicted softwre suite. The surprise is the need to go into more detil to come to meningful conclusions. If itar were the more direct nd engging interfce, it should demonstrte cler trend towrd this in ll tests. But, s stted in the conclusion, this is fr from the cse. Despite being the overll winner in terms of speed, errors, intuitiveness nd being fun to use; most users stted tht they would still prefer to work with the trditionl pproch. A similr study into immersive content uthoring by Gun Lee 9 showed similrly mixed results, nd further reserch by Okley Buchmnn 18 finds tht n AR system for conducting geogrphic surveys gin fils to deliver the expected usbility gins. uch results indicte tht the reltionship is n inequlity rther thn the trditionlly ssumed equlity: is+ i+ os+ Considertion must therefore be given to the spects of directness tht re responsible for this inequlity. 2.2 Tech Bis Tech bis (T) is mesure of how well given device succeeds in the role for which it is intended. Mture technologies re effective t providing their intended experience nd s such hve high tech bis. Conversely, less commonplce technologies often hve reltively low tech bis. As n exmple, in typicl grphics worksttion sitution, the intended role of modern CRT disply is simply to provide high qulity 2D imge. In the sme sitution, mouse is intended to trck the user s hnd movements. These devices re very good t fulfilling these roles the imges displyed by o (1)
modern CRTs re generlly of high resolution nd hve mny colours nd high refresh rte, nd modern mice trck user s input very ccurtely. These devices would therefore ll exhibit very high tech bis in the role described. In typicl AR scenrio, the user my wer hed mounted disply (HMD) with n LCD disply for ech eye nd n ttched cmer. The system will process the view from the cmer, overlying computer generted grphics on the scene nd displying the result to the user. In this cse, the HMD s intended role is to ugment the user s perceptions nd give the impression tht the imgery is ctully present in the rel world. Due to the reltively bd imge qulity nd field of view of current HMDs, this senstion is not s strong s might be hoped, mening tht in this cse the HMD will exhibit very low tech bis. On the other hnd, if the intended role of the HMD ws to simply provide stereoscopic imge then its tech bis would be higher lthough still not s high s in the cse of CRT disply displying 2D imge. By ssigning vlue T:(0 < T < 1) for the tech bis of the gulfs of execution (Ti) nd evlution (To) we chieve the following: is + i i= Ti os + o o= To The minimum ttinble distnce is therefore determined by the semntic nd rticultory components, nd the degree to which it is possible to chieve this theoreticl minimum is governed by the tech bis of the hrdwre used. This pper uses two lyers of interction semntic nd rticultory, but other common configurtions could be used 14, 15, 16, 17. 2.3 User Fctors DM is reltionship between the user nd the system 2. Therefore there is need to tke humn fctors into ccount. There re mny user-relted fctors tht my ffect perceived distnce, but the most importnt of these tht my predictbly be modelled is user experience. The user s sense of directness will be inversely proportionl to their level of experience 1, 2 with the system becuse, s users become fmilir with the interfce, less cognitive effort is required to express their desires 2. This is, in effect, n cquired bridging of the gulfs by the user. U = 1 F (2) (3) (4) 2.4 Index of Distnce These reltionships cn be expressed with two indexes. The first of these is the Index of Distnce (), which my be used on its own to predict the distnce proposed user interfce my present. In most cses the primry im of developing n interfce is to minimise distnce irrespective of user experience. is + = Ti i os+ + To 2.5 Index of Directness o The second index is the Index of Directness (D), which scles the index of distnce by user fmilirity F:(0 < F < 1). D= F Which, when expnded, gives us the following: 1 is+ D= F Ti i os+ o + To This describes how direct given user perceives given implementtion of given user interfce to be, rther thn n indiction of the theoreticl cognitive distnce between the user nd the interfce. This is n importnt mesure when deling with specific, specilised user scenrio, where the overll directness my be more relevnt thn the cognitive distnce lone. 3 Appliction Due to the inherent difficulties of deriving vlues for coefficients in the model, evlution of indices using this model need to be reltive rther thn bsolute. For exmple, n index of directness computed for one cse cn be directly compred with nother, only when cre is tken to use the sme scles, ssumptions nd methodology in both cses. 3.1 Velocity of Mixed Distnce Interfces (5) (6) (7) In pplictions where the user is exposed to mixed distnce interfces, vrious elements of the interfce hve differing distnces. A good exmple is tht of recording studio ppliction, where prt representing the most commonly performed subset of ctivities is implemented tngibly s mixing desk using motorised sliders, nd the remining functionlity is implemented vi trditionl GUI, mouse nd keybord.
order to reduce distnce. It would be possible to engineer new interfce tht gretly reduced distnce using the Index of Distnce; but in doing so, much of the cquired directness of the system by the controller my be lost. In this cse the index of directness should be used insted, in order to ssess the improvements in light of the inerti of the controller using the system. Figure 4: A typicl recording studio configurtion provides good exmple of n effective mixed distnce user interfce. uch mixed-distnce interfces re sensible pproch to improving directness, s they llow commonly used subset of tsks or opertions to hve lessened cognitive distnce without scrificing the flexibility of more trditionl user interfce for the less common tsks. In such cses, it is useful to consider the chnge of distnce tht the user must overcome when switching focus between the interfce elements. uch vritions in distnce within n interfce cn be described s velocity. By tking weighted verge of the Index of Distnce for ech of the interfce types, we cn derive single overll Index of Distnce nd Index of Directness for the whole interfce. This in turn mens the theoreticlly optiml blend of interfce types cn be determined using liner progrmming. 3.2 Inerti If user interfce is significntly ltered in order to improve distnce, it must be determined if the gins in directness due to decresed distnce re greter thn the loss of directness cused by the decresed user fmilirity. A smll improvement in the distnce of system used by very expert users my not be enough to counter the expertise lost in chnging the interfce, resulting in net loss of perceived directness to the user. Thus, ny reductions of distnce in n existing user interfce must be lrge enough to overcome the inerti of the users experience if it is to be worthwhile improvement without requiring relerning by the users. For exmple, ir trffic controllers spend long time ttining expertise in using their systems. Becuse these systems re complex nd becuse the sfety of hundreds of lives relies on their effective use, there is much reserch on improving the user interfces in It is possible to rgue tht the primry focus should lwys be tht of directness, s new systems my be re-lerned nd thus, with time, new expertise my be joined with the decresed distnce to chieve the most optiml possible usbility. But consider tht in some cses the user my hve so much inerti tht it is lmost impossible to overcome. For exmple, surgeons re provided importnt informtion vi uditory cues during n opertion, such s hert rte. urgeons become so expert t using this system tht their use of the interfce is lmost completely subconscious. If the interfce were re-engineered in such wy tht this informtion ws no longer provided, it could result in life-thretening performnce decreses for the surgeon tht re unble to be re-lerned. Any replcement would in essence be substitute, rther thn replcement, for the uditory pproch. 4 Conclusion This pper proposes mthemticl model for relting vrious spects of Direct Mnipultion in order to gin better understnding of how to mximise perceived directness in user interfces nd design the most effective user interfce. The model introduces n Index of Distnce nd n Index of Directness, s well s the novel concepts of tech bis, velocity, inerti nd definition of mixed user interfces. The gol is to further understnd how to optimise nd quntittively compre nd predict user interfces. This model is being further reserched with the development nd nlysis of specific cse studies for ongoing verifiction of the reltionships described, identifying vlues for the vrious coefficients of the model by isolting ech fctor nd to determine ny further contributing fctors to directness. This reserch is lso pursuing investigtions to further define vlues for vrious coefficients to enble direct comprison of the cse studies used. Initil reserch focuses on the forml verifiction nd identifiction of the reltionships described, but emphsis will be plced on mixed user interfces nd the role of velocity in UIs.
5 References [1] hneidermn, B., The Future of Interctive ystems nd the Emergence of Direct Mnipultion, Behviour nd Informtion Technology 1982, v.1 n.3, pp. 237-256 [2] Hutchins, E. Holln, J. Normn, D., "Direct Mnipultion Interfces", Humn-Computer Interction, Volume 1, 1985. pp. 311-338 [3] Fitzmurice, G. "Grspble User Interfces", Grdute Deprtment of Computer cience, University of Toronto, 1996 [4] Ishii, H. nd Ullmer, B., "Tngible Bits: Towrds emless Interfces between People, Bits nd Atoms," Proceedings of Conference on Humn Fctors in Computing ystems (CHI '97), ACM, Atlnt, Mrch 1997 [5] Fitzmurice, G., Ishii, H., Buxton, W., "Bricks: Lying the Foundtions for Grspble User Interfces," Proceedings of Conference on Humn Fctors in Computing ystems (CHI '95), ACM, Denver, My 1995 [6] Antifkos,. Improving Interction with Context-Awre ystems, elected Redings in Vision nd Grphics, volume 35, 2005 [7] Ishii, H., Kobyshi, M., Grudin, J. "Integrtion of Interpersonl pce nd hred Workspce: ClerBord Design nd Experiments," ACM Trnsctions on Informtion ystems (TOI), ACM, Vol. 11, 1993 [8] Nelles, C. Grphicl vs Tngible User Interfce, unpublished PhD thesis, eingereicht m Fchochschul- Diplomstudiengng, Medientechnik und Design, Hgenburg, 2005 [9] Lee, G. Immersive Authoring of Virtul Worlds, Division of Electricl nd Computer Engineering, Pohng University of cience nd Tehcnology, Kore, 2005 [10] "Illuminting Cly: A 3-D Tngible Interfce for Lndscpe Anlysis"- Ben Piper, Crlo Rtti* nd Hiroshi Ishii, CHI 2002, Conference on Humn Fctors in Computing ystems, Minnepolis, Minnesot 20-25 April 2002 [11] Rosenthl, M., A. tte, J. Lee, G. Hirot, J. Ackermn, K. Keller, E. D. Pisno, M. Jiroutek, K. Muller, nd H. Fuch - "Augmented Relity Guidnce for Needle Biopsies: A Rndomized, Controlled Tril in Phntoms," Proc. Medicl Imge Computing nd Computer-Assisted Intervention 2001. [12] Fuchs, H., M. A. Livingston, R. Rskr, D. Colucci, K. Keller, A. tte, J. R. Crwford, P. Rdemcher,. H. Drke, nd A. A. Meyer, MD. Augmented Relity Visuliztion for Lproscopic urgery, Proc. First Interntionl Conference on Medicl Imge Computing nd Computer- Assisted Intervention, 934 943. [13] tte, A., M. A. Livingston, G. Hirot, W. F. Grrett, M. C. Whitton, H. Fuchs, nd E. D. Pisno. Technologies for Augmented- Relity ystems: Relizing Ultrsound- Guided Needle Biopsies. Computer Grphics: Proc. IGGRAPH 96, 1996 [14] Frohlich, D, The history nd future of direct mnipultion" Behviour & Informtion Technology 12, 6, 1993. pp. 315-329. [15] Neilsen, J, A Lyered Interction Anlysis of Direct Mnipultion, 1992 [16] Hix, D. nd Hrtson, H., Developing User Interfces. John Wiley & ons, Inc, 1993 [17] Tylor, M., Lyered protocol for computerhumn dilogue, I: Principles. Interntionl Journl of Mn-Mchine tudies, 28, 1988. [18] Okley s PHD