The Cognitive Coprocessor Architecture for Interactive User Interfaces

Transcription

1 The Cognitive Coprocessor Architecture for Interactive User Interfaces George G. Robertson, Stuart I<. Card, and Jock D. Mackinay Xerox Pao Ato Research Center 3333 Coyote Hi Road Pao Ato. CA Abstract The graphics capabiities and speed of current hardware systems aow the exporation of 3D and animation in user interfaces, whie improving the degree of int,eraction as we. In order to fuy utiize these capabiities, new software architectures must support mutipe, asynchronous, interacting agents (the Mutipe Agent Probem), and support smooth interactive anima.tion (the Animation Probem). The Cognitive Coprocessor is a new user interface architecture designed to sove these two probems, whie supporting highy interactive user interfaces that have 2D and 3D animations. This a.rchitecture incudes 3D Rooms, a 3D anaogy to the Rooms system with Rooms Buttons extended to Interactive Objects that dea with 3D, animation, and gestures. This research is being tested in the domain of I?tformation Visuaization, which uses 2D a.nd 3D animated artifa.cts to represent the structure of informa.tion. A prototype, caed the Information Visuaizer, has been buit. 1 Introduction Rapid improvements in computer and periphera hardware in recent years have dra.matica.y increased the opportunities for buiding highy intera.ctive and sophisticated user interfaces for a wide variety of appications. The graphics capabiities a.nd speed of current, hardwa.re systems aow the exporation of 3D and a.nimation in user interfaces, whie improving the degree of interaction as we. However, in order to fuy utiize these Permission to copy without fee a or part of this materia is granted provided that the copies are not made or distributed for direct commercia advantage, the ACM copyright notice and the tite of the pubication and its date appear. and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to repubish, requires a fee and/or specific 1989 ACM O /89/001 /O010 $ capabiities in a systematic way, new software architectures are needed. Two probems, in particuar, need to be addressed by these new architectures. First, the degree (and compexity) of interaction increases rapidy as interfaces begin to dea with mutipe interacting agents (the Mutipe Agent Probem), Second, there is a trend toward increased use of interactive animation in interfaces because animation can decrease the cognitive oad on the user (the Animation Probem). This paper describes a new user interface architecture, caed the Cognitive Coprocessor, which provides systematic ways of deaing with both the Mutipe Agent Probem and the Animation Probem. After describing the Mutipe Agent Probem and Animation Probem in more detai, we describe an appication, caed Information Visuaization, which uses 2D and 3D animation to expore information and its structure. We then describe the basis, background, and structure of the Cognitive Coprocessor architecture. Managing coections of visuaizations, or virtua workspaces, is done with a 3D anaogy to the Rooms system [4], caed 3D Rooms. Interaction is accompished through Interactive Objects, which are simiar to Rooms Buttons, but have been extended to dea with 3D, animation, and gestures. A of these components are being integrated into a prototype system, caed the Information Visuaizer. The paper concudes with a discussion of open impementation issues and the benifits of the architecture. 2 The Mutipe Agent Probem The behavior of an interactive system can be described as the product of the interactions of (at east) three agents [a]: a user, a user discourse nzachine, and a task ma.chine or appication (see Figure 1). The user and the user discourse machine engage in a form of diaogue communication. This diaogue is not, of course, in the form of natura anguage, but in terms of utterances pecuiary suited to conversation between human

2 and machine. An interesting feature of such conversa? tions is that, unike human-human conversations, the utterances are between fundamentay dissimiar agents - a human on one end, and a machine on the other. The machine sends its utterances to the human prima.riy through graphica presentations (pictures or text) or maybe by sound. The human sends his or her utterances to the machine through manipuations of some sort of input device, such as a keyboard or a mouse. The asymmetries in the conversation refect (1) the highy deveoped perceptua abiities of the huma.n compa.red to the reativey crude perceptua abiities of machines, (2) the abiity of many machines to generate (at high speed) graphica information matched to this human perceptua abiity, and (3) the greater cognitive resources for semantic processing on the pa.rt of humans. In addition to widey dissimiar diaogue anguages, these a.gents operate with very different time constants. For exa.mpe, a search process in an appication a.nd the gra.p1ica. dispay of its resuts may be sow, whie the user s perception of dispayed resuts may be quite fast. The user interface must provide a form of impeda.nce ma.tching (deaing with different time constants) between the various agents as we as transate between different anguages of interaction. In addition to the three agents of Figure 1, there are increasing numbers of user interfaces that provide cognitive assistance to the user by adding inteigent agents of various sorts (e.g., an agent to fiter and sort your mai [S]). These additiona agents have their own time constants and angua.ges of interaction tha.t must be a.ccomodated by the user interface. Impedance matching can be difficut to a.ccompish architecturay because a agents invoved want rapid interaction with no forced waiting on other agents, and the user wants to be abe to rapidy change his or her focus of attention as new informa.tion becomes avaia.be. For exampe, if a user initiates a ong sea.rch tha.t provides intermediate resuts as they become avaiabe, the user shoud be abe to abort or redirect the sea.rch at any point (e.g., based on perception of the int,ermedia.te resuts), without waiting for a disp1a.y or search prc+ cess to compete. Another exampe, from the doma.in of text editors, is the abiity of an editor to skip a or part of a dispay update when sequences of comma.nds are typed ahead. The user interface a.rchitecture must provide a systematic way to manage the interactions of mutipe asynchronous agents that can interrupt a.nd redirect each other s work. 3 The Animation Probem Over the ast sixty-five years [$I, animation has come from a primitive art form to a very compex and effective discipine for communication. Wat Disney has been quoted as saying, Animation can expain whatever the mind can conceive. In fact, when couped with computers, animation has been used for a wide range of things, from the concrete of simuated reaity (ike animating the fight of a simuated airpane), through metaphor (ike animating the opening of windows on a computer desktop), to the abstract (ike animating agorithms [1,3] and Scientific Visuaization). Interactive animation is the most difficut of these techniques, because of its extreme computationa requirements. An important property of interactive animation is that it can shift a user s task from cognitive to perceptua activity, freeing cognitive processing capacity for a.ppication tasks. For exampe, interactive animation supports object constancy. Consider an animation of a compex object that represents some compex reationships. When the user rotates this object, or moves around the object, animation of that motion makes it possibe (even easy, since it is at the eve of perception) for the user to retain the reationships of what is dispayed. Without animation, the dispay woud jump from one configuration to another, and the user woud have to spend time (and cognitive effort) reassimiating t he new dispa,y. By providing object constancy, animation significanty reduces the cognitive oad on the user. The Animation Probem arrises when buiding a system that attempts to provide smooth interactive animation and sove the Mutipe Agent Probem. The difficuty is that smooth animation requires a fixed rate of guaranteed computationa resource, whie the highy interactive and redirectabe support of mutipe asynchronous agents with different time constants has widey va.rying computationa requirements. The user interface architecture must baance and protect these very different computationa requirements. 4 Appication: In format ion Visuaizat ion In order to put the Cognitive Coprocessor in a framework that addresses the probems set forth above, we has focused on a cass of appications that require effective soutions to those probems. The appication area is Information Visuaization, anaogous to Scientific Visuaization. In Information Visuaization, 2D and 3D a.nimated objects (or visuaizations) are used to represent botch information and the structura reationships 11

3 TASK USER USER MACHINE DISCOURSE MACHINE Figure 1: Tripe Agent, Mode of Human-Computer Interaction. of information. Direct manipua.tion of these objects causes changes in the actua structure of the information or changes in the a.ctua information. These object,s are paced in simuated 3D environments, and the user s task is to move around those environments, int,eracting with and manipuating the artifacts that he or she finds. Some interactions invoke time consuming operations (ike searching a database). Other interactions merey change the user s orienta.tion or some object s orientation in the environment. Figure 2 shows an exampe of a simua.ted 3D environment, caed the Ezporaiory, which has a whiteboard on the right wa and a tabe with some object paced on it. In this simpe exampe, the user s task is to move around the room, interacting with the objects encountered in order to discover their content and function. Information Visuaization can make use of a 3D environment because the aspect ratio and perspective of 3D make much better use of imited screen space when dispaying arge information structures. It requires a good soution to the Mutipe Agent Probem because of the varying time constants and computationa requirements of search, manipuation, and movement. And, it requires a good soution to the Anima.tion Probem, because animation is crucia to both movement of the user about the environment and movement of objects in the environment (crucia because the user needs the benefit of object constancy that animation provides in order to make sense out of movement of compex objects representing compex structura reationships). 5 Cognitive Coprocessor Sheridan [7], in his studies of interactive embedded automation, was the first to arrive at the tri-part division of Figure 1 and to expicate the possibie paradigms of interaction. But it seems cear that the three agent mode can be adapted and enhanced to serve as a mode for human-machine interaction generay [a]. From this point of view, interactions with computer workstations are simpy a specia case of a more genera framework that incudes the contro of automated airiners, eectric power pants, and space probes. Human interfaces to network-based computation, for exampe, introduce deays, bandwidth imitations, and interactive automation contro paradigms previousy encountered in embedded automation system contro. It has aso been pointed out [2] that inteigent agents can be introduced at different sites of the three agent mode, creating different casses of inteigent systems. The Cognitive Coprocessor is a user interface architecture that supports the three agent mode, the addi- 12

4 Figure 2: Exampe 3D-Room: The Exporatory. tion of inteigent agents, aradsmooth interactive animation. It incudes management of mutipe asynchronous agents that operate with different time constants and need to interrupt and redirect each other s work. It s name is derived from the cognitive assistance it provides the user and other agents, as we as the coprocessing of mutipe interacting agents that it supports. Figure 3 shows the basic structure of the Cognitive Coprocessor Architecture. The three ayers correspond to the agents of the three agent mode (Figure 1). The basic contro mechanism (inner oop) of the Cognitive Coprocessor is caed the Animation Loop. It m&ntains a Task Queue, a Dispay Queue, and a Go,vemor. Buit on top of the Animation Loop is: an informa,tion workspace manager (and support for 3D simuated environments), caed SD Room; support for navigating around 3D environments; and support for Inferncive Objects, which provide basic I/O mecha.nisms for the user interface. The task ma.chine (which, for the Information Visuaization appica.tion, is a coection of visuaizers) coupes with the Cognit,ive Coprocessor in various ways, as iustrated in Figure Animation Loop The Animation Loop is the basic contro mechanism for the Cognitive Coprocessor Architecture, and maintains: (1) a Task Queue, which contains pending computationa tasks from various agents; (2) a Dispay Q,ueue, which contains pending instructions from various agents about how the screen shoud be painted on the next animation oop cyce; and (3) a Governor, which keeps track of time and aows for adjustments to anima.tions to keep them smooth. The architecture provides a fra.mework for agents. Bowever, it is cear that agents must be designed carefuy to fit in that framework. There are guideines for defining agents that specify how they shoud interact with and use the animation oop. For instance, these guideines indicate the size of tasks and dispay instructions permitted during any one cyce of the animation oop. Agents must be prepared to be interrupted during a task and redirect or abort their efforts. This can be accompished by either decomposing tasks into sma pieces and aowing one piece to be executed during each animation cyce (ike coroutines), or by spawning processes that can be interrupted by interprocess communication (ike mutiprogrammed systems). In either case, the architecture provides the mechanisms and guideines to make it easy to construct agents that function we in this environment, but does not force agents to conform to the guideines. The Tusk Qveze may contain three different kinds of tasks: procedures, ight-weight process spawn requests, and heavy-weight process spawn requests. The heavyweight processes are supported by the underying operating system (e.g., Unix), share no memory with the rest of the architecture, are communicated with by means of interprocess communication, and may be aborted at any time by the architecture (by kiing the process). These are most appropriate for ong running tasks or for parts of agents that aready exist in some other software package (e.g., a search procedure in a database manager). The ight-weight processes are supported by the underying anguage (e.g., extended Common Lisp), share memory with the architecture, and are communicated with by means of either shared memory or ightweight interprocess communication. These are most appropriate for medium and ong term tasks that are more 13

5 Task Machine 3D Navigation ++ 3D Rooms User Discourse Machine Task Queue Dispay Queue User Figure 3: Cognitive Coprocessor Architecture. 14

6 fuy integrated with the rest of the architecture (e.g., a search procedure over a data structure buit by an integrated visuaizer). Finay, task procedures are executed to competion, and are expected to be very short. They are most appropriate for monitoring the behavior of spawned processes and for short processing tasks (e.g., checking on the status of a spawned search pro cess). Conceptuay, the Dispay Queue contains procedures for painting the various aspects of t,he disp1a.y. In pra.ctice, the architecture is object oriented, hence the Dispay Queue contains a of the objects to be drawn. The, architecture is taiored for doube-buffered auimation (required for smooth animation), hence a objects are redrawn on each animation oop cyce. When singebuffered animation is necessary, the a.rchitecture requires appication agents to provide a. method for drawing ony the changes for each object, and to pace ony changed objects on the Disp1a.y Queue. The Cognitive Coprocessor and agents shoud be designed so that the animation oop runs a.t ten cyces per second or faster (thirty is idea), to provide smooth a.nimation. The Governor provides a, timing mechanism so that agents can te when the a.nimation oop cyce is taking onger than desirabe for smooth animation. Agents are expected to ta.ior their anima.tions to run at a fixed rea-time rate, hence shoud use the Governor to determine how much dista.nce to cover in an anima.- tion for each animation oop cyce. This prevent,s an animation from running either too fast or t(oo sow. For exampe, a ba bouncing in a room shoud be designed to bounce at a fixed rate (e.g., one bounce per secoud). The distance the ba moves on each a.nimation step depends on how fast the system a.s a whoe is running (i.e., the ba must move further in one step if the syst,em is running sower than norma). Given the other components of the architecture, the Animation Loop is quite simpe. It has two major states: the ide state and the anima.tion state. In the ide sta.te, nothing is changing on the dispay, and the task queue is empty, so the animation oop simpy suspends, waiting for input from the user or one of the spa.mned agent task processes. In the animation state, the foowing steps are taken: 1. Update the Governor s cock; 2. Process any pending input from the user or spawned task processes; 3. Process any pending tasks on the Task Queue; 4. Draw the new dispay: Cear the next dispay buffer (doube-buffered animation), Draw each of the objects on the Dispay Queue, Swap the dispay buffers; 5. Loop (go to step 1). This basic architecture with its guideines for agent construction provides the support we need for managing smooth interactive animation and mutipe asynchronous agents. 5.2 Virtua Workspaces: 3D Rooms Most of the Information Visuaization appication described earier can be supported by the Cognitive COprocessor Architecture described above. However, two additions are needed to fuy support the appication. First, we need support for simuated 3D environments in which to pace our 2D and 3D visuaizations. And second, we need a way to manage arge coections of visuaizations. Both of these are provided by a system we ca 3D Rooms. The probem of managing arge coections of information visuaizations is very simiar to the probem of managing arge numbers of windows on a computer desktop. The Rooms system [4] takes advantage of the fact that a person s dynamic access to information tends to exhibit ocaity of reference, accessing sma custers of information at a time. Henderson and Card use this fact as the basis for managing windows in custers, or virtua workspaces caed Rooms, to significanty reduce spa,ce contention in window-based user interfaces. In our Information Visuaization appication, the virtua workspaces are sma custers of 2D and 3D visuaize tions. Since we wish to pace our visuaizations in 3D environments, it seems appropriate to use a metaphor of a physica room, and make each 3D environment be a t,hree-dimensiona simuated room. The extension of the Rooms system to the 3D Rooms system is then natura. The 3D Rooms system manages coections of 3D artifacts (representing abstractions or simuations of informatiou or information structures) instead of windows, a.nd keeps each coection in a simuated 3D room D Rooms Cass Hierarchy The 3D Rooms system is object oriented. The basic object cass hierarchy is shown in Figure 4. There is a basic cass, caed Room, and two major sub-casses, 2D-Room and SD-Room. Sub-casses are created for ea.ch widey used coection of visuaizations. From each visuaization-coection cass, individua instances of rooms are created. 15

7 Room Visuaization A Room D isuaization B Room E Visuaization C < Room F Room G Figure 4: Exampe 3D Rooms Object Cass Hierarchy. A basic Room has the foowing characteristics (object sots): a name, a ist of interactive objects (described in the next section), and a ma.pping from keystrokes to seectabe interactive objects. A basic Room has methods for: 0 Initiaization, Enter-Room, Exit-Room, Draw-Room, Draw-Contents-Of-Room, Draw-Interactive-Objects, Hande-Mouse-Seection, and Hande-Menu-Seection. A 2D-Room adds some characteristics: a size (x and y) and a background coor. A 3D-Room adds: a size (x, y, and z), coors for four was, the foor, a.nd the ceiing, a foca ange (for 3D perspective contro), a.nd home positions for the body (see na.viga.tion discussion beow), direction the body is facing, and direction the gaze is facing. Visuaization casses a.dd their own cha.racteristics and shadow the basic methods to a.chieve the desired resuts. The Exporatory in Figure 2 is an exampe of a 3D- Room. In addition to the whiteboard and tabe with some object on it, there is a door on the back wa (not shown), which eads to another room. The user moves around this room interacting with the objects (the whiteboard, the object on the tabe, and the door). 5.4 Navigation in 3D Rooms In order to make interaction with 3D artifacts in 3D Rooms effective, we need a gracefu way for the user to move about a simuated 3D room. We have chosen to use the metaphor of waking around a room, and have kept room sizes and egocentric motion contros as cose to famiiar physica settings and movement as possibe. The particuar motion contros were partiay derived from a theory of input devices [5]. They provide three virtua joysticks with one mouse button. The defaut joystick contros body movement forward and backward, and body rotation eft and right. The other two virtua joysticks are tied to icons on the screen (see Figure 2). One icon represents the direction the body is facing, and its joystick contros body movement up, down, eft, and right. Cicking on the body icon wi return the body to its norma standing position. The second icon represents the direction the gaze is facing (the orientation of the head), and its joystick contros head movement up and down and head rotation eft and right, with natura human imitations (e.g., you cannot turn your head more than about 80 degrees). Cicking on the gaze icon wi bring the head to a face forward orientation. These navigationa contros are uniform across the entire Cognitive Coprocessor Architecture. In order to provide proper feedback and smooth animation of egocentric movement in a 3D-Room, the navigation mecha.nism is integrated into the basic animation oop in the draw step (step (4) above). 16

8 5.5 Interactive Objects (Generaized Buttons) The Rooms system [4] h as devices, caed Buiions, which are used for a variety of purposes, such a.s movement, new interface buiding bocks, and task assistance. A Button has an appearance (typicay, a bitmap) and a seection action (a procedure to execute when the Button is pressed ). The most typica Button in Rooms is a door - when seected, the user passes from one Room to another. Buttons are abstractions that ca.n be pa.ssed fromone Room to another, and from one user to another via eectronic mai. In our 3D Rooms system, we provide a generaization of Rooms Buttons, which we ca.11 Interactive Objects. Interactive objects are extended to dea with gestures, animation, 2D or 3D a.ppea.ra.nce, and an extensibe set of types. An interactive object ha.s an appea.ra.nce; but. has a, draw method instead of a bitma.p, since it. ma.y a.ppear as a 2D or 3D object. The notion of seection is generaized to aow mouse-based gestura input in addition t,o simpe pressing. Whenever a user gestures a.t a.11 interactive object, a gesture parser is invoked that interprets mouse movement and cassifies it as one of a sma set of easiy differentiated gestures (e.g., press, rubout, check, and throw eft, right, up or down). Once a. gesture has been identified, the interactive object dispa.tches t,o the appropriate method (i.e., there is a method for ea.& of the gestures). These gesture methods are specified when the interactive object is created. The gesture parser ca.n be easiy extended to aow additiona gestures a.nd gesture methods, as ong as the new gestures are easiy differentiated from a other gest,ures. Interactive objects can be animated by specifying a home position and a seected position. When t,he object is seected, it fys to its seected position (which may invove a number of rota,tions and tra.osa.tions). One exampe of an animated intera.ctive object. is an editabe text input object which is at home on the foor of the room, but fys to a vertica.1 orienta.tion when seected so that editing is easier. Another exa.mpe of an animated interactive object is a door t,hat, opens when the doorknob is touched. There are a number of types of interactive object,s. In the current impementation, these incude sta.tic text, editabe text, date entry, number entry, set seection, checkmark, simpe button, doors, and thermometers (for feedback and progress indicators). The set of types supported for interactive objects can be easiy extended. 6 An Information Visuaizer: Discussion The Information Visuaizer is a prototype system that incorporates a of the architectura features described for soving the Mutipe Agent Probem and Animation Probem, in the context of the Information Visuaiza tion a.ppication. It provides the integration of: 2D and 3D animated visuaizations; 3D Rooms to manage the virtua workspaces and provide a 3D environment; interactive objects to provide generaized extensions to Buttons for user interaction; and the Cognitive Coprocessor Architecture as the basic underying architecture to support mutipe asynchronous agents and smooth interactive animation. To date, we have contructed six 3D rooms in this system, providing various 2D and 3D representations of information and the structure of information. The Exporatory in Figure 2 is an exampe of one of those rooms. The system we have buit runs on a hardware base (Siicon Gra,phics Iris) that supports 3D graphics and doube-buffered animation. There are a number of impementation issues surrounding the probem of porting the Information Visuaizer to other hardware bases. One major probem has to do with 3D graphics standards. It is desirabe to use a standard to ease the porting probems, but undesirabe if the standard imposes impementation ayers that sow the system down a.nd bender smooth animation (with its rea-time constraints). One soution that we are exporing is the use of a portabe 3D graphics ayer that compies out intermediate impementation ayers. Another major probem has to do with support for both singe-buffered and doube-buffered animation. As mentioned earier, we a.re exporing the required extensions to 3D Rooms draw methods to support singe-buffered animation. Whether it can be done effectivey enough to support smooth animation is sti an open question. The basic Cognitive Coprocessor Architecture has proven to be very effective in providing a good framework for working on the Information Visuaization appica.tion. It has been easy to define new visuaizations once they were conceived (conceiving new visuaizations is quite a difficut task, and is not addressed here). And, it has been easy to achieve the desired kind of responsive redirectabe interaction with mutipe agents, whie aso a.chieving the desired smooth interactive animation. References [] Brown, M.H. (1987). Agorithm animation. MIT Press. 17

9 [2] Card, SK. (1989). H uman Factors and Artificia Inteigence. In P.A. Hanccock & M.H. Chigne (Eds.), InteIZigent interfaces: Theory, research and design. Esevier Science Pubishers B.V. (North- Hoand). [3] Duisberg, R.A. (1988). Animation using tempora constraints: an overview of the animus system. Human-Computer Interaction, 3, , Lawrence Erbaum. [4] Henderson, D.A., & Card, S.K. (1986). Rooms: the use of mutipe virtua workspaces to reduce space contention in a window-based graphica user interface. ACM Transactions on Graphics, 5, 3, , Juy, [5] Mackinay, J.D., Card, S.K., & Robertson, G.G. (in preparation) A semantic anaysis and taxonomy of input devices. To appear in Human-Computer Interaction, Lawrence Erbaum. [S] Maone, T.W., Grant K.R., Turbak, F.A., Brobst, S.A., &z Cohen, M.D. (1987). Inteigent information-sharing systems. Communica- -tions ACM, 30, 5, May 1987, [7] Sheridan, T.B. (1984). Supervisory contro of remote manipuators, vehices and dynamic processes: experiments in command and dispay aiding. Advances in Man-Machine System Research, 1, , JAI Press. [S] Thomas, F., & Johnston, 0. (1981). Disney animation - the iusion of ife. Abbevie Press (New York). 18