Getting More Out Of Programming-By-Demonstration

Transcription

1 Papers CHI MAY 1999 Getting More Out Of Programming-By-Demonstration Richard G. McDanie and Brad A. Myers HCI Institute, Schoo of Computer Science Carnegie Meon University 5000 Forbes Avenue Pittsburgh, PA USA { richm, barn ABSTRACT Programming-by-demonstration (PBD) can be used to create toos and methods that eiminate the need to earn difficut computer anguages. Gamut is a PBD too that nonprogrammers can use to create a broader range of interactive software, incuding games, simuations, and educationa software, than they can with other PBD toos. To do this, Gamut provides advanced interaction techniques that make it easier for a deveoper to express a aspects of an appication. These techniques incude a simpified way to demonstrate new exampes, caed nudges, and a way to highight objects to show they are important. Aso, Gamut incudes new objects and metaphors ike the deck-of-cards metaphor for demonstrating coections of objects and randomness, guide objects for demonstrating reationships that the system woud find too difficut to guess, and tempora ghosts which simpify showing reationships with the recent past. These techniques were tested in a forma setting with nonprogrammers to evauate their effectiveness. Keywords End-User Programming, User Interface Software, Programming-by-Demonstration, Programming-by-Exampe, Appication Buiders, Inductive Learning, Gamut. INTRODUCTION Gamut is an innovative too for buiding interactive software ike games, simuations, and educationa software. Much of the effort invoved in producing software in this domain is not in programming the appication s ogic but in providing the engaging background, artwork, and gamepay that keeps the users interested. Artists and educators who coud produce such materia are often unabe to program computers. Thus, toos which eiminate the burden of programming whie providing a wide range of capabiities are desirabe. Traditiona deveopment toos for producing interactive software require extensive programming knowedge. Programming graphics in common environments ike Visua C++ or Visua Basic can be difficut even for seasoned programmers. Toos such as interface buiders can hep deveopers design the visua appearance of an appication but sti require programming to make the interface actuay work. Appication buiders such as Cick & Create [3] eiminate programming but impose severe imits on the kinds of programs that can he Pem~ission to make digita or hard copieb of a or part of this work -or persona or cassroom use is granted without fee provided that copies arc not made or distributed for proiit or commercia advantage and that copies bear this notice and the fu citation 011 the first page. 1 0 CopV otherwise, to repubish, to post on serwrs or to redistribute to ists. requires prior specific permission and!or a fee. CHI 99 Pittsburgh PA USA Copyright ACM /99/05...$5.00 created. Authoring toos ike AuthorWare [] or Director [S] are simiary imited and cannot produce compex Ibehaviors and payer interactions without using their buit-in scripting anguages. One method for simpifying the programming process has been programming-by-demonstration (PBD). Rather than using a textua notation, the deveoper buids the program by providing exampes of the intended interactions between the user and the appication. Exampes are demonstrated using the same interface normay used to create and manipuate the appication s data. The system uses the exampi& to infer the deveoper s intention and creates the code to execute the program. Our research is aimed at significanty improving and expanding what can be accompished using PBD. Gamu.t has the abiity to infer compex reationships through the use of improved interaction techniques. The interaction techniques aow the deveoper to give Gamut a of the required information without resorting to a written programming anguage. These interaction techniques provide severa benefits: A simpified method for producing exampes. An understandabe way to create negative exampes. The abiity to give the system specific and direct hints. Objects and metaphors that can describe compex behaviors concisey. DOMAIN Gamut can create games and simuations simiar to board games. These are two-dimensiona games with a board-ike background that uses paying pieces to represent the game s state. The domain extends we beyond Chess and Monopoy, however. By having objects react autonomousy and by adding payer interaction, one can create video game behaviors such as moving monsters and shooting aiens. Educationa games ike Reader Rabbit [ 161 and Payroom [6] and video games ike PacMan can a be made using Gamut. The board game domain provides severa chaenges for a PBD system: The created games are interactive and require payer input. Some PBD systems ony assist in editing static data such as a text document. Board games have a arge number of states and modes. Game behavior can be triggered by a variety d events and can have compicated reationships. 442

2 CHI MAY 1999 Papers Figure 1: Gamut s main window. On the eft are the too paette and mouse icons. Aong the bottom is the behavior diaog area. Reationships between objects and actions are often formed as ong chains of other reationships which buid upon each other. For exampe, the destination square where a piece is moved in Monopoy coud be described as the square that is the dice s number of squares away from the square where current payer s piece currenty resides. This description depends on the configuration of the board, the number on the dice, and the payer whose turn it is. Each object in the reation forms a ink in the chain. Furthermore, an object such as the turn indicator is not necessariy graphicay or temporay connected to the other objects. Current PBD systems cannot infer this form of reationship. Gamut can be taught the rues of a game, but generay cannot create computer opponents. The difference here is the difference between rues and strategy. Paying a compex game we requires strategy which is often not easiy encoded as a set of rues. Gamut is designed to assembe games for humans to pay, not to pay the games, itsef. The deveoper has to show the system a reationships upon which a behavior depends. EXAMPLE To motivate the design of Gamut s interaction techniques, we wi show how to buid the simpe board game appication shown in Figure 1. This game was aso used as a task in the usabiity study discussed ater. In the game, two pieces coored red and bue foow the path of squares around the edge. The first piece to reach the end wins. The pieces aternate turns and move the number of spaces as shown on the die in the center. As an added compication, whenever one piece ands on another, the anded-on piece must go back to the beginning. As each of the foowing interaction techniques is presented, we wi show how the deveoper uses that technique to buid this board game. INTERACTION TECHNIQUES The key to Gamut s interaction techniques is that the deveoper can demonstrate not ony the surface activity of the interface, but the semantics behind that activity. The tech- niques aow the deveoper to express a the reevant reationships of an entire appication. The techniques can be divided into three categories: deveoper generated objects, such as guide objects, cards, and decks of cards; interaction methods, which incudes nudges and hint highighting; and system generated objects, such as tempora ghosts. Guide Objects Guide objects are graphica objects and widgets that are visibe whie the deveoper is creating an appication but are hidden when the appication runs. Gamut supports two kinds of guide objects. The first is derived from Mausby s Metamouse [9] and Fisher et a s Demo II [4] which aowed certain graphica objects to be made invisibe on demand. Onscreen guide objects show graphica reationships between other objects on the screen, visibe or invisibe. Onscreen guide objects can be used to demonstrate distances, ocations, and even speeds. For instance, in the exampe game, the path that each piece foows around the board can be represented with arrow ine guide objects as shown in Figure 2. Guide objects are drawn in paste coors so they wi be distinct from appication objects. At any time, the deveoper can make the guide objects invisibe by switching a mode. Without the path, the system woud not know how the squares around the board were connected, and might not even be abe to see that the squares exist at a. Aowing the deveoper to draw the graphica connections saves the system from having to provide sophisticated machine vision heuristics to achieve the same effects. Figure 2: The deveoper uses arrow ines as guide objects to represent the path the pieces foow. Other guide objects are paced offscreen. The payer s view is shown as a bue window frame in the midde of the deveoper s drawing area as seen in Figure 1. Objects that are drawn outside of the bue frame cannot be seen by the payer and are offscreen. Offscreen guides objects are used to represent the appication s data that is not stored directy on the board. Timers, counters, togge buttons, and other widgets are a used as offscreen objects. In our exampe game, the deveoper needs to represent the payer turn order. The deveoper decides to use a checkbox and paces one outside of the frame window and abes it Red s Turn. (In the Motif ook-and-fee, a checkbox ooks ike a raised or owered rectange next to a text abe.) The widget begins with its checkmark on. The purpose of guide objects is to enabe the deveoper to show reationships that are neary impossibe to infer. In AI, this is caed the hidden object probem [IS]. A hidden object is a dependency or variabe upon which a behavior depends that is not incuded as part of the appication s visi- 443

3 Papers CHI MAY 1999 be state. Gamut cannot infer hidden objects without the deveoper s hep because it has no way to determine what such objects coud be. The number of possibe things a hidden object coud be is virtuay infinite. However, it is possibe to recognize when a reationship requires more than the deveoper has shown. Gamut s inferencing agorithm can detect when reationships have not been fuy specified and asks the deveoper to te the system about missing objects. Deck Widget Cards and decks are the two major data structures in Gamut. Many modem board games use decks of cards to simuate a arge variety of behaviors. In games ike Monopoy, cards are a source of random events ike the Chance deck as we as the means for storing game state such as knowing which payer owns each property. In Gamut, decks may be used to represent ists of numbers, objects, coors, etc., and they provide a randomization feature (shuffing) which is usefu for constructing random behaviors. A deck may aso be used to produce video game behaviors. For exampe, a deck can provide aternating images for an animated character. To make a character move randomy, its position can be tied to a deck containing an arrow for each direction that gets shuffed each time the character moves. Gamut s deck of cards is not the same card metaphor found in HyperCard [7]. In HyperCard, cards are the whoe appication. In genera, a HyperCard stack is a set of screen dispays with inks between them to denote the method and order in which dispays are presented. A Gamut deck is a widget within the appication. To use a deck in Gamut, one drags objects into it. The deck wi store and maintain the order of a objects it contains. Figure 3: The card and deck editor. This shows the contents of the deck in Figure 1. Each face of a die is presented as a card. The current face is shown in the main drawing area. Card Widget Gamut s card widget acts as a arge drawing surface, independent from the main window. Naturay, cards may be paced in decks, but they can aso be used on their own. The deveoper uses Gamut s card editor (see Figure 3) to draw on a card. The raised frame in the card s drawing area shows what is visibe within the card s widget. Objects drawn outside the visibe region act as offscreen guide objects simiar to offscreen objects in the main window. In the exampe game, the deveoper uses a deck I:O represent the die in the center of the board (see Figure 1). The deck editor shows each item in the assembed deck which in this exampe is a set of cards each representing a face of the die (see Figure 3). The deveoper can demonstrate roing the die by shuffing the deck. In the drawing portion of the deck editor, the deveoper draws the pips for each face in the card s visibe region. Beow the visibe region, the deveoper adds a number box and types the numeric vaue of the die face. By incuding the number box, the system wi not have to count the dots in order to infer the vaue of the card. Demonstrating Behavior Gamut introduces a new way to demonstrate behavior which we ca nudges. The idea is that when the system makes a mistake or needs to earn new materia, the deveoper gives the system a nudge teing the system immediatey where it went wrong. In other words, when the appication is supposed to do something but does nothing, or does something when it is not supposed to, the deveoper nudges the system and corrects the behavior. Gamut defines two kinds of nudges. The first is caed Do Something. The deveoper uses Do Something to demonstrate new behaviors. When the deveoper sees the system miss a cue, the deveoper pushes the Do Something button. The system then becomes ready to accept the deveoper s new exampe permitting the deveoper to modify the appication s state appropriatey. The second nudge is caed Stop That which tes the system that one or more objects did something wrong. The deveoper, when noticing a deviant action, s.eects the affected object and presses the Stop That button. The system immediatey undoes a actions just performed. on that object. If the object was supposed to do nothing, the deveoper is finished at this point. If the object was supposed to perform a different action, the author may modify objects to show the system the correct behavior. In our exampe, the deveoper wants to demonstrate that when the payer pushes the appication s Move button, the game wi respond by moving the current payer s piece. The deveoper first pushes the Move button (it is to the eft of the die in Figure 1). Though the deveoper pushes the button, the appication wi do nothing because no behavior has yet been demonstrated. So the deveoper pushes Do Something at which point the system prepares to accept a new exampe. The system dispays tempora ghosts to show how objects have changed from the previous state, activates hint highighting so the deveoper may give hints, and presents a diaog asking the deveoper to compete the exampe and press the Done button when compete. The appication is supposed to ro the die and move the piece the corresponding number of paces. It aso must update the payer turn. The deveoper pushes the Shuffe button on the die s deck, moves the red piece the corresponding number of squares, and togges the turn. indicator to be unchecked. The view woud now ook ike IFigure 4a. At this point, before pressing Done to finish the (exampe, the deveoper shoud give the system hints. 444

4 CHI MAY 1999 Papers Figure 4a: The deveoper has just moved the red piece which used to be in the midde of the top row. The red piece s ghost shows where it was. Figure 4b: The deveoper has highighted the arrow ines, die, and ghost of the togge switch as hints. Hint Highighting A hint highight is a specia form of seection where the author points out key eements that are important to a demonstration thereby focusing the system s attention on those objects. Mausby impemented a simiar feature in Cima [o] by having the user seect a word or phrase and use a menu command to make it a hint. Hints are used to reduce the size of the inferencing agorithm s search space. The number of features upon which a singe reationship may depend can be immeasuraby arge. Finding the correct features can require an exponentia amount of search time without hints. Providing hints can reduce the search to near constant time. In Gamut, the deveoper highights objects as hints by pressing the right mouse button over them. Gamut marks hinted objects with green rectanges. Highight marks around ines are seen as a thin rectange that foows the ine s direction as seen in Figure 4b. Highighting is different from norma seection which is caused by the eft mouse button and presented as a conventiona set of square handes. (The circe on the right of Figure 4a and b is seected.) Seection is used to move, resize, and recoor objects. Since it is common for the deveoper to want to hint highight an object and sti perform other operations, highighting is made an independent operation from seection. In the exampe, the deveoper knows that the path of the piece is important as we as number shown on the die; so, the deveoper highights the ines in the path and the die object. The deveoper wi aso need to highight something that shows what vaue to set the current payer checkbox. The checkbox ony togges back and forth between true and fase so its ony dependency is its own vaue. Thus, the new vaue of the checkbox depends on its origina vaue. To highight the origina vaue, the deveoper highights the checkbox s tempora ghost which is shown on the right in Figure 4b. Tempora Ghosts A common probem for hint highighting arises when the objects that need to be highighted do not exist anymore. Interactive games are dynamic: objects are created, moved, and destroyed constanty. Tempora ghosts are a technique for keeping objects that change onscreen so that they may be highighted. Ghosts aso make the recent past visibe so that the author can understand what changes have occurred. Though the concept of ghost objects is not new, Gamut is the first to use it for PBD. Gamut dispays tempora ghosts as dimmed, transucent images of objects seen in their past state. If an object is moved, a ghost wi appear in the object s origina position. If the object changes coor, the ghost wi appear directy beow the object but offset so the deveoper can sti see it. When the deveoper togges the turn indicator in the exampe, a ghost appears beow it, offset to show that it used to be checked (see the right portion of Figure 4a). The deveoper highights this ghost to show that its vaue is important. Gamut wi be abe to see that the new vaue is different from the od vaue and use that to describe how the togge changes. Finay, the deveoper pushes Done because the exampe is finished. For the second exampe, the deveoper pushes the Move button again. The system is abe to incorporate enough information to know how far to move the pieces but it does not know that it is supposed to move the bue piece and so it moves the red piece. The deveoper notices that this behavior is wrong, seects the red piece and presses Stop That. Gamut immediatey undoes the move action performed on the red piece. Stop That paces the system into the same mode as Do Something so tempora ghosts are dispiayed once again and hint highighting is made active. The deveoper moves the bue piece the correct number of spaces and aso changes the payer turn indicator if the system did not aready do so. When finished, the deveoper presses Done. This time, the deveoper did not highight any objects. As a resut, when the system finds ambiguities, it wi ask the deveoper questions in order to resove them. Question Diaogs Questions occur when the system finds a contradiction or suspects that there is a reationship where an object was not highighted. The system wi generate one question at a time in the behavior diaog region of the window. The questions ask about the objects or vaues that are immediatey affected by the behavior. Deveopers have three choices for response. First, they may highight the object upon which reationship depends and press the Learn button. The system then tries to incorporate this new information to generate a description. Second, they may choose the Repace button in order to directy repace the od vaue with the new. Repace is used to correct mistakes or to modify a behavior from its origina form. Finay, the third button is caed Wrong and is used in cases where Gamut asks a question that makes no sense. Gamut generates its questions using heuristics which can sometimes fai. The Wrong button tes the computer that it has generated a bad question and that it shoud try a different ine of reasoning. 445

5 Papers CHI MAY 1999 In the deveoper s second exampe in the exampe appication, the system sees the bue piece move instead of the red piece but it does not know why. The system asks the deveoper to highight the object that best describes why the bue piece has moved and not the red. The deveoper highights the ghost of the turn indicator and presses Learn. Highighting the turn indicator tes the system that the bue piece moves when the checkbox is unchecked. To finish this portion of the exampe, the deveoper woud need to test the Move button one more time and correct the system (without needing further highighting) to compete the behavior. Mouse Input Icons The exampe appication does not directy use mouse input in the window since the button widget handes the mouse automaticay. However, it is worth mentioning how demonstrating mouse events is accompished in Gamut. Gamut has a paette of mouse events beow the main too paette (see Figure 1). To create a mouse event, the deveoper seects an event and drops it onto the window as though it were a graphica object. This aows the deveoper to demonstrate mouse events without entering a specia mode. The system wi respond as though the payer had just produced the seected event. Keyboard and other sorts of events, though not impemented, coud be incuded in a simiar way. Gamut uses the same icons to represent mouse events as we used in our Marquise system [14]. Cicking events are shown with an arrow pointing up as we as down whereas button down events ony point down. Doube cicks are shown as two arrowheads pointing down and movingdragging uses a wavy ine. The icons are shown in Figure 5. Figure 5: Icons that are used to represent mouse events. Contrasting Nudges With Other Systems Techniques In an abstract sense, Do Something and Stop That represent positive and negative exampes. New demonstrations are positive exampes and are performed using Do Something. Stop That signifies a negative exampe since it asks that no action be performed in that given instance. Evidence from Frank [5] suggested that deveopers found negative exampes difficut to understand, but we suspect that those users had difficuty with the demonstration techniques in that particuar system. Frank s system and others require the deveoper to demonstrate negative exampes using specia modes. This requires the deveoper to understand a priori when a negative exampe is required and it draws attention to the exampe instead of to the behavior that it represents. With nudges, the deveoper stays focused on the behavior of the appication. Each nudge provides an incrementa improvement to the behavior that the deveoper is testing. The deveoper does not have to know whether a particuar exampe is positive or negative, and must ony te the system when objects are not behaving correcty. Negative exampes permit the earning of disjunctive ogic statements which in turn permits program structures such as if-then statements to be earned without the author having to create conditions by manuay changing the inferred code. Another advantage of nudges is that they reduce the number of system modes. Some PBD systems require a separate recording mode to enter stimuus events. This is part of the Stimuus/Response mode distinction used in various system incuding Pavov [19]. A Stimuus/Response stye: interface is normay impemented as an extended macro recorder. With a macro recorder, pressing record causes the system to record a subsequent actions. Paying the macro ater wi execute those actions in a new context. An extended macro recorder adds an extra stimuus phase where the deveoper demonstrates the event of the behavior. Revising behaviors can be more tedious using the extended macro recorder technique than Gamut s nudges approach. Since the macro recorder requires the deveoper to perform the event during the stimuus recording phase, the deveoper must know beforehand what he or she intends to demonstrate. If the deveoper were to find a probem whie testing the appication, the stimuus event that occurred during the test woud have to be recreated for the macro recorder. Furthermore, the initia state for a the objects invoved woud have to be reset. With the nudges approach, the (deveoper can create new exampes as the need occurs. Whe:n a probem appears during testing, the deveoper can immediatey nudge the system and use the current appication Istate in the exampe. INFERENCING To manage the various interaction techniques, Ga.rnut needs an inferencing agorithm that can understand them. The agorithms must hande mutipe exampes inc:rementay whie incorporating hint highighting of various objects, guide objects, and tempora ghosts. Furthermore, the agorithms must be abe to generate the behaviors characteristic of board games ike chains of reationships with c:onditiona components. For a more compete description of Gamut s inferencing agorithm see our previous paper [ 111. Gamut s action anguage is based on the commcznd object structure found in Amuet [13], the deveopment environment used to impement Gamut. A command ob.ject represents an atomic action such as Move, Create, Cut, Copy, and Paste. In Amuet, user actions are queued onto an undo history which Gamut uses as the input to the first stage of inferencing (see Figure 6). Task Data Stage 1 Convert to actions Exampe actions B+fiL Stage 2 Stage 3 Match origina and exampe actions Origina actions And/Or tree of changes Figure 6: Stages in Gamut s inferencing agorithm. Assembe behavior 9 Fina behavior I In the first stage, the commands from the undo history are reduced to a canonica form which removes repetition and 446

6 CHI MAY 1999 Papers order dependencies. The events found in the undo ist are often compicated. A group command, for instance, not ony creates a group object but aso moves and reparents the objects being grouped. This stage converts the events into a sma set of basic actions. This reduced set of commands is passed on to the matching stage. The matching stage uses a pan recognition agorithm as its basis [ 181. If the appication aready has behavior defined for the event being demonstrated, the od behavior is used as a tempate for recognizing features in the new exampe. (If there is no previous behavior, the agorithm moves to stage three.) The agorithm can foow chains of descriptions in the origina behavior and determine which of the parameters shoud be changed in order to make the od behavior perform the actions in the new exampe. Output from this phase is the set of differences from the existing behavior and the new exampe. The set of changes is captured in an And/Or tree which represents the various ways the changes might be appied. And-nodes represent changes that must occur together whie Or-nodes are used to store aternatives. The fina stage of inferencing resoves the differences found in stage two and describes new parameters. This stage uses the objects highighted by the deveoper to search for descriptions which resove differences stored in the And/Or tree. The agorithm empoyed by this phase is heuristic and is based on the agorithm in Marquise [14]. When Gamut does not find a suitabe description, it asks the deveoper a question in the behavior diaog. The text of the question is based on the current unresoved difference and refers to the vaue in the origina behavior, the new vaue in the exampe, and some context information stating what the vaue affects. The deveoper is asked to highight the appropriate objects to answer the question. If Gamut sti fais to find a suitabe description, it wi use a decision tree to choose between the od and new vaues. The precise way that Gamut appies decision trees is beyond the scope of this artice. The basic idea is that Gamut generates attributes for the decision tree agorithm using the objects the deveoper highights. The agorithm (specificay ID3 [15]) wi, in turn, decide which attributes to appy and in what order. This aows Gamut to generate conditiona expressions with objects that are not directy affected by the behaviors they contro. USER TESTING We tested Gamut under forma conditions to see how we the techniques woud be understood by nonprogrammers. In a short three-hour session, the test participants had to earn the system and buid two tasks using Gamut. One participant was aso invited back to attempt a third task which was onger and required mouse input. Overa, Gamut performed fairy we. Three of the four participants were abe to compete the tasks and the one person who attempted the third task competed it as we. Participants The study s participants were contacted through eectronic buetin boards and emai at Carnegie Meon University. The subjects were required to be nonprogrammers. Specificay, the subjects were aowed to have taken a ow-eve cass in programming, but were not aowed to program computers for a iving or as a hobby. Participants were required to be famiiar with typica computer interface metaphors as we as drawing editors. Tasks In the three hour sessions, the participants had to compete an hour-ong tutoria and two tasks. We asked the participants to use a think-aoud protoco [ 171 to articuate their thoughts. The sessions were videotaped and an experimenter was present to answer the participant s questions. The first task was based on a matching test simiar to some educationa games ike Reader Rabbit and is shown in Figure 7. It was caed Safari and it consists of two decks of cards. One contained a ist of animas ike Zebra, and the other contained a ist of questions about animas ike Does it have stripes? The goa of the task was to put guide objects into the deck that woud te Gamut the correct answers and to demonstrate the behavior of a pair of buttons abeed Yes and No. The second task was the board game task that we used in this paper as an exampe and can be seen in Figure 1. The participants did not have to draw the board or create the die, but did have to create guide objects to represent the path the pieces foowed, create something to represent the turn indicator, and demonstrate a the behaviors. The third task which ony one participant attempted was based on the video game Q*bert and is aso shown in Figure 7. It had a character which jumps from cube to cube in a pyramid and coects objects. There is aso an enemy ba which fas down from the top of the pyramid in random directions. The third task required a onger time to compete than the others so it had its own session. The participant was given a shortened tutoria which described the new interaction techniques required in the third task. Observations The purpose of the study was to see whether nonprogrammers coud use Gamut to demonstrate behavior. Few PBD systems in the past have ever been tested with actua users so we were mosty interested in proving that PBD coud actuay work. As shown in Tabe 1, each participant used a different set of Gamut s techniques in order to compete the tasks. For instance, some participants preferred to use Do Something excusivey, others preferred to use Stop That. Ony one participant was unabe to earn enough of Gamut s techniques to buid the appications. One of the probems the participants had concerned highighting ghost objects. Most participants showed a pecuiar reuctance to highight a ghost object. Severa preferred to highight the origina object and not its ghost even though the origina was modified and dispayed a different state from what the participant wanted the system to earn. The Highight Ghosts ine on the tabe shows that ony one participant was truy comfortabe highighting ghosts whie the others woud choose not to most of the time. 447

7 Papers CHI MAY 1999 I techniques IP IP2IP3IP41 Do Something X X Stop That X The most significant probem we discovered in Gamut concerns guide objects. Deveopers were aways reuctant to create guide objects to use in their demonstrations. The piot study showed we had to incude specific instructions to te the participant to draw guide objects. However, once tod, participants were usuay abe to create objects that were suitabe for the task they needed to demonstrate. The ony successfu participant who had difficuty creating guide objects after being tod to do so was P4. It turns out P4 eventuay did create a the needed guide objects, but ony after asking the experimenter so many questions that it was not cear whether P4 had designed the objects hersef or whether the experimenter had given away the answers. A probem that a participants shared (though to different degrees) was highighting inappropriate objects as hints when Gamut asked a question. Gamut requires that the deveoper provide a hint when it asks a question, but there seemed to be situations where certain objects were considered too obvious to highight. Ghost objects seemed to be one instance. Visua paths and ines connected to objects seemed to be another. When the system needed to have an object highighted that was too obvious for the participant s taste, the participant woud often choose to highight an object which had a ess obvious connection to the behavior. Gamut is designed to be resiient to bady highighted objects so the participant usuay had mutipe opportunities to answer the same question, though the system s performance becomes consideraby worse each time the participant gives a bad hint. Eventuay, most participants woud highight the appropriate object and the system coud proceed. Payer Mouse Icons* X I I Tabe 1: Different participants earned and used different sets of Gamut s techniques. A participants except the first were. abe to successfu compete the tasks. * Ony participant two was given a task that required the Mouse Icons to compete. The Cards and Decks and Payer Mouse Icons ines of the tabe show that two participants used decks in their tasks and one used the mouse icons. The first two tasks came with decks of cards aready prepared. A participants had itte difficuty using the decks. However, P2 and P3 created and used an origina deck of cards as their own widget. The Payer Mouse Icons were ony needed for task 3 and were ony used by P2 who had itte troube. It is not entirey cear why P1 was unabe to use the system. According to the participant s own comments, he was too tired to earn how the system worked. The poor resut may have simpy been due to fatigue from a ong day at work. The sorts of errors P1 woud make were mosty caused by forgetting how to demonstrate using nudges. For instance, sometimes he might not demonstrate the event for the exampe and sometimes he woud not push Do Something or Stop That. RELATED WORK A number of toos exist for buiding games. Most construct a specific cass of games such as Bi Budge s Pinba Construction Set [2] which makes pinba simuations. A recent product is Cick & Create [3] in which the deveoper first draws the game objects and cassifies each as background, characters, or other objects. Then the author assigns behavior to the characters by picking from a ist of stock behaviors. These behaviors can be customized by changing some parameters, but the author is imited to the buit-in methods. Gamut most resembes our previous system, Marquise [ 141. Like Gamut, Marquise s goa was to create whoe appications. Marquise had the abiity to recognize paettes of objects and coud quicky infer operations such as seecting and dragging. Marquise s major deficiency was an inabiity to correct guesses by demonstration. It aso had a imited set of expressions for describing objects and ocations which caused it to make poor inferences. The ony means for correcting the system was editing the inferred code using a set of unwiedy diaog boxes. PBD systems such as Wober s Pavov [19] and Frank s Grizzy Bear [5] have shown that simpe heuristics can be used to infer many forms of graphica constraints and simpe behaviors. Both of these systems infer inear reation-

8 CHI MAY 1999 Papers ships between objects with numeric parameters (ike an object s screen position). Unfortunatey, inear constraints cannot be used to infer conditiona expressions based on modes and many other kinds of behavior needed to buid whoe appications. Pavov requires users to annotate their demonstrated behaviors with conditiona guard statements in order to overcome these probems. Gamut s inferencing abiity is simiar to Mausby s Cima system [o]. Cima aso has the abiity to earn from hints and can earn concepts incrementay. Cima s description anguage is not as powerfu as Gamut?. Cima s statements are restricted to ogic statements in disjunctive norma form (DNF) which it uses to recognize passages in a body of text. Aso, Cima currenty cannot do work on behaf of the user: it just recognizes strings of text. STATUS AND FUTURE WORK Gamut is impemented using Amuet [12] and wi run on Unix, Windows, or the Macintosh. Gamut is a prototype system impemented as the first author s thesis project. It is not a commercia software product and it is not avaiabe for reease except for research purposes. Though Gamut is functiona, more work woud be needed to make it usabe for typica deveopers. We have used Gamut to demonstrate behaviors that other systems cannot produce. For instance, we have created a Turing machine emuation, compete Tic-Tat-Toe and Hangman games, various video game behaviors such as a PacMan-ike monster. Behaviors from educationa games such as matching words as in Reader Rabbit have aso been created. Thus, Gamut has been used to demonstrate a broad range of behaviors without resorting to a written programming anguage at any point. Though Gamut s input techniques work we, the interface sti needs to provide better feedback. Currenty, the system has ony the behavior diaog to te the deveoper about the behavior being demonstrated. More work is needed to inform the deveoper about what the system knows and what it can infer. The system needs to note graphica constraints and to have a diaog mechanism for dispaying the inferred code in an understandabe format. CONCLUSION Gamut has the abiity to infer compex behaviors which can be used to buid compete interactive appications. This new capabiity derives from an innovative coection of interaction techniques couped with inductive earning agorithms that can take advantage of the techniques. The nudges interaction simpifies exampe recording and provides a simpe manner to create negative exampes. Hint highighting is a means for improving the system s guessing by aowing the software author to point out important objects in a behavior. The deck-of-cards metaphor aows compicated behaviors to be specified that can invove sets of data and randomness. Guide objects permit demonstration of objects and reationships which the system coud not guess by itsef. Finay, tempora ghosts aow the author to directy form reationships with the recent past. These techniques were tested in a usabiity study where we found that nonprogrammers were abe to use them to buid reaistic appication behaviors effectivey. Thus, Gamut s techniques are an effective method for demonstrating a broader range of appications with a minimum of programming expertise and woud be appropriate for use in a wide range of future PBD systems. ACKNOWLEDGEMENTS This research was partiay sponsored by NCCOSC under Contract No. N C-6037, Arpa Order No. B326, and partiay by NSF under grant number IRI The views and concusions contained in this document are those of the authors and shoud not be interpreted as representing the officia poicies, either expressed or impied, of the U.S. Government. REFERENCES Authorware. Authorware Inc Normandae Lake Bvd., Suite 430, Minneapois MN 55437, , B. Budge. Pinba Construction Set. Exidy Software. Core1 Cick & Create. Core1 Corporation and Europress Software Ltd G. L. Fisher, D. E. Busse, D. A. Wober. Adding Rue-Based Reasoning to a Demonstrationa Interface Buider. Proceedings of UIST 92, pp M. Frank. Mode-Based User Interface Design by Demonstration and by Interview. Ph.D. thesis. Graphics, Visuaization & Usabiity Center, Georgia Institute of Technoogy, Atanta, Georgia. 6. L. Grimm, D. Caswe, and L. Kirkpatrick. Payroom. Broderbund Software, 500 Redwood Bvd., Novato, CA , HyperCard. Appe Computer Inc., Cupertino, CA, Macromedia, Director, 600 Townsend Street, San Francisco, 9. CA 94103, macropr@macromedia.com, D. Mausby, I. Witten. Inducing Procedures in a Direct- Manipuation Environment. Proceedings SIGCHI 89, Apri, pp D. Mausby. Instructibe Agents. Ph.D. thesis. Department of Computer Science, Univ. of Cagary, Cagary, Aberta, June R.G. McDanie, B.A. Myers. Buiding Appications Using Ony Demonstration. Proceedings of IUI 98. pp B. A. Myers et a. The Amuet Environment: New Modes for Effective User Interface Software Deveopment. IEEE Transactions on Software Engineering, Vo. 23, no. 6. June pp B. A. Myers, D. S. Kosbie. Reusabe Hierarchica Command Objects. Human Factors in Computing Systems, Proceedings SIGCHI 96, Denver, CO, Apri, 1996, pp B. A. Myers, R. G. McDanie, and D. S. Kosbie. Marcuise: Creating Compete User Interfaces by Demonstration. Pro- ceedings of INTERCHI 93: Human Factors in Computing Systems, 1993, pp J. R. Quinan. Induction of Decision Trees. Machine Learning, Kuwer Academic Pubishers, Boston, Vo. 1, 1986, pp Reader Rabbit. The Learning Company, M. Rettig. Prototyping for tiny fingers. Communications of the ACM 37,4 (Apri 1994). pp K. VanLehn. Learning One Subprocedure per Lesson. A@- cia Inteigence, Vo. 31, 1987, pp D. Wober. Pavov: Programming By Stimuus-Response Demonstration. Human Factors in Computing Systems, Proceedings SIGCHI 96, Denver, CO, Apri, 1996, pp