Automatic Generation of Web Interfaces from Discourse Models Institut für Computertechnik ICT Institute of Computer Technology Hermann Kaindl Vienna University of Technology, ICT Austria kaindl@ict.tuwien.ac.at Background Outline AI theories underpinning discourse modeling for HCI Other theories underpinning discourse modeling for HCI Interaction design based on discourse modeling Exercise Sketch of automated user-interface generation Hermann Kaindl 1
Based on toolkits employing widgets Traditional UI development Widgets grouped according to their graphical appearance Highly-specialized designers and programmers needed Lots of UI code Error-prone, low maintainability Expensive Interactive objects presented on the display windows buttons scroll bars User interface elements Classification hierarchy of widgets Widgets Widget Widget for Selecting an Action Control Tool Container Widget Hermann Kaindl 2
Interaction design Design of interactions between human and computer Relation to requirements engineering Relation to task analysis No commitment to specific user interface Scenarios Stories and narratives For representation of cultural heritage explanations of events everyday knowledge Human understanding in terms of specific situations Human verbal interactions by exchanging stories Hermann Kaindl 3
Background Outline AI theories underpinning discourse modeling for HCI Other theories underpinning discourse modeling for HCI Interaction design based on discourse modeling Exercise Sketch of automated user-interface generation Scripts Schank and Abelson Script: structure that describes appropriate sequences of events in a particular context Handles well-known everyday situations Predetermined and stereotyped sequence of actions Hermann Kaindl 4
Scripts Restaurant script example Sketch of stereotypical sequence of actions in (U.S.) restaurant: A customer enters a restaurant and waits to be seated. A waiter guides the customer to an empty table and hands over a menu. The customer studies the food list in the menu and chooses something. The waiter comes to the table and takes the order. Rhetorical Structure Theory (RST) Mann and Thompson Linguistic theory Internal relationships among text portions and associated constraints and effects Relationships in a text are organized in a tree structure Rhetorical relations associated with non-leaf nodes, and text portions with leaf nodes Hermann Kaindl 5
RST Taxonomy (selection) Tom Gruber Actually, the old Greeks Domain models Conceptualizations of a domain Ontologies Often using taxonomies and object-based ideas Ontology languages based on knowledgerepresentation theories E.g., OWL based on description logic Hermann Kaindl 6
Ontologies Model of domain of discourse for online shop example Background Outline AI theories underpinning discourse modeling for HCI Other theories underpinning discourse modeling for HCI Interaction design based on discourse modeling Exercise Sketch of automated user-interface generation Hermann Kaindl 7
John R. Searle Theory from philosophy of language Speech acts Human speech also used to do something with intention to act Speaking a language is performing speech acts, act such as making statements, giving commands, asking questions and so on Speech acts: basic units of language communication Communicative acts: abstraction from speech Communicative Acts Taxonomy (selection) Hermann Kaindl 8
Harvey Sacks; Luff, Gilbert and Frohlich Theory from sociology Conversation Analysis Focus on sequences of naturally-occurring talk turns To detect patterns that are specific to human oral communication Adjacency pair: e.g., a question should have a related answer Inserted sequence: subordinate interactions Background Outline AI theories underpinning discourse modeling for HCI Other theories underpinning discourse modeling for HCI Interaction design based on discourse modeling Exercise Sketch of automated user-interface generation Hermann Kaindl 9
Discourse Example Discourse Model Discourse atoms and molecules Metaphorical view Communicative acts as atoms Adjacency pairs as molecules Communicative acts instead of RST text portions Interaction instead of text Two dimensions Tree with discourse relations (monologue) Adjacency pair (dialogue) Integration of RST and procedural constructs with Conversation Analysis Hermann Kaindl 10
Communicative Acts Open & Closed Question Open Questions enable asking for a particular type of information, respectively, an instance of a domain class. Closed Questions restrict the possible answer to a list of provided domain instances to choose from. Communicative Acts Informing & Answer Both are used to convey information. Answer communicative acts are always directly related to questions, whereas Informing is uttered standalone or together with acknowledgment. Hermann Kaindl 11
Communicative Acts Request Used to request the communication partner to act. Thus, the propositional content of a request is always an action that has to be carried out. The action can be defined either for the given application, or it can be the request to utter a particular communicative act. Communicative Acts Offer Offers to carry out an action or to add information to the shared knowledge. Hermann Kaindl 12
Communicative Acts Accept & Reject Accept and Reject provide for accepting or rejecting a particular request or offer. Communicative Acts Taxonomy CommunicativeAct Assertive Commissive Directive Answer Informing Offer Accept Reject Request Question adjacent to adjacent to adjacent to adjacent to adjacent to OpenQuestion ClosedQuestion adjacent to Hermann Kaindl 13
Adjacency Pair Relates an initial communicative act with one subsequent communicative act or two alternative subsequent communicative acts. Typical adjacency pairs of communicative acts are: ClosedQuestion Answer, OpenQuestion Answer Offer Accept, Offer Reject Request Informing, Request Accept, Request Reject RST relations (in our approach) Nucleus: the main part of the communication Satellite: the helper part Communicative acts instead of text portions Hermann Kaindl 14
RST relation Joint Relates independent subtrees with communicative acts of the same kind. It does not imply any order. So, it is also possible to issue both nuclei concurrently (e.g., on a GUI). RST relation Contrast Relates similar subtrees and compares them with respect to differences. Hermann Kaindl 15
RST relation Background General information of any sort that is likely to help understand the nucleus. Thus, satellite of the background relation shall only contain Informing communicative acts. RST relation Elaboration Satellite contains additional detail about some element of subject matter which is presented in the nucleus, in one or more of the ways listed below (nucleus :: satellite): set :: member abstraction :: instance whole :: part process :: step object :: attribute generalization :: specific The communicative acts can also be questions, for example, if one communicative partner wants to figure out additional details about the subject matter. Hermann Kaindl 16
RST relation Elaboration (cont.) RST relation Title Specialization of Elaboration, restricting the additional detail of some element of subject matter to a short description, either title or caption. Hermann Kaindl 17
RST relation Annotation Another specialization of Elaboration, restricting the additional detail of some element of subject matter to meta information. Taxonomy of RST relations Discourse Relation RST Relation + nucleicondition + nucleussatellitecondition + satellitecondition Nucleus-Satellite Relation Multi-Nucleus Relation Result Elaboration Back ground Contras t Joint Annotation Title Hermann Kaindl 18
Procedural construct Sequence Defined order of uttering the communicative acts or subtrees. Procedural construct IfUntil If-statement combined with a conditional loop Utterance of the <Then> subtree depends on successful execution of the related Condition. Repetition of the <Tree> branch until Condition becomes fulfilled, while RepeatCondition is fulfilled Hermann Kaindl 19
Procedural construct IfUntil (cont.) Conceptual Discourse Metamodel Hermann Kaindl 20
Domain representation Speech act usually talks about something in the domain of discourse Model of the domain Integration and use of ontologies Example model Hermann Kaindl 21
Background Outline AI theories underpinning discourse modeling for HCI Other theories underpinning discourse modeling for HCI Interaction design based on discourse modeling Exercise Sketch of automated user-interface generation Exercise Fitness Studio Interaction design model according to our approach, for the website of a fitness club which should allow registered users to book the various courses that the club offers. Try to understand the model sketch of a discourse for this application! Hermann Kaindl 22
Fitness Studio Discourse Model Questionnaire Voluntarily, please fill in the subjective questionnaire at http://ontoucp.org/cms/technology/questionnaire.html Hermann Kaindl 23
Background Outline AI theories underpinning discourse modeling for HCI Other theories underpinning discourse modeling for HCI Interaction design based on discourse modeling Exercise Sketch of automated user-interface generation Unified Communication Platform Domain Specification Workflow Repositories Communicative Act Types depend on based on request inform propose uses uses uses Rendering Specification uses Communication Definition Language Human Machine Interface based on Human Machine Interface Communication System 1 Platform System 3 uses System 2 Machine Machine Interface Machine Machine Interface System 4 System-of-Systems Hermann Kaindl 24
Automated generation of user interfaces Essential steps Generation of structural UI model Generation of finite state machine Rendering of UI Even for multiple platforms Generation of structural UI model Transformation process according to MDA (model-driven architecture) Hermann Kaindl 25
Finite State Machine Maximum presentation unit state machine for the online shop example Rendered User Interface Guide in Kunsthistorisches Museum Wien Hermann Kaindl 26
Summary and Conclusion Human-computer interaction can be based on discourse modeling. Discourse model represents interaction design. Discourse model used for automatic user-interface generation. Thank you for your attention!??? Hermann Kaindl 27
Literature Carroll, J. M., (editor), Scenario-Based Design: Envisioning Work and Technology in System Development. New York, NY: John Wiley & Sons, 1995. Luff, P., Gilbert, N., Frohlich, D., (eds.), Computers and Conversation, Academic Press, 1990. Mann, W.C., and Thompson, S.A. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3): 243 281, 1988. Searle, J.R. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge, England, 1969. Schank, R. C., and Abelson, R. P., Scripts, Plans, Goals and Understanding. Hillsdale, NJ: Lawrence Erlbaum, 1977. Selected work of this tutorial presenter Bogdan, C., Falb, J., Kaindl, H., Kavaldjian, S., Popp, R., Horacek, H., Arnautovic, E., and Szep, A., "Generating an Abstract User Interface from a Discourse Model Inspired by Human Communication. In Proceedings of the Forty-first Annual Hawaii International Conference on System Sciences (HICSS-41), p. 10, Hawaii, 2008, IEEE Computer Society Press. Bogdan, C., Kaindl, H., Falb, J., and Popp, R., "Modeling of interaction design by end users through discourse modeling. In Proceedings of the 2008 ACM International Conference on Intelligent User Interfaces (IUI 08), Gran Canaria, Spain, 2008. ACM Press, pp. 305 308. Falb, J., Kaindl, H., Horacek, H., Bogdan, C., Popp, R., and Arnautovic, E., "A discourse model for interaction design based on theories of human communication". In CHI 06 Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2006. ACM Press, pp. 754 759. Falb, J., Kavaldjian, S., Popp, R., Raneburger, D., Arnautovic, E., and Kaindl, H., "Fully Automatic User Interface Generation from Discourse Models". In Proceedings of the 2009 ACM International Conference on Intelligent User Interfaces (IUI 09), ACM. Sanibel Island, Florida, USA, 2009. ACM Press. Tool demo paper. Hermann Kaindl 28
Selected work of this tutorial presenter (cont.) Falb, J., Popp, R., Röck, T., Jelinek, H., Arnautovic, E., and Kaindl, H., "Fully-automatic generation of user interfaces for multiple devices from a high-level model based on communicative acts". In Proceedings of the Fortieth Annual Hawaii International Conference on System Sciences (HICSS-40), p. 10, Hawaii, 2007, IEEE Computer Society Press. Falb, J., Popp, R., Röck, T., Jelinek, H., Arnautovic, E., and Kaindl, H., "UI Prototyping for Multiple Devices Through Specifying Interaction Design". In Proceedings of IFIP INTERACT 2007, LNCS 4662, Part I. Heidelberg, Germany, 2007. Springer, pp. 136 149. Kavaldjian, S., Bogdan, C., Falb, J., and Kaindl, H., "Transforming Discourse Models to Structural User Interface Models". In MoDELS 2007 Workshops, LNCS 5002. Heidelberg, Germany, 2008. Springer, pp. 77 88. Popp, R., Falb, J., Arnautovic, E., Kaindl, H., Kavaldjian, S., Ertl, D., Horacek, H., and Bogdan, C., "Automatic Generation of the Behavior of a User Interface from a High-level Discourse Model". In Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS-42), p. 10, Hawaii, 2009, IEEE Computer Society Press. Hermann Kaindl 29