Abstract In recent years, interactive digital media has made a rapid development in human computer interaction. However, the amount of communication or information being conveyed between human and the computer is still considered quite inadequate, which result in the enjoyment and satisfaction of human is restricted. Intelligent Interactive Multimedia has brought interactive multimedia into new era. By adopting the human computing approach and machine learning algorithm that enable the computer to be capable to deduce the initial intention of human through the recognition of human s behaviors, it will further to meet the needs of human. Eventually, intelligent interactive multimedia will be completely developed towards this direction. This research deploy a conceptual plan as an application call Intelligent Bubble Interactive Multimedia. The contribution of this research included classification of the human behavior, using machine learning algorithm that helps to analyze and learn the human behavior based on certain features that is proposed and the most significant contribution will be the digital media installation that manage to response intelligently and automatically based on the human s interaction. In this research, a study on human computing approach in user modeling through intelligent interactive multimedia that models the human interactive behavior with the digital media content is proposed. The intelligent interactive multimedia s content will adapt to the human behavior from different users. Computer Vision Technology will act as an observer to captures the user s interactive behaviors and in return, behaves intelligently as if the content is communicating with the user. When an interaction takes place, a logical feedback is sensed and to a certain extent, an intelligent feedback either knowing what the user s initial intention was or to suggest an intelligent reaction based on the user s input actions. A test of interaction and data capturing for machine learning to analyze will provide the answer to the current spectator s behavior. The outcome of this research is a prototype of intelligent interactive multimedia s content that adapt to human behavior based on the interactions between human and digital content that can be extended into intelligent agents represented with bubbles to interactive multimedia display.
Motivation and objectives of the proposed study Motivation of the proposed study is to achieve the computer/machine which has the capable to understand the intention or needs of the Human when interact with the machine. And then, the machine will react or response intelligently according to the human s intention or need and present it to the human through the multimedia s content adaption. For example, user A is surfing at the ticket reservation website, when an interesting promotion s banner is sliding in front of the user A. At the moment, user A feels promotion B is a best deal for him/her. Simultaneously, the machine is capable to understand the user A s intention and response intelligently by linking user A to the proper page which show more details about the promotion B automatically. Instead of, user A gives the instruction to the machine, so that, the machine only can make a response to link user A to the promotion s page of B. Objectives of the proposed study are 3I objectives which are included Interaction, Information technology, and Intelligent. Interaction is to adopt human computing approach to model interactive communication between a user and an interactive multimedia. And hence, the content of Interactive Multimedia will be adapted to the interaction of the user intelligently. Information technology is to deploy vision technology as an observer to observe user s behavior in front of the monitor, and then to capture user s gesture in communicating with an interactive multimedia. Machine Learning is to process and analyze the captured information from the vision technology and to tell the particular user is belonging to which category of human behavior. Lastly, intelligence is the machine/computer is capable to understand the user s intention and response intelligently according to the user s input through human behavior recognition. Literature Review Early 1900 computer was completely developed in system centric. Computers were designed primarily to solve preformulated problems or to process data according to predetermined procedures. And then, J.C.R.Linklider brought in the idea of cooperative interaction between man and electronic computer to enhance the flexibility of the computer which was performed in the limitation context [1]. Man and Computer
symbiosis should be in the way computer facilitate formulative thinking as computers now facilitate the solution of formulated problems, and to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs [1]. Basics for the achievement of the effective, cooperative involvement include developments in memory components, in memory organization, in programming languages, in computer time sharing, and in input and output techniques necessary to directly facilitate the exchange of information between a man and a computer system. These techniques include both hardware and software, and usually its function is closer to the man-computer interface [1] [2]. Rapidly growth of the man and computer study, interface being introduced and number of researches being carried out to integrate the interactive between Human and Computer. Creative and innovative development of interface in Text, in Graphic, in Speech or the combination of text, graphic and speech has been concerned in Human and Computer study to enhance the interactive between man and computer. Efficiency and reliability in human-computer interaction are pointed out to be dependent on complexity and flexibility of interface. Human Computer Interaction s is responsible for the interaction between the humans and the machines, modeling the human-computer interfaces is a key task in Man Machine System theory and practice [3]. Massive attention of Computer Vision technology, Machine Learning and Human features exploration in Man and Machine study is paid. The use of hand gestures gives an attractive alternative to bulky interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for Human Computer Interaction. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures [4]. The new technique for direct visual matching of images for face recognition and image retrieval, is adopting Bayesian (MAP) analysis of image differences by using a probabilistic measure of similarity [5]. Current research for future man and machine development s trend, human behavior is engaged and a central concern to let the machine to understand or know
man s intention through man s behavior [6]. Successful user modeling relies on capturing the correct meaning from user s activity, preference, choice made and satisfaction. These involves sensing human behavioral signals especially in multimodal channels like visual (gestures), motor (activities), perceptions (intelligent and emotional deduction) and the like. The main objective is to bridge the gap of interactive applications as different users have different needs, different types of queries to be answered, different expectation and different intentions. The goal of human computing is to go beyond the interface and to explicitly address human factors at all levels of computations [7] [8]. References: 1. Recognition. Massachusetts Institute of Technology, Cambridge, MA 02139, USA. 1999. J.C.R.Licklider. Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics. 1960. 2. Robert W.Taylor. Man-Computer Input-Output Techniques. IEEE Transactions on Human Factors in Electronic, VOL.HFE-8, No.1, March 1967. 3. L.Balint. The Role of Models in Handling Complexity, Flexibility and Reliability of Human-Computer Interfaces. Department of Natural Sciences Hungarian Academy of Sciences Budapest, Munich F.u.y. Hungary. 1989 4. Vladimir I.Pavlovic, Rajeev Sharma, Thomas S.Huang. Visual Interpretation of Hand Gestures for Human-Computer Interaction. The Beckman Institute and Department of Electrical and Computer Engineering, University of Illinois, Urbana. 1997. 5. Baback Moghaddam, Tony Jebara, Alex Pentland. Bayesian Face Recognition. Massachusetts Institute of Technology, Cambridge, MA 02139, USA. 1999. 6. Ernest Edmonds and Greg Turner. Approaches to Interactive Art Systems. Creativity and Cognition Studios, Faculty of Information Technology, University of Technology, Sydney.2004. 7. Pantic, M., Pentland, A., Nijholt, A., Huang,T. Human computing and machine understanding of human behavior: a survey. 8th International conference on multimodal interfaces (ICMI2006). 8. Elgammal, A. Human-centered multimedia: Representations and challenges. ACM International workshop on human-centered multimedia (HCM2006).
Problem and Justification Problem is found in the current proposed research is Interactive Multimedia acts meaningless and not intelligent, which every changes take place in the current developed interactive multimedia is predicted, due to, the interactive is changed according to its internal mechanism when influenced by spectator s interaction. By the following, spectator needs to know how to operate or interact with the Interactive Multimedia under the inflexibility condition of Interactive Multimedia. Lastly, spectator s behavior is hardly to be predicted, until today there is not a proper ways or rules to determined human behaviors. So, the uncertainty of the human behavior from the spectator is high. Justification to solve the problems found is implementing machine learning to learn and analyze the human behavior to tell the machine what to response. The Intelligent Interactive Multimedia is managed to communicate and understand the spectator intention and response accordingly by adapting spectator s intention into the Interactive Multimedia s content. Movements, Distance, Speed, Time, and Position are the features to classify the spectator are belonging to which category of Human Behavior. Bayesian network help to determine the uncertainty of Human Behavior through method of probability. Proposed Research Methodologies 1. Literature review which in the area of Man and Computer study, Human Computer Interaction, Human Computer Interface and Interactive Digital Media. 2. Using webcam / infrared camera to model user s gesture when the user communicating with the interactive multimedia. 3. Data gathering based on the features proposed from different group of users. 4. Machine learning to learn the data gathered from different group of users. And then, machine learning will process and analyze the user s behavior is belong to which category of human behavior. 5. Prototype of interactive multimedia. (ie. Intelligent Bubbles Interactive Multimedia).
Gantt chart for 2 Years Planning
For the complete Gantt Chart, please refer to the attachment Microsoft Office Project ResearchMilestone.pdf
List of recent references related to the study. 1. William A.S.Buxton. Human Skills in Interface Design. University of Toronto & Xerox Palo Alto Research Center. 1994. 2. Yun Long Lay. Hand Shape Recognition. Department of Electronic Engineering, National Chin-Yi Institute of Technology, Sec 1, Chung-Shan Road, Taiping, Taichung 411,Tawan. 200. 3. Lars Bretzner, Ivan Laptev, Tony Lindeberg. Hand Gesture Recognition using Multi-Scale Colour Features, Hierarchical Models and Particle Filtering. Centre for User Oriented IT Design (CID) Dept of Numerical Analysis and Computer Science, KTH, 100 44 Stockholm, Sweden. 2002. 4. Dina Goren-Bar. Intelligent System for Interactive Entertainment. Ben-Gurion University of the Negev and Center for Scientific and Technological Research, Trento. 2006. 5. Liu Yun, Zhang Peng. An Automatic Hand Gesture Recognition System Based on Viola-Jones Method and SVMs. College of Information Science and Technology, Qingdao University of Science and Technology. 2009