GUIBDSS Gestural User Interface Based Digital Sixth Sense The wearable computer

Transcription

1 2010 GUIBDSS Gestural User Interface Based Digital Sixth Sense The wearable computer By: Abdullah Almurayh For : Dr. Chow UCCS CS525 Spring /4/2010

2 Contents Subject Page 1. Abstract 2 2. Introduction 3 4. Extra Sensory Perception 4 5. Gesture Recognition 5 6. Digital 6th sense 6 7. Conclusion References 13 P a g e 1

3 1. Abstract: Most people have been using the computer for long time; however, people are still finding the best ways to make their actions with computers much faster and easier. Some people do not like typing; all the while they still have to type commands or interesting writs. Reporters, as an example, do have a lot of things to do at the same moment. They need to report an event, by recording, typing, picturing, videoing, and asking or presenting the report as much as possible. In fact, a reporter cannot do all of them at the same time since each job has its procedure. Therefore, we need techniques that can gather these tasks and execute them as much needed.. In the last several years, we have been able to act with the computer and command it by just talking to it, such as auto representative. In addition, we have seen how much the computer is commanded to do all sort of tasks by programming it and applying new schemes that involve enhancing its abilities. Now let us think about this question: Can the computer sense what we intend to do? That means whatever I want the computer to do, can it be done by just acting and behaving as we do with each other in human being sense. In this paper we are going to see how this question was already answered by the technology that is called Digital Sixth Sense. P a g e 2

4 2. Introduction: Since the computer was discovered, it has been evolved rapidly to meet the needs of humans. Recently, we have seen how much the computer has been involved dealing efficiently with businesses, industries, education, sciences, medicine, and the like. Although, the computer has become the strongest scientific revolution of enormous breadth, the Internet has become the future. The computer is the most significant assistant device since it has most of the senses that human have. Computer experts have made and gathered all sorts of hardware and software that are acting like humans. Therefore, it processes inputs and outputs as human do, but by different ways. It speaks using audio components to produce all kind of sounds. It listens via microphones and responds based on what it hears. In addition, it recognizes smells by analyzing them based on what a smell consists of and responds appropriately. It sees through digital cameras and responds depending on which purpose it is made for. The computer can touch things and recognize what they are. Moreover, the computer is able to know what things taste like by using a sort of specially programmed hardware using computations. The interesting thing is that, it can move since there have been many kinds of robots that move everywhere independently. It is true that the computers do not look like humans since it does not have a soul, and cannot feel naturally or exactly as the human, but it is getting closer to doing what humans can do. It even responds rapidly and intelligently. According to the computer evaluations, we can expect much more than what we have so far. Graphical user interfaces (GUI) have improved computer usability that makes it work in many fields and environments. GUI gives people the opportunity to interact with programs in more ways than just typing. Many examples include hand-held devices such as IPods, MP3 Players, portable media players or gaming devices; household appliances and office equipment with images rather than text commands. Graphical user interfaces use visual indicators and graphical icons with the manipulation of the graphical elements to represent the information and actions available to a user. However, digital cameras capture photographs or take video digitally by recording an image or series of images recording the sounds at the same time via electronic image sensors. Digital cameras, on the other hand, have been re-evaluated to do many amazing roles, such as surveillance, observation, monitoring, examination, and so on. Recently, digital cameras have been incorporated into many devices ranging from personal digital assistants (PDA), mobile phones, vehicles, games, and more. Wikipedia addresses that The Hubble Space Telescope and other astronomical devices are essentially specialized digital cameras [1]. P a g e 3

5 3. Extra Sensory perception Extra sensory perception (ESP) means recognizing physical senses that involve predicting information that has not yet been gained or discovered. This information is sensed with the mind, but not actually scientifically proven. Wikipedia tells, regarding to extra sensory perception phenomenon, that The term was coined by German psychical researcher, Rudolf Tischner, and adopted by Duke University psychologist J. B. Rhine to denote psychic abilities. [2] In fact, extra sensory perception is well known commonly as the sixth sense for human beings. Wikipedia also addresses that The term implies acquisition of information by means external to the basic limiting assumptions of science, such as that organisms can only receive information from the past to the present [2]. Since the sixth sense is the other term for extra sensory perception, and since this idiom is well known and widely used, we are going to use it in this paper as a reference of extra sensory perception. The expression "sixth sense" is considered an incorrect because besides the traditional five senses sight, hearing, touch, smell, and taste, there is only one additional sense that called Senses of act [4], a classification attributed to Aristotle. According to Aristotle, there are three great senses of act: movement, form, and knowledge rather than the other senses, like nature, habit, and so on, that can be reduced to these three. However, neurologists do not agree that there are a certain number of senses since they have differing definitions of based on what constitutes a sense. They have criticized the categorization for human senses by saying it is restrictive as it does not include categories for newly accepted senses. The prime purpose of introducing that is to agree with saying sixth sense because this name would be appended with the word digital as a reference for the computer involved in using it. P a g e 4

6 4. Gesture Recognition: Gesture recognition relates to movements that are reflected from human body origins, usually the face and hands. Any bodily motion or state is related to a special response. There have been many schemes that have been made based on cameras and computer algorithms to analyze the meaning of motion language. The idea of having computers to begin to understand human body language defines GestureRecognition. Because of that, we can make a strong relationship between computers equipment and humans. This should be an interaction that is more than primitive text or graphical user interfaces in which techniques are still usually the majority of using keyboard, mouse, pen, scanner, and the like. In fact, Gesture Recognition allows humans to interact with machines naturally without any mechanical devices by interfacing them using human-machine interface techniques. The purpose of this complicated work with using the concept of gesture recognition use parts of the human body such as a finger to point at the computer screen so that the cursor will move accordingly. We have seen such that these techniques work with touch screens that have been used widely in many industrial concepts such as ATM machines, factories, medicines, phones, and so on. Accordingly, we can realize that we can go on and see the human being interact with the computer much more smartly that the computer can figure out what a human wants based on his behavior. The key is that Gesture Recognition can be conducted with techniques from computer vision and image processing. Computer vision is the science and technology of machines that are enabled to see with hardware and software technology. As a scientific discipline, computer vision is concerned with the theory behind artificial systems that extract information from images. The image data can take many forms, such as video sequences, views from multiple cameras, or multi-dimensional data from a medical scanner. Then, the data can be used for human interaction. [5] Wikipedia said. P a g e 5

7 5. Digital Sixth Sense: 5.1. Understanding Digital Sixth Sense: Digital Sixth Sense is gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information. It can be a wearable, portable, locatable, and settable. Digital Sixth Sense decreases the gap between physical things and digital things that can bring intangible or digital information out into the physical world. The Digital Sixth Sense has viable and flexible system that apply different kinds of applications to demonstrate that they are very useful. Recently, we have had gestures supported by multi-touch based systems that let the user zoom in, zoom out or pan using intuitive hand movements. Therefore, it would be possible for the map application to let the user navigate a map displayed on walls, smooth surfaces and so on, using hand gestures. There have been drawing applications that let users draw on any surface just via tracking the fingertip. Regarding our topic Digital Sixth Sense, it basically recognizes the user s freehand status. For instance, it implements a gestural camera that takes quick picture of the scene the user is looking at and by understanding what the movement of the user s fingers means the computer responds based on what the user intends. In fact, this invention has been implemented by Pranav Mistry, who has been working as a Research Assistant at Fluid Interfaces, Massachusetts Institute of Technology (MIT) Media Lab since Sixth Sense aims to integrate information and technology into everyday life Digital Sixth Sense architecture Architecturally, the digital sixth sense consists of external equipment, but in a different way. Pranav Mistry has combined four fundamental hardware devices: P a g e 6

8 1. Mobile Computer: this computer is an extremely popular cell phone that is called IPhone, which was produced by Apple Company. Sometime it is called mobile wearable computer. This device is characterized by the presence of thirdgeneration 3G service for there Internet that it can grab the information from the Internet when needed. 2. Pocket Projector: this device is basically the way of outputting the information that it can be seen and watched on surfaces such as walls, doors, boards, and the like. For this implementation the projector is tiny and it can be carried easily. Pranav Mistry used 3M MPro110 projector since it is designed to be carried and produce the best show. 3. Camera: This particular equipment is the main input device that recognizes the human fingers and send the shape of the movement as an input signal the computer based in some special application that can analyze what the movement means and how it should respond to the user. He actually used a Logitech QuickCam which is one of the most commonly used cameras. 4. Colored Markers: these markers are rolled on the user s fingers two markers per hand. They are colored so that the camera can recognize their movements and focus on them without being distracted by other nearby movements P a g e 7

9 5. Mirror: A mirror is used to reflex the outputs coming from projector into surfaces. According to Mistry, he says that The SixthSense prototype is comprised of a pocket projector, a mirror and a camera. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to the mobile computing device in the user s pocket. The projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks user's hand gestures and physical objects using computer-vision based techniques [9]. That hardware or devices cannot without software that links them talk to each other and process inputs to show the output. Pranav Mistry used three important software applications: 1. Microsoft Windows platform: It focuses on windows graphical user interface programming using Microsoft C# language. 2. WPF: This software is named The Windows Presentation Foundation and it is defined as a graphical subsystem for rendering user interfaces in applications for windows environments. Wikipedia has stated that It aims to unify a number of application services: user interface, 2D and 3D drawing, fixed and adaptive P a g e 8

10 documents, advanced typography, vector graphics, raster graphics, animation, data binding, audio and video.[10] 3. OpenCV: It stands for Open Source Computer Vision which is a library of programming functions that used for real time computer vision. Pranav Mistry in addition said The software program processes the video stream data captured by the camera and tracks the locations of the colored markers (visual tracking fiducials) at the tip of the user s fingers using simple computer-vision techniques. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces. The maximum number of tracked fingers is only constrained by the number of unique fiducials, thus SixthSense also supports multi-touch and multi-user interaction [9]. P a g e 9

11 He has made reference that these components of SixthSense are a prototype and costs 350 dollars to be implemented. In fact, it so easy that anyone can have a simple web cam, portable battery-powered projector, and a small mirror and link them to the device cell and then hung in the neck. However, this procedure can do more than we expect [9] Digital Sixth Sense work It works simply by having the camera recognize and track user s hand gestures and physical objects using computervision based techniques. After that, based on the incoming video stream data, captured by the camera, the specialized programs or applications process and address the data. Then, the computer immediately responds to the user based on indexing results of movements by the corresponding meaning or action. Finally, regardless of what information is going to be the result, the mini projector projects visual information enabling surfaces, walls and physical objects around us to be used as interfaces. Pranav Mistory represented an example of this procedure saying drawing a magnifying glass symbol takes P a g e 10

12 the user to the map application or drawing symbol lets the user check his mail [9]. He also said a newspaper can show live video news or dynamic information can be provided on a regular piece of paper. The gesture of drawing a circle on the user s wrist projects an analog watch to effectively tell time of day [9]. P a g e 11

13 6. Conclusion: Since human beings have first started to use computers, there have bee great advancements in technology. The benefits have been good but there is also some bad. The user must learn how to work the computer. It is a difficult task for many, but there are people that have learned ever since computers were first used. Users wanted and easy way to work with computers and developments in technology began looking for ways to make human interaction with computers easier. The Graphical User Interface was one such advancement. The user did have to use a Command Line Interface any more and many users began to work with computers using an easier interface. Developers, engineers and vendors are constantly researching and looking for easier ways to interface with computers. There have been many prototypes that have tested but none have really been accepted. Many reasons can cause this, such as cost, complex hardware and software, customer acceptance, etc. However, there is the SixthSense device that may well be the next major advancement since the GUI. Mobile users may have the device available sooner than they think. The trend with PDA s, cell phones, and Blackberry s is not over but it may that the SixthSense may possibly take a portion of the market share. Although, SixthSense shows promising benefit, it will still need to go through the test of end user acceptance. The device in not expensive, it is easy to use which leads to customer acceptance, and the hardware and software are simple and already developed technologies. The idea is simple and easily implemented that makes learning the techniques for motion recognition a small task. Attention in the future should be given to this device. It has the potential for a desired product for mobile users, but still cannot replace a high end workstation required for many enterprise requirements. P a g e 12

14 7. References [1] Digital Camera, [2] Extrasensory Perception, [3] What Is The Sixth Sense, [4] Potentiality and Actuality, [5] Gesture Recognition, [6] MIT researchers make 'sixth sense' gadget, [7] MIT's 6th Sense device could trump Apple's multitouch, [8] SixthSense the wearable computer, by Jon Trux, ENDS 400 WDS SCIENCE PHOTO LIBRARY, 2009 [9] SixthSense the wearable computer, [10] Windows Presentation Foundation WPF, [11] Open Source Computer Vision, P a g e 13