Symbiotic Human-Computer Interaction

Transcription

1 Symbiotic Human-Computer Interaction Emil M. Petriu University of Ottawa, Ottawa, ON, Canada Thomas E. Whalen CRC, Ottawa, ON, Canada

2 Abstract The presentation will address basic principles and discuss examples of symbiotic human-computer interaction for the next evolutionary stage of the computing technology. It is expected that current "human-computer interaction" (HCI) interfaces will evolve into more efficient multimedia HCI symbiotic partnerships in which humans will contribute human-specific capabilities complementing those of the computer. It will be a symbiotic relationship in which each partner will lead in some cases and provide assistance in others. The leader/assistant role of a partner will be decided on the basis of maximizing the overall efficiency of the symbiotic team.

3 2.83 Kg TRANSDUCER MEASURAND (weight, voltage, power, ) The classic measurement process: an early example of human-transducer cooperation

4 Robots emulate humans

5 The definition given by M. Bradley: "Robotics is the intelligent connection of the perception to action" considers robotics from a system integration perspective, indicating how robots are doing things. Programmable robots (manipulators,vehicles) provide the action function. A variety of sensors provide the perception capability. Computers provide the framework for integration/connection as well as the intelligence needed to coordinate in a meaningful way the perception and action capabilities.

6 Robot Controller - Hierarchical Architecture Artificial intelligence level, where the program will accept a command such as Pick up the bearing and decompose it into a sequence of lower level commands based on a strategic model of the task. Control mode level where the motions of the system are modelled, including the dynamic interactions between the different mechanisms, trajectories planned, and grasp points selected. From this model a control strategy is formulated, and control commands issued to the next lower level. Servo system level where actuators control the mechanism parameters using feedback of internal sensory data, and paths are modified on the basis of external sensory data. Also failure detection and correction mechanisms are implemented at this level.

7 Model (human s image) of the real world as he/ she perceives it trough his/her sensory organs Off-Line Programing by Human ROBOT CONTROL SYSTEM SENSORS SENSORS (EXTEROCEPTORS) (EXTEROCEPTORS) ACTUATORS MULTI-SENSOR FUSION WORLD MODEL HIGH LEVEL INFORMATION EXCHANGE MECHANISMS: XML, Grammars,... HYBRID DELIBERATIVE/REACTIVE CONTROL MULTI-CARRIER COMMUNICATION MECHANISMS DISTRIBUTED COMPUTING FRAMEWORK: WIRELESS COMMUNICATION NETWORK Traditional human-computer interaction

8 Tele-Robotic Manipulation Head Mounted Display Haptic Feedback Virtual model of the object manipulated in the physical world Video Camera Robot Arm Tactile Sensors Manipulated Object Video and haptic virtual reality interfaces allow a human operator to interactively control a remote robot manipulator equipped with video camera and tactile sensors.

9 Early (80s) Video and HapticTelerobotic System E.M. Petriu, D.C. Petriu, V. Cretu, "Control System for an Interactive Programmable Robot," Proc. CNETAC Nat. Conf. Electronics, Telecommunications, Control, and Computers, (in Romanian), pp , Bucharest, Romania, Nov System architecture Robot hand and tendon driven compliant wrist (a) The robotic tactile sensor, and (b) the human tactile feedback

10 Distributed Interactive Telemanipulation System under development at the University of Ottawa Virtual Operation Theater O B J ( N )... O B J ( 1 )... O B J ( j ) 3D Geometric & Elastic Composite Model of Object {( x, y, z,e ) p = 1,..,P } p p p p A V A T A R H A N D ( k ) A p p l i c a t i o n S p e c i f i c I n t e r a c t i v e A c t i o n S c e n a r i o { [ 3 D ( j ) & F ( k, j ) ], t } Composite Haptic Interaction Vector between User (k) and Object (j) Haptic Human Interface U S E R ( k ) Haptic Robot Interface ROBOT(k) NETWORK OBJ(i) CyberGrasp CyberTouch Robot Arm Controller Tactile Sensor Interface It allows any user USER(k) (k=1,...,nu) to experience the haptic handson feeling while dexterously manipulating remote physical objects via a given robotic telemanipulator ROBOT(k) (k=1,...,nr) equipped with haptic and video sensors. An impedance-reflecting virtual operation theatre contains the composite geometric & haptic models of the kinematic and dynamic interaction behaviour of all manipulated objects. Haptic transport protocols allow to compensate for network lag and preserve synchronization in shared object manipulation.

11 Human-Computer Interaction in the Tele-Robotic Manipulation System The robotic telemanipulation system has a bilateral architecture allowing to connect the human operator and the telerobotic manipulator as transparently as possible. Conformal (1:1) mapping of human & robot sensory and perception frameworks Using a head mounted display for augmented visual virtual reality and a haptic feedback system, the human operator controls the operation of a remote robot manipulator equipped with video camera and tactile sensors placed in robots hand. The tactile sensors provide the cutaneous information at the remote robotic operation site. The joint sensors of the robot arm provide the kinesthetic information. A tactile human-computer interface provides the cutaneous feedback allowing the human operator to feel with his/her own sense of touch the same sensation as that acquired by the remote robot hand from its artificial tactile sensor. A robot-like kinematic structure provides the kinesthetic feedback for the haptic human-computer interface.

12 Model (human s image) of the real world as he/ she perceives it trough his/her sensory organs Off-Line Programing by Human MULTI-MEDIA HCI INTERFACES Real-Time Interactive Programing by Human ROBOT CONTROL SYSTEM SENSORS SENSORS (EXTEROCEPTORS) (EXTEROCEPTORS) ACTUATORS MULTI-SENSOR FUSION WORLD MODEL HIGH LEVEL INFORMATION EXCHANGE MECHANISMS: XML, Grammars,... HYBRID DELIBERATIVE/REACTIVE CONTROL MULTI-CARRIER COMMUNICATION MECHANISMS DISTRIBUTED COMPUTING FRAMEWORK: WIRELESS COMMUNICATION NETWORK Human-Computer interaction for symbiotic operations

13 The time clutch concept is used to disengage synchrony between operator specification time and tele-robot operation time during path specifications. In order to avoid fatal errors and reduce the effect of the communication delay, we are using a distributed virtual environment allowing to maintain a shared world model of the physical environment where the telemanipulation operation is conducted]. Another critical requirement is the need to maintain the synchronism between the visual and the haptic feedback. While being two distinct sensing modalities, both haptic perception and vision measure the same type of geometric parameters of the 3D objects that are manipulated.

14 Visual-, geometric-, force-, and touch domain human -feedback devices

15 Biologically inspired robotic manipulator consists of a robot arm an instrumented passive-compliant wrist and a 16-by-16 tactile probe. Position sensors placed in joints and on the instrumented passive-compliant wrist provide the kinesthetic information. The compliant wrist allows the robot-hand equipped with tactile sensors to accommodate the constraints of the explored object surface and thus to increase the amount of cutaneous tactile information.

16 The tactile probe is based on a 16-by-16 matrix of Force Sensing Resistor (FSR) elements spaced 1.58 mm apart on a 6.5 cm2 (1 square inch) area. The FSR elements exhibit exponentially decreasing electrical resistance with applied normal force: the resistance changes by two orders of magnitude over a pressure range of 1 N/cm2 to 100 N/cm2.

17 The elastic overlay has a protective damping effect against impulsive contact forces and its elasticity resets the transducer system when the probe ceases to touch the object. However, they may cause considerable blurring distortions in the sensing process if they are not properly controlled. We avoided it by replacing the one-piece pad with a custom-designed elastic overlay consisting of a relatively thin membrane with protruding round tabs.. 16-by-16 median filtered tactile image of a washer.

18 Human Kinesthetic Perception The kinesthetic sensing provides information about the positions and velocities of the kinematic structure of the hand. The human kinesthetic capability has a much lower frequency band than the cutaneous perception provided by the fingertips.

19 Immersionn_3D Interaction < CyberGlove Uses linear sensors electrical strain gauges; Angles are obtained by measuring voltages on a Wheastone bridge; 112 gestures/sec filtered. Sensor resolution 0.5 degrees, but errors accumulate to the fingertip (open kinematic chain); Sensor repeatability 1 degree Needs calibration when put on the hand

20 Immersionn_3D Interaction < CyberGrasp CyberGrasp Pack CyberForce

21 Commercial Virtual Hand Toolkit for CyberGlove/Grasp providing the kinesthetic human-computer interface

22 Human Cutaneous/Tactile Perception The cutaneous/tactile sensing provides information about contact force, contact geometric profile and the temperature of the touched object. There are various cutaneous mechanoreceptors located in the outer layers of the skin. These receptors are specialized nervous cells or neurons. The free nerve endings are the most numerous and play an active role in the perception of pain, cold and warmth. These mechanoreceptors have preferential frequency response characteristics: the highest sensitivity for the Pacinian Corpuscles (PC) units is around Hz but they respond from 30 Hz to very high frequencies. The Rapidly Adapting (RA) units effective frequency range is between 10 and 200 Hz, with more sensitivity below 100 Hz. The Slowly Adapting (SA) units respond at low frequencies, under Hz. ( The human kinesthetic function has a much lower frequency band.)

23 [Burdea& Coiffet 2003] G. Burdea and Ph. Coiffet, Virtual Reality Technology, (second edition), Wiley, New Jersey, 2003 Two-point limen test: 2.5 mm fingertip, 11 mm for palm, 67 mm for thigh (from [Burdea& Coiffet 2003] ).

24 Immersionn_3D Interaction < CyberTouch

25 Tactle/Cutaneous Feedback to the Human Operator R O B O T - H A N D TS TS TACTILE IMAGE ACQUISITION H U M A N - H A N D TM TM TACTILE SENSATION RECONSTRUCTION TS = Tactile Sensor TM = Tactile Monitor A tactile monitor placed on the operator's palm should allow the human teleoperator to virtually feel by touch the object profile measured by the tactile sensors placed in the jaws of the robot gripper.

26 Cutaneous tactile HCI developed at the University of Ottawa in the early 90s. It consists of an 8-by-8 array of electromagnetic vibrotactile stimulators. The active area is 6.5 cm2 (same as the tactile sensor). The figure shows a curved edge tactile feedback.

27 Human-Computer Symbiont (Cyborg) Systems Human operator and intelligent sensor-based systems work together as symbionts (cyborgs) each contributing the best of their specific abilities. Proper control of these operations require humnan-computer interfaces capabilities allowing the human operator to experience the feeling of virtual immersion in the working environment.

28 Interfaces enhancing the performance figures of the human s natural capabilities Brain PDA Eye eye glasses, binoculars, IR night vision device, HMD for augmented VR,.. Ear hearing amplifier NORMAL BASIC HUMNAN BEING Hand gloves (baseball glove), hand tools (pliers) Leg footwear, skates, bike, exoskeleton,.. ARTIFICIALLY ENHNCED HUMAN = SYNBIONT?

29 Prosthetic interfaces enhancing the performance figures of the human s natural capabilities (including survivability) Eye + Artificial Cornea Ear + Hearing Aid (Implant) CRIPPLED HUMNAN BEING Pacemaker Artificial Hand Knee Joint + Titanium Knee Joint ARTIFICIALLY ENHNCED HUMAN = SYNBIONT

30 Interfaces enhancing the performance figures of the human s natural capabilities Brain PDA Eye eye glasses, binoculars, IR night vision device, HMD for augmented VR,.. Eye + Artificial Cornea HEALTY OR CRIPPLED HUMNAN BEING Ear + Hearing Aid (Implant) Pacemaker Hand Artificial Hand gloves (baseball glove), hand tools (pliers) Knee Joint + Titanium Knee Joint footwear, skates, bike, exoskeleton,.. ARTIFICIALLY ENHNCED HUMAN = SYNBIONT!

31 Human beings are only valuable in this symbiotic partnership to the degree that their capabilities complement those of the computers: (i) (ii) (iii) (iv) (v) humans are still far more intelligent than any computer, are able to act on incomplete or ambiguous instructions, able to adapt to a variety of computer interfaces, and able to interact directly with other humans, humans are mobile being able to leave the vicinity of the computer, perform complex tasks in a variety of different environments, and then return to report on the outcome, humans can recognize visual, auditory, olfactory, gustatory, and haptic stimuli, humans are dexterous, which allows them to precisely manipulate a wide variety of objects, humans are emotional, varying the characteristics of response, depending on the global state of each individual. E.M. Petriu, T.E. Whalen, "Computer-Controlled Human Operators," IEEE Instrum. Meas. Mag., Vol. 5, No. 1, pp , 2002.

32 Symbionts will combine intrinsic machine-sensing reactive behavior with higher-order human-oriented world-model representations of the immersive virtual reality. Humans are very high-bandwidth creatures: - their visual system is capable of perceiving more than a hundred megabits of information per second, and - their largest sense organ, the skin is capable of perceiving nearly that much as well. - human speech conveys information in the form of intonation and inflection as well as the actual words uttered. People communicate through "body language which includes facial expressions and eye movements.

33 Human-to-human communication and cooperation require a common language and an underlying system of shared knowledge and common values. In order to achieve a similar degree of human-to-computer interaction and cooperation, a symbiotic framework should be developed to allow for the management of heterogeneous functions and knowledge.

34 Cyber/Machine Concept Representation Language Cyber/Machine Society/World {Intelligent Robot Agents} Human Society/World {Human Beings} Human Concept Representation Language

35 Asimov s laws of the robotics: 1 st law: A robot must not harm a human being or, through inaction allow one to come to harm. 2 nd law: A robot must always obey human beings unless that is in conflict with the 1 st law. 3 rd law: A robot must protect itself from harm unless that is in conflict with the 1 st and 2 nd law. Cyber/Machine Concept Representation Language Cyber/Machine Society/World {Intelligent Robot Agents} Human Society/World {Human Beings} Human Concept Representation Language

36 Cyber/Machine Concept Representation Language Cyber/Machine Society/World {Intelligent Robot Agents} Symbiont Concept Representation Language Symbiont Society/ World {Cyborgs} Human Society/World {Human Beings} Human Concept Representation Language

37 Cyber/Machine Concept Representation Language Symbiont Concept Representation Language Multi-Cultural Human & Cyber& Symbiont Hyper-Society World Hyper-Society Common Concept Representation Meta-Language Human Concept Representation Language

38 Asimov s laws of the robotics: 0 th law: "A robot may not injure humanity or, through inaction, allow humanity to come to harm." 1 st law- updated: A robot must not harm a human being or, through inaction allow one to come to harm, unless this would violate the 0 th law." 2 nd law: A robot must always obey human beings unless that is in conflict with the 1 st law. 3 rd law: A Human robot must protect itself Concept from harm unless that Representation is in conflict with the 1Language st and 2 nd law. [*] I. Asimov, Robots and Empire, Doubleday & Co., NY 1985, p.291 Multi-Cultural Human & Cyber& Symbiont Hyper-Society World Hyper-Society Common Concept Representation Meta-Language

39 BORG Hyper-Society World BORG Hyper-Society Common Concept Representation Meta-Language

40 Moral, Ethical, Theological, Legal, Biological, Psychological Social, Economic,. Challenges in a BORG Hyper-Society World [Normal Human Partner] + [Pacemaker-fitted Human Partner] = [Acceptable Married (incl. Lovers)_Couple] [Normal Human Partner] + [Advanced Augmented Symbiont Partner] = [Acceptable Married (incl. Lovers)_Couple]? [Normal Human Partner] + [Robot Partner] = [Acceptable Married (incl. Lovers)_Couple]??? => Absolutely! according to Asimov s Prelude to the Foundation

41 .. or an alternative No Robots!!! No Symbionts!!! Human ( no machine allowed) Society/World {Human Beings} Human Concept Representation Language

42 ... The GALACTIC EMPIRE: Robots banned.. but Eto Demerzel. Hari Seldon s Psychohistory. The FOUNDATION => SECOND GALACTIC EMPIRE Human Society/World {Human Beings} Human Concept Representation Language

43 Publications A.-M. Cretu, J. Lang, E.M. Petriu, A Composite Neural Gas-Elman Network that Captures Real-World Elastic Behavior of 3D Objects, Proc. IMTC/2006, IEEE Instrum. Meas. Technol. Conf., pp , Sorrento, Italy, April A.-M. Cretu, E.M. Petriu, Neural-Network Based Adaptive Sampling of Three- Dimensional-Object Surface Elastic Properties, IEEE Trans. Instrum. Meas.," Vol. 55, No. 2, pp , A.-M. Cretu, E.M. Petriu, G.G. Patry, Neural-Network-Based Models of 3- Objects for Virtualized Reality: A Comparative Study, IEEE Trans. Instrum. Meas.," Vol. 55, No. 1, pp , P. Payeur, C. Pasca, A.-M.Cretu, E.M. Petriu, Intelligent Haptic Sensor System for Robotic Manipulation, IEEE Trans. Instrum. Meas., Vol. 54, No. 4, pp , T.E. Whalen, D.C. Petriu, L.Yang, E.M. Petriu, Interactive Scripts for Haptic Virtual Environments, Proc. HAVE IEEE Intl. Workshop on Haptic, Audio and Visual Environments and their Applications, pp , Ottawa, ON, Canada.t E.M. Petriu, S.K.S. Yeung, S.R. Das, A.-M. Cretu, H.J.W. Spoelder, Robotic Tactile Recognition of Pseudorandom Encoded Objects, IEEE Trans. Instrum. Meas., Vol. 53, No. 5, pp , E.M. Petriu, T.E. Whalen, R. Abielmona, A. Stewart, "Robotic Sensor Agents," IEEE Instrum. Meas. Mag., Vol. 7, No. 3, pp , T.E. Whalen, D.C. Petriu, L. Yang, E.M. Petriu, M.D. Cordea, Capturing Behaviour for the Use of Avatars in Virtual Environments, CyberPsychology & Behavior, Vol. 6, No. 5, pp , P. Rusu, E.M. Petriu, T.E. Whalen, A. Cornell, H.J.W. Spoelder, Behavior-Based Neuro-Fuzzy Controller for Mobile Robot Navigation, IEEE Trans. Instrum. Meas., Vol. 52, No. 4, pp , E.M. Petriu, "Neural Networks for Measurement and Instrumentation in Virtual Environments, in Neural Networks for Instrumentation, Measurement and Related Industrial Applications, (S. Ablameyko, L. Goras, M. Gori, V. Piuri Eds.), NATO Science Series, Computer and System Sciences Vol. 185, pp , IOS Press, E.M. Petriu, T.E. Whalen, "Computer-Controlled Human Operators," IEEE Instrum. Meas. Mag., Vol. 5, No. 1, pp , 2002.

44 E.M. Petriu, T.E. Whalen, V.Z. Groza, Haptic Perception System for Robotic Tele-Manipulation, Proc. INES 2002, 6th International Conference on Intelligent Engineering Systems 2002, pp , Opatija, Croatia, May E.M. Petriu, D. Ionescu, D.C. Petriu, S.K. Yeung, Ph. Lavoie, N. Trif, Absolute Position Measurement Applications of Pseudo-Random Encoding, Proc. ETIM'96 IEEE Intl. Workshop on Emergent Technol. for Instrum. Meas., pp , Como, Italy, S.K. Yeung, E.M. Petriu, W.S. McMath, D.C. Petriu, "High Sampling Resolution Tactile Sensor for Object Recognition," IEEE Trans. Instrum. Meas., Vol. 43, No. 2, pp , W.S. McMath, M.D. Colven, E.M. Petriu, S.K. Yeung, "Tactile Pattern Recognition Using Neural Networks," Proc. IEEE&SICE IECON'93 Conf., pp , Maui, Hawaii, E.M. Petriu, W.S. McMath, "Tactile Operator Interface for Semi-autonomous Robotic Applications," Proc. Int. Symposium on Artificial Intell. Robotics Automat. in Space, i-sairs'92, pp.77-82, Toulouse, France, E.M. Petriu, W.S. McMath, S.K. Yeung, N. Trif, "Active Tactile Perception of Object Surface Geometric Profiles," IEEE Trans. Instrum. Meas., Vol. 41, No. 1, pp.87-92, S.K. Yeung, W.S. McMath, E.M. Petriu, N. Trif "Three Dimensional Object Recognition Using Integrated Robotic Vision and Tactile Sensing," Proc. IEEE&RSJ Int. Workshop on Intell. Robots and Systems IROS'91, pp , Osaka, Japan, E.M. Petriu, D. Petriu, V. Cretu, "Multi-Microprocessor Control System for an Experimental Robot with Elastic Joints," Proc. Nat. Conf. Cybernetics, (in Romanian), Bucharest, Romania, 1981