Design Principles for Safety in Human-Robot Interaction

Transcription

1 Noname manuscrpt No. (wll be nserted by the edtor) Desgn Prncples for Safety n Human-Robot Interacton Manuel Gulan Claus Lenz Thomas Müller Markus Rckert Alos Knoll the date of recept and acceptance should be nserted later Abstract The nteracton of humans and robots has the potental to set new grounds n ndustral applcatons as well as n servce robotcs because t combnes the strengths of humans, such as flexblty and adaptablty, and the strengths of robots, such as power and precson. However, for a successful nteracton the safety of the human has to be guaranteed at all tmes. Ths goal can be reached by the use of specalsed robot hardware but we argue that safety n human-robot nteracton can also be done wth regular ndustral robots, f they are equpped wth addtonal sensors to track the human s poston and to analyse the human s verbal and non-verbal utterances, and f the software that s controllng the robot s especally desgned towards safety n the nteracton. For ths reason, we propose three desgn prncples for an ncreased safety n robot archtectures and any other software component that controls a robot for human-robot nteracton: robustness, fast reacton tme, and context awareness. We present a robot archtecture that s based on these prncples and show approaches for speech processng, vson processng, and robot control that also follow these gudelnes. Keywords Human-Robot Interacton, Safety Issues, Speech Processng, Vson Processng, Robot Control Technsche Unverstät München Robotcs and Embedded Systems Boltzmannstraße Garchng be München Germany E-mal: {gulan,lenz,muelleth,rckert,knoll}@n.tum.de 1 Introducton Robots are used n factores all over the world because they are fast, they can lft heavy weghts, they can operate n envronments that are dangerous for humans, and they can repeat the same movements over and over n a precse way. However, the applcaton of robots has one man dsadvantage: robots need to be reprogrammed for every new task they should execute. Of course, one could argue that then robots should be used for repettve tasks that need to be executed many tmes, such as weldng a car body, but would t not be useful to combne the obvously excellent capabltes of a robot wth the flexblty and adaptablty of humans? The combnaton of the strengths of humans and robots could potentally lead to new ways n producton n whch customers can choose a product that s especally customsed for them but stll was made wth robot precson. When humans should work together wth ndustral robots there s one bg lmtaton: ndustral robots are heavy machnes that can move very fast and can easly hurt a human user. Haddadn et al. report n [21] the possble dangers arsng for a human who s nteractng wth an ndustral robot. They conducted a systematc seres of tests wth several ndustral robots httng crash test dummes and lsted the njures that mght be caused by uncontrolled robot movements, whch ncluded fractures and shearng of lmbs. Ths s the reason why n factores robots usually are set up behnd fences and the only nteracton of human and robot takes place when the robot s turned off for mantenance. But how can robots be made safer for the nteracton wth humans? One way would be to buld safe robot hardware, whch was for example done for Justn,

2 2 a lght-weght robot especally desgned for nteracton wth unknown envronments and wth humans [32]. Ths approach has two man dsadvantages: frst, robots that are especally bult for a safe nteracton as a trade-off often lose one or more of ther abltes that makes them nterestng. For example they cannot lft heavy loads any more or have to move slowly. Second, the development and money that companes nvested n regular ndustral robots would be lost f they had to swtch to new safer robots for human-robot nteracton (HRI). Therefore, n order to make ndustral robots safe for HRI, we argue that these robots need to be equpped wth addtonal sensors that enable the robot to determne the human s poston and to understand the human s utterances (whch ncludes not only speech and gestures, but also also other cues, ncludng affectve states). Addtonally, the robot needs to be controlled wth a robot archtecture and software that s especally talored for safety n HRI. The safety of the human that nteracts wth a robot s one of the prerequstes to buld socal robots, snce a frendly socal nteracton s not possble f one of the partners s a threat to the other. For ths reason, we propose three elementary desgn prncples for every software that controls a robot n a safe way: robustness, fast reacton tmes, and context awareness. These desgn prncples are the outcome of development work n several projects for HRI (for example JAST 1 and JAHIR 2 ) and have proven to be essental for any software component n ths feld. The remander of ths publcaton s organsed as follows: Secton 2 ntroduces the three desgn prncples for safety n HRI and shows how modules for nput processng and robot control can be combned n a robot archtecture that follows these prncples. After that, we show how the safety desgn prncples can be appled n approaches for speech processng (Secton 3), vson processng (Secton 4), and robot control (Secton 5). Fnally, Secton 6 concludes ths publcaton and dscusses future research drectons. 2 Safety Desgn Prncples In ths secton, we descrbe three desgn prncples for robot archtectures for safe HRI and show an example of such an archtecture. Our research should extend prevous work by the CoSy project [22], whch developed three desgn prncples for cogntve systems. We also regard these prncples as essental for any goaldrected robotc system that shows cogntve abltes: Concurrent modular processng. Robot archtectures have to consst of sngle modules that run n parallel. Wth ths desgn, complcated tasks can be splt nto subtasks that are executed n parallel by specalsed parts of the system, for example the vson system may track a human whle the actuator control makes sure that the robot does not collde wth the human. Ths desgn prncple also mples that the system conssts of several specalsed subarchtectures, whch can be renewed or extended easly due to the modular archtecture desgn. Structured management of knowledge. Ths prncple organses the nformaton representaton n the system and the flow of nformaton between the sngle parts of the system. The nformaton nsde the archtecture s defned by subarchtecture ontologes and general ontologes, were the term ontology s a general expresson that stands for a representaton format. Ths means that each subcomponent of the system has ts own representaton for ts knowledge. For example, the vson system mght have an nternal representaton for objects that conssts of vectors that contan colour and shape nformaton whle externally t mght translate ths nformaton nto a strng format for other subarchtectures. Dynamc contextual processng. Ths prncple says that n order to mplement a goal-orented behavour n the cogntve system there have to be control mechansms that steer the nformaton flow of the concurrently workng system components. The CoSy project proposed to mplement ths behavour n a way were subcomponents have to announce ther ntent to process nformaton whle a controllng system component tells them f they are allowed to do that or not. We agree wth these desgn prncples. However, the CoSy project developed these prncples for cogntve robots n general wthout havng n mnd the safety of the human user workng wth the robot. Therefore, we propose three addtonal desgn prncples that are necessary to ensure the safety of the human: Robustness. In a complex system for HRI there are many potental sources for errors; at the nput and output modules of the robot as well as at the reasonng modules that control the robot s behavour. For example, one cannot assume that nput recognton modules work correctly all the tme; n fact, many tmes the opposte wll be true and recognton errors and mssng nput can be antcpated. From ths, two consequences follow to make sure that the safety n the nteracton s ncreased: the

3 3 system must be able to compensate for erroneous or mssng nput, and when errors occur the system has to be able to dentfy these errors and to develop strateges to solve them. Fast reacton tme. Robots have to process and react fast to the nput they are gettng from ther envronment. Ths has two obvous reasons: on the one hand, humans wll only accept robots f they are reactng fast to what they say or do. Most confuson n the nteracton between human and robot arses when the robot s ether not reactng at all or f t needs too much tme to react to the human s utterances. On the other hand, the robot also needs to quckly detect dangerous stuatons for the human, for example by the use of sensors or when the human says certan keywords that sgnal the robot to stop. Context awareness. When the robot processes the nput of ts sensor modules, t wll nevtably have stuatons n whch the nput s ambguous or where t has not enough nput nformaton to compute a complete hypothess of whch acton to execute next. In these stuatons, the robot has to make use of context nformaton, for example t could recognse the human s gestures or look at the objects n ts vcnty. Therefore, the robot needs sutable representaton formats for ts knowledge about the world and defned nterfaces to the software modules that provde the nformaton about the robot s envronment. The desgn prncples we are dscussng here are of course also mportant for robotcs n general, so why are we regardng these prncples as crucal for HRI? Especally, when we are talkng about robustness and fast reacton tme here, we are thnkng of specalsed forms of these propertes that a robot desgned for a safe nteracton wth a human needs. Usually, robustness n an ndustral robot means that the robot s workng even f parts of the system are malfunctonng. Ths can for example be reached by duplcatng crtcal system components. What we mean by robustness s that the robot shows a robust behavour. It must be aware that n the nteracton wth a human there mght be unclear or unexpected stuatons, to whch the robot has to react n a reasonable way, for example by at least not hurtng the human. Smlarly, n general every robot should execute the tasks t was programmed for as fast as possble, but that s not what we mean by fast reacton tme. We argue that n order to ncrease the safety for the human, the robot has to react to unknown stuatons fast and also wth a reacton that takes nto account the partally avalable context of ths stuaton as fast as pos- Fg. 1 Robot archtecture for safe human-robot nteracton. sble. Ths means that the robot has to understand the current stuaton rather than to just react to sgnal nput. Also, only wth a reasonable fast reacton tme of the robot the human can judge the robot s actons, whch also ncreases the safety for the human, as we already mentoned above. Therefore, wth these desgn prncples we propose a robot archtecture that combnes approaches that are reacton-based or low-level, whch could be seen as part of the embodment movement as descrbed by Pfefer and Bongard [33], and hgh-level methods for reasonng, whch are part of more tradtonal artfcal ntellgence (AI). Ths mxture of new and old approaches s also n agreement wth Sloman [39], who argues that an archtecture for a truly cogntve techncal system has to combne methods from embodment and tradtonal AI, whch also shows smlartes to how human perceve and reason about ther envronment. Fgure 1 shows a schematc overvew of our proposed robot archtecture. In ths archtecture, nput processng modules, whch are depcted by ovals, send the processed data to a central buffer whch we call data flow. In ths schematc archtecture overvew we show nput modules for speech, object, and gesture recognton, and for hand and person trackng; however, due to the archtecture desgn new nput modules can always be added to the system. Reasonng modules, depcted by rounded rectangles, get the data from all nput modules that are relevant for them from the data flow. In our example, we have reasonng modules for speech processng, whch gets data from speech recognton, an object nventory, whch represents results from object and gesture recognton for reference resoluton n the speech processng module, and robot control, whch uses nformaton mostly from the trackng modules but also gets nput from speech processng and uses the object nventory to locate objects n the robot s envronment.

4 4 After ths short ntroductory overvew, we dscuss were the desgn prncples for safety n HRI can be found n the archtecture and how they affect ts desgn. As the reader wll see n the remander of ths publcaton, we wll do ths dscusson for all the approaches we are presentng here. Robustness. Robustness seems to be mostly a matter for the nput processng modules that have to nterpret the robot s envronment n a stable way. However, the robot archtecture tself can also contrbute to ncreasng the overall robustness of the system. For ths, the archtecture has to guarantee a robust transmsson of nformaton n the data flow as well as to provde mechansms that help subcomponents of the archtecture to recover n case of breakdowns. Fast reacton tme. The robot archtecture s mportant for fast reacton tmes n two ways: frst, wthout a sutable nfrastructure none of the system s subcomponents can provde fast reacton tmes. The archtecture has to take care that nformaton between components flows fast and relably. Second, the archtecture has to provde dedcated fast processng channels for securty-related parts of the system. For example, the archtecture needs specal channels for robot control, whch s represented by the two broad arrows n Fgure 1. Over these channels the robot can be stopped at all tmes. Ths can be done for example from outsde the system by pressng an emergency button or by dedcated nput processng modules. For nstance a specalsed module that measures the loudness of the human utterances could stop the robot n case the loudness reaches a certan threshold, whch would ndcate an emergency stuaton. Context awareness. We thnk that context awareness has to be a bult-n feature of a robot archtecture n order for the robot to nteract wth ts envronment n a reasonable and safe way. Therefore every robot archtecture has to be developed already wth the context n mnd n whch the robot has to nteract. In our archtecture, context awareness s ncluded n two ways: Frst, the archtecture provdes the nfrastructure for the nput modules so that they can publsh ther recognton results system-wde and thus n a sense generate the context for the robot. For example, the object nventory that represents results from object and gesture recognton s a fxed part of the system. Second, snce the safety of the human co-worker of the robot s also part of the context the robot s workng n, the archtecture has bult-n mechansms to ncrease the safety for the human. For example, we already mentoned the dedcated subcomponents that are used to stop the robot n case of emergency stuatons. Now that we ntroduced the desgn prncples for safety n HRI and outlned how these apply to robot archtectures, we dscuss n the followng sectons how the prncples nfluence our approaches for speech processng (Secton 3), vson processng (Secton 4), and robot control (Secton 5). 3 Speech Processng Speech s one of the cogntve abltes that sets humans apart from other mammals. Wthout speech, humans would not be able to jont-actly work together so effectvely and safely as they do. Thus, a robot that should nteract wth a human n a safe way needs to be able to comprehend speech and other human utterances, ncludng gestures or emotons. However, the problem wth natural speech s that humans do not speak grammatcally correct and they also tend to leave out parts of the sentences they want to say. Addtonally, as Foster et al. showed n [12], many spoken expressons that refer to objects n the world can only be understood together wth the gestures that accompany the spoken part of the message. For example, Foster et al. present a real dalogue from a jont constructon task, n whch two humans assemble tangram models together. The dalogue between the two s shown n Fgure 2. human 1: And I ll get ths human 1: And then the red one human 2: Kay I ve got the yellow human 1: Cool Fg. 2 Example of a real dalogue between two humans durng a jont constructon task. In the example, the two humans are talkng about several trangles, whch both of them can see. None of them ever mentons the word trangle, because they can see whch object the other person s handlng and therefore they know whch object the other person s talkng about. The dalogue also shows how both of them never say a full grammatcally correct sentence. But how can speech contrbute to ncrease the safety n HRI? If the robot can understand what the human s sayng, for example when t knows whch object the human s gong to pck up next, then t can plan ts actons so that t does not necessarly has to operate n the same area as the human. Also, f the robot detects that the human s n the way of ts planned path, then t mght warn the user by producng an adequate speech output. Addtonally, speech s a natural way of

5 5 communcaton for humans, so f the robot can understand what the human s sayng then ths ncreases the perceved safety feelng of the human. In speech processng, the man desgn prncple we have to work on s the ncrease of the robustness of the methods. That starts already wth usng relable and fast hardware for speech recognton, but experence shows that the recognton process s always hard to control and that t can produce erroneous nput easly. Therefore, the methods for speech processng have to take nto account that the nput from speech recognton s erroneous and fragmentary. In the next sectons, we wll explan n more detal our approach for robust speech processng. 3.1 Processng Overvew Our approach for speech processng enables the robot to robustly recognse and understand what the human s sayng, to resolve whch objects the human s talkng about, and to generate an adequate response to the human. In the current verson of the approach the robot s not able to have a sophstcated conversaton wth the human. However, due to the modular setup of the speech processng archtecture, the ntegraton of a dalogue manager to the system would be a potental extenson n the future. Fgure 3 shows the speech processng subarchtecture for the robot wth the sngle steps of our approach. Rectangular boxes represent processng modules, rounded rectangular boxes are standng for the context nformaton that s avalable to the processng modules. The sngle processng steps are as follows: we recognse the speech by a human user wth a commercal speech recognser, n our case Dragon NaturallySpeakng 10. Followng speech recognton, the output of the speech recognser s preprocessed to ncrease the robustness of the system. After preprocessng, the nput s parsed by a grammar to translate t nto a logcal representaton that can be used to resolve the references n the human utterance (.e. to fnd out whch objects the human s talkng about) and to generate a response from the robot to the human. In ths response the robot can ether confrm that t understood the whole utterance correctly or ask for clarfcaton f t was not able to resolve all references. The context nformaton that s avalable to the system conssts on the one hand of a grammar that s used n the parsng and output generaton steps and from whch sentence lsts and vocabulary for preprocessng and speech recognton can be automatcally generated. On the other hand, the robot gets nformaton from the Fg. 3 Speech processng subarchtecture; rectangular boxes stand for processng steps, rounded boxes stand for context nformaton. vson system about objects n the world and about gestures the human made. In the followng sectons we wll gve a more detaled descrpton of the steps for speech processng. Snce we use a commercally avalable software for speech recognton as already mentoned above we wll skp speech recognton and start ths descrpton wth the second step n the processng chan. 3.2 Preprocessng In our speech processng approach we nether want to lmt the words or sentences the human can say to the robot nor want to gve the users nstructons on what they can say. However, we stll want to be able to use a grammar to parse the human utterances, whch has many advantages over smpler approaches such as keyword spottng. For ths reason, we transform the nput by the human nto sentences whch a grammar can parse. Ths way, we can use a grammar formalsm that has already been proven to work well for human-robot nteracton (e.g. n JAST [17], and CoSy [26]) wth only slght adjustments to the approach. For the nput transformaton, we compare the nput sentence we got from speech recognton wth a set of

6 6 nput take a take take a cube take a cube 1 1 take a bolt 1 2 take a green cube 2 2 take a yellow cube 2 2 take a red cube 2 2 take a green bolt 2 3 take a yellow bolt 2 3 take a red bolt 2 3 Fg. 4 Sentence lst wth Levenshten dstances for the nput sentences take a and take take a cube. sentences that can be generated automatcally wth the grammar we are usng n the parsng step (see Secton 3.3). For the sentence comparson we currently use an extended Levenshten algorthm [28]. Fgure 4 shows an example for two nput sentence comparsons. The left column of the table shows the lst of sentences that has been generated wth a grammar. The two columns on the rght contan the dstances of the two nput sentences take a and take take a cube to the lst of sentences. For example, the dstance between take a and take a cube s 1 because we need one nserton to yeld take a cube from take a. The two nput sentences show good examples for typcal errors n speech recognton, whch are mssng words or recognsng short words twce. In the example, the two most lkely full sentences that would be chosen for nput sentence 1 take a are take a cube and take a bolt. In ths case, the robot would have to ask the user for clarfcaton whch object t should grasp or get the mssng nformaton from some other modalty. However, snce t occurs qute often that some nformaton n the nput sentence s mssng, we added sentences wth placeholders, whch we call empty words, to the lst of sentences. In Fgure 5 you can see the lst of sentences extended by the sentences wth empty words and ther dstances to the two nput sentences from the example n Fgure 4. If we compare the nput sentence take a to the extended sentence lst that contans the empty words we get three sentences that have dstance 1 to the nput sentence. One of these sentences s take a emptynoun whch would be chosen for parsng as we wll show n the next secton. 3.3 Parsng The preprocessng step yelds an nput sentence that can be parsed by a grammar. In our speech processng mplementaton, we are usng combnatory categoral grammar (CCG) to parse and represent speech nput. CCG was ntroduced by Ades [1] and Steedman [40]. It s an extenson to the categoral grammar, whch s also nput take a take take a cube take a cube 1 1 take a bolt 1 2 take a green cube 2 2 take a yellow cube 2 2 take a red cube 2 2 take a green bolt 2 3 take a yellow bolt 2 3 take a red bolt 2 3 emptyverb a cube 2 2 take a emptynoun 1 2 take a emptyadjectve 2 2 cube emptyverb a emptyadjectve 3 3 cube emptyverb a emptyadjectve 3 4 emptynoun Fg. 5 Extended sentence lst wth empty words and Levenshten dstances for two nput sentences. called lexcalsed grammar, of Ajdukewcz [2] and Bar- Hllel [4]. Tradtonal context-free grammar formalsms use a top-down approach for parsng sentences, whle combnatory grammars utlse a bottom-up approach, whch brngs advantages n computablty and grammar development. Due to the addton of combnatory logc to the grammar formalsm, CCGs produce a semantc representaton of a sentence durng the parsng process. We are usng a CCG that was mplemented wth OpenCCG. OpenCCG [43] s a Java-based mplementaton of the CCG formalsm. It s capable of both, parsng and realsng sentences; that means t can translate utterances nto a logcal form as well as take a gven logcal form and convert t back to a sentence. OpenCCG generates hybrd logc expressons for the parsed sentence nstead of combnatory logc, as explaned n [3]. Fgure 6 shows such a hybrd logc formula that was parsed wth our grammar and represents the sentence take ths yellow cube.. The grammar we are usng for our current work s based on a grammar that was developed for the JAST t1:acton (take-verb mood mp Actor x1 : anmate beng Patent ( c1 : thng cube-np det dem-prox num sg HasProp ( y1 : proposton yellow))) Fg. 6 Hybrd logc formula that was generated wth a combnatory categoral grammar for the sentence take ths yellow cube.. In hybrd logc all actons and enttes n the sentence have socalled nomnals, whch can be seen as dentfers. To understand ths logc formula, we have to llustrate the two concepts of nomnals and damond operators that are part of hybrd logcs. Nomnals can be

7 7 seen as dentfers that are used to name parts of the logcal form. In the present case we used nomnals to name the actons expressed n a sentence and the enttes that are nvolved n the acton. In the example, the nomnal t1:acton s used to name the take acton expressed n the sentence, whle the two nomnals x1:anmate-beng and c1:thng name the actor that should execute the requested acton and the cube that should be taken, respectvely. The use of nomnals to dentfy actons and enttes s very useful for reference resoluton, as we wll see n the next secton, whch was one reason to use the CCG formalsm for our approach. In the logcal formula we can also see the damond operators mood, Actor, Patent, det, and num. These operators represent the syntactc propertes of the parsed sentence, ncludng such nformaton as that the sentence was uttered n mperatve mood or that a proxmal demonstratve was used as determner to further specfy a certan cube. For a more detaled descrpton of the use of CCG, OpenCCG, and hybrd logc, please refer to [17]. Our grammar contans the vocabulary that s necessary for the doman n whch our robot s workng. Ths vocabulary conssts of verbs that express the robot actons, nouns that descrbe the enttes (.e. humans, robots, and objects) n the envronment of the robot, as well as adjectves that descrbe propertes of the enttes (e.g. colours and shapes). As we have already started to explan n the prevous secton, we added so-called empty words to the grammar. These words stand for actons, enttes and propertes that have not been recognsed correctly. For example, the logcal form n Fgure 7 expresses the fact that the user requested the robot to take an object that s not further specfed at the moment. Ths s sgnalled by the keyword t1:acton (take-verb mood mp Actor x1 : anmate beng Patent ( e1 : thng emptynoun-np det ndef num sg )) Fg. 7 Logcal form wth an empty object that has not been recognsed correctly. Wth the addton of empty words, the robot s able to parse utterances by the human even f the sentence was not recognsed correctly. However, the robot then needs strateges to resolve the actons and enttes n the logcal form as we wll see n the next secton. 3.4 Reference Resoluton As t was shown n the former secton, the parsng step generates a logcal form of the spoken utterances that contans the structure of the sentence as well as markers (nomnals) for the actons and enttes that are contaned n the sentence. Therefore, the logcal form s perfectly suted to map all the enttes the human s talkng about to objects and persons n the world. Ths process s called reference resoluton or bndng. In our approach, reference resoluton follows the followng steps: (1) snce the structure of the sentence s known, we can determne from t the knd of drectve sentence that was uttered. Accordng to Qurk [34] (pp. 827) n the Englsh language there are seven ways to utter drectve sentences (or commands) that are relevant to our approach. (2) When the sentence structure has been determned, the enttes of the sentence can be extracted, ths ncludes n our approach only physcal objects wth ther propertes snce our robot s only workng wth one human and has not to know about any other potental co-workers. (3) In the next step, the logcal form has to be analysed f t contans any dectc expressons that are cues for the robot that t has to check for pontng gestures by the human as well. Dectc expressons are for example demonstratve determners lke ths n the expresson take ths yellow cube. (4) Now that all objects and gestures that need to be resolved are known, the object nventory, whch was ntroduced n Secton 2, can be used to map objects and gestures respectvely. For that the object nventory has nterfaces over whch other modules can make requests f the robot can see objects or gestures wth certan propertes n ts envronment. Fnally, reference resoluton generates a hypothess about the spoken utterance that contans the logcal form of the utterance together wth the referenced objects. Every hypothess has one of four types: Resolved. When a hypothess s resolved then the system was able to fnd a mappng for each object the human was talkng about. Unresolved. If a hypothess s unresolved, then the system was not able to fnd a mappng for each of the objects. In ths case, the system has to ask for clarfcaton, as we wll dscuss n the next Secton. Ambguous. When the hypothess s ambguous, one or more of the found mappngs between talked about objects and physcal objects s not clear. Ths means that there are several objects that match the object propertes of the logcal form. In that case the robot can ether choose one of the objects randomly or also ask for clarfcaton.

8 8 Conflctng. The system generates a conflctng hypothess when the human s talkng about an object but pontng to another object that has not the same propertes as the object he/she s talkng about. 3.5 Output Generaton The applcaton of CCGs n HRI has another advantage: as t was already mentoned n Secton 3.3, OpenCCG can produce a natural language expresson from a gven hybrd logc formula, a process that s called output generaton or realsaton. Therefore, the logc formula that was produced n the parsng and reference resoluton steps can be transformed to generate an output expresson by the robot to the human. In the easest case, when everythng was understood correctly, ths can be used to confrm that the order by the human was understood correctly. Snce the robot knows the objects t s talkng about after reference resoluton, t can also refer to these objects by usng gestures, for example t could pont to the objects or look at them f t s equpped wth an anmatronc head. Ths behavour has postve effects on the acceptance of the robot by the human, whch was reported n [13]. Output generaton can also be used to ask for specfc parts of an utterance that have not been understood correctly durng the frst three processng steps. At ths pont, the empty words we ntroduced n the grammar desgnate the not understood parts of the logcal form and can be fltered out easly. Ths clarfcaton process was recently dscussed by Krujff et al. [25], who specfy four forms of clarfcaton requests: attenton when the communcaton channel between the two conversaton agents has to be clarfed, dentfcaton when the utterance was not understood on an acoustc level, recognton whch refers to clarfcaton n the analyss of the utterance, and consderaton that s used to analyse the meanng of an utterance. Wth our approach we can solve clarfcaton requests from the recognton level, whch means that the robot can ask for clarfcaton when t has a lexcal problem ( what s a slat? ), a grammatcal problem ( should I move the cube tself or move myself to ts poston? ), or referental problems ( whch object should I take?, what should I do wth the green cube? ). 3.6 Dscusson After ths overvew of our approach for speech processng n HRI we wll dscuss the qualfcaton of ths approach to ncrease the safety for the human. We wll show how the safety desgn prncples for robot archtectures we stated n Secton 2 are appled for speech processng. Addtonally, we show how our approach could be mproved to ncrease the safety level for the human even further. Robustness. In speech processng for HRI, robustness s weak because most of the tme methods from computatonal lngustcs cannot be appled, snce unlke wrtten language, natural language s often ungrammatcal and ncomplete. Also, speech recognton s stll unrelable and needs a lot of tunng although there have been contnuous mprovements n the feld. In our approach, we react to these facts wth the preprocessng step before we parse the nput by the human. Ths allows us to use methods from computatonal lngustcs afterwards (parsng wth CCG) that have proven to work robustly and fast. Addtonally, we added the empty words to the grammar to cope wth the uncertanty that comes wth usng nput from speech recognton. At the moment, we react to the underspecfcaton that s expressed by the empty words wth clarfcaton requests by the robot. Ths ensures that the robot does not execute actons the user dd not asked t to do and so ncreases the safety of the human. However, an mprovement to ths approach would be the addton of a learnng component to gve the robot the opportunty to learn new words from the human. Currently we are workng on a more sophstcated method to compare the nput sentence to the sentences that can be parsed by the grammar. Ths method should also consder the semantcs of the sentences as well as ther valdty n the current context. The approach that was ntroduced by L et al. [29], whch uses semantc nets and corpus statstcs for comparson of short sentences, could be a bass for ths. Fast reacton tme. The usage of CCG n our approach ensures that the nput s processable at all tmes and processng tmes are much faster n comparson to other grammar formalsms. However, speech processng n general s far away from reachng real tme processng, whch s mostly due to the fact that most methods for speech recognton and parsng are based on the assumpton that ther nput are whole sentences. To mprove on ths problem, ncremental speech processng how t s proposed by [26] and [5] could be an answer. Context awareness. The usage of grammars for speech processng and output generaton mplctly provdes context nformaton for the robot by defnng the vocabulary of the applcaton. Of course, one can argue that ths way the vocabulary of the robot s constraned and some nformaton of the orgnal sentence the human sad mght get lost because t contans words that are not n the sentence lst. However, we thnk that a

9 9 robot whch s programmed for a specfc task should also be equpped wth a vocabulary for ths task. For example a robot that bulds cars does not need a vocabulary for hotel reservatons. A possble mprovement to ncrease the robot s vocabulary could be the use of WordNet [9], whch s a lexcal database that stores Englsh words that are stored nto sets of synonyms. WordNet could be used to automatcally retreve synonyms for the words n the grammar to ncrease the robot s vocabulary. In ths secton, we showed how the desgn prncples for safety n HRI nfluences speech processng; n the next secton, we swtch to another channel of nput processng and show how we apply the desgn prncples to vson processng. 4 Vson Processng Every robot that should safely nteract wth a human needs to be able to nterpret the natural envronment of the human n a fast and relable way. Therefore, we present a multthreaded vson subarchtecture that shows a non-blockng and wat-free behavour. Wth ths archtecture we reach an almost lnear speed up n performance, whch enables the robot to recognse objects as well as human gestures n realtme. Specfcally, we postulate the followng requrements that have to be met by the vson subarchtecture: Object and gesture recognton. The vson system needs to be able to recognse objects, object assembles, gestures of the human collaborator (e.g. pontng gestures), and parts of the robot. Parallel processng. The system has to be capable of explotng a multthreaded envronment, ndependent from actual hardware based degree of parallelsaton. Subarchtecture communcaton. The communcaton of the sngle parts of the vson subarchtecture needs to be effcent and fast. Publcaton of recognton results at dfferent processng stages. Prelmnary recognton results from early processng stages need to be publshed n realtme; fnal recognton results have to be publshed as fast as possble, but not necessarly n realtme The desgn of our vson subarchtecture s ndependent from hardware assumptons or operatng system specfcatons. In order to acheve optmal performance, we have to take the requrements of multcore computers nto account. In the followng sectons, we wll frst gve an overvew of the vson subarchtecture n Secton 4.1. After that, we explan the sngle steps of our vson processng approach, whch are Examne vson processng steps and data. We determne a functonal decomposton of the overall recognton process. Optons for herarchcal organsaton of processng modules are dscussed as well. In addton, we model occurrng data n the dentfed stages as tems or parttons n order to allow smultaneous analyss. Ths step s dscussed n detal n Secton 4.2 and Secton 4.3. Organse parallel processng. In ths step we decde how to apply the asynchronous concept of communcaton, how to mplement a sutable data structure, and we desgn a task schedulng algorthm. Secton 4.4 provdes more detaled nformaton on ths. Data mantenance. Fnally, we show a data management layer for storage and mantenance of data. Ths layer has to provde threadsafe access to data tems, therefore necessary operatons for processng modules must be desgned. Here, synchronsaton and consstency ssues as well as error management must be taken nto account, whch s explaned n detal n Secton 4.5. To conclude the secton about vson processng, we dscuss n Secton 4.6 how the desgn prncples for safety n HRI nfluenced our vson processng approach. 4.1 Vson Subarchtecture Overvew In order to reach the goal of realtme vsual processng we need to speed up the nter-module communcaton and smultaneous data access n our vson subarchtecture. We propose a multthreaded vson system based on a hgh level abstracton from hardware, operatng system, and low level vson tasks ncludng morphologcal operatons. Ths allows us to mnmse the overhead for communcaton tasks, as the amount of data needed to be transferred decreases n an abstract representaton wth a magntude of a few hundred. Fgure 8 shows an overvew of the computer vson (CV) subarchtecture. On the left, the processng layer ncludng ts decomposton n the functonal doman (see Secton 4.2) and data doman (see Secton 4.3) s depcted. The rght column n the fgure shows the data management layer (detals follow n Secton 4.5), whch handles nternal nformaton representaton and flow of the CV subsystem accordng to the second desgn prncple of CoSy ( structured management of knowledge ). The vson system apples an asynchronous communcaton mechansm (ACM), whch leads to a nonblockng behavour and guarantees the requred fre-

10 10 Table 1 Functonal decomposton of tasks for vson processng. Task Input Output Preprocessng Image Loadng Image Segmentaton / Image Regons of Attenton Early Processng Analyss and Interpretaton Object Recogntoplates Regon and tem- Objects Gesture Recognton Regon Gesture Robot Detecton Regon Robotpart Postprocessng Result Publshng Objects, gestures and robotparts Result Vsualsaton Image, regons, objects, gestures and robotparts Fg. 8 Vson processng subarchtecture. quency for result publshng, as publshng ncomplete, prelmnary analyss results s tolerated (detals n Secton 4.4). Derved from common standards [11], ntermedate vson data s managed n lmted-sze prortyqueues. 4.2 Processng Modules Accordng to the frst desgn prncple of the CoSy project ( concurrent modular processng ), the vson subarchtecture s organsed as a combnaton of functonal enttes, where each module can operate ndependently of the others. In ths parallel subarchtecture, we have to desgnate anchor ponts for dstrbuted computaton the processng modules. Here, the level of abstracton consderng computatonal tasks matters n terms of parallelsaton. In order to avod unnecessary overhead regardng communcaton and take full advantage of the multcore envronment, we model concurrent computaton wth a moderate level of abstracton. For functon doman parallelsaton, we clam that the dvson nto well-defned functonal submodules s feasble. In the processng layer of the proposed CV system ths s obvously the case, as we can dentfy three major functonal stages: preprocessng, analyss and nterpretaton, and postprocessng. Table 1 shows the modulewse decomposton of the CV subarchtecture and lsts the data tems that are used n the decomposed tasks. As the recognton process s decomposable n the functon doman (see Task column), now, n order to prove our clam, we must acheve data doman parallelsaton. Thus, we have to specfy the functonal tasks accordng to the need of multple nstantaton of the processng modules. 4.3 Multple Instantaton of Processng Modules Consder that the data management layer successfully stored extracted regons of attenton that were extracted by the early processng modules. Analyss operatons from the second stage are desgned to only alter one specfc regon (or data tem) at a tme. Therefore, we can smultaneously serve requests for a regon from several concurrently actve analyss module nstances. Also, the system does not need to wat for the result of an analyss because results are publshed as soon as they are provded by the data management. We publsh perodcally n realtme, thus t s of mnor relevance to publsh a sngle result at the exact moment of ts completed analyss. Generally, we derve the followng approach from the non-blockng paradgm of ACMs: we publsh ncomplete analyss results of a scene rather than watng for a complete analyss that would block the system n the meantme. Ths approach allows us to run multple concurrent module nstances for the analyss of data tems as long as the data management s mplemented n a threadsafe way, whch wll be explaned n more detal n Secton 4.5. Thus, we are able to mplement data doman parallelsaton n a smlar way as t was reported n [6]. 4.4 Asynchronous Communcaton and Schedulng Orgnatng from functonal and data doman decomposton, we have to consder communcaton strateges for a system runnng processng modules n parallel. In parallel computaton we can generally apply ether synchronous or asynchronous communcaton strateges for data exchange between processes or threads [7]. Wth synchronous communcaton the partners wat for con-

11 11 frmaton of sent data tems, so ths strategy has ts man applcatons where the correct transmsson of data s essental. Though beng robust, due to ts blockng nature a synchronous approach can cause problems especally for realtme systems where mmedate responses have to be guaranteed. For ths reason, asynchronous non-blockng communcaton mechansms (ACMs) have been proposed. Wth ACMs nformaton or data s dropped when capactes exceed whch s acceptable as long as the system does not block. Non-blockng algorthms can be dstngushed as beng lock-free and wat-free [41]. Lockfree mplementatons guarantee at least one process to contnue at any tme though starvaton s a rsk, because an operaton may never fnsh due to the progress of other operatons. On the other hand, wat-free mplementatons exst that avod starvaton as they guarantee completon of a task n a lmted number of steps [23]. Generally, t s essental for systems utlsng an ACM to stay responsve, not to guarantee data transmsson. Accordng to Smpson [38], ACMs can be classfed based on the destructveness of data accesses. Table 2 shows protocols wth names gven by Smpson for dfferent levels of destructon (N-DR stands for nondestructve readng, etc.) and was taken from [37]. Another classfcaton of ACM protocols by Yakovlev [44] dstngushes data access wth respect to ther overwrtng and re-readng permsson. Table 2 Common classfcaton for asynchronous communcaton mechansms (ACM). DR N-DR DW Sgnal Pool N-DW Channel Constant Concernng ACMs for the proposed CV system the mplementaton of a pool ACM n ether classfcaton scheme s relevant. On the one hand, we take nto account the Smpson classfcaton as we do have nondestructve read operatons but wrte operatons nclude deleton of tems. On the other hand, we use Yakovlev s classfcaton, as we allow overwrtng n a wrte operaton and do not delete tems when readng them from the storage. There s one dsadvantage n such an mplementaton of the ACM: we rsk that a module requests data from the data management, but there s no such data tem at the moment, for example because the specfc queue s empty. In ths case the data management delvers a null data tem, so modules have to deal wth these tems as well. Therefore, we propose an algorthm that whenever a Null data tem s receved tres to suspend module nstances (.e. sngle threads) for an optmal amount of tme untl a correct data tem s expected to be delvered agan. An ncremental back off tme b(c) may be calculated for the nstance of a module as follows: b(c) = mn(c, ( a j )) (1) n In (1) the parameter c denotes the counter for the number of tres snce the last correct data tem has been receved by the module, denotes the predefned back off ncrement n mllseconds, a s the maxmum age of a data tem untl t s deleted, j the number of module nstances operatng on the same task and n the current number of tems n the queue. If a Null data tem s retreved, c s ncremented and the module s mmedately suspended for a tme b(c) agan. In case a correct tem could be delvered, c s reset to 0 and the tem s processed. The back off strategy tres to optmally calculate suspenson perods for nstances not needed at the moment but at the same tme to provde an nstance whenever needed. In detal, the frst argument of mn calculates an ncremental amount of tme for the module nstance to sleep and the second argument represents the expected mean tme untl the next correct data tem can be delvered. Ths value s then used as the maxmum amount of tme to suspend a module nstance. The optmal back off strategy ensures that suspendng a sngle module never blocks the entre vson system, as (1) calculatng a maxmum for the suspenson tme guarantees wat-free behavour (t states an upper lmt); and (2) the vson system guarantees, that new camera mages are contnuously acqured, and thus new data s constantly fed nto the system for processng. 4.5 Data Management A crucal pont s the mplementaton of an adequate data access strategy for concurrent requests. The strategy has to ensure ntegrty and consstency of data and needs to provde error management polces as well. One also has to consder prorsaton whenever a module requests to wrte whle another module smultaneously wants to read data from or wrte data to the storage. Another mportant pont s the deleton of data tems when they expre. Consderng modularty, we organse data access n a data management layer. A natural approach for the mplementaton s based on the sngleton desgn pattern [16]. Sngleton mplementatons only provde a sngle

12 12 nstance of an object to the overall system; so n our case any request from an analyss module must call the sngle nstance of the data management. Error handlng of the data management layer can be mplemented straght forward as the layer delvers Null data tems whenever an erroneous request was receved, a queue was empty or no sutable data tem could be found. The error handlng approach utlsng Null data tems s even wat-free because t completes n a lmted number of steps [41]. Organsng the sngleton nstance n a threadsafe manner concernng read and wrte accesses ensures ntegrty and consstency. In order to acheve ths, the data management module s organsed as a bundle of queues Data Operatons Threadsafe concurrent data access s realsed by encapsulatng synchronsaton. An nstance of a processng module sends a request for storage or retreval of a data tem of a certan knd by callng one of the three operatons provded to the processng layer. In detal, these operatons are defned as follows: A Wrte operaton requests storage of the passed data tem from the data management layer. In pseudo code we can defne ths access operaton as follows: wrte<queue>(item):vod The Read operaton requests delvery of a data tem from the data management. A Read request enables the module from the processng layer to specfy the type of data to be delvered but the concrete tem s stochastcally determned. The pseudo code notaton for ths operaton s: read<queue>():item The ReadAll operaton requests delvery of all data tems from a specfc queue. Ths can be specfed as: readall<queue>():item[] Randomsed Weghted Data Retreval Consderng a request from the processng layer, accordng to what strategy should the data management select an tem from the specfed queue? The retreval strategy mplemented n data management selects a data tem to delver accordng to the evaluaton of a stochastcal functon. The functon s based on the assumpton that a data tem (re-)detected n the near past must be prortsed to one that last occurred many cycles ago as t may have already dsappeared or removed. Snce each tem n a queue Q has a tmestamp, we wegh the tems Q accordng to ther age a = now tmestamp() such that the weght ncreases, the younger tems are: Q : w = 1 a maxage (2) A new queue of ponters to data tems from the orgnal queue s bult afterwards. The new queue on whch the actual retreval operaton s performed wth a random selecton s flled wth at least one ponter to each data tem. In fact, accordng to the weght w of an tem, a number of duplcates d of each ponter s pushed to the queue: Q : d = 1 argmn j Q (w j ) w (3) Afterwards the random selecton on the ponter queue s performed where more recent tems are prortsed automatcally as more ponters to the correspondng data tems exst. Before applyng the weght to the tems of a queue, we have to exclude elements that match certan precondtons. As an tem cannot be altered by two processng modules concurrently, we ntroduce a lockng mechansm for tems. The non-blockng nature of data access can though stll be guaranteed due to the error handlng approach descrbed earler. Before a data tem s delvered to the processng layer, the state of the tem s changed to locked. Locked tems are not allowed to be delvered to any other nstance and so are excluded from the weghtng step. Releasng the lock s n responsblty of the module processng the tem Queue Lockng Another mportant queston to dscuss s the behavour of the system n case of concurrent Wrte or Read operatons concernng a specfc queue. Concurrent Read operatons are allowed at any tme, but n case a Wrte operaton s requested all retreval requests and concurrent Wrte requests must be blocked meanwhle. Therefore, the system has to mplement a mechansm that uses cascaded mutual exclusons. Thus, each data queue storng a specfc knd of data tem s protected by a ReadWrte mutex, whch exactly matches the above requrements. Though a sngle operaton may be blocked, the overall system s not. If a mutex can not be acqured at the moment, n case of a Read operaton a null tem s delvered and n case of a Wrte operaton no operaton s executed. Ths behavour s agan conform to the defnton of an asynchronous non-blockng algorthm as t s wat-free.

13 Storage Cleanup Due to system lmtatons n a realtme envronment we only allow the data management to store a confgurable amount of data from the past. Thus, data tems must be marked wth a tmestamp to allow age determnaton. Consequently there s no Delete operaton provded to the processng layer, but tems are deleted automatcally durng a CleanUp step when they expre. The CleanUp operaton s nvoked perodcally by a separate data management thread. The thread s scheduled to wat for a certan predefned perod of tme before t nvokes the operaton. After performng the operaton on all data, the thread s suspended agan. Of course, as ths operaton s equvalent to a Wrte operaton, t must be protected by the same mutex mechansm. Smlar to the Wrte operaton, no operaton s performed n case the mutex for a certan queue can not be acqured Enhancements In order to enhance the performance of Read and ReadAll operatons we ntroduce the concept of nterpretaton preselecton. We assume that certan data tems are not relevant for dedcated tasks. For example a gesture recognton module could only be nterested n a regon that comes nto the scene from the bottom or an vsualsaton module s only dsplayng objects from wthn the last 100ms, but skppng gestures totally. In order to completely leave the relevance decson to the processng modules, we propose a mechansm that evaluates a predcate that s passed wthn a request. Accordng to the predcate, the excluson step before weghtng the tems of a queue s adapted. Now not only locked tems but also tems that do not match the predcate are removed from the set of tems from whch ponters are duplcated accordng to ther weght. In ths way, the search space for retreval can be restrcted, but the non-determnstc selecton algorthm can stll be appled. Havng defned the retreval method stubs n Secton 4.5.1, we now extend these defntons: read<queue>(predcate):item readall<queue>(predcate):item[] Predcate s a non-empty bnary predcate that evaluates to True or False on each data tem of the specfed queue. If the set of tems after preselecton s empty, a Null data tem s delvered. As some tems also support state attrbutes for trackng (tracked / new) or nterpretaton (statc / movng), a processng module can use these attrbutes for the mplementaton of ts own predcate. Moreover, a processng module can n prncple specfy arbtrary predcates on members of the data tems n a queue. For example, consderng that data tems n a queue are tmestamped, t s possble to track them from one cycle to the followng. Therefore, we defne a comparson predcate that s appled when a storage Wrte query s requested. The predcate evaluates symbolc or meta attrbutes, ncludng classfcaton, colour, approxmate pose, number of ponts, or wdth, heght, and depth. When the data management receves a Wrte request for a formerly recognsed tem, only necessary attrbutes may be updated, all other attrbutes (especally the unque d) nstead can be kept. For example, consderng an tem statc and fully analysed, the exstng tem just gets all ordnary attrbutes (such as the tmestamp, poston, etc.) updated, but the updated tem s not marked for analyss agan. 4.6 Dscusson Followng ths overvew of our approach for vson processng, we dscuss how the desgn prncples for safety n HRI nfluence the method. Robustness. To ncrease the robustness of object and gesture recognton, we are usng multple modaltes n the recognton process: the use of colour, shape, and sze nformaton makes sure that a good recognton result can be guaranteed at all tmes, even n cases were one of the object propertes cannot be determned correctly. Fast reacton tme. For a decreased processng tme we took two actons: frst, we make use of the multcore archtectures of current computers to parallelse our approach for object and gesture recognton. Because parallelsaton scales very well consderng the proposed approach, wth ts ntroducton, the processng tmes for analyss of a complete frame could approxmately be cut nto 1/n, where n = 4 on the quad-core computer utlsed [31]. Second, we are usng trackng to follow already recognsed objects and gestures. If the sze and poston of an object only changes slghtly, we assume that t stll s the same object and thus do not ntate the whole recognton process for ths object. Consderng ths, we are able to ncrease processng tmes agan, as the actual object recognton s the most tme-consumng task n the vson system. As a future step, we plan to realse a mechansm called attenton-based early processng, whch has been ntroduced n [30]. In ths method, object recognton publshes ts prelmnary recognton results at an early processng stage. Ths way, the robot already gets nformaton about the rough poston and sze of an object

14 14 but does not know the exact type and propertes of the object yet. Ths can for example be used to avod collson wth movng objects. Context awareness. For object recognton, context s mostly mportant to further ncrease the robustness and processng tme of the approach. In our case we are usng the nformaton about the poston of the robot s arm to determne whether t s movng wthn the feld of vew of one of the used cameras. In that case, the vson system may nterrupt the recognton process whch decreases squanderng of computng power and avods false recognton results. Now that we have hghlghted the nput sde of HRI and showed our work on speech and vson processng, we want to swtch to the output sde and explan how the desgn prncples for safety nfluence our approach for robot control. 5 Robot Control The consderaton of the safety desgn prncples s of utmost mportance controllng a collaboratve robot. Especally workng n a shared workspace mght sooner or later lead to stuatons where human and robot collde wth each other. That ths can be very harmful to the human especally when standard ndustral robots are used, was shown n [19,20,18]. In ths secton, we dscuss how the safety desgn prncples nfluence the controller module of a robot. We are manly concerned to speed up the reacton tmes of the robot and to ncrease the context awareness of the system. For that, on the one hand we are splttng up the robot actons n several tasks that can be stacked up n a task herarchy. Ths speeds up the onlne calculaton of the robot movements and allows the robot to be reconfgured for varous applcatons easly. On the other hand, we equp the robot wth an nternal representaton of ts envronment n whch the sensor data from vson processng, person trackng, and hand trackng s updated contnuously n an asynchronous fashon, so that the robot always knows about ts surroundng. In the remander of ths secton we show n Secton 5.1 how sngle robot tasks, ncludng posture, operatonal poston, and collson avodance, can be arranged n a task herarchy so that the safety for the human s ncreased wthout constranng the robot s capabltes. Secton 5.2 gves an overvew of the nternal robot representaton, and Secton 5.3 presents how our approach for robot control can be appled n an applcaton for HRI n an ndustral settng. Secton 5.4 concludes the secton about robot control and dscusses how the safety desgn prncples nfluence our approach. 5.1 Robot Task Herarchy Accordng to [36], actons consst of several atomc tasks that are arranged n a task-orented way. That means on the reverse that varous dfferent actons can be generated by rearrangng the same set of atomc tasks. However, an encodng of prortes n the task herarchy s of mportance, because t needs to be prevented that contrary tasks nterfere wth each other and lead to uncontrollable or unwanted behavour of the robot, whch mght lead to stuatons n whch the safety of a human s n danger. Based on the syntax of [45], an acton A can be formulated as a composton of tasks T k wth a projecton rule k that ensures the behavour of the task: A = T n 0 = T n n T n 1 n T 0. (4) As most ndustral robots are only controllable on the poston or velocty level, we transform the task descrptons n terms of jont veloctes ( q): q Acton = q Tn n q Tn 1 n q T0. (5) Accordng to the defned control structure, we need to take care of only a few ponts for each task: compute the velocty to solve the sngle task, set the constrants accordng to the current stuaton or use statc constrants, and fnally transform the lower prorty task velocty nto a safe subspace respectng the actve constrants. For the latter we are usng null spaces wth orthogonal projectors as n [36,35] n the posture and the operatonal poston task and a constrant least square optmsaton for the collson avodance task. Wth these projectors we are able to decouple the tasks from each other whch s also shown n Fgure 9. velocty task 1 velocty task 2... velocty task N constrant velocty... constrant velocty constrants... constrants Task 1 Task 2... Task N prorty level Output Fg. 9 Actons are defned through task compostons. The assocated constrants are respected through projectons nto correspondng subspaces, wth executon prortes rangng from lowest (left) to hghest (rght) Respectng Hardware Lmts An mportant ssue that needs to be respected are the lmtatons of jont angles, veloctes, and acceleratons. The resultng velocty q as well as ntermedate results needs to be n a certan boundng regon l q u, that

15 15 does not volate these lmts. Therefore, we adaptvely recompute the boundng lmts n every tme step accordng to l = max( q mn, q lowerjl, q AL ) (6) for the lower boundary and u = mn( q max, q upperjl, q DL ) (7) for the upper boundary. The values for q max and q mn are the maxmum and mnmum velocty for jont as defned by the manufacturer of the robot. q lowerjl = qmn q (t) t (8) s the velocty that s needed to reach the lower jont lmt of jont n the next tme step t + 1 and q upperjl = qmax q (t) t (9) s the velocty to reach the upper jont lmts respectvely. These veloctes converge to zero as the jont angle s approachng the jont lmt. To respect that the acceleraton and the deceleraton abltes of the motors of the robot are also lmted, the maxmum veloctes of the jonts need to be constrant. Therefore, we approxmate ths factor wth q AL = q acc t + q (t 1) (10) for the acceleraton and q DL = q dec t + q (t 1) (11) for the deceleraton of the jonts. Wth the lmt nformaton for each jont, a lmt method C( q) can be defned that scales the velocty vector to respect all above mentoned lmts wthout changng the trajectory of the moton: C( q) = s(l, u) q. (12) In the followng sectons, we explan the tasks posture, operatonal poston, and collson avodance. These tasks can be stacked n an arbtrary herarchy to perform varous actons Task: Jont Poston (Posture) The goal of the posture task s to set the robot to a certan jont confguraton q goal. Wth ths posture controller the robot can be set to specfc postures, for example to a human frendly or uprght posture. The velocty for ths task s calculated by ( ) qgoal q(t) q po = C. (13) t To constran certan degrees of freedom n jont space that cannot be nfluenced by lower prorty tasks a n n matrx S po s used to select them, wth n beng the number of jonts. Ths means the guaranteed velocty can be expressed as q po = S q po. (14) Usng the projector N P o of the selected constrants, we get as output for the projecton of an nput velocty q n : q = q po + N po q n (15) wth N po = I S po. (16) Task: Operatonal Poston The operatonal poston task drves the tool centre pont of the robot to a defned goal poston and orentaton x goal n Cartesan coordnates accordng to: ẋ = x goal x(t). (17) t After calculatng the velocty n Cartesan space, constrants can be set usng a dagonal selecton 6 6 matrx S Op to select the degrees of freedom (n Cartesan space) that should not be nfluenced by lower prorty tasks. Transformed nto lmted jont veloctes usng a sngularty robust pseudo-nverse J e of the Jacoban J e of the end effector, we get q op = C(J e S op ẋ op ). (18) Usng the null space N op of the selected constrants, we get as output for the orthogonal projecton of an nput velocty q n : q = q op + N op q n (19) wth N op = I J e S op J e. (20)

16 Task: Collson Avodance Wth ths task, the robot can be controlled so that t avods collson wth statc enttes (for example a workbench) and dynamc enttes (for example the human or movng objects) n ts envronment. In ths task, the avodance s done n a reactve way wth dynamcally updated collson scenes that s nterfaceable wth a varety of sensors. The vson system we presented n Secton 4 also connects to the robot controller va ths nterface. The man challenge that arses here, s that the planned moton and the avodance moton must be handled n a way such that they do not nterfere wth each other. Therefore, we fuse potental feld methodology to repel the robot from the obstacle wth a constrant least square optmsaton that restrcts the moton of the robot to safe orthogonal subspaces of the collson avodance. Vrtual Forces To compute the velocty that repels the robot from surroundng obstacles, we need to compute the mnmum dstances of all objects n the envronment model (ncludng self collson) to all body parts of the robot. Fgure 10 shows the body parts of the robot we are usng n our examples n dfferent colours along wth an example of the mnmum dstances d (red lnes) from an obstacle to a gven jont confguraton. Opposte to smplfed and only approxmated models of manpulators (whch was for example used n the skeleton algorthm presented n [8]), we measure the dstances of arbtrary shapes to a convex verson of the real CAD model of the robot n order to reach a hgh precson of the vrtual forces. Wth an effcent mplementaton of the GJK algorthm [42], we compute these dstances faster than the update rate of the robot controller. After calculatng the mnmum dstance vectors v x, n Cartesan space (.e. the drecton of the appled vrtual force), we need to transform them to veloctes n jont space and fnd the overall moton of the robot to avod the collson. Ths s done accordng to q = I q = I JP T U r() rep,(q) v x, (q), (21) wth I beng the number of bodes of the robot, the current jont confguraton of the robot q, the Jacoban of the mnmum dstance pont on the robot J Pr(), and the repellng potental functon U rep, : { ( ) 2 1 U rep, (q) = 2 η 1 d 1 (q) Q f d (q) Q, 0 f d (q) > Q wth Q beng the dstance at whch the potental feld functon s appled, whch s represented by the transparent bubble n Fgure 10. Constrant Least Square Mnmsaton To ensure that we project the lower prorty task n an orthogonal subspace of the collson avodance task, we use the mathematcal framework of quadratc programmng [10] to mnmse the quadratc error between optmal velocty of the lower prorty task subject and the constrants of the hgher prorty task. The low prorty task executon ( q n ) s optmsed regardng must have constrants of the hgher prorty task. The projecton s descrbed accordng to: mn q n J e q n ẋ t 2, (22) where ẋ t s the deal lnear and angular velocty to solve the lower prorty task subject to the lnear constrants of the form C T q t 0 (23) Fg. 10 Computng the repellng forces of an obstacle: the red lnes llustrate the mnmum dstances of an obstacle to the body parts of the robot n a gven jont confguraton. If a dstance s below a chosen securty threshold (transparent bubble), the dstance s used to compute vrtual forces on the robot usng potental felds. wth the constrant matrx q T C T = 1. (24) q T I

17 17 and q calculated accordng to (21). Ths means, only those veloctes are vald, whch are orthogonal to the drecton of the collson avodance veloctes or pont n a drecton that leaves the defned safety regon. The output of the mnmsaton process s then equal to the constraned jont velocty q. Quadratc Programmng n combnaton wth collson avodance on the level of jont acceleraton was used n [15, 14]. 5.2 Internal World Representaton The robot controller has an nternal representaton of ts surroundng envronment (nternal world representaton, IWR) that s kept up to date wth changes n the world that are of nterest for the robot. Snce the robot n the shown collaboraton scenaro s statonary, many thngs n the envronment can be added to the IWR as statc components. Ths ncludes the worktable, the cage, the assembly belt n the background, as well as the robot s poston n relaton to the table. Multple sensor modules perceve dynamc objects ncludng the human and add, update or remove them at runtme. Object propertes such as the colour or the locaton of the object, as well as the robot s posture, whch s also constantly updated when the robot moves are also part of the IWR. the nput processng modules have to organse the IDs they are usng by themselves, whch also means that they have to take care that they remove enttes from the IWR when they do not track them anymore. Fgure 5.2 shows the nterfaces for handlng enttes n the IWR. addsngleobject(groupid, bodyid, shape, pose); updatesngleobject(bodyid, pose); removesngleobject(bodyid); addgroupobject(groupid, object, pose); updategroupobject(groupid, pose); removegroupobject(groupid); Fg. 12 Interfaces for the nternal robot representaton; nput processng modules use these nterfaces to add, remove, or update enttes n the representaton. Robot control uses the IWR on the one hand to keep track of the current status of the envronment. On the other hand, t also forwards the nformaton from the nput processng modules to a vrtual scene, n whch the controller also presents nformaton about the robot s appearance and poston. For ths, the vrtual scene frst loads nformaton about the statc envronment of the robot and then gets the addtonal nformaton about dynamc enttes n the world from the controller. Ths vrtual scene can be dsplayed to the human users so that they also get a quck overvew of the robot s belef about the current state of the world. Fgures 13(b) to 13(d) show an example for a 3D representaton of a vrtual scene. 5.3 Applcaton Example: Moble Storage Box Fg. 11 Schema for the nternal world representaton component. Fgure 11 shows a schematc overvew of ths representaton. The ID generator s a central component of the IWR. Input processng modules, for example an object recognton or a person trackng module, request globally unque IDs from the generator. Wth these IDs the nput processors can ntroduce, update, and remove new enttes (.e. persons or objects) n the IWR. Here, Wth the task-based herarchcal control structure we can solve a wde range of tasks that can be used n producton scenaros n whch human and robot work together as a team. In the followng secton, we present an applcaton of our robot control approach that was used on the demonstraton platform JAHIR (Jont-Acton for Humans and Industral Robots) [27], whch s embedded n a factory settng [46]. The JAHIR platform conssts of an ndustral robot arm that s mounted behnd a workbench. A human who stands face to face to the robot shares the workspace wth the robot durng collaboratve tasks. Fgure 13(a) shows the system nteractng wth a human. Durng nteracton, the human wears a jacket on whch a set of nfrared markers are attached on the palm and on the lower and upper arm. An nfrared trackng system tracks these markers and sends the sensor data to the nternal representaton of the robot. Ths way, the robot constantly updates ts knowledge about the poston of

18 18 the human s arm and hand. Fgures 13(b) to 13(d) show a 3D model of the nternal robot representaton n varous nteracton stuatons. In the representaton, the human s body s roughly estmated by several cylnders whch ensures that the human s safety zone s bg enough. Our applcaton example s set n a manual producton settng. In manual producton human workers need to have all parts they need for an assembly wthn reach so that they can buld t effcently. At the same tme, the workers need a place were they can put already assembled subcomponents of an assembly, from whch they can retreve the subcomponent n a later producton stage. Thus, we programmed the JAHIR robot to assst the human as a moble storage box. The robot s able to supply assembly parts as well as to take already fnshed subcomponents of the assembly and to keep them n range of the human. For ths, the robot holds a storage box and keeps t near the human s hand. To do ths n a safe way, the robot has to brng n lne the tasks to follow the human s hand and to avod collson wth the human and wth ts envronment at the same tme. Gven these requrements, we confgure the robot controllers as presented n Equaton (25) to compose acton A 1. A 1 = T orentaton T avodance T poston T posture. (25) T orentaton s the task wth the hghest prorty, whch takes care of keepng the box always n a horzontal orentaton. The operatonal poston controller s used here wth the selecton matrx S orentaton = dag (0, 0, 0, 1, 1, 1) (26) to fx the orentaton of the box. Task T avodance avods collsons wth the surroundng envronment and the human hand. The poston task T poston follows the human hand through updates of the hand trackng system to the goal poston x goal, that should be 0.1 m n front and below of the hand. To keep the poston fxed, the selecton matrx S poston = dag (1, 1, 1, 0, 0, 0) (27) s used. In the posture task, we defned that the robot should have an uprght jont confguraton. Because the posture task has the lowest prorty, we can nclude all jonts n the velocty calculaton. The orentaton has the hghest prorty n ths example, because the spare parts, that are nsde the box, should not fall to the ground. Although ths seems to lmt the safety for the human at frst glance, each task n the controller can suspend the whole movement of the robot. Ths means, that f the human comes to close to the robot and the collson shapes touch, the robot stops untl the collson s cleared. The collson shapes have a bgger sze than the real objects, for example the hand s approxmated as a sphere. Another ssue that should be consdered s, that although the orentaton s fxed wth hghest prorty, the robot has three postonal degrees of freedom left to avod the collson. Fgure 13 shows some mpressons from the moble storage box applcaton. The robot s carryng a red box wth assembly subcomponents n ts grpper, so that the human can retreve parts of the box, whch s shown n Fgure 13(a). In order to allow the human to grasp a subcomponent from the box, the moton of the robot needs to be stopped. For ths the robot controller uses nformaton from the avodance task that measures the dstance between robot and human. The controller stops the current moton, f the dstance of robot and human falls below a defned threshold. 5.4 Dscusson To conclude the secton about robot control, we dscuss how the desgn prncples for safety n HRI nfluence our approach. Addtonally, we show how the approach can be even further mproved. Robustness. To ncrease robustness n robot control, we take three steps: frst, we compute the trajectores of the robot actuator onlne every seven mllseconds. Ths way, the robot can react quckly on new events sent by the nput processng modules n an asynchronous fashon. Second, the decomposton of robot tasks nto sngle actons decreases the complexty of the trajectory computaton and therefore ncreases robustness. Thrd, n the nternal world representaton we chose shapes to represent human body parts, whch are bgger than the body part n realty. Together wth the computed vrtual repellng forces for collson avodance, ths ensures that the robot stops ts movement long before an actual collson can occur. All tasks can suspend the moton of the robot, for example when a collson or an error n the computaton occurs. To further mprove our approach and to ncrease the stablty of the collson avodance, we plan to use more dstances than only the mnmum dstance to compute the vrtual forces. Addtonally, we want to add more atomc tasks to the controller, ncludng sngularty and jont lmt avodance, and we plan to track the whole human body pose at best wthout the use of any markers. Fast reacton tme. Fast and effcent computaton s very mportant for robot control. Therefore, n

19 19 (a) (b) (c) (d) Fg. 13 Moble storage box: The robot has a storage box as tool whch follows the human hand n safe dstance and avods collson wth the hand and the surroundng, so that the human can pck up parts or place assembled products n there. (a) real scene; (b), (c) and (d) 3D representaton wth robot behavour. our approach we compute robot trajectores faster than the robot update tme, even on a standard dual core personal computer. To mprove our approach n the future, we plan to parallelse the computaton by sourcng out computatonally ntensve parts of the approach e.g. computaton of collson avodance to other computers. Context awareness. We ncreased the context awareness n our approach by addng the nternal world representaton to robot control. Ths way, the robot always has an exact nformaton about the current status of ts envronment and can make sure that t does not collde wth a human or any objects. In the next step, we plan to ncrease the context awareness of the robot even further by addng full body trackng of the human to the nternal world represen- taton. Addtonally, we plan to equp the setup wth a depth sensor camera. Ths wll enable the robot to measure the sze of objects, even f they are unknown, so that t can avod collson. 6 Concluson In ths publcaton, we presented three desgn prncples for safety n human-robot nteracton, robustness, fast reacton tme, and context awareness, whch are essental for every software component that controls the behavour of a robot that nteracts wth a human. We showed how these desgn prncples can be appled to robot archtectures and how they nfluence approaches for speech processng, vson processng, and robot control. We also presented an example applcaton that n-