Safe and Efficient Robotic Space Exploration with Tele-Supervised Autonomous Robots

Transcription

1 Safe and Efficien Roboic Space Exploraion wih Tele-Supervised Auonomous Robos Albero Elfes*, John M. Dolan, Gregg Podnar, Sandra Mau, Marcel Bergerman *Je Propulsion Laboraory, 4800 Oak Grove Drive, Pasadena, CA, Roboics Insiue, Carnegie Mellon Universiy, 5000 Forbes Avenue, Pisburgh, PA, Absrac A successful plan for space exploraion requires he commissioning of flees of robos o prospec, mine, build, inspec and mainain srucures, and generally assis asronaus, rendering he overall mission as safe as reasonably achievable for human beings, he mos precious resource. The auhors are currenly developing, under he suppor of NASA, a Robo Supervision Archiecure (RSA) which will allow a small number of human operaors o safely and efficienly elesupervise a flee of auonomous robos. This represens a significan advance over he sae of he ar, where currenly one robo is overseen by a group of skilled professionals. In his paper we describe some aspecs of his work, including he archiecure iself for coordinaion of human and robo work, failure and coningency managemen, high-fideliy elepresence, and operaion under limied bandwidh. We also presen highlighs of our firs applicaion: wide area prospecing of minerals and waer in suppor of susained ouposs on he Moon and on Mars. Inroducion 1 NASA has iniiaed he implemenaion of is Vision for Space Exploraion by planning o reurn human beings o he Moon by 2018 and hen proceed o Mars by This bold, risky, and cosly enerprise will require ha all possible acions be aken o maximize he asronaus safey and efficiency. The auhors believe ha his can be faciliaed by flees of robos auonomously performing a wide variey of asks such as in-space inspecion, mainenance and assembly; regional surveys, mineral prospecing and mining; habia consrucion and in-siu resource uilizaion (ISRU); ec. These robos will be elesupervised by a small number of human ground conrollers and/or asronaus, who will be able o share conrol wih and eleoperae each individual robo whenever necessary, all from a safe, shirsleeve environmen. In his paper we presen he Robo Supervision Archiecure (RSA), a mulilayered archiecure for human elesupervision of a flee of mobile robos. This research is suppored by he Advanced Space Operaions Technology Program of NASA s Exploraion Sysems Mission 1 This work is suppored by NASA under Cooperaive Agreemen No. NNA05CP96A. Direcorae. Our objecive is o demonsrae ha he RSA enhances he elesupervisor s efficiency in a real-world mineral prospecing ask (see Figure 1) while allowing supervision of he robo flee from he relaive safey of a lander, orbier or ground saion. The archiecure is general enough o accommodae a wide range of applicaions and o span he enire specrum of so-called sliding auonomy levels, from pure eleoperaion o supervised conrol o fully auonomous [6]. Figure 1. Aris s concepion of elesupervised wide-area roboic prospecing on he Moon. Our Robo Supervision Archiecure addresses he following imporan challenges: 1. Coordinaion of human and robo work: The RSA is designed boh o minimize he need for humans o perform cosly and risky exra-vehicular aciviies (EVA), in which asronaus in space suis execue asks in orbi or on lunar/planeary surfaces; and o muliply heir effors beyond direc consan eleoperaion of one robo. I does so by allowing he robos o operae as auonomously as echnically possible, receiving assisance whenever necessary; and by allowing he human o assume parial or oal conrol of a robo or is ask-specific insrumenaion whenever appropriae. While we do no explicily address issues of ighly-coupled coordinaion beween collocaed humans and robos (i.e., join ask performance), he archiecure suppors augmening he robos in hese scenarios when auonomy is insufficien and coninued assisance is criical.

2 2. Enhanced human safey and efficiency: Allowing a human o elesupervise a flee of auonomous robos from he safey of a shirsleeve environmen is a pruden and efficien approach o fuure space exploraion. As our research progresses, we will es he efficiency gains over he baseline of a single human in a space sui performing EVA by asking a flee of robos o execue wide-area prospecing; and we will compue sysem performance merics ha ake ino accoun he area covered, prospecing accuracy, errain difficuly, human safey, number of roboic rovers, human effor defined as he degree of human ask inervenion in full-ime equivalens, and ask compleion ime. 3. Failure and coningency managemen: The archiecure explicily defines a Hazard and Assisance Deecion subsysem which operaes on muliple levels. A he robo level, i assesses deviaions from sandard or expeced operaing condiions, boh wih respec o he robo s healh and is assigned ask. A he worksaion level, he Hazard and Assisance Deecion subsysem is responsible for queuing and prioriizing all robo alers and requess, based on a crierion ha akes ino accoun he ype of hazard, esimaed ime for he elesupervisor o fix i, and he prediced ime o reach criical danger. 4. Remoe operaions wih bandwidh consrains: Our sysem relies on boh high-bandwidh radio links for geomerically-correc, high-fideliy elepresence and eleoperaion of any robo in he flee, and lowerbandwidh links for command and elemery exchange beween he robos and he elesupervisor worksaion. Should he high-bandwidh video link malfuncion, he sysem design provides for graceful fallback o lowerbandwidh communicaions. I is imporan o noe ha our approach is opimal when he human elesupervisor and he robo flees are near each oher, meaning ha hey are separaed by a roundrip communicaion delay of no more han abou 300 milliseconds (28,000 mi / 45,000 km disan). I is also applicable wih shor elecommunicaion delays, such as beween he Earh and Moon. This paper is srucured as follows. In he nex secion we discuss he novely of our work wih respec o he sae of he ar. In he following secion we describe he archiecure iself, as an overarching paradigm under which all oher subsysems reside. I addresses specifically challenges #1 and #2 above. In he sequel we address challenge #3, presening he Hazard and Assisance Deecion subsysem and is underlying proocols. In he nex-o-las secion we presen deails of our firs applicaion area, he wide-area mineral prospecing ask, including a ask-specific performance meric. The las secion presens conclusions and fuure work. For compleeness, we noe ha our approach o challenge #4 is presened in more deail in anoher paper [10]. Relaed Work The mos imporan aspec of our work, namely, creaing a Robo Supervision Archiecure ha allows a human safely and efficienly o elesupervise a flee of auonomous robos, encompasses a variey of roboic echnologies. We review here only research focused on human-robo ineracion for space exploraion, or which is srongly applicable o he area. The sae of he ar in roboic space exploraion is he Mars Exploraion Rover (MER) mission [8]. Spiri and Opporuniy combined have logged over 10 km and operaed for over 1200 sols (Marian days). This is achieved by assigning a large eam of highly skilled professionals o download elemery and imagery, inerpre he daa, plan he rover aciviies, and program and upload commands every sol, in addiion o a large science eam o selec science arges and asks. In conras, we are muliplying one human s capabiliy o elesupervise a large number of robos, while sill allowing he human o perform oher asks. Anoher difference beween MER and his work is ha, because of he long communicaion delays beween Earh and Mars, he only possible way of operaing he rovers is via bach command sequences which are execued in auonomous mode, whereas he RSA accommodaes a large variey of operaion modes. The sliding auonomy aspec of space exploraion is one of grea imporance. Heger e al. [6], in paricular, have developed an archiecure geared owards humans and robos joinly performing ighly coordinaed asks. They focus on how small eams of robos in space and a few humans on Earh could work ogeher o assemble large orbial srucures, while we focus on maximizing an asronau s efficiency by coordinaing a large flee of robos. From he poin of view of direc assisance o and collaboraion wih asronaus, a relevan projec is Robonau [1]. Robonau s focus is a space robo designed o approach he dexeriy of a space suied asronau. From he poin of view of RSA, a Robonau would be anoher roboic device whose operaion could be coordinaed using our archiecure. When eleoperaed, Robonau s main similariy wih our work is he elepresence capabiliy implemened wih sereo cameras. However, Robonau s use of a head-mouned display and converged cameras differs from our geomerically-correc remoe viewing sysem. Oher human-mulirobo archiecures are hose of Nourbakhsh e al. [9] and Sierhuis e al. [12]. The former focuses on urban search and rescue operaions; heir archiecure allows for simulaneous operaion of realworld and simulaed eniies. The laer have creaed a Mobile Agens Archiecure (MAA) inegraing diverse mobile eniies in a wide-area wireless sysem for lunar and planeary surface operaions. Our work is concepually

3 similar o hese, bu i differs in ha we focus on human safey and efficiency in a planeary exploraion environmen by providing high-fideliy elepresence and a hazard and assisance deecion mehodology ha seeks o opimize he use of human aenion resources given a se of robo assisance requess. Finally, we noe he work of Fong e al. [5], where he auhors also develop an archiecure for supervision of muliple mobile robos. Their work and ours differ in he assisance reques proocols and our use of sereoscopic elepresence. The Robo Supervision Archiecure The RSA is implemened as a mulilayered muli-robo conrol and coordinaion archiecure ha can accommodae differen configuraions of roboic asses based on previous work by Elfes [3], [4]. Here, mulilayered means ha robo sysem conrol is performed a muliple levels of resoluion and absracion and a varying conrol raes. Likewise, replicaed means ha he fundamenal aciviies of percepion, decisionmaking and acuaion occur a each layer of he archiecure. A diagram of he overall RSA archiecure is shown in Figure 2 and explained below. The Auonomous Navigaion Sysem (ANS) is replicaed on each robo for local rover navigaion. In he same way, each robo's Hazard and Assisance Deecion (HAD) sysem is ighly coupled wih he local ANS, and is suppored up hrough he layers of he RSA archiecure for high-level decision-making and handover o he elesupervisor. The Human Telesupervisor oversees he enire mission planning and execuion, being able o assume a wide range of roles from pure supervision while monioring he progress of he assigned asks; o monioring he performance of he flee of auonomous robos; o pure eleoperaion of any robo vehicle or is subsysems. This means ha he RSA covers he enire sliding auonomy specrum as defined in [6]. The reader should noe ha his is no o be confused wih he so-called levels of ineracion engagemen [11], as we are no dealing wih he issue of robos ineracing wih humans in a social way. Task Planning and Monioring and Robo Flee Coordinaion lie a he core of he Robo Supervision Archiecure. The high-level mission plans are creaed and edied wih he Planning Tools, and are hen assigned o he Robo Flee Coordinaion module, which decomposes hem ino asks and assigns hese o he individual robo conrollers (see Figure 3). Figure 2. RSA sysem-level block diagram and main daa pahs.

4 mobiliy, manipulaion, and oher engineering or science subsysems, he corresponding robo conroller may be implemened as a collecion of modules responsible for each one of hem. Each robo conroller is currenly subdivided ino he Auonomous Navigaion Sysem and Prospecing Task Suppor modules, respecively responsible for conrolling he mobiliy and prospecing subsysems. The ANS is responsible for decomposing he navigaion pah assigned o i and reporing is progress. The iniial Prospecing Task Suppor sofware a he robo level is very simple: i merely suppors he conrol and monioring of he prospecing ools. Figure 3. RSA Planning, Monioring, and Coordinaion. Robo Flee Coordinaion also collecs operaional resuls from all robos and inegraes hem for convenien human monioring and analysis. The Monioring Level also includes Percepion Decision-Making Acuaion sequences o monior muli-robo operaions a he sysemlevel, and o analyze for high-level hazard and assisance deecion. Robo Flee Coordinaion imagery and elemery are combined for building regional imagery and maps, and are also presened graphically o he elesupervisor, as shown in Figure 4. Figure 4. Telesupervisor worksaion (concep). A suie of ediing ools for plans and maps, as well as manipulaion ools are available a boh he Planning and Coordinaion levels. The elesupervisor uses hese ools and oher display-oriened ools such as overlay maps, grids, and images for viewing. All hese daa srucures are mainained in a daa sore as depiced a he righ of Figure 3. As depiced in Figure 5, each individual Robo Conroller subsysem is responsible for receiving a collecion of asks from he Robo Flee Coordinaion and monioring is execuion. The robo conroller has direc access o all of he robo s subsysems o drive acuaors and read sensor daa. When a robo is a relaively complex combinaion of Figure 5. Robo Conroller. Jus like he Robo Flee Coordinaion module, he Robo Conroller includes a se of robo-specific Combining Tools. These include funcions such as generaing composie elevaion maps and images along he pah of he individual robo by combining he separae deph maps and images capured by he robo s cameras. Each Hazard and Assisance Deecion subsysem is responsible for he robo s and he mission s inegriy, including vehicle monioring and healh assessmen and failure deecion, idenificaion, and recovery. I assesses he robo s sensor daa o infer poenial hazards or he machine s inabiliy o complee a ask, and communicaes wih he Robo Flee Coordinaion, which mus ake he appropriae acion(s). Robo Telemonioring allows each robo o be consanly moniored a low bandwidh by he human elesupervisor wih imagery and daa updaed regularly from each prospecing robo vehicle. As shown on he righ in Figure 4, each robo has a dashboard which includes sreaming images from one of he robo s cameras, and graphical depicions of saus daa such as baery charge, aiude, moor emperaures, and any oher moniored elemery. The Hazard and Assisance Deecion (HAD) sysem (see nex secion) auomaically moniors each robo. When he elesupervisor should be made aware of a hazardous condiion, i is on ha robo s dashboard ha i is indicaed. This is exemplified for Robo #2 in Figure 4 by an orange surround of he robo view.

5 Telepresence and Teleoperaion subsysems: We make a disincion beween monioring he operaion of each robo, and elepresenly aking conrol of a robo. Whereas monioring is suppored by simulaneous low-bandwidh daa sreams from each of he robos, elepresence is suppored by high-bandwidh sereoscopic imagery and oher elesensory modaliies one robo a a ime. I provides no only he sereoscopic visual, bu also aural, and aiude propriocepive feedback ha allows for more immersive elepresence. Teleoperaion involves direc human conrol of a single robo when a vehicle mus be remoely driven raher han operaing under is Auonomous Navigaion Sysem; and when he ask-specific ools mus be operaed manually. Joysick, keyboard, and ask-specific inpu devices suppor his. This subsysem is implemened over a dual-pah daa communicaion infrasrucure, where he low-bandwidh pah is used for communicaion of commands and elemery daa, and he high-bandwidh pah is used for sereoscopic video. In addiion o he described funcionaliy, each subsysem is a source for daa which are boh archived for laer analysis, and also provided in par as a sream for access by a Disan Exper who can consul as required. Hazard and Assisance Deecion The Hazard and Assisance Deecion (HAD) subsysem is responsible for he following high-level capabiliies: Single-rover hazard and assisance assessmen. Muli-rover assisance reques prioriizaion and managemen. Is overall goal is o assess siuaions where a robo requires assisance, aler he operaor abou hem, and opimally prioriize asks for he operaor such ha he overall pause-ime of rovers in he eam is minimized wihou endangering hem and, consequenly, he mission. Pause-ime is he duraion of ime when he rover pauses while waiing for elesupervisor aenion. Task-specific assisance requess and auomaically-idenified science opporuniies boh benefi from, and are handled wihin he hazard assisance and scheduling ools; erminology which is used hroughou our curren discussion. The deecion of hazards involves a fusion of various sensor inpus o generae a comprehensive picure of he siuaion. Boh he lower-level HAD locaed on each rover and he higher-level HAD locaed in he elesupervisor worksaion are implemened as a percepion decisionmaking acuaion sequence. A he lower level, he percepion aspec includes receiving inpus from he various sensors and subsysems, including such daa as moor curren, baery levels, and odomery. The decision-making aspec involves assessing he whole specrum of available sensor daa and deermining wheher here is a poenial for hazard. If such a hazard is idenified, he urgency of he siuaion is evaluaed by examining which hazard sae i is in (normal green, warning yellow, or high-aler red) and by calculaing a predicion of how quickly i will degrade (maximum allowable neglec ime) and he esimaed ime o fix he problem (fix-ime). The acuaion aspec involves responding o he deecion of a red aler hazard by haling he acuaors on he robo and alering he supervisor wih a hazard flag and he imebased esimaes menioned above. If he flag is merely a yellow warning and no a high-aler red, hen he supervisor is cauioned abou he siuaion, bu he operaion of he rover remains uninerruped. A he higher level on he supervisor side, HAD percepion receives cauionary (yellow) and high-aler (red) hazard flags, and heir relaed elemery from he rovers. The decision-making aspec prioriizes he hazard flags in he Assisance Queue such ha oal pause-ime for he rover eam is minimized while no endangering any rover. The resuling acions (acuaion) are o inform he elesupervisor of he hazard or poenial hazard hrough he supervisor worksaion conrol panel. Since a elesupervisor oversees muliple rovers, i is no feasible for he operaor o be fully aware of each rover s siuaion a all imes. When a hazard is flagged, he operaor mus be made aware of exacly wha hazard was deeced and why i occurred. To ge a beer picure of he siuaion, daa from various sensors and he video feed are moniored by he HAD algorihm. The HAD subsysem was designed as in Figure 6 o address hese requiremens. Each subsecion of he diagram is explained in a op-down fashion in he following secions. Robo Flee Coordinaion Level: Prioriized Assisance Queue The major moivaion for developing he HAD algorihm is o be able o idenify, aler and prioriize requess for he elesupervisor s aenion. I also opimizes and recommends a paricular order o he operaor for fixing muliple rovers efficienly. The goal ha our algorihm focuses on is o minimize overall robo pause-ime o ge he leas number and shores overall duraion of inoperaive rovers wihou neglecing any long enough o endanger hem. These alers are sored and prominenly lised for he elesupervisor in he Assisance Queue on he worksaion. The following simplifying assumpions were adoped in he prioriizaion algorihm: (i) robos operae independenly of each oher, (ii) conex swiching from one robo being serviced o anoher is insananeous, and (iii) a rover is considered rescued as long as is associaed parameers are back ino he green sae. The auhors are aware ha real-world siuaions are more complex and demand a more sophisicaed implemenaion; for he ime being, however, hese assumpions are no overly resricive, and hey will be addressed in fuure work.

6 Robo Flee Coordinaion - Level HAD Robo Conroller- Level HAD Robo Healh-Relaed Sensor Daa HAD Flag Soring and Prioriizaion HAD Telemery HAD flags o Telesupervisor wih elemery daa and imagery HAD Flag Color Fix-Time Neglec-Time Max Allowable NT Navigaion- Relaed Sensor Daa Prioriized Assisance Queue Task-Specific Sensor Daa Figure 6. Hazard and Assisance Deecion block diagram. Prioriizaion Minimize Pause-Time & Harm Operaor acions Telemery Daa Imagery HAD Log Communicaion Sysem To ANS: Hal robo if Flag = RED Crandall and Goodrich [2] proposed he idea of a cycle of robo effeciveness where effeciveness is modeled o degrade during auonomous operaion and resored o original values hrough human eleoperaion. The ime periods during which he rover acs auonomously on he one hand and under human inervenion on he oher are respecively dubbed Neglec-Time and Ineracion-Time. We adop a similar concep such ha when a hazard flag occurs, Neglec-Time is he amoun of ime he operaor does no address he issue, and Fix-Time (similar o Ineracion-Time) is he amoun of ime i akes he operaor o resolve he hazard siuaion. We assume ha during he repair he rover is no accomplishing is prescribed goal, hus while no down i is paused for he duraion of he Fix-Time. An example of he mos basic case is given o demonsrae he derivaion of he prioriizaion scheme. Imagine wo rovers wih hazards flagged simulaneously and consan esimaed Fix-Times. Rover 1 requires a Fix-Time of 10 unis of ime and Rover 2 requires 5 unis. Assuming ha he performance of Robo 1 is independen of ha of Robo 2, prioriizing robos by shores Fix-Time firs minimizes pause-ime (see Figure 7, where he noaion : 10 indicaes i akes 10 unis of ime o fix robo 1 and wais: 10 indicaes robo 2 wais for aenion for 10 unis of ime ). firs firs Figure 7. Simulaneous arrival of assisance requess. When we incorporae he idea ha flags may arrive a differen imes, hen he ime delay beween requess,, has o be aken ino accoun. Since one arrives before he oher, he problem is no longer which robo o address firs, bu wheher i is more efficien o swich asks in he mids of servicing he firs robo. As a simplificaion, we neglec he possible ime-cos in swiching asks midway. As illusraed in Figure 8, if he Fix-Time of he new hazard from Robo 2 ( ) is shorer han he remaining Fix-Time of Robo 1 ( - ), hen swiching would be more efficien (see Figure 8a, where R 1 : 10, R 2 : 3, : 5). If, on he oher hand, he Fix-Time of Robo 2 is longer han he remaining Fix-Time of Robo 1, swiching is less efficien, as in Figure 8b, where R 1 : 10, R 2 : 5, : 7. The crierion used for he swiching decision is hus < -. (a) : 10, : 3, : 5 Finish firs Swich To Finish firs : 10 : 5 wais: 5 Toal Pause-ime = 18 Toal Pause-ime = 16 : 3 : 5 Wai: 3 : 5 : 5 (b) : 10, : 5, : 7 Swich o : 10 wais: 10 : 5 Toal Pause-ime = 25 : 5 : 10 : 7 : 3 Wai: 3 Toal Pause-ime = 18 : 5 : 7 Wai: 5 : 3 : 7 : 5 Toal Pause-ime = 20 wais: 5 : 10 Toal Pause-ime = 20 Figure 8. Asynchronous arrival of assisance requess.

7 The above examples assume ha he severiy and herefore he required Fix-Time of unaddressed hazards do no increase wih ime. In fac, hazards may worsen during periods of neglec. We borrow he erm Neglec-Time (wai ime in Fig. 7 and 8) from [2], bu noe ha i refers here o he enire ime beween periods of human inervenion, whereas our Neglec-Time denoes only he ime beween he flagging of a hazard and he poin of human inervenion. The oal ime from flagging a hazard o resoring i o he safe, green zone can hen be expressed as he sum of Neglec- and Fix-Time ( neglec + fix ). This is illusraed in Figure 9, where is a generic hazard parameer. max hreshold A Figure 9. Neglec and Fix-Time. Wih his addiional complexiy of Neglec-Time added, he crierion for deciding o swich asks for maximum efficiency changes o _neglec + _fix < _neglec + _fix - [1] B fix = F - B Example :{ fix =10, neglec =5}; :{ fix =3, neglec =8}; :7 negl.: 5 : 7 neglec = B - A Hazard Flagged fix: 10 neglec: 8 Human Inervenes fix: 3 C F Hazard Fixed Figure 9 also shows he rapezoidal model we adoped o esimae Neglec and Fix Times linearly. The jusificaion for his model is given in a subsequen secion. In he general case, he hazard severiy increases during neglec, as refleced in Figure 9 by moving closer o he maximum allowable value.. Given he rae of change of we can make an esimae of he maximum allowable neglec ime remaining, max_neglec, before his his ceiling as seen in Figure 11 ( max_negl_yellow = D - B and max_negl_red = G - E ). Anoher imporan consideraion for scheduling is herefore ha his deadline for fixing be me. This means ha on op of evaluaing he swiching crieria for maximum efficiency, here needs o be an evaluaion for feasibiliy of swiching wihou endangering any robos. Thus hese wo crieria emerge: _max_neglec > _neglec + _fix + [2a] _max_neglec > _neglec + _fix [2b] There are four possible scenarios given hese wo equaions as seen in Figure 10. The prioriizaion scheme for each of hese 4 cases becomes: Case 1: Swich from o if Equaion [1] is rue. Case 2: Say wih. Case 3: Swich o. Case 4: Swich from o if _max_neglec < _max_neglec Case 1 is based purely on minimizing pause-ime. Cases 2 and 3 are invoked when one of he rovers would reach is criical limi if pause-ime is minimized. The reasoning for Case 4 is o noify he elesupervisor of which problem is mos likely o reach criical limis firs. The human can bes judge which siuaion is more easily fixable despie poenially reaching hose criical limis or wheher i would be hopeless o address one or he oher. Case 1: Equaion [2a]&[2b] saisfied Case 2: Equaion [2a] NOT saisfied Case 3: Equaion [2b] NOT saisfied Case 4: Neiher equaion saisfied Max Allowable Neglec Time Neglec + Fix Time Figure 10. Maximum allowable neglec-ime cases.

8 The HAD scheme for prioriizing flags from muliple rovers is hus o place he firs flag ha arrives ino he firs posiion in he Assisance Queue and swich he order wih a subsequen incoming flag only if i mees he crieria based on Equaions [1] and [2] and he cases oulined. In essence, Equaion [1] minimizes pause-ime while Equaion [2] prevens furher harm. In a dynamic siuaion, he Fix-Time as well as a maximum allowable Neglec-Time can be esimaed based on curren values and known models for resolving a hazard. This is done on he rover side and communicaed o he operaor side for he prioriizaion scheme given above. Soring - Urgency Levels The HAD prioriizaion scheme is furher caegorized ino hree levels of urgency: normal green, warning yellow and aler red. The regions are separaed by hreshold values for he parameers being moniored, as shown in Figure 11. These hresholds are based on mission and robo design. In green, he rover operaes auonomously and repors elemery a regular inervals o he elesupervisor worksaion. In yellow, he elesupervisor is warned of a poenial upcoming hazardous siuaion. Auonomous operaions coninue bu elemery is repored more frequenly and roubled sensors highlighed. In red, everyhing is he same as yellow excep ha he rover hals iself immediaely and wais for elesupervisor aenion. The red and yellow flags are prioriized by he above scheme separaely. Then he red flags are lised firs in he Assisance Queue followed by he yellow. The reason for having a yellow warning region is o have a buffer and addiional warning for he operaor. I is always beer o be more cauious especially when i comes o he expensive endeavor of planeary exploraion. max red_hreshold yellow_ hreshold Figure 11. Monior and flag. Robo Flee Coordinaion Level: Hazard Logs from HAD Telemery on he Robo Conroller Level B A Scieniss and engineers may wan laer o sudy he deeced hazards, so all relevan daa in he hazard siuaion mus be recorded and saved. This log includes a sequence of evens saring when he HAD flag was raised and lasing unil he operaor hands back conrol o he rover. D C E F G Each even receives a ime samp. The relevan elemery from vision and sensors are saved, as well as a record of he acions aken. The operaor is asked o briefly sae wha and why s/he is aking such an acion eiher hrough ex or voice recording. Robo Conroller Level: Fix-Time for Hazards As inroduced above, he esimaed amoun of ime necessary for a human o remedy a hazard is known as Fix-Time. The prioriizaion descripion above implies ha i is more efficien in erms of oal pause-ime o ackle a hazard wih shorer Fix-Time before urning o rovers wih longer Fix-Times. As shown in Figure 9 and Figure 11, we propose a rapezoidal model as an approximaion for Neglec- and Fix-Time. This allows for a linear simplificaion of he curve. Fix-Time consiss of wo linear porions, fix = base + H* neglec where he fla porion of Figure 9 is he firs erm ( base = C - B ) and he sloped porion is he second erm (H* neglec = F - C ), wih he coefficien H being he raio of he raes of degradaion and repair (! & Neglec /! & Fix ). The rae of repair! & Fix is based on specific models for he HAD siuaion; for example, for he case of a low baery,! & would be he rae of recharge. The rae of degradaion Fix! & on he oher hand is Neglec dependen on he curren siuaion; for example, he case where a baery is discharging normally would be differen from a case where here is a shor circui discharging i. base describes he ineviable urnaround ime i will ake once he human eleoperaor sars o address he problem. Also, in he mos basic case where HAD monioring is binary, for example he camera is on or off, he sloped erm would obviously be zero. Conversely, for cases where he urnaround is nearly insananeous, hen base may equal zero insead. The sloped porion H* neglec described above gives an esimae based on curren parameers of approximaely how long he repair will ake o bring back ino green based on he siuaion (! & ) and he hazard ype (! & ). Fix Robo Conroller Level: Neglec-Time Neglec The Neglec-Time consiss of he firs sloped porion in he rapezoidal approximaion of he curves in Figure 9 and Figure 11. This value is derived simply by racking he amoun of ime elapsed beween when ransiioned from yellow o green and when he elesupervisor begins o work on he rover. One reason for keeping rack of Neglec-Time is for deermining approximaely wha he Fix-Time should be. Anoher criical reason is o ensure ha he ask is addressed before a cerain deadline as described in he nex secion.

9 Robo Conroller Level: Maximum Allowable Neglec Time Remaining There has been much work on predicion for navigaion, in paricular, for obsacle avoidance during navigaion. However, here is much less discussion of predicive algorihms for oher aspecs of rover healh and performance. We propose HAD as a general, allencompassing predicive and opimizing algorihm for monioring and handling any expeced, or unexpeced, hazards. For each possible HAD case evaluaed, each will monior a leas one sensor value (such as pich angle) or calculaed value (such as odomery), as well as heir respecive raes of change. Le denoe any variable being moniored. When a rover is raversing, monioring jus he value iself will only inform he rover of a hazard afer he fac (e.g., rover already suck). However, if we also monior he rae of change of his value, we can cach any ineresing rends of decline/incline and ge a ime esimae of when will reach some hreshold value. Two parameers define he yellow hazard zone:! yellow_ hreshold and! red _ hreshold. The red aler zone is similarly defined by! red _ hreshold and! max. Thus he predicion equaions become for yellow,! red _ hreshold "! curr # yellow_ max _ neglec =! & [3a] and for red, curr! max "! curr # red _ max _ neglec =! & [3b] Wha we ge is a prediced ime, based on curren rends, before he rover s will reach a dangerous hreshold. This advance warning will give he operaor ime o poenially address he issue before i degrades furher and i will also influence how HAD flags are prioriized in he Assisance Queue on he operaor side. This esimae of maximum allowable neglec ime remaining before hiing a hreshold (Equaions [3a] and [3b]) provide a deadline by which he operaor mus address he issue. I does his hrough Equaions [2a] and [2b] o evaluae he feasibiliy for swiching HAD asks on he operaor side during he prioriizaion process. curr Wide-Area Prospecing Our firs-year es for he RSA is o auonomously search an area for in siu resources wih assisance from a human elesupervisor when needed. A se of onboard insrumens for each rover represens an analysis sysem ha will funcion as a sand-in for a suie of insrumens for he Moon. I is no he purpose of his projec o design a complee, inegraed, chemical analysis sysem, bu raher o demonsrae he ineracions beween human elesupervisors and prospecing robos, and validae heir performance in idenifying resources in he field. In he fuure, he prospecing insrumens will be expanded o include sampling ools. A chosen area will be prospeced using a predeermined search algorihm defined prior o he es (see Figure 12). This simple grid-search algorihm is akin o any iniial erresrial prospecing ask where no a priori mineral informaion is known abou he area. I is also analogous o sub-surface sample prospecing, such as core boring, which is one of our arge prospecing asks in he fuure. Figure 12. Robo pahs wihin prospecing area. To provide a racable prospecing ask for developmen and validaion, we have eleced o do a surface sudy wih limied manipulaion and non-conac sensor-based sampling. Because in he fuure he prospecing ask will involve discree prospec sies wih physical soil and/or core samples colleced and reurned o base for analysis, he iniial sudy is designed o be an analogue, wih surface sensor daa being aken in a similar fashion a discree prospec sies. This surface sensor will provide daa ha can be analyzed wihou he need o deploy expensive sampling ools (e.g., a mass specromeer) on each of he robos. This will be accomplished by adaping a video camera o collec visual sample daa for an area of approximaely 0.25 m 2 a each sampling sie. The arge maerial densiy measuremens colleced a each sampling sie ogeher wih he coordinaes of each sie will be used o build a map of maerial densiy over he prospecing area. An algorihm ha infers a disribuion over he whole prospecing area will be employed. This map will hen be compared o he resource map mainained by our geologis who seeded he prospecing area o characerize he accuracy of he prospecing aspec of he sysem. To quaniaively assess how well he sysem performs in he prospecing ask, we propose a basic performance meric based on he following noions: 1) greaer area, accuracy, errain difficuly, and safey of prospecing coverage mean increased performance; 2) greaer effor and ime required mean decreased performance. Given hese facors, we propose he following meric: ACTS P = ( R / w + H E )

10 where: P: performance in unis of (area accuraely prospeced)/(effor-ime). A: area covered. C: prospecing accuracy; C = 1 corresponds o he highes possible accuracy and C = 0 corresponds o he lowes possible accuracy. T: errain difficuly facor (T 1) wih T = 1 corresponding o he easies errain (a fla surface wihou obsacles). S: safey facor (S 1) wih S = 1 corresponding o he leas safe ask performance, i.e., via 100% EVA. R: number of roboic rovers (ineger). H: number of humans (ineger, bu does no occur in he performance formula); noe ha alhough our projec s focus is on a sysem in which a single human asronau conrols muliple rovers, he meric is general enough o allow for muliple humans. H E = human effor defined as he degree of human ask inervenion in full-ime equivalens (0 H E H); e.g., if one human inervenes 30 min. during a 1-hr. ask, H E = (30/60) = 0.5; if hree humans inervene 15, 30, and 45 min. respecively during a 1-hr. ask, H E = (15/60) + (30/60) + (45/60) = 1.5. w: facor allowing commensurabiliy of human and rover ime by giving he relaive value of he former o he laer; e.g., w = 4 ses human ime o be four imes as valuable as rover ime R/w + H E : combined human-rover effor. = ime required o cover A. We will repor on he resuls obained wih his meric afer we conclude our indoor and oudoor ess in he Fall of Conclusion The work presened in his paper summarizes pars of a larger echnology developmen effor being underaken by he auhors under NASA suppor and in cooperaion wih NASA ceners. Oher aspecs of his effor include he Auonomous Navigaion Sysem, based currenly on sandard binocular vision [13]; he Telepresence and Teleoperaion Sysem [10]; and oher ask-specific elemens. Our ulimae goal is o deliver he enire Robo Supervision Archiecure o NASA a echnology readiness level 6 [7], afer exensive field ess where one human will elesupervise a flee of eigh o en auonomous robos performing mineral prospecing and core sample drilling.. Specifically wih respec o HAD, in he near fuure more sophisicaed assessmen proocols will be implemened, possibly using a Bayesian framework for dynamic sae esimaion [13]. This will improve he abiliy o idenify obsacles and will also aid in he performance of opporunisic science in ha feaures of ineres can be deeced wih greaer accuracy and frequency. References [1] R.O. Ambrose, R.T. Savely, S.M. Goza, e al. Mobile Manipulaion Using NASA s Robonau. Inl. Conference on Roboics and Auomaion, New Orleans, USA, April 2004, pp [2] J. W. Crandall, M. A. Goodrich, D. R. Olsen, and C. W. Nielsen. Validaing Human-Robo Ineracion Schemes in Muli-Tasking Environmens. Submied o SMC Par-A: Special Issue on Human-Robo Ineracion. [3] A. Elfes. Incorporaing spaial represenaions a muliple levels of absracion in a replicaed mulilayered archiecure for robo conrol. in Inelligen Robos: Sensing, Modelling, and Planning, R. C. Bolles, H. Bunke, H. Nolemeier (eds.), World Scienific, [4] A. Elfes. Dynamic conrol of robo percepion using muli-propery inference grids. Inl. Conference on Roboics and Auomaion, Nice, France, May 1992, pp [5] T. Fong, C. Thorpe, and C.; Baur. Muli-robo remoe driving wih collaboraive conrol. IEEE Transacions on Indusrial Elecronics, Vol. 50, No. 4, Augus 2003, pp [6] F. Heger, L. Hia, B.P. Sellner, R. Simmons, and S. Singh, Resuls in Sliding Auonomy for Muli-robo Spaial Assembly. 8h Inernaional Symposium on Arificial Inelligence, Roboics and Auomaion in Space, Munich, Germany, Sepember, [7] J.C. Mankins. Technology Readiness Levels: A Whie Paper. NASA Office of Space Access and Technology, April hp://advech.jsc.nasa.gov/ downloads/ TRLs.pdf. [8] NASA Mars Exploraion Rover Mission. hp://marsrovers.jpl.nasa.gov/home/index.hml. [9] I. Nourbakhsh, K. Sycara, M. Koes, e al. Human-Robo Teaming for Search and Rescue. Pervasive Compuing, Vol. 4, No. 1, Jan-Mar 2005, pp [10] G. Podnar, J. Dolan, A. Elfes, M. Bergerman, H.B. Brown and A.D. Guisewie. Human Telesupervision of a Flee of Auonomous Robos for Safe and Efficien Space Exploraion. Submied o Conference on Human- Robo Ineracion, Sal Lake Ciy, USA March [11] T.E. Rogers, J. Peng, and S. Zein-Sabao. Modeling Human-Robo Ineracion for Inelligen Mobile Roboics. IEEE Inernaional Workshop on Robos and Human Ineracive Communicaion, Nashville, USA, Augus 2005, pp [12] M. Sierhuis, W.J. Clancey, R.L. Alena, e al. NASA s Mobile Agens Archiecure: A Muli-Agen Workflow and Communicaion Sysem for Planeary Exploraion. 8h Inernaional Symposium on Arificial Inelligence, Roboics and Auomaion in Space, Munich, Germany, Sepember [13] A. Soo and P. Khosla. Probabilisic adapive agen based sysem for dynamic sae esimaion using muliple visual cues. Carnegie Mellon Universiy, Pisburgh, PA, 2001.