Benchmarking Intelligent Service Robots through Scientific Competitions. Luca Iocchi. Sapienza University of Rome, Italy

Transcription

1 Benchmarking Intelligent Service Robots through Scientific Competitions Luca Iocchi Sapienza University of Rome, Italy

2 Motivation Development of Domestic Service Robots Complex Integrated Systems AI&Robotics Large variety of tasks Human-Robot Interaction

3 About Starts in international competition per year Many regional competitions Largest competition for domestic and service robots

4 Large variety of tasks University of Bonn, Germany

5 Benchmarking Robotic Abilities Functional benchmarking Planning domains (e.g., IPC) SLAM (e.g., Radish, RAWSEEDS) Person tracking (e.g., PETS) Object recognition (e.g., PASCAL) Navigation (e.g., MoVeMe) Mobile manipulation Integrated system benchmarking Benchmarking the entire system => robotic competitions Combining Functional and Integrated system benchmarking => trend for the future

6 Robotic scientific competitions DARPA / EUROC Challenges RoboCup AAAI / ICRA / IROS robot competitions RoboCup Junior, Eurobot Advantages of Competitions Difficulties for DSR Set up of common test-beds Attractive for many teams (research groups) Collaboration and knowledge sharing Evolution over time Human involved Real environments Integration of several functionalities Large variety of tasks

7 Observations from other Robot Competitions Little HRI involved Limited application orientation No real world environment Very specific rules and regulations for robots and environment Often requires many resources (special environment, many robots) Danger of developing towards local optima

8 Fixed task and performance improvement over time Local optimum (overfitting) Performance Local Optima in Benchmarking Given a fixed task Increasing difficulty of tasks and maintaining performance Increase general applicability Performance Time Tasks

9 Scenario and Concepts

10 Scenario and Concepts Autonomous robots Human-Robot Interaction Non-standardised realistic domestic environment and real public areas Many tests related to desired functionalities Changing tests over the years to keep performance "constantly good" Statistical evaluation for measuring league progresses

11 Scenario and Concepts Teams (research groups) are provided with specifications of tasks, environmental setting and performance metrics to evaluate and compare integrated research in domestic and service robots.

12 Current focus of Functional abilities: Navigation Mapping Person recognition Person tracking Object recognition Object manipulation Speech recognition Gesture recognition Cognition

13 Current focus of System properties: Ease of use Fast calibration and setup Natural and multi-modal interaction Attractiveness and ergonomics of the robot Adaptivity and general intelligence Robustness General applicability

14 Implementation of General rules 2 stages with different focus Stage 1 for basic tasks Stage 2 for more complex, integrated tasks High level of uncertainty in the environment (no standardization) Only natural interaction allowed Very short setup time (usually 1 minute) Partial score system for tests

15 Stage 1 Robot Inspection & Poster: Autonomous registration to the competition, TC inspection, team poster Follow me: Lead the robot quickly on a path through an external scenario Cocktail Party: Deliver drinks to people in the apartment Clean up: Clean up a room in the apartment Emergency Situation: React to an unknown emergency situation Technical Challenge: Defined by TC every year Open Challenge: Demonstrate most important (scientific) achievements (team choice)

16 Stage 2 Enduring General Purpose Service Robot: Solve multiple tasks not known beforehand upon request Restaurant: Mapping and serving drinks and food in a real unknown restaurant Demo Challenge: Topic-based free demonstration (e.g., health-care, cooking, ) Finals: Open demonstration with external jury + Exec evaluation

17 Final

18 Apartment, People and Objects

19 Person names

20 Objects

21 Object categories and default locations

22 Test evolution: 'follow me' example 2007: proof of concept, special markers on the walker allowed 2008: walker known, but no special markers 2009: walker unknown 2010: outside the arena (in the RoboCup venue) 2011: pre-defined interferences (people passing between walker and robot) : crowded and complex environment (changing floor through an elevator) future: public environment with crowd and unpredictable interferences

23 Benchmarking Robot Cognition: General Purpose Service Robot The test is about how much the robot can understand and reason about the environment and its task No predefined task Task goals are randomly generated at runtime Task goals can include multiple objects/locations, underspecified objects/locations and wrong information GPSR incorporates the abilities tested in all previous tests.

24 Benchmarking Robot Cognition: General Purpose Service Robot Task goal is not predefined! Given a set of known objects, known locations and known persons, execute a randomly generated task from a set of templates.

25 Evaluation of the League Year by year statistical analysis to: Measure overall performance Drive developments Plan for rule changes

26 Score system of Navigation Follow Me 49 % Mapping Person Recogn. 6% Person Tracking Object Recogn. Object Manipul. Speech/ Gesture Recogn. 39 % Clean Up Cocktail Party Emergency Situation Defined by the Technical Committee General Purpose Service Robot Restaurant Open Challenge Demo Challenge Defined by the teams Final 6% Cognition

27 Functionality distribution Each test includes a set of the functional abilities Distribution of functional abilities over tests evolves over time allowing for proper analysing and planning.

28 Functionality distribution for a test Example from Follow me 2012 test Navigation Object Person Person Recognition Recognition Tracking Object Speech Gesture Mapping Cognition Manipulation recognition recognition Distribution over tests evolves over of functional abilities 0.5 time allowing for proper analysing and planning. CP1 CP CP Complete

29 Evaluation Best/average score of the finalist teams.

30 Evaluation

31 Evaluation Performance metrics of the league over the years Performance do not always increase because of changes in the rules (major changes in 2008, 2010, 2012). Good: we are not going towards a local optimum!!!

32 Community Resources Web site (information and Wiki (> 50 teams active worldwide) HW/SW/Papers Mailing lists (active rule discussion)

33 Scientific Achievements Speech understanding in noisy environments Speaker localization for following human guides Detecting and tracking human operators using laser and RBGD cameras Detecting, learning and recognizing objects Complex two-hands object manipulation Demonstrated within an integrated system

34 Future directions of More general tests in the real world (also outdoor) Longer operation times Less pre-defined pre-planned behaviors Improved cognitive and social skills (language skills, social behaviors) Human-robot and Inter-team robot-robot cooperation Detecting and dealing with failures and unpredicted situations

35 Conclusions can drive the development of effective intelligent robots Benchmarking methodology based on the definition of several variable tests Cognitive abilities can be actually tested in a real setting Statistical analysis can drive fast achievements of the league. Research groups can exploit RoboCup@Home to develop, test, evaluate and disseminate DSR solutions.

36 References Tijn van der Zant and Thomas Wisspeintner. Creating and Benchmarking Tomorrows Service Robot Applications. IN: Lima, P. (Ed) Robotic Soccer, Vienna: I-Tech Education and Publishing. pp , Thomas Wisspeintner, Tijn van der Zant, Luca Iocchi, Stefan Schiffer. Scientific Competition and Benchmarking for Domestic Service Robots. Interaction Studies. 10 (3) pp , ISSN: Jörg Stückler, Dirk Holz, and Sven Behnke: Demonstrating Everyday Manipulation Skills in IEEE Robotics & Automation Magazine 19(2):34-42, Dirk Holz, Luca Iocchi, Tijn van der Zant, "Benchmarking Intelligent Service Robots through Scientific Competitions: The Approach", In AAAI Spring Symposium: Designing Intelligent Robots, 2013.

37 Thank you for your attention Questions?