Benchmarks and Good Experimental Methods in

Transcription

1 Benchmarks and Good Experimental Methods in Robotics Research Angel P. del Pobil Universitat Jaume I, Spain RSS 08 Workshop Zürich June 28,

2 Research Benchmarks: Goals a) Consolidate currents efforts in benchmarking b) Dissemination of existing benchmarks efforts in the community through discussions coordinated with international events, integrating experts in workshops and studies, and through the website c) Focus on disseminating good experimental methodology together with benchmarking d) Increase awareness by organizing and participating in meetings, discussions, workshops,... on benchmarking, performance metrics and good experimental practice 2

3 Research Benchmarks: Work Redesigned and up-to-date website of existing on-going benchmarking initiatives and other relevant efforts Promote good experimental practice through discussions, workshops and guidelines -as suggested by reviewers Many workshops, discussions and meetings Five international workshops organized: 1. IROS'07 Workshop on Performance Evaluation and Benchmarking for Intelligent Robots, November EURON'08 Workshop on Good Experimental Methodology and Benchmarking in Robotics, Prague, March with EUROS Workshop on Good Experimental Methodology & Benchmarks in Cognitive Robotics, Karlsruhe, April with CogSys RSS'08 Workshop on Experimental Methodology and Benchmarking in Robotics Research, Zürich, June IROS'08 Workshop on Performance Evaluation and Benchmarking for Intelligent Robots, Nice, September 2008 (accepted) 3

4 Redesigned website New version of up-to-date exhaustive, detailed repository of current existing efforts in comparative research: competitions, benchmarks, conferences, data sets, etc. Result of a long process of information gathering, from different sources or provided by a number of persons. This information was previously unavailable, scattered in different sources and merged with uninteresting issues. Available as a 242-page document (Deliverable DR 2.7) -with IROS 07 Proc. Also at completely redesigned website: Number 1 in a search in Google for -robot & benchmarks- out of 426K hits 4

5 IROS 07 Workshop on Performance Evaluation and Benchmarking for Intelligent Robots San Diego, November 2, 2007 co-organized with Raj Madhavan Workshop outcome: Full-day event Presentations by 7 speakers, 21 authors Well attended, lively discussions Full-paper proceedings at website and as appendix in DR 2.7 5

6 IROS 07 Workshop Program Synthetic Approach to Cognitive Systems: A Perspective from Cognitive Robotics, Kaz Kawamura Benchmarking Urban 6D SLAM, O. Wulf, A. Nuchter, J. Hertzberg, and B. Wagner The Jacobs Test Arena for Security, Safety, and Rescue Robotics (SSRR), A. Birk, K. Pathak, J., S. Schwertfeger, M. Pfingsthorn and H. Bulow Towards Quantitative Comparisons of Robot Algorithms: Experiences with SLAM in Simulation and Real World Systems, B. Balaguer, S. Carpin, S. Balakirsky Reliability Testing for Embodied Autonomous Systems, L. F. Gunderson and J. P. Gunderson Advances in the Framework for Automatic Evaluation of Obstacle Avoidance Methods, J.L. Jimenez I. Rañó, J. Minguez Good Experimental Methodologies in Robotics: State of the Art and Perspectives, F. P. Bonsignorio, J. Hallam, and A. P. del Pobil 6

7 EURON Workshop on Good Experimental Methodology and Benchmarking March 25/26 in Prague with EUROS 2008 Formal presentations, discussions, Lecture Notes to appear Workshop outcome: Two-day event Presentations by 8 speakers, 14 authors Well attended, lively discussions 7

8 EURON 08 Workshop Program Day 1 Open discussion about benchmarks and architectures for robustness and autonomy, H. Bruyninckx. Day 2 Plenary discussion. Proposals for benchmarking SLAM, G. Fontana, M. Matteucci, J. Neira, D. Sorrenti. Motion Planning vs. Automated Planning in benchmarking, M. Reggiani, E. Pagello. Benchmarking mobile robots' motion, A. Marjovi, L. Marques. Plenary discussion. The Hydra-Shiva concept for GEM and Benchmarking in robotics, A Moshaiov. Plenary discussion Advances in the Framework for Automatic Evaluation of Obstacle Avoidance Methods, J Minguez. RoSta - A Brief View Over Benchmarking Activities In Service Robotics, K. Pfeifer. GEM and Benchmarking in robotics, where we are? Serious? Science?, F. Bonsignorio, J. Hallam, A. P. del Pobil. Plenary discussion. 8

9 Workshop on Good Experimental Methodology & Benchmarks in Cognitive Robotics April 1 in Karlsruhe -collocated with CogSys 2008 Informal presentations and discussions Half-day event: GEM and Benchmarking in robotics, F. Bonsignorio, J. Hallam, A. P. del Pobil. Quality Measures for Mapping: from Test Environments to Analysis Tools, M. Pfingsthorn, A. Birk. Benchmarking in the DEXMART project, G. Grünwald Experiences in evaluating human-robot interaction - COGNIRON, I. Lütkebohle Open Discussion 9

10 RSS 08 Workshop on Experimental Methodology and Benchmarking in Robotics Research Zürich, June 28, Workshop details: Half-day event -due to competition Presentations by 7 speakers, 21 authors Full-paper proceedings 10

11 RSS 08 Workshop Program Introduction, A.P. del Pobil The Jacobs Map Analysis Toolkit, I. Varsadan, A. Birk M. Pingsthorn, S. Schwertfeger, K. Pathak RobotStadium: Online Humanoid Robot Soccer Simulation Competition, O. Michel Can We Benchmark The Influence of Information-Processing Architectures on Intelligent Systems? N. Hawes, J. Wyatt The RAWSEEDS Proposal for Representation-Independent Benchmarking of SLAM, G. Fontana, M. Matteucci, D. G. Sorrenti A unified benchmark framework for autonomous Mobile robots and Vehicles Motion Algorithms (MoVeMA benchmarks), D. Calisi, L. Iocchi, D. Nardi Performance metric for vision based robot localization, E. Frontoni, A. Ascani, A. Mancini, P. Zingaretti The EURON GEM Review guidelines, F. Bonsignorio, J. Hallam, A. P. del Pobil Open discussion 11

12 IROS 08 Workshop on Performance Evaluation and Benchmarking for Intelligent Robots Nice, September 26, 2008, co-organized with Raj Madhavan Call for papers still open. Topics include: Knowledge representation, perception and learning Uncertainty management in robot navigation Cognitive manipulation Benchmarking autonomy and robustness to changes Capability-led understanding of cognitive robots Shared ontologies based on performance capabilities Relationships between different cognitive robotics capabilities Experimental scenarios to evaluate performance, generality and robustness Benchmarking and replication of experiments with robots 12

13 SIG on Good Experimental Methodology Proposal accepted. Chaired by John Hallam, Fabio Bonsignorio and A. P. del Pobil Co-organized most of the workshops. Working group meetings: Benicassim, Spain, 9/1/2008, collocated with RISE 08 Genoa, Italy, 31/1/2008, Zaragoza, Spain, 4/3/2008 Prague, 28/3/2008, as part of the EURON Annual meeting Valencia, Spain, 30/4/2008 Main result is a document entitled General Guidelines for Robotics Papers Involving Experiments, available from the website and as an appendix in DR2.7 With particular descriptions for the following domains: 1. SLAM 2. Mobile Robots Motion Control 3. Obstacle Avoidance 4. Grasping 5. Visual Servoing 6. Autonomy/Cognitive Robotics 13

14 General Guidelines for Robotics Papers Involving Experiments 1. Is it an experimental paper? 2. Are the system assumptions/hypotheses clear? 3. Are the evaluation criteria spelled out explicitly? 4. What is being measured and how? 5. Do the methods and measurements match the criteria? 6. Is there enough information to reproduce the work? 7. Do the results obtained give a fair and realistic picture of the system being studied? 8. Are the drawn conclusions precise and valid? 14

15 On-going Benchmarking & GEM Initiatives 30+ on-going initiatives in Europe on benchmarking scenarios and data sets in 14 (8+6) different categories (below) Many websites and some got independent funding (RAWSEEDS, RoSta, Rat s Life, Jacobs Arena ), others have applied as full projects or parts of a project (DEXMART) Details in website and report (DR2.7) 1. Motion planning 2. Manipulation and Grasping 3. Visual servoing 4. Mobile robotics 5. Image processing and computer vision in robotics 6. Networked robotics 7. Cognitive Robotics 8. Good Experimental Methodology New in Year 4: 1. Human-robot interaction 2. Security, Safety and Rescue Robotics - SSRR 3. SLAM 4. Service robotics 5. Software reliability in Robotics 6. Information-Processing Architectures in Robotics 15

16 On-going Benchmarking & GEM Initiatives At least 26 groups active in Europe in benchmarking and GEM Albert-Ludwigs-Universität Freiburg, Germany CYBERBOTICS, Switzerland DLR, Germany Fraunhofer Institute, Germany Heron Robots, Italy Jacobs University Bremen, Germany Jaume I University, Spain Lappeenranta Univ. of Technology, Finland Leibniz Universität Hannover, Germany Politecnico di Milano, Italy Royal Institute of Technology, Sweden SINTEF, Norway Syddansk Universitet, Denmark Tel-Aviv University - Israel Università di Padova - Italy Università di Siena - Italy University of Bielefeld, Germany University of Coimbra, Portugal University of Freiburg, Germany University of Genoa - Italy University of Karlsruhe, Germany University of Milano-Bicocca, Italy University of Osnabrück, Germany University of Parma, Italy University of Zaragoza, Spain Utrecht University, Netherlands 16

17 Impact of Benchmarking & GEM Initiatives Awareness, present in many independent activities: A community exists and new people with proposals in new areas keep showing up in organized events As part of FP7 projects (DEXMART, Synthetic Forager, ) Competent workshops: ICRA 08 on Network Robot Systems: benchmarks and platforms toward Human-Robot Interaction (organized by A. Sanfeliu, UPC, et al.) RSS 08 on Quantitative Performance Evaluation of Navigation Solutions for Mobile Robots, (A. Kleiner, Albert-Ludwigs-Universität Freiburg, et al.) Document General Guidelines for Robotics Papers Involving Experiments discussed at last IEEE Transactions on Robotics Editorial Board Meeting We hope to see impact soon in editorial criteria for reviews: IJRR, 17