University of Bologna, May 21, 2018

Transcription

1 University of Bologna, May 21, 2018 Alessandro Saffiotti AASS Center for Applied Autonomous Sensor Systems Cognitive Robotic Systems Laboratory University of Örebro, Sweden

2 1968 [ Source: SRI International ]

3 Roadmap

4 A bit of history 1949: Enter the Computer Perform mechanical operations on symbols Computer 93

5 A bit of history 1956: Artificial Intelligence Did you say symbols? The red flower Computer La fleur rouge Allen Newell Herbert Simon

6 A bit of history 1956: Artificial Intelligence box(x) banana(y) Symbol System push(x,y) climb(x) grab(y) Encoding Decoding

7 A bit of history 1956: Artificial Intelligence Perception box(x) banana(y) Symbol System push(x,y) climb(x) grab(y) Action From signals to symbols From symbols to symbols From symbols to actions

8 A bit of history 1968: Shakey, the first robot using AI programs Perception Symbol System Action [ Source: SRI International ]

9 A bit of history 1968: Shakey, the first robot using AI programs Did great reasoning planning unexpected events But lived in fake world objects with perfect shape pure and uniform colors uniform and constant light on a flat and regular floor Why? [ Source: SRI International ]

10 A bit of history Perception box(x) banana(y) Symbol System push(x,y) climb(x) grab(y) Action From signals to symbols From symbols to symbols The PASCAL Challenge From symbols to actions [ Source: Everingham et al, IJCV ]

11 A bit of history Perception is difficult Perception box(x) banana(y) Symbol System push(x,y) climb(x) grab(y) Action From signals to symbols From symbols to symbols From symbols to actions

12 A bit of history Perception is difficult Perception box(x) banana(y) Symbol System push(x,y) climb(x) grab(y) Action From signals to symbols From symbols to symbols From symbols to actions [ Source: Siciliano et al, 2014]

13 A bit of history Perception is difficult Action is difficult Perception box(x) banana(y) Symbol System push(x,y) climb(x) grab(y) Action From signals to symbols From symbols to symbols From symbols to actions??

14 A bit of history Circa 1970: AI and Robotics separate Perception Symbol System Action A brain without a body AI Robotics A body without a brain Perception Symbol System Action Perception Symbol System Action

15 A bit of history AI Robotics Perception Symbol System Actuation Perception Symbol System Action

16 A bit of history Circa 2000: AI and Robotics ready to re-unite Perception Symbol System Actuation Perception Symbol System Action AI Robotics Symbol System Perception Action

17 An example

18 cross(door-1) follow(corridor-3) cross(door-2) reach(alex) Symbol System Perception Action

19 Symbol System cross(door-1) follow(corridor-3) follow(corridor-4) cross(door-7) reach(alex) Perception Action

20 Roadmap

21 Why integration is difficult very difficult! Integrating the high-level and low-level different abstraction levels (particular / general) different space & time views (local / global) different information carriers (data / symbols) different formal tools (continuous / discrete) Symbol System Methods and techniques from AI Perception Action Methods and techniques from Robotics Robot & Environment

22 The magical middle layer approach Integrating the high-level and low-level different abstraction levels (particular / general) different space & time views (local / global) different information carriers (data / symbols) different formal tools (continuous / discrete) Interface between existing methods Symbol System Methods and techniques from AI Perception? Action Methods and techniques from Robotics Robot & Environment

23 My message today Integrating the high-level and low-level different abstraction levels (particular / general) different space & time views (local / global) different information carriers (data / symbols) different formal tools (continuous / discrete) Interface between existing methods Symbol System Methods and techniques from AI New methods that span across layers Perception Action Methods and techniques from Robotics Robot & Environment

24 Two case studies Semantic Maps Combined Task & Motion Planning Knowledge representation Task planning Symbol System (goto table3) (IF (on table3 cup22)) (T (approach cup22) (pickup cup22)) (F (goto cupboard1)... Perception Action Perceptual data Robot & Environment Motion planning

25 Roadmap

26 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Magical middle layer Symbol System Task planning (goto table3) (IF (on table3 cup22)) (T (approach cup22) (pickup cup22)) (F (goto cupboard1)... Perception Action Robot & Environment Motion planning

27 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Magical middle layer Symbol System Task planning (goto table3) (IF (on table3 cup22)) (T (approach cup22) (pickup cup22)) (F (goto cupboard1)... Perception Action Motion planning

28 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner Executor Motion planner box cup1 Robot

29 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner 1. PickUp(cup1, leftarm) 2. Place(cup1, box, leftarm) Executor Motion planner Robot

30 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner failed (1) 1. PickUp(cup1, leftarm) 2. Place(cup1, box, leftarm) Executor Motion planner Robot

31 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner 1. PickUp(milkbox, leftarm) 2. Place(milkbox, table, leftarm) 3. PickUp(cup1, leftarm) 4. Place(cup1, box, leftarm) Executor Motion planner Robot

32 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner failed (4) 1. PickUp(milkbox, leftarm) 2. Place(milkbox, table, leftarm) 3. PickUp(cup1, leftarm) 4. Place(cup1, box, leftarm) Executor Motion planner Robot

33 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner 1. PickUp(milkbox, leftarm) 2. Place(milkbox, table, leftarm) 3. PickUp(cup1, rightarm) 4. Place(cup1, box, rightarm) Executor Motion planner Robot

34 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner failed (4) 1. PickUp(milkbox, leftarm) 2. Place(milkbox, table, leftarm) 3. PickUp(cup1, rightarm) 4. Place(cup1, box, rightarm) Executor Motion planner Robot

35 Case study 1: task and motion planning Hierarchical architectures Task planner decides what actions to do Motion planner decides how to realize each action Goal: put cup1 in the box Task planner No plan Executor Motion planner Robot

36 Case study 1: task and motion planning CTAMP: Combined Task and Motion Planning Task planner and motion planner cooperate Only geometrically valid plans are sent to executor Task planner Motion planner Symbol System Perception Action Executor Robot & Environment Robot

37 Case study 1: task and motion planning CTAMP: Combined Task and Motion Planning Task planner and motion planner cooperate Only geometrically valid plans are sent to executor Goal: put cup1 in the box Task planner Motion planner 1. PickUp(milkbox, leftarm, topgrasp) 2. Place(milkbox, table, leftarm) 3. PickUp(cup1, rightarm, topgrasp) 4. Place(cup1, box, rightarm) Executor Robot Success

38 What s really happening? (3, 56, -2, 85, 40, 0) (3, 6, -2, 10, -70, 0) (90, 6, 50, -10, 40, 0) Geometric search space pick-up(cup1) move(cup1,place2) move(cup1,place3) pick-up(box4)

39 What s really happening? Search in the hybrid task-geometry space Distribute search across the two spaces Geometric search space

40 Hierarchical approach Task planner Geometric search space Executor Geometric planner

41 Hierarchical approach Task planner Geometric search space Executor Geometric planner

42 Hierarchical approach Task planner Geometric search space Executor Geometric planner

43 Hierarchical approach Task planner Geometric search space Executor Geometric planner

44 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

45 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

46 Hybrid, interleaved search Task planner Symbolic backtracking Geometric planner Geometric search space Executor

47 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

48 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

49 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

50 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

51 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor

52 Hybrid, interleaved search Geometric backtracking Task planner Geometric planner Geometric search space Executor

53 Hybrid, interleaved search Task planner Geometric planner Geometric search space Executor 2017 A. Saffiotti

54 CTAMP: more variations ( ) Bring geometric failures into ASP solver [Lagriffoul & Andres] Consider geometric ramifictions [desilva, Pandey, Gharbi, Alami] Geometric search guides symbolic search [Cambon, Alami, Gravot] Symbolic search in belief space [Kaelbling & Lozano-Perez] Simplified geometric reasoning instead of search [Gaschler et al]... and so on! Task planner Geometric planner Geometric search space Executor

55 CTAMP: more variations ( )

56 What do they all have in common? Integration achieved through combined algorithms Task planning Symbol System (goto table3) (IF (on table3 cup22)) (T (approach cup22) (pickup cup22)) (F (goto cupboard1)... Perception Action Robot & Environment Motion planning 2017 A. Saffiotti

57 A hybrid planning algorithm [ Source: Bidot, Lagriffoul, Karlsson ] 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 graspable(cup1) empty(rh) empty(lh) cup(cup1)... cup1 = (0.65, -0.3, 0.11) tray = (0.76, 0.61, 0.01) right_arm = (0.34, -2.13, 1.02,......

58 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1)

59 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1)

60 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 Symbolic backtracking (!pick right top cup1) (!pick right top milk_box1)

61 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1)

62 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1)

63 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1) (!pick left top cup1)

64 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1) (!pick left top cup1) (!place_regrasp left top cup1)

65 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1) (!pick left top cup1) (!place_regrasp left top cup1) No path! (!pick_regrasp right bottom cup1)

66 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 Geometric backtracking (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1) (!pick left top cup1) higher (!place_regrasp left top cup1) (!pick_regrasp right bottom cup1)

67 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1) (!pick left top cup1) (!place_regrasp left top cup1) (!pick_regrasp right bottom cup1)

68 A hybrid planning algorithm 1. maintain a hybrid state (s,c) 2. select a symbolic action a with symbolic preconditions true in s 3. select a geometric instance m of a with geometric preconditions true in c 4. update (s,c) to (s,c ) 5. if not final state, go to 2 (!pick right top cup1) (!pick right top milk_box1) (!place right top table milk_box1) (!pick left top cup1) (!place_regrasp left top cup1) (!pick_regrasp right bottom cup1) (!place right bottom tray cup1)

69 A real example Goal: put both cups in the box

70 A real example Perception (!pick right top z1 milk_box_h2) (!place right top table z1 milk_box_h2) (!pick left top z1 large_cup1) (!place_regrasp let top y1 large_cup1) (!pick_regrasp right bottom y1 large_cup1) (!place right bottom boxano z2 large_cup1) (!pick right top z1 small_cup1) (!place right top boxano z1 small_cup1)

71 The key move Task planning Symbol System (goto table3) (IF (on table3 cup22)) (T (approach cup22) (pickup cup22)) (F (goto cupboard1)... Perception Action Robot & Environment Motion planning

72 The key move Task planning Symbol System (goto table3) (IF (on table3 cup22)) (T (approach cup22) (pickup cup22)) (F (goto cupboard1)... Perception Action Robot & Environment Motion planning

73 Roadmap

74 Case study 2: Semantic maps Knowledge representation Symbol System Perception Action Perceptual data Robot & Environment

75 Case study 2: Semantic maps [ Source: Galindo et al, 2008 ]

76 The added value Make implicit knowledge explicit (defconcept Kitchen :is (:and Room (:some at stove)...)) (room area-1) (stove obj-1) (at obj-1 area-1) GOAL: Go to kitchen Infer: (kitchen area-1) Plan: (GOTO area-1)

77 The added value Make implicit knowledge explicit Infer the existence of instances (defconcept Livingroom :is (:and Room (:some at tvset)...) (livingroom area-2) (room area-3) (at me area-3) GOAL: Approach TV-set Infer: (tvset S001) (at S001 area-2) Plan: (APPROACH S001)

78 The added value Make implicit knowledge explicit Infer the existence of instances Deal with contraditions (defconcept Towel :is (:and Utility (:all location Bathroom)...)) (kitchen area-4) (towel obj-22) (at obj-22 area-4) INCONSISTENCY Possible causes: obj-22 is not towel area-4 is not kitchen obj-22 is not at area-4 obj-22 must be relocated

79 More value: get knowledge from the Web The RoboHow project [ Source: ]

80 More value: get knowledge from the Web [Source: Tenorth, Bartels, Beetz ]

81 Semantic maps in the literature

82 What do they all have in common? So far, mostly hyerarchical approach Signal-level reasoning Provides input to

83 Roadmap

84 What these case studies have in common Working in AI and Robotics is not about putting existing pieces together Symbol System? Perception Action Robot & Environment

85 What these case studies have in common Working in AI and Robotics is not about putting existing pieces together It is about building new pieces that span across layers Symbol System Perception Action Robot & Environment

86 Three extensions

87 Three extensions Semantic maps reason about past states (in relation to the current one) CTAMP reasons about future states (in relation to the current one) Signal-level reasoning Geometric search space

88 Three extensions Why not look at the whole picture? Signal-level past now future

89 Three extensions The combination can extend to N types of knowledge Geometric search space

90 Three observations The combination can extend to N types of knowledge RACE: Robustness by Autonomous Competence Enhancement [ Source: Mansouri & Pecora 2013 ]

91 Three observations [ Source: Konečný & Pecora 2013 ] Combined reasoning also needed for execution monitoring RACE: Robustness by Autonomous Competence Enhancement

92 Three extensions Human-centered AI Symbol System Human Perception Action Robot & Environment

93 Three extensions Human-centered AI Symbol System Human Perception Action Robot & Environment

94 Three extensions Human-centered AI Symbol System Human Perception Action Robot & Environment

95 Did we say humans? Human-centered AI a wider perspective what consequences for humans? what consequences for our environment? An example: our ethical stance We refuse collaborations with actors (companies, institutions, or funding organizations) involved in the development or use of military technology, or which engage in or promote armed conflict

96 Thank You! And thanks to: Julien Bidot Mathias Broxvall Marcello Cirillo Silvia Coradeschi Maurizio Di Rocco Cipriano Galindo Amy Loutfi Lars Karlsson Federico Pecora Ali Abdul Khaliq Pär Buschka Jasmin Grosinger Štefan Konečný Fabien Lagriffoul Kevin LeBlanc Robert Lundh Masoumeh Mansouri Andreas Persson Contact: