Learning to Order Objects using Haptic and Proprioceptive Exploratory Behaviors

Transcription

1 Learning to Order Objects using Haptic and Proprioceptive Exploratory Behaviors Jivko Sinapov, Priyanka Khante, Maxwell Svetlik, and Peter Stone Department of Computer Science University of Texas at Austin, Austin TX 78712, USA {jsinapov,pkhante,maxwell,

2 Building-Wide Intelligence Project: 2

3 Building-Wide Intelligence Project: 3

4 Building-Wide Intelligence Project: 4

5 Motivation: Grounded Language Learning Robot, fetch me the green empty bottle 5

6 Object Category Recognition in Robotics Sridharan et al Rusu et al Collet et al Lai et al

7 Object Category Learning in Robotics Thomason, J., Sinapov, J., Svetlik, M., Stone, P., and Mooney, R. (2016). Learning Multi-Modal Grounded Linguistic Semantics by Playing I, Spy Robotics and Vision 3 Session 7

8 Now, when and where does this fail... Consider the word, weight - how should it be grounded? 8

9 How do humans ground such words? Sample Montessori toys designed to teach children about the ordinal properties of object weight, height, and size 9

10 Object Ordering in Psychology 10

11 Object Orderings in Human Environments 11

12 Problem Formulation Order Recognition: what property is a given series of objects ordered by? height width 12

13 Problem Formulation (2) Order Insertion: given an object series, insert a new object into the correct position series test object 13

14 Three-Stage Approach Stage 1: Object Exploration Stage 2: Unsupervised Order Discovery... Stage 3: Semantic Grounding... weight width height 14

15 Stage 1: Object Exploration 32 common household and office items The objects vary along three ordinal properties: 1) Weight 2) Width 3) Height 16

16 Exploratory Behaviors 17

17 Video 18

18 Video 19

19 Video 20

20 Haptic and Proprioceptive Feature Extraction... Joint Positions (Prorioception) Joint Efforts (Haptics)... Time 21

21 Haptic and Proprioceptive Feature Extraction... Joint Positions (Prorioception) Joint Efforts (Haptics)... Time 22

22 Haptic and Proprioceptive Feature Extraction... Joint Positions (Prorioception) Joint Efforts (Haptics)... Time 23

23 Stage 2: Unsupervised Order Discovery Sensory Modalities haptics proprioception grasp Behaviors lift hold lower drop push press 24

24 Unsupervised Order Discovery Example with Synthetic Data Input Relational Count Matrix Object order with highest likelihood using the method of [Kemp and Tennenbam, 2008] 26

25 Example Relational Count Matrix with the Press action and Haptic features Similarity between objects i and j in the press-haptic sensorimotor context 27

26 Resulting Order (Press behavior and Haptic modality) The number corresponds to the object's height in millimeters 28

27 Stage 2: Unsupervised Order Discovery Sensory Modalities haptics proprioception grasp Behaviors lift hold lower drop push press 29

28 Stage 3: Order Grounding Stage 30

29 Order Grounding Example: height Positive Examples: Negative Examples:

30 Object Order Representation Training Example:... Object Orders Discovered During Stage 2 32

31 Object Order Representation Training Example:... Object Orders Discovered During Stage 2 33

32 Object Order Representation Training Example:... Object Orders Discovered During Stage 2 34

33 Object Order Representation Training Example: x1... Object Orders Discovered During Stage 2 35

34 Object Order Representation Training Example: x1... Object Orders Discovered During Stage 2 36

35 Object Order Representation Training Example: x1 x2... Object Orders Discovered During Stage 2 37

36 Object Order Representation Training Example: x1 x2... xn... Object Orders Discovered During Stage 2 38

37 Results: Order Recognition 39

38 Sample Learned Decision Trees weight Hold Haptics Lower Haptics Lift width height Grasp Proprioception Press Proprioception Press Haptics Haptics 40

39 When does the robot make mistakes? 41

40 When does the robot make mistakes? difficult easy 42

41 When does the robot make mistakes? difficult easy 43

42 Object Order Insertion Results 44

43 Object Order Insertion Results 45

44 Object Order Insertion Results 46

45 Object Order Insertion Results 47

46 Conclusion A behavior-grounded framework for learning object ordering concepts The robot grounded three ordering concepts, weight, height, and width Future Work: Active action selection Learn object ordering concepts in conjunction with object categories, pairwise object relations, etc. Learn from humans (for a preview, see our next talk at Robotics and Vision III) 48

47 Thank you! Jivko Sinapov Priyanka Khante Maxwell Svetlik Peter Stone 49