Learning the Proprioceptive and Acoustic Properties of Household Objects. Jivko Sinapov Willow Collaborators: Kaijen and Radu 6/24/2010

Transcription

1 Learning the Proprioceptive and Acoustic Properties of Household Objects Jivko Sinapov Willow Collaborators: Kaijen and Radu 6/24/2010

2 What is Proprioception? It is the sense that indicates whether the body is moving with required effort, as well as where the various parts of the body are located in relation to each other. - Wikipedia

3 Why Proprioception? 3

4 Why Proprioception? Full Empty 4

5 Why Proprioception? Hard vs Soft

6 Lifting: gravity, effort, etc.

7 Pushing: friction, mass, etc.

8 Squeezing: compliance, flexibility

9 Power, Play and Exploration in Children and Animals, 2000

10 Why Audio?

11 Why Audio? What actually happened: the robot dropped a can

12 The Importance of Natural Sound natural sound is as essential as visual information because sound tells us about things that we can't see, and it does so while our eyes are occupied elsewhere. Sounds are generated when materials interact, and the sounds tell us whether they are hitting, sliding, breaking, tearing, crumbling, or bouncing. Moreover, sounds differ according to the characteristics of the objects, according to their size, solidity, mass, tension, and material.

13 Sound Producing Event [Gaver, 1993]

14 Why should robots use natural sound and proprioceptive feedback? Human environments are cluttered with objects that generate sounds Help robot perceive events and objects outside of field of view Help robot perceive properties of objects that cannot be inferred by visual systems

15 Related Work: Proprioception Learning Haptic Representations of Objects : [ Natale et al (2004) ]

16 Related Work: Acoustics Material Recognition (5 Materials) Object Recognition (4 test objects) [Richmond and Pai, 2000] [Torres-Jara, Natale and Fitzpatrick, 2005]

17 Previous Work Interactive Object Recognition Using Proprioceptive Feedback Key Question: Can objects be recognized using only joint-torque feedback from manipulation experience? [IROS SPMM Workshop, 2009]

18 Exploratory Behaviors Lift: Crush: Shake: Push: Drop:

19 Video

20 50 household objects Different materials: metal, paper, plastic, wood, etc. Some objects have contents inside of them (e.g., pill bottle) All are graspable by the Barrett Hand

21 Object Recognition Pipeline r Joint-Torque Data Object Probability Estimates Dimensionality Reduction Discrete Proprioceptive Sequence Proprioceptive Recognition Model

22 Object Recognition Results

23 Acoustic Object Recognition Behavior Execution: WAV file recorded: Discrete Fourier Transform:

24 Frequency Bins Fourier Transform Time

25 Acoustic Object Recognition r Fourier Transform Object Probability Estimates Dimensionality Reduction Discrete Auditory Sequence Auditory Recognition Model

26 Cross-Modal Object Recognition Proprioception sequence Audio sequence Proprioceptive Recognition Model Auditory Recognition Model Weighted Combination

27

28 Application: Recognition of Dropped Objects

29 Summary of Previous Work Natural sound for object recognition (ICRA 2009) Natural sound can be used to detect an object s material and form object categories which group similar objects together (RSS MM workshop. 2009, ICDL 2010) Proprioceptive feedback can provide high object recognition accuracy when coupled with multiple behaviors applied on object (IROS SPMM 2009) Multiple exploratory behaviors are important for recognition ( AAAI 2010)

30 Perception Problem for PR2: Is the bottle full or empty?

31 General Approach Let the robot experience what full and empty bottles feel like Use prior experience to classify new bottles as either full or empty

32 Behavior: Power, Play and Exploration in Children and Animals, 2000

33 Data Representation Behavior Execution: Recorded Data: [J i, E i, C i ] Joint Positions Efforts Class Label {full, empty}

34 Training Procedure Objects: Procedure: Place object on table Robot grasps it and moves it in random 3D positions in space Robot puts object back down on table in random position; repeat. Training procedure is nearly autonomous

35 Classification Procedure Pr( empty ) Pr( full ) [J i, E i,?] Recognition Model Feature Extraction

36 Recognition Model X =[J i, E i,?] Recognition Model

37 Recognition Model X =[J i, E i,?] Recognition Model Find N closest neighbors to X in joint-feature space

38 Recognition Model X =[J i, E i,?] Recognition Model Find N closest neighbors to X in joint-feature space Train k-nn classifier C on the N neighbors that maps effort features to class label

39 Recognition Model X =[J i, E i,?] Recognition Model Find N closest neighbors to X in joint-feature space Train k-nn classifier C on the N neighbors that maps effort features to class label Use trained classifier C to label X

40 Application to Sorting Task Sorting task: Place empty bottles in trash Move full bottles on other side of table

41 Application to Sorting Task

42 Application to Sorting Task

43 Application to Sorting Task Ask me for demo

44 Classification Results Cross-Validation Accuracy with a single measurement: ~92 % (estimated with 800 training data points) Classification accuracy during sorting task with 5 or more measurements: 100% (estimated from 20 individual runs)

45 Can the same method be used to directly estimate the bottle s weight?

46 Can the same method be used to directly estimate the bottle s weight? Variable number of marbles in each bottle, from 0 to 80, at increments of 10

47 Can the same method be used to directly estimate the bottle s weight? Variable number of marbles in each bottle, from 0 to 80, at increments of 10 Mean abs. error: lbs ~7 marbles

48 Can the same method be used to directly estimate the bottle s weight? Variable number of marbles in each bottle, from 0 to 80, at increments of 10 Mean abs. error: lbs ~7 marbles

49 Applications to other tasks e.g., pushing a box to find out if it s full

50 Can auditory feedback be used to distinguish between empty and full bottles?

51 Can auditory feedback be used to distinguish between empty and full bottles? The sound generated when a bottle is dropped is dependent on whether the bottle is full or not

52 Can auditory feedback be used to distinguish between empty and full bottles? Empty Bottle Full Bottle

53 Auditory Event Segmentation Segmentation Event Tokens

54 Auditory Event Feature Extraction

55 Auditory Event Feature Extraction

56 Feature Extraction Auditory Event Features in R 18

57 Auditory Event Recognition Feature Extraction Auditory Recognition Model Pr( empty ) Pr( full )

58 Auditory Classification Results 5 drops of an empty bottle and 5 of a full bottle 24 auditory events w/ empty bottles and 13 w/ full Recognition Accuracy for individual events: k-nn: % SVM: 100 %

59 Conclusion The auditory and proprioceptive modalities can be useful for tasks that cannot be solved through vision Objects properties can be learned over the course of manipulation

60 Software Contributions Demo: sorting bottles Runs on cturtle Uses grasping pipeline for pick and place

61 Software Contributions Demo: sorting bottles Runs on cturtle Uses grasping pipeline for pick and place Package proprioception: Service for capturing and classifying proprioceptive feedback

62 Software Contributions Demo: sorting bottles Runs on cturtle Uses grasping pipeline for pick and place Package proprioception: Service for capturing and classifying proprioceptive feedback Package audio_classification: service that captures microphone input; computes Fast Fourier Transform, and extracts features for input to classifiers

63 Software Contributions Demo: sorting bottles Runs on cturtle Uses grasping pipeline for pick and place Package proprioception: Service for capturing and classifying proprioceptive feedback Package audio_classification: service that captures microphone input; computes Fast Fourier Transform, and extracts features for input to classifiers Service for general purpose classification and recognition on the fly

64 Future Work Apply proprioceptive and auditory recognition models to other types of objects: e.g., push a box to check if it s full shake an object to check if something is inside Generalize sorting task to other object types: e.g., sort by material type Implement routines for autonomous object exploration

65 Thank you 6 5