Perception. Read: AIMA Chapter 24 & Chapter HW#8 due today. Vision

Transcription

1

2 Perception Vision Read: AIMA Chapter 24 & Chapter 25.3 HW#8 due today

3 visual aural haptic & tactile vestibular (balance: equilibrium, acceleration, and orientation wrt gravity) olfactory taste magnetoreception others?

4 Body with 25 degrees of freedom (DOF) whose key elements are electric motors and actuators Sensor network, including 2 cameras, 4 microphones, sonar rangefinder, 2 IR emitters and receivers, 1 inertial board, 9 tactile sensors, and 8 pressure sensors Various communication devices, including voice synthesizer, LED lights, and 2 high-fidelity speakers Intel ATOM 1,6ghz CPU (located in the head) that runs a Linux kernel and supports Aldebaran s proprietary middleware (NAOqi) Second CPU (located in the torso) 27,6-watt-hour battery that provides NAO with 1.5 or more hours of autonomy, depending on usage

5 NAO has two cameras and can track, learn, and recognize images and faces. NAO sees using two 920p cameras, which can capture up to 30 images per second. The first camera, located on NAO s forehead, scans the horizon, while the second located at mouth level scans the immediate surroundings. The software lets you recover photos and video streams of what NAO sees. But eyes are only useful if you can interpret what you see. That s why NAO contains a set of algorithms for detecting and recognizing faces and shapes. NAO can recognize who is talking to it or find a ball or, eventually, more complex objects.

6 NAO is equipped with two sonar channels: two transmitters and two receivers. They allow NAO to estimate the distances to obstacles in its environment. The detection range is 0 70 cm. Less than 15 cm, there is no distance information; NAO only knows that an object is present.

7 Omnidirectional walking NAO's walking uses a simple dynamic model (linear inverse pendulum) and quadratic programming. It is stabilized using feedback from joint sensors. This makes walking robust and resistant to small disturbances, and torso oscillations in the frontal and lateral planes are absorbed. NAO can walk on a variety of floor surfaces, such as carpeted, tiled, and wooden floors. NAO can transition between these surfaces while walking. Whole body motion NAO's motion module is based on generalized inverse kinematics, which handles Cartesian coordinates, joint control, balance, redundancy, and task priority. This means that when asking NAO to extend its arm, it bends over because its arms and leg joints are taken into account. NAO will stop its movement to maintain balance.

8 Fall Manager The Fall Manager protects NAO when it falls. Its main function is to detect when NAO's center of mass (CoM) shifts outside the support polygon. The support polygon is determined by the position of the foot or feet in contact with the ground. When a fall is detected, all motion tasks are killed and, depending on the direction, NAO's arms assume protective positioning, the CoM is lowered, and robot stiffness is reduced to zero.

9 Audio NAO uses four microphones to track sounds, and its voice recognition and text-to-speech capabilities allow it to communicate in 8 languages.

10 Sound Source Localization One of the main purposes of humanoid robots is to interact with people. Sound localization allows a robot to identify the direction of sounds. To produce robust and useful outputs while meeting CPU and memory requirements, NAO sound source localization is based on an approach known as Time Difference of Arrival. When a nearby source emits a sound, each of NAO s four microphones receives the sound wave at slightly different times.

11 Sound Source Localization, continued For example, if someone talks to NAO on its left side, the corresponding sound wave first hits the left microphones, then the front and rear microphones a few milliseconds later, and finally the right microphone. These differences, known as interaural time difference (ITD), can then be mathematically processed to determine the current location of the emitting source. By solving the equation every time it hears a sound, NAO can determine the direction of the emitting source (azimuthal and elevation angles) from ITDs between the four microphones.

12 Signal Processing In robotics, embedded processors have limited computational power, making it useful to perform some calculations remotely on a desktop computer or server. This is especially true for audio signal processing; for example, speech recognition often takes place more efficiently, faster, and more accurately on a remote processor. Most modern smartphones process voice recognition remotely. Users may write their own signal processing algorithms directly in the robot.

13

14 to know what is where, by looking. (Marr) What are the characteristics of visual perception in people? What are the key open issues in computer vision?

15 Vision is a Constructive Process Your conscious perception of the visible world is an illusion manufactured by your brain (at great cost). Examples: brightness, color, and size constancy Vision Solves Specific Tasks in Specific Contexts Generality in visual skills tied directly to needs and context.

16 Vision is inferential: Light

17 Checker Shadow

18 Pixel color strongly affected by illumination. Perception of color constancy maintained by the brain (Images courtesy of David Heeger) Sunlight Fluorescent light

19 Object size vs. object depth (Images copyright John H. Kranz, 1999)

20 In each of the previous examples, the brain s goal is to maintain invariance to environmental effects: The amount and color of light in the scene The distance between the observer and an object of interest. How to achieve this invariance computationally? All other sensory experiences, such as audition and touch, are also the result of a constructive process.

21 question: Which aspects of the rich visual stimulus should be considered to help the agent make good action choices, and which aspects should be ignored? approaches 1. feature extraction computations applied directly to the sensor observations 2. recognition each agent distinguishes among the objects it encounters based on visual and other information 3. reconstruction agent builds a geometric model of the world from an image or set of images

22 edge detection texture analysis computation of optical flow (video sequence)

23 process of breaking an image into regions of similar pixels (a) Original image (b) Boundary contours (c) & (d) Segmentation into regions

24 Geometry-based Objects and images modeled as set of point/surface/volume elements Example real-time method: store geometric relationships in hash table Appearance-based Objects and images modeled as set of features closer to raw image Example real-time method: use histograms of simple features (e.g. color)

25 sources of appearance variation

26 (a) object models: images of a shoe and a telephone (b) a test image (c) the shoe and telephone have been detected by: finding points in the image whose features match a model

27 Object recognition in its full generality is a very hard problem. There are brightness-based and feature-based approaches Other possibilities? poking (i.e. experimenting) cf. Paul Fitzpatrick, formerly of MIT CSAIL, currently at Robot Rebuilt

28 Robot: We have got to find out where this object s boundary is. Camera: How are we going to do that? There's no way. Robot: Well there is one way. Looks reachable I say, let s poke it. (brandishes the Giant Poking Limb)

29 object segmentation poking affordance exploitation (rolling) edge catalog object detection (recognition, localization, contact-free segmentation) manipulator detection (robot, human)

30 Learning from an activity Poking: to learn to recognize objects, manipulators, etc. Chatting: to learn the names of objects Learning a new activity Searching for an object Then back to learning from the activity