Knowledge-Sharing Techniques for Egocentric Navigation *

Transcription

1 Knowledge-Sharing Techniques for Egocentric Navigation * Turker Keskinpala, D. Mitchell Wilkes, Kazuhiko Kawamura A. Bugra Koku Center for Intelligent Systems Mechanical Engineering Dept. Vanderbilt University Middle East Technical University Nashville, TN Ankara, Turker turker.keskinpala@vanderbilt.edu bugra@bugra.net {wilkes, kawamura}@vanderbilt.edu Abstract Teams of humans and robots working together can provide effective solutions to problems. In such applications, effective human-robot teaming relies on being able to communicate information about the current perception or understanding of the environment. In this paper, human-robot teaming on navigational tasks is discussed. The role of the human user will be to specify the goal point(s) for the robot, and also to interact with the robot in the event of perceptual errors. A novel navigation method, Egocentric Navigation (ENav), has been developed based on egocentric representations. In this paper, two knowledge sharing methods are described which exploit the characteristics of ENav. In the first method, robots share Landmark EgoSpheres, and in the second method the robot shares it Sensory EgoSphere with the human user for visual perception correction.. Keywords: Human-robot teaming, Egocentric Navigation, Sensory EgoSphere, Landmark EgoSphere. 1 Introduction Recently, the concept of humans and robots working together in cooperative teams has beeen receiving greater attention. In such an application, communicating information about the current perception or understanding of the environment is very important for effective humanrobot teaming. Since navigation is fundamental to mobile robotics, in this paper we consider human-robot teaming on navigational tasks. The role of the human user will be to specify the goal point, or points, for the robot, and also to interact with the robot in the event of perceptual errors. At the heart of this problem are the representations of the state of the world that are used by the human user and the robots in the team. Representation plays a very important role in perceptual knowledge sharing in a robot-robot and a robothuman team. At the Intelligent Robotics Laboratory at Vanderbilt University, we have been using egocentric * /03/$ IEEE. representations for the perceptual knowledge of the robot. One egocentric representation is called the Sensory EgoSphere (SES). The second representation is called the Landmark EgoSphere (LES). The LES contains information about the angular distribution of the landmarks that the robot expects to see at a goal position, and is similar to the SES in structure [5]. For navigational tasks we often use a simplified EgoSphere structure, in which the original 3-D shape of the EgoSphere is projected down to a 2-D structure. In this work, the SES is used to represent the robot s perception of its environment at its current state and the LES is used to represent the robot s expected perception of its environment at a target position that the robot is trying to reach. The SES is a result of robot s perception. The LES may be obtained in several ways. For example, it may be the result of the robot s perception on a previous navigational mission, the result of another robot s perception at the target point, or it may be derived from a rough map of the area. In this paper, in association with our Egocentric Navigation algorithm (ENav), we describe two methods that demonstrate the communication of perceptual knowledge, represented by the SES and the LES, among humans and robots. The first method will be a knowledge sharing method in a team of two heterogeneous robots with the emphasis given on sharing LES information between the robots. The second method will be a visual perception correction method where the robot shares its understanding of the environment, via its current SES, with the human so that the human may correct possible misperceptions by the robot. Our approach takes inspiration from the qualitative navigation method of Levitt and Lawton [7], Dai and Lawton [3], and Sutherland and Thompson [11]. Additional inspiration comes from the work of Pinette [8] and Cartwright and Collet [2]. This paper is organized as follows: In the next section the importance of the choice of representation will be discussed along with some reasons why egocentric representations were chosen over Cartesian representations. In the third section, the egosphere concept will be

2 described. The fourth section will briefly introduce the Egocentric Navigation algorithm. The knowledge sharing methods in robot-robot and human-robot teams will be described in the fifth section. Finally, future work and conclusions will follow. 2 Representation As humans, when we try to remember a place, we typically remember the geometric relationships between the objects in the environment more naturally than the metric relationships between them. Recent studies suggest that humans prefer angular information over distance information in the context of learning places and performing self-localization [4]. Such angular information is explicitly shown in egocentric representations. Research by Scholl [10] revealed that the cognitive spatial representations employed by human beings produce environment-centered representations of the spatial relations between objects. The environment-centered representations are derived from the sequence of local egocentric views that the human being experiences while moving through the environment. Cartesian representations require that the more natural egocentric information be transformed into Cartesian coordinates, thus making the representation less natural. When knowledge sharing between robot-robot and human-robot teams is considered, having a natural representation that can be understood both by the robots and the humans facilitates effective communication. Sharing a robot s raw sensory data in a heterogeneous team would not make sense to the other robots. Similarly, the human would not understand the raw sensory data of the robot, nor would the robot understand the sensory data sent to the human brain. For this reason, it is important to use a representation that makes sense to any member of the team. We chose egocentric representations over Cartesian representations, since a robot can easily form an egocentric representation of its surrondings using its sensors, and humans can naturally create and understand egocentric representations. In addition, egocentric representations are directly exploited in the Egocentric Navigation algorithm (Enav). 3 Egospheres There are two egocentric representations that are used in this work. One egocentric representation is called the Sensory Egosphere (SES). Albus proposed the egosphere in He envisioned it as a dense map of the visual world, a virtual spherical shell surrounding the robot onto which a snapshot of the world was projected [1]. Our definition and use of the egosphere differs from that of Albus. We call it the Sensory EgoSphere and define it as a database a spherical data structure, centered on the coordinate frame of the robot that is spatially indexed by azimuth and elevation [9]. Its implicit topological structure is that of a Geodesic Dome, each node of which is a pointer to a distinct data structure. The SES is a sparse map of the world that contains pointers to descriptors of objects or events that have been detected recently by the robot. As the robot operates within its environment, events, both external and internal, stimulate the robot s sensors. Upon receiving a stimulus the associated sensory processing module writes its output data (including the time of detection) to the SES at the node that is closest to the direction from which the stimulus arrived. Since the robot s sensory processing modules are independent and concurrent, multiple sensors stimulated by the same event will register the event to the SES at about the same time. If the event is directional, the different modules will write their data at the same location on the SES. Hence, sensory data of different modalities coming from similar directions at similar times will register close to each other on the SES. Given that the sensors on a robot are discrete with regard to angular positioning, there is nothing to be gained by defining the SES to be a continuous structure. Moreover, the computational complexity of using the SES increases with its size which is, in turn, dependent on its density (number of points on its surface). We use a (virtual) geodesic dome structure for the SES since it provides a uniform tessellation of vertices such that each vertex is equidistant (along geodesics) to six neighbors. The tessellation frequency is determined by the highest angular resolution of the SONAR array. The SES is a multiple-linked list of pointers to data structures. There is one pointer for each vertex on the dome. Each pointer record has seven links, one to each of its six nearest neighbors and one to a tagged-format data structure. The latter comprises a terminated list of alphanumeric tags each followed by a time stamp and another pointer. A tag indicates that a specific type of sensory data is stored at the vertex. The corresponding time stamp indicates when the data was stored. The pointer associated with the tag points to the location of a data object that contains the sensory data and any function specifications (such as links to other agents) associated with it. The type and number of tags on any vertex of the dome is completely variable. Often in practice, the SES is not a complete geodesic dome, instead it is restricted to only those vertices that fall within the directional sensory field of the robot. Imagery or image features can be stored at the vertex closest to the direction of the object identified in the image as shown in Figure 1. SONAR and LIDAR work only in the equatorial plane of our robot and so their data is restricted to the vertices near the dome's equator [9]. For navigational tasks we have used a simplified egosphere structure, in which the original 3-D shape of the egosphere is projected down to a simpler 2-D structure, i.e., a circle. In this paper, the SES is used to represent the robot s perception of its environment at its current state.

3 on the current perception of the robot without explicitly requiring any range information. In the absence of range information, the robot uses the SES to represent its current perception by using the angular separation between perceived objects. Figure 3 shows sample egocentric representations. Target Current Figure 1. Geodesic dome around the robot and mapping of objects to egosphere [5] The second representation that is used in this work is called the Landmark Egosphere (LES). The LES is a robocentric representation of environmental features expected at a target position [5]. In other words, the LES contains information about the angular distribution of the landmarks that the robot expects to see at a goal position, and is similar to the SES in structure [5]. The LES can be extracted from a map of the area, or it can be derived from some other description of the region. Furthermore, a previously acquired SES stored in the memory of any robot in a team of robots can be used as an LES. In order for the robot to form egocentric representations, we needed to get 360 vision from the robot. For this purpose, we designed a vision system consisting of seven cameras mounted in a ring formation, and a 8-by-1 video multiplexer. The multiplexer were used to switch between seven cameras and we processed the images from each camera one by one to for the egocentric representation of the robot s environment. Figure 2 shows the vision system mounted on the Pioneer 2 AT robot used in our experiments. Figure 2. Vision system on the Pioneer 2 AT robot 4 Egocentric Navigation Egocentric navigation (ENav) is a basic navigational behavior designed to operate based only on egocentric representations, namely, the SES and the LES [6]. The ENav algorithm moves the robot to a target location based Figure 3. Sample egocentric representations. The current representation in sensory data is labeled as an SES, the target representation is labeled as an LES Navigation depends heavily on perception since the goal is to move the robot to a location where the perception of the robot closely matches a target representation defined by LES [6]. The navigation algorithm takes an SES and an LES as input and compares the SES created by the perception process with the LES. This comparison results in an error term that represents the difference between the current SES and the target LES. A heading is computed using the SES and the LES depending on the error s being satisfactorily above a threshold. As the robot iteratively moves along this heading, it is taken towards the target point at which the error drops below the threshold [6]. Error and heading computations are based on pair-wise analysis of landmarks. First, landmarks not common to both SES and LES are removed. Next, from the common landmarks in both representations, landmark pairs are formed and these pairs are compared between the LES and SES. In this comparison, the smaller magnitude angle between two landmarks is used to define the separation between these landmarks. A unit vector along the bisector of these two landmarks is created by each pair depending on this comparison. The unit vector can either be toward or away from the landmarks. Finally, the final heading vector for the situation defined by the SES and LES pair is created by adding all the resulting unit vectors. ENav algorithm can also be described by vector algebra. First, a unit vector is created pointing to every landmark on the SES and the LES. The landmarks on the SES are represented with unit vectors u c i (the superscript C indicates current perception) and the landmarks on the LES are represented with unit vectors u t i as shown in Figure 4 (the superscript t indicates a target location). Variables i and j index over individual landmarks in

4 distinct pairs formed from the SES and LES. The pair-wise vector analysis is carried out as follows [5]: dc ij = uc i. uc j dt ij = ut i. ut j Cij= uc i x uc j, where i j Tij= ut i x ut j Aij = sgn(dcij dtij) (1) Bij = [sgn(cij. Tij) + 1] / 2 (2) uij = (1 + Bij(Aij -1) )(uci + ucj / uci + ucj ) (3) h = Σ uij (4) Figure 4. Computation of heading based on SES and LES [5] There is a relationship between ENav, EgoSpheres and memory models. The navigation system is composed of local and global paradigms. Local navigation uses only the information within the immediate sensing region of the robot, and is reactive. On the other hand, global navigation is deliberative and uses information beyond the robot s sensory horizon. Dividing the system this way implicitly organizes the robot s memory into long-term and shortterm memory structures. The SES provides short-term memory (STM) for reactive navigation. On the other hand, long-term memory (LTM) contains global layout information and supports global navigation. In ENav, landmarks assumed to be around the robot are represented on the LES, which is a LTM structure. Within the ENav scheme, there is also a task-specific memory module which is task-dependent and holds descriptors of via regions that point to transition points for navigation. [5]. 5 Knowledge Sharing In this work, knowledge sharing techniques for ENav are studied. Knowledge sharing is important when robotrobot and human-robot teaming is considered, since some tasks require cooperation between the members of the team. We address both cases where perceptual knowledge is shared in robot-robot and human-robot teams. 5.1 Knowledge Sharing in a Robot-Robot Team Knowledge sharing in mobile robots is an important aspect of mobile robot research. In a multi-robot team, some tasks require cooperation between robots which may be in the form of knowledge sharing. The content of the knowledge shared depends on the task that the robots must perform. However, the nature of the knowledge that is shared is also important. In a heterogeneous robot team, sharing one robot s raw sensory data would not make sense to the other robots. For this reason, the representation plays an important role for knowledge sharing in a robot team. The shared knowledge should be in a form that can be understood by all the robots in the team. Thus it should be at a higher level of abstraction than raw sensory data. The LES is a very suitable representation that can be shared among robots, because it is a natural representation of the objects in the robot s environment. As previously mentioned, there may be various sources for the LES. It can be extracted from an a priori map, can be provided to the robot by a human user, or it can be provided to the robot by another robot in the team. In this work, effort was made to have two heterogeneous robots share LES information enabling one robot to navigate by ENav. ENav requires that two representations be present in order for the robot to navigate. In this case, the navigating robot creates its own SES using its perception capabilities, and receives a target LES from the other robot in the team who has the knowledge of the target location. For example, the other robot may have been at the target location previously and can use the SES it formed at that location as a target LES for the navigating robot. In general, sharing this knowledge in a robot team is analogous to knowledge sharing in humans. Consider the case where Person A needs to meet with Person B at a location that Person A has never visited before. In this case, Person A may call Person B from his cell phone and ask for directions. Knowing the starting region where Person A is located, Person B can give qualitative descriptions to direct Person A to his region. Using these descriptions Person A can find Person B using his own senses along the way and by comparing what he sees with what Person B described and deciding which way to go. In this analogy, the interaction between Persons A and B is knowledge sharing. In the robotics sense, Robot A can find Robot B if Robot B is in a region to which Robot A does not know how to navigate. Knowing which region Robot A is in, Robot B may provide qualitative descriptions that will help Robot A to navigate to its region. These qualitative descriptions are LESs that can be generated from Robot B s long-term memory. Robot A can then use ENav to reach Robot B. Robot A reaches Robot B by using Robot A s own sensors to create its own perceptual knowledge, and uses the LES knowledge transferred from Robot B to

5 decide on its heading. It must be noted that Robot B does not need to know the precise position of Robot A since no heading calculation is made that explicitly depends on the relative positions of the robots. The ENav algorithm generates the heading directions for Robot A. This knowledge sharing method can be further analyzed using the three memory structures used in ENav. Short-term memory holds robo-centric topological regions (SES), long-term memory holds global layout information, and task specific memory holds robo-centric topological regions in terms of LES representations that indicate transition points, i.e., a sequence of waypoints, for navigation. In this sense, what is really shared between the robots for one robot to navigate by using the knowledge of the other one is task specific memory. R A (actually a Pioneer AT2 robot called Skeeter) does not know how to go the region where R B (a RWI ATRV-Jr. robot called Scooter) is. R B has knowledge of the environment and can generate a sequence of waypoints, described by LESs, to help R A navigate using ENav. To start ENav, Skeeter is able to use its own vision sensors to create an SES. However, since it does not know the environment, it cannot create a target LES. It asks Scooter for an LES. Scooter sends a waypont LES that describes the location L 1. Upon receiving this LES, Skeeter internalizes that LES to use it in ENAV and starts navigating. Skeeter computes its own heading and decides its direction by itself. After Skeeter decides that it has arrived at the LES location, it asks for the next LES. At this point, Scooter sends the second waypoint LES that describes location L 2. Once again, Skeeter navigates creating a chain of SESs and comparing them to the LES for L 2. When Skeeter decides that it has arrived at the second LES, it asks for another LES. This continues until Scooter sends the target LES to Skeeter and notifies Skeeter that this is the final target. Skeeter can ask for as many LESs as are necessary to reach to target. For the LES sharing method to work, the robots need not see each other, and need not have the same absolute heading because no heading computation is done relative to the robot positions. In addition, robots can be positioned in the environment without any constraints except that they must be able to sense at least one landmark. Green Cone Pink Cone Pink Cone Blue Cone Yellow Cone Blue Cone 5.2 Knowledge Sharing in Human-Robot Team As in knowledge sharing in a robot-robot team, representation is also important in knowledge sharing in a human-robot team. It is potentially even more difficult to have human and robot understand each other. This is due to the fact that many robots use knowledge representations that are very unfamiliar for humans. Since humans prefer angular information over distance information while learning places and localizing themselves [4], an egocentric representation is a natural representation for humans. Our robots are also capable of representing their environmental knowledge in an egocentric structure. This suggested that the robot could share its egocentric representation with the human user. Green Cone Yellow Cone Figure 5. Illustration of LES sharing method Figure 5 illustrates how this method works. First, it has to be kept in mind that a robot should have a goal state, and current state of its environment to use ENav. That is, the robot should have an LES and an SES to start navigating with ENav. In the case illustrated in Figure 5, As pointed in Section 4, ENav is highly dependent on perception since the goal is to move the robot to a location where the perception of the robot closely matches a target representation defined by an LES. However, the robot s perception of the objects around it is not perfect. Sometimes the robot does not detect the landmarks around it correctly, potentially affecting the ENav algorithm s heading computation and, as a result, affecting the performance of the navigation system. In addition to trying to solve the problems in the robot s perceptual systems, we allowed for the inclusion of the human user in the decision

6 loop of the robot during SES creation. This way, the system was able to take advantage of the superior visual perception capabilities of the human team member Perception System and Perception Problems Before going into the details of the problems with perception and this method, it is useful to describe the perception system we used. We developed a visual attention, or saliency, algorithm based on defining the concept of bright colors in the Hue, Saturation and Value (HSV) color space. This method is promising for a limited number of landmarks. Since we use color to detect landmarks, both single-color or multi-color landmarks can be used. However, using single-color landmarks has some disadvantages such as limiting the number of unique landmarks with the number of colors that can robustly be detected in the environment. Moreover, the environment may have colors that are more salient than the objects used as landmarks [5]. We adopted a multi-color landmark structure in our perception system. In this structure, two colors and a unit vector pointing from the first color to the second color defines a pair. Figure 6 shows basic color pair structure used in forming multi-color landmarks, and Figure 7 shows the sample color patterns used [5]. y x Green Pink Figure 6. Basic color pair structure used in forming multicolor landmarks: green_pink_1,0 counterparts in the multicolor landmark affect the correct detection of landmarks. A false positive is the case where a non-existent landmark is detected in the environment, or a different landmark is detected instead of the one in the scene. If single color landmarks are used, the probability of false positives is higher than for multi-color landmarks. The perception system detects the most salient color in the environment as a landmark. A three color landmark reduces the occurrence of false positives, since, for something to be detected as a landmark three colors must be detected and the color pairs should agree with the structure of the landmark Perception Correction by Human Intervention In order to address the visual perception problems originating from the robot s relatively poor visual perception capability, we included the human user in the loop during SES creation. We developed a visual perception correction system, and an interface that enables the human user to correct the perception mistakes made by the robot during the SES creation process. In order to navigate using ENav, the robot creates SES and compares this to the target LES. In the perception correction method, the robot creates an SES and before using this SES for navigation, it shares the SES with the user for confirmation. The robot s SES is presented to the user in the perception correction interface where each camera frame is positioned as shown in Figure 8. If anything is detected on any frame, an icon of the detected landmark is placed above the corresponding frame at the position where the landmark was detected. This is illustrated in Figure Horizontal Bar L Vertical Bar Figure 8. Position of camera frames on the interface Figure 7. Sample color patterns used in multi-color landmarks Although this visual attention system has the potential to reduce the occurrence of perception problems associated with dynamic light conditions, perception problems still occur. The problems with visual perception can be categorized in two groups: False negatives and false positives. A false negative means that the robot cannot detect a landmark that is in the scene. This situation occurs because of several factors. First, since the perception system depends on color, changing lighting conditions can affect detection performance. Second, the presence of colors in the environment that appear to be more salient than their Figure 9. Perception Correction Interface after SES creation As can be seen in Figure 9, the robot was able to detect two landmarks in frames 1, and 7. These two landmarks were posted on the SES as shown in Figure 10. From the perception correction interface, the human user can see that the robot was not able to detect some of the landmarks that were visible to it. Using the functionality provided with the interface, the human user corrects the

7 current SES of the robot by adding two of the landmarks seen in frame 2. To do this, the user opens up the Object List by clicking the button on the interface and selects the landmark that is to be added. As selected, the landmark icon appears in the upper left corner of the screen. Then, the user drags and drops the icon on the landmark in the frame. After this operation, the landmark icon is placed on the frame indicating a detected landmark. This illustrates correction of false negatives using this method. Figure 11 shows the Object List, the robot s SES and the perception correction interface after two corrections made by the human user. Figure 10. 2D SES view of the robot detected landmarks are removed from the SES. After correction by the human user, the robot uses the new SES for navigation. 6 Future Work In future work, the LES sharing method can be extended to LES and SES sharing where the robot receiving the LES can send its SES back to the other robot. In this way the robot sending the LES can localize the other robot on its map and may forward it to different targets by sending appropriate LESs. In addition, a backtracking task can be assigned to the robot receiving the sequence of LESs. The robot can perform this task by learning the LES sequence that it received and use this to find its way back to it s starting point by running through this sequence backwards. The human-robot interaction aspect of the knowledge sharing in a human-robot team can also be improved. In our ongoing work, the robot is more involved in the interaction process. This work can lead to a mixed initiative vision system where the human and robot carry out a more intelligent dialog-based interaction. In our approach the robot can be trained to recognize a large set of features extracted from regions in an image. By constraining the features to have clearly identifiable semantic meaning, the robot and human have with a shared semantic context for discussing these features. 7 Conclusion In this paper, two knowledge sharing methods based on ENav were presented. ENav was defined as a basic navigational behavior based only on egocentric representations. Egocentric representations are crucial for both the navigation and the knowledge sharing methods that we developed. The egocentric representations, the SES and LES, represent the objects around the robot by only using their angular distributions relative to the robot. The SES is used to represent the robot s perception of its environment at its current state, while the LES represents the expected state of the world at a target location. As a qualitative navigation method, ENav, without explicitly requiring range information, provides a basic navigation method where a heading is computed by comparing the two egocentric representations. Figure 11. Object List, the robot s corrected SES and perception correction interface after the correction The interface can also be used to correct a false positive. In this case, the falsely detected landmark icons above the associated frames can be dragged and dropped on the trash can icon on the interface. This change is immediately reflected on the robot s SES and the falsely Two knowledge sharing methods were described. In the first method, it was shown that LESs can be shared in a robot-robot team, where one robot navigates using LESs provided by another robot in the team while the navigating robot uses its own visual perception sensors to create the required SESs. In the second method, it was shown that the robot could share its SES with the human user in order for the human to check and correct the SES created by the robot. Using the proposed perception correction interface, the user was able to monitor the SES creation of the robot,

8 and correct false negatives and false positives. After correction by the user, the robot used the updated SES and navigated using ENav. In conclusion, the success of the two knowledge sharing methods we demonstrated showed that it is advantageous to use egocentric representations and sensory features that are understood by both humans and robots, when knowledge sharing in robot-robot and human-robot teams is concerned. Geoforum (Special Issue on Geography, Environment and Cognition), Vol 23 No. 2, pp , [11] K.T. Sutherland and W.B. Thompson, Inexact navigation, Proc. of the IEEE Int. Conf. on Robotics and Automation, Atlanta, GA, pp. 1-7, Acknowledgments This research has been partially funded under a grant from DARPA (Grant #DASG ) and under Grant #DASG References [1] J.S Albus, Outline for a Theory of Intelligence, IEEE Transactions on Systems, Man and Cybernetics, Vol 21, pp , [2] B.A Cartwright and T. S. Collett, Landmark Learning in Bees, Journal of Comparative Physiology, Vol 151, pp , 1983 [3] D. Dai and D.T. Lawton, Range-free qualitative navigation, Proc. of the IEEE Int. Conf. on Robotics and Automation, Atlanta, GA, pp , [4] S. Healy, Spatial Representation in Animals, Oxford University Press, pp. 6-8, [5] K. Kawamura, A.B. Koku, D.M. Wilkes, R.A. Peters II and A. Sekmen, Toward Egocentric Navigation, Int. J. of Robotics and Automation, vol. 17, no. 4, pp , [6] A.B. Koku, Egocentric Navigation and Its Applications, Ph.D. Dissertation, May [7] T.S. Levitt, & D.T. Lawton, Qualitative navigation for mobile robots, Artificial Intelligence, Vol. 44, pp , [8] B. Pinette, Qualitative Homing, Proc. of IEEE International Symposium on Intelligent Control, Alexandria, VA, pp , [9] R.A. Peters II, K.E. Hambuchen, and K. Kawamura, The Sensory EgoSphere as a Short-Term Memory for Humanoids, Proc. of the IEEE-RAS International Conference on Humanoid Robots, Waseda University, Tokyo, Japan, pp , [10] M.J. Scholl, Landmarks, places, environments: multiple mind-brain systems for spatial orientation,