Three Years of Using Robots in the Artificial Intelligence Course Lessons Learned

Transcription

1 Three Years of Using Robots in the Artificial Intelligence Course Lessons Learned Abstract Amruth N. Kumar Ramapo College of New Jersey 505 Ramapo Valley Road Mahwah, NJ We have been using robots in our Artificial Intelligence course since fall We have been using the robots for open-laboratory projects. The projects are designed to emphasize high-level knowledge-based AI algorithms. After three offerings of the course, we paused to analyze the collected data and see if we could answer the following questions: (i) Are robot projects effective at helping students learn AI concepts? (ii) What advantages, if any, can be attributed to using robots for AI projects? (iii) What are the downsides of using robots for traditional projects in AI? In this paper, we will discuss the results of our evaluation and list the lessons learned. 1. Introduction Working with robots is exciting. Many teachers and researchers have attempted to translate this excitement into learning. In the last few years alone, numerous faculty have attempted to incorporate robots into the undergraduate curriculum, and in various capacities: for non-majors, in a survey course, across the Computer Science curriculum, for recruitment of women (e.g., [Haller and Fossum 2001]), in Computer Science I (e.g., [Fagin 2000]) and in the Artificial Intelligence course (e.g., [Kumar and Meeden 1998; Harlan et al 2001; Klassner 2002]). We have been using robots in our Artificial Intelligence course [Kumar 2001] for assigning projects in the course. Our Artificial Intelligence course is a junior/senior level course, taken by Computer Science majors in a liberal arts undergraduate school. The course is traditional in its content: it covers representation and reasoning, with emphasis on search, logic and expert systems. Our objective in using robots was to reinforce the learning of these AI tools using an embodied agent. This is similar to the approach initially used by [Harlan et al 2001], and that by [Greenwald and Artz 2004] for soft computing topics. Other objectives for which robots have been used include: as an organizing theme for the various AI concepts [Kumar and Meeden 1998], as an empirical testbed for philosophical issues in a graduate course [Turner et al, 1996], as a bridge between abstract AI theory and implementation [Shamma and Turner 1998], and as an exploration of the relationships between hardware, environment and software in agent design [Klassner 2002]. We chose the LEGO Mindstorms ( robot for the course because it is truly "plug-and-play". Students do not have to design circuits, or even solder components to build LEGO robots. Therefore, the projects can focus on AI algorithms rather than robot construction. LEGO Mindstorms robot is also inexpensive. Therefore, we could ask students to buy their own kit, either individually or in groups of 2 or 3 students. 1

2 In order to help students construct their robots, we recommended [Baum, 2002] as a reference. This book describes several robots, including how to construct and program them. Students can easily adapt these robots for their projects, and focus on building intelligent behaviors into them using AI algorithms. Finally, we used Java and LeJos ( with the robot. This enabled us to utilize a larger part of the on-board main memory for programs, allowing our students to write larger programs. 2. Our Goals and Objectives The following principles guided how we used robots in our AI course: AI, not Electrical Engineering: We wanted to emphasize AI algorithms and not robot construction. In other words, we wanted to minimize the time our students spent constructing the robots, and maximize the time they spent implementing AI algorithms to test high-level behavior of the robots. Constructing robots from components has greater pedagogical value to engineering students than to Computer Science students. Some knowledge of engineering principles is desirable for constructing robots. Constructing robots can be time-consuming, and frustrating to the uninitiated. Since our students were Computer Science majors in a liberal arts college, we decided to simplify robot construction by using a "plug-and-play" robot such as LEGO MindStorms. AI, not Robotics: We wanted to use robot projects to teach AI, not robotics. We wanted to emphasize knowledge-based AI algorithms and not reactive algorithms specific to robotics [Brooks 1986] - we wanted to minimize the time students spent implementing low-level reactive behavior in the robot, and maximize the time they spent building high-level knowledge-based behavior traditionally considered intelligent in AI. For this reason, we stayed away from the many interesting problems specific to robotics, such as localization, mapping, odometry, landmarking, object detection, etc., that have been addressed by other practitioners, e.g., [Dodds et al 2004; Mayer et al, 2004]. Open, not Closed Labs: Finally, we wanted to use the robots in open laboratory projects [Tucker et al 1991] projects that students carry out on their own, after class. Open-labs have some advantages over closed labs: students can spend as much time as they need to finish a project. Not only is this often necessary to properly design and test robot behavior, it also encourages students to be more creative in their design and implementation of robots. Traditionally, closed lab courses are worth more credits than open-lab courses. So, using open labs in a course helped us keep down the number of credits in the curriculum. Clearly defined, not open-ended: We chose to assign closely tailored (as opposed to openended) projects in our course. It is especially interesting to assign open-ended robot projects in a course and let students fully exercise their imagination. Such open-ended projects can be combined with contests to foster a healthy atmosphere of learning while playing (e.g., [Sklar et al 2002; Verner and Ahlgren 2004]). But, many robot behaviors can be implemented using purely reactive algorithms just as well as using knowledge-based algorithms. Since our emphasis was on using AI algorithms, we chose to closely specify the requirements of our projects. This not only gives students a clear idea of what is expected of them, but also helps us formulate a clear grading policy. Our approach differs from other current approaches for using robots in the AI course along the lines of these objectives. 2

3 2.1 Robots for Traditional Projects Why use robots for traditional knowledge-based AI projects? This has many pedagogical benefits: Using robots promotes hands-on, active learning, which increases the depth of the student s knowledge [McConnell 1996]; Students can use robots to test their implementation of AI algorithms in a situated environment rather than an abstract symbolic environment, which is known to promote learning [Brown et al, 1989]; Students have tangible ways to assess the success or failure of their implementation. Since robots are tactile, using them helps visual and kinesthetic learners, thereby addressing the needs of more and different types of learners than a traditional AI course. Finally, robots excite and motivate students. We believe that using robots in the Artificial Intelligence course also helps students better understand topics such as: Algorithm Analysis: Students get a better feel for the time and space complexities of algorithms, and an appreciation of algorithm complexity issues. Real-time programming: A topic not well represented in typical undergraduate liberal arts Computer Science curricula is real-time programming (non-determinism, concurrency, promptly responding to changes in the environment). Using robots addresses this issue, briefly, but effectively. Group Projects: [Maxwell and Meeden, 2000] state that what students learned the most from their robotics course was about team work. Assigning robot projects as group projects is a great way to offer additional opportunities for collaborative learning in the curriculum. Some questions worth considering in this context are: (i) Are robot projects effective at helping students learn traditional AI concepts? (ii) Are there downsides to using robots for traditional projects in AI? (iii) What other advantages, if any, can be attributed to using robots for AI projects? We will attempt to answer these questions. 3. Typical Projects The topics that we identified as candidates for robot projects in the introductory Artificial Intelligence course are: Blind searches depth-first search and breadth-first search; Informed searches hill climbing, best-first search and A* search; Expert systems - forward chaining and backward chaining; Game playing - Minimax search and alpha-beta cutoffs. For each topic, we designed a project that engaged the robot in a meaningful task from the real world. Clearly, all these projects could be implemented just as easily, as a purely symbolic solution. But, robot-based solutions are more natural and intuitively appealing than symbolic solutions for these problems. Blind searches: The robots had to use blind search algorithms to either traverse a two-dimensional tree (See Figure 1), or a maze. Although these problems can also be solved purely reactively, we required that students use knowledge-based algorithms in their robots. 3

4 Informed searches: The robots had to use informed search algorithms to either traverse a maze (See Figure 2) whose layout is known, or to clean a grid of rooms while minimizing travel. A successful robot not only reaches the end of the maze (which can be accomplished through reactive behavior alone), but also explores the maze in the order dictated by the algorithm, foregoing parts of the maze that are sub-optimal. Expert Systems: The robots had to use forward and backward chaining to scan a pixel grid and determine the character represented by the grid. We have used two versions of the pixel grid one where the robot traverses the grid (See Figure 3) and another where it scans the grid with an overhanging scanner arm (See Figure 4). A successful robot not only correctly identifies the character, but also traverses the pixel grid in the order dictated by the algorithm. Game Playing: The robot had to play mini-chess with a human opponent. The chess board was 5 x 5, contained only pawns, and the robot learned of its opponent s moves through keyboard input. In each case, the students could practice on one version of the prop, but had to demonstrate their robot on a different version not available to them till the day of submission. Human assists were discouraged, but penalized only infrequently. The size of the props, e.g., the number of rooms in a grid or maze, the number of nodes in a tree, and the number of characters displayable on a pixel grid was limited by the main memory limitations of the robot. Students could either work individually or in teams of two. Figure 1: A Two-Dimensional Tree for Blind Searches 4

5 Figure 2: The Maze for Blind and Informed Searches Figure 3: Pixel Grid for Forward and Backward Chaining Expert Systems 4. Evaluation of Robot Projects Figure 4: Pixel Grid and Scanner Arm of a Robot We have offered our AI course with robot projects three times so far. In this section, we will discuss the results of evaluating the robot projects each semester and compare them to draw conclusions that span all three years. 4.1 Fall 2000 This was the first time we offered the AI course with robot projects. We conducted an anonymous survey of our students at the end of the semester. In the survey, we asked them to compare robot projects with traditional LISP projects, or projects in other courses they had taken with the same 5

6 instructor (all but 3 of the respondents had taken other courses with the same instructor). There were 16 students in the class, all of whom responded to the survey. Compared to projects in other courses, students rated the robot projects in AI as: hard, i.e., 3.85 on a Likert scale of 1 (very easy) to 5 (very hard). taking a lot more time, i.e., 4.56 on a scale of 1 (lot less time) to 5 (lot more time) more interesting, i.e., 4.18 on a scale of 1 (lot less interesting) to 5 (lot more interesting) So, our initial hunch in using robots in the course was right: even though students spent a lot more time doing robot projects, they also enjoyed the projects a lot more. Students agreed that the robot projects: helped them learn/understand AI concepts better (2.06 on a scale of 1 (Strongly Agree) to 5 (Strongly Disagree)) gave them an opportunity to apply/implement AI concepts that they had learned (1.93 on the above scale) They rated the various components of the projects on a scale of 1 (easy) to 3 (hard) as follows: the assigned problems tended to be hard: 2.4. putting together robot hardware was easy to moderate: 1.71 writing software for the robot was moderate: 2.00 getting the robot to work reliably was hard: 2.87 Clearly, building the robot was the easiest part of the project, which validated our choice of LEGO robots for the course. Students were nearly neutral on whether the grade they received on the projects accurately reflected how much they had learned from the projects (2.61 on a Likert scale of 1 (Strongly Agree) to 5 (Strongly Disagree)) and on whether their grades were an accurate reflection of how much time they had spent doing the projects (3.30 on the same scale). Students were unanimous in recommending that we continue to assign robot projects in future offerings of the course. Over 90% said that they would recommend such a course to friends. Both of these indicated that robot projects had captured the imagination of the students, and were therefore effective. 4.2 Fall 2001 We used similar projects in fall 2001 as in fall We evaluated the projects using an anonymous survey at the end of the semester. 9 out of the 11 students in the class responded. Compared to projects in other courses, students rated the robot projects in AI as: hard, i.e., 4.33 on a Likert scale of 1 (very easy) to 5 (very hard). taking a lot more time, i.e., 4.78 on a scale of 1 (lot less time) to 5 (lot more time) more interesting, i.e., 3.44 on a scale of 1 (lot less interesting) to 5 (lot more interesting) They rated the various components of the projects on a scale of 1 (very easy) to 5 (very hard) as follows: neutral about the assigned problems:

7 putting together robot hardware was easy: 2.44 writing software for the robot was between neutral and hard: 3.56 getting the robot to work reliably was very hard: 4.78 Students were neutral on whether the grade they received on the projects accurately reflected how much effort they had put into the projects (3.43 on a Likert scale of 1 (Strongly Agree) to 5 (Strongly Disagree)). They were neutral on whether their grades were an accurate reflection of how much they had learned from the projects (3.25 on the same scale). These figures are interesting in how consistent they are with the figures from fall 2000, as shown in Table 1. In order to facilitate comparison, we have scaled the Section B scores of fall 2000 from a three-point scale to a five-point scale. N refers to the number of students who evaluated the projects. Table 1: Comparison of evaluation results from fall 2000 and fall 2001 Criterion Fall 2000 N=16 Fall 2001 N=9 A. Robot projects compared to traditional projects: The ease of robot projects: Scale: 1 (very easy) 5 (very hard) Time taken by robot projects: Scale 1 (lot less time) 5 (lot more time) How interesting robot projects were: Scale 1 (lot less) 5 (lot more) B. Student rating of the components of the robot projects: Scale: 1 (very easy) 5 (very hard) The assigned problems Assembling robot hardware Writing software for the robot Getting the robot to work reliably C. Whether project grades reflected: Scale: 1 (very easy) 5 (very hard) The effort put in by students on the project How much students had learned from doing the project Even though students consistently rated robot projects as being harder and a lot more timeconsuming than traditional projects, they also rated robot projects as being a lot more interesting than traditional projects, highlighting one advantage of using robots for traditional projects in AI that they are more engaging, and therefore, more effective. Students were consistent in thinking that getting the robots to work reliably was the hardest part of a robot project. They were also consistently neutral about whether project grades reflected the effort they put into the projects. That they reported spending a lot more time on robot projects may have tempered their opinion on this issue. 4.3 Fall 2003 This was the third time we offered the Artificial Intelligence course with robot projects. Instead of a single end-of-the-semester evaluation of the use of robots in the course, we evaluated every project 7

8 individually. We believed that evaluating each project individually, and as soon as it is completed would provide a more accurate picture than an end-of-semester evaluation of all the projects together. Previous evaluations had shown that getting the robot to work reliably was the hardest part of any robot project. Therefore, in fall 2003, we relaxed the project requirements in several ways: (i) the environment/props were more flexible; (ii) students could assist their robot without being penalized as long as the robot announced the correct state. The state announcements of a robot were now used to judge its success/failure. (These changes are described in the next Section Lessons Learned.) We wanted to assess the impact of using robots on students' knowledge of analysis of algorithms. So, we drafted a test consisting of 10 multiple choice questions. We administered the test both at the beginning of the semester, and again at the end of the semester. Students did not have access to the test in the interim. The scores of 5 students increased from pretest to post-test; the scores of 3 students stayed the same, and the scores of 2 students decreased. We discarded the scores of students who took one test and not the other. We cannot draw any definitive conclusions because of the small sample size, but the numbers are encouraging. On our evaluation of the first project, 11 students responded (class size was 12). Respondents rated the various aspects of the project as follows, on a scale of 1 (very easy) to 5 (very hard): Neutral about building the robot: 2.82 Neutral about writing the program: 2.9 Getting the robot to work reliably was hard: 4.18 Students rated the components of the projects as follows on a scale of 1 (Strongly agree) to 5 (Strongly disagree): For Depth-first search: o helped them understand the algorithm: 2.55 o helped them learn how to implement the algorithm: 2.45 o helped them apply the algorithm to a problem 2.0 For Hill-Climbing: o helped them understand the algorithm: 2.18 o helped them learn how to implement the algorithm: 2.55 o helped them apply the algorithm to a problem: 2.27 On a concept quiz, students who did not attempt the project did poorly as compared to those who did. On our evaluation of the second project, 12 students responded. Respondents rated the various aspects of the project as follows on a scale of 1 (very easy) to 5 (very hard): Building the robot was easy: 2.08 Neutral about writing the program: 2.92 Getting the robot to work reliably was neutral to hard: 3.58 Students rated the components of the project as follows on a scale of 1 (Strongly agree) to 5 (Strongly disagree): For Best-first search: o helped them understand the algorithm: 2.17 o helped them understand how to implement the algorithm: 2.08 o helped them apply the algorithm to a problem:

9 For A* search: o helped them understand the algorithm: 2.25 o helped them understand how to implement the algorithm: 2.25 o helped them apply the algorithm to a problem: 2.25 On a concept quiz, students who did not attempt the project did poorly as compared to those who did. On our evaluation of the third project, 10 students responded. Respondents rated the various aspects of the project as follows on a scale of 1 (very easy) to 5 (very hard): Building the robot: 3.3. This is to be expected since students had to build a scanner arm, which had an overhang and used rack and pinion gears (See Figure 4). Writing the program: 2.9 Getting the robot to work reliably: 4.0 Students rated the components of the project as follows on a scale of 1 (Strongly agree) to 5 (Strongly disagree): For forward chaining: o helped them understand the algorithm: 2.2 o helped them understand how to implement the algorithm: 2.2 o helped them apply the algorithm to a problem: 2.2 For backward chaining: o helped them understand the algorithm: 2.1 o helped them understand how to implement the algorithm: 2.1 o helped them apply the algorithm to a problem: 2.2 On the fourth project, since there were only 3 respondents, we did not analyze the results. Table 2 summarizes the student responses for the first three projects. N refers to the number of students who evaluated the projects. Table 2: Comparison of evaluation results from fall 2003 Criterion Project 1 N=11 Project 2 N=12 Project 3 N=10 Rating the components of the project: Scale: 1 (very easy) to 5 (very hard) Building the robot Writing the program Getting the robot to work reliably For the first algorithm, the project helped: Scale: 1 (Strongly agree) to 5 (Strongly disagree) Understand the algorithm Understand how to implement the algorithm Apply the algorithm to a problem For the second algorithm, the project helped: Scale: 1 (Strongly agree) to 5 (Strongly disagree) Understand the algorithm Understand how to implement the algorithm Apply the algorithm to a problem

10 Students were again consistent in noting that getting the robot to work reliably was the hardest part of a robot project. For all six algorithms, students agreed that the robot project helped them understand the algorithm, how to implement it, and how to apply it to a problem. Clearly, students believe that robot projects help them learn the underlying AI concepts. The results were most consistent, and hence, most definitive on that robot projects help students learn how to apply the algorithm to a problem. We believe that this is one of the advantages of using robots for traditional projects in AI in the traditional symbolic world, concepts such as state and operators are exact, whereas in the robot world, they are subject to approximation and interpretation. This forces students to focus on the boundary between real-life problems and their AI solutions, and helps them better develop their skills of operationalizing AI algorithms, i.e., applying them to solve problems. We are pleased that this is borne out by the results of the evaluation. We analyzed the results of the midterm and final exam to see if there was a correlation between project completion and grades. We considered anyone who scored at least 60% on a project as having completed the project. Table 3 summarizes the results. For instance, the first project was on depth-first search and hill climbing. On the midterm exam, 40% of the first question was on depthfirst search. The sole student who did not attempt the first project scored 3 (out of 10) points on this question. The scores of the rest of the students ranged from 7 through 10, with an average of 8.4. N/A in the table indicates that the student did not attempt the question - students were asked to answer 6 out of 7 questions on the midterm exam and the final exam. It is clear from the table that there is a positive correlation between project completion and student scores on relevant sections of the tests. Table 3: Comparison of the test scores of the students who completed the projects, versus those who did not. Problem Topic & Points Scores (out of 10) of students who attempted the project Did not attempt project Project 1: Depth-first search and Hill Climbing Midterm 4 points on Range: 7 10 (11 scores) 3 Problem 1 Depth-first search Average: 8.4 Midterm Problem 3 6 points on Hill climbing Range: 3 10 (11 scores) Average: 7.5 N/A Project 2: Best-first search and A* Midterm Problem 5 8 points on A* search Range: 6 9 (9 scores) Average: 7.7 4,5,9 Project 3: Forward and backward chaining Final Exam Problem 5 6 points on forward / backward chaining Range: N/A 10 (10 scores) Average: 5.7 4, N/A 4.4 Analysis Across the Semesters We considered any student who scored at least 1 point on a project as having attempted the project, and anyone who scored at least 60% of the points as having completed the project. Table 3 lists the 10

11 number of students who attempted and completed each project from fall 2000 through fall N refers to class size, not including any students who audited the course. Table 4: Project completion rates from fall 2000 through fall 2003 Semester Project 1 Project 2 Project 3 Project 4 Fall 2000 Attempted N/A (N=15) Completed N/A Fall 2001 Attempted (N=11) Completed Fall 2003 Attempted (N=12) Completed Most students who attempted a project also completed it. 47% of the students completed the projects in fall 2000, 75% in fall 2001, and 65% in fall Not many students attempted project 4 in fall 2003 because it was assigned in the last three weeks of the semester. If we discount it, project completion rate jumps to 83% in fall These improvements in completion rates over the years may be attributed to the changes described in the Lessons Learned section that we made to our projects. At over 65%, the completion rates are comparable to those of traditional projects in an upper-level elective, suggesting that project completion rates do not suffer when robots are used for AI projects. Students reported spending large amounts of time on the projects. In fall 2001, they spent an average of 30.5 hours on the first project (which had four components, and was arguably the largest project), 20.2 hours on the second project and 12.6 hours on the third and arguably the easiest project. In fall 2001, they reported spending an average of 8.2 hours on the first (the easiest) project, hours on the second project, 32.5 hours on the third project, and 12.2 hours on the fourth project (on Expert Systems). Within each project, students reported wildly different amounts of time for instance, on the fourth project in fall 2001, the range was 3 hours to 30 hours! The complexity of the robot, its programming and testing all play a role in how much time a student spends on each project. For instance, students who took advantage of callbacks wrote significantly shorter programs (and spent commensurately less time) than those who tried to deterministically model the entire behavior of the robot, because callbacks permit modular separation of reactive and planning components. Students who used a three-wheel construction spent longer hours testing the robot than those who built the robot with dual differentials and four wheels. In fall 2000, there was a trend from individual effort towards team effort as the semester progressed: whereas 50% of the students attempted the first project by themselves, 60% or more attempted the second and third projects in groups. The students strongly recommended that group projects be allowed in future offerings of this course (1.42 on a scale of 1 (Strongly recommend) to 5 (Strongly do not recommend)). Yet, in fall 2003, all but two students chose to work on the first three projects individually. Simplifying the environment and the criteria for the success of the robot project may have contributed to this trend. 5. Lessons Learned 11

12 It is clear from our evaluations that one of the hardest parts of robot projects is to make the robot behave reliably, time after time. Students often found that their robot would work correctly on trial runs, but would fail to work during submission. Many factors contribute to the unpredictability of a robot, including: The many analog components in a robot, such as sensors and motors; The batteries driving a robot, and their level of charge; The mechanical design of the robot. Hardware Guidelines: The following are some mechanical design considerations that can alleviate the problem of unpredictable robots: Treads and thick wheels should be avoided they generate more friction than thin/bicycle wheels and adversely affect turning and motion, especially as the batteries discharge. Three-wheel design should be avoided the third wheel makes turning unpredictable, especially when it is a swivel wheel. Two wheels and a stub arrangement should also be avoided the stub generates additional friction as it drags on the floor. Preferably, a fourwheel design with dual differentials should be used, and the robot should be turned by changing the direction of rotation of front and rear wheels. The load/weight should be balanced on the front and rear wheels to avoid skateboard effect. Light sensors should be mounted as close to the ground as possible. Preferably, the sensor should be calibrated at run-time. While our experience has been specific to LEGO MindStorms robots, we believe these design guidelines are applicable to most student-assembled robot platforms. Project Design Guidelines: The following are some guidelines for designing robot projects that alleviate the problem of unpredictable robot behavior: The robot should externalize its state before each move. This will help the instructor and the student evaluate the correctness of the underlying knowledge-based algorithm, even if the robot goes off-course using dead reckoning. For instance, a robot that scans a pixel grid should announce the next pixel it intends to examine at each step. A robot that cleans rooms should announce the coordinates of the next room it will visit, before traveling to it. The environment/prop should be flexible, not rigid. A flexible environment can significantly alleviate the problems arising from the unpredictability of robot behavior. For instance, consider the design of a maze. A maze with fixed walls (e.g., Figure 2) is less forgiving than one that is constructed with walls that can be moved during the demonstration (e.g., Figure 5). In a fixed-wall maze, if a robot turns by 75 instead of 90 degrees, the robot may end up in a room other than the one it intended to visit. In a moveable-wall maze, if a robot is found to turn incorrectly, the wall that it should have encountered can be moved into its path, essentially making the room it visits the one it intended to visit! As long as the focus of the robot project is a knowledge-based algorithm (and not robotics), and the robot announces its state and intention before each movement, moving walls to address errors in the navigation of a robot is inconsequential to the correctness of the project. Finally, the students should be provided the option of submitting an unedited videotape of their robot in action, instead of demonstrating the robot in the instructor s presence. In our experience, in-person demonstrations often turn into marathon sessions for both the students 12

13 and the instructor, since the students do not want to give up trying to get their robot to work correctly. Finally, it is not always possible or advisable to transplant a symbolic AI project into the robot domain. For instance, consider the issue of chronological backtracking as a robot tries to find its way out of a room. Unless the robot uses reliable dead reckoning, it will fail to backtrack to the correct prior location in the room. Even if it can backtrack to the correct location, depending on the placement of obstacles in the room, stopping to recalculate its next move at this location may not be the most reasonable next course of action. Using search on a two-dimensional tree (Figure 1) is an example of a symbolic project transplanted into the robot domain it adds to the complexity of the solution (because of the unreliability of robot motion) without adding to the engagement factor of the problem. Recasting the same problem as a search through a maze (Figure 2) not only makes the problem more interesting, but also engages the strengths of the robot processing input from multiple sensors, collision detection, the ability to explore and map a space, etc. Our experience has been that providing a real-life context is a good way to adapt a traditional project for robots. Real-life contexts typically demand approximation instead of exactness, allow recovering from a mistake instead of undoing one, and reward dynamically reacting to a changing environment, all strengths of robots. As added bonuses, real-life contexts typically make projects more interesting and engaging, and help students understand how to operationalize an algorithm. 6. Discussion Figure 5: A Maze Built with Moveable Walls Analysis of the results from fall 2003 clearly shows that there is a positive correlation between project completion and test scores. Moreover, students themselves believe that robot projects help 13

14 them learn the underlying AI concepts. Therefore, robot projects are effective at helping students learn AI concepts. In both fall 2000, and fall 2001, students consistently rated robot projects as being harder and a lot more time consuming than traditional projects. Across all three years, they confirmed that getting the robots to work reliably was the hardest part of a robot project. These are clearly some drawbacks of using robots for traditional projects in the AI course. In both fall 2000, and fall 2001, students consistently rated robot projects as being a lot more interesting than traditional projects. This confirms one advantage of using robots for traditional projects in AI they are more engaging, and therefore, more effective. In fall 2003, students consistently agreed that robot projects help them learn how to apply the algorithm to a problem. This is another advantage of using robots for traditional projects in AI robot projects help students better operationalize AI algorithms than symbolic projects. Finally, our evaluation in fall 2003, while preliminary, seems to support the assertion that using robot projects promotes the understanding of algorithm complexity issues. So, is it worth using robots for traditional projects in the AI course? The answer is depends - yes, if we consider the excitement and engagement generated among students; may be, if we only go by how much students learn; and no, if we consider the time and effort that robot projects demand from both the students and the instructor. We plan to continue to assign robot projects in our AI course. We plan to develop additional knowledge-based robot projects on advanced topics. Finally, we plan to continue to evaluate the use of robot projects in our AI course. Acknowledgments Partial support for this work was provided by the National Science Foundation s Course, Curriculum and Laboratory Improvement Program under grant DUE The author thanks the anonymous reviewers whose comments helped significantly improve this paper and its findings. References Cited [1] Baum, D., Dave Baum's Definitive Guide to Lego MindStorms. Apress Publishers. 2 nd Edition, [2] Brooks, R.A., A Robust Layered Control System for a Mobile Robot, IEEE Journal of Robotics and Automation, Vol. 2(1), March 1986, pp [3] Brown, J.S., Collins, A., and Duguid, S., Situated Cognition and the Culture of Learning, Educational Researcher, 18(1), 1989, [4] Dodds, Z., Santana, S., Erickson, B., Wnuk, K., Fisher, J., and Livianu, M. Teaching Robot Localization with the Evolution ER1. Accessible Hands-on Artificial Intelligence and Robotics Education, AAAI Spring Symposium Technical Report SS-04-01, 2004,

15 [5] Fagin, B., Using Ada-Based robotics to teach Computer Science. In Proceedings of the 5 th Annual Conference on Innovation and Technology in Computer Science Education (ITICSE 2000), New York, NY: The Association for Computing Machinery, 2000, [6] Fossum, T.V., Haller, S.M., Voyles, M.M., and Guttschow, G.L., A gender-based study of elementary school children working with Robolab, Technical Report of the AAAI Spring Symposium Workshop on Robotics in Education, Stanford University, March [7] Greenwald, L. and Artz, D. Teaching Artificial Intelligence with Low-Cost Robots, Accessible Hands-on Artificial Intelligence and Robotics Education, AAAI Spring Symposium Technical Report SS-04-01, 2004, [8] Harlan, R., Levine, D., and McClarigan, S. The Khepera Robot and the krobot Class: A Platform for Introducing Robotics in the Undergraduate Curriculum. Proceedings of 32 nd SIGCSE Technical Symposium on Computer Science Education, 2001, [9] Klassner, F. A Case Study of LEGO MindStorms Suitability for Artificial Intellifence and Robotics Courses a the College Level, Proceedings of 33 rd SIGCSE Technical Symposium on Computer Science Education, 2002, [10] Kumar, D. and Meeden, L., A Robot Laboratory for Teaching Artificial Intelligence. In Proceedings of the Twenty-Ninth ACM SIGCSE Technical Symposium (SIGCSE '98), New York, NY: The Association for Computing Machinery, 1998, [11] Kumar, A. Using Robots in the Undergraduate Artificial Intelligence Course: An Experience Report, Proceedings of FIE 2001, 2001, Session T4D. [12] Maxwell, B.A., and Meeden, L.A., Integrating Robotics Research with Undergraduate Education, IEEE Intelligent Systems, November/December 2000, 2-7. [13] Mayer, G.R., Weinberg, J.B., and Yu, X. Teaching Deliberative Navigation Using the LEGO RCX and Standard LEGO Components. Accessible Hands-on Artificial Intelligence and Robotics Education, AAAI Spring Symposium Technical Report SS-04-01, 2004, [14] McConnell, J.J., Active Learning and its use in Computer Science, In Proceedings of the Conference on Integrating Technology into Computer Science Education (ITICSE 1996), New York, NY: The Association for Computing Machinery, 1996, [15] Shamma, D.A. and Turner, C.W Teaching the Foundations in AI: Mobile Robots and Symbolic Victories. In Proceedings of the Eleventh International Florida Artificial Intelligence Research Symposium Conference (FLAIRS 98), Menlo Park, CA: AAAI Press. [16] Sklar, E., Eguchi, A., and Johnson, J. Robocupjunior: Learning with Educational Robotics. In Proceedings of the 6 th Robocup Symposium, [17] Tucker, A. et al.: Computing Curricula 1991, Communications of the Association for Computing Machinery, 34 (June 1991): [18] Turner, C.; Ford, K.; Dobbs, S.; Suri, N.; and Hayes, P Robots in the Classroom. In Proceedings of the Ninth Florida Artificial Intelligence Research Symposium (FLAIRS '96), Florida AI Research Society. 15

16 [19] Verner, I.M. and Ahlgren, D.J. Robot Contests: Promoting Experiential Engineering Education. Accessible Hands-on Artificial Intelligence and Robotics Education, AAAI Spring Symposium Technical Report SS-04-01, 2004,