Peter Ostafichuk, Mechanical Engineering, UBC With: Goran Fernlund Sherry Green Carol Jaeger Bill McCutcheon Dirk van Zyl

Transcription

1 A Shift to Outcomes-Based Assessment in Engineering How changes in the national accreditation requirements in professional engineering are changing the way we evaluate our programs. Peter Ostafichuk, Mechanical Engineering, UBC With: Goran Fernlund Sherry Green Carol Jaeger Bill McCutcheon Dirk van Zyl Thomas Froese Oldrich Hungr Mark Martinez Fariborz Taghipour Joseph Yan

2 1 Canadian Engineering Accreditation Engineering accreditation in Canada has traditionally been inputbased, with a focus on metrics such as contact hours, course content, class sizes. Through international agreements, Canada other countries have agreed to shift to outcomesbased accreditation systems. By 2014, Canadian universities must demonstrate that graduates of their programs possess attributes in twelve specified areas. Schools across the country, including UBC, are working hard to develop tools processes to meet these requirements.

3 2 CEAB Graduate Attributes The Canadian Engineering Accreditation Board (CEAB) is responsible for Canadian engineering accreditation. They have adopted 12 Graduate Attributes to specify the knowledge abilities required of engineers at the time of graduation: 1. Knowledge base for engineering 2. Problem analysis 3. Investigation 4. Design 5. Use of engineering tools 6. Individual team work 7. Communication skills 8. Professionalism 9. Impact on society environment 10. Ethics equity 11. Economics project management 12. Lifelong learning

4 3 Process Flow Used at UBC 1. Defining Purpose Outcomes Plan 4. Analysis

5 Outcomes 4 Defining Outcomes Plan 4. Analysis The graduate attributes broadly define the required outcomes, but they are difficult to measure directly. For example: #2 Problem Analysis: An ability to use appropriate knowledge skills to identify, formulate, analyze, solve complex engineering problems in order to reach substantiated conclusions. With inspiration from other schools, we began by defining the attributes in terms of Indicators. These are the 4-7 measurable descriptors of what students in UBC Engineering must do to be considered competent in an attribute. For example, the first of five indicators for Problem Analysis was phrased as: 2.1 Problem Identification: Identify known unknown information, uncertainties, biases when presented with a complex ill-structured problem

6 Outcomes 5 Program Mapping Plan 4. Analysis A survey was sent to faculty teaching in each program. For each course, instructors indicated to what degree each attribute was developed: E U I Emphasized (taught assessed) Utilized (required for the course but not taught) Introduced (appeared in the course but not assessed) Not developed Knowledge Base Problem Analysis Investigation Design Course Number APSC 150 I I I I I I U I I MATH 100 E U I U I I MATH 101 E U I U I I MATH 152 E I E E I Engineering Tools Individual / Team Work Communication Professionalism Impact of Engineering Ethics / Equity Econ. / Project Management Life-long Learning PHYS 153 E E E I I E U U U U I U PHYS 170 E E U I U I I APSC 201 U E U U U E E E E I U APSC 261 U U I U U U E E E I I MATH 253 E E I E I U I U U MATH 256 E E U I I MECH 220 E I U U E U I I I I I MECH 221 E E E I E U U I I I I MECH 222 E E E U E U U I I I I I MECH 223 E E E E E E U U E I E I EECE 365 E U I I U I U I I MECH 305 E E E I U E E I I I I MECH 325 E E E E U E U E MECH 326 E U I U MECH 327 E E U E E I E U I I MECH 328 E E E E E U E U U U E U MECH 329 E E U E U U U E U I I MECH 364 E E E U I I E I I MECH 368 E U U U E I I I I MECH 375 E E I I I I I I I MECH 380 E E U U I MECH 392 E E E U U I I APSC 450 U U U I I U U E E E I MECH 431 I U U U U I I I U I E I MECH 457 E E E E E E U U U U U I

7 Outcomes 6 Program Mapping Plan 4. Analysis Course 1 Knowledge Base In the survey, instructors also indicated which Emphasis Exams Quizzes Assignments In class Reports Project / lab Presentations No Assesmt Other Course Number MATH 100 E X X X MATH 101 E X X X APSC 150 I MATH 152 E X X X X X PHYS 153 E X X X X X PHYS 170 E X X X X APSC 201 U MECH 220 E X X X X MECH 221 E X X X X X X MECH 222 E X X X X X X MECH 223 E X X X X X X X X MATH 253 E X X X X MATH 256 E X X assessment tools they used for any attributes the emphasized in their course. With the map assessment information, several courses were targeted in each program for data collection.

8 Outcomes 7 Plan 4. Analysis A hful of courses were targeted in each program data collection is ongoing. Existing course evaluation tools are being used wherever possible. In addition to embedded questions on exams, extensive use is being made of marking rubrics. Shown is a rubric for prototype evaluation. Five criteria in the left column map directly to the Indicators used to define the attributes. The results are used in the course as well as documented for accreditation.

9 Outcomes 8 Plan 4. Analysis To complement the instructor-gathered data from courses (exams, rubrics, etc.) various student surveys are being piloted. Shown is an example of part of a survey where students selfrate their competency with the different attributes. Similar surveys have been used in courses to gather data at the indicator level.

10 Outcomes Plan 4. Analysis 9 Analysis & Data is being summarized by year, indicator, attribute. At this stage the effort is to identify strengths weaknesses within the programs. Attribute 4: Design An ability to design solutions for Overall 1st Year complex, open ended engineering problems to design systems, components or processes that meet 5 5 specified needs with appropriate 75% attention to health safety risks, 54% applicable stards, economic, 5 environmental, cultural societal 3rd Year 28% considerations. 25% 15% Below Expectations: 3% 3% 5 5 Marginal: 28% Meets Expectations: 54% Exceeds Expectations: 15% Indicator Summary Courses elements assessed 4.1 Use of Process MECH 223 Report 1 & 2 Adapt apply general design process to design system, component, or process to solve openended complex problem. 5 MECH 223 MECH 223 MECH 325 MECH 45X 4.2 Need Identification Identify customer, user, enterprise needs, applicable constraints 5 MECH 223 MECH 325 MECH 328 MECH 45X Presentation 1 & 2 Final Exam 2 Assignments 1 5 Report 1, 2, & 3 Final Exam 1 & 2 Assignments 1 5 Project Report Report 1 2nd Year 4th Year

11 Outcomes 10 Plan Plan 4. Analysis We are at the early stages of this process, are piloting what we believe to be a workable assessment framework. We have initial data on how our programs are performing, but have not yet begun implementing curriculum changes or program improvements. We are striving to implement a meaningful yet sustainable process. In particular, we hope to incorporate data collection as part of regular course program activities (as opposed to the accreditation scramble that used to happen every five or six years). In addition, we are working hard to educate students about the coming changes, to include them other stakeholders in the process.

12 11 National Collaboration UBC is one of seven Canadian schools collaborating on supporting this process. Reflecting the scale of task ahead of us, we have collectively chosen the acronym EGAD based on the Engineering Graduate Attribute Development Project. EGAD provides two main support mechanisms: Workshops: training workshops are offered on a regular basis, on request. Since 2010, 12 workshops have been delivered in six provinces. Online Resources: online outcomes-based accreditation training materials resources are available through the EGAD website: Included on the website are vetted resources examples drawn from the participating institutions.