WORLD TRADE ORGANIZATION

Transcription

1 WORLD TRADE ORGANIZATION 20 June 2012 ( ) Committee on Trade and Development TECHNICAL COOPERATION AUDIT REPORT FOR 2011 Note by the Secretariat

2 Page 2 TABLE OF CONTENTS I. EXECUTIVE SUMMARY... 4 II. INTRODUCTION... 5 III. AUDIT METHODOLOGY... 5 IV. OUTPUT... 6 A. EXISTING OUTPUT INDICATORS... 6 B. REVIEW OF SPECIFIC ACTIVITIES... 8 V. OUTCOME... 9 A. IMMEDIATE OUTCOME... 9 B. LONGER-TERM OUTCOME... 9 ANNEX 1: ANNEX 2: RECOMMENDATIONS ON OUTPUT RECOMMENDATIONS ON OUTCOME ANNEX 3:... 14

3 Page 3 ACRONYMS ATPC BTOR DDA GTF MIP MP NAMA NTP PEF Q&A RBM RFC RTA RTPC TA TCA ToT TRTA WTO Advanced Trade Policy Course Back to Office Report Doha Development Agenda Global Trust Fund (synonym of DDAGTF) Mission Intern Programme Member of Parliament Non-Agricultural Market Access Netherlands Trainee Programme Participant's Evaluation Form Questions and answers Result-based Management Reference Centre Regional Trade Agreement Regional Trade Policy Course Technical Assistance (includes workshop, seminars, courses, conferences, etc.) Technical Cooperation Audit Unit Training of Trainers Trade-related Technical Assistance World Trade Organization

4 Page 4 I. EXECUTIVE SUMMARY 1. In anticipation of the introduction in the WTO of Result-based Management (RBM) for TA activities as of 2012, the Technical Cooperation Audit Report for 2011 already focuses on the main components of the logical framework presented to Members as part of the TA plan for : it deals with the TA output (number of activities, duration, number of participants, subjects covered, teaching methods, etc.), before addressing the immediate and longer-term outcome of the activities for beneficiary countries. 2. As far as WTO TA output is concerned, indicators of volume, number of participants and average duration of activities showed minor variations in 2011, some positives, others less so. The statistics on the various training methods used in the activities showed marginal evolutions towards more interactivity in the activities compared to 2010, but the data series are too short for the moment to detect any real trend. Lectures represented just under half of the total training time, with more interactive methods (Q&A, exercises, simulations, etc.) accounting for the balance. Compared to 2010, other methods tended to be used in a lower proportion of activities, but were allocated a larger share of the programme when used. In a context of lower WTO resources for TA, work is going on to improve the coordination, coherence and management of reduced TA resources (through innovative partnerships, prioritisation, etc.). 3. The immediate outcome of the TA activities i.e. the difference that an activity made for its participants is traditionally measured through the self-evaluation presented by the trainers in the back-to-office report (BTOR). Most such reports simply presented the views of the participants on the activity even when objective indicators, such as results of quizzes or exercises, were available. The Secretariat has undertaken to improve this as part of its current move to RBM. According to participants, 100% of the TA activities achieved their stated objectives completely or almost completely. These objectives generally included improving participants' knowledge and understanding on one or several WTO agreements, on DDA negotiations, etc. 4. As in previous years, the initial analysis of the longer-term outcome of the TA provided by the WTO over the years in a limited number of countries showed some very positive effects and impact, in particular on the compliance with their WTO obligations. The introduction of RBM in 2012 should lead to more solid evidence in this domain. 5. In 2011, the Technical Co-operation Audit (TCA) reviewed more specifically the follow-up of the recommendations contained in BTORs and performed an ex-post evaluation of the Geneva Weeks for non-resident Members and Observers. It recommended the implementation of a formal mechanism to follow-up on BTOR recommendations, as approximately half of the past recommendations had not been followed-up. This mechanism will be implemented through the introduction of RBM. Geneva Weeks have been reasonably successful so far. The current changes in WTO circumstances is prompting a reconsideration of the main parameters of the programme, to make sure that WTO resources are used in a way which best satisfies the needs of non-residents. The objectives and programme of the Geneva Week are currently being revised in this perspective.

5 Page 5 II. INTRODUCTION 6. In anticipation of the implementation of Result-based Management (RBM) for its technical assistance and training activities in 2012, since last year the Technical Cooperation Audit (TCA) Report has been structured around the key components of the WTO logical framework. After a short reminder of the methodology used by the evaluators, a first section on Output discusses certain key aspects of the activities implemented in 2011 and TCA's main findings on them. A distinction is then made in a second section between the immediate and longer-term Outcome of the activities. Implementing RBM for WTO technical assistance and training activities is still work in progress. In this context, it was not considered appropriate to formulate any new recommendation at this stage. 7. For the sake of concision, the information on Input and Output presented by the Secretariat in its report on the implementation of the technical assistance and training plan has not been repeated here. 1 It has been assumed that Members were already acquainted with it. III. AUDIT METHODOLOGY 8. This report is based on a compilation and analysis of several types of information: comments gathered through interviews of government officials, documents and responses to questionnaires that were available to TCA about the various TA activities organized by the WTO in 2011: (a) (b) (c) (d) Interviews of government officials, based either in the capitals or in Geneva, provided invaluable information on the effects and impact of WTO TA in the countries concerned. Back-to-Office Reports (BTORs) prepared by the WTO officers in charge of the activity. BTORs have a standardized format and content identifying the objectives of the activity and performance indicators in terms of effectiveness and efficiency. They should take into account the Evaluation Forms (see (d)) filled by the participants, but constitute essentially an auto-evaluation by the trainers. The main focus of BTORs should be the assessment of the immediate outcome of the activity (i.e. whether and the extent to which the immediate objectives of the activity have been achieved). They generally contain recommendations for improvement. Two hundred and fourteen BTORs have been produced in 2011 (i.e. for 77 per cent of the activities). Memoranda prepared by the WTO officers in charge of the activity. The main difference with BTORs is that the format and content of such memos is not standardized. They tend to be shorter and do not necessarily address all the issues found in BTORs. For example, they generally limit themselves to a presentation of the output and rarely discuss the immediate outcome of the activity. Memos are used when the standardized format of the BTOR would not be appropriated, due to the specificities of the activity. Thirty-one such Memos have been produced in 2011 (i.e. for 11 per cent of the activities). Participant's Evaluation Forms (PEFs) completed by participants at the end of many activities. PEFs ask participants to rate a series of items from 1 (bad) to 5 (very good), such as achievement of the objectives, quality of the activity, level of knowledge prior and after the activity, etc. Participants can add comments to their responses. 1 See document WT/COMTD/W/186.

6 Page 6 9. The analysis of these documents was complemented by field visits to a small sample of activities. During these visits, the evaluators observed the activity and interviewed the participants. They also organized a final evaluation session in which participants were asked for improvement suggestions. Field visits used to be programmed in a way which rotated the subjects, regions and trainers concerned, with a view to cover most of them over a three-year cycle. Field visits target in priority regional activities, so that evaluation costs remain commensurate to the cost of the activity itself. In 2011, these plans were disrupted by the cancellation of many of the activities initially identified for a field visit, due to political instability in several countries in 2011 and to liquidity problems in the Global Trust Fund (GTF). As a result, TCA could only perform three field visits in Whenever necessary or useful, the evaluators also discussed with the various Secretariat officers involved in the organization or delivery of an activity. IV. OUTPUT 11. TCA has formulated in the past a number of recommendations concerning the TA output of the Secretariat. 3 Output can be analysed in terms of volume of activities organised, duration of such activities, number of participants in each activity, subjects covered, teaching methods used, etc. The statistics in Annex 3 show how some of these indicators have evolved in TCA also focused more specifically during the year on the Geneva Week for non-resident Members and Observers and on the follow-up on the recommendations included in previous year BTORs. A. EXISTING OUTPUT INDICATORS 1. Management of TA activities 12. In 2011, at the request of senior management, TCA conducted an analysis of the follow-up of recommendations contained in 499 Back to Office Reports (BTORs) issued in More than half of recommendations related to the organisation of the activities (e.g. on design, content, training methods, duration) and a quarter to trainer issues (mainly on partnerships). Three quarters of all recommendations were calling for some sort of adaptation. 39% of all recommendations had not been followed-up. Lack of follow-up was mainly due to oversight, whilst the implementation of a number of recommendations was still pending. 14. This showed the limits of the current informal approach towards follow-up of BTOR recommendations, which led TCA to propose the implementation of a formal mechanism. The new mechanism holds the potential to enhance the efficiency, effectiveness and sustainability of the results related to TA activities, as well as to improve overall accountability and transparency. Its implementation should lead to a more systematic monitoring of the TA offer. 2. Promotion of Quality over Quantity 15. Reduce the number of activities and, hence, prioritise: Even if there is no official policy to reduce the number of activities organised each year, 4 the table in Annex 3 shows that 2011 was the third year in a row where the total number of national and regional activities organised by the Secretariat went down (by 21 per cent in 2011 from a total of 338 in 2010 to 266 in 2011). The decrease accelerated in Whilst the total number of national and regional activities organised by 2 See list attached as Annex 1. 3 See list of past recommendations attached as Annex 2. 4 See explanation in section II (paragraph 17) in document WT/COMTD/W/186.

7 Page 7 WTO remained somewhat stable during the period , at 385 per year on average, it has decreased by 30% over the past three years. In 2011, the decrease was essentially due to the reasons already mentioned above in paragraph As financial necessity more than virtue led to this reduction, TCA could not detect any attempt to prioritise in The lower volume of activities generated only limited real time savings, since the Secretariat continued to organise all the planned activities before cancelling them in the last stages of the process. As a result, it was not possible to dedicate more time to the activities which actually took place to improve their quality. 17. Limit the number of participants (30 max): The statistics on the average number of participants in regional activities (see Annex 3) show that the mild growth of the average number of participants per activity continued in At 32.8, this average exceeded the recommended limit of 30 by 9%. The standard deviation from this average also continued its slow growth to reach almost 13 participants, which means that the size of certain groups tends to become bigger. Even if the actual average size of the groups has not yet reached a level where the use of interactive teaching methods would become problematic, the trend over the past six years is worrying since this average has grown 58%over that period. If this trend is not reversed, we might soon reach a stage where the size of the groups will become a real problem in an increasingly high number of activities. 18. On average, there was a selection of the participants in 45% of the activities for which TCA received a BTOR (19% of national activities, 64% of regional activities and up to 100% of internships, RTPCs and Geneva-based courses). This shows that an increasing proportion of regional or global activities organised by the WTO involve a selection of participants. However, in a number of cases, part of the countries invited to an activity fail to nominate enough candidates for the organisers to make any selection. This inevitably reduces the benefits that should accrue to the group from a selection of the most appropriate candidates. 19. Lengthen the duration of activities (full weeks): TCA has collected statistics on the average duration of activities since 2002 (see Annex 3). In 2011, the average duration of national activities was somewhat stable at 2.6 days, after the sharp fall registered in The average duration of regional activities increased in 2011 to reach 3.5 days. This means that the duration gap between national and regional activities keeps increasing. Regional activities are on average almost one day longer than national ones. In any event, the average duration of both national and regional activities in 2011 still remained rather short. The insufficient duration of WTO activities is the criticism most frequently voiced by participants. This would lead to an insufficient coverage of certain parts of the programme or limit the use of interactive training methods designed to maximise the acquisition of knowledge and skills by participants. It is worth recalling that, up to a total duration of five days, the cost of extending the duration of an activity by half a day or a full day remains fairly marginal. 3. Improvement of Delivery 20. More interactive training methods (more exercises, case studies, quizzes, etc.): 2011 is the second year for which data on the use of interactive training methods are available. The small variations detected between 2010 and 2011 may consequently not be real trends. If lectures remain the predominant medium used to deliver TA activities, their average share of the programme shrank marginally. They were still used in all activities for which statistics were available and represented on average just under half of the total training time. The rest of the training time remained shared on average almost equally between Q&A (15%), exercises (14%) and case studies / simulations (11%), with minor ups and downs compared to the previous year. The latter were used in just over half of the activities (51%), however, a much lower occurrence than Q&A (83%) and exercises (71%). Compared to 2010, these interactive training methods tended to be used in a lower proportion of

8 Page 8 activities, but were allocated a larger share of the programme when used. These averages hide huge variations between activities, as some of them (e.g. thematic courses on trade negotiations) are almost entirely based on simulations, whilst at the other end of the spectrum, 10% of national activities remain almost entirely based on lectures. B. REVIEW OF SPECIFIC ACTIVITIES 1. Geneva Weeks for non-residents 21. The Geneva Week programme has been successful so far. It achieved its three stated objectives in a satisfactory manner, as it improved the knowledge of participants about the DDA negotiations, the work of the WTO and its TA offer; it exposed participants to the work of certain WTO bodies; and it allowed participants to interact with WTO staff members, with other Genevabased organisations and other delegations. 22. Limited adjustments have progressively been introduced in the Geneva Weeks' programme (more time dedicated to TA-related issues and less to WTO bodies) and in their presentation (involvement of negotiating groups' chairmen, occasional use of interactive methods, etc.). Documentation is now available in advance on-line. These changes have been welcomed by the participants, but failed to address their main concern about the lack of time available for consultations. 23. Following the evaluation in 2011 of the Geneva Week activity for non-resident WTO Members and Observers, an inter-divisional task force led by the Development Division was set up to implement some of the actions recommended. Some of these changes have already been introduced and will be reflected in the next Geneva Week event scheduled for July They include organizing Geneva Week events around major topics and themes of systemic importance to the WTO and its Members, such as the WTO's recent 8 th Ministerial Conference in December 2011 or the upcoming Trade Facilitation Symposium in July Other changes include allocating more time for non-resident participants to hold individual consultations with WTO Secretariat staff, other Members or with experts from Geneva-based inter-governmental organizations. Additionally, the management of Geneva Week will be improved with the use of Results-Based Management principles, which will be in keeping with the other Technical Assistance activities of the WTO. Meanwhile, the objectives and aims of Geneva Week have been updated. Achieving the objectives will now be measured through newly developed indicators. Further changes include encouraging participants to be more active users of e-training to prepare for their visit to Geneva and a more careful and systematic monitoring of attendance of registered participants at the various committee meetings and briefings conducted over the five days of Geneva Week. 2. Other activities reviewed in previous years (a) E-Training 24. As a result of the move to a permanent availability of existing e-training courses, of the upgrade of the e-training platform and of the introduction of two new courses on market access for trade in goods and trade & environment, the WTO offer boomed in The total number of days on which e-training was available was multiplied by seven on average. The increase was even more impressive for the availability of e-training in French and Spanish, which were less frequently offered in the past than e-training in English. 25. This increased availability made it possible to offer to 40% more participants to follow an e- training course in 2011, in spite of a reduction in the number of applications received. Whilst in the past WTO could barely satisfy half of the applications, this limitation has been removed in This

9 Page 9 logically led to a comparable increase in the number of participants effectively following an e-training course, and eventually to a 20% increase in the number of successful participants. 26. The introduction course now used as a prerequisite for many TA activities was recast in The new version of the course taking on board a number of the suggestions formulated in 2010 became available towards the end of the year. V. OUTCOME 27. The main focus of the WTO evaluation methodology has historically been on the immediate outcome of the TA activities, i.e. on whether the immediate objectives assigned to an activity have been fulfilled. This approach has been progressively complemented over time to start addressing the longer-term outcome of WTO TA. A first step in that direction was the introduction of ex-post evaluations of certain programmes in 2006, followed by the initiation in 2009 of the assessment of the effects and impact of TA activities at country level. Further progress on this front should be expected in 2012 as a result of the introduction of RBM. A. IMMEDIATE OUTCOME 28. The immediate outcome of WTO TA activities is the achievement of the immediate objectives assigned by the trainers to each activity. Significant progress has been made in the formulation of the objectives, which are now almost always presented in a way which catches the difference the activity should make for the participants (e.g. improved knowledge or skills, etc.). 29. A back-to-office-report (BTOR) prepared by the trainers who organised and delivered a TA activity is the main instrument through which the immediate outcome of WTO TA activities is evaluated. In principle, they constitute a self-evaluation of the activity by the trainers themselves. However, in 2011 most BTORs limited themselves to the presentation of the opinion of the participants, as expressed in the evaluation questionnaire at the end of the activity, even when other more objective indicators (e.g. results of quizzes, of exercises or simulations, etc.) were available. In preparation of the implementation of RBM, TCA has proposed changes to the BTOR template which should contribute to fixing this shortcoming of the past practice. 30. The increase in the number of reports produced has been one of the main achievements of the last three years. TCA continued to draw the attention to the missing reports on a quarterly basis. The overall reporting rate for the year was 89%, marginally better than in 2010 but still lower than in 2009 (see graph in Annex 3). B. LONGER-TERM OUTCOME 31. Since 2009, on the occasion of its field visits, TCA performs country reviews, whereby it assesses with the authorities of the host country the effects and impact of the TA received from WTO over the past five years. In 2012, due to the cancellation of most of the field visits, TCA performed only one country review, in Botswana. This review was guided by a standard list of questions trying to measure the impact for the officials who participated in the activities, for the teams to which they belonged and for the country as a whole. A list of the officials of the country who participated in WTO activities since 2005 was provided to the authorities in advance to guide the discussion. 32. Whilst the anecdotal evidence gathered over the years cannot be considered as representative at this stage, especially given the contrasts between national situations, it provides a very valuable insight into the concrete results produced. It also led to the formulation of interesting suggestions on how to improve WTO TA in the future.

10 Page In 2012, the main findings were the following: (a) Better compliance with Members' WTO obligations (e.g. notifications, implementation of accession commitments, submission to a TPR) was the most visible effect or impact, even if significant variations remain according to the areas or the Members concerned. (b) (c) (d) (e) WTO TA activities had allowed governments to increase their participation in certain targeted WTO negotiations (trade facilitation in all cases, followed by NAMA and agriculture) and in RTAs. TA activities have also helped improve or reinforce the internal organisation and capabilities of the governments to deal with WTO matters, although absorption capacity and staff turn-over remained a real challenge in the smallest countries. This was critical to the conduct of the accession process. The structures developed for the accession had a lasting role afterwards. They were particularly useful to implement commitments and explain the policy of the government to national stakeholders (private sector, MPs, etc.). The formulation and implementation of a coherent development-enhancing trade policy, although still a real challenge, also benefited indirectly from the improved trade culture generated by TRTA. Governments were generally satisfied with the assistance received from WTO, which they perceived as useful and of good quality, especially the advanced and regional three-month trade policy courses and long-term internship programmes. However, in many cases the TRTA received by the governments was provided predominantly by bilateral donors and the assistance received from WTO was somewhat marginal.

11 Page 11 ANNEX 1: On site evaluations carried out by TCA in Botswana Regional Seminar on Regional Trade Agreements for English-Speaking African Countries (17-19 May 2011) 2. Vienna Seminar on WTO Accessions for Central and Eastern Europe, Central Asia and the Caucasus Region (12-14 July 2011) 3. Geneva Reference Centre Training for NTP and MIP Trainees (23-24 November 2011)

12 Page 12 ANNEX 2: List of past TCA main recommendations RECOMMENDATIONS ON OUTPUT 1. Management of TA Activities Have contingency plans for slack financial times Promote cross-fertilisation amongst trainers Use video conference to replace short long-distance missions Formulate division-wide TA objectives Follow-up on prior recommendations Transform the regional desks Modify the current planning method for regional activities Benchmark each of the three levels of progressive learning, subject by subject 2. Promotion of Quality over Quantity Prepare the activities properly (e.g. planning) Reduce the number of activities and, hence, prioritise Limit the number of participants (30 max.) Lengthen the duration of activities (full weeks) Adopt an in-depth focus and use time efficiently Share lessons learned with colleagues and adjust programmes, presentations, exercises, structures, etc. accordingly 3. Improvement of Delivery More interactive training methods (more exercises, case studies, quizzes, etc.) Create a Secretariat database of pedagogical tools Improve training skills (training of trainers ToT) Identify and disseminate best practice 4. WTO Reference Centres (RFCs) Develop long-term strategy for RFCs; identify those in need of permanent support, such as Internet connection support Reduce the number of missions; assess Members' needs and plan the missions better Develop guidelines on the operation of a RFC Integrate RFCs in other activities / programmes Promote network of RFCs Focus training on the five main WTO databases; train RFC coordinators in information management Enforce annual reporting by RFCs 5. e-training Have a comprehensive offer covering each of the WTO agreements and subjects Consider those who have completed an e-training for other activities Raise level of other activities, as basics provided through e-training Develop synergies between e-training and other TA products Beef-up the resources dedicated to e-training and upgrade the internet platform Review the objectives, content and exams of e-training courses to adjust them to the three-level progressive learning structure

13 Page 13 Review the methodology and pedagogy of the courses (greater use of examples, of the book of legal texts, etc.) 6. Trade Facilitation Self-Assessment of Needs and Priorities Report to all Members about the achievement of objectives Improve coordination in the scheduling of WTO TA activities in the countries concerned Promote the participation of all stakeholders at the appropriate level Secure an appropriate level of political support from governments Develop and implement a follow-up mechanism 7. Regional Trade Policy Courses (RTPCs) Organize the WTO partnerships with regional institutions for their entire duration Gradually increase the partners' responsibilities over the years Strengthen the coordination mechanisms between regional and WTO trainers to improve the effectiveness of the joint lecturing arrangements Systematise reporting on each RTPC Cooperate with all stakeholders on course design, selection and briefing of regional trainers, training methods, pedagogical tools, etc. Facilitate networking between former and current RTPC partners, regional trainers and WTO experts 8. Internship Programmes Assess Missions' needs for interns and regroup their funding in the GTF 9. Advanced Trade Policy Course (ATPCs) Define the concepts or provisions that participants are assumed to already master Adopt a horizontal approach (instead of the current subject by subject approach) based on a series of case studies or simulations cutting across a variety of subjects Discontinue the Project segment Reinstate weekly debriefings or revisions and a final wrap-up of the course RECOMMENDATIONS ON OUTCOME Increase the share of demand-driven activities so as to better respond to Members' needs Assess Member's needs beforehand Respect WTO reporting guidelines Define objectives, performance indicators, evaluation methods Use TCA's services to assist in their formulation Measure the impact of the activities and analyse performance in this light Insert new indicators in the BTORs

14 Page 14 ANNEX 3: Output Statistics on WTO TA

15 Page 15