THE USABILITY OF SCIENCE ADVICE TO INTERNATIONAL ENVIRONMENTAL CONVENTIONS

Transcription

1 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 1/42 THE USABILITY OF SCIENCE ADVICE TO INTERNATIONAL ENVIRONMENTAL CONVENTIONS by Pia M. Kohler Department of Urban Studies and Planning Massachusetts Institute of Technology, Cambridge, MA September 2003 Abstract: Many international environmental conventions have designated a body responsible for providing decision makers with science advice. Yet, how usable is this advice being provided to decision-makers? I suggest that the usability of this science advice will depend on four factors: autonomy, diversity, process and transparency (in order of decreasing importance). These factors are evaluated for three international environmental conventions: the 1971 Ramsar Convention on Wetlands, the 1985 Vienna Convention for the Protection of the Ozone Layer and its 1987 Montreal Protocol on Substances that Deplete the Ozone Layer, and the 1992 Convention on Biological Diversity. Decision-makers from these conventions were surveyed to gain a greater insight into what factors are significant contributors to a more usable output. Through these surveys, three factors are identified as most important in determining usability of science advice: scientist reputation, autonomy from government influence and transparency. The results also imply that the size of the science advisory body may be a critical factor. The next step in this avenue of research is to examine how thee attributes can be improved in existing science advisory bodies, and how these lessons can be applied to the creation of new science advisory processes for international settings. Prepared for presentation at the Open Meeting of the Global Environmental Change Research Community, Montreal, Canada, October, 2003 Forthcoming publication: Kohler, Pia M The Usability of Science Advice to International Environmental Conventions. in Negotiating a Sustainable Future: Innovations in International Environmental Negotiation - Volume 12. Lawrence E. Susskind, William Moomaw and Kristen Kurczak. Eds. Cambridge, MA: Program on Negotiation Books.

2 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 2/42 THE USABILITY OF SCIENCE ADVICE TO INTERNATIONAL ENVIRONMENTAL CONVENTIONS PROLOGUE The global community has recently come together in an effort to prevent further degradation and loss of X, a cherished environmental commodity found throughout the world and whose existence is vulnerable to industrial development. Non-governmental organizations, grassroots activists, politicians and experts have drafted the text of a convention on X which will serve as a basis for global action to preserve X. This convention will function as most other international environmental conventions. The day-to-day operations of the convention will be overseen by a small secretariat which will report to the governments that have decided to become parties to the convention (Parties). The Parties to this convention will meet regularly as a Conference of the Parties (COP). Parties will be called upon to make decisions on the setting, implementation and monitoring of targets and measures. While governments have agreed on a final text for the convention, it is now necessary to wait for a predetermined minimum number of governments to ratify the convention so that it can enter into force. Meanwhile, governments are meeting regularly to iron out organizational and implementation details of the convention. The monitoring of X, and the necessary steps to preserve X all come under the decision-making authority of the COP. As the convention is implemented, these decisions will be based on science knowledge. Following the example of most recent international conventions, governments are designing a science advisory body to provide guidance to Parties on the setting, monitoring and implementation of targets and measures pertaining to the effective application of the Convention.

3 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 3/42 In setting the parameters for this science advisory body, governments will need to define its mandate. Will it only be asked to provide guidance on setting targets, or will it also be asked to provide analyses of policy options for meeting these targets? Will this body oversee compliance by Parties, and should its members be responsible for assessing infractions or extenuating circumstances? Governments will have to agree on the required qualifications for participation in the science advisory body, on the number of participants, as well as on any controls over participants nationalities or institutional affiliation. This science advisory body will represent a significant budgetary commitment, and governments will have to decide how much can be spent on providing science advice, including details such as convening meetings and providing translation (in meetings or of documents). Throughout this decision-making process, governments will seek an organizational structure that will provide science advice most usable to Parties in making decisions for the preservation of X. In facing the challenge of designing this science advisory body, there is a need to learn from existing science advisory bodies which organizational elements contribute to a more usable output. INTRODUCTION Some issues, in particular those concerning public goods, or common pool resources, are often better suited to preservation, or to being addressed in general through multilateral, or even global action. Yet, few of these efforts have been successful in achieving the improvement sought in the first place. Many factors may be to blame for this lack of efficacy. These may include concerns over maintaining national sovereignty and the associated limits to implementing and enforcing international agreements, difficulties arising from the sheer scale and scope of the problems being tackled through these means, a lack of sufficient widespread

4 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 4/42 commitment or interest to concretely improve the situation. One factor increasingly blamed for the lack of effectiveness of international environmental conventions is the provision of science advice. Among others, Oran Young and George Demko identified the integration of scientific knowledge as one of the necessary components of an effective international environmental regime (Young and Demko, 1996). In order to address this flaw of international environmental systems, it is necessary to first gain a greater understanding of the science advisory process associated with different international environmental conventions. In most international environmental conventions, the incorporation of science into decision-making has been provided for through the creation of science advisory bodies associated with specific conventions. Science advice has been provided using a variety of structures, including large subsidiary science advisory bodies, principal scientific panels, and groupings of expert panels. The differences in the organizational structures of these science advisory bodies will influence the extent to which decision-makers will feel confident in accepting and using the products from these bodies. In this context, decision-makers are government delegates attending the Conference of the Parties (COP) the decision making body of an international environmental convention. In this paper, I suggest that the usability will depend on four factors (in order of decreasing importance): autonomy, diversity, process, and transparency. I will evaluate these factors for three international environmental conventions: the 1971 Ramsar Convention on Wetlands, the 1985 Vienna Convention for the Protection of the Ozone Layer and its 1987 Montreal Protocol Substances that Deplete the Ozone Layer, and the 1992 Convention on Biological Diversity. This evaluation will first be carried out using a weighted score sheet, and

5 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 5/42 further understanding will be sought by surveying decision-makers who have taken part in recent COPs of these three conventions. BACKGROUND The improved integration of scientific knowledge has been identified as a key element in developing effective international policy by a number of researchers whom have explored the effectiveness of international institutions (Birnie, 1996; Guppy, 1996). There is also an emerging literature available assessing the effectiveness of current international environmental conventions and in a few cases providing recommendations for improvement (Haas, Keohane and Levy, 1993). In Environmental Diplomacy: Negotiating More Effective Global Agreements (Susskind, 1994), Lawrence Susskind concludes by presenting suggestions for improvements, including the ten recommendations from the Salzburg Initiative, the result of a series of seminars held in 1990 and 1991 bringing together more than 120 governmental, nongovernmental, and corporate representatives from thirty-two countries. In her 1999 work, Elisabeth Corell focused on the specific role of expert knowledge in the negotiation of the Convention to Combat Desertification (Corell, 1999). There is little research available focusing in particular on the means of providing science advice, identifying its weaknesses, or developing opportunities for improvement. Such research is of particular interest as new conventions are being created and specifics of their scientific advisory bodies are under negotiation. For example, the recent Stockholm Convention on Persistent Organic Pollutants has yet to have a clear definition of its science advisory body responsible for reviewing chemicals for inclusion under this convention. Gaining a greater understanding of what will make the products of such bodies more usable for decision-makers is an important consideration in designing, or even improving, science advisory bodies. For the

6 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 6/42 purpose of this research, three international environmental conventions are being studied. These conventions were established in different decades and tackle problems of varying scope and scale. The Ramsar Convention on Wetlands was adopted in Ramsar, Iran in As of August 2003, 138 states were Contracting Parties to this agreement (Ramsar Convention Bureau, 2003b). The Ramsar Convention is implemented by a Conference of the Contracting Parties (COP) which meets every three years. The last meeting was held in November 2002 in Valencia, Spain. The COP elects a Standing Committee to guide the Convention's progress and oversee the work of the Ramsar Bureau in between meetings of the COP. The Standing Committee has 16 members, ensuring geographic representation among the Convention s 6 regions: Africa, Asia, Europe, Neotropics, North America, and Oceania. There are also 6 permanent observers to the Standing Committee: the Netherlands, Switzerland, BirdLife International, IUCN-The World Conservation Union, Wetlands International, and WWF-International (Ramsar Convention Bureau, 2003a). The Scientific and Technical Review Panel (STRP) provides science advice to the Ramsar COP and Standing Committee (Ramsar Convention Bureau, 2003d). The Vienna Convention for the Protection of the Ozone Layer was adopted in 1985 and has been ratified by 185 Parties as of July 2003 (Ozone Secretariat, 2003c). In 1987, the Convention was supplemented by the Montreal Protocol on Substances that Deplete the Ozone Layer, which 184 Parties have ratified to date (Ozone Secretariat, 2003c). Four subsequent amendments have been made to the Montreal Protocol, in 1990, 1992, 1997 and The Conference of the Parties (COP) and the Meeting of the Parties (MOP) oversee the implementation of the Convention and the Protocol respectively. While the MOP meets annually, the COP meets every three years. These two events are usually held in conjunction with each

7 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 7/42 other. Each of these groups has the authority to mandate the creation of expert panels to provide them with science advice. Three panels are currently providing science advice: a Scientific Assessment Panel (SAP), an Environmental Effects Assessment Panel (EEAP) and a Technology and Economic Assessment Panel (TEAP). Their work is supplemented by seven Technical Options Committees (TOC) (Ozone Secretariat, 2000). The Convention on Biological Diversity was adopted in 1992 and has been ratified by 187 parties as of August 2003 (Secretariat of the CBD, 2003c). In 2000, the Convention was supplemented by the Cartagena Protocol on Biosafety which to date has been ratified by 57 Parties and will be entering into force in September The Conference of the Parties (COP) to the Convention is advised in science matters by the Subsidiary Body on Scientific, Technical and Technological Advice (SBSTTA) (Secretariat of the CBD, 2003d). The COP has the authority to create special working groups to supplement the work of the SBSTTA. A roster of experts is also maintained as a resource for science advice to the parties (Secretariat of the CBD, 2003a). METHOD Score sheet In order to gain a greater understanding of which organizational components will influence the output from a science advisory body, it is necessary to determine whether some design aspects are more important than others in influencing the usability of the output. In light of the numerous variables that must be decided upon in structuring a science advisory body, it was necessary to focus on components more likely to significantly influence the usability of the output.

8 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 8/42 In developing a score sheet to assess these factors for different conventions, it was also necessary to focus on factors that would exhibit some variation between different international environmental regimes. To a great extent, the factors applied in the score sheet were chosen based on knowledge of the international environmental conventions, and from a series of interviews with both decision-makers and providers of science advice in global environmental policy. Insight into the issues at stake was also drawn from sitting in on the negotiations to design a science advisory body 1. The autonomy of the body is an evaluation of the extent to which its output is influenced by one country in particular, or one particular organizational viewpoint. Along with the reputation of the scientists involved, this independence contributes significantly to improving the general perception of the science advisory body s work. The reputation of scientists involved was most cited as the most important factor of success in a series of interviews, while Rajendra K. Pachauri added that the autonomy from government and institution influence was also an important contributor to success 2. In his recent book, Edward Parson argues that science assessments achieve what he calls authoritativeness by involving a critical mass of the most respected experts on the issue (Parson, 2003). The diversity of the panel is a key component of creating a product that is of relevance to delegates from all parties to the convention. Environmental issues are interdisciplinary in nature, and as such, the product of the science advisory body will benefit from a panel which brings together scientists from all the disciplines concerned. Also, scientists may vary in their viewpoints as their institutional affiliation varies. Thus institutional diversity, i.e. whether a 1 Contact group on the Persistent Organic Pollutants Review Committee, Geneva, Switzerland. June more information available at: Earth Negotiations Bulletin Report of the Sixth Meeting of the Intergovernmental Negotiating Committee on Persistent Organic Pollutants (POPS INC-6) 2 Dr. Rajendra Pachauri is the chairman of the Intergovernmental Panel on Climate Change. The interview took place in Cambridge, MA on November 16, 2002.

9 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 9/42 participant is primarily affiliated with industry, a government authority, academia, an intergovernmental organization, or a non-governmental organization will also enhance the output s usability. Procedural factors will also affect the extent to which the product from the body will be used by decision-makers in the COP. This includes ensuring that potential participants to the science advisory body have received sufficient notice prior to participation. The results of the science advisory body will be of greater use to decision-makers if they address issues that reflect their principal concerns. Finally, the extent to which any advice is taken into consideration by decision makers will depend on the accessibility of the format in which the advice is presented. A more transparent process will contribute to the extent to which science advice is considered by decision-makers. While the success of the body is dependent on accessibility to its reports and products, transparency will also be enhanced by the extent to which information is made available on the preliminary proceedings leading to the final output as these can better reflect the dialogue involved. Thus transparency would be increased by making available minutes from the meetings or preparatory and preliminary reports. Acknowledging the global nature of these international environmental conventions, a key to transparency of the proceedings and results is the availability of these documents in a range of language that makes them accessible to a wider proportion of participants. In most international environmental conventions, proceedings and documents are available in the six official UN languages (English, French, Spanish, Russian, Chinese and Arabic) yet this is not always true of science advisory proceedings. Transparency will be enhanced by making materials available in the languages of relevance to the decision-makers concerned with the issue at hand.

10 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 10/42 The following score sheet, with four components, was used in assigning a usability score to science advisory bodies. This score is a function of autonomy, diversity, process and transparency. Each component is made up of criteria evaluated using a scale of 1 to 5, where: 5 = very good 4 = good 3 = average 2 = poor 1 = very poor The Autonomy component (A) has three criteria: resistance to influence by a single organization, resistance to influence by a single country, reputation of scientists. The maximum score for this component is 15. The Diversity component (D) has three criteria: disciplinary representation, country representation, and variety of institutional affiliations. The maximum score for this component is 15.The Process component (P) has three components: prior notice of meetings, ease of use of product, relevance of agenda. The maximum score for this component is 15. The Transparency component (T) has four components: availability of information on participating scientists, availability of minutes and proceedings, availability of final product, translation of final product. The maximum score for this component is 20. A Usability score (U) was derived using the following formula: Usability = 4* Autonomy + 3*Diversity + 2*Process + Transparency or U = 4A +3D + 2P + T The relative weights used in this formula reflect the relative importance assigned to each of these factors by interviewees and in the relevant literature. The maximum usability score is 155 points. According to this scoring system, science advisory bodies with the best usability would be

11 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 11/42 expected to have a score ranging between 124 and 155. A score of 124 is achieved by earning a 4 (good) in every criteria in the score sheet. Science advisory bodies with average usability would be expected to have a score ranging between 84 and 123. A score of 84 is achieved by earning a 3 (average) in every criteria in the score sheet. Science advisory bodies with worst usability would be expected to have a score of less than 84 points. Surveys Surveys were sent to participants of the most recent COPs of each of the conventions being studied. The survey was made up of two parts. The full surveys sent out for each convention are available in appendices 1 to 3. The first half of the survey gathered information about the respondent. Questions 1 to 3 focused on the respondent s experience and institutional affiliation. Questions 5 to 7 sought information on participation by scientists from the respondent s region and country in the science advisory process. Question 4 asked for the primary source of science knowledge while Question 8 asked respondents to rank a list of factors in order of their importance for a science advisory body whose output they would trust and use. The second half of the survey (questions 9 to 21) asked the respondent to fill out the same score sheet described above. Each criteria was set out as a statement and respondents used scores from 1 to 5, where 5 signified strong agreement, and 1 indicated strong disagreement. Questions 9 to 11 contributed to the diversity score, Questions 12 to 15 addressed issues of transparency. Questions 16 to 18 reflected the process and questions 19 to 21 related to autonomy. Surveys were sent to decision-makers associated with each of these conventions. Thirty surveys were sent via to randomly selected government delegates who had recently attended the 8 th Meeting of the Conference of the Contracting Parties (COP-8) to the Ramsar

12 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 12/42 Convention on Wetlands, held in November Thirty surveys were also sent to randomly selected government delegates who had recently attended the 14 th Meeting of the Parties to the Montreal Protocol on Ozone Depleting Substances and the 6 th Conference of the Parties to the Vienna Convention to Protect the Ozone Layer (MOP14/COP6), also held in November Contact information was not available for participants of a recent Conference of the Parties (COP) to the Convention on Biological Diversity and the survey was sent to thirty randomly selected government representatives listed as National Focal Points on issues relating to the CBD. The list had last been updated in December RESULTS AND ANALYSIS Score sheet Using the information available on each of the three Conventions and their subsidiary bodies provided in their official documents, on their website, and from other sources, I evaluated each of the criteria in the score sheet described above. Results are presented in table 2 at the end of this section. Ramsar Convention on Wetlands The Ramsar Convention promotes wetland conservation through their wise use, which is defined as sustainable utilization for the benefit of mankind in a way compatible with the maintenance of the natural properties of the ecosystem (Ramsar Convention Bureau, 2001d). The Scientific and Technical Review Panel (STRP) to the Ramsar Convention on Wetlands was established in 1993 as a subsidiary body to the Conference of the Parties, the Standing Committee, and to the Secretariat. Its composition and duties were further defined in Resolution VII.2. The STRP is currently composed of fifteen members from across Ramsar s

13 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 13/42 geographic areas and also includes representatives from the Convention s four partner organizations: BirdLife International, IUCN-The World Conservation Union, Wetlands International and WWF and other invited observer organizations (Ramsar Convention Bureau, 2003d). AUTONOMY When assessing autonomy, the STRP s geographically constrained memberships ensures that representatives from Ramsar s six regions are represented. Such distribution contributes to preventing any one country from exercising disproporational influence on the outcomes of the STRP. Ramsar obtains a score of good (4) for no influence by one country. The Convention s four partner organizations participate in the STRP. While these four organizations may at first be perceived as being similar environmental non-governmental organizations, their differing agendas help ensure that one of these organizations will not overly influence the results of the STRP. Institutional diversity is also enhanced by the participation of representatives from many observer organizations, such as from the science advisory bodies to other international environmental conventions (Ramsar Convention Bureau, 2001b). Ramsar obtains a score of average (3) for no influence by an organization. The STRP is set up to encourage the consultation of outside experts in developing their science advice, which can help ensure that the best scientists in the field of wetland management play a role in advising the COP and Standing Committee. In addition, the members on the STRP are often their country s specialist on wetlands issues thus providing a certain level of expertise (Ramsar Convention Bureau, 2001a). Finally, every party is asked to designate a national focal point for STRP matters which can contribute further to identifying the best scientists for

14 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 14/42 participation in the STRP (Ramsar Convention Bureau, 2000a). Ramsar obtains a score of very good (5) for reputation of scientists. A = = 12 DIVERSITY The members of the STRP bring to the panel a very wide range of areas of expertise, which is enhanced by access to outside experts and participation by the four partner organizations (Ramsar Convention Bureau, 2001b). Ramsar obtains a score of very good (5) for disciplinary diversity. The STRP s geographically constrained membership ensures that experts from Ramsar s six regions are represented. The triennial nomination to the panel ensures that over time different countries will be represented on the STRP. However, country diversity is in effect limited by the small size of the panel. Ramsar obtains a score of good (4) for country diversity. The participation in the STRP by the Convention s four partner organizations, along with the STRP s freedom in consulting outside experts helps ensure a certain diversity of affiliations. However, while academia, government institutes, and environmental NGOs are all represented, experts associated with private interests are rarely included in the STRP. Ramsar obtains a score of good (4) for institutional affiliation diversity. D = = 13 PROCESS The STRP meets annually, and the Ramsar secretariat maintains an extensive website with a detailed list of meetings (Ramsar Convention Bureau, 2003c). The last STRP meeting was held in June 2001, and members of the STRP also participated in the recent 8 th Meeting the

15 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 15/42 Conference of the Parties in November The date of the next meeting of the STRP has not yet been announced. Ramsar obtains a score of average (3) for prior notice of meetings. The STRP produces many reports submitted to the secretariat, the Standing Committee and finally to delegates to the COP meetings. The output and work of the STRP is presented to the COP in a large overview document, and the outcome of more specialized working groups overseen by the STRP is presented to the COP as information documents prepared by the STRP (Ramsar Convention Bureau, 2002). These documents are presented in a straightforward manner and successfully draw attention to policy implications of relevance to the COP. In some instances, the output is made easier to apply through the effective use of case studies (STRP, 2002a). These outputs would benefit from more succinct policy-maker summaries, especially in light of the many documents being submitted to the COP. The reports themselves are easily accessible on the website in multiple digital formats (Ramsar Convention Bureau, 2002). Ramsar obtains a score of good (4) for product in usable format. The STRP addresses issues set to it by the COP and is overseen by the Standing Committee, and as such can organize its work to investigate issues of priority to the COP (Ramsar Convention Bureau, 2000b). However, the STRP in its last meeting expressed concern at the lack of manpower for covering in a timely manner the issues on its agenda (Ramsar Convention Bureau, 2001c). Ramsar obtains a score of good (4) for address topics of priority to COP. P = = 11 TRANSPARENCY Information on participants is widely available on the Ramsar website and in its official documentation. Representatives to the STRP are listed by name, along with information about

16 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 16/42 their affiliation and areas of expertise. Further details on participants to the STRP are also available in minutes and documents from its meetings. Participants and leaders of the various working groups overseen by the STRP are also listed in STRP reports and documents. Ramsar obtains a score of very good (5) for information on participants. Detailed minutes, as well as summaries of the minutes and decisions, from STRP Meetings are available on the Ramsar website (Ramsar Convention Bureau, 2001c). Detailed information on the working groups is also made available (STRP, 2002b). Ramsar obtains a score of very good (5) for minutes and proceedings available. Final reports from the STRP are made available to delegates at the meetings of the COP. The STRP submits a general report of its work to the COP, along with numerous information papers on specific scientific concerns. These reports are prepared to be available to delegates 6 months prior to a COP meeting and are available on the Convention website and from the Secretariat. Ramsar obtains a score of very good (5) for final reports available. Minutes of the STRP meetings, and the general report to the COP are available in Ramsar s three official languages: French, English and Spanish. However, the information papers submitted to the COP which contain the more substantial science advice are only available in English. Ramsar obtains a score of poor (2) for translation. T = = 17 USABILITY U = U = 4A +3D + 2P + T = 126 The scores above combine to give Ramsar a final score of 126 points out of 155, which fits in the range of good science advisory bodies.

17 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 17/42 Vienna Convention to Protect the Ozone Layer The Vienna Convention for the Protection of the Ozone Layer was adopted in 1985 and was supplemented in 1987 by the Montreal Protocol on Substances that Deplete the Ozone Layer. Expert panels were initially put in place leading up to the negotiating of an Ozone layer treaty. Four panels were created: a science panel to report projected ozone losses under various emission scenarios, a technology panel to report the availability of substitutes and the feasibility of larger cuts, an environmental effects panel, and an economic panel (Parson, 2003). Now that the Convention has entered into force, these panels evolved into a Scientific Assessment Panel (SAP), and Environmental Effects Assessment Panel (EAAP) and a Technology and Economic Assessment Panel (TEAP). Their work is supplemented by seven Technical Options Committees (TOC). These panels and committees meet regularly and produce reports and recommendations to the Conference of the Parties of the Vienna Convention and the Meeting of the Parties to the Montreal Protocol. The members are independent scientists from organizations and institutions such as The World Bank, the US National Oceanic and Atmospheric Administration, and various academic institutions and private corporations from around the world (Ozone Secretariat, 2003a). AUTONOMY There are no requirements or controls to prevent undue influence by a particular country or organization in the work of the panels and technical options committees associated with the Vienna Convention and Montreal Protocol. However, there are no structures in place to systematically grant influence to any one particular actor involved. The Ozone Panels obtain a score of average (3) for no influence by one country and no influence by an organization.

18 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 18/42 The Ozone Panels function under a peer review system and involve many respected scientists in this field of ozone research. The Ozone Panels obtain a score of very good (5) for reputation of scientists. A = = 11 DIVERSITY The many panels and committees under the Ozone regime have ensured widespread disciplinary diversity in developing science advice. However, disciplinary diversity has been limited in the past, especially in terms of including a focus on interactions with climate change issues. This latter restriction however was reduced at MOP14/COP6 (Earth Negotiations Bulletin, 2002c). The Ozone Panels obtain a score of good (4) for disciplinary diversity. As described above, there are no controls on country affiliation and as a result panels and committees can easily be skewed towards overrepresentation by a region or country. The Ozone Panels obtain a score of poor (2) for country diversity. Scientists from many institutional affiliations are involved in the science advice provision to the Ozone regime, and include representatives from governments, academia, the private sector and non-governmental organizations. The Ozone Panels obtain a score of very good (5) for institutional affiliation diversity. D = = 11 PROCESS The dates of upcoming meetings of panels and committees are not announced far in advance, and are sometimes not listed on the Ozone secretariat website. The upcoming meetings section is rather limited, and one must generally refer to other sources to find out

19 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 19/42 about upcoming events (Ozone Secretariat, 2003b). The Ozone Panels obtain a score of poor (2) for prior notice of meetings. The assessment panels provide extensive reports to the MOP/COP that generally include succinct and accessible policy-maker summaries (Ozone Secretariat, 2002). The reports are also presented at length to participant of the MOP/COP thus making the product of the Panels all the more usable (Earth Negotiations Bulletin, 2002c). The Ozone Panels obtain a score of very good (5) for product in usable format. The work of the ozone panels and committees is in direct response to guidelines set for it by the MOP/COP. The Ozone Panels obtain a score of very good (5) for address topics of priority to COP. P = = 12 TRANSPARENCY Information on participants of panels is not easily available, and the most information available in some cases is only for the chairs of the panels and committees. In such cases, nationality, affiliation and qualifications are rarely disclosed (Ozone Secretariat, 2003a). The Ozone Panels obtain a score of poor (2) for information on participants. Proceedings of the meetings are not available, and the only output are in the form of reports published or submitted to the MOP/COP. The Ozone Panels obtain a score of very poor (1) for minutes and proceedings available. The final reports are available on-line and at meetings of the MOP/COP and are sometimes published in between meetings. The Ozone Panels obtain a score of very good (5) for final reports available.

20 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 20/42 The proceedings of the panel reports to the MOP/COP benefit from simultaneous translation in the six UN languages. The summary report submitted to the MOP/COP is also available in six languages, but more detailed reports from the panels are only available in English. The Ozone Panels obtain a score of good (4) for translation. T = = 12 USABILITY U = U = 4A +3D + 2P + T = 113 The scores above combine to give the Ozone regime a final score of 113 points out of 155, which fits in the range of average science advisory bodies. Convention on Biological Diversity The Convention on Biological Diversity was drafted in 1992 and its objectives include the conservation of biological diversity, the sustainable use of its components and the fair and equitable sharing of the benefits arising out of the utilization of genetic resources (Secretariat of the CBD, 1992b). The Conference of the Parties (COP) to the Convention on Biological Diversity is advised in science matters by the Subsidiary Body on Scientific, Technical and Technological Advice (SBSTTA) (Secretariat of the CBD, 2003d). The COP has the authority to create special working groups to supplement the work of the SBSTTA. A roster of experts is also maintained as a resource for science advice to the parties (Secretariat of the CBD, 2003a). AUTONOMY The SBSTTA meetings take place in a plenary format similar to that followed for COP meetings, and as such the SBSTTA is vulnerable to being held up by a minority of countries in trying to reach a consensus. The CBD obtains a score of very poor (1) for no influence by one country.

21 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 21/42 The CBD is a recent convention and has such is still prone to power struggles in its management. In particular, the United Nations Environment Programme has exercised an increasingly significant role in the management of the Secretariat to the CBD. The mandate of the CBD, and its SBSTTA is also influenced by actions of the World Trade Organization, especially as relating to biosafety issues. The CBD obtains a score of average (3) for no influence by an organization. The SBSTTA includes delegates from every party and some observer countries. These delegates often include both scientists and government negotiators, and experts are also hired as consultants by the Secretariat to provide science input to the SBSTTA. The CBD obtains a score of average (3) for reputation of scientists. A = = 7 DIVERSITY While the field of biological diversity is broad, there are few efforts within the SBSTTA to explore the full range of issues involved. In addition, each meeting of the SBSTTA is limited in its scope by the agenda set for it by the COP (Secretariat of the CBD, 2002). The CBD obtains a score of poor (2) for disciplinary diversity. Every party to the CBD is allowed to send as many delegates as they wish to meetings of the SBSTTA, and even governments that are not parties to the Convention are allowed to send delegates. The CBD obtains a score of very good (5) for country diversity. Due to the SBSTTA s plenary format, participation is mostly limited to scientists with government affiliations. While non-governmental organizations, intergovernmental organizations and industry are allowed as observers their participation is limited. There is little input from

22 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 22/42 scientists affiliated with academia. The CBD obtains a score of very poor (1) for institutional affiliation diversity. D = = 8 PROCESS Participants are given ample notice of SBSTTA meetings, the dates and venue of the next SBSTTA meeting generally being announced by the end of the preceding meeting (Secretariat of the CBD, 2003b). The CBD obtains a score of very good (5) for prior notice of meetings. The product of the SBSTTA is a detailed report (Secretariat of the CBD, 2001) submitted to the COP in the format of decisions. In a sense this output is already of a very political nature and does not contain scientific information that can easily be referred to in decision-making at the COP. The CBD obtains a score of average (3) for product in usable format. The SBSTTA does address issues set for it by the COP, yet the agenda is set with such advance that the SBSTTA is often ill equipped to advise the COP on issues of urgency. The CBD obtains a score of average (3) for address topics of priority to COP. P = = 11 TRANSPARENCY While a list of participants to the SBSTTA is distributed to those physically present at the meeting, participants list are not available on the CBD website and are even hard to obtain from the Secretariat. In any event, in this listing there is little if any information on affiliation or qualifications. While the CBD does maintain a roster of experts (Secretariat of the CBD, 2003a) that could be called upon by delegates, which does include information on nationality, affiliation and expertise, these experts are not involved in the outcome of the SBSTTA. The CBD obtains a score of very poor (1) for information on participants.

23 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 23/42 The proceedings of the SBTTA are available to a limited extent in the meeting report put out by the Secretariat (Secretariat of the CBD, 2001). A more detailed record is made available through daily reports form the Earth Negotiations Bulletin (Earth Negotiations Bulletin, 2001). The CBD obtains a score of good (4) for minutes and proceedings available. Final reports from the SBSTTA are available on the CBD website and are also distributed by the Secretariat. These reports also benefit from being available in all six UN languages. Yet it is more difficult to gain access to any technical reports presented to the SBSTTA (prepared by parties or the Secretariat) in which case they are only available in English. The CBD obtains a score of good (4) for final reports available. The CBD obtains a score of good (4) for translation. T = = 13 USABILITY U = U = 4A +3D + 2P + T = 87 The scores above combine to give the CBD a final score of 83 points out of 155, which fits in the lower end of the range for average science advisory bodies. Table 1 presents an overview of the score sheet results for the three conventions. Using these score sheets, the output from the STRP was found to be the most usable, while the output from the SBSTTA was found to be least usable. Wetlands Ozone Biodiversity No influence by organization (/5) No influence by country(/5) Reputation of scientists => AUTONOMY (/15) Disciplinary diversity (/5) 5 4 2

24 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 24/42 Country diversity (/5) Institutional affiliation diversity (/5) => DIVERSITY (/15) Prior notice of meetings (/5) Product in usable format (/5) Address topics of priority to COP (/5) => PROCESS (/15) Information on participants (/5) Minutes and proceedings available (/5) Final reports available (/5) Translation (/5) => TRANSPARENCY (/20) USABILITY (U = 4A + 3D + 2P + T) Table 1: results of score sheet Surveys: Thirty surveys were sent by to decision-makers associated with each Convention. Of the 90 surveys sent out, 8 surveys were unanswerable because the addresses were faulty. 22 surveys were returned, providing a response rate of 27%. Some survey respondents expressed difficulty in filling out the score sheet portion of the survey. This was the case in particular for respondents associated with the CBD who expressed doubt as to whether the SBSTTA could be considered a scientific body. Table 2 presents the range of results returned for the score sheet portion of the survey. If one were to consider the mean usability scores obtained by respondents, the Ozone regime and the Biodiversity Convention are identified as having the more usable science advice output. However, contrary to the results obtained when filling out the score sheet using the information made available by the Conventions themselves, the Wetlands Convention is found to have the least usable science advice output. It is most interesting to note that all three of the conventions had mean score sheet results falling in the average range.

25 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 25/42 Wetlands Ozone Biodiversity Influence by organization (/5) Influence from countries (/5) Reputation of scientists (/5) => AUTONOMY (/15) Disciplinary diversity (/5) Country diversity (/5) Institutional affiliation diversity (/5) => DIVERSITY (/15) Prior notice of meetings (/5) Product in usable format (/5) Address topics of priority to COP (/5) => PROCESS (/15) Information on participants (/5) Minutes and proceedings available (/5) Final reports available (/5) Translation (/5) => TRANSPARENCY (/20) USABILITY (U = 4A + 3D + 2P + T) Number of surveys sent out Number of surveys returned Mean Usability Score Table 2. Range of usability scores from survey respondents Since respondents who received the survey were only asked about their experiences with one convention process, it is difficult to compare the score sheet results obtained for these conventions. However, it is necessary to determine whether these inconclusive score sheet results are influenced by changing the formula for usability used above. Overall ranking of each factor 1. Reputation of scientists involved 2. Availability of result in accessible format 2. Autonomy from government influence Rankings assigned by respondents Wetlands Ozone Biodiversity 5, 1, 1, 9, 3, 2, 1, 1 1, 1, 4, 1 3, 1, 1 5, 6, 1, 6, 1, 8, 4, 1, 5, 1 5, 2, 2 1, 9, 2, 3 2, 10, 2, 8, 6, 1, 1, 2, 4 7, 6, 2, 4 1, 4, 2, 1

26 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 26/42 4. Diversity of 4, 8, 1, 3, 2, 7, 1, 3 3, 3, 4, 7, 3 3, 7, 1, 2 disciplines 4. Transparency of 7, 2, 2, 4, 11, 2, 4 2, 8, 1, 3, 7 1, 2, 1, 1 Process 6. Autonomy from 3, 7, 5, 2, 4 5, 7, 1, 5 1, 5, 2, 1 other organizations 7. Responsiveness to 9, 2, 4, 3, 4, 1, 2 6, 2, 3, 5, 6 2, 3, 4 policy-maker requests 8. Lack of 3, 2, 11, 5, 3, 3, 6, 3 4, 4, 9 5, 11, 2, 1 bureaucracy 9. Diversity of 10, 8, 6, 7 9, 6, 8 2, 6, 1, 3 nationalities 10. Translation of 4, 3, 5, 10, 10, 2, 5, 2 11, 11 2, 10, 5 results in multiple languages 11. Conducting 3, 9, 7, 9, 2, 5 10, 10 3, 8, 4 proceedings in multiple languages Other: autonomy from 1 political pressure from government and ngos Other: Experience 1 relevant to the issue Other: Authenticity of information 1 Table 3: rankings of factors of importance for trust and use of output. In ranking the most important factors for a science advisory body whose results will be trusted and used (table 3), it is interesting to note that overall respondents ranked the reputation of scientists involved as most significant. This, combined with the high ranking of autonomy from government influence, concurs with the formula used in developing the score sheet, which includes scientists reputation and autonomy from government within the most significant component, autonomy. More surprising, was the high ranking of transparency, which in contrast to its least significant role in the score sheet formula, was frequently cited as an important factor. These rankings were used to derive another formula in an attempt to better reflect these responses.

27 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 27/42 Based on the rankings of factors from respondents, the score sheet results were computed using the following formula: USABILITY = 10*(reputation of scientists) + 8*(product in usable format + autonomy from country) + 6*(information on participants + minutes and proceedings available + diversity of disciplines + autonomy from organization) + 4*(topics of priority to COP) + 2*(diversity of countries) + translation of documents Using this new formula, a good score would be in the range of 228 to 285 points, while an average score would fall in the range of 171 to 227 points. Any score below 171 would fall in the poor range. When this formula is applied to the evaluations I carried out of the three science advisory bodies, the STRP obtains a score of 246 out of 285, in the good range. The Ozone panels obtain a score of 202 out of 285, well in the average range. The SBSTTA obtains only 148, well in the poor range. Applying the formula derived from respondent preferences accentuated the variance in usability for the three conventions. When this new formula is applied to the survey responses for the score sheets, the STRP obtains a mean score of out of 285 points. The Ozone panels obtain a mean score of out of 285, while the SBSTTA obtains a mean score of 210 out of 285. All three of these results fall within the average score range. The first half of the survey provided insight in the sources of science advice commonly used by decision-makers at the COP. As shown in table 4, a majority of decision-makers sought their science knowledge from their government s science authority. The second most cited source of science knowledge is scientific peer-reviewed literature. It is interesting to note that, despite the high score achieved by the SBSTTA in the second half of the surveys, none of the respondents associated with the CBD chose the SBSTTA as a source of science knowledge.

28 Kohler Usability of Science Advice IHDP Open Meeting 2003 p. 28/42 Wetlands Ozone Biodiversity Government science authority Regional science authority Scientific peer-reviewed literature Mainstream media The convention s science advisory body Other: information distributed by convention secretariat Other: Universities, institutes Table 4: Number of times source of science knowledge chosen by respondents (some respondents chose multiple sources) The first half of the survey also provided information on respondents knowledge of participation by scientists from their region and country in the science advisory body. Table 5 highlights the emphasis on regional representation within the STRP to the Wetlands Convention. Wetlands Ozone Biodiversity From region: YES From region: NO From country: YES From country: NO Table 5: Are scientists from your region/your country involved in the provision of science advice? CONCLUSION The survey results highlighted that the factors identified by respondents as most significant do not vary greatly across the three conventions. Thus it may be possible to determine through further research which specific organizational details are most efficient at meeting the needs of decision-makers in providing usable advice. In particular the results above indicate that three factors may be most influential in determining the usability of the output of a science advisory body. These are: scientist reputation, autonomy from government influence and