IMMEDIATE, MEDIUM AND LONGER-TERM STRATEGY IN SCIENCE AND TECHNOLOGY STATISTICS

Transcription

1

2 IMMEDIATE, MEDIUM AND LONGER-TERM STRATEGY IN SCIENCE AND TECHNOLOGY STATISTICS International Review of Science and Technology Statistics and Indicators UNESCO Institute for Statistics, Montréal, 2003

3 UNESCO The constitution of the United Nations Educational, Scientific and Cultural Organization (UNESCO) was adopted by 20 countries at the London Conference in November 1945 and entered into effect on 4 November The Organization currently has 189 Member States. The main objective of UNESCO is to contribute to peace and security in the world by promoting collaboration among nations through education, science, culture and communication in order to foster universal respect for justice, the rule of law, and the human rights and fundamental freedoms that are affirmed for the peoples of the world, without distinction of race, sex, language or religion, by the Charter of the United Nations. To fulfill its mandate, UNESCO performs five principal functions: 1) prospective studies on education, science, culture and communication for tomorrow's world; 2) the advancement, transfer and sharing of knowledge through research, training and teaching activities; 3) standardsetting actions for the preparation and adoption of internal instruments and statutory recommendations; 4) expertise through technical co-operation to Member States for their development policies and projects; and 5) the exchange of specialized information. UNESCO is headquartered in Paris, France. UNESCO Institute for Statistics The UNESCO Institute for Statistics (UIS) is the statistical office of UNESCO and is the UN depository for global statistics in the fields of education, science and technology, culture and communication. UIS was established in It was created to improve UNESCO's statistical programme and to develop and deliver the timely, accurate and policy-relevant statistics needed in today's increasingly complex and rapidly changing social, political and economic environments. UIS is based in Montréal, Canada. UNESCO Institute for Statistics P.O. Box 6128, Succursale Centre-Ville Montreal, Quebec H3C 3J7 Canada Tel: (1 514) Fax: (1 514) uis@unesco.org UIS 2003 Ref: UIS/TD/03-02

4 Executive Summary This document was prepared in collaboration with the UNESCO Division of Science Policy and Analysis and the UNESCO Regional Office for Science and Technology in Latin America and the Caribbean. It presents issues and approaches that constitute the immediate, medium and longer-term strategy of the UNESCO Institute for Statistics (UIS) in the field of Science and Technology (S&T) statistics. This document describes the priority areas of S&T policy for which the UIS will collect and provide indicators, based on the results of an international consultation carried out during The issues have been prioritised according to the results of the consultation and taking into account the availability of resources needed. The first and immediate priority in this area will be to focus on data on human resources in S&T, establishing at the same time systems of input indicators, including financial and institutional resources for R&D. The development of appropriate innovation indicators for developing countries constitutes the main medium-term priority for the S&T statistics programme. In the longer term, the work programme will incorporate output indicators publications and patent counts. It will address methodological issues related to the measurement of the social impact of S&T, as well as how other areas, such as agriculture, health, energy, or environment, are impacted. In addition, this document covers issues related to data collection, analysing the factors to take into account in order to increase the coverage and quality of the current S&T statistics database of the UIS. It proposes a redesign of the complete data collection process and a greater focus on the production of analytical reports. The general strategy of the document is based on proposing ways for carrying out future work in the area, focusing on statistical capacity building, fostering demand and use of S&T statistics, collaborating with existing programmes, and increasing networking activities in the regions. Finally, more detailed activities for the periods , , and 2008 onwards are proposed, in line with the priorities outlined above. * * *

5 Table of Contents SECTION 1 Introduction 7 SECTION 2 Information needs and priorities for data and indicators Immediate term data and indicator priorities Human Resources Indicators Financial Resources Indicators Indicators regarding other resources Classifications related to S&T activities Medium term data and indicator priorities Innovation indicators Longer term data and indicators priorities Output indicators Impact indicators...18 SECTION 3 Data Collection, Quality, and Methodological Issues Data quality Data collection and production of statistics and indicators Dissemination and diffusion Missing data...23 SECTION 4 Strategy and Action Plan Capacity Building Fostering demand and use of S&T statistics and indicators Collaboration, co-operation and networking activities Methodologies Stages in the implementation of the strategy First stage Second stage Third stage

6 SECTION 1 Introduction 1. The World Conference on Science held in Budapest in 1999 placed emphasis on the necessity for countries to have access to reliable and policy-relevant statistics on science and technology (S&T): Governments should promote the further development or setting up of national statistical services capable of providing sound data, disaggregated by gender and disadvantaged groups, on science education and R&D activities that are necessary for effective S&T policymaking. Developing countries should be assisted in this respect by the international community, using the technical expertise of UNESCO and other international organizations. (Science Agenda - Framework for Action, World Science Conference, Budapest, 1999) 2. Following this recommendation, the UNESCO Institute for Statistics, jointly with UNESCO s Division of Science Analysis and Policy (SC/AP) and the Regional Office for Science and Technology in Latin America and the Caribbean (ROSTLAC), has carried out a fundamental international review of policy priorities and information needs in the fields of science and technology. 3. The objective was to identify current and emerging information needs and to prioritise these to serve as a basis for development of strategies for improving the relevance, availability and quality of S&T statistics in countries around the world, and, through this, to help define a new UNESCO statistical programme in the field of S&T. The ultimate goal is to revitalize efforts at both the international and national levels to build S&T statistical systems that are highly responsive to policy information needs, and to strengthen UNESCO s role in stimulating and facilitating this revitalization process, especially through its activities in standard setting and in the gathering of cross-nationally harmonised data. 7

7 4. UNESCO designed this fundamental review as a three-stage process. The first stage began with a meeting held in April 2002 with a group of international experts in the field of S&T statistics and policy to discuss policy information needs. The main outcome of this meeting was a Background Paper 1, which reviewed some of the current policy concerns and existing S&T data areas, systems and indicators. 5. The Background Paper was used as a starting point for the second stage of the review, aimed at obtaining a worldwide reflection and feedback from relevant institutions and experts within UNESCO Member States, as to the key S&T policy issues and information needs, in order to define the priority areas in which the UIS should develop its S&T statistical programme. The main instrument was a questionnaire sent to a large number of relevant stakeholders all over the world. The principal outcome of this consultation was a paper called Science & Technology Policy Priorities and Information Needs - Issues for a New Strategy in S&T Statistics 2, which analyses the priorities given by respondents to S&T policy issues, and the information needs related to these policy issues. The analysis is conducted by region, institutional background, type of involvement in S&T policy, and gender of respondents, showing interesting patterns. 6. The consultation paper presented a series of 19 recommendations for action. These recommendations are based both on the statistical analysis of responses received to the questionnaire, as well as on the comments and proposals contributed by respondents and experts. These recommendations relate to the following broad content areas: o Human resources o Financial resources o Innovation o Social impact of S&T o Public perception of S&T o Technology output o Bibliometric tools o Links between industry and university and non-university research centres o Patent statistics o Information related to S&T institutions, mechanisms and policies o Higher education statistics o Information society statistics. 1 Background Paper: Science & Technology Statistics and Indicators in Developing Countries: Perspectives and Challenges (UIS/S&T/2/BD1), Montreal, Document UIS/S&T/2/CP. 8

8 7. The recommendations also pointed out that the future UIS strategy in S&T statistics and indicators should focus on data quality, and should work through regional S&T indicators networks in developing countries, taking into account that the analysis found the region to be the most significant variable from the point of view of the design of a strategy. 8. In line with the recommendations, the UIS should organize capacity building programmes, diffuse, develop and adapt methodologies appropriate for developing countries in key priority areas, and carry out some analyses of the resulting data. 9. During the International Meeting Towards a New UNESCO Strategy in Science and Technology Statistics, held in Paris 24 to 26 March 2003, further advice was provided by experts both from producers and users communities, mainly from developing countries. The experts helped to define priorities within the context of the whole set of recommendations presented in the Consultation Paper, taking into account the limited resources available at UIS and the necessity of defining a clear niche for UIS work in the international context. 10. The present document introduces the basic elements for a new UNESCO Institute for Statistics immediate, medium and longer-term strategy in the field of Science and Technology Statistics. It builds on all the previous steps of the consultation process, taking into account complementary inputs and suggestions received from key experts in the field of S&T policy and indicators, as well as recommendations already stated in the external evaluation of UNESCO s S&T statistics activities produced by Rémi Barré in Section 2 of this document addresses the question of what issues should the UIS embrace within the broad field of S&T statistics and in the framework of the S&T policy priorities identified in the consultation process. The different areas of indicators selected are presented - grouped in immediate, medium and longer-term priorities - building especially upon the discussions held at the International Meeting held in Paris. 3 Rémi Barré, Les activités de l UNESCO dans le domaine des Statistiques de la Science et de la Technologie, Evaluation externe, Rapport final, UNESCO, décembre 1996 (BPE-97/WS/1). 9

9 12. Section 3 discusses how the UIS should conduct its activities regarding S&T indicators, in relation to the whole survey process. It analyses the strengths and weaknesses of the UIS S&T Statistics Database and discusses the importance of a dialogue with policy-makers. Statistical capacity building activities for the development of S&T indicators production systems at local level are proposed, as well as future UIS networking and co-operation activities. Finally, strategies for the diffusion, adaptation and development of methodologies in key areas of interest for groups of countries are discussed. 13. Section 4 briefly describes the stages for the implementation of the UIS strategy, relating the future activities to the different priorities presented in previous sections. 10

10 SECTION 2 Information needs and priorities for data and indicators 14. In the framework of the UNESCO International Consultation on S&T Policy Priorities and Information Needs, a number of different Science and Technology policy issues were found to be extremely important. 4 These issues were also the main focus of the information needs expressed during the consultation, and therefore have informed the main priority lines for the production of statistics and indicators. 15. The different thematic areas considered as highly important for the UIS programme need specific strategies, given not only the varying nature of the problems and the differences in existing S&T capacities in the Member States, but also the limitations in the amount of resources available at the UIS. 16. Therefore, three levels of priority were established immediate, medium, and longer term - taking particular account of inputs received during the International Meeting Towards a New UNESCO Strategy in Science and Technology Statistics, held in Paris in March Immediate term data and indicator priorities 17. In the immediate term, the Work of UIS will focus on activities related to collection and capacity building in the area of Human resources for Science and Technology, as well as other Input Indicators. 18. The current UIS S&T Statistics Database 5 is composed of data on R&D personnel and R&D expenditure. The collection of such data will continue to be carried out by UIS, giving special attention to statistical capacity building, with the aim of improving quality and coverage of the UIS database. 4 For detailed results of the consultation, please see the Consultation Paper (UIS/S&T/2/CP). 5 An analysis of the current contents of this Database is presented in section

11 19. UIS will also continue to take part in the follow-up of the Frascati Manual 6 revision process carried out by OECD s group of National Experts in Science and Technology Indicators (NESTI), as well as in projects aimed at developing complementary methodologies for applying and/or adapting this manual to the specific conditions of developing countries Human Resources Indicators 20. Human Resources is the highest S&T policy priority, obtaining a worldwide consensus. A programme to develop and collect better human resources statistics, focusing both on personnel active in R&D and issues related to the broader concept of human resources devoted to S&T (HRST) is needed. This last concept includes the measurement of stocks and flows of professionals and practitioners (and not only of researchers). 21. Therefore the scope of data collection and studies will be broadened to fields, such as S&T employment; science education; recruitment to science, engineering and technology careers; retention of scientists; and international mobility of highly qualified people. The development of indicators in the field of science education is especially significant, given the generally recognised need to improve, strengthen and diversify science education at all levels and for all sectors, and therefore integrate science into the general culture, contributing to the formation of open and critical thinking, as well as to the improvement of people s ability to meet the challengers of the knowledge society In order to study the inflows into the set of Human Resources devoted to S&T, higher education statistics will be analysed, in close co-operation with the UIS staff specialising in education statistics, building on data provided by existing education surveys. Trends in secondary -and where appropriate primary- education will also be taken into account in these analyses. Complementary, alternative sources of HRST data will be sought, such as labour force surveys or national population censuses. 6 The Frascati Manual, Proposed Standard Practice for Surveys on Research and Experimental Development, OECD, Paris, 2002, presents the most developed and worldwide-accepted standard methodology for the collection of R&D statistics. 7 See UNESCO-CASTME, Science, Technology and Mathematics Education for Human Development Framework for Action, Paris, December

12 23. Complementary to the production of survey-based statistics on R&D personnel, the establishment of national registers of researchers possibly in the framework of more comprehensive national S&T information systems- will be promoted, especially in countries lacking a robust S&T statistical system. These tools may contribute to the production of indicators disaggregated by gender, age or other demographic characteristics, as well as for studies of careers development and migration patterns. In this regard, they constitute instruments for the better production of national statistical data and, therefore, for improving the quality of statistics provided to the UIS for the design and follow-up of S&T policies, as well as for many different kinds of studies. Exchange between existing initiatives in this area (such as Scienti 8 in Latin America) will be fostered, exploring the possibility of establishing international standards. 24. Special attention will be given to the problems involved in the measurement of international mobility of highly skilled people, in the search for adequate methodologies and data sources to permit a quantitative approach to the analysis of this process, frequently called brain drain. 25. A first evaluation will be carried out in order to assess to what extent Human resources in S&T data (in its broader sense) are collected worldwide, and the level of international comparability of available data. This study will take into account the different criteria presented in the Canberra Manual 9, and will propose strategies for expanding to interested countries the measurement of these variables, especially the measurement of the stock of professionals. 26. UIS will participate in OECD s and Eurostat s efforts to revise the Canberra Manual, and will also promote the inclusion, in national studies, of appropriate questions and the utilization of international classifications in different types of statistical instruments, including population censuses, household surveys, and employment surveys or registers. 8 The International Network of Information and Knowledge Sources for Science, Technology, and Innovation Management (Scienti) supports research and development, implementation, and operation of information systems, portals, and information technology tools in the areas of science, technology, and innovation. One of its projects is the CvLAC System (Latin American and Caribbean Curricula Vitae), considered the base for a national S&T management tool. 9 Manual on the Measurement of Human Resources Devoted to S&T "Canberra Manual", OECD/GD(95)77, OECD, Paris

13 27. Special attention will be paid to gender mainstreaming in S&T, including the production and analysis of indicators and disaggregated data by gender Financial Resources Indicators 28. Financial Resources also emerged from the consultation as a highly important S&T policy issue. Indicators on financial resources provide a direct measure of the S&T effort of Member States. At the same time, they constitute a necessary complement to human resources data. They present, together with data related to infrastructure and institutions, a proxy to the quality of the environment in which researchers develop their work. Many issues included under Human Resources, such as the international mobility of researchers, are closely related to the availability of financial resources in developing countries. Hence, there is a need to strengthen the collection of data on expenditure on R&D, providing help where appropriate in order to create local capacities for this type of data collection. In this regard, ways of adapting Frascati Manual guidelines to the characteristics of certain groups of developing countries need to be evaluated. International comparability of R&D data should nevertheless be sought as widely as possible Indicators regarding other resources 29. In some regions, especially in countries with a lower level of development, the consultation revealed an important demand for indicators related to infrastructure, institutions and policy mechanisms. Therefore, information on institutional capacities will be collected and in the medium term some methodological guidelines will be proposed for establishing databases of infrastructure and institutions, in close relation to the work on databases and registers of researchers Classifications related to S&T activities 30. In close collaboration with OECD and Eurostat, UIS will be active in the revision and diffusion of pertinent classifications: o Field of Science classifications (in close co-operation with the development of an ISCED Field of Studies classification); and o Socio-Economic Objectives (this revision will be conducted in collaboration with Eurostat s revision of NABS Nomenclature for the analysis and comparison of scientific programmes and budgets proposed for the 4 th quarter of 2003 and the first half of 2004). 14

14 2.2 Medium term data and indicator priorities 31. In the medium term, UIS will devote attention to the problems involved in the development and collection of data in the field of Innovation Indicators, devoting special attention to the particularities of innovation activities in developing countries Innovation indicators 32. The high importance given during the consultation process to issues such as Economic Impact of S&T, and Innovation is clearly related to a general perception of the fundamental contributions that Science and Technology bring to development. Nevertheless, the use of data to explore this relationship has not been that widespread. Recent economic theories propose that certain characteristics of Systems of Innovation and of the technological behaviour of the firms are useful to explain the different paths of developments countries take. Therefore, indicators related to these issues need differentiated treatment by groups of countries, taking account of marked differences in national systems of innovation and in capacities for producing high technology output throughout the world. 33. As a first step to the introduction of innovation statistics into the UIS working plan, priority will be given to the participation in international activities in this area. One of the lines will be to continue the involvement in activities of OECD and Eurostat, such as the Oslo Manual 10 revision process, with a particular interest in contributions from the perspective of developing countries. Another line of action will be to analyse the existence of innovation surveys in developing countries, and in partnership with others, the analysis of regional experiences, such as the application of RICYT s Bogotá Manual 11 in Latin America, in order to evaluate promoting its possible application in other regions. 10 Oslo Manual, Proposed Guidelines for Collecting and Interpreting Technological Innovation Data, OECD and Eurostat, Paris, Standardisation of Indicators of Technological Innovation in Latin American and Caribbean Countries, Bogotá Manual, prepared by Hernán Jaramillo, Gustavo Lugones, and Mónica Salazar, Ibero American Network of Science and Technology Indicators (RICYT) / OAS / CYTED COLCIENCIAS / OCYT, Bogotá and Buenos Aires,

15 34. In a second step in this direction, UIS will study the ways of adapting international methodologies and indicators to measure innovation in all sectors of the economy in a way more appropriate to developing countries. This involves issues such as: Measurement of innovation in agriculture and other sectors not always considered in current innovation measurement practices; Promoting the use of indicators reflecting sub-national (also called regional) innovation systems, including those related to linkages among the actors and the outcomes of the activity of innovation; and Measuring minor or incremental innovation 12, which in some developing countries can be the most frequent type of innovations, as well as innovative applications of existing products or processes, using tools such as surveys of use of technologies and management practices. 2.3 Longer term data and indicators priorities 35. Some of the issues which obtained highest priority during the consultation, especially those related to the measurement of the impact of Science and Technology will have to be addressed in the longer term, due to the need to choose a small number of key tasks affordable for a team with limited resources. 13 Importantly as well, robust indicators systems of S&T inputs and outputs are first needed to provide framework in which impact indicators are used to explain the consequences of S&T research on different societal and economic variables. 36. Since some degree of work is being carried out in more developed countries on these issues, especially in the context of OECD, UIS will carefully follow developments in this area. 12 Following the Oslo Manual (paragraph 63), radical innovations shape big changes in the world, whereas incremental innovations fill in the process of change continuously. While incremental innovations are considered in the theoretical framework of the Oslo Manual, they are not always taken into account when it comes to specific methodologies and practical measurement issues in actual surveys. 13 Nevertheless, some of these issues might be incorporated into previous stages of the S&T statistics programme if additional resources are obtained for such projects. 16

16 2.3.1 Output indicators 37. Current tools for the production of Scientific output indicators, mainly publication and citation counts, are biased towards the scientific production of developed countries. The utility of bibliometric indicators based on the Science Citation Index and other indexes produced by the Institute for Scientific Information (ISI) 14 is often questioned since scientific production in developing countries does not necessarily follow the same path towards publication as it does in other countries in the world. Not every discipline in developing countries publishes results and scientific communications in journals registered by the ISI, nor have scientific journals in developing countries the profile to be included in the international databases. In spite of these difficulties, these indicators are not easily replaceable for measuring scientific outputs because of the absence of other data. The more consolidated or internationalised areas of science in some developing countries are gradually accepting the use of bibliometric indicators The design and application of bibliometric tools adapted for the analysis of scientific output in developing countries is therefore one of the lines of work to be carried out in the longer term. The work on this issue will include exploring the possibilities of utilizing different sets of databases for the production of indicators. 16 In order to advance in this area of work, partnerships with organizations experienced in the work with bibliometric indicators will be established. Another possible aspect of this work would be to analyse ways of promoting the creation and expansion of national and regional (peer-reviewed and high standard) publication databases, which could help to improve the measurement of developing countries scientific output. 14 ISI ( is the publisher of the Science Citation Index, Social Science Citation Index and Arts and Humanities Citation Index, the most widespread tools used for bibliometric analysis, as well as the Web of Science. 15 See Background paper (UIS/S&T/2/BD1). 16 Work in this direction has been carried out by RICYT (Principales Indicadores de Ciencia y Tecnología Iberoamericanos / Interamericanos 2001, in Latin America, using 8 international databases, and by academic researchers as in Rigas Arvanitis et al. Science in Africa: A Bibliometric Panorama Using PASCAL Database, Scientometrics 47 (3): , March - April, 2000, or David Roessner et al. Indicators of Technology-based Competitiveness: Incorporating Recent Changes in the Concept, "High-technology," and in Data Availability, Final Report to the National Science Foundation, Georgia Institute of Technology, December 2001, which uses INSPEC and Compendex for bibliometric studies in different countries of the world. 17

17 39. The incorporation of indicators of Technology Output, like the Technology Balance of Payments, indicators regarding International Trade of High Technology Goods and Services, or Patent Statistics in the UIS database also require specific data collection and capacity building activities, as well as increased partnership with international organizations in charge of the production of other international statistics, such as the UN Statistics Division, World Intellectual Property Organization (WIPO), European Patent Office (EPO), United States Patent and Trademark Office (USPTO), and Japan Patent Office (JPO). Data from these last three institutions reflect the most important patents worldwide, and feed the OECD s Triadic Patent Family 17 project. Activities related to these issues will also be part of the UIS longer-term strategy Impact indicators 40. An important demand for indicators related to Social impact of S&T has been evident throughout the consultation process. Since there is currently no agreement about the exact definition of such indicators, nor about the actual way of measuring social impact of S&T, a great deal of research and development activity would be needed in this field. In the longer term, an R&D programme will be prepared in order to develop indicators of social impact of S&T. This research programme would bring together interested groups from different regions, including particularly researchers from Africa and Latin America, since these regions gave higher priority to this subject throughout the consultation process. The possibility of launching an open call for projects on this issue will be evaluated. 41. The necessity of quantifying different aspects of the Impact of S&T on agriculture is another one of the issues related to impact to which high importance was given by respondents from Africa and Latin America & the Caribbean during the consultation. Specific indicators in this field have still to be developed, in close co-operation with International and local agencies specialized in this field, such as Food and Agriculture Organization of the United Nations (FAO), International Fund for Agricultural Development (IFAD), or regional organizations. In other sectorial priority areas, such as health, environment, or energy, similar strategies will be developed in the longer term A patent is a member of the Triadic Patent Families if and only if it is filed at the European Patent Office (EPO), the Japanese Patent Office (JPO) and is granted by the US Patent & Trademark Office (USPTO). 18 Issues like the impact on industry or on small and medium enterprises are included under the term economic impact and treated in section

18 42. Public perception of S&T is one of the areas of the social impact of S&T that was frequently stated as high priority. Public and private institutions in countries with different levels of development have carried out exercises in order to measure these variables. Nevertheless, up to now it has been difficult to construct comparable cross-national statistics on this issue, particularly taking into account the different cultural issues involved. Studies will be carried out in order to assess the possibility of developing crossnationally comparable statistics on public perception of S&T drawing on the experience of the US National Science Foundation, as well as other institutions and organizations active in this field in countries with different levels of development. 43. The UIS programme in S&T statistics will be closely related to the work on information society statistics 19, carried out by UIS staff specialising in communication statistics, feeding the corresponding programme with S&T data related to information society topics. 19 The relation between S&T and information society statistics can even be tracked back to the definition of Scientific and Technological Activities by UNESCO in the sixties. An updating of this concept long overdue- would naturally include in the definition of S&T Services many of the aspects included today under the Information Society framework. 19

19 SECTION 3 Data Collection, Quality, and Methodological Issues 44. As discussed during the International Meeting Towards a New UNESCO Strategy in Science and Technology Statistics held in Paris, in addition to consideration of content issues - this is, what type of information will be collected - it is important to consider the whole survey process, encompassing the questionnaire design, data collection, problems with vast quantities of missing data including lack of regular data reporting for significant numbers of countries, the need for an improved database for data access and dissemination, and the need for improved analytical outputs and publications. 45. Regarding data collection, quality, and methodological issues, UIS actions in the field of S&T statistics will be guided by the following main principles: o Foster data quality (through investing in and expanding the capacity of the human resources devoted to the programme, and through attention to sound methodological development of indicators, means of data collection, processing and analysis); o Investment in systems and vehicles for dissemination and diffusion of program outputs (including data and indicators via the Internet, methodological and analytical publications available via the Internet and in print format); and o Increase over time the number of countries providing high quality data with sensitisation of policy-makers, and statistical capacity building and technical assistance being key elements. 3.1 Data quality 46. Improving data quality is one of the main priorities in all areas of statistics gathered at UIS. In the case of S&T statistics, the improvement of data quality is especial priority, since it has been somewhat neglected over the years. Therefore, work in the area will take into account the following dimensions of quality: o Relevance to policy, through continuing dialogue and consultation with policy-makers, the UNESCO Science Sector and policy experts; o Validity and reliability; o Consistency over time and space; o Comparability through standards; o Accessibility and affordability, through free access to the information by different means; o Potential for disaggregation; 20

20 o Currency and punctuality; o Clarity and transparency; o Coherence across sources; and o Efficient use of resources. 47. In order to formalize consideration of data quality issues, the possibility of extending the Data Quality Assessment Framework (DQAF) developed and applied to a number of subject matters by the International Monetary Fund (IMF) 20 to S&T statistics or other methodologies will be studied. 3.2 Data collection and production of statistics and indicators 48. In order to contribute to the objective of producing high quality statistics and indicators, the UIS will improve its own capacities both in the area of S&T statistics and S&T indicators. 21 Step by step, the team responsible of S&T statistics and indicators will be strengthened, both from the quantitative and qualitative point of view. In conjunction with these developments, consideration will be given to establishment of a scientific committee, in order to advise and assess the UIS S&T statistics and indicators activities, from the point of view of scientific relevance and methodological quality. Alternately, use of time- and task-limited expert committees on special topics as the need arises will be considered. 49. Moreover, all stages of current data collection and production of statistics and indicators will be reviewed in the immediate term, as listed and elaborated below: o Questionnaire design and survey strategy, reflecting the information needs and priorities identified in this report, based on the consultation findings; o Professional relations with data users and producers, including a survey of Statistical Capacity Building needs as a preliminary step towards initiation of an appropriate Programme; o Procedures for data processing, including identification of the need for development of procedures for dealing with missing or erroneous data; o Production of indicators, including identification of needs for documentation of methodologies of existing indicators, and for development of new indicators; 20 See Carson, Carol S. "Toward a Framework for Assessing Data Quality," IMF Working Paper, WP/01/25. Washington: International Monetary Fund, 2001, 21 In drafting this item, special attention was given to the recommendations included in the abovementioned report (BPE-97/WS/1) prepared by Rémi Barré. 21

21 o Design of a database for dissemination and retrieval of data and indicators via the Internet; o Analysis, diffusion and publication, including opportunities for collaboration with partners; and o Quality assurance and the incorporation of quality at all stages of the process. 50. The existing R&D questionnaire will be revised and simplified, in order to motivate the respondent with questions which are unambiguous and relevant to their circumstances. The revision will take into account itemnon-response rates experienced to date. The questionnaire will be harmonized with the ones of partner organizations, and will be based on a modular approach, with universal core questions and different additional groups of questions for different regional groups of countries. Alongside this, a new user-friendly electronic questionnaire will be developed, giving respondents the option of responding electronically or via the paper questionnaire. The periodicity of regular surveys will be re-examined to consider the trade-offs between annual versus less-frequent data collection. Regular data collection may be an important element in building and retaining capacity within countries. The possibility of carrying out special surveys for some new indicator areas will be analysed. Taking into account the intimate relation existing between R&D and higher education in many countries, the possibility of integrating some questions on R&D into tertiary education questionnaires will also be studied. 51. In order to establish mutually beneficial relations with data producers, direct contact and communication with institutions responsible for national S&T data will be established, except in regions where a UIS has agreements with regional organizations to undertake the collection of national data. This dialogue will be ongoing, and will have the aims to deepen the UIS understanding on local circumstances, and to help to increase quality and commitment to data production in the countries. Special attention will be devoted to the identification of proper dataproviders in the different countries. With this information, a new register of respondents to the UIS S&T questionnaire will be constructed and maintained, in order to improve the links with institutions and professionals directly involved in the production of S&T data at national level. 52. The UIS database will be revised, especially regarding its technological implementation, and ways of improving data access through the Internet will be developed. 22

22 53. Quality assurance methods will be incorporated into the whole process of data collection, processing, analysis, publication and responding to external demands. 54. The production of statistics is based on the collection of results of S&T surveys carried out by Member States. The production of indicators includes the use of statistics from the UIS database, as well as from other sources. Some of the data used may be related to national statistical systems, or those collected by UNESCO; OECD, or Eurostat, and others might come from other sources, especially on scientific publications, patents, and international trade. 55. In some cases, the production of indicators using external, and especially commercial, databases will be subcontracted to research groups having the capacity and the sources to carry out this work. This will depend critically upon the financial resources. When deemed appropriate, open calls will be launched, encouraging also the presentation of proposals from developing countries R&D groups. The UIS team will closely manage any subcontracted activities, in order to increase its capacities and carry out quality control. 3.3 Dissemination and diffusion 56. The information produced by the UIS will be widely diffused, especially making use of electronic methods including the Internet. Diffusion activities might include regional meetings to discuss results of data collection with producers. 57. Based on UIS data supplemented by other sources, a series of analytical publications will be prepared focussing on key issues of the S&T agenda (i.e. S&T workforce and mobility, gender in S&T, innovation in developing countries). 3.4 Missing data 58. The structure of the UIS S&T Database includes indicators on human and financial resources. Pilot data collection started in 1967, and regular data were compiled from 1970 onwards. The scope of the database includes all 189 UNESCO Member States. The UIS S&T Database is therefore the most comprehensive database on these issues available worldwide. One of the strengths of UIS mandate is that Member States are obliged to provide data to UNESCO, even if this requirement does not always translate to reality. 23

23 59. The following statistics are included in the database: Human resources (number of personnel engaged in R&D) o by category of personnel (researchers, technicians and equivalent staff and other supporting staff); o by sector of performance; o by field of science and technology; and o by branch of economic activity. Financial resources (gross domestic expenditure on R&D) o by type of expenditure; o by sector of performance; o by field of science and technology; o by source of funds; o by branch of economic activity; o by major socio-economic objective; and o Current intramural expenditure by type of R&D activity. Government budget appropriations or outlays for R&D o by major socio-economic objective; and o by sector of destination. 60. Based on these statistics, S&T indicators are produced and published, together with basic metadata and definitions, on the UIS website 22. The current list, which will be thoroughly revised following the recommendations included in this Strategy Paper, is: Human resources: o Total R&D personnel (Full time equivalent FTE); o Researchers, by gender (FTE); o Researchers (FTE) per million inhabitants; o Technicians, by gender (FTE); o Technicians (FTE) per million inhabitants ; o Other supporting staff (FTE); and o Number of Technicians by researcher. Financial resources: o Total Gross Domestic Expenditure on R&D (national currency); o Gross Domestic Expenditure on R&D by source of funds (percentage); o Total Gross Domestic Expenditure on R&D as a percentage of GNP; o Total Gross Domestic Expenditure on R&D per capita (national currency); and o Average Gross Domestic Expenditure on R&D per researcher (national currency)

24 61. From 1998 on, data were not collected from OECD Members in order to reduce reporting burden. 23 For these countries, the OECD Secretariat provided data collected by them. As presented in the following table, non- OECD countries can therefore be grouped based on the number of times they responded to UNESCO questionnaires over the period from 1990 to 2001 as follows: (a) countries responding in 4 or more years; (b) countries responding in 1 to 3 years; and (c) countries responding in 0 years In 1996 OECD countries already received an abridged questionnaire, but still provided data directly to the then UNESCO Statistics Division. 24 It should be noted in this analysis the threshold for being counted as a respondent was the return of a questionnaire with any data completed. Hence countries considered as respondents might nevertheless have suffered from large amounts of missing data. 25

25 Responses to UIS S&T questionnaire received from Member States*, 1990 to 2001 Countries responding in 4 or more years AFRICA (11) Zambia Kyrgyzstan Latvia Bolivia St. Helena Burkina Faso ASIA (15) Malaysia Moldova Chile Uruguay Cameroon Afghanistan Mongolia Romania Colombia Venezuela Central African Republic Bahrain Oman Russian Federation Costa Rica OCEANIA (2) Congo China Singapore Serbia and Montenegro Cuba French Polynesia Egypt Cyprus Thailand Slovenia Dominican Republic Samoa Guinea Georgia EUROPE (12) Former Yug. Rep. of Macedonia Ecuador Madagascar India Belarus Ukraine El Salvador Senegal Israel Bulgaria LATIN AMERICA&CAR.(16) Guyana Tunisia Kazakhstan Croatia Argentina Panama Uganda Kuwait Estonia Brazil Peru TOTAL=56 Countries responding in 1 to 3 years AFRICA (6) Togo Iran, Islamic Republic of Qatar Lithuania OCEANIA (1) Burundi ASIA (15) Jordan Sri Lanka Monaco Guam Mauritius Azerbaijan Macau Syrian Arab Republic LATIN AMERICA&CAR.(3) Rwanda Bangladesh Maldives Tajikistan Bermuda Seychelles China, Hong Kong SAR Pakistan Uzbekistan Nicaragua South Africa Indonesia Philippines EUROPE (2) Trinidad and Tobago TOTAL=27 Countries responding in 0 years AFRICA (37) Lesotho ASIA (16) Andorra Grenada American Samoa Algeria Liberia Armenia Bosnia and Herzegovina Guadeloupe Cook Islands Angola Libyan Arab Jamahiriya Bhutan Gibraltar Guatemala Fiji Benin Malawi Brunei Darussalam Holy See Haiti Kiribati Botswana Mali Cambodia Liechtenstein Honduras Marshall Islands Cape Verde Mauritania Iraq Malta Jamaica Nauru Chad Morocco Korea, Dem. People's Rep. San Marino Martinique New Caledonia Comoros Mozambique Lao People's Dem. Rep. LATIN AMERICA&CAR.(29) Montserrat Niue Congo, Dem. Rep. of the Namibia Lebanon Anguilla Netherlands Antilles Norfolk Island Côte d'ivoire Niger Myanmar Antigua and Barbuda Paraguay Pacific Islands Djibouti Nigeria Nepal Aruba Puerto Rico Papua New Guinea Equatorial Guinea Reunion Palestine Bahamas St. Kitts and Nevis Solomon Islands Eritrea Sao Tome and Principe Saudi Arabia Barbados St. Lucia Tokelau Ethiopia Sierra Leone Turkmenistan Belize St. Pierre and Miquelon Tonga Gabon Somalia United Arab Emirates British Virgin Islands St. Vincent and the Grenadines Tuvalu Gambia Sudan Viet Nam Cayman Islands Suriname Vanuatu, Republic of Ghana Swaziland Yemen Dominica Turks and Caicos Islands Guinea-Bissau United Republic of Tanzania EUROPE (8) Falkland Islands (Malvinas) U. S. Virgin Islands Kenya Zimbabwe Albania French Guiana OCEANIA (16) TOTAL=106 Note: * OECD countries (Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Netherlands, New Zealand, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland, Turkey, United Kingdom, United States) are not included. All other Member States and territories that received the questionnaire in the corresponding timeframe are included (including some non-independent territories), and Singapore (non-member State). 62. However, the database does not include data for every indicator for all the countries and for every year. In fact, data density is rather low. Moreover, there is a marked lack of data from many developing countries and in particular African countries. 26

26 63. Besides availability issues, the quality of the data contained in the database is not homogeneous. The following analysis is based on the responses given by Member States to the two last questionnaires, sent out in 1998 and During the two last rounds of UIS data collection, around 50 non-oecd countries filled in the questionnaires each year. Among these countries, 10 were from Africa, 16 from Asia, 10 from Latin America and Caribbean, 11 from Europe and 2 from Oceania. In terms of regional coverage, only 18% of African countries filled the questionnaires, 10% of Oceania, 32% of Asia, 35% of Europe (non OECD) and 20% of Latin America and Caribbean countries. 65. Nevertheless, regional publications, such as RICYT s Principales Indicadores de Ciencia y Tecnología Iberoamericanos / Interamericanos indicate that many Latin American countries do collect R&D data. This might show that the channels through which UIS has been operating up to now need to be revised. This issue will be further discussed in Section Although 90% of the questionnaires provided data at an aggregate level with relatively good quality 25, disaggregated data were of poor quality and were not often available at all (only around 40% of the questionnaires provided disaggregated data on R&D expenditure). 67. The lack of data in many countries could have different origins and causes: Lack on information on methodologies, and difficulties for applying the Frascati Manual (definitions, concepts and nomenclatures); Weaknesses in data collection mechanisms (no unit responsible for the collection and centralisation of S&T data within the countries); Lack of data in countries stemming from low demand by S&T policymakers for S&T data for decision-making; and The complexity of the UIS questionnaires (i.e. being too detailed for some countries). 25 The quality criteria refer to coherent and consistent data across tables within the questionnaire. 27