Exploring the Missing Links Between Causes and Effects

Transcription

1 Exploring the Missing Links Between Causes and Effects New Conceptual Framework for Understanding Micro- Macro Conversions in Programme Evaluation By Petri Virtanen & Petri Uusikylä Please do not quote without authors permission An paper to be presented in 5 th EES Conference in Seville, Spain, October Contact information: Net Effect Ltd., Mikonkatu 13 a 34, FIN Helsinki, Finland Tel , Fax petri.virtanen@neffect.net, petri.uusikyla@neffect.net Summary Programme evaluation has become a widely applied mode of systematic inquiry for making judgements over public policies. Although evaluation, as a form of systematic inquiry, has provided crucial feedback information for the policy-makers, it still too often comes up with self-evident answers to complex and multidimensional societal problems. In this paper, the authors take a close look on the ontological premises of the of the program (as well as project) evaluation, the discussion on causality, and their relations to the rational theory of action. Considering the role of causal reasoning in different evaluation paradigms is actually a question of evaluation paradigm crisis. It seems that different evaluation paradigms differ quite considerably as a whole (as a research strategy), but particularly in terms of mapping out causal links between causes and effects. According to the authors view, a new understanding is needed for causal interpretation of causes and effects, and this kind reasoning would at least to some extent strengthen the core ideas which lie behind the currently well-spread and consolidated realistic evaluation tradition. Traditional cause effect logic totally undermines the fact that effects of a programme are always caused by real actors rather than constructed ideal actors. A programme always influences through societal actors that base they courses of actions on interpretations on expected logic of a programme. The paper concludes with a proposed research agenda to be implemented by the authors during Key words Programme evaluation, evaluation paradigms, causality, micro-macro conversion

2 1. Introduction Programme evaluation has become a widely applied mode of systematic inquiry for making judgements over public policies. Evaluations have been used ex-ante (testing the coherence and applicability of the policy), ex nunc (assessing the implementation and intermediate results of the policy) as well as ex post (analysing the final results and outcomes of the policy). Although evaluation, as a form of systematic inquiry, has provided crucial feedback information for the policy-makers, it still too often comes up with self-evident answers ( that we already knew but now it has been proven ) to complex and multidimensional societal problems. Why is that? Are evaluators not qualified enough to capture the true dynamics of public policies or understand the cause-effect chains of those policies? Or is the reason for niceto-know-evaluations that policy makers want to have easy answers or results that tend to give support to the decisions made? Or is the whole concept of programme evaluation mission impossible, i.e. is it simply impossible to measure causal effects caused by public policies? Our answer would be partly all of these but none alone. To come up with a satisfactory answer we need to explore the ontological premises of the of the program (as well as project) evaluation, the discussion on causality, and their relations to the rational theory of action. This paper first discusses the epistemic nature of causality in programme evaluation starting from the original ideas presented by Francis Bacon and David Hume centuries ago. Secondly, we highlight some axiomatic postulates of programme management theory and thereafter discusses the relationship between these postulates and in paradigmatic terms three alternative models of public policy evaluation. These foundations are then, thirdly, critically challenged with some of the core arguments of social theory of action and its commentaries, mainly coined by James S. Coleman (1990). Fourthly, linking all of the previous approaches and starting points together, a new conceptual model for programme evaluation is presented in order to highlight the ideas introduced in this paper. The last part of the paper also includes concrete research agenda. 1

3 2. The Strong and Weak Postulates of Causality Assumption in Programme Evaluation Science does not just describe the world. It also gives us explanations of how and why things are as they are. This starting point is essential for trustworthy programme evaluation practice and in understanding the logic of programme evaluation. In addition, many people are in favour of the view that scientific and philosophical theories should have definite connection to what can be observed, measured, recorded and ultimately given a theoretical description in terms of laws and causes. The reverse side from this is speculation. What is the philosophy behind this kind of thinking? To understand this, a short retrospective look to the history of science is needed next. Francis Bacon ( ) and his Novum Organum (published 1620) paved way four hundred years ago for modern science. The method of Bacon rested on two key pillars, observation (the scientist record the world of the data of sensory experience) and induction (the scientist, then, generalize from a whole collection of particular instances to general conclusion) (see, e.g. Ladyman, 2002: 18-29). In scientific terms, then, what is causality? The classic discussion of the problem of induction and causality is coined by David Hume ( ). He related induction to the nature of causation and the laws of nature. According to Hume, causal relation connect ideas that have at first sight no logical relation (Ladyman, 2002: 32-37). Hence the basic idea is, that scientific explanation is causal explanation. Whatever is being explained is the explanandum and that which does the explaining is the explanans. But what if, taking into the consideration the compex nature of modern society and public interventions, the definitions of explanandum and explanans remain obscure? What then? Where to look? Can the Truth be approximate? Is it approved by the users of evaluations? (Cf. Van Fraassen, 1998: ; Vedung, 1997: ; Pawson & Tilley, 1997, 30-34; Mohr, 1995: ). According to the seminal work by Karl R. Popper (1980: 59-60), to give a causal explanation of an event means to deduce a statement which describes it, using as premises of the deduction one or more universal laws, together with certain singular statements, the initial conditions. As Popper underlines, the initial conditions describe what is usually labelled as the cause of the event in question and the prediction describes what is usually referred as effect. Thus, according to Popper, the principle of causality is 2

4 the assertion that any event whatsoever can be causally explained i.e. it can be deductively predicted. What we as authors of this article try to achieve in discussing the role of causality? It would be tempting to use the terms coined by Thomas S. Kuhn to describe the situation as far as the major trends in the evaluation methods and methodologies are concerned. Are the current methods and methodologies used in various evaluation activities forms of what Kuhn calls as normal science? Are the problems of modern evaluation practices puzzle-solving activity where rules for solving puzzles are quite strict and determined by various guidelines, standards and method books? Are the current evaluation procedures conservative in a sense that new novelties in methodological sense cannot be produced? Are new methodological novelties restricted by the common notions related to incommensurability (lack of common measure indicates that new tools cannot be introduced)? Considering the role of causal reasoning in different evaluation paradigms is actually a question of paradigm crisis. According to our view, a new understanding is needed for causal interpretation of causes and effects, and this kind reasoning would at least to some extent strengthen the core ideas which lie behind the currently well-spread and consolidated realistic evaluation tradition (Pawson & Tilley, 1997). Naturally, we are realistic in writing our arguments. We are not offering any new theory, but perhaps an alternative view. In our case, we are at the moment able to present only but very few confirming elements to back up our argumentation. Therefore, demands for falsifying our point remains to be received in the future. As Popper argued very strongly, theories can be ranked only according to their degree of falsifiability and this a true measure of their empirical content. Hence, In writing this article we are moving somewhere in the realm of context of discovery not in the realm of justification. 3. How Evaluation Paradigms Treat Programmes Three Examples Causality concerns also different evaluation paradigms. In the following, three distinct interpretations are considered to consolidate the authors view on causality issue. The first paradigm can be labelled as a representative of goal-free evaluation, and the 3

5 second one as goal-bound evaluation, to apply the well-known classification of Michael Scriven (1981). The first interpretation is Guba s and Lincoln c classic, The Fourth Generation Evaluation (1989). For them, evaluation is fundamentally social, political and value-oriented enterprise and therefore they do not treat evaluation as a scientific process. Their point is: fourth generation evaluation moves definitely far from existing measure-oriented, description-oriented and judgement-oriented paradigm into something new, which is probably most easily to be coined with the term negotiation (ibid., 7-8). At the outset, it is reasonably to consider Guba s and Lincoln s point of view here. Let us consider the above-mentioned definitions by negation and see what kind of questions open up? The opposite to these definitions would be non-measurable-oriented, non-description-oriented and non-judgement-oriented. Polemically, we might even further elaborate these notions as it follows: biased, speculative and aggregate? Guba and Lincoln do not spend so much time in defining what is programme or project to be evaluated (see, Guba & Lincoln, 1989: 71-74). As a matter of fact, they focus on various stakeholder groups which should taken into close scrutiny and with which the evaluation agenda should be developed together. Co-operation is the key word and it is something to be dealt with in every single evaluations. Needless perhaps to mention, Guba and Lincoln take a strong reluctant view on the principle of causality and generalization, since the conventional evaluation methodologies, according to them (ibid., 59-61), do not take into account, for instance, of contextual factors unless by physically and statistically controlling them. At the time, Guba and Lincoln offered fresh view to the evaluation community. Underlining the role of negotiation, putting emphasis on hermeneutic methodology and stressing the importance of relativistic ontology brought new winds to the discussion related to evaluation tools, techniques, methods and methodologies. But, at the end, the magnum opus of Guba and Lincoln is a child of it s era. Reading it after 13 years of it s publication, you cannot avoid being irritated by the harsh attacks on quantitative methods it almost feels that this was the main point Guba and Lincoln wanted to make. The result of this was in methodological terms severe one-eyed interpretation of how evaluation should be carried out. Alternative models were considered to be pejorative. 4

6 Next we turn shortly on two other examples. Second interpretation is actually an example of normal (evaluation) science that prevailed during the 1960s, 1970s and 1980s, before Guba s and Lincoln s contribution. Evert Vedung (1997) and Peter Rossi and others (1999) are here used as an examples of goal-bound tradition of programme evaluation. Vedung (1997: ), for instance, consideres programme as an entity to be understood by means of intervention theory consisting of causes (hypothesis underlying intervention logic), intervention per se (programme), planned conversion (expected results), intended output (in concerete measurable terms), intended immediate outcomes, intended intermediate results and intended ultimate (societal) outcomes. The difference of approach when compared to that of Guba and Lincoln is much more than semantic. Hence, in a word, according to this paradigm it is important to map out inter-linkages between outcomes (or effects), results, outputs, activities, inputs and objectives. This approach has been labelled, as put forth by Petri Uusikylä and Ville Valovirta (2002), for instance, as Result-Chain thinking. Finally, we have the third paradigm, which derives its origins from the realistic evaluation paradigm and it tries to combine the elements of the above-mentioned goal-free and goal-bound approaches. Ray Pawson and Nick Tilley (1997: 63-64) argue that social programmes are undeniably, unequivocally, unexceptionally social systems. They ( ) comprise the interplays of individual and institution, of agency and structure, and of micro and macro social processes. Therefore, much seems to be learned from the inspecting the social nature of programmes, and Pawson and Tilley (ibid.) suggest that an evaluator should focus on certain themes in carrying out evaluation missions: these include embeddedness, mechanisms, contexts, regularities, and change. Key idea in Pawson s and Tilley s reasoning is the understanding between mechanisms, contexts and outcomes. This is what they call the CMO-principle. In other words, as we understand this logic, an evaluator should always focus on the intervention as such but this is far from enough. He/she should also construct the hypothesis, implement data collection strategy and understand the data in terms of CMO-principles. This means in practice that an evaluator tries to understand what might work for whom in certain circumstances, use multi-method data collection operations and data analysis, and finally consolidate his/her findings with the context the programme has been functioning (what dis work for whom in what circumstances). 5

7 Pawson and Tilley come actually quite close to the classic ideas of causality presented by David Hume almost four hundred years ago. Namely, Hume further examied the concept of causality and underlined that an important feature of it is that of contiguity the relation of being connected in space and time (see Hume 1963). This, you can call it situationality if you like, is in board terms one of the key ideas put forward by Pawson and Tilley in their well spread Realistic Evaluation (1997). Table 1 sums up the main methodological differences between alternative approaches in programme evaluation. Table 1. Main Approaches in Programme Evaluation. Philosophical Base Goal-free approach Goal-bound approach Phenomenology Positivism Realism Author example Guba & Lincoln 1989 Vedung 1997 Rossi et al Mohr 1995 Disiplinary Base Sociology; Anthropology Economic theory Political Theory Focus of Methodology Methodology Causality interpretation Variables Control or comparison groups Participants role in carrying out Describe program holistically and from perspective of the participants Ethnography, case studies; participant observation; triangulation Reluctant to map out out the connection between causes and effects Emerging in course of evaluation Judge worth of programme Indentify causal links Experimental and quasi-experimental designs Cost-benefit analysis Primary concern Predetermined as input-output Realistic Approach Pawson & Tilley 1997 Social Policy Sociology To identify and analyse mechanism-context-outcome relationships Context + mechanism=outcome An issue on the research agenda, special emphasis on the CMO-framework Predetermined and emerging Not necessarily Yes Not necessarily Active None Active 6

8 evaluation Evaluator s role Interactive Independent of programme Political pressures (internal / external) Focus of evaluation report Observing Describe Controlled in design Describe and explain Present holistic portrayal of programme in process Render go / no go decision To understand and report CMO- configurations Next we turn to the question of micro-macro conversion and use the ideas by James Coleman as a conceptual bridge in finding out new conceptual understanding related to the causality issue in programme evaluation. In addition to that, we link conceptual ideas originally developed by Franz-Xaver Kaufman (1987) related to the intermediate sphere between the micro and macro to the ideas presented by Coleman in order to make better understanding new framework needed to understand the logic and usefulness of programme evaluation. 4. The Use of Metatheories in Finding Causal Explanations in the Implementation of Programmes Alternative approaches in program evaluation vary on how they treat programmes and actions that are taken to carry out the activities of the programme. Economic models could be labelled as heavily under-socialised explanations 1 of societal reality i.e. they focus on programs and their societal goals while forgetting the individual action through which - and the context in which - the programmes are implemented. This may in the worst case lead into false and mechanistic interpretation of programmes actual results and impacts. On the other hand, it seems that constructivist and (and partly also realist) approach to evaluation represent over-socialised interpretations of programme reality. These approaches do tend to neglect the program goals and focus too much on negotiations 1 This term is borrowed from Denis Wrong (1961) and Mark Granovetter (1985) who classify sociological (Wrong) and economic theories (Granovetter) according to their emphasis on agency and structure. 7

9 between stakeholders and consensus-building. While doing this they seem in constant danger of lapsing into wholesale relativism. Behavioralist approach fall somewhere between under- and over-socialised fields. Although behavioralists examine action and human behavior it is been carried out in a rather pre-deterministic way. In practice this means that model take into account the programme objectives and results of an experimental study while being ignorant on those mechanisms that transfer programme objectives into programme outputs. One of the main problem behind the mechanistic causal effect models is that they neglect the fact that action always takes place in situations and thus their success depends upon the way the action is performed by specific actors in specific situations (as the realistic school of evaluation has emphasised). Kaufman (1987) have highlighted that fact long before the rise of realistic evaluation tradition, originally very heavily coined by Ray Pawson and Nick Tilley, approach by the following two chains of logic. Figure 1. The true and false social intervention logic of a program (adopted from Kaufmann,1987: 10-11). FALSE PROGRAMME LOGIC A PROGRAMME initiates A CAUSAL PROCES ELABORATED PROGRAMME LOGIC A PROGRAMME interferes with AN INTERVENTION FIELD that produces whose reactions produce EFFECTS EFFECTS 8

10 Figure 1 describes the fact that traditional cause effect logic totally undermines the fact that effects of a programme are always caused by real actors rather than constructed ideal actors. A programme always influences through societal actors that base they courses of actions on interpretations on expected logic of a programme that is, actions are always socially constructed rather than objectively derived from abstract programme logic. Any programme always interferes with an intervention field of social actors whose reactions produce final outcomes or effects. Figure 2. The socially embedded course of programme intervention (adopted from Kaufmann, 1987: 11-12). ACTORS (with intentions and on the basis of a defined situations) develop A PROGRAMME which is implemented BY OTHER ACTORS ADRESSEES (with their intensions and definitions of the situation) (with their intensions and definitions of the situation) (whose reactions produce) EFFECTS Figure 2 elaborates the model further. First, it starts with the understanding that the programmes themselves are socially constructed, i.e. certain group of actors (with certain intentions and their best understanding of a societal needs) develop programmes, which 9

11 thereafter will be implemented by other actors who might and most often do have their own interpretations of programme logic and its situational validation. Secondly, after that there will be a third set of actors who try to adopt themselves with the expected goals and objectives (based on their own subjective understanding) and finally implement the programme. Following through this logic of programme implementation constructed by multiple set of actors it seems to be self-evident that we are dealing with several rather than one single interpretation of a program. Therefore we need more precise methodology to understand how social programmes are turned into real outcomes and effects. These problems between programme and action resemble the classical micro -macro (or agency structure) problem in modern sociological theory. This debate revolves round the problem of how structures determine what individuals do, how structures are created, and what are the limits, if any, on individuals capacity to act independently of structural constraints; what are the limits in other words on human agency. These questions can be translated into programme evaluation jargon: the questions would be: how programmes determine what projects (or individuals belonging to those) do, how programmes are created and what are the limits created by the programmes on projects capacity to act independently. 5. Proposal for a research agenda In the existing programme evaluation models there is imminent gap existing between programme-level results and project-level results. This is a classical problem of aggregation: programme-level results cannot simply be aggregated from project-level results, although most of the programme evaluators tend to be believe so. This is quite close to what Frank Fischer (1995) has called as mapping out situational validity in programme evaluation. Let us next demonstrate this aggregation problem by a single example using European Social Fund (ESF) as an example and applying the model proposed by James Coleman (1990) and further elaborated by Kaufmann (1987). In the positivist or behavioralist evaluation approaches it is often implicitly assumed that introduction of the socioeconomic programme leads to certain expected change in society for example the 10

12 increased employment or regional competitiveness. Also the evaluation of the actual impacts follow the same logic (see for example the evaluation guidelines the so-called MEANS Handbook series adopted and supported by the European Commission). The major problem for explanations of system behaviour based on actions and orientations at the level below that of the system is that of moving from the lower level to the system level. 2 The use of the example presented in Figure 3 raises further questions about what kind of proposition was attempted to demonstrate and in particular, what unit or units were involved in the proposition. Should the proposition be specified at the individual level? If so, then the proposition of Figure 3 must be revised. Figure 3. Macro-proposition: ESF-Programme increases employment. ESF - Obj. 3 PROGRAM A Increased employment B The single proposition now breaks into three (see Figure 4): one with an independent variable characterizing the individual (i); a second with both independent and dependent variable characterizing the individual (ii); and third with the independent variable characterizing the individual and dependent variable charactering the system (iii). Figure 4 shows a way of diagramming such multilevel systems of propositions. The upper horizontal arrow represents our baseline macro-proposition. The three connected arrows 2 In economics, for example, there is microeconomic theory and there is macroeconomic theory; and one of the central deficiencies in economic theory is the weakness of the linkage 11

13 of which the first begins from the macro-level proposition and goes down to lower level (individual level). This arrow (1) represents the transformation function that converges the objectives of the ESF-programme (macro-level) into local- (or regional-) level understanding of the societal goals that programme serves. Thereafter, local actors (through bargaining and exchange of information) transform these internalised programme-values into conrete project-ideas and local priorities (2). Local projects (micro-level) try to formulate their project ideas in such a way that projects would meet local needs and articulated programme goals (such as innovative approaches to local employment, customer orientation, quality principles etc.). Second lower level arrow (3) represents the function that turns individual or project values into action (i.e. projects or their stakeholders start acting according to differently, providing new innovative services etc.). After that (4) these results and experiences can be aggregated to local results (innovations, increased employment etc.). Last arrow (5) is the most important and interesting, because it moves transfers the local results into programme (i.e. systemic) results. In concrete terms it verifies the aggregation of changes taking place at the individual level behaviour and then resulting in the local (or regional) level changes (i.e. increased employment, better competitiveness in the region etc.) which then can be aggregated into programme results. between them, a weakness papered over with the idea of aggregation and with ubiquitous concept in macroeconomic theory, that of the representative agent (Coleman 1990, 6-7). 12

14 Figure 4. Macro- and Micro-level propositions: Impacts of the European Social Fund Programme on Employment (applied from Coleman, 1990: 8, and conceptual ideas further inspired by Kaufmann, 1987) Obj. 1 ESF-Programme A Program created with certain set of actors that share common understanding of the program goals 1 Local actors transform the program goal into local priorities through local networks Individuals in various projects transform program goals and values into project goals and values i ii 2 MACRO- LEVEL MESO- LEVEL MICRO- LEVEL 3 Increased employment 4 These goals are transformed into activities and outputs that benefit clients iii iv 5 B Local innovations shape local modes of activities Local results are transformed into macro effects Derived from the figure 4, the practical research questions in this evaluation mission are briefly summarized in table 2. Table 2. Concrete research agenda including research questions to be tackled (based on Figure 4). Set of questions (reflecting the numbers presented in the Figure 4). Relation 1 Programme Evaluation Research Theme (examples of evaluation questions) How the programme was planned and it s goals set? What was the level of consensus in mapping out key priorities at programme level? Empirical case examples Strategy space of Finnish ESFframework (Appendix 1.) 13

15 Relation 2 Relation 3 Relation 4 Relation 5 What is the local structure for implementing the programme? Who are the key actors in the network and what is the volume and quality of connections within the network? What is the local focus in implementing the programme? What themes and issues are tackled in particularly? What kind of selection criteria for projects are been applied? What is the competence of the project managers and project-teams? What is the human, organisational and client capital of the projects? How the projects develop global objectives, mission statements, expected results and design their activities? How efficiently the projects are carried out? What is the relation between expected activities and occurred activities? How the project self-monitor its success? How the local innovations developed function in local settings? What is the logic of local innovation process and who are the key actors in developing new ideas? How the local innovations are being transferred into wider contexts to be utilised? How the utilization and dissemination aspects can be strengthened? (Appendix 1.) Centre of Exellece programme utility network (Appendix 2) Self-assessment mind map of the city of Helsinki ESFproject (Appendix 3) Finland s Objective 5b regional results (Appendix 4) Finland s 5b programme results (Appendix 5) 6. Conclusions Modern society is so complex and multidimensional to be captured by mechanistic evaluation approaches. Therefore evaluators need specialized knowledge to calibrate their evaluation studies and design experiments. (Evaluation) science is about confirmation as well as falsification. Our ideas remain to be further confirmed and thereafter, if all, falsificated. However, it seems that the debate on causality in programme evaluation has not perhaps been so active as it should has been, keeping in mind the challenges facing programme evaluations in the on-going programming period ( ) in structural funding not to speak of the changes in structural funding framework when the 14

16 enlargement of the European Union takes place. When there are more countries and lesser funds to be utilised, the more important it is to produce trustworthy programme evaluations on the links between causes and effects of structural interventions. In this research paper we have tried to elaborate traditional cause-effect evaluations towards a multilevel research agenda. Further empirical studies should be carried out to test whether out multilevel logic provides better answers to difficult question of what has been the real value added caused by an individual programme. 7. References Coleman, James (1990). Foundations of Social Theory. Cambridge, Mass: The Belknap Press of Harvard University Press. Fischer, Frank (1995). Evaluating Pulic Policies. Chicago: Nelson-Hall Publishers. Van Fraassen, Bas, C. (1998). The Pragmatics of Evaluation. Pp in Klemke, E.D. et al. (eds.). Introductionary Readings in the Philosophy of Science. New York: Prometheus Books. Granovetter, Mark S. (1985). Economic Action and Social Structure: The Problem of Embeddedness. American Journal of Sociology 91: Guba, Egon G., Lincoln, Yvonna S. (1989). Fourth Generation Evaluation. Thousand Oaks: Sage. Hume, David (1963). An Enquiry Concerning Human Understanding. Oxford: Oxford University Press. Kaufman, Franz-Xaver (1987). Prevention and Intervention in the Analytical Perspective of Guidance. Pp In Hurrelmann, Klaus et al. (eds.) Social Interventions: Potential and Constraints. Berlin & New York: Walter de Gruyter. Kuhn, Thomas S. (1962). The Structure of Scientific Revolutions. Chicago: The University of Chicago Press. Ladyman, James (2002). Undestanding Philosophy of Science. London: Routledge. Mohr, Lawrence B. (1995). Impact Analysis for Program Evaluation. Thousand Oaks, CA: Sage. Pawson, Ray, Tilley, Nick (1997). Realistic Evaluation. London: Sage. Popper, Karl R. (1980). The Logic of Scientific Discovery. Cambridge: Unwin & Hyman. Rossi, Peter H., Freeman, Howard, E., Lipsey, Mark (1999). Evaluation. A Systematic Approach. New York et al.: Sage. Scriven, Michael (1981). Evaluation Thesaurus. Newbury Park: Sage. Talmage, H. (1982). Evaluation of Programs. In h. Mitzel (ed.) Encyclopedia of Educational Research. New York: Free Press. Uusikylä, Petri, Valovirta, Ville (2002). Osaoptimoinnista tulosketjuihin. Helsinki: Valtiovarainministeriö (From Partial Optimizing to Result-Chains, available only in Finnish, published by the Ministry of Finance). Vedung, Evert (1997). Public Policy and Program Evaluation. New Brunswick & London: Transaction Publishers. 15