What is it to evaluate the evaluators? A fairly formal reflexive analysis Malcolm Ashmore Loughborough and Bogotá
What is reflexivity? Basic division between reflexivity as a Bad Thing and a Good Thing As a Bad: the analytic philosophers bête noir Used to hoist perceived relativists with their own petard As (basically) a Good Many versions, many typologies Self-awareness; self-reference; the (ethnomethodological) constitutive circularity of accounts (Ashmore 1989) On a continuum from benign introspection to radicalconstitutive (Woolgar 1988) Positional; textual; the (ethnomethodological) constitutive reflexivities of ordinary life (Macbeth 2001) Mechanical; substantive; methodological; meta-theoretical; interpretative (including the radical referential); ethnomethodological (Lynch 2000)
Reflexivity as/in research practice Three stances/desires 1. Ethnomethodological studies of (others ) constitutive reflexivities Seldom interested in self-reference see Pollner 1991 2. The autobiographic-confessional Maps onto Ashmore s self-awareness, Woolgar s benign introspection, Macbeth s positional, Lynch s methodological Done to improve methodological adequacy and moral self-satisfaction The most common and popular; almost obligatory in contemporary qualitative social science inquiry 3. Interrogation of object/subject similarities and their significances Maps onto Ashmore s self-reference, Woolgar s radicalconstitutive, Macbeth s textual, Lynch s radical referential Done as performance/provocation; and/or to improve empirical adequacy The stance of the SSK reflexivists
The SSK reflexivists (late 1980s) revisited Reflexivity understood as a turning (it) back on itself ; a form of self-referentiality SSK reflexivists attempted to take it seriously treat it positively (as an opportunity, not a threat nor a liability) practice reflexivity Work based on the positive recognition of a similarity: between SSK s topic and resource or object and subject i.e. knowledge of knowledge (the X-of-X relation) What is it to know knowledge, to study studies, to explain explanation, to understand understanding, etc.? What is the significance of this X-of-X relation? Selected contributions from a very small group Malcolm Ashmore, 1989. The Reflexive Thesis. Michael Mulkay, 1985. The Word and the World. Steve Woolgar, 1988. Reflexivity is the ethnographer of the text, in his Knowledge and Reflexivity.
Quotes In refusing to separate knowledge of things out there and knowledge of the self in here, the reflexive knower, while reading the Book of Nature, simultaneously writes a piece of his or her autobiography... This constitutive inseparability of knower and known redefines knowledge as a matter of interdefinition or mutual genesis of what the world is and what you are; as an incessant circulation between representer and represented which can find a secure foundation neither in the external world nor in the knowing subject. (Pels 2000: 2) the researcher is required to participate, in the course of her research, in activities which are also the object of that research. She produces knowledge claims about the production of knowledge claims; she aims to explain how explanation is done, to understand how understanding is produced, and so on. (Woolgar 1988: 23) the reflexive move is double-edged. In that it entails a process of re-it-eration (the it being that which is actively deployed in the turning as well as that which is the passive recipient of such deployment), reflexivity acts as much to preserve as to destroy. (Ashmore 1989: xxviii)
The X-of-X relation Examples: Knowledge of knowledge the reflexive application of sociology of scientific knowledge (SSK) to itself (Ashmore 1989, Mulkay 1985, Woolgar 1988) The analysis of debunking practices (Ashmore 1993, 1994, 1995) Or, as I argue next, debunking (of) debunking And evaluating the evaluators?
Analysing debunking Three empirical studies of the finding of fraud/gross error (i.e. debunking) in science 1. The N-ray phenomenon (1900-1904) discovered to be (debunked as) non-existent in 1904 (Ashmore 1993) 2. The Piltdown Man paleoanthropological finds of the 1910s, discovered to be (debunked as) fraudulent in the 1950s (Ashmore 1995) 3. The Cyril Burt scandal (Ashmore 1994) The work of Sir Cyril Burt, British psychologist (twin studies supporting the inheritance of intelligence) posthumously debunked as fraud in the early 1970s, leading to strong fraud consensus for a decade Two books defending Burt and accusing his accusers of fraudulently manufacturing fraud charges ( metadebunking ) published independently (Joynson 1989 and Fletcher 1991) So what is the stance of the analyst in these studies? Must analysing debunking mean debunking debunking?
Debunking, debunking debunking, analysing debunking Debunking: a negative evaluative practice leading, if successful, to a radical change of cultural status for the target from true/genuine/useful/respected/famous to false/fake/useless/scandalous/infamous Debunking debunking (metadebunking): a move of the same kind and motivation but with a different target, namely the agents of (the first) debunking practice; e.g. Joynson and Fletcher accusing Burt s accusers Analysing debunking: results in a negative evaluation (a debunking) of its topic and thus a positive re-evaluation of its topic s target Ashmore 1993 attacked the N-ray debunker (Wood) and thus rescued the N-ray proponent (Blondlot) Ashmore 1995 attacked the Piltdown debunkers (Weiner et al.) and thus rescued the Piltdown proponents/discoverers (Dawson et al.) Ashmore 1994 attacked Burt s debunkers debunkers (Joynson, Fletcher) and thus rescued Burt s debunkers (Kamin, Gillie, et al.) and thus endorsed their attack on Burt Is this reverse switching effect of going meta inevitable? Is analysis (taking X as topic) itself a form of debunking such that (as the science warriors insist) STS/SSK just is critical of science? Is symmetrical-as-even-handed analysis possible?
Evaluating the evaluators an X-of-X relation? Not strictly; evaluating the evaluators has (subtly) different terms The formally purer relation is evaluating evaluating Verbing verbing Where the verb/action/practice is (precisely) evaluative/normative/judgemental Debunking debunking Evaluating evaluating
Conclusion the (inevitable?) results of evaluating the evaluators An analysis of (say) a negative TripAdvisor review of a hotel, will, according to my analysis, be bound to be (read as) itself negative however nonevaluative it wants to be After all, as symmetrical, uninvolved, disinterested analysts you can/will never do agreement with the reviewer And thus the target of the review (the hotel) will be rescued And thus the analytic efforts will be equivalent to giving the hotel a positive review Is this what you want, Malte and Steve? And a final question
and will this business ever end? Infinite regress of rating rating rating? What is the role of this research effort (and of its product, HowsMyFeedback.org) in the temporal extent of the culture of evaluating/rating/ranking/reviewing? Will it shorten the life of this culture? Or extend it?
References Ashmore, M. (1989). The Reflexive Thesis: Wrighting Sociology of Scientific Knowledge. Chicago: The University of Chicago Press. Ashmore, M. (1993). The theatre of the blind: Starring a promethean prankster, a phoney phenomenon, a prism, a pocket, and a piece of wood. Social Studies of Science 23: 67-106. Ashmore, M. (1994). Debunking, metadebunking and analysis: Cyril Burt s accusers, their detractors and me. Unpublished. Presented at the EASST conference, Budapest, Hungary. Ashmore, M. (1995). Fraud by numbers: Quantitative rhetoric in the Piltdown forgery discovery. South Atlantic Quarterly, special issue on Mathematics, Science and Postclassical Theory, B. Herrnstein-Smith and A. Plonitsky, eds., 94 (2): 591-618. Fletcher, R. (1991). Science, Ideology and the Media: The Cyril Burt Scandal. New Brunswick, NJ and London: Transaction. Joynson, R.B. (1989). The Burt Affair. London: Routledge. Latour, B. (1988). The politics of explanation: An alternative. In S. Woolgar, ed. Knowledge and Reflexivity. London, Newbury Park, CA, and New Delhi, Sage. Lynch, M. (2000). Against reflexivity as an academic virtue and source of privileged knowledge. Theory, Culture and Society 17(3): 26-54. Macbeth, D. (2001). On reflexivity in qualitative research: two readings, and a third. Qualitative Inquiry 791): 35-68. Mulkay, M. (1985). The Word and the World: Explorations in the Form of Sociological Analysis. London: Allen & Unwin. Pels, D. (2000). Reflexivity: One step up. Theory, Culture and Society 17(3): 1-25. Pollner, M. (1991). Left of ethnomethodology: The rise and decline of radical reflexivity. American Sociological Review 56: 370-80. Woolgar, S. (1988). Reflexivity is the ethnographer of the text. In Knowledge and Reflexivity: New Frontiers in the Sociology of Knowledge, edited by S. Woolgar, 14-34. London and Beverly Hills, Ca.: Sage.