Methods for SE Research This material is licensed under the Creative Commons BY-NC-SA License
Methods for SE Research Practicalities
Course objectives To help you with the methodological aspects of your thesis Trying to be concrete enough Basics of research methodology Main principles, approaches, concepts, generalisation, validity Overview of some research methods that can be used Except for, e.g., statistical methods, controlled experiments Particular help for study design Components of a study design (research design, research process) Writing and evolving Research Questions, and Title Discussion
Learning objectives (see Moodle)
Structure and teaching methods Content sessions (lectures) Lecturing the basics and background, discussions Selected research methods: (Systematic) Literature Review, Design Science, Case Study, and Action Research Reading material tasks, some involve study circle sessions (reading on your own, discussion in groups or a specific task) Shadish et al. (selected pages from the beginning of the book) Understanding the basic concepts, such as causality and validity in particular Selected (parts of) papers or theses on each research method Workshop on RQ writing & Study Design; more details later Learning Portfolio Reflections on sessions Articulation of validity issues in the domain of SE Own study design Short presentation of your RQs and Study Design Guidelines will be in Moodle
Study circle, some practicalities
Study circle some practicalities See Moodle for details In Finnish: Lukupiiri reading circle You will be assigned into small groups PhD students please go to different groups No teacher as such, some facilitation may be in order collective learning Everyone should get equal opportunity to be involved
Study circle before the session Get the material from Moodle Check the instructions what to read from Moodle Make notes while reading Think about what you would like to discuss with others
SAMPLE, READ the INSTRCUTIONS FROM MOODLE
Study circle in the session Some points for the discussion Use the reading instructions to guide your discussion Discuss whether there is something you did not understand or would otherwise bring to discussion Concentrate on central concepts or topics, themes and avoid getting too long in details Use concrete examples from SE when applicable to concretise Try to keep steady progress so that you manage to go through all the material
Methods for SE Research Introduction This material is licensed under the Creative Commons BY-NC-SA License
Some Background to Research Methodology
Philosophy of Science Mapping Research Methods: Lähdesmäki, T., Hurme, P., Koskimaa, R., Mikkola, L., Himberg, T., Methods Paths for Humanists. University of Jyväskylä, Faculty of Humanities. http://www.jyu.fi/mehu. (Referred 03.08.2012.)
Clearly, all of this stuff is not for the faint-of-heart (William M.K. Trochim, http://www.socialresearchmethods.net/kb) I've seen many a graduate student get lost in the maze of philosophical assumptions that contemporary philosophers of science argue about. And don't think that I believe this is not important stuff. Philosophers have been debating these issues for thousands of years and there is every reason to believe that they will continue to debate them for thousands of years more. Those of us who are practicing scientists should check in on this debate from time to time (perhaps every hundred years or so would be about right). But in the meantime, we can't wait for the philosophers to settle the matter. After all, we do have our own work to do!
Nature of knowledge In epistemology (theory of knowledge) the interest is in discussions on what and how people can know things and what kind of knowledge is right knowledge. In ontology the interests are in discussions on what reality is and what kinds of phenomena are real. https://koppa.jyu.fi/avoimet/hum/menetelmapolkuja/en/methodmap/philosophy-of-science
Epistemology epistemology ɪˌpɪstɪˈmɒlədʒi, ɛ- noun [ mass noun ] Philosophy the theory of knowledge, especially with regard to its methods, validity, and scope, and the distinction between justified belief and opinion. Knowledge is objective hard data, can be measured Knowledge is subjective relies on interpretation social world is different from natural world
Positivism interpretivism (constructivism, post-positivism) Positivism Scientific, objective, robust Involves identifying causes Tests hypotheses, often quantitative Key method (controlled) experiment discover laws through direct manipulation and observation Interpretivism Social world different from natural world Stresses interpretations and interpreting processes in the formation of knowledge The researcher s job is to interpret events and phenomena in terms of how people involved understand their own experience Typically qualitative
Theory Interpretivism Induction Data collection Research question(s) Data analysis Synthesis Results, findings, lessons learnt Hypothesis Positivist Deduction Observations Confirmation / falsification Research problem Phenomenon
There will be methodological issues and debates in SE after your thesis! (given you finish yours, of course) So, do not wait. How about you trying to solve these issues once and for all in your thesis well, cannot really suggest that either. Yet, this does not mean that you couldn t contribute to the methodological maturation of SE.
Research Strategies
Research strategies Mapping Research Methods: Lähdesmäki, T., Hurme, P., Koskimaa, R., Mikkola, L., Himberg, T., Methods Paths for Humanists. University of Jyväskylä, Faculty of Humanities. http://www.jyu.fi/mehu. (Referred 03.08.2012.)
Quantitative methods Mixed methods Qualitative methods [Creswell 2009 ] Research approaches Approaches studying reality Mathematical Research stressing what is reality Research stressing utility of artifacts Conceptual analytical Approaches for empirical studies Artifactsbuilding Artifactsevaluating Theory testing Theory creating [Järvinen 2004]
Many approaches and research methodologies In this course: Pragmatic view to multitude of approaches Selecting methods to match the phenomenon of interest Arguments for and against methods often have a secondary, more or less hidden, agenda
Basic vs. Applied Research and Sciences of Artificial
Some history (From D. E. Stokes. 1997. Pasteur s Quadrant: Basic Science and Technological Innovation, The Brookings Institution.) The Second World War Vannevar Bush: Science: the Endless Frontier, 1945 President Roosevelt's science advisor and head of the Office of Scientific Research and Development A report for the US science policy in the post-war years Founding of Atomic Energy Commission 1946, Department of Defence 1947, National Science Foundation 1950 A general way of thinking about the nature of basic science and its relationship to technological innovation Turned out to be profoundly important in the longer run The Bush s view has since been skilfully employed
Canons of Vannevar Bush 1. Basic science is performed without thought of practical ends Kind a Gresham s Law: mixing will results in applied driving out the pure 2. Basic research is the pacemaker of technological improvement Advances of basic science will be converted into technology by the processes of technology transfer, moving from basic to applied research 3. A nation will recapture the technological benefit of its investment in basic science A nation which depends upon others for its new basic scientific knowledge will be slow in its industrial progress and weak in its competitive position in world trade, regardless of its mechanical skills
Basic research Applied research Development Production and operation Linear model (Stokes 1997) Illustration by Timo Halko
Research is inspired by: Considerations of use? Quest for fundamental understanding? Quadrant model of Scientific Research (Stokes 1997) Illustration by Timo Halko
Dynamic model of Scientific Research (Stokes 1997) Illustration by Timo Halko
Toward an Ecological Model of Research and Development The choice between basic and applied research is a false one (Ben Shneiderman, The Atlantic, Apr 23, 2013) It is ironic that Bush saw little possibility that applied research could stimulate pure or basic research. The irony is striking since his book is filled with examples of how war-time needs, especially in medicine produced remarkable basic research payoffs. Of course the most compelling example from Bush's own experience is the Manhattan Project, in which the practical goal of building an atomic bomb invigorated basic research in physics, materials engineering, mathematics, chemistry, and many other fields.
Toward an Ecological Model of Research and Development The choice between basic and applied research is a false one (Ben Shneiderman, The Atlantic, Apr 23, 2013) Addition to Stokes: Theory-inspired applied research http://www.theatlantic.com/technology/archive/2013/04/toward-an-ecological-model-of-research-and-development/275187/
Research is inspired by: Considerations of use? Quest for fundamental understanding? THEORY- INSPIRED APPLIED RESEARCH Quadrant model of Scientific Research (Stokes 1997) Illustration by Timo Halko
Herbert A. Simon (1996): Sciences of the Artificial Historically and traditionally, it has been the task of the science disciplines to teach about natural things: how they are and how they work. It has been the task of engineering schools to teach about artificial things: how to make artifacts that have desired properties and how to design. Engineering, medicine, business, architecture, and painting are concerned not with the necessary but with the contingent not with how things are but how they might be...
Natural Social Natural Artificial
Software Engineering
Software engineering (SE) is about building software systems or products that are genuinely useful and usable and meet the requirements set for them. (ACM Computing Curricula 2005: The overview report.)
For SE, there is reality Research approaches reality for SE is more social than natural Research stressing what is reality Approaches studying reality Mathematical Research stressing utility of artifacts Conceptual analytical Approaches for empirical studies Artifactsbuilding Artifactsevaluating Theory testing Theory creating [Järvinen 2004] IS Social Sci SE ~ CS Math
Research in software engineering The phenomena investigated happen in the real world environment Messy in many ways, particularly when humans are heavily involved Interaction of technological systems and social systems In many aspects, not well understood, hard to put in numbers Relevance Problems are relevant in the environment Utility of results can be evaluated in the environment Does not mean that research should be messy Understanding and use of rigorous methods Crisp discussion on merits and deficiencies of available methods The research discipline still immature Research methods are still evolving and theories are rare Utilise research methodology that goes beyond the traditional computer science, such as case study, action research, ethnography from social sciences. Aim towards development of theories.
SE Methodological Tool Box Case study Action research Design science Constructive research Quasi-experiment Controlled experiment Survey photo by Per Erik Strandberg, Wikimedia Commons
References Creswell JW, 2009. Research Design Qualitative, Quantitative and Mixed Methods Approaches. 3rd ed.: Sage Publications. Järvinen P, 2004. Research Questions Guiding Selection of an Appropriate Research Method, University of Tampere. Niiniluoto, I, 1993. The aim and structure of applied research. Erkenntnis, 38(1), pp.1 21. Shadish WR, Thomas CD, Campbell DT, 2002 Experimental and quasi-experimental designs for generalized causal inference. pp. 1 102 Stokes DE, 1997. Pasteur s Quadrant: Basic Science and Technological Innovation, The Brookings Institution. Yin RK, 2009. Case study research: design and methods. 4th ed.: Sage Publications. Shneiderman B, 2013 (Apr 23). Toward an Ecological Model of Research and Development The choice between basic and applied research is a false one. The Atlantic. http://www.theatlantic.com/technology/archive/2013/04/toward-an-ecological-model-ofresearch-and-development/275187/ (Referred 30.10.2013) Mapping Research Methods: Lähdesmäki, T., Hurme, P., Koskimaa, R., Mikkola, L., Himberg, T., Methods Paths for Humanists. University of Jyväskylä, Faculty of Humanities. http://www.jyu.fi/mehu. (Referred 03.08.2012)