Crowdsourcing: Best Practices and Open Issues

Transcription

1 Crowdsourcing: Best Practices and Open Issues Matthias Hirth, Tobias Hossfeld University of Würzburg WG2 Mechanisms and Models

2 Motivation Best Practices and (current) Challenges Reliability: common approaches do not work Hidden influence factors: e.g. demographics, download speed, Impact of time when launching a campaign The future of Crowdsourcing (studies) Upcoming Challenges: Dagstuhl Seminar Reproducible Crowdsourcing studies 2

3 Joint Publication Best Practices Filtering Training Lab Tester Demographics Hardware requirements Reliability Tobias Hoßfeld, Christian Keimel, Matthias Hirth, Bruno Gardlo, Julian Habigt, Klaus Diepold, Phuoc Tran-Gia. CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation.s Pseudo reliable crowd Phase 1 QoE - Test Software based screening mechanisms Content questions, reliability checks Incentive design, variable payments Post processing Statistical analyses Phase 2 Why does content- and rating based filtering differ so much? How to improve existing filter mechanisms? 3

4 Impact of Payments and Country Martin Varela, Toni Mäki, Lea Skorin-Kapov, Tobias Hoßfeld Increasing Payments in Crowdsourcing: Don't look a gift horse in the mouth. PQS 2013, Vienna, Austria, September g p y Oceania How to report crowdsourcing results correctly? Which information need to be published to make an experiment reproducible? How to deal with hidden influence factors? Europe Asia Americas Africa groups have means significantly different from Asia 4

5 Experiment begin changes demographics! Cumulated daily activity pattern of the users Users mainly work during regular office hours Due to different time zones: Time of task submission influences participating workers Can we remove all crowd-based issues just by increasing the sample/number of experiments? (Probably not) Still: How can we tackle issues using statistical approaches?

6 Dagstuhl Seminar Crowdsourcing: Group Work Group 1: Long-Term Perspectives, Ethical issues, social aspects, Impact in 5 years on economics Moderator: Phuoc Tran-Gia; Scribe: Christian Keimel Martha A. Larson, Christian Keimel, Nhatvi Nguyen, Klaus Diepold, Gianluca Demartini Group 2: Theory, Taxonomy, Evaluation Frameworks Moderator: Tobias Hossfeld; Scribe: Abraham Bernstein Munindar P. Singh, Deniz Iren, Abraham Bernstein, Joseph Davis Group 3: Industry Use Cases Moderator: Maja Vukovic; Scribe: Maja Vukovic Florian Zeiger, Vassilis Kostakos, Kathrin Borchert, Shinichi Konomi Group 4: Crowdsourcing Mechanisms and Design Moderator: Alessandro Bozzon; Scribe: Matthias Hirth Babak Naderi, Andreas Hotho, Matthias Hirth, Markus Krause, Cristina Cabanillas

7 Results from Group on Long-Term Perspectives Discussion Group: Long-Term Perspectives, Ethical Issues, Social Aspects & Impact in 5 Years on Economics Crowdsourcing is sustainable in the future but emerging public perception or legal regulation may come into play. Workers, requesters, lawyers and platform providers should work together to define a vision of crowdsourcing ahead of regulations. More crowd-building, resulting in specialized crowds. Crowdsourcing is opening new and thrilling aspects for empirical research, but commercial aspects need to be added. Microworking should strive towards a marketable skill development. Building economic capacity in geographical regions and social segments where needs exist. 7

8 Reproducible Crowdsourcing Studies Is the current approach feasible? One experiment (with repetitions/refining), one crowd (-source) Generalize results? No detailed information about the crowd or the tasks design (which questions where asked?) included in most paper How to report a crowdsourcing experiment in order to make it repeatable for other researchers/justify the generalization? Common template proposed by Qualinet for this? Which information to include in publications: Demographics, Environment settings, etc.? How to minimize crowd specific factors? Which and how can specific results be generalized? 8