computational social media lecture 07: crowdsourcing

Size: px

Start display at page:

Download "computational social media lecture 07: crowdsourcing"

Myron Harvey
6 years ago
Views:

1 computational social media lecture 07: crowdsourcing daniel gatica-perez

2 reminders HW3: Algorithmic Bias Check (also on course website) Due Thu Last lecture of the semester Fri Same time

3 this lecture 1. introduction 2. definitions & categorization of crowdsourcing systems 3. understanding crowdsourcing research analyzing crowdsourcing workers & systems designing crowdsourcing systems open issues

4 1. introduction

6 Human Computation (Luis von Ahn, 2004) espgame.org, gwap.com L. von Ahn & L. Dabbish, Labeling Images with a Computer Game, in Proc. ACM CHI

7 image:

8 2. definitions & categorization of crowdsourcing systems

9 term coined by Jeff Howe (2005)

10 definition of crowdsourcing source: cooltownstudios crowdsourcing is an online, distributed problem-solving and production model that leverages the collective intelligence of online communities to serve specific organizational goals crowds are given the opportunity to respond to tasks promoted by the organization motivated to respond for a variety of reasons. D. C. Brabham, Crowdsourcing, MIT Press, 2013

key ingredients of crowdsourcing 1. an organization (or entity) that has a task it needs performed 2. a community that is willing to perform the task voluntarily 3.

11 key ingredients of crowdsourcing 1. an organization (or entity) that has a task it needs performed 2. a community that is willing to perform the task voluntarily 3. an environment that allows the work to take place and the community to interact with the organization 4. mutual benefit for organization and community D. C. Brabham, Crowdsourcing, MIT Press, 2013 credit (cc):

12 a problem-focused categorization type how it works ideal kind of problems examples knowledge discovery & management crowd has to find & collect info into a common format info gathering, reporting problems, creation of collective resources peer-to-patent.org D. C. Brabham, Crowdsourcing, MIT Press, 2013 seeclickfix.com

16 a problem-focused categorization type how it works ideal kind of problems examples knowledge discovery & management crowd has to find & collect info into a common format info gathering, reporting problems, creation of collective resources peer-to-patent.org broadcast search crowd has to solve empirical problems ideation problems with innocentive.com empirical provable solution like scientific problems D. C. Brabham, Crowdsourcing, MIT Press, 2013 seeclickfix.com

18 a problem-focused categorization type how it works ideal kind of problems examples knowledge discovery & management crowd has to find & collect info into a common format info gathering, reporting problems, creation of collective resources peer-to-patent.org broadcast search crowd has to solve empirical problems ideation problems with innocentive.com empirical provable solution like scientific problems peer-vetted creative production crowd creates & selects creative ideas ideation problems where solutions are matter of taste such as design & aesthetics D. C. Brabham, Crowdsourcing, MIT Press, 2013 seeclickfix.com threadless.com crashthesuperbowl.com nextstopdesign.com

21 one world (MadV, Nov. 2006)

22 video replies: most responded video

26 a problem-focused categorization type how it works ideal kind of problems examples knowledge discovery & management crowd has to find & collect info into a common format info gathering, reporting problems, creation of collective resources peer-to-patent.org broadcast search crowd has to solve empirical problems ideation problems with innocentive.com empirical provable solution like scientific problems peer-vetted creative production crowd creates & selects creative ideas ideation problems where solutions are matter of taste such as design & aesthetics threadless.com distributed human intelligence tasking crowd analyzes large amounts of information large-scale data analysis where human intelligence is more effective/efficient than machine intelligence mturk.com D. C. Brabham, Crowdsourcing, MIT Press, 2013 seeclickfix.com crashthesuperbowl.com nextstopdesign.com crowdflower.com

27 HIT: Human Intelligence Task

connections between crowdsourcing & social media crowdsourcing & social media face similar challenges + motivate / incentivize users & workers + sustainability of site activity + community management

31 connections between crowdsourcing & social media crowdsourcing & social media face similar challenges + motivate / incentivize users & workers + sustainability of site activity + community management social media as implicit crowdsourcing crowdsourcing can use social media as channels + use Twitter to crowdsource city reports + post videos on YouTube + use Facebook for recruitment for workers crowdsourcing can involve online social interaction + vote / rate / comment on other people s contributions credit (cc):

32 3. understanding crowdsourcing research

33 categorizing crowdsourcing research analyzing crowdsourcing workers & systems designing crowdsourcing systems crowdsourcing uses & applications

34 3.1 analyzing crowdsourcing workers & systems

35 example 1: who are the crowdworkers? J. Ross, L. Irani, M.S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the Crowdworkers? Shifting Demographics in Mechanical Turk, in Proc. ACM CHI Extended Abstracts, 2010.

36 J. Ross, L. Irani, M.S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the Crowdworkers? Shifting Demographics in Mechanical Turk, in Proc. ACM CHI Extended Abstracts, 2010.

37 reported annual household income by country (USD) J. Ross, L. Irani, M.S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the Crowdworkers? Shifting Demographics in Mechanical Turk, in Proc. ACM CHI Extended Abstracts, 2010.

40 how well do crowdworkers do their job?

41 example 2: crowdworker modeling issues: 1 - annotators might not be of same quality 2 objects might not have same difficulty 3 ground-truth might not exist at all this and the next 3 slides are taken from: Marco Tagliasacchi, Crowdsourcing for Multimedia Retrieval, Summer School on Social Media Modeling and Search,

42 aggregating judgments a classifier M. Tagliasacchi, Crowdsourcing for Multimedia Retrieval, Summer School on Social Media Modeling and Search, Sep

43 the basic idea (Dawid & Skene, 1979) i: object j: annotator (a.k.a. sensitivity) (a.k.a. specificity) A. P. Dawid and A. M. Skene, Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, J. of the Royal Statistical Society. Series C, Vol. 28, No. 1, pp , 1979 M. Tagliasacchi, Crowdsourcing for Multimedia Retrieval, Summer School on Social Media Modeling and Search, Sep

44 the basic idea (2) (Dawid & Skene, 1979) Estimates of the true labels are produced in the E-step Estimates of the alpha parameters are produced in the M-step The learned parameters can be used to remove poor annotators in a crowdsourced annotation task and to better inform the process M. Tagliasacchi, Crowdsourcing for Multimedia Retrieval, Summer School on Social Media Modeling and Search, Sep

45 further reading if you who want to know more Aashish Sheshadri and Matthew Lease SQUARE: A Benchmark for Research on Computing Crowd Consensus In Proc. 1st AAAI Conference on Human Computation (HCOMP) 2013

46 3.2. designing crowdsourcing systems

47 let s say you have a job for MTurk

48 crowdsourcing tasks in MTurk: tricks of the trade ownership: host your data on Mturk or not? biases: MTurkers aren t fair sample of world population engagement: is the task fun? entertaining? complexity: how difficult or time-involved is the task? payment: what is a fair rate for workers? interaction: create community of trusted workers geography: limit workers to one country/region? spam: use tricks to verify no spammers fill tasks qualifications: engage workers of recognized quality quality: use model to reduce effect of bad responses W. Mason and S. Suri, Conducting behavioral research on Amazon s Mechanical Turk, Behavior Research Methods, Volume 44, Vol. 1 pp 1-23, Mar. 2012

49 for further details paper: W. Mason and S. Suri, Conducting behavioral research on Amazon s Mechanical Turk, Behavior Research Methods, Volume 44, Vol. 1 pp 1-23, Mar slides:

50 why do crowds participate? motivations (Deci & Ryan, 1985) intrinsic: doing an activity for its inherent satisfactions rather than for some separable consequence + fun + challenge extrinsic: an activity that is done in order to attain some separable outcome + financial reward + social pressure extrinsec motivations tend to undermine intrinsic ones E. L. Deci and R. M Ryan, Intrinsic Motivation and Self-determination in Human Behavior, Plenum Press, 1985 D. C. Brabham, Crowdsourcing, MIT Press, 2013

51 why do crowds participate? to earn money to develop creative skills to network with other creative professionals to build a portfolio for future employers to challenge oneself to solve a tough problem to socialize and make friends to pass time when bored to contribute to a larger project of common interest to share with others to have fun D. C. Brabham, Crowdsourcing, MIT Press, 2013 credit (cc):

52 3.3. (a few other) open issues

53 innovation: crowdsourcing & google glass OpenGlass project: applications to help blind and visually impaired users identify objects via crowdsourcing

54 discovery: citizen science "Foldit attempts to predict the structure of a protein by taking advantage of humans' puzzle-solving intuitions and having people play competitively to fold the best proteins,"

56 crowdsourcing as practice intellectual property & copyright + who owns my idea if it wins? if if doesn t? unfair business practices + crowdsourcing used for manipulation: fake reviews of products & services labor rights + fair payment, spec work, transnational jobs, + does crowdsourcing dump salaries? credit (cc): D. C. Brabham, Crowdsourcing, MIT Press, 2013

57 Mary L Gray (MSR): Ethnographic study of the people who offer projects and the workers who provide labor through Mechanical Turk. There's a conversation in media studies right now about "immaterial labor" as part of the new economy of digital media I want to look at this other world of labor to find out who "Mechanical Turks" are, what motivates "buyers" and "sellers" of MTURK skills, and what kind of labor they feel they're exchanging. The project offers a way to think through the politics and ethics of MTURK and the material labor it reflects.

58 what to remember crowdsourcing & social media have many common points from online participation to incentives & sustainability crowdsourcing is a very active research field understanding of workers & systems models & frameworks for system design new uses & applications many issues remain open both technical & societal

59 questions? @dgaticaperez

Web Science & Technologies University of Koblenz Landau, Germany. Human Computation. Cristina Sarasua

Web Science & Technologies University of Koblenz Landau, Germany Table of contents Human computation A definition for human computation Different models for human computation Games with a purpose Crowdsourcing