Web Science & Technologies University of Koblenz Landau, Germany. Human Computation. Cristina Sarasua

Transcription

1 Web Science & Technologies University of Koblenz Landau, Germany

2 Table of contents Human computation A definition for human computation Different models for human computation Games with a purpose Crowdsourcing Microtask crowdsourcing Microtask crowdsourcing platforms Challenges in microtask crowdsourcing How to design microtask crowdsourcing CrowdMAP, an example Lessons learned

3 The problem There are some things that only humans can do humans do much better than computers Flowers Yellow tulip blue sky Dark orange chrysanthemum Not the same kind of flower

4 What is human computation?

5 Definition Human computation is a paradigm for utilizing human processing power to solve problems that computers cannot yet solve [Luis von Ahn, 2005] computation that is carried out by humans [Law & von Ahn, 2011] the study of systems where humans perform a major part of the computation or are an integral part of the overall computational system [HCOMP2012] Careful! not every system that has human interaction is a human-based computation system.

6 Example: recaptcha Pictures:

7 Models for human computation We can design a human computation system embedding it in something else (e.g. recaptcha in login) as a stand-alone application Incentivizing with games Incentivizing with money People can be motivated by the idea of (for example) belonging to a community competing against other participants having fun gaining reputation

8 game

9 Games with a purpose (GWAP) A GWAP is a system that outsources a task to humans by means of a game. The human plays and implicitly he is computing People wish to be entertained (von Ahn & Dabbish) Competition among players Scores Players ranking

10 GWAP example I: ESP game & Google Image labeler

11 GWAP example II: Duolingo

12 crowd

13 Crowdsourcing Crowdsourcing: outsourcing a task to the crowd (large group of people) Crowdfunding: funding shared by the crowd. Each participant contributes with some money. (E.g. used to record a music CD) Challenges and contests: competitions where participants are able to win a prize (E.g. used to obtain new business ideas). Sometimes there is only a winner, and sometimes there are several winners. Usually, participation does not guarantee receiving a reward. Citizen Science: scientific tasks carried out by non-professionals. (E.g. used to classify gathered data). Microtask crowdsourcing: simple tasks that are done by the crowd in a labor marketplace, in exchange for a financial reward Macrotask crowdsourcing: large tasks that are accomplished (sometimes by teams) within workplaces. They take more time than microtasks and the reward is proportionally higher.

14 Contest example I: Innocentive

15 Contest example II: TopCoder

16 Citizen science example: Zoouniverse: GalaxyZoo, Seafloor and WhaleFM

17 Microtask crowdsourcing example I: Video annotation Picture:

18 Microtask crowdsourcing example II: Book translation DE EN FR ES

19 Crowdsourcing

20 Microtask crowdsourcing platforms Labor marketplace: an online site where requesters publish the microtasks they would like to be executed by the crowd. The humans who work on the tasks are the so-called workers. Example: Amazon MTurk There are other crowdsourcing platforms that are meant for improving the management of microtasks and workers, which requesters can use to publish their microtasks at marketplaces. Example: CrowdFlower And many others

21 Amazon Mturk (I)

22 Amazon MTurk (II) Mturk is a crowdsourcing platform with a labor marketplace Workers - browse available microtasks (HITs) for them - select microtasks to work on - work and submit responses - get paid Requesters - log in - create microtasks (UI, data, configuration) - publish microtasks - retrieve results Requesters can create microtasks using User interface (manually) API (programmatically) Demo

23 CrowdFlower (I)

24 CrowdFlower (II) CrowdFlower is a crowdsourcing platform without a marketplace, but it allows requesters to publish microtasks in several third-party marketplaces at the same time (e.g. MTurk) Requesters can create microtasks using User interface (manually) API (programmatically) Requesters - log in - create microtasks (UI, data, configuration) - publish microtasks - retrieve results Quality assurance Gold standard: known answer, used to evaluate the performance of workers (trusted vs untrusted workers) Demo

25 Comparing Mturk and CrowdFlower Mturk CrowdFlower Deployment Marketplace Yes No +30 channels Workers Any (filters: geolocation, language, qualification tests) Tasks Any Any Any (filters: geolocation, language, qualification tests) Quality control Reputation, task design Gold standard, task design API Yes Yes

26 CrowdFlower API REST services with methods for Uploading data Creating, ordering, pausing, resuming, deleting jobs Creating, reading units Creating webhooks Reading judgments URL: RESTClient

27 How many people is using this? Mturk: screenshot 17/01/ :14 CrowdFlower: screenshot 17/01/ :18

28 Challenges in microtask crowdsourcing Running complex works (decomposed as simple ones) steps forward: CrowdForge Turkomatic Quality assurance steps forward: TurKit Separation between bias & error (Iperoitis) Microtask scheduling & recommendation The time needed to obtain all the results steps forward: Real-time crowdsourcing

29 Design of microtask crowdsourcing experiments (I) STEP 1: check whether it is suitable for you Does the task look amenable to microtask crowdsourcing? Can the problem be divided into small and simple tasks? Can multiple workers work independently? May workers be incentivized with money? Are the results of the workers measureable? IF answer == yes THEN go on ELSE Sorry, try another kind of technique

30 Design of microtask crowdsourcing experiments (II) STEP 2: plan the content of microtasks and #microtasks Identify the purpose of your microtasks Select the data for the microtasks Individual microtasks vs templates Define a problem statement Define a form to gather the answers from workers Write instructions and give a title to the microtasks Plan how many microtasks / how to divide data Decide whether to create groups of microtasks

31 Design of microtask crowdsourcing experiments (III) STEP 3: plan microtask configuration & result management Define the reward per microtask Estimate total budget Calculate the time needed to complete a microtask Think about the expiration date Define who will be able to work on your microtasks (e.g. geolocation restrictions, skills) Define how to detect/filter SPAM Define how to assess whether a result is correct or not Define how to aggregate responses

32 Design of microtask crowdsourcing experiments (IV) STEP 4: identify tools to use Choose a marketplace where you want to publish Decide how to create microtasks (requester UI vs API) Select the crowdsourcing platform to use sometimes they use different terminology Is there any other tool needed? Plugin Create your own extension

33 Design of microtask crowdsourcing experiments (V) STEP 5: publish Think of when to publish Publish in test mode CrowdFlower internal interface) and check Publish some trials in order to check the wording and see how much time it takes to get microtasks completed Update things if results are not good enough Publish the real microtasks Observe microtask evolution (web server / manually) Interact with microtasks in runtime (e.g. possible to update gold standard errors)

34 CrowdMAP: crowdsourcing ontology alignment with microtasks A detailed example of microtask Crowdsourcing

35 CrowdMAP: crowdsourced ontology alignment (I) What is ontology alignment? Person Agent Document Document mynewontology Foaf {mynewontology:person, rdfs:subclassof, foaf:agent} {mynewontology:document, owl:equivalentclass, foaf:document} It can be done manually or automatically It is important for interoperability, integration

36 CrowdMAP: crowdsourced ontology alignment (II) Motivation Problem Automatic tools do not perform perfectly Human intervention is desirable (e.g. review) owl:equivalentclass Conference Congress A mapping Why microtask crowdsourcing? We need people to compute the result Big ontologies might need many people Verifying a mapping is something than can be translated into a simple task Workers might not need much context to find the correct answer (*specific domains)

37 crowdsourced ontology alignment

38 CrowdMAP: crowdsourced ontology alignment (III) The concept Extend current alignment technology with human computation Microtasks containing Do you think there is a relationship between Conference and Congress? Yes / no Pay $ to contributors Publish at marketplace (MTurk) Ontology O1 Ontology O2 Automatic Ontology Alignment Candidate Mappings {cm1 cmn} CrowdMAP Verified mappings {vm vmn} Configuration (param, CML)

39 CrowdMAP: crowdsourced ontology alignment (IV) What workers see (workflow)

40 CrowdMAP: crowdsourced ontology alignment (V) CrowdMAP architecture

41 CrowdMAP: crowdsourced ontology alignment (VI) Experiments at Mturk Ontologies and solutions to the alignments Set of golden mappings microtask = 6 mappings + 1 gold mapping Mturk batches of max. 50 microtasks (HITs) Trusted worker 2 gold mappings Used templates and data $0.01 $ % HITs template X, 50% HITs template X_turned

42 CrowdMAP: crowdsourced ontology alignment (VII) Results Improved precision without loosing recall Average #workers per batch == 11 Average agreement was 87,40% We spent $50 in total, including pilots The set of mappings had size ~ 600 / 40 / 20 Presented Mappings Precision Recall All possibilities (CartP) % correct 50% incorrect % correct 50% incorrect % correct 50% incorrect % correct 50% incorrect Result from algorithm

43 Lessons learned in microtask crowdsourcing It is important to do trials with users, to see how humans react when accomplishing the microtasks before workers provide the final result Workers tend to work on the newest microtasks good to observe published microtasks and restart if necessary Workers talk about requesters reputation keep your workers happy, listen to their With crowdsourcing techniques it is easy and cheap to obtain human contributions Good results require appropriate management of microtasks and workers

44

45 References (I) Luis von Google Techs, Luis von TED, _collaboration.html Thomas W. ISWC, nce/

46 References (II) Ambati, V., Vogel, S., Carbonell, J.G.: Towards task recommendation in micro-task markets. In:. (2011) Cavallo, R., Jain, S.: Efficient crowdsourcing contests. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 2. (2012) Designing games with a purpose. von Ahn, L. and Dabbish, L. In: Commun. ACM 51(2008) Extended Abstracts on Human Factors in Computing Systems. (2010) G. Little, L. Chilton, M.G., Miller, R.: TurKit: tools for iterative tasks on mechanical Turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation. (2009) Games with a purpose. von Ahn, L. In: IEEE Computer 39 (2006) Howe, J. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group. (2008) Howe, L.: Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. 1 edn. Crown Publishing Group, New York, NY, USA (2008) Ipeirotis, P., Provost, F., Wang, J.: Quality management on Amazon Mechanical Turk. In:Proceedings of the ACM SIGKDD Workshop on Human Computation. (2010) Khazankin, R., Psaier, H., Schall, D., Dustdar, S.: Qos-based task scheduling in crowdsourcing environments. In: Proceedings of the 9th international conference on Service-Oriented Computing. (2011) Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: crowdsourcing complex work. In: Proceedings of the 24th annual ACM symposium on User interface software and technology. (2011) Oleson, D., Sorokin, A., Laughlin., G., Hester, V., Le, J., Biewald, L.: Programmatic gold: targeted and scalable quality assurance in crowdsourcing. In: AAAI Workhop on. (2011) Quinn, A.J., Bederson, B.B. Human computation: a survey and taxonomy of a growing field (2011). Ranade, G., Varshney, L. To crowdsource or not crowdsource. In: AAAI Workshops (2012) Ross, J., Irani, L., Silberman, M.S., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers?: shifting demographics in mechanical turk. In: CHI '10 Sarasua, C. Simperl, E. Noy, N.F. Human computation: a survey and taxonomy of a growing field. In: International Semantic Web Conference 1, volume 7649 of Lecture Notes in Computer Science, (2012) Turkomatic: automatic recursive task and workflow design for Mechanical Turk. In: Human factors in computing systems (CHI). (2011) Von Ahn, L.. Doctoral Thesis. School of Computer Science, Carnegie Mellon University (2005)