Information Services with Social Components

Size: px

Start display at page:

Download "Information Services with Social Components"

Jody Wiggins
5 years ago
Views:

1 Information Services with Social Components Wolf-Tilo Balke Institute for Information Systems Technische Universität Braunschweig

Semantic Metadata Most Information services

created just in time also for unexpected

2 Semantic Metadata Most Information services still rely on metadata But how to know what metadata will be needed at content creation time? Can missing metadata be efficiently created just in time also for unexpected queries? Joint work with Joachim Selke & Christoph Lofi 2

3 Information Services Complete missing information For targeted information provisioning For better indexing & searching For personalization Where to get this information? Extract it from the Social Web? 3

4 Information Services Example: recommender systems 4

5 Information Services Example: ESP Game & Google Image Labeler Idea: Games with a purpose Image Labeling: Guess your partner s tags, and both score. No payment necessary 5

Aims at tapping the the Wisdom of the Crowd Under certain conditions

6 Crowdsourcing Hot and emerging paradigm Vaguely defined concept: Concepts for fostering human collaboration to solve complex problems. Aims at tapping the the Wisdom of the Crowd Under certain conditions large crowds of people are able to perform highly effective decisions 6

7 Crowdsourcing Examples: Building complex artefacts Knowledge: Wikipedia.org Software: Linux, Apache Content Creation YouTube, Flickr User opinions IMDb, Netflix, Amazon Networking Facebook. LinkedIn etc. 7

8 Crowdsourcing Four challenges need to be overcome How to recruit and retain users? What contributions can users make? How to combine the contributions to solve the target problem? How to evaluate users and their contributions? 8

Problem: Mission cannot easily be changed, only specialized tasks solvable on

9 Crowdsourcing Most platforms rely on volunteers Intrinsically motivated Users believe in the mission of the platform Users directly profit from the platform Problem: Mission cannot easily be changed, only specialized tasks solvable on each platform Communities have to be carefully fostered, but are hard to control 9

10 Generic Crowdsourcing Generic Task-Based Crowdsourcing General purpose platforms can facilitate virtually any task for anybody Workers are attracted and retained by paying money 10

Generic Crowdsourcing Clients can initiate a large crowd-sourcing task Define the user interface Define how the task is broken down to individual

11 Generic Crowdsourcing Clients can initiate a large crowd-sourcing task Define the user interface Define how the task is broken down to individual work packages: HITs (Human Intelligence Tasks) Define the overall workload Define how individual results are aggregated Define payment per HIT 11

12 Generic Crowdsourcing Workers solve task Short description of task Transparent payment per HIT Solves task using user interface provided by client Can provide feedback with respect to task and its initiator 12

com/ Laboral Centro de Arte, Gijon, Spain Japan Media Arts Festival,

13 Generic Crowdsourcing Popular example from art: Aaron Koblin Laboral Centro de Arte, Gijon, Spain Japan Media Arts Festival, Tokyo, Japan Apex Gallery, New York, USA ElectroFringe, New Castle, Australia Media Art Friesland, The Netherlands 13

14 Generic Crowdsourcing You get what you pay for sheep = 200 USD 14

15 Generic Crowdsourcing 15

16 Generic Crowdsourcing Popular examples from art reloaded How about more detailed instructions? 16

17 Generic Crowdsourcing 17

18 Real World Applications Crowd-Enabled Databases Core idea: Build a database engine which can dynamically crowdsource certain operations Complete missing data during query time Incomplete tuples (CNULL values) Elicit completely new tuples Use human intelligence operators Entity resolution Similarity rankings etc. 18

19 Crowd-Enabled DB 19

Classification of CS Tasks The ease-of-use and reliability of crowdsourcing tasks varies with the respective use case In general, three variables have to be controlled Answer/Solution Quality,

20 Classification of CS Tasks The ease-of-use and reliability of crowdsourcing tasks varies with the respective use case In general, three variables have to be controlled Answer/Solution Quality, impacted by Worker diligence Worker maliciousness Worker quality and skills Execution Time Job attractiveness (payment vs. time) Worker pool size Costs Number of HITs costs per HIT (affected by time and skill needed) Quality control overhead 20

Classification of CS Tasks Two general discriminating properties impacting these variables can be identified Ambiguity of the tasks solutions For a given solution, can we indisputably decide if it is

21 Classification of CS Tasks Two general discriminating properties impacting these variables can be identified Ambiguity of the tasks solutions For a given solution, can we indisputably decide if it is correct or wrong? Factual tasks (best case) Can we at least reach a community consensus? i.e. answer is considered correct by most people Consensual tasks (not-so-good case) Is there no correct answer? Answers completely subjective? Opinionated tasks (luckily, uninteresting case for most computer science tasks) 21

22 Classification of CS Tasks Required level of worker expertise / skill Can anybody solve the tasks? General worker pool can be used Are special skills / background knowledge required? Worker pool must be filtered Expert users must be found 22

23 factual level of answer ambiguity / agreement consensual opinionated Classification of CS Tasks Examples: What is the nicest color? What political party will you vote? Examples: What is the best operating system? II Examples: Ambiguous classification Does the person on this photo look happy? Is this YouTube Video funny? IV Examples: Is Vertigo a violent movie? Is the VW Amarok a car suited for families? I Examples: Find information on the Web When was Albert Einstein born? Manual OCR Simple cognitive classification Is there a person on this photo? III Examples: Find specific information What is the complexity class of deciding Horn logic? any user some users only question answerable by 23

24 Experiment Basic Settings: Amazon Mechanical Turk Judge 1000 random movies Consider only movies which have consensual genre classifications in IMDb, Rotten Tomatoes, and Netflix Only 10,562 movies overall Use these movies as truth Majority vote of 10 workers each No Gold questions $0.02 per HIT with 10 movies 24

Experiment Result (stop after $20; 10,000 answers) 105 minutes (1:45 hours) 89% reached a consensus 59% of these movies are classified correctly What went wrong? Malicious workers!

25 Experiment Result (stop after $20; 10,000 answers) 105 minutes (1:45 hours) 89% reached a consensus 59% of these movies are classified correctly What went wrong? Malicious workers! 62% selected comedy (first choice in form) 30% of all movies in test set are indeed comedies 24% selected no comedy 70% of all movies in test set are no comedies 14% selected I don t know this movie 25

Example Observation: the test set contains some very obscure movies Quick survey among students: knew only 10%-20% But: Many workers claimed to know all movies Judged 56% of

26 Example Observation: the test set contains some very obscure movies Quick survey among students: knew only 10%-20% But: Many workers claimed to know all movies Judged 56% of all movies as comedies, 44% as no comedy Originate just from two distinct countries All others workers: Knew only 26% of all movies 32% comedy 68% no comedy Realistic values! 26

Example Adjusted Settings: Similar to experiment above, but exclude all workers from the two offending countries Hopefully, only trustworthy workers remain Result (stop after $20; 10,000 look-ups)

27 Example Adjusted Settings: Similar to experiment above, but exclude all workers from the two offending countries Hopefully, only trustworthy workers remain Result (stop after $20; 10,000 look-ups) 116 minutes (1:56 hours) 63% of all movies reached consensus Of those, 79% are classified correctly Result still disappointing Obscure movies do not reach consensus Consensus still not reliable 27

Hybrid Approaches How to perform better? Employ hybrid techniques combining crowdsourcing, information extraction, machine learning and the Social Web!

28 Hybrid Approaches How to perform better? Employ hybrid techniques combining crowdsourcing, information extraction, machine learning and the Social Web! Tackle the following challenges Performance Drastically speed up crowdsourcing times Costs Require just few crowdsourcing HITs for obtaining a large number of judgements Data Quality Circumvent the impact of malicious workers Reliably obtain judgements for even obscure and rare items 28

29 Hybrid Approaches Reconsider crowd-enabled databases Large table with movies e.g. like IMDb, ~2 Million movies Task Introduce new column with a rating for humour (0-10) Traditional approach Create crowd-sourcing task asking users for judgement Consensual result requiring background knowledge Extremely challenging (and expensive) task! 29

30 Hybrid Approaches Can we do better? Let s take all the social web information into account! Massive amount of data in acceptable quality Already successfully used for generating recommendations 30

31 The Social Web The Social Web as a Data Source has become common-place Collect information before buying products (reviews) Recommend news articles, movies, books, Mostly all this data is aggregated into a rating Easy to do, rich in information, and rather ubiquitous Valuable to extract: collaborative filtering, etc. 31

32 Perceptual Spaces Idea: Each user has personal likes/dislikes, preferences, etc. that explain the respective rating behaviour Ratings of each individual will be rather consistent regarding likes/dislikes a systematic bias How to dissemble ratings into the individual biases? Let users and items be d-dimensional points Coordinates of a user represent his/her personality (bias) Coordinates of an item represent its profile regarding personality traits 32

ratings for the purpose of recommending new unrated items Ratings are seen as a

33 Perceptual Spaces Building the perceptual space Possible from ratings, review texts, tags, Factor Models Developed to estimate the value of non-observed ratings for the purpose of recommending new unrated items Ratings are seen as a function of user vectors and item vectors Prominent factor models: SVD, Euclidian embedding, 33

34 Perceptual Spaces 34

Perceptual Spaces Intuition There are good and bad movies Average rating per movie Movie bias compared to the average e.g. The Good, the Bad, and the Ugly is a good movie: 9.1 vs. 6.

35 Perceptual Spaces Intuition There are good and bad movies Average rating per movie Movie bias compared to the average e.g. The Good, the Bad, and the Ugly is a good movie: 9.1 vs. 6.3 on average (Bias +1.8) There are good-natured and bad-natured users i.e. providing generally more positive or more negative ratings Average per-user rating e.g. user Bob always rates movies worse than the average user (negative bias) 35

36 Perceptual Spaces Modeling A user without any preference regarding a movie s properties should rate a movie as Average rating of all movies + Movie Bias + User Bias If a rating diverges from this estimation, then he/she expresses preferences There are some properties he specifically likes, leading to better ratings e.g. like Science Fiction and Giant Monsters 36

37 How to use a Perceptual Space? Extract the correct distances regarding the topic of interest from the perceptual space However, the data is hidden in the space! What dimensions should contribute to the distances? Main idea: train a classifier via crowdsourcing Provide training set via the crowd: positive and negative examples for humorous movies, good books, Non-linear SVM for classification Non-linear regression for values 37

38 How to use a Perceptual Space? Tags Reviews The Social Web Ratings Links Extract Perceptual space Query Result Crowdenabled DB HITs Crowdsourcing service 38

39 Experimental Results Yelp.com Large, but only mildly motivated community Restaurant rating of San Francisco 3.8k restaurants, 128k users, 626k ratings No additional tuning, evaluate against truth provided by human expert editors 39

40 Summary Discussion of crowdsourcing for real world applications Information systems, recommendations, etc. Quality of crowdsourcing tasks needs to be addressed Correctness, time, and costs What type of task, possible quality assurance, Training classifiers over perceptual spaces can solve the problem to some degree 40

41 References Anhai Doan, Ramakrishnan, R., Halevy, A.Y.: Crowdsourcing Systems on the World-Wide Web. Communications of the ACM (CACM), No. 54, Franklin, M., Kossmann, D., Kraska, T., Ramesh, S., Xin, R.: CrowdDB: Answering Queries with Crowdsourcing. ACM SIGMOD Int. Conf. on Management of Data, Athens, Greece, Selke, J., Lofi, C., Balke, W.-T.: Pushing the Boundaries of Crowd-Enabled Databases with Query-Driven Schema Expansion. 38th Int. Conf. on Very Large Data Bases (VLDB), in PVLDB 5(2), Istanbul, Turkey, Demartini, G., Difallah, D. E., Cudré-Mauroux, P.: ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-Scale Entity Linking. 21st Int. Conf. on World Wide Web (WWW), Lyon, France,

42 Thanks for Your Attention Questions?

Outline. Collective Intelligence. Collective intelligence & Groupware. Collective intelligence. Master Recherche - Université Paris-Sud

Outline. Collective Intelligence. Collective intelligence & Groupware. Collective intelligence. Master Recherche - Université Paris-Sud Outline Online communities Collective Intelligence Michel Beaudouin-Lafon Social media Recommender systems Université Paris-Sud mbl@lri.fr Crowdsourcing Risks and challenges Collective intelligence Idea