Ranking the annotators: An agreement study on argumentation structure

Transcription

1 Ranking the annotators: An agreement study on argumentation structure Andreas Peldszus Manfred Stede Applied Computational Linguistics, University of Potsdam The 7th Linguistic Annotation Workshop Interoperability with Discourse ACL Workshop, Sofia, August 8-9, 2013

2 Introduction classic reliability study 2 or 3 annotators authors, field experts, at least motivated and experienced annotators measure agreement, identify sources of disagreement

3 Introduction classic reliability study 2 or 3 annotators authors, field experts, at least motivated and experienced annotators measure agreement, identify sources of disagreement crowd-sourced corpus 100-x annotators crowd bias correction [Snow et al., 2008] outlier identification, find systematic differences [Bhardwaj et al., 2010] spammer detection [Raykar and Yu, 2012]

4 Introduction classic reliability study 2 or 3 annotators authors, field experts, at least motivated and experienced annotators measure agreement, identify sources of disagreement classroom annotation annotators students with different ability and motivation, obligatory participation do both: test reliabilty & identify and group characteristic annotation behaviour crowd-sourced corpus 100-x annotators crowd bias correction [Snow et al., 2008] outlier identification, find systematic differences [Bhardwaj et al., 2010] spammer detection [Raykar and Yu, 2012]

5 Outline 1 Introduction 2 Experiment 3 Evaluation 4 Ranking and clustering the annotators

6 Experiment Task: Argumentation Structure Scheme based on Freeman [1991, 2011] node types = argumentative role proponent (presents and defends claims) opponent (critically questions) link types = argumentative function support own claims (normally, by example) attack other s claims (rebut, undercut)

7 Experiment Task: Argumentation Structure Scheme based on Freeman [1991, 2011] node types = argumentative role proponent (presents and defends claims) opponent (critically questions) link types = argumentative function support own claims (normally, by example) attack other s claims (rebut, undercut)

8 Experiment Task: Argumentation Structure Scheme based on Freeman [1991, 2011] node types = argumentative role proponent (presents and defends claims) opponent (critically questions) link types = argumentative function support own claims (normally, by example) attack other s claims (rebut, undercut)

9 Experiment Task: Argumentation Structure Scheme based on Freeman [1991, 2011] node types = argumentative role proponent (presents and defends claims) opponent (critically questions) link types = argumentative function support own claims (normally, by example) attack other s claims (rebut, undercut) This annotation is tough! fully connected discourse structure unitizing ADUs from EDUs is already a complex text-understanding task

10 Experiment Data: Micro-Texts Thus, we use micro-texts: 23 short, constructed, German texts each text exactly 5 segments long each segment is argumentatively relevant covering different argumentative configurations A (translated) example [ Energy-saving light bulbs contain a considerable amount of toxic substances. ] 1 [ A customary lamp can for instance contain up to five milligrams of quicksilver. ] 2 [ For this reason, they should be taken off the market, ] 3 [ unless they are virtually unbreakable. ] 4 [ This, however, is simply not case. ] 5

11 Experiment Data: Micro-Texts Thus, we use micro-texts: 23 short, constructed, German texts each text exactly 5 segments long each segment is argumentatively relevant covering different argumentative configurations A (translated) example [ Energy-saving light bulbs contain a considerable amount of toxic substances. ] 1 [ A customary lamp can for instance contain up to five milligrams of quicksilver. ] 2 [ For this reason, they should be taken off the market, ] 3 [ unless they are virtually unbreakable. ] 4 [ This, however, is simply not case. ] 5

12 Experiment Setup: Classroom Annotation Obligatory annotation in class with 26 undergraduate students: minimal training - 5 min. introduction - 30 min. reading guidelines (6p.) - very brief question answering 45 min. annotation Annotation in three steps: identify central claim / thesis decide on argumentative role for each segment decide on argumentative function for each segment

13 Experiment Setup: Classroom Annotation Obligatory annotation in class with 26 undergraduate students: minimal training - 5 min. introduction - 30 min. reading guidelines (6p.) - very brief question answering 45 min. annotation Annotation in three steps: identify central claim / thesis decide on argumentative role for each segment decide on argumentative function for each segment

14 Evaluation: Preparation Rewrite graphs as a list of (relational) segment labels 1:PSNS(3) 2:PSES(1) 3:PT() 4:OARS(3) 5:PARS(4)

15 Evaluation: Results level #cats κ A O A E α D O D E role+type+comb+target (71) unweighted scores in κ [Fleiss, 1971], weighted scores in α [Krippendorff, 1980] low agreement for the full task varying difficulty on the simple levels other complex levels: target identification has only small impact hierarchically weighted IAA yields slightly better results

16 Evaluation: Results level #cats κ A O A E α D O D E role typegen type comb target (9) role+type+comb+target (71) unweighted scores in κ [Fleiss, 1971], weighted scores in α [Krippendorff, 1980] low agreement for the full task varying difficulty on the simple levels other complex levels: target identification has only small impact hierarchically weighted IAA yields slightly better results

17 Evaluation: Results level #cats κ A O A E α D O D E role typegen type comb target (9) role+typegen role+type role+type+comb role+type+comb+target (71) unweighted scores in κ [Fleiss, 1971], weighted scores in α [Krippendorff, 1980] low agreement for the full task varying difficulty on the simple levels other complex levels: target identification has only small impact hierarchically weighted IAA yields slightly better results

18 Evaluation: Results level #cats κ A O A E α D O D E role typegen type comb target (9) role+typegen role+type role+type+comb role+type+comb+target (71) unweighted scores in κ [Fleiss, 1971], weighted scores in α [Krippendorff, 1980] low agreement for the full task varying difficulty on the simple levels other complex levels: target identification has only small impact hierarchically weighted IAA yields slightly better results

19 Evaluation: Category confusions studying all individual confusion matrices not feasible: 26 annotators, 325 different pairs of annotators Cinková et al. [2012]: sum up all confusion matrices and build a probabilistic confusion matrix

20 Evaluation: Category confusions studying all individual confusion matrices not feasible: 26 annotators, 325 different pairs of annotators Cinková et al. [2012]: sum up all confusion matrices and build a probabilistic confusion matrix

21 Evaluation: Category confusions studying all individual confusion matrices not feasible: 26 annotators, 325 different pairs of annotators Cinková et al. [2012]: sum up all confusion matrices and build a probabilistic confusion matrix

22 Evaluation: Category confusions studying all individual confusion matrices not feasible: 26 annotators, 325 different pairs of annotators Cinková et al. [2012]: sum up all confusion matrices and build a probabilistic confusion matrix PT PSN PSE PAR PAU OSN OSE OAR OAU? PT PSN PSE PAR PAU OSN OSE OAR OAU ? for the role+type -level;? = missing annotations

23 Evaluation: Category confusions studying all individual confusion matrices not feasible: 26 annotators, 325 different pairs of annotators Cinková et al. [2012]: sum up all confusion matrices and build a probabilistic confusion matrix PT PSN PSE PAR PAU OSN OSE OAR OAU? PT PSN PSE PAR PAU OSN OSE OAR OAU ? for the role+type -level;? = missing annotations

24 Evaluation: Category confusions studying all individual confusion matrices not feasible: 26 annotators, 325 different pairs of annotators Cinková et al. [2012]: sum up all confusion matrices and build a probabilistic confusion matrix PT PSN PSE PAR PAU OSN OSE OAR OAU? PT PSN PSE PAR PAU OSN OSE OAR OAU ? for the role+type -level;? = missing annotations

25 Evaluation: Comparison with Gold-Data

26 Evaluation: Comparison with Gold-Data Distribution of annotator s F1 score per level, macro-averaged over categories role typegen type comb target role+typegen role+type ro+ty+co ro+ty+co+ta central-claim

27 Ranking and clustering the annotators Questions: What range of agreement is possible in this group of annotators? How to give structure to this inhomogenous group of annotators? How to identify subgroups of good annotators, how to sort out bad ones without too much gold data?

28 Ranking and clustering the annotators Questions: What range of agreement is possible in this group of annotators? How to give structure to this inhomogenous group of annotators? How to identify subgroups of good annotators, how to sort out bad ones without too much gold data?

29 Ranking and clustering the annotators Questions: What range of agreement is possible in this group of annotators? How to give structure to this inhomogenous group of annotators? How to identify subgroups of good annotators, how to sort out bad ones without too much gold data? Ranking by thesis F1

30 Ranking the annotators: by central claim F1

31 Ranking the annotators: by central claim F1 Agreement for the n-best annotators ordered by central claim F1 1.0 role+type+comb+target role+type+comb target typegen role role+type comb type role+typegen

32 Ranking and clustering the annotators Ranking by thesis F1 still requires some gold data identifies bad annotators identifies good annotators

33 Ranking and clustering the annotators Ranking by thesis F1 Ranking by cat. distr. still requires some gold data identifies bad annotators identifies good annotators

34 Ranking the annotators: by category distributions Deviation from average category distribution: no attacks, only support anno PT PSN PSE PAR PAU OSN OSE OAR OAU? gold A A A A A A A A A A A A A A A A A A A A A A A A A A gold

35 Ranking the annotators: by category distributions Deviation from average category distribution: no proponent attacks anno PT PSN PSE PAR PAU OSN OSE OAR OAU? gold A A A A A A A A A A A A A A A A A A A A A A A A A A gold

36 Ranking the annotators: by category distributions Deviation from average category distribution: missing annotations anno PT PSN PSE PAR PAU OSN OSE OAR OAU? gold A A A A A A A A A A A A A A A A A A A A A A A A A A gold

37 Ranking and clustering the annotators Ranking by thesis F1 still requires some gold data identifies bad annotators identifies good annotators Ranking by cat. distr. no gold data required identifies outliers but beware: outliers could also be above average good annotators

38 Ranking and clustering the annotators Ranking by thesis F1 still requires some gold data identifies bad annotators identifies good annotators Ranking by cat. distr. no gold data required identifies outliers but beware: outliers could also be above average good annotators Clustering by agreement

39 Clustering the annotators Agglomerative hierarchical clustering: initialize clusters as singletons for each annotator while clusters > 1 do: calc κ for all pairs of clusters merge cluster pair with highest agreement

40 Clustering the annotators Agglomerative hierarchical clustering: initialize clusters as singletons for each annotator while clusters > 1 do: calc κ for all pairs of clusters merge cluster pair with highest agreement

41 Clustering the annotators Agglomerative hierarchical clustering: 0.6 initialize clusters as singletons for each annotator while clusters > 1 do: calc κ for all pairs of clusters merge cluster pair with highest agreement N-#00 N-#03 N-#05 N-#01 N-#06 N-#09 N-#04 N-#02 N-#07 N-#08 simulation: noise and systematic differences F-#07 F-#01 F-#09 F-#03 F-#08 F-#05 F-#00 F-#02 F-#04 F-#06

42 Clustering the annotators Agglomerative hierarchical clustering: initialize clusters as singletons for each annotator while clusters > 1 do: calc κ for all pairs of clusters merge cluster pair with highest agreement N-#05 N-#03 N-#15 N-#14 N-#01 N-#07 N-#12 N-#13 N-#02 N-#09 simulation: noise but no systematic differences N-#10 N-#04 N-#18 N-#16 N-#00 N-#06 N-#08 N-#11 N-#17 N-#19

43 Clustering the annotators: Results for role+type linear growth, no strong clusters range from κ=0.45 to κ=0.84 conforms with central claim ranking in picking out the same set of reliable and good annotators conforms with both rankings in picking out similar sets of worst annotators A21 A20 A04 A18 A25 A10 A09 A11 A15 A16 A07 A23 A14 A22 A17 A01 A13 A26 A06 A02 A08 A24 A03 A12 A05 A19

44 Clustering the annotators: Results for role+type linear growth, no strong clusters range from κ=0.45 to κ=0.84 conforms with central claim ranking in picking out the same set of reliable and good annotators conforms with both rankings in picking out similar sets of worst annotators A21 A20 A04 A18 A25 A10 A09 A11 A15 A16 A07 A23 A14 A22 A17 A01 A13 A26 A06 A02 A08 A24 A03 A12 A05 A19

45 Clustering the annotators: Results for role+type linear growth, no strong clusters range from κ=0.45 to κ=0.84 conforms with central claim ranking in picking out the same set of reliable and good annotators conforms with both rankings in picking out similar sets of worst annotators A21 A20 A04 A18 A25 A10 A09 A11 A15 A16 A07 A23 A14 A22 A17 A01 A13 A26 A06 A02 A08 A24 A03 A12 A05 A19

46 Clustering the annotators: Results for role+type linear growth, no strong clusters range from κ=0.45 to κ=0.84 conforms with central claim ranking in picking out the same set of reliable and good annotators conforms with both rankings in picking out similar sets of worst annotators A21 A20 A04 A18 A25 A10 A09 A11 A15 A16 A07 A23 A14 A22 A17 A01 A13 A26 A06 A02 A08 A24 A03 A12 A05 A19

47 Clustering the annotators: Results for all levels A21 A15 A16 A20 A04 A18 A11 A14 A23 A17 A25 A09 A12 A22 A13 A26 A02 A01 A07 A06 A10 A19 A05 A08 A24 A role A21 A18 A20 A04 A10 A15 A16 A25 A14 A11 A17 A22 A07 A09 A06 A02 A26 A13 A01 A23 A19 A08 A12 A05 A24 A typegen A21 A18 A20 A04 A10 A25 A15 A16 A07 A09 A14 A11 A22 A06 A26 A17 A01 A13 A23 A02 A08 A24 A03 A19 A12 A type A20 A04 A07 A10 A18 A21 A09 A15 A16 A11 A25 A17 A22 A14 A13 A06 A26 A23 A02 A01 A05 A08 A03 A19 A24 A comb A18 A20 A04 A21 A10 A11 A15 A16 A25 A07 A09 A22 A06 A14 A02 A17 A26 A01 A13 A23 A12 A05 A19 A08 A24 A target A21 A20 A04 A18 A25 A10 A09 A11 A15 A16 A07 A23 A14 A22 A17 A01 A13 A26 A06 A02 A08 A24 A03 A12 A05 A role+type A21 A20 A04 A18 A09 A10 A25 A11 A15 A16 A23 A07 A17 A14 A22 A26 A01 A13 A06 A02 A08 A05 A19 A12 A24 A ro+ty+co A20 A04 A21 A18 A10 A09 A25 A11 A07 A23 A17 A15 A16 A22 A14 A26 A01 A06 A13 A02 A05 A19 A08 A12 A24 A ro+ty+co+ta

48 Ranking and clustering the annotators Ranking by thesis F1 still requires some gold data identifies bad annotators identifies good annotators Ranking by cat. distr. no gold data required identifies outliers but beware: outliers could also be above average good annotators Clustering by agreement no gold data required identifies subgroups with characteristic annotation behaviour identifies good & bad annotators but beware: high agreement best annotators

49 Clustering the annotators: And then? For strong clusters pairs, investigate what makes them so different: 0.6 compare their category distribution compare their typical confusions compare their Krippendorff diagnostics 1.0 N-#00 N-#03 N-#05 N-#01 N-#06 N-#09 N-#04 N-#02 N-#07 N-#08 F-#07 F-#01 F-#09 F-#03 F-#08 F-#05 F-#00 F-#02 F-#04 F-#06...

50 Clustering the annotators: And then? For steadily growing clusters: partial order on annotators by path from best to maximum cluster investigate confusion rate on the growing cluster path conf c1,c 2 = c 1 c 2 c 1 c 1 + c 1 c 2 + c 2 c A21 A20 A04 A18 A25 A10 A09 A11 A15 A16 A07 A23 A14 A22 A17 A01 A13 A26 A06 A02 A08 A24 A03 A12 A05 A19

51 Clustering the annotators: And then? For steadily growing clusters: partial order on annotators by path from best to maximum cluster investigate confusion rate on the growing cluster path PAR+PAU OAR+OAU PT+PSN PSN+PAU PSN+PSE OAU+OSN 0.10 c 1 c conf c1,c 2 = c 1 c 1 + c 1 c 2 + c 2 c

52 Conclusions analyse the possible interpretations of the guidelines in a fine-grained manner by using more annotators learn about the task difficulty identify subgroups of good & reliable annotators, even if overall agreement is dissatisfactory Thank You!

53 Conclusions analyse the possible interpretations of the guidelines in a fine-grained manner by using more annotators learn about the task difficulty identify subgroups of good & reliable annotators, even if overall agreement is dissatisfactory Thank You!

54 Literatur I Vikas Bhardwaj, Rebecca J. Passonneau, Ansaf Salleb-Aouissi, and Nancy Ide. Anveshan: a framework for analysis of multiple annotators labeling behavior. In Proceedings of the Fourth Linguistic Annotation Workshop, LAW IV 10, pages 47 55, Stroudsburg, PA, USA, Association for Computational Linguistics. Silvie Cinková, Martin Holub, and Vincent Kríž. Managing uncertainty in semantic tagging. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL 12, pages , Stroudsburg, PA, USA, Association for Computational Linguistics. Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): , James B. Freeman. Dialectics and the Macrostructure of Argument. Foris, Berlin, James B. Freeman. Argument Structure: Representation and Theory. Argumentation Library (18). Springer, Klaus Krippendorff. Content Analysis: An Introduction to its Methodology. Sage Publications, Beverly Hills, CA, Vikas C. Raykar and Shipeng Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research, 13: , Rion Snow, Brendan O Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 08, pages , Stroudsburg, PA, USA, Association for Computational Linguistics.

55 Evaluation: Krippendorff s Category Definition Test Krippendorff [1980] diagnostics: systematically compare agreement on the original tagset with that on a reduced tagset category definition test: one category of interest against the rest compare the resulting κ values to see which category is distinguished better from the rest category κ A O A E PT PSE PSN OAR PAR OSN OAU PAU level role+type ; base κ=0.45

56 Evaluation: Krippendorff s Category Distinction Test Krippendorff [1980] diagnostics: systematically compare agreement on the original tagset with that on a reduced tagset category distinction test: only collapse one pair of categories κ tells you how much you loose due to confusions between those two categories category pair κ A O A E OAR+OAU PAR+PAU OAR+OSN PSN+PSE OAR+PAR PSN+OSN PAR+OSN level role+type ; base κ=0.45

57 Evaluation: Text-specific agreement κ for the full task ( role+type+comb+target )

58 Scores for the 6-best annotators role+type ro+ty+co+ta F κ α PT PSN PSE PAR PAU OSN OSE OAR OAU? PT PSN PSE PAR PAU OSN OSE OAR OAU ? for the role+type -level