Outliers Detection vs. Control Questions to Ensure Reliable Results in Crowdsourcing. A Speech Quality Assessment Case Study

Transcription

1 Outliers Detectin vs. Cntrl Questins t Ensure Reliable Results in Crwdsurcing. A Speech Quality Assessment Case Study Rafael Zequeira Jiménez, Laura Fernández Gallard, Sebastian Möller Quality and Usability Lab, Technische Universität Berlin

2 Crwdsurcing Page 2

3 Mtivatin Speech quality is imprtant fr the Quality f Experience (QE) in: audi bks virtual r rbtic cnversatinal agents Page 3 * The cllected ratings can be used t train AI systems t predict the speech quality autmatically

4 Mtivatin Speech quality experiments traditinally cnducted in Labratry - Prfessinal audi equipment - Sundprf rm - Limited number f participants Page 4

5 Crwdsurcing Study Cnducted a speech quality assessment experiment Crwd-wrkers were presented with 20 speech stimuli Opinin abut verall quality gathered in a 5-pint scale

6 Speech Material: Database number 501 frm ITU-T Rec. P Germans were recrded per cnditin 200 speech stimuli (9s lng n avg.) 50 degradatin cnditins: narrw- & wide- band tempral clipping signal-crrelated nise, cmbinatins f these degradatins The database cntains quality ratings t the 200 stimuli made by 24 different native German listeners, in accrdance t ITU-T Rec. P.800 Page 6

7 Study Cnditins: Address the study t native Germans Cllect 24 ratings per stimulus frm different listeners Experiment in accrdance with the ITU-T Rec. P.800 Page 7

8 Crwdsurcing Platfrm: German based CS platfrm Reprted 1 millin glbal users in September 2017 Mst f their users are frm German speaking cuntries Page 8

9 Crwdsurcing Experiment Screening task t recruit German listeners Speech quality assessment task: Qualificatin phase Speech quality assessment Page 9

10 Crwdsurcing Experiment Qualificatin Speech Quality Assessment cnsent request use f headphne audi Math trapping questin 5 stimuli as an anchr intrductin envirnment recrd up t 15s 20 stimuli t rate 2 trapping Questin Page 10

11 Crwdsurcing Experiment Qualificatin Speech Quality Assessment cnsent request use f headphne audi Math trapping questin 5 stimuli as an anchr intrductin envirnment recrd up t 15s 20 stimuli t rate 2 trapping Questin Page 11

12 Results 87 wrkers participated in the study 8 wrkers failed the Qualificatin phase 53 unique listeners: 60,4% males 96,2% native Germans prvided 4840 ratings the cllected ratings accunt fr 24 t 26 assessment frm different listeners per file Page 12

13 Crwdsurcing vs. Labratry Spearman s rank-rder crrelatin: rh = 0,864 (p<0,001) Mntnic relatinship between Lab- and CS- MOS Rt Mean Square Errr: RMSE=0,474 Page 13

14 Filtering frm unreliable wrkers Wrk in [1] and [2] recmmends: the use f trapping questin, t catch inattentive users when the user fail, then all f their ratings are discarded This apprach was effective in [1] and imprved slightly the results in [2] [1] B. Naderi, T. Plzehl, I. Wechsung, F. Köster, and S. Möller, Effect f Trapping Questins n the Reliability f Speech Quality Judgments in a Crwdsurcing Paradigm, in Interspeech, 2015, pp [2] R. Zequeira Jiménez, L. Fernández Gallard, and S. Möller, Scring Vice Likability using Pair-Cmparisn: Labratry vs. Crwdsurcing Apprach, in Ninth Internatinal Cnference n Quality f Multimedia Experience (QMEX), 2017, pp Page 14

15 Filtering frm unreliable wrkers A wrker is unreliable r untrustwrthy when: s/he fails the trapping questin in the SQAT s/he fails the Qualificatin mre than nce Page 15

16 Filtering frm unreliable wrkers A wrker is unreliable r untrustwrthy when: s/he fails the trapping questin in the SQAT s/he fails the Qualificatin mre than nce Page 16

17 Filtering frm unreliable wrkers Discarded 320 ratings in ttal frm W4, W5, W7 W6 did nt cnduct the SQAT Methd: filtering by trapping questin" (F-TQ) Spearman s rank-rder crrelatin n 4520 ratings: rh = 0,862 (p<0,001) When discarding all the wrkers (F-TQ ) : rh = 0,854 (p<0,001) Page 17

18 Outlier Detectin utliers: ratings abve 1,5 interquartile range (IQR) depicted by circles extreme utliers: ratings at 3,0 IQR r abve depicted by asterisks Page 18

19 Outlier Detectin Discarded 122 ratings identified as extreme utliers Methd: filtering by utlier detectin 1" (F-OD1) Spearman s rank-rder crrelatin: rh = 0,863 (p<0,001) still nt better than the first cefficient when n data was discarded Page 19

20 Outlier Detectin 2 Discarded 1480 ratings frm 12 wrkers that were utliers r extreme utliers three times r mre [5]. Methd: filtering by utlier detectin 2" (F-OD2) Spearman s rank-rder crrelatin: rh = 0,867 (p<0,001) [5] B. Naderi, Mtivatin f Wrkers n Micrtask Crwdsurcing Platfrms. Springer, Page 20

21 Alternative Apprach Applied F-OD1 and F-OD2 and discarded 1529 ratings in ttal. Identify the utliers made by all the wrkers that failed the trapping questins. Then remved 17 ratings. Methd: F-TQ-OD Spearman s rank-rder crrelatin n 3294 ratings: rh = 0,868 (p<0,001) Page 21

22 Results Overview *p < 0,001 Apprach Ratings discarded rh RMSE - 0 0,864* 0,474 F-TQ 320 0,862* 0,476 F-TQ 780 0,854* 0,480 F-OD ,863* 0,477 F-OD ,867* 0,474 F-TQ-OD ,868* 0,479 Page 22

23 Results Cmparisn Apprach Methd Wrkers Discarded Ratings Discarded [6] [7] F-TQ-OD gld standard questins verificatin questins trapping questin + utliers detectin 25% 75% - 34,3% 22% 31,9% [6] T. Hßfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-Gia, and R. Schatz, Quantificatin f YuTube QE via Crwdsurcing, in 2011 IEEE Internatinal Sympsium n Multimedia, 2011, pp [7] J. Redi and I. Pva, Crwdsurcing fr Rating Image Aesthetic Appeal: Better a Paid r a Vlunteer Crwd?, in Internatinal ACM Wrkshp n Crwdsurcing fr Multimedia, 2014, pp Page 23

24 Discussin Apprach Ratings discarded rh RMSE - 0 0,864* 0,474 F-TQ 320 0,862* 0,476 F-TQ 780 0,854* 0,480 F-OD ,863* 0,477 F-OD ,867* 0,474 F-TQ-OD ,868* 0,479 We recmmend t emply F-OD1 in case high crrelatin is nt a pririty. This is the mst cst effective apprach. We recmmend t use F-TQ-OD fr mre accurate results. Page 24

25 Cnclusin Adapted successfully a Labratry listening test t Crwdsurcing Obtained a strng and statistically significant Spearman crrelatin: r=0.868 Tested utliers detectin and trapping questin t filter the data frm unreliable ratings Prpsed a cmbinatin f utlier detectin and trapping questin that leads t mre accurate results Further testing is required t determine fr which type f experiment ur apprach can be applied. Page 25

26 Thank yu fr yur Attentin! Rafael Zequeira