Chapter 4 Human Evaluation

Transcription

1 Chapter 4 Human Evaluation Human evaluation is a key component in any MT evaluation process. This kind of evaluation acts as a reference key to automatic evaluation process. The automatic metrics is judged good or bad by viewing its correlation with human assessments. In a human evaluation, human evaluators look at the output and judge them by hand. They check whether the ouptut is correct or not. Bilingual human evaluators who can understand both input and output are the best qualified judges for the task. Figure 4.1 shows the process of human evaluation. Figure 4.1 Human Evaluation Process Here, the output of the system is provided to the human evaluator, who evaluates the output on the basis of a subjective questionnaire/metric, based on which an evaluator can judge the output. This is done for each of the sentences which are going to be evaluated. But, judging MT output merely on the basis of correctness is a very harsh method of evaluation. So, MT Researchers have developed metrics onto which an output can be checked. Most of the human evaluation campaigns judge MT outputs based on either comprehensibility or adequacy or fluency or meaning maintenance or 46

2 clarity. In recent years, almost all the evaluation campaigns have been using adequacy and fluency as a measure to register human judgments. These two factors can be incorporated in a single metric or can be provided as separate metrics. Moreover a human evaluator has to adjudge the output on the basis of some scale which may range from metric to metric. Table 4.1 shows adequacy scale for evaluation of sentences and Table 4.2 shows fluency scale. Score Description 5 All Information 4 Most Information 3 Much Information 2 Little Information 1 None Table 4.1: Interpretation of Adequacy on Scale 5 Score Description 5 Flawless English 4 Good English 3 Non Native English 2 Disfluent English 1 Incomprehensible Table 4.2: Interpretation of Fluency on Scale 5 In this chapter we will describe the human evaluation process. We shall describe a novel human evaluation metric that we have developed and its evaluation results on seventeen MT Engines. We shall also compare this metric with Human Adequacy and Fluency using Scale 5 metrics and would analyze its results. 4.1 HEval: Subjective Human Evaluation Metric We have used a human evaluation metric which captures the linguistic features of a language and provides qualitative assessments of the MT engine outputs. The linguistic features that we have considered in the metric are: 1. Translation of Gender and Number of the Noun(s). 2. Identification of the Proper Noun(s). 3. Use of Adjectives and Adverbs corresponding to the Nouns and Verbs. 4. Selection of proper words/synonyms (Lexical Choice). 5. Sequence of phrases and clauses in the translation. 47

3 6. Use of Punctuation Marks in the translation. 7. Translation of tense in the sentence. 8. Translation of Voice in the sentence. 9. Maintaining the semantics of the source sentence in the translation. 10. Fluency of translated text and translator s proficiency. 11. Evaluating the translation of source sentence (With respect to syntax and intended meaning). This metric was first proposed by late Prof. Rekha Govil. We have only perfected this metric and hence named it after her. We have employed a five point scale to measure the quality of the translations. Table 4.3 provides the description of the scales. Score Description 4 Ideal 3 Perfect 2 Acceptable 1 Partially Acceptable 0 Not Acceptable Table 4.3: Interpretation of HEval on Scale 5 All the above features are scored using these scores. An average is computed on all the eleven scores which give us a single objective value. Figure 4.2 shows the working prototype that has been developed. This prototype is being used to evaluate MT Systems developed under English-Indian Languages Machine Translation Systems Project which is sponsored by TDIL Programme, Government of India (Lata and Kumar, 2010). Since the current human evaluation metrics were unable to provide the complete assessment of the dimensions of translation quality as they only focused on just one or two quality measures. Most of the human evaluation campaigns focused on limited features of evaluations. In these evaluations adequacy and fluency being the most important. This kind of evaluation is inappropriate as in practice a human translator does not translate a text just on superficial ratings. While translation, a translator has to look at many factors like gender being addressed in the text, the proper nouns being used, use of adjectives and adverbs etc. Then only a translator comes out with a good translation. Our metric intuitively captures this phenomenon and provides an objective score. Most of the MT systems might not be able to provide appropriate 48

4 results for all the eleven features used in our metric, but they may provide good or acceptable results for some of the features. Judging an MT output on just one/two features/factors might deviate a human judge from making a sound judgment, as one judge, who has looked at all the features, may give a moderate rating to a translation whereas another judge, who might have overlooked some of the features, may give low rating to a translation. This causes an inter-annotator disagreement while evaluating MT Systems. Figure 4.2: HEval: Subjective Human Evaluation Metric Moreover, if a human judge is asked to reevaluate a particular translation again, then he might not assign the same rating to the translation. This poses a serious problem while ascertaining quality of MT systems. HEval addresses these issues by clearly laying down parameters onto which an evaluation can be performed. This provides a crisp and repeatable assessment of MT outputs. Since, each feature clearly states what needs to be done and how to access the same; it reduces the possibility of assigning a low score to a good translation. Table 4.4 provides an example to justify this claim, where we show some of the good and bad translations and compare them with adequacy and fluency scores. These scores are calculated using the score presented in Table 4.1 and 4.2 respectively. 49

5 The scores of HEval Metric for these four translations are shown in table 4.5 along with adequacy and fluency scores. From the table it is evident that HEval metric can provide results in line with Human Adequacy and Fluency measures. Moreover, this metric can even justify its results as it provides more qualitative information about the evaluation. This metric also provide consistent scores when more than one judge evaluate a translation. All the judges came out with almost same scores for the translations produced by different engines. The good translations were consistently given a higher score whereas bad translations were given lower scores. English Most impressive are the black and white chessboard marble floor, the four tall minarets (40 m high) at the corners of the structure, and the majestic dome in the middle. E1 सबस भ वश ल ह क ल और सफ द बस त स गमरमर फश, च र ल ब म न र (40 म टर ऊ च ) स रचन ह, और ब च म र जस ग बद क क न म E2 सबस भ वश ल बस त क ल और सफ द स गमरमर क फश, स रचन क क न पर च र ल ब म न र (40 म टर ऊ च ), और ब च म र जस ग बद ह. Most क क क ल और सफ द स गमरमर chessboard floor, impressive ह E12 च र ल ब minarets क corners पर ( 40 म. ऊ च ) स रचन, म ज टक ग बद म य म और E15 सबस ह क क ल और सफ द स गमरमर म जल, च र ऊ च म न र ( म. ऊ च ) क क न -क न क स रचन, और क म ज टक ग बद क म य म 40 क पर ह Table 4.4: A Case of English-Hindi MT Evaluation (Sent # 150) Engine Overall Adq Flu E1 E2 E12 E15 Human Human Human Human Human Human Human Human Table 4.5: Results of Human Evaluation for Case of Table

6 In the following sections we have provided results of two judges for all seventeen MT systems and have also compared our metric with Human Adequacy and Fluency Metrics on a five point scale. For document and system levels we have averaged the scores for the entire document and system respectively. For sentence level evaluation, we have generated the rank of the translation. 4.2 Evaluation of MT Systems using HEval We used 1300 sentences for all 17 MT Engines which were distributed into 13 documents of 100 sentences each. Combined document wise score of all 17 MT engines is shown in table 4.6. Combined system wise score of all 17 engines are shown in table 4.7. The values in bold represent the highest score achieved by an engine with respected to a particular human evaluator. At document level, out of 13 documents, 8 times all human judges gave the same consistent score, i.e. they all gave a high rank to the same engine. At system level too, the results were repeatable. All the judges unanimously gave high score to Engine E2. This proves the consistency of the metric to provide same results across different judges. This inter-annotator agreement at document and system level was also repeated at sentence level as well. At sentence level, it was not possible to provide results of all the engines for all 1300 sentences, so, the results were consolidated into ranks. At this level, we classified the systems into four categories, (i) combined, which had ranks of all the systems; (ii) web, which only had the ranks of four web based MT engines; (iii) toolkit, which had the ranks of four MT toolkits used in our experiments and (iv) hybrid, which had system combinations based on MT toolkits. Table 4.8 shows the results combined study. Out of 1300 sentences 522 times, E1 was ranked as the best by judge 1. This was repeated with Judge 2, who 473 times gave a higher rank to E1. In all the cases E2 performed second best. Figure 4.3 clarifies this claim. In Web Based MT category, out of 1300 sentences, E1 was given the highest rank by all the judges, which was followed by engine E2. Table 4.9 shows the detailed figures of this study and is summarized by Figure 4.4. In MT Toolkits category, out of 1300 sentences, E15 was 51

7 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E17 Human Doc 1 Human Human Doc 2 Human Human Doc 3 Human Human Doc 4 Human Human Doc 5 Human Human Doc 6 Human Human Doc 7 Human Human Human Doc 8 Human Human Human Doc 9 Doc 10 Human Human Human

8 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E17 Doc 11 Doc 12 Doc 13 Human Human Human Human Human Human Table 4.6: Combined Document Level Scores of HEval Metric Human 1 Human 2 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E Table 4.7: Combined System Level Scores of HEval Metric 53

9 given the highest rank by all the judges, this was followed by E9. Table 4.10 shows the ranks scored by different engines and figure 4.5 summarizes the results. In hybrid Category, out 1300 sentences, E16 was adjudged as the highest ranker followed by E17. Results of this study are shown in table 4.11, and are summarized by figure 4.6. Engine Human 1 Human 2 E E E E E E E E E E E E E E E E E Table 4.8: System Level Ranking of HEval Metric for All MT Engines Engine Human 1 Human 2 E E E E Table 4.9: System Level Ranking of HEval Metric for Web Based MT Engines Engine Human 1 Human 2 E E E E Table 4.10: System Level Ranking of HEval Metric for MT Toolkit Based MT Engines 54

10 Engine Human 1 Human 2 E E E E E E E E E Table 4.11: System Level Ranking of HEval Metric for Hybrid MT Engines Human1 Human E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E17 Figure 4.3: Ranks Given by Human Judges to Combined MT Engines Human1 Human E1 E2 E3 E4 Figure 4.4: Ranks Given by Human Judges to Web Based MT Engines 55

11 Human1 Human E5 E9 E12 E15 Figure 4.5: Ranks Given by Human Judges to Toolkit Based MT Engines E6 E7 E8 E10 E11 E13 E14 E16 E17 Human1 Human2 Figure 4.6: Ranks Given by Human Judges to Hybrid MT Engines Combined Web MT Toolkit Hybrid Agreement Percentage Agreement Percentage Agreement Percentage Agreement Percentage H1 H % % % % Table 4.12: Pairwise Agreement between Judges In order to strengthen our claim we also did pair wise comparisons of MT engines. We took the highest ranking engines from four different categories. In this study, we compared the results of judge 1 with judge 2. In combined category, out of 1300 sentences, 689 times, judge 1 agreed with judge 2 and by giving a higher rank to an MT engine. This accounted for 52% agreement between the two judges. All the agreement scores and their respective percentages are show in Table From the 56

12 table it is clear that the metric can show reasonable agreement at sentence level as well which can further be explained by analyzing the scores given by the judges to different features. Human 1 Human 2 E E E E E E E E E Table 4.13: Spearman s Rho Correlation between HEval and Human Adequacy Scale Correlation of HEval with Human Adequacy Score We compared the results of our metric with Human Adequacy score on scale 5. We took the results of some of the engines from various categories and correlated them with Adequacy score. For this study we used only one human judge to provide adequacy scores of MT engines. We used Spearman s rho statistic to measure correlation. We used Spearman s rho correlation as it is based on ranks and does not show sensitivity towards outliers. Table 4.13 shows the results of this study. Adequacy judgment was done on ten MT Engines, top two ranking engines from web based category. Top three MT engines from toolkit category and four high ranking MT engines from hybrid category. In all the cases E1 scored higher with significant correlation at 99% confidence interval. E2 showed significant correlation at 95% confidence interval. E5 showed significant correlation between Adequacy and Judge2 at 95% confidence interval. It showed positive correlation with Judge1. E6 and E9 showed just showed a positive correlation. E10 showed positive correlation with Judge1 and negative correlation with Judge2. E15 showed significant correlation at 95% confidence interval with all the judges. E16 showed slight negative correlation with judges 2 and positive correlation with judge 1. E17 showed positive correlation with judge 1 and 2. Mostly all the correlation coefficients either showed positive or 57

13 significantly positive correlations. Thus, through this we can deduce that HEval can be a good replacement for Adequacy based evaluations. Human 1 Human 3 E E E E E E E E E Table 4.14: Spearman s Rho Correlation between HEval and Human Fluency Scale Correlation of HEval with Human Fluency Score We compared the results of our metric with Human Fluency score on scale 5. We took the results of same engines from various categories that were used for Adequacy scores. For this study also, we used only one human judge to provide fluency scores of MT engines. Here too, we used Spearman s rho statistic to measure correlation. Table 4.14 shows the results of this study. In all the cases E1 showed positive correlation with all the judges. E2 also showed positive correlation with all the judges. E5 showed significant correlation with all the judges at 95% confidence interval. E6 showed significant correlation with all the judges at 99% confidence interval. E9 showed just showed a significant positive correlation with 95% confidence inerval. E10 showed positive correlation with all the judges. E15 showed positive correlation with judges 2 and showed negative correlation with judge 1. E16 showed slight negative correlation with judges 2 and positive correlation with judge 1. E17 showed negative correlation with judges 2 and positive correlation with judge 1. Since, here also, mostly all the correlation coefficients either showed positive or significantly positive correlations. Thus, through this we can deduce that HEval can be a good replacement for Fluency based evaluations too. 58

14 4.5 Conclusion In this chapter, we showed the working of a new human evaluation metric which was implemented in this research. We showed the design of the metric along with the scoring criteria and the mechanism for scoring the translations using an example. We also compared the results of this metric with the popular human evaluation metrics (Adequacy and Fluency) which have been implemented in almost all the evaluation campaigns. We also correlated the results of our metric with these two metrics with some of the good and bad MT Engines. The metric showed significant correlation with both Human Adequacy and Human Fluency evaluation metrics. 59