Attacking Quality Limitations:

Transcription

1 Attacking Quality Limitations: New Approaches to Translation Technologies Hans Uszkoreit German Research Center for Artifical Intelligence (DFKI) and Saarland University Coordinator META-NET

2 My Background linguist and computer scientist first active in MT in as research associate for the MT project METAL at U. of Texas at Austin (today known as Lucy MT) responsible for deep linguistic processing in the German flagship MT project VerbMobil Coordinator of EuroMatrix and EuroMatrixPlus Coordinator of TaraXÜ Coordinator of META-NET since 2010 Coordinator of QTLaunchPad since 2012 co-founder and advisor of several start-up companies

3 Research Transfer through People Several of our researchers went to places where MT is applied Thorsten Brants started Google Translate and hired Franz Och Andreas Eisele is project manager at Christian Federmann became program manager for MT at Microsoft Research just a few weeks ago

4 Examples of MT Success Stories! Free Online Translation: Google Translate, Bing Translator, In-house Online Translation Services: VW, Newspaper Translation: Spanish to Catalan, Specialized Text Translation: VW User Manuals

5 And still! When it comes to high-quality outbound translation, " MT has not yet even scratched the surface. The human translation business LSP has an annual " growth rate of more than 10%. The main remaining barrier for MT is quality.

6 Just to be clear: Even today s technology has many more applications Especially by better combination with human translation: machine-aided human translation But much greater disruptive changes are still ahead of us

7 Seven Ingredients for Progress 1. analytical investigation of quality barriers 2. inclusion of professional translators 3. improvement of automatic quality estimation 4. specialized technologies for different purposes 5. creative combination of different approaches 6. exploitation of the best monolingual processing technologies 7. putting a strong focus on semantics

8 On the one hand analytical investigation of quality barriers unified metrics for measuring quality of HT and MT participation of translators in the investigation

9 1. analytical investigation of quality barriers concentration on nearly good translations unified metrics for measuring quality of HT and MT separating barriers for some techologies from barriers for all approaches

10 Where do we stand w.r.t. real quality? DEà EN Total: 1400 Acceptable: 231 DE- EN Can easily be fixed: 768 None of both: ENà DE Total: 2402 Acceptable: 126 EN- DE Can easily be fixed: 948 None of both: % 20% 40% 60% 80% 100% Acceptable Can easily be fixed None of both

11 A New Approach to MT Research 5-70% 10-75% 5-70% good bad ugly

12 A New Approach to MT Research good bad ugly

13 A New Approach to MT Research Recognize the truly good High Quality Es<ma<on help the translator to use or improve the bad: Computer Assisted Transla<on Make the nearly good truly good: High Quality MT help the end user to make the ugly at least understandable: Improve Gist Transla<on

14 RBMT vs. SMT Errors (in pilot corpus) RBMT (inner ring) and SMT (outer ring) Terminology 6% 7% MistranslaTon Omission AddiTon Untranslated 26% 33% 3% 24% 21% Style 14% Spelling Typography Grammar Unintelligible 1% 12% 6% 9% 3% 1% 0% 10% 9% 2% 13% Basic issues (error classes)

15 RBMT vs. SMT in accuracy and fluency (excluding terminology) Fluency 48% Fluency 60% Accuracy 40% Accuracy 52% Fluency 51% Accuracy 21% Accuracy 49% Fluency 79% RBMT (inner ring) and SM (outer ring) 15

16 2. inclusion of professional translators Involvement of translation professionals in all stages of research as providers of data, insights and evaluation judgements but also as the second central component in a high quality translation process responsible for creativity and knowledge.

17 3. improvement of quality estimation recent improvements of automatic quality estimation (Specia et al.) conservative triage also for estimating the quality of the source

18 4. specialized technologies for different purposes Too much research in Europe follows patterns set by US research instead of concentrating on our own, demands, strengths and opportunities Most of MT research has concentrated on in-bound content overview translation Nearly all existing translation markets are for out-bound translation

19 Currently Overwhelming Dominance of Inbound InformaTonal MT Predominant research funding: DARPA research Google Translate, Microsoft Bing Translate, Asia Online Security Applications, Intelligence Services Business Intelligence Applications MT for crosslingual IR Most MT applications in social media

20 Three Main Markets for AutomaTc TranslaTon A) Inbound informaton translaton of text: Google Translate, Bing Translator B) Outbound machine- assisted translaton of text: Language Weaver, Lucy Translator, C) TranslaTon in personal communicaton: Yocoy, Jibbigo 2

21 Differences need for full coverage need for accuracy need for completeness means for correction inbound information translation outbound translation dialogue translation/ interpretation unrestricted low high human translation unrestricted medium medium postediting, human translation restricted very high medium none (sublanguage) 2

22 ACT Always Correct TranslaTon A novel approach to conversatonal translaton. ACT only translates sentences, it can translate correctly. Based on sublanguages that are large enough for nearly all situatons. AutomaTc rule- learning approach, imported from informaton extracton. 2

23 5. creative combination of different approaches rule-based MT is far from being dead hybrid MT: especially combination SMT and RBMT from handcrafted rules to automatically learned language models new development: large volume of data produced by humans

24 6. exploitation of the best monolingual processing technologies progress in named entity recognition, morphology, parsing also in NL generation

25 7. putting a strong focus on semantics Warren Weaver (1949): Thus it may be true that the way to translate from Chinese to Arabic, or from Russian to Portuguese, is not to attempt the direct route, shouting from tower to tower. Perhaps the way is to descend, from each language, down to the common base of human communication the real but as yet undiscovered universal language and then re-emerge by whatever particular route is convenient. Kevin Knight (2012): As long as we get the who did what to who wrong, we are optimizing with respect to the wrong metric (BLEU)!

26 Semantic Resources?! They are growing fast:" linked open data, FreeBase, DBPedia, YAGO, ClueWeb WordNet, and the VerbNet, FrameNet, BabelNet, UWN, PropBank, OntoNotes

27 Conclusion We need to work in parallel: major push in establishing large web-based translation services major push in systematic research on overcoming quality barriers do this with research, IT industry, EC, EP, language industry the European Union can support it by research funding service procurement policy making

28 Thank you! EP STOA Workshop State of the Art of Machine Translation Brussels, Dec. 3rd 2013