NLP course project Automatic headline generation. ETH Spring Semester 2014

Transcription

1 NLP course project Automatic headline generation ETH Spring Semester 2014

2 Project description The content of the course will include the most fundamental parts of language processing: Tokenization, sentence boundaries Morphology Syntax Semantics etc. For the practical project, we are going to focus on a very concrete application scenario where you can apply most of the material taught in the course.

3 Project description Headline generation: Given a newswire article, generate a very short summary containing the main fact in the article.

4 Headline generation With the possibility of commercial space travel becoming a reality in the near future, we are facing some serious safety hurdles before the industry takes off. Chief among these issues is something called the Kessler Syndrome. Named after a NASA scientist, it s a theory that as the space around the planet gets more crowded, collisions are more likely to occur, adding even more debris orbiting the globe. If left unchecked, the amount of junk floating around our planet could theoretically block out the sun, let alone make for safe space travel. Swiss scientists don t want this to happen and have decided to do something about it.

5 Headline generation With the possibility of commercial space travel becoming a reality in the near future, we are facing some serious safety hurdles before the industry takes off. Chief among these issues is something called the Kessler Syndrome. Named after a NASA scientist, it s a theory that as the space around the planet gets more crowded, collisions are more likely to occur, adding even more debris orbiting the globe. If left unchecked, the amount of junk floating around our planet could theoretically block out the sun, let alone make for safe space travel. Swiss scientists don t want this to happen and have decided to do something about it.

6 Headline generation Switzerland working to clean up massive amounts of orbiting debris With the possibility of commercial space travel becoming a reality in the near future, we are facing some serious safety hurdles before the industry takes off. Chief among these issues is something called the Kessler Syndrome. Named after a NASA scientist, it s a theory that as the space around the planet gets more crowded, collisions are more likely to occur, adding even more debris orbiting the globe. If left unchecked, the amount of junk floating around our planet could theoretically block out the sun, let alone make for safe space travel. Swiss scientists don t want this to happen and have decided to do something about it.

7 Headline generation Switzerland Announces Plan to Clean Up Space Junk With the possibility of commercial space travel becoming a reality in the near future, we are facing some serious safety hurdles before the industry takes off. Chief among these issues is something called the Kessler Syndrome. Named after a NASA scientist, it s a theory that as the space around the planet gets more crowded, collisions are more likely to occur, adding even more debris orbiting the globe. If left unchecked, the amount of junk floating around our planet could theoretically block out the sun, let alone make for safe space travel. Swiss scientists don t want this to happen and have decided to do something about it.

8 Headline generation Swiss Plan "Janitor Satellite" to Clean Up Space Junk With the possibility of commercial space travel becoming a reality in the near future, we are facing some serious safety hurdles before the industry takes off. Chief among these issues is something called the Kessler Syndrome. Named after a NASA scientist, it s a theory that as the space around the planet gets more crowded, collisions are more likely to occur, adding even more debris orbiting the globe. If left unchecked, the amount of junk floating around our planet could theoretically block out the sun, let alone make for safe space travel. Swiss scientists don t want this to happen and have decided to do something about it.

9 Headline generation Swiss looks to tidy up space junk With the possibility of commercial space travel becoming a reality in the near future, we are facing some serious safety hurdles before the industry takes off. Chief among these issues is something called the Kessler Syndrome. Named after a NASA scientist, it s a theory that as the space around the planet gets more crowded, collisions are more likely to occur, adding even more debris orbiting the globe. If left unchecked, the amount of junk floating around our planet could theoretically block out the sun, let alone make for safe space travel. Swiss scientists don t want this to happen and have decided to do something about it.

10 Automatic Text Summarization A summary is: "a concise representation of a document's content to enable a reader to determine its relevance to a specific information" (Johnson, 1995). "a text produced from one or more texts, that contains a significant portion of the information in the original text(s), and is no longer than half of the original text(s) (Hovy, 2003). Automatic text summarization: Methods and techniques for obtaining summaries of textual documents in a fully automated way.

11 Headline Generation Single-document summarization producing very short summaries. Examples (February 19, 2012): Syria 'disintegrating under crippling sanctions' NASA celebrates 50th anniversary of John Glenn's flight Iranian warships sail into the Mediterranean Latvians say "no" in Russian language vote Dow nears psychological milestone: Greek cabinet backs extra austerity measures Whitney Houston's journey 'home' ends with private burial

12 Headline Generation Single-document summarization producing very short summaries. Advantages (I): The output is just one sentence: not so much need to worry about coherence and cohesion. But he said President Bashar al-assad's government would fight to the end. Human rights groups have put the figure at more than 7,000, while the government says at least 2,000 members of the security forces have been killed combating "armed gangs and terrorists".

13 Headline Generation Single-document summarization producing very short summaries. Advantages (II): The first sentence in news articles is usually a summary. Syria 'disintegrating under crippling sanctions' One of Syria's leading businessmen says its economy is being crippled by foreign sanctions and that the government is slowly disintegrating. NASA celebrates 50th anniversary of John Glenn's flight Veterans of NASA's Project Mercury reunited Saturday to celebrate the 50th anniversary of John Glenn's orbital flight, visiting the old launch pad and meeting the astronaut himself.

14 Headline Generation Single-document summarization producing very short summaries. Advantages (III): Selecting only one sentence from one document we do not have to worry about removing redundancy in the summary.

15 Headline Generation Single-document summarization producing very short summaries. Main challenges: How to find the most relevant information in the source document. Cannot always benefit from redundancy. How to generate a very short, yet grammatical summary. Many headlines are equally good. How to evaluate discriminating valid variations of headlines from irrelevant ones.

16 DUC and TAC competitions International competitions of automatic summarizers: Document Understanding Conferences (DUC) from 2000 to Text Analysis Conferences (TAC), from Each competition defines one or several tasks and provides: Development data for participants to tune their summarizers. Test data (usually available the last week). Manual and automatic evaluation.

17 5. Multi-document query-focused summarization, e.g. as answer of [who is X?]. DUC Tasks that year: 1. Very short single-document summaries (<= 75 bytes). (Headline Generation) 2. Multi-document summaries (<= 665 bytes). 3. Very short cross-lingual single-document summaries. 4. Cross-lingual multi-document summaries.

18 DUC-2004 Very Short Summaries A total of 500 documents. Generate a headline for each summary (<= 75 bytes). Summaries over the length limit are truncated. Evaluation using ROUGE n-gram matching.

19 DUC-2004 Very Short Summaries ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Inspired by BLEU for Machine Translation evaluation. Developed by Chin-Yew Lin. Compares the automatic summary with a set of manual summaries. Counts the number of overlapping n-grams. Many different variations of ROUGE metrics.

20 DUC-2004 Very Short Summaries BLEU = precision of n-grams ROUGE = recall of n-grams M: N-grams in manual summaries A: N-grams in automatic summary

21 DUC-2004 Very Short Summaries Example: ROUGE-1 M1: Switzerland working to clean up orbiting debris M2: Switzerland Announces Plan to Clean Up Space Junk A1: Swiss Plan "Janitor Satellite" to Clean Up Space Junk Tokens in manual summaries: 15 Tokens in manual summaries which appear in the automatic summary: 9 Recall = 9 / 15

22 DUC-2004 Very Short Summaries Example: ROUGE-1 M1: Switzerland working to clean up orbiting debris M2: Switzerland Announces Plan to Clean Up Space Junk A2: Swiss looks to tidy up space junk Tokens in manual summaries: 15 Tokens in manual summaries which appear in the automatic summary: 5 Recall = 5 / 15

23 DUC-2004 Very Short Summaries Example: ROUGE-1 M1: Switzerland working to clean up orbiting debris M2: Switzerland Announces Plan to Clean Up Space Junk A3: Space Junk Plan to Clean Up Switzerland Tokens in manual summaries: 15 Tokens in manual summaries which appear in the automatic summary: 11 Recall = 11 / 15!

24 DUC-2004 Very Short Summaries Other evaluation procedures: Extrinsic evaluation (e.g. long clicks on documents with different titles or snippets; summaries for open-ended answers in QA systems). Manual intrinsic evaluations (readability, informativeness, coherence, overall quality, etc.) Many automatic metrics proposed, but ROUGE-SU4 (a variation of ROUGE) ranks well in terms of rank correlation to overall quality scores.

25 Techniques Ideally, a summarizer would be implemented according to the following architecture: 1. Process the input documents. 2. Understand them and transform them into a semantic representation encoding what happened, what the entities involved where, the role of each entity, and additional modifiers (time, location, etc.) 3. Reason or Inference on the semantic representation to decide what will appear in the summary. 4. Generate the summary with Natural Language Generation But the technology is not ready yet......to achieve > 95% readable, informative summaries for every possible domain

26 Techniques Most existing summarizers use extractive technologies: Weight the sentences based on a combination of features: Sentence position. Sentence length. Lexical cohesion scores (words in common with other sentences, graph-based scoring algorithms, etc.). Relevant entities mentioned in the sentence. etc. Extract the top N sentences (N=1 for headlines) and remove redundant sentences. Postprocessing. Sentence rewriting and compression.

27 Techniques DUC proceedings: TAC proceedings: 2008: : : ACL Anthology:

28 Getting started 1. Download the ROUGE evaluation script from (you will need to apply and wait for the response ): com/default.aspx 2. Download the data files at: 3. Decompress the file: tar xvf - nlp-ethz.tgz gunzip -c 4. Go to the docs directory and look at the input documents and their format: cd nlp-ethz/docs ; ls 5. Go to the eval directory and browse the manual headlines (manual/) and the participants proposed headlines (peers/): cd nlp-ethz/eval/manual ; ls cd nlp-ethz/eval/peers ; ls

29 Getting started 6. Run ROUGE on the data: ROUGE/RELEASE-1.5.5/ROUGE pl \ -e ROUGE/RELEASE-1.5.5/data \ -a -c 95 -b 75 -m -n 4 -w 1.2 t1.rouge.in > output

30 You can share the tools you develop that are useful to other teams, e.g. to parse the input format of the files (this will be valued in the project's final score). Organization aspects You can work as individuals or 2-person groups. We now provide the development data on which you can improve your systems. At the end of the term we will provide to you the test data to verify that the systems were not over fitting the development set. We will keep track of progress on these sets. We will provide pointers to some additional off-the-shelf tools that you can download and use, such as parsers.