NEURAL TEXT GENERATION IN STORIES USING ENTITY REPRESENTATIONS

Transcription

1 1 / 17 NEURAL TEXT GENERATION IN STORIES USING ENTITY REPRESENTATIONS AS CONTEXT Elizabeth Clark Yangfeng Ji Noah A. Smith Paul G. Allen School of Computer Science & Engineering University of Washington NAACL 2018

2 2 / 17 MOTIVATION Context: All of a sudden, Emily walked towards the dragon. Current Sentence: Seth yelled at her to get back but

3 2 / 17 MOTIVATION Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but

4 2 / 17 MOTIVATION Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but [she] 1 ignored [him] 3.

5 3 / 17 OVERVIEW Can we use entity representations as a form of context to improve text generation for stories? Three evaluations: 1. Mention generation 2. Sentence selection 3. Human evaluation

6 4 / 17 SEQ2SEQ WITH ATTENTION Current Sentence: Seth yelled at her to get back but h c,1 h c,2 h c,3 h c,4 h c,5 h c,6 h c,7 h c,8

7 4 / 17 SEQ2SEQ WITH ATTENTION Current Sentence: Seth yelled at her to get back but h c,1 h c,2 h c,3 h c,4 h c,5 h c,6 h c,7 h c,8 Previous Sentence: Emily walked towards the dragon.... h p,t 5 h p,t 4 h p,t 3 h p,t 2 h p,t 1 h p,t

8 4 / 17 SEQ2SEQ WITH ATTENTION Current Sentence: Seth yelled at her to get back but h c,1 h c,2 h c,3 h c,4 h c,5 h c,6 h c,7 h c,8 Previous Sentence: Emily walked towards the dragon.... h p,t 5 h p,t 4 h p,t 3 h p,t 2 h p,t 1 h p,t α T 5 α T 4 α T 3 prev sent α T 2 αt 1 α T

9 5 / 17 EXAMPLE STORY GENERATION Legend: Seq2Seq Human Clark et al. (2018)

10 5 / 17 EXAMPLE STORY GENERATION This is ridiculous, said Duke. Yesterday I felt fine, and now you re telling me I m at death s door?! Legend: Seq2Seq Human Clark et al. (2018)

11 5 / 17 EXAMPLE STORY GENERATION This is ridiculous, said Duke. Yesterday I felt fine, and now you re telling me I m at death s door?! We ll take care of Furble tomorrow, the doctor said. Legend: Seq2Seq Human Clark et al. (2018)

12 5 / 17 EXAMPLE STORY GENERATION This is ridiculous, said Duke. Yesterday I felt fine, and now you re telling me I m at death s door?! We ll take care of Furble tomorrow, the doctor said. You ve named my tumor?! Duke shrieked. Legend: Seq2Seq Human Clark et al. (2018)

13 5 / 17 EXAMPLE STORY GENERATION This is ridiculous, said Duke. Yesterday I felt fine, and now you re telling me I m at death s door?! We ll take care of Furble tomorrow, the doctor said. You ve named my tumor?! Duke shrieked. Yeah, replied the doctor coolly, we ve found that anthropomorphizing tumors helps people in your position come to terms with their condition more easily. Legend: Seq2Seq Human Clark et al. (2018)

14 5 / 17 EXAMPLE STORY GENERATION This is ridiculous, said Duke. Yesterday I felt fine, and now you re telling me I m at death s door?! We ll take care of Furble tomorrow, the doctor said. You ve named my tumor?! Duke shrieked. Yeah, replied the doctor coolly, we ve found that anthropomorphizing tumors helps people in your position come to terms with their condition more easily. Lance yells over the speakers no sudden hammering Legend: Seq2Seq Human Clark et al. (2018)

15 COHERENT MENTION GENERATION Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but. Option A: [she] 1 ignored [him] 3. Option B: [Emily] 1 ignored [Seth] 3. Grosz et al. (1995) Hobbs (1979) 6 / 17

16 7 / 17 THREE FORMS OF CONTEXT The current sentence h t 1 The previous sentence prev sent The entities entity

17 8 / 17 ENTITY REPRESENTATIONS Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but Ji et al. (2017)

18 8 / 17 ENTITY REPRESENTATIONS Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but Ji et al. (2017)

19 8 / 17 ENTITY REPRESENTATIONS Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but Ji et al. (2017)

20 8 / 17 ENTITY REPRESENTATIONS Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but Ji et al. (2017)

21 8 / 17 ENTITY REPRESENTATIONS Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but Ji et al. (2017)

22 9 / 17 ENTITY REPRESENTATIONS FOR GENERATION Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but entity

23 9 / 17 ENTITY REPRESENTATIONS FOR GENERATION Context: All of a sudden, [Emily] 1 walked towards [the dragon] 2. Current Sentence: [Seth] 3 yelled at [her] 1 to get back but entity

24 10 / 17 COMBINING THE CONTEXT REPRESENTATIONS h t 1 Full Model prev sent max_pool context t S2SA entity Entity Model

25 10 / 17 COMBINING THE CONTEXT REPRESENTATIONS h t 1 Full Model prev sent max_pool context t S2SA entity Entity Model

26 10 / 17 COMBINING THE CONTEXT REPRESENTATIONS h t 1 Full Model prev sent max_pool context t S2SA entity Entity Model

27 11 / 17 CORPUS Toronto Book Corpus: Adventure books 390 books split into 42,000 segments 43 million tokens, 35,000 types Annotations from Stanford CoreNLP Zhu et al. (2015) Clark and Manning (2016a,b)

28 12 / 17 EVALUATION #1: MENTION GENERATION Passage: All of a sudden, towards. yelled at back but... walked to get Candidates: [Emily] 1 (gold)

29 12 / 17 EVALUATION #1: MENTION GENERATION Passage: All of a sudden, [Emily] 1 walked towards. yelled at to get back but... Candidates: [Emily] 1 [the dragon] 2 (gold)

30 12 / 17 EVALUATION #1: MENTION GENERATION Passage: All of a sudden, [Emily] 1 walked towards [the dragon] 2. yelled at to get back but... Candidates: [Emily] 1 [the dragon] 2 [Seth] 3 (gold)

31 12 / 17 EVALUATION #1: MENTION GENERATION Passage: All of a sudden, [Emily] 1 walked towards [the dragon] 2. [Seth] 3 yelled at to get back but... Candidates: [Emily] 1 [the dragon] 2 [Seth] 3 [her] 1 (gold)

32 12 / 17 EVALUATION #1: MENTION GENERATION Passage: All of a sudden, [Emily] 1 walked towards [the dragon] 2. [Seth] 3 yelled at [her] 1 to get back but... Candidates: [Emily] 1 [the dragon] 2 [Seth] 3 [her] 1 [she] 1 (gold)

33 12 / 17 EVALUATION #1: MENTION GENERATION Passage: All of a sudden, [Emily] 1 walked towards [the dragon] 2. [Seth] 3 yelled at [her] 1 to get back but... Candidates: [Emily] 1 [the dragon] 2 [Seth] 3 [her] 1 [she] 1 (gold) Mean Average Precision Mention Generation S2SA Entity Model Full Model

34 13 / 17 EVALUATION #2: SENTENCE SELECTION Context:... All of a sudden, [Emily] 1 walked towards [the dragon] 2. Gold sentence: [Seth] 3 yelled at [her] 1 to get back but [she] 1 ignored [him] 3. Distractor sentence: [She] 1 patted [its head] 4 and [it] 2 curled up outside [the cave] 5.

35 13 / 17 EVALUATION #2: SENTENCE SELECTION Context:... All of a sudden, [Emily] 1 walked towards [the dragon] 2. Gold sentence: [Seth] 3 yelled at [her] 1 to get back but [she] 1 ignored [him] 3. Distractor sentence: [She] 1 patted [its head] 4 and [it] 2 curled up outside [the cave] 5. Full Model Sentence Selection Accuracy 0.57 Entity Model 0.53 S2SA Accuracy

36 14 / 17 EVALUATION #3: HUMAN EVALUATION Context Full Model S2SA Model Explanation

37 14 / 17 EVALUATION #3: HUMAN EVALUATION Context Full Model 27 passages S2SA Model 23 passages Explanation

38 14 / 17 EVALUATION #3: HUMAN EVALUATION Context Scared of us hoodlums, they are. Like we ever would touch em, and he spat on the pavement. The big city gang go where we want, when we want, no stopping us, he stated proudly. These people here are all just sheep. Baa, baa, one after the other. Full Model S2SA Model Explanation

39 14 / 17 EVALUATION #3: HUMAN EVALUATION Context Scared of us hoodlums, they are. Like we ever would touch em, and he spat on the pavement. The big city gang go where we want, when we want, no stopping us, he stated proudly. These people here are all just sheep. Baa, baa, one after the other. Full Model There is nothing I can do. S2SA Model She didn t know what happened Explanation

40 14 / 17 EVALUATION #3: HUMAN EVALUATION Context Scared of us hoodlums, they are. Like we ever would touch em, and he spat on the pavement. The big city gang go where we want, when we want, no stopping us, he stated proudly. These people here are all just sheep. Baa, baa, one after the other. Full Model There is nothing I can do. S2SA Model She didn t know what happened Explanation She isn t mentioned in the paragraph.

41 14 / 17 EVALUATION #3: HUMAN EVALUATION Context He was wearing brown slacks and a tan button-down shirt, with wool slippers. He looked about sixty, a little paunchy, with balding brown hair and a bushy mustache. Ice blue eyes observed Alejo keenly, then drifted over to Wara. Welcome to my home. The man s voice was deep and calm. Full Model S2SA Model Explanation

42 14 / 17 EVALUATION #3: HUMAN EVALUATION Context He was wearing brown slacks and a tan button-down shirt, with wool slippers. He looked about sixty, a little paunchy, with balding brown hair and a bushy mustache. Ice blue eyes observed Alejo keenly, then drifted over to Wara. Welcome to my home. The man s voice was deep and calm. Full Model I m proud of you, he said. S2SA Model What s going on? Explanation

43 14 / 17 EVALUATION #3: HUMAN EVALUATION Context He was wearing brown slacks and a tan button-down shirt, with wool slippers. He looked about sixty, a little paunchy, with balding brown hair and a bushy mustache. Ice blue eyes observed Alejo keenly, then drifted over to Wara. Welcome to my home. The man s voice was deep and calm. Full Model I m proud of you, he said. S2SA Model What s going on? Explanation The introduction makes the man sound like he is a stranger, so I m proud of you seems out of place.

44 15 / 17 FUTURE DIRECTIONS Deeper entity knowledge: social commonsense, modeling inter-entity relationships Structure in story generation: discourse structure, semantics, story structure New domains: news articles, recipes Rashkin et al. (2018) Bosselut et al. (2018)

45 16 / 17 THANK YOU! Questions? eaclark7@cs.washington.edu

46 17 / 17 REFERENCES Bosselut, A., Levy, O., Holtzman, A., Ennis, C., Fox, D., and Choi, Y. (2018). Simulating action dynamics with neural process networks. In ICLR. Clark, E., Ross, A. S., Tan, C., Ji, Y., and Smith, N. A. (2018). Creative writing with a machine in the loop: Case studies on slogans and stories. In IUI. Clark, K. and Manning, C. D. (2016a). Deep reinforcement learning for mention-ranking coreference models. In EMNLP. Clark, K. and Manning, C. D. (2016b). Improving coreference resolution by learning entity-level distributed representations. In ACL. Grosz, B. J., Weinstein, S., and Joshi, A. K. (1995). Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2): Hobbs, J. R. (1979). Coherence and coreference. Cognitive Science, 3: Ji, Y., Tan, C., Martschat, S., Choi, Y., and Smith, N. A. (2017). Dynamic entity representations in neural language models. In EMNLP. Rashkin, H., Bosselut, A., Sap, M., Knight, K., and Choi, Y. (2018). Modeling naive psychology of characters in simple commonsense stories. In ACL. Zhu, Y., Kiros, R., Zemel, R. S., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV.