Information Extraction. CS6200 Information Retrieval (and a sort of advertisement for NLP in the spring)

Transcription

1 Information Extraction CS6200 Information Retrieval (and a sort of advertisement for NLP in the spring) 1

2 Informa(on Extrac(on Automa(cally extract structure from text annotate document using tags to iden(fy extracted structure We ve briefly men(oned one example But part of speech tagging is so low- level it usually doesn t count as IE Named en(ty recogni(on iden(fy words that refer to something of interest in a par(cular applica(on e.g., people, companies, loca(ons, dates, product names, prices, etc. 2

3 Named En(ty Recogni(on Example showing seman(c annota(on of text using XML tags Informa(on extrac(on also includes document structure and more complex features such as rela(onships and events 3

4 Named Entity Recognition The Persian learned men say that the Phoenicians came to our seas from the so-called Red Sea, and having settled in the country which they still occupy, at once began to make long voyages. Among other places to which they carried Egyptian and Assyrian merchandise, they came to Argos, which was at that time preeminent in every way among the people of what is now called Hellas. The Phoenicians came to Argos, and set out their cargo. On the fifth or sixth day after their arrival, when their wares were almost all sold, many women came to the shore and among them especially the daughter of the king, whose name was Io (according to Persians and Greeks alike), the daughter of Inachus. As these stood about the stern of the ship bargaining for the wares they liked, the Phoenicians incited one another to set upon them. Most of the women escaped: Io and others were seized and thrown into the ship, which then sailed away for Egypt. 4

5 Named Entity Recognition The Persian learned men say that the Phoenicians came to our seas from the so-called Red Sea, and having settled in the country which they still occupy, at once began to make long voyages. Among other places to which they carried Egyptian and Assyrian merchandise, they came to Argos, which was at that time preeminent in every way among the people of what is now called Hellas. The Phoenicians came to Argos, and set out their cargo. On the fifth or sixth day after their arrival, when their wares were almost all sold, many women came to the shore and among them especially the daughter of the king, whose name was Io (according to Persians and Greeks alike), the daughter of Inachus. As these stood about the stern of the ship bargaining for the wares they liked, the Phoenicians incited one another to set upon them. Most of the women escaped: Io and others were seized and thrown into the ship, which then sailed away for Egypt. Person 5

6 Named Entity Recognition The Persian learned men say that the Phoenicians came to our seas from the so-called Red Sea, and having settled in the country which they still occupy, at once began to make long voyages. Among other places to which they carried Egyptian and Assyrian merchandise, they came to Argos, which was at that time preeminent in every way among the people of what is now called Hellas. The Phoenicians came to Argos, and set out their cargo. On the fifth or sixth day after their arrival, when their wares were almost all sold, many women came to the shore and among them especially the daughter of the king, whose name was Io (according to Persians and Greeks alike), the daughter of Inachus. As these stood about the stern of the ship bargaining for the wares they liked, the Phoenicians incited one another to set upon them. Most of the women escaped: Io and others were seized and thrown into the ship, which then sailed away for Egypt. Person Location 6

7 Named Entity Recognition The Persian learned men say that the Phoenicians came to our seas from the so-called Red Sea, and having settled in the country which they still occupy, at once began to make long voyages. Among other places to which they carried Egyptian and Assyrian merchandise, they came to Argos, which was at that time preeminent in every way among the people of what is now called Hellas. The Phoenicians came to Argos, and set out their cargo. On the fifth or sixth day after their arrival, when their wares were almost all sold, many women came to the shore and among them especially the daughter of the king, whose name was Io (according to Persians and Greeks alike), the daughter of Inachus. As these stood about the stern of the ship bargaining for the wares they liked, the Phoenicians incited one another to set upon them. Most of the women escaped: Io and others were seized and thrown into the ship, which then sailed away for Egypt. Person Location Ethnic 7

8 Named Entity Recognition The Persian learned men say that the Phoenicians came to our seas from the so-called Red Sea, and having settled in the country which they still occupy, at once began to make long voyages. Among other places to which they carried Egyptian and Assyrian merchandise, they came to Argos, which was at that time preeminent in every way among the people of what is now called Hellas. The Phoenicians came to Argos, and set out their cargo. On the fifth or sixth day after their arrival, when their wares were almost all sold, many women came to the shore and among them especially the daughter of the king, whose name was Io (according to Persians and Greeks alike), the daughter of Inachus. As these stood about the stern of the ship bargaining for the wares they liked, the Phoenicians incited one another to set upon them. Most of the women escaped: Io and others were seized and thrown into the ship, which then sailed away for Egypt. Classes could also be, e.g., Wikipedia articles Person Location Ethnic 7

9 8

10 8

11 8

12 Named Entity Recognition Rule-based Uses lexicons (lists of words and phrases) that categorize names e.g., locations, peoples names, organizations, etc. Rules also used to verify or find new entity names e.g., <number> <word> street for addresses <street address>, <city> or in <city> to verify city names <street address>, <city>, <state> to find new cities <title> <name> to find new names 9

13 Named Entity Recognition Rules either developed manually by trial and error or using machine learning techniques Statistical uses a probabilistic model of the words in and around an entity probabilities estimated using training data (manually annotated text) Hidden Markov Model (HMM) is one approach Conditional Random Fields: similar structure, often higher accuracy, more expensive to train 10

14 HMM for Extraction Resolve ambiguity in a word using context e.g., marathon is a location or a sporting event, boston marathon is a specific sporting event Model context using a generative model of the sequence of words Markov property: the next word in a sequence depends only on a small number of the previous words 11

15 HMM for Extraction Markov Model describes a process as a collection of states with transitions between them each transition has a probability associated with it next state depends only on current state and transition probabilities Hidden Markov Model each state has a set of possible outputs outputs have probabilities 12

16 HMM Sentence Model Each state is associated with a probability distribu(on over words (the output) 13

17 NER as Sequence Tagging The Phoenicians came from the Red Sea 14 14

18 NER as Sequence Tagging O B-E O O O B-L I-L The Phoenicians came from the Red Sea 14 14

19 Sequence Tagging Fed raises interest rates 15 15

20 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 15 15

21 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 15 15

22 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 15 15

23 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 16 16

24 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 16 16

25 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 17 17

26 Sequence Tagging NN NNS NNP VB VBZ Fed raises interest rates 17 17

27 Sequence Tagging NN NNS NNP VB VBZ T n = 5 4 = 625 possible paths! Fed raises interest rates 17 17

28 Sequence Tagging Efficient (linear time) Shortest path = Viterbi algorithm NN NNS NNP VB VBZ T n = 5 4 = 625 possible paths! Fed raises interest rates 17 17

29 Sequence Tagging Efficient (linear time) Shortest path = Viterbi algorithm NN NNS NNP VB VBZ T n = 5 4 = 625 possible paths! Fed raises interest rates Can we specify that Fed always has the same tag in this document? 17 17

30 NER as Sequence Tagging The Phoenicians came from the Red Sea 18 18

31 NER as Sequence Tagging O B-E O O O B-L I-L The Phoenicians came from the Red Sea 18 18

32 NER as Sequence Tagging Capitalized word O B-E O O O B-L I-L The Phoenicians came from the Red Sea 18 18

33 NER as Sequence Tagging Capitalized word Ends in s O B-E O O O B-L I-L The Phoenicians came from the Red Sea 18 18

34 NER as Sequence Tagging Capitalized word Ends in s Ends in ans O B-E O O O B-L I-L The Phoenicians came from the Red Sea 18 18

35 NER as Sequence Tagging Capitalized word Ends in s Ends in ans Previous word the O B-E O O O B-L I-L The Phoenicians came from the Red Sea 18 18

36 NER as Sequence Tagging Capitalized word Ends in s Ends in ans Previous word the Phoenicians in gazetteer O B-E O O O B-L I-L The Phoenicians came from the Red Sea 18 18

37 NER as Sequence Tagging O B-E O O O B-L I-L The Phoenicians came from the Red Sea 19 19

38 NER as Sequence Tagging Not capitalized O B-E O O O B-L I-L The Phoenicians came from the Red Sea 19 19

39 NER as Sequence Tagging Not capitalized Tagged as VB O B-E O O O B-L I-L The Phoenicians came from the Red Sea 19 19

40 NER as Sequence Tagging B-E to right Not capitalized Tagged as VB O B-E O O O B-L I-L The Phoenicians came from the Red Sea 19 19

41 NER as Sequence Tagging O B-E O O O B-L I-L The Phoenicians came from the Red Sea 20 20

42 NER as Sequence Tagging Word sea O B-E O O O B-L I-L The Phoenicians came from the Red Sea 20 20

43 NER as Sequence Tagging Word sea preceded by the ADJ Word sea O B-E O O O B-L I-L The Phoenicians came from the Red Sea 20 20

44 NER as Sequence Tagging Word sea preceded by the ADJ Hard constraint: I-L must follow B-L or I-L Word sea O B-E O O O B-L I-L The Phoenicians came from the Red Sea 20 20

45 Great Ideas in ML: Message Passing adapted from MacKay (2003) textbook 21 21

46 Great Ideas in ML: Message Passing Count the soldiers adapted from MacKay (2003) textbook 21 21

47 Great Ideas in ML: Message Passing Count the soldiers 3 behind you 2 behind you 1 behind you adapted from MacKay (2003) textbook 21 21

48 Great Ideas in ML: Message Passing Count the soldiers there s 1 of me 3 behind you 2 behind you 1 behind you adapted from MacKay (2003) textbook 21 21

49 Great Ideas in ML: Message Passing Count the soldiers there s 1 of me 5 behind you 4 behind you 3 behind you 2 behind you 1 behind you adapted from MacKay (2003) textbook 21 21

50 Great Ideas in ML: Message Passing Count the soldiers 1 before you 2 before you 5 behind you 4 behind you 3 behind you 2 behind you 1 behind you adapted from MacKay (2003) textbook 21 21

51 Great Ideas in ML: Message Passing Count the soldiers 1 before you 2 before you 3 before you 4 before you 5 before you 5 behind you 4 behind you 3 behind you 2 behind you 1 behind you adapted from MacKay (2003) textbook 21 21

52 Great Ideas in ML: Message Passing Count the soldiers 2 before you there s 1 of me only see my incoming messages 3 behind you adapted from MacKay (2003) textbook 22 22

53 Great Ideas in ML: Message Passing Count the soldiers 2 before you there s 1 of me Belief: Must be = 6 of us only see my incoming messages 3 behind you adapted from MacKay (2003) textbook 22 22

54 Great Ideas in ML: Message Passing Count the soldiers 2 before you there s 1 of me Belief: Must be = 6 of us only see my incoming messages 3 behind you adapted from MacKay (2003) textbook 22 22

55 Great Ideas in ML: Message Passing Count the soldiers 1 before you there s 1 of me Belief: Must be = 6 of us only see my incoming messages 4 behind you adapted from MacKay (2003) textbook 23 23

56 Great Ideas in ML: Message Passing Count the soldiers 1 before you there s 1 of me Belief: Must be = 6 of us only see my incoming messages 4 behind you adapted from MacKay (2003) textbook 23 23

57 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree adapted from MacKay (2003) textbook 24 24

58 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here adapted from MacKay (2003) textbook 24 24

59 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here 1 of me 11 here (= 7+3+1) adapted from MacKay (2003) textbook 24 24

60 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree adapted from MacKay (2003) textbook 25 25

61 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 3 here adapted from MacKay (2003) textbook 25 25

62 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here (= 3+3+1) 3 here adapted from MacKay (2003) textbook 25 25

63 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree adapted from MacKay (2003) textbook 26 26

64 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 7 here 3 here adapted from MacKay (2003) textbook 26 26

65 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 11 here (= 7+3+1) 7 here 3 here adapted from MacKay (2003) textbook 26 26

66 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree adapted from MacKay (2003) textbook 27 27

67 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree Belief: Must be 14 of us adapted from MacKay (2003) textbook 27 27

68 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here 3 here Belief: Must be 14 of us adapted from MacKay (2003) textbook 27 27

69 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here 3 here Belief: Must be 14 of us adapted from MacKay (2003) textbook 28 28

70 Great Ideas in ML: Message Passing Each soldier receives reports from all branches of tree 3 here 7 here 3 here Belief: Must be 14 of us wouldn t work correctly with a loopy (cyclic) graph adapted from MacKay (2003) textbook 28 28

71 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward v n a v n a v n a v n a find preferred tags 29 29

72 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward belief v 1.8 n 0 a 4.2 v n a v n a v n a v n a find preferred tags 29 29

73 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward message α belief v 1.8 n 0 a 4.2 β message v n a v 3 v n 1 n a 6 a v 2 v n a nv an a find preferred tags 29 29

74 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward message α belief v 1.8 n 0 a 4.2 β message v n a v 3 v n 1 n a 6 a v 2 v n a nv an a v 0.3 n 0 a 0.1 find preferred tags 29 29

75 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward α message α belief v 1.8 n 0 a 4.2 β message v 7 n 2 a 1 v n a v 3 v n 1 n a 6 a v 2 v n a nv an a v 0.3 n 0 a 0.1 find preferred tags 29 29

76 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward α message α belief v 1.8 n 0 a 4.2 β message v 7 n 2 a 1 v 3 n 1 a 6 v 2 v n a nv an a v 0.3 n 0 a 0.1 find preferred tags 29 29

77 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward α message α belief v 1.8 n 0 a 4.2 β message v 7 n 2 a 1 v 3 n 1 a 6 v 2 n 1 a 7 v 0.3 n 0 a 0.1 find preferred tags 29 29

78 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward α message α belief v 1.8 n 0 a 4.2 β message v 7 n 2 a 1 v n a v 3 v n 1 n a 6 a v 2 n 1 a 7 v 0.3 n 0 a 0.1 find preferred tags 29 29

79 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward α message α belief v 1.8 n 0 a 4.2 β message β v 7 n 2 a 1 v n a v 3 v n 1 n a 6 a v 2 n 1 a 7 v 3 n 6 a 1 v 0.3 n 0 a 0.1 find preferred tags 29 29

80 Great ideas in ML: Forward-Backward n In the CRF, message passing = forward-backward α message α belief v 1.8 n 0 a 4.2 β message β v 7 n 2 a 1 v n a v 3 v n 1 n a 6 a v 2 v n a nv an a v 3 n 6 a 1 v 0.3 n 0 a 0.1 find preferred tags 29 29

81 Named En(ty Recogni(on Accurate recogni(on requires about 1M words of training data (1,500 news stories) may be more expensive than developing rules for some applica(ons Both rule- based and sta(s(cal can achieve about 90% effec(veness for categories such as names, loca(ons, organiza(ons others, such as product name, can be much worse 30

82 Interna(onaliza(on 2/3 of the Web is in English About 50% of Web users do not use English as their primary language Many (maybe most) search applica(ons have to deal with mul(ple languages monolingual search: search in one language, but with many possible languages cross- language search: search in mul(ple languages at the same (me 31

83 Interna(onaliza(on Many aspects of search engines are language- neutral Major differences: Text encoding (conver(ng to Unicode) Tokenizing (many languages have no word separators) Stemming Cultural differences may also impact interface design and features provided 32

84 Chinese Tokenizing 33