Relation Extraction, Neural Network, and Matrix Factorization

Size: px

Start display at page:

Download "Relation Extraction, Neural Network, and Matrix Factorization"

Christiana Evans
5 years ago
Views:

1 Relation Extraction, Neural Network, and Matrix Factorization Presenter: Haw-Shiuan Chang UMass CS585 guest lecture on 2016 Nov. 17 Most slides prepared by Patrick Verga

2 Relation Extraction

3 Knowledge Graph

4 Entities Common Types NIL Person Organization Location country state city Numbers/Date You can have different ontology more fine-grain types Common nouns (common sense) Classifiers is called Named Entities Recognizer (NER) Features POS tagging Lexicon dictionary (gazetteer) word embedding or topic models

5 Relations Brendan, birthplace, USA Brendan born in USA Brendan traveled to Pittsburgh Brendan likes Amherst Brendan thinks midterm is hard Relation Extraction Event Extraction Sentiment Extraction Belief Extraction

6 January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Gates will now focus on the charitable foundation he runs with his wife Melinda Gates. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Edges = { spouse, located in, lives in founded }

7 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Gates will now focus on the charitable foundation he runs with his wife Melinda Gates. Edges = { spouse, located in, lives in founded } Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.

8 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Gates will now focus on the charitable foundation he runs with his wife Melinda Gates. Edges = { spouse, located in, lives in founded } Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.

9 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. Gates will now focus on the charitable foundation he runs with his wife Melinda Gates. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Bill Gates Microsoft Bill Gates Bill Gates Microsoft Seattle Melinda Gates Seattle

10 Bill Gates Microsoft Edges = { spouse, located in, lives in founded } Melinda Gates Seattle B.M. Gates Foundation

11 Bill Gates Microsoft Edges = { spouse, located in, lives in founded } Melinda Gates Seattle B.M. Gates Foundation

12 Edges = { spouse, located in, lives in, Bill Gates Microsoft } founded is married to, became the CEO of, is headquartered in, helped to start, Melinda Gates B.M. Gates Foundation Seattle } resides in the city of,..

13 Research Questions How to extract relations? Relation Extraction How to perform reasoning on relations? Knowledge Base Completion What can be qualified as a relation? Schema/Ontology Definition

14 Challenges in Relation Extraction The problem could be very general Many different relations Many ways to expressing one relation in different context Hard to get annotated data Hard to get training data Hard to evaluate the results completely Unlabeled data: Missing positive or False?

15 Challenges in Relation Extraction Different ways of defining what is correct Correct in reality, correct in author s belief, probably correct? Error propagation at different stages Entity detection: F1 0.8~0.9 Entity Linking: F1 0.5~0.6 Relation Extraction: F1 0.2~0.3

16 Neural Networks

17 Or any Noun Adj other tags Naive Bayes HMM chairs end with s comfortable end with s Noun Adj Logistic Regression CRF chairs end with s comfortable end with s

18 Discrete Model Input/output: Noun sequence/sequence The feature is discrete Adj f(x): [0,, 1, 0, ].... x: [, chairs, ] The state is discrete V y=adj P(y=Adj)

19 Neural Networks { Noun Adj V LSTM GRU CNN word embedding chairs comfortable

20 Continuous State Transition We can extend the size of the state to model longer dependency v2 v3 Words with similar meaning v1 v2 v1 Traverse the state space similarly v3

21 Applications of Sequence Learning Tags POS tag Dependency tags Entity types Event types Words Other language Applications POS tagging Dependency parsing Entity detection Event detection Language modeling Translation

22 How Discriminative you want? Model flexibility: Interpretability: HMM<CRF<LSTM HMM>CRF>LSTM Number of parameters: Optimization easiness HMM<CRF<LSTM HMM>CRF>LSTM Performance (give sufficient data): Robustness against Overfitting HMM<CRF<LSTM HMM>CRF>LSTM

23 Constraints and Problems Prior knowledge of the problem Training Data External Resources Model Feature Engineering Prediction? Trying many models? Getting more data? Describing precisely?

24 Matrix Factorization

25 Interpretations of Word Embedding Unsupervised learning estimate topics words words infer documents documents ~= topics Word similarity documents -> short context, semantic -> syntax Document similarity Topic modeling (LSA)

26 Structure on Interactions words words key words users entities entity pairs near word (context) documents documents documents near entities relations

27 Applications words words key words users entities entity pairs word embedding topic modeling summarization recommendation coreference relation extraction near word (context) documents documents documents near entities relations

28 Matrix Factorization plus Neural Networks users entity pairs sentence Sentence Embedding LSTM GRU CNN word embedd words

29 An example

30 Universal Schema How to extract relations? Neural Network (LSTM) How to perform reasoning on relations? Matrix Factorization What can be qualified as a relation? Everything could be a schema

31 Knowledge Base (KB) Construction Text Text docs Text docs docs Wei Li studies at Xinghua U. Her 2008 publications include W. Li. Scalable NLP ACL, Queries Wei Li, Member,? Entity Extraction (NER) Entity Mentions Wei Li W. Li Xinghua U. (PER) (PER) (ORG) Resolution (Coref) Entities, Relations Wei Li W. Li Xinghua U. Relation Extraction Relation Mentions Member( Wei Li, Xinghua U.) KB Xinghua U. 31

32 Relation Extraction Text Text docs Text docs docs Wei Li studies at Xinghua U. Her 2008 publications include W. Li. Scalable NLP ACL, query Entity Extraction Entity Mentions Wei Li W. Li Xinghua U. (PER) (PER) (ORG) Resolution (Coref) Entities, Relations Wei Li W. Li Xinghua U. Relation Extraction Relation Mentions Member( Wei Li, Xinghua U.) KB answer 32

33 Compositional Universal Schema Verga, P., Belanger, D., Strubell, E., Roth, B., and McCallum, A. (2016). Multilingual relation extraction using compositional universal schema. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).

34 Universal Schema January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle 34 Relation Types structured spouse born-in friend per:co-worker per:lives-in org:top-members org:member textual 50k columns CEO chairman president leader-of head-of head-of-state HQ-in

35 January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, ~x Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle of Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle vector ~y embedding distributed representation semantics learned parameters Relation Types structured per:co-worker org:top-members org:member chairman textual just outside Seattle. P ((s, r, o)) = u > s,ov r 35 spouse born-in friend per:lives-in CEO president leader-of head-of 50k columns head-of-state HQ-in

36 January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle 36 Bayesian Personalized Ranking (BPR) ~x u T s,o vr > u T s,o vr vector embedding ` distributed representation of ~y semantics learned parameters spouse born-in friend per:co-worker per:lives-in org:top-members org:member P ((s, r, o)) = CEO chairman president leader-of head-of head-of-state HQ-in u > s,ov r

37 January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle 37 ~x vector embedding distributed representation of ~y semantics learned parameters spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in

38 Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle 38 ~x vector embedding distributed representation of ~y semantics learned parameters spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in

39 January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle 39 ~x vector embedding distributed representation of ~y semantics learned parameters spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle.

40 January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant. His long-time friend, Steve Balmer, will take over as CEO of Microsoft. Gates will now focus on the charitable foundation he runs with his wife Melinda French Gates. Bill and Melinda were married in a ceremony in Hawaii, rather than her hometown of Dallas. Gates moved his family into their 55,000-square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle 40 ~x vector embedding distributed representation of ~y semantics learned parameters spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Pattern LSTM Encoder

41 query: spouse Joe NewEntity worked with his wife Jane NewEntity on relation extraction. spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Pattern LSTM Encoder Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle ~x vector embedding distributed representation of ~y semantics learned parameters 41

42 query: spouse Joe NewEntity worked with his wife Jane NewEntity on relation extraction. spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Pattern LSTM Encoder Joe NewEntity worked with his wife Jane NewEntity on relation extraction. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle ~x vector embedding distributed representation of ~y semantics learned parameters 42

43 query: spouse Joe NewEntity worked with his wife Jane NewEntity on relation extraction. spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Pattern LSTM Encoder Joe NewEntity worked with his wife Jane NewEntity on relation extraction. Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle Joe Jane 43 ~x vector embedding distributed representation of ~y semantics learned parameters

spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54

44 spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of head-of-state HQ-in Gates moved his family into their 55,000- square-foot, $54 million house on the shore of Lake Washington, just outside Seattle. Pattern LSTM Encoder Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Hawaii Bill, Seattle Joe Jane ~x vector embedding distributed representation of ~y semantics learned parameters Joe NewEntity worked with his wife Jane NewEntity on relation extraction. 44

45 Conclusions Relation extraction is one step further toward bridging the gap between text and knowledge Neural networks are powerful, but data is usually the bottleneck in relation extraction Matrix factorization can alleviate the requirements of large amount of annotations Universal Schema is an example of combining complex methods to solve complex problem

46 Multilingual Relation Extraction

47 Multilingual Relation Extraction English Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in Bill, Microsoft Melinda, Dallas Obama, Michelle Bill, Seattle Spanish esposa presidente vive en residente fundador 47

48 Multilingual Universal Schema Relation Types structured English Spanish Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle spouse born-in friend per:co-worker per:lives-in org:top-members org:member textual textual CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador 48

49 ~x Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle of Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle vector ~y embedding distributed representation semantics learned parameters spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador LSTM 49

50 spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador LSTM Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x vector embedding distributed representation of ~y semantics learned parameters 50

51 spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador LSTM Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x vector embedding distributed representation of ~y semantics learned parameters 51

52 Entity pair Relation spouse spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador Melinda, Bill Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x ~y vector embeddin distribute representati of semanti learned paramete LSTM 52 Michelle, Obama married to esposa

53 Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x ~y vector embeddin distribute representati of semanti learned paramete spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador 53 resident presidente married esposa spouse gato cat casado LSTM

54 married esposa presidente cat casado spouse born-in friend per:co-worker per:lives-in org:top-members org:member resident CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador gato spouse Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x ~y vector embeddin distribute representati of semanti learned paramete LSTM 54

55 married cat gato spouse esposa casado Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x ~y vector embeddin distribute representati of semanti learned paramete spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador LSTM 55 resident presidente

56 married cat gato spouse esposa spouse / esposa casado Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x ~y vector embeddin distribute representati of semanti learned paramete spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador LSTM resident presidente

57 cat gato married casado spouse / esposa presidente Steve, Microsoft Bill, Steve Bill, Microsoft Melinda, Bill Microsoft, Seattle Melinda, Dallas Obama, US Obama, Michelle Bill, Seattle ~x ~y vector embeddin distribute representati of semanti learned paramete spouse born-in friend per:co-worker per:lives-in org:top-members org:member CEO chairman president leader-of head-of married-to HQ-in esposa presidente vive en residente fundador LSTM resident

Row-less Universal Schema. Patrick Verga and Andrew McCallum

Row-less Universal Schema. Patrick Verga and Andrew McCallum Row-less Universal Schema Patrick Verga and Andrew McCallum January 15, 2000 Tech pioneer Bill Gates stepped down today as chief executive officer of Microsoft, the Seattleheadquartered software giant.