Lecture 4: n-grams in NLP. LING 1330/2330: Introduction to Computational Linguistics Na-Rae Han

Transcription

1 Lecture 4: n-grams in NLP LING 1330/2330: Introduction to Computational Linguistics Na-Rae Han

2 Objectives Frequent n-grams in English n-grams and statistical NLP n-grams and conditional probability Large n-gram resources 2/2/2017 2

3 For fun: most frequent bigrams? of the in the to the on the and the to be for the at the in a do n't with the from the of a that the is a going to by the as a with a I think Source: 2/2/2017 3

4 Most frequent trigrams? I do n't one of the a lot of the United States do n't know out of the as well as going to be I did n't to be a Source: 2/2/2017 4

5 n-grams and statistical NLP You have a good intuition as a native speaker. Beyond intuition, it is possible to obtain a highly detailed & accurate set of n-gram statistics. How? Through corpus data. Corpus-sourced, large-scale n-grams are one of the biggest contributors to the recent advancement of statistical natural language processing (NLP) technologies. Used for: spelling correction, machine translation, speech recognition, information extraction... JUST ABOUT ANY NLP APPLICATION 2/2/2017 5

6 n-grams vs. conditional probability Suppose 'is' is the current word. What is the most likely next word? How likely are 'you' and 'your' as the next word? Questions of conditional probability Can be answered through n-gram data 'is' occurs 4,705,743,816 times (1)* 'a' is the most likely next word with (2) / (1) = 0.10 probability. 'your' as the next word has (3) / (1) = probability. 'you' as the next word has (4) / (1) = probability. *Source: **Source: is a ** (2) is the is not is an is to is your (3) is you (4)

7 Extremely large "All our N-gram are Belong to You" Google Web 1T 5-Gram, released in August 2006 through LDC (Linguistic Data Consortium) 1-5 grams Compiled from 1-trillion words of running web text 24 GB of compressed text Source of Norvig's 1- and 2-gram frequency lists Publication of this data triggered huge advances in NLP technologies and applications. 2/2/2017 7

8 Even larger Google Books Ngram Corpus 2.html Basis for Google Books Ngram Viewer 1-5grams Freely downloadable (for those who can) Compiled from over 5 million books, published up to 2008 Data has publication dates; good for charting historical trend Books were digitized using OCR In multiple languages American/British English, Chinese, French, German, Hebrew, Italian, Russian, Spanish 2/2/2017 8

9 Large-ish COCA n-gram lists Word 2-5 grams, each containing top ~1 million entries Based on COCA (The Corpus of Contemporary American English) ( 520 million words as of Jan 2017 COCA's full unigram list is not free. COCA's top 5000 words/lemmas Contains lemma and POS of top 5,000 words 2/2/2017 9

10 Excerpted, manageable Natural Language Corpus Data: Beautiful Data by Peter Norvig Has lists of large-scale English n-gram data: character ( 1- & 2- grams) and word level (1, 2, 3 grams) Data derived/excerpted from Google Web 1T 5-Gram corpus ¼ million most frequent bigrams Google's original data is 315 mil 2/2/

11 1-grams/word list: Norvig vs. ENABLE count_1w.txt the of and to a in for is on that by this with i goofel gooek gooddg gooblle gollgo golgw Total # of entries: 333K vs. 173K Overlap? Usefulness? enable1.txt aa aah aahed aahing aahs aal aalii aaliis aals aardvark aardvarks aardwolf aardwolves aargh aarrgh zymotic zymurgies zymurgy zyzzyva zyzzyvas 11

12 2-grams: Norvig vs. COCA count_2w.txt you get you getting you give you go you going you gone you gonna you good you got you gotta you graduate you grant you great you grep you grew you grow you guess you guessed you guys you had you hand Compiled from: 1 trillion words vs. 500 million words w2_.txt you get 30 you gets 31 you gettin 861 you getting 263 you girls 24 you git 5690 you give 138 you given 169 you giving 182 you glad 46 you glance you go 70 you god 54 you goddamn 115 you goin 9911 you going 1530 you gon 262 you gone 444 you good 25 you google you got 12

13 2-grams: Norvig vs. COCA count_2w.txt you get you getting you give you go you going you gone you gonna you good you got you gotta you graduate you grant you great you grep you grew you grow you guess you guessed you guys you had you hand Total # of entries: ¼ million* vs. 1 million *NOT google's fault! Norvig only took top 0.1% of 315 million. Usefulness? w2_.txt you get 30 you gets 31 you gettin 861 you getting 263 you girls 24 you git 5690 you give 138 you given 169 you giving 182 you glad 46 you glance you go 70 you god 54 you goddamn 115 you goin 9911 you going 1530 you gon 262 you gone 444 you good 25 you google you got 13

14 Know your data When using publicly available resources, you must evaluate and understand the data. Origin? Domain & genre? Size? Traits? Merits and limitations? Fit with your project? 2/2/