Introduction to Markov Models

Transcription

1 Itroductio to Markov Models But first: A few prelimiaries o text preprocessig Estimatig the probability of phrases of words, seteces, etc. What couts as a word? A tricky questio. How to fid Seteces?? CIS 42/52 - Itro to AI 3 CIS 42/52 - Itro to AI 4 Q: How to estimate the probability of a give setece W? A crucial step i speech recogitio (ad lots of other applicatios First guess: bag of words : Give word lattice: form subsidy for farm subsidies far Pˆ( W P ( w ww Uigram couts (i.7 * 0 6 words of AP text: form 83 subsidy 5 for 885 farm 74 subsidies 55 far 570 Most likely word strig give PW ˆ( is t quite right CIS 42/52 - Itro to AI 5 Predictig a word sequece II Next guess: products of bigrams For W=w w 2 w 3 w, Give word lattice: farm subsidies far Bigram couts (i.7 * 0 6 words of AP text: form subsidy 0 subsidy for 2 form subsidies 0 subsidy far 0 farm subsidy 0 subsidies for 6 farm subsidies 4 subsidies far 0 Much Better (if ot quite right (Q: the couts are tiy! Why? CIS 42/52 - Itro to AI 6 Pˆ( W P ( w w i i i form subsidy for

2 How ca we estimate P(W correctly? Problem: Naïve Bayes model for bigrams violates idepedece assumptios. Let s do this right. Let W=w w 2 w 3 w. The, by the chai rule, P( W P( w * P( w w * P( w w w *...* P( w w... w We ca estimate P(w 2 w by the Maximum Likelihood Estimator ad P(w 3 w w 2 by ad so o Cout( ww 2 Cout( w Cout( ww 2w3 Cout( w w CIS 42/52 - Itro to AI 7 2 ad fially, Estimatig P(w w w 2 w - Agai, we ca estimate P(w w w 2 w - with the MLE Cout( w w... w Cout( w w w So to decide pat vs. pot i Heat up the oil i a large p?t, compute for pot Cout("Heat up the oil i a large pot" Cout("Heat up the oil i a large" UNLESS OUR CORPUS IS REALLY HUGE BOTH COUNTS WILL BE 0, yieldig 0/0 CIS 42/52 - Itro to AI 8 The Web is HUGE!! (206 versio But what if we oly have 00 millio words for our estimates?? 48.9/403=0.2 CIS 42/52 - Itro to AI 9 CIS 42/52 - Itro to AI 0 A BOTEC Estimate of What We Ca Estimate What parameters ca we estimate with 00 millio words of traiig data?? Assumig (for ow uiform distributio over oly 5000 words So eve with 0 8 words of data, for eve trigrams we ecouter the sparse data problem.. CIS 42/52 - Itro to AI Review: How ca we estimate P(W correctly? Problem: Naïve Bayes model for bigrams violates idepedece assumptios. Let s do this right. Let W=w w 2 w 3 w. The, by the chai rule, P( W P( w * P( w w * P( w w w *...* P( w w... w We ca estimate P(w 2 w by the Maximum Likelihood Estimator Cout( ww 2 Cout( w ad P(w 3 w w 2 by Cout( ww 2w3 Cout( ww 2 ad so o CIS 42/52 - Itro to AI 2 2

3 The Markov Assumptio: Oly the Immediate Past Matters The Markov Assumptio: Estimatio We estimate the probability of each w i give previous cotext by P(w i w w 2 w i- = P(w i w i- which ca be estimated by Cout( wi wi Cout( w i So we re back to coutig oly uigrams ad bigrams!! AND we have a correct practical estimatio method for P(W give the Markov assumptio! CIS 42/52 - Itro to AI 3 CIS 42/52 - Itro to AI 4 Markov Models Review (ad crucial for upcomig homework: Cumulative distributio Fuctios (CDFs The CDF of a radom variable X is deoted by F X (x ad is defied by F X (x=pr(x x F is mootoic odecreasig: x y, F x F y If X is a discrete radom variable that attais values x, x 2,, x with probabilities p(x, p(x 2, the FX ( xi p( xi ji CIS 42/52 - Itro to AI 5 CIS 42/52 - Itro to AI 6 CDF for a very small Eglish corpus Corpus: the mouse ra up the clock. The spider ra up the waterspout. P(the=4/2, P(ra=P(up=2/2 P(mouse=P(clock=P(spider=P(waterspout=/2 Arbitrarily fix a order: w=the, w2=ra, w3=up, w4=mouse, / 0/ 9/2 8/2 7/2 6/2 5/2 ` 4/2 3/2 2/2 F(the=4/2 F(ra=6/2 F(up=8/2 F(mouse=9/2 ` Visualizig a -gram based laguage model: the Shao/Miller/Selfridge method To geerate a sequece of words give uigram estimates: Fix some orderig of the vocabulary v v 2 v 3 v k. For each word positio i, i Choose a radom value r i betwee 0 ad Choose w i = the first v j such that F V v r j i i.e the first v j such that P( v r m m i /2 The Ra Up Mouse Clock Spider waterspout CIS 42/52 - Itro to AI 7 CIS 42/52 - Itro to AI 8 3

4 Visualizig a -gram based laguage model: the Shao/Miller/Selfridge method The Shao/Miller/Selfridge method traied o Shakespeare To geerate a sequece of words give a st order Markov model (i.e. coditioed o oe previous word: Fix some orderig of the vocabulary v v 2 v 3 v k. Use uigram method to geerate a iitial word w For each remaiig positio i, 2 i Choose a radom value r i betwee 0 ad j Choose w i = the first v j such that P( vm wi ri m CIS 42/52 - Itro to AI 9 (This ad ext two slides from Jurafsky CIS 42/52 - Itro to AI 20 Wall Street Joural just is t Shakespeare Shakespeare as corpus N=884,647 tokes, V=29,066 Shakespeare produced 300,000 bigram types out of V 2 = 844 millio possible bigrams. So 99.96% of the possible bigrams were ever see (have zero etries i the table Quadgrams worse: What's comig out looks like Shakespeare because it is Shakespeare CIS 42/52 - Itro to AI 2 CIS 42/52 - Itro to AI 22 The Sparse Data Problem Agai Eglish word frequecies well described by Zipf s Law Zipf (949 characterized the relatio betwee word frequecy ad rak as: f r C (for costat C r C/f log(r log(c - log (f Purely Zipfia data plots as a straight lie o a loglog scale So we smooth. How likely is a 0 cout? Much more likely tha I let o!!! CIS 42/52 - Itro to AI 23 *Rak (r: The umerical positio of a word i a list sorted by decreasig frequecy (f. CIS 42/52 - Itro to AI 24 4

5 Word frequecy & rak i Brow Corpus vs Zipf Zipf s law for the Brow corpus Lots of area uder the tail of this curve! From: Iteractive mathematics CIS 42/52 - Itro to AI 25 CIS 42/52 - Itro to AI 26 Exploitig Zipf to do Laguage ID #The followig filters out arabic words that are also frequet i Spaish ad Eglish... arabic_top_2 = [ '7ata', 'aa', 'ma', 'w', 'bs', 'fe', 'b3d', '3adou', 'm', 'ka', 'me', 'ahmed' ] #The followig filters out urdu words commo i Eglish urdu_top_7 = ['hai', 'ko', 'ki', 'mai', 'a', 'se', 'ho', 'bhi', 'mei', 'ka', 'tum', 'ahi', 'meri', 'jo', 'wo', 'dil', 'hai'] spaish_top_6 = ['de', 'la', 'que', 'el', 'e', 'y', 'es', 'u', 'los', 'por', 'se', 'para', 'co'] eglish_top_20 = ['the', 'to', 'of', 'i', 'i', 'a', 'is', 'ad, 'you', 'for', 'o', 'it', 'that', 'are', 'with', 'am', 'my', 'be', 'at' 'ot', 'we'] All the code you eed. #TO GET BEST LANGUAGE AS STRING: lid_pick_best(lid_process_tweet(tweet couts=collectios.couter( def lid_process_tweet(tweet: couts.clear( for word i re.split(r'[\.?!,]*\s+, tweet.ecode('ascii','replace.strip(.lower(: if ot re.match(r' for lag i laguages: if word i topwords[lag]: #( eglish, arabic,... couts[lag]+= #dict of word lists idexed by lad retur couts.most_commo( def lid_pick_best (cout_list: if cout_list: retur cout_list[0][0] else: retur 'UNKNOWN' CIS 42/52 - Itro to AI 27 CIS 42/52 - Itro to AI 28 5