Recap from previous lecture. Information Retrieval. Topics for Today. Recall: Basic structure of an Inverted index. Dictionaries & Tolerant Retrieval

Transcription

1 Recap from previous lecture nformation Retrieval Dictionaries & Tolerant Retrieval Jörg Tiedemann Department of Linguistics and Philology Uppsala University nverted indexes dictionary & postings type/token distinction terms = normalized types put in the dictionary Boolean Model return exact matches for Boolean queries Jörg Tiedemann 1/1 Jörg Tiedemann 2/1 Topics for Today Recall: Basic structure of an nverted index Continue with the basics (inverted indexes) 1. Phrase queries 2. Dictionary data structures 3. Tolerant retrieval wild-card queries spelling correction For each term t, we store a list of all documents that contain t. BRUTUS! CAESAR! CALPURNA! {z } {z } dictionary postings Jörg Tiedemann 3/1 Jörg Tiedemann 4/1

2 Phrase queries Query: stanford university as a phrase (10% of web queries = phrase queries) dea one: Biword index! index every word bigram longer phrases: stanford university palo alto : Problems? 1. STANFORD UNVERSTY AND UNVERSTY PALO AND PALO ALTO 2. post-filtering of all hits Phrase queries dea two: Positional indexes! (store positions in posting list) Example query: to 1 be 2 or 3 not 4 to 5 be 6 TO, : h 1: h 7, 18, 33, 72, 86, 231i; 2: h1, 17, 74, 222, 255i; 4: h 8, 16, 190, 429, 433i; 5: h363, 367i; 7: h13, 23, 191i;...i BE, : h 1: h 17, 25i; 4: h 17, 191, 291, 430, 434i; 5: h14, 19, 101i;...i Document 4 is a match! Jörg Tiedemann 5/1 Jörg Tiedemann 6/1 Proximity search Data Structures for Dictionaries For each term t, we store a list of all documents that contain t. Second advantage of positional indexes: Can also use them for proximity search. For example: employment /4 place Find all documents that contain EMPLOYMENT and PLACE within 4 words of each other. BRUTUS! CAESAR! CALPURNA! {z } {z } dictionary postings Jörg Tiedemann 7/1 Jörg Tiedemann 8/1

3 Naive Dictionary: array of fixed-width entries Dictionary Data structures term document pointer to frequency postings list a 656,265! aachen 65! zulu 221! space needed: 20 bytes 4 bytes 4 bytes Two main classes of data structures: hashes and trees Criteria for when to use hashes vs. trees: s there a fixed number of terms or will it keep growing? What are the relative frequencies with which various keys will be accessed? How many terms are we likely to have? How do we store a dictionary in memory efficiently? How do we look up an element in this array at query time? Jörg Tiedemann 9/1 Jörg Tiedemann 10/1 Hashes Trees: Binary tree Each vocabulary term is hashed into an integer. Try to avoid collisions At query time, do the following: hash query term, resolve collisions, locate entry in fixed-width array Pros: Lookup is fast (faster than in a search tree) Lookup time is constant Cons no way to find minor variants (resume vs. résumé) no prefix search (all terms starting with automat) need to rehash everything periodically if vocabulary keeps growing Jörg Tiedemann 11/1 Jörg Tiedemann 12/1

4 Trees: Binary tree Trees: B-tree simplest tree structure efficient for searching Pros: solves the prefix problem (terms starting with automat) Cons: slower: O(log M) in balanced trees (M is the size of the vocabulary) re-balancing binary trees is expensive! Alternative: B-tree B-tree definition: every internal node has a number of children in the interval [a, b] where a, b are appropriate positive integers, e.g., [2, 4]. Jörg Tiedemann 13/1 Jörg Tiedemann 14/1 Trees: B-tree Tolerant Retrieval What s the difference? slightly more complex still efficient for searching same features as binary trees (prefix search!) need for re-balancing is less frequent! Wildcard queries Spelling correction Jörg Tiedemann 15/1 Jörg Tiedemann 16/1

5 Tolerant Retrieval: Wildcard queries How to handle * in the middle of a term Prefix queries: mon*: find all docs containing any term beginning with mon Easy with B-tree dictionary: retrieve all terms t in the range: mon apple t < moo Suffix Queries: *mon: find all docs containing any term ending with mon Maintain an additional tree for terms backwards Then retrieve all terms t in the range: nom apple t < non Example: pro*cent We could look up pro* and *cent in the B-tree and intersect the two term sets. Expensive! Alternative: permuterm index! How can we find all terms matching pro*cent? Jörg Tiedemann 17/1 Jörg Tiedemann 18/1 Permuterm index Permuterm index Rotate wildcard query, so that the * occurs at the end. introduce special symbol $ to indicate end of string! pro*cent becomes cent$pro* Add all rotations to B-tree: HELLO! hello$, ello$h, llo$he, lo$hel o$hell How does this help when matching queries with the index? Jörg Tiedemann 19/1 Jörg Tiedemann 20/1

6 Permuterm index & term mapping Processing a lookup in the permuterm index all rotated entries map to the same string... For X, look up X$ For X*, look up X* For *X, look up X$* For *X*, look up X* For X*Y, look up Y$X* Example: For hel*o, look up o$hel* To sum up: Rotate query wildcard to the right Use B-tree lookup as before What is the problem with this structure? Permuterm more than quadruples the size of the dictionary compared to a regular B-tree. (empirical observation for English) Jörg Tiedemann 21/1 Jörg Tiedemann 22/1 Alternative: Bigram (k-gram) indexes Postings list in a 3-gram inverted index Enumerate all k-grams (sequence of k characters) occurring in a term! more space-efficient than permuterm index Example: from April is the cruelest month we get the 2-grams (bigrams): $a ap pr ri il l$ $i is s$ $t th he e$ $c cr ru ue el le es st t$ $m mo on nt h$ $ is a special word boundary symbol, as before. Maintain a second inverted index from bigrams to the dictionary terms that contain the bigram... etr - BEETROOT - METRC - PETRFY - RETREVAL retrieve all postings of matching k-grams intersect all lists as usual Jörg Tiedemann 23/1 Jörg Tiedemann 24/1

7 Processing wildcarded terms in a bigram index Query mon* can now be run as: $m AND mo AND on Gets us all terms with the prefix mon, but also many false positives like MOON.! Must post-filter these terms against query. Surviving terms are then looked up in the term-document inverted index.! k-gram are still fast and more space efficient than permuterm indexes. Processing wildcard queries As before, we must potentially execute a large number of Boolean queries Most straightforward semantics: Conjunction of disjunctions Recall the query: gen* AND universit* (geneva AND university) OR (geneva AND université) OR (genève AND university) OR (genève AND université) OR (general AND universities) OR...! Very expensive!! Requires query optimization! Do we need to support wildcard queries? Users are lazy! f wildcards are allowed! Users will love it (?) Does Google allow wildcard queries? (And other engines?) Jörg Tiedemann 25/1 Jörg Tiedemann 26/1 Tolerant Retrieval: Spälling Korrection Correcting documents Two general uses of spelling correction: Correcting documents being indexed Correcting user queries (more common) Two different methods: solated word spelling correction Check each word on its own for misspelling Will not catch typos resulting in correctly spelled words, e.g., an asteroid that fell form the sky Context-sensitive spelling correction Look at surrounding words Can correct form/from error above primarily for OCR ed documents tuned for OCR mistakes (trained classifiers) may use domain-specific knowledge confusion between O and D etc... but also: correct typos in web pages and low quality documents! fewer misspellings in our dictionary and better matching general R philosophy: don t change the documents Jörg Tiedemann 27/1 Jörg Tiedemann 28/1

8 Correcting queries solated word correction more common in R than document correction typos in queries are common people are in a hurry users often look for things they don t know much about example: al quajda strategies: (also) retrieve documents with the correct spelling return alternative query suggestions ( Did you mean...? ) Premise 1: There is a list of correct words from which the correct spellings come. Premise 2: We have a way of computing the distance between a misspelled word and a correct word. Simple spelling correction algorithm: return the correct word that has the smallest distance to the misspelled word. Example: informaton! information Jörg Tiedemann 29/1 Jörg Tiedemann 30/1 solated word correction solated word correction Two choices: use a standard lexicon Webster s, OED etc.... industry-specific dictionary (for domain-specific R) advantage: correct entries only! vocabulary of the inverted index all words in the collection! better coverage but: include all misspellings! compute weights for all terms (based on frequencies) Task: Return lexicon entry that is closest to a given character sequence Q What is closest? Several alternatives: 1. Edit distance and Levenshtein distance 2. Weighted edit distance 3. k-gram overlap Jörg Tiedemann 31/1 Jörg Tiedemann 32/1

9 Edit distance Levenshtein distance: Computation The edit distance between string s 1 and string s 2 is the minimum number of basic operations that convert s 1 to s 2. Levenshtein distance: basic operations = insert, delete, and replace Levenshtein distance dog-do: 1 Levenshtein distance cat-cart: 1 Levenshtein distance cat-cut: 1 Levenshtein distance cat-act: 2 Damerau-Levenshtein: additional operation = transpose Damerau-Levenshtein distance cat-act: 1 Recursive definition & dynamic programming: start with upper-left table cell fill table with edit costs f a s t c a t s Jörg Tiedemann 33/1 Jörg Tiedemann 34/1 Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Jörg Tiedemann 35/1 Jörg Tiedemann 36/1

10 Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Jörg Tiedemann 37/1 Jörg Tiedemann 38/1 Levenshtein distance: algorithm Each cell of Levenshtein matrix LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy cost of getting here from my upper left neighbor (copy or replace) cost of getting here from my left neighbor (insert) cost of getting here from my upper neighbor (delete) the minimum of the three possible moves ; the cheapest way of getting here Jörg Tiedemann 39/1 Jörg Tiedemann 40/1

11 Levenshtein distance: Example Weighted edit distance c a t s f a s t As above, but weight of an operation depends on the characters involved. Meant to capture keyboard errors, e.g., m more likely to be mistyped as n than as q. Therefore, replacing m by n is a smaller edit distance than by q. We now require a weight matrix as input. Modify dynamic programming to handle weights. Jörg Tiedemann 41/1 Jörg Tiedemann 42/1 Using edit distance Using edit distance given a query: get all sequences within a fixed edit distance intersect this list with the list of correct words return spelling suggestions Alternatively: use all corrections to retrieve doc s! slow! (and accepted by user?) use single best correction for retrieval Problems: lot s of possible strings even with few edit operations intersection with dictionary is slow Possible solution: use N-gram overlap can replace edit distance for spelling correction Jörg Tiedemann 43/1 Jörg Tiedemann 44/1

12 k-gram indexes for spelling correction Get all k-grams in the query term Use the k-gram index to retrieve correct words that match query term k-grams (recall wildcard search) Threshold by number of matching k-grams (e.g., only terms that differ by at most 3 k-grams) or use Jaccard coefficient > threshold Example: Jaccard(A, B) = A \ B A [ B Bigram index, misspelled word bordroom Bigrams: bo, or, rd, dr, ro, oo, om k-gram indexes for spelling correction: bordroom... BO - aboard - about -boardroom - border OR - border - lord - morbid - sordid RD - aboard - ardent -boardroom - border! boardroom exists in 6 out of 7 lists! Jaccard = 6/( ) =6/ Jörg Tiedemann 45/1 Jörg Tiedemann 46/1 Context-sensitive spelling correction Context-sensitive spelling correction Another approach: Hit-based spelling correction One approach: break phrase query into conjunction of biwords look for biwords that need only one term corrected get phrase matches and rank them... Example: flew form munich Try all phrases with possible corrections: Try query flea form munich Try query flew from munich Try query flew form munch The correct query flew from munich has most hits. Many alternatives!! Not very efficient! try to correct only if few hits returned tweaking with query logs and expected hits Jörg Tiedemann 47/1 Jörg Tiedemann 48/1

13 To sum up Using a positional inverted index with skip pointers efficient dictionary storage wild-card index spelling correction Soundex: Find phonetic alternatives We can quickly run a query like (SPELL(moriset) /3 tor*to) OR SOUNDEX(chaikofski) Jörg Tiedemann 49/1