Recap from previous lectures. Information Retrieval. Recap from previous lectures. Topics for Today. Dictionaries & Tolerant Retrieval.

Transcription

1 Recap from previous lectures nformation Retrieval Dictionaries & Tolerant Retrieval Jörg Tiedemann Department of Linguistics and Philology Uppsala University nverted indexes dictionary & postings type/token distinction terms = normalized types put in the dictionary Boolean Model return exact matches for Boolean queries Jörg Tiedemann 1/55 Jörg Tiedemann 2/55 Recap from previous lectures Topics for Today Vector Space Model term weights (TF-DF) ranked retrieval dimensionality reduction (LSA/R) Link Analysis Mining in linked data sets (networks) PageRank, Hubs, Authorities Web Crawling Going back to the basics (inverted indexes) 1. Skip pointers 2. Phrase queries 3. Dictionary data structures 4. Tolerant retrieval wild-card queries spelling correction Soundex Jörg Tiedemann 3/55 Jörg Tiedemann 4/55

2 Recall: Basic structure of an nverted index Faster intersection with skip pointers For each term t, we store a list of all documents that contain t. BRUTUS! skip sections of postings with short cuts more efficient with long posting lists CAESAR! CALPURNA! {z } {z } dictionary postings How do we use the skip pointers? Where do we put skip pointers? Jörg Tiedemann 5/55 Jörg Tiedemann 6/55 Where do we place skips? Phrase queries Tradeoff: number of items skipped vs. frequency of skips More skips: Each skip pointer skips only a few items, but we can frequently use it. Fewer skips: Each skip pointer skips many items, but we can not use it very often. Simple heuristic: for postings list of length P, use p P evenly-spaced skip pointers (ignoring term distribution) Easy for static index, harder for dynamic ones! Gain is not much with today s fast CPUs! Query: stanford university as a phrase (10% of web queries = phrase queries) dea one: Biword index! index every word bigram longer phrases: stanford university palo alto : Problems? 1. STANFORD UNVERSTY AND UNVERSTY PALO AND PALO ALTO 2. post-filtering of all hits Jörg Tiedemann 7/55 Jörg Tiedemann 8/55

3 Phrase queries Proximity search dea two: Positional indexes! (store positions in posting list) Example query: to 1 be 2 or 3 not 4 to 5 be 6 TO, : h 1: h 7, 18, 33, 72, 86, 231i; 2: h1, 17, 74, 222, 255i; 4: h 8, 16, 190, 429, 433i; 5: h363, 367i; 7: h13, 23, 191i;...i BE, : h 1: h 17, 25i; 4: h 17, 191, 291, 430, 434i; 5: h14, 19, 101i;...i Document 4 is a match! Jörg Tiedemann 9/55 Second advantage of positional indexes: Can also use them for proximity search. For example: employment /4 place Find all documents that contain EMPLOYMENT and PLACE within 4 words of each other. Jörg Tiedemann 10/55 Data Structures for Dictionaries Naive Dictionary: array of fixed-width entries For each term t, we store a list of all documents that contain t. BRUTUS! CAESAR! CALPURNA! {z } {z } dictionary postings term document pointer to frequency postings list a 656,265! aachen 65! zulu 221! space needed: 20 bytes 4 bytes 4 bytes How do we store a dictionary in memory efficiently? How do we look up an element in this array at query time? Jörg Tiedemann 11/55 Jörg Tiedemann 12/55

4 Dictionary Data structures Hashes Two main classes of data structures: hashes and trees Criteria for when to use hashes vs. trees: s there a fixed number of terms or will it keep growing? What are the relative frequencies with which various keys will be accessed? How many terms are we likely to have? Each vocabulary term is hashed into an integer. Try to avoid collisions At query time, do the following: hash query term, resolve collisions, locate entry in fixed-width array Pros: Lookup is fast (faster than in a search tree) Lookup time is constant Cons no way to find minor variants (resume vs. résumé) no prefix search (all terms starting with automat) need to rehash everything periodically if vocabulary keeps growing Jörg Tiedemann 13/55 Jörg Tiedemann 14/55 Trees: Binary tree Trees: Binary tree simplest tree structure efficient for searching Pros: solves the prefix problem (terms starting with automat) Cons: slower: O(log M) in balanced trees (M is the size of the vocabulary) re-balancing binary trees is expensive! Alternative: B-tree Jörg Tiedemann 15/55 Jörg Tiedemann 16/55

5 Trees: B-tree Trees: B-tree What s the difference? B-tree definition: every internal node has a number of children in the interval [a, b] where a, b are appropriate positive integers, e.g., [2, 4]. slightly more complex still efficient for searching same features as binary trees (prefix search!) need for re-balancing is less frequent! Jörg Tiedemann 17/55 Jörg Tiedemann 18/55 Tolerant Retrieval Tolerant Retrieval: Wildcard queries Wildcard queries Spelling correction Soundex Prefix queries: mon*: find all docs containing any term beginning with mon Easy with B-tree dictionary: retrieve all terms t in the range: mon apple t < moo Suffix Queries: *mon: find all docs containing any term ending with mon Maintain an additional tree for terms backwards Then retrieve all terms t in the range: nom apple t < non How can we find all terms matching pro*cent? Jörg Tiedemann 19/55 Jörg Tiedemann 20/55

6 How to handle * in the middle of a term Permuterm index Example: pro*cent We could look up pro* and *cent in the B-tree and intersect the two term sets. Expensive! Alternative: permuterm index! Rotate wildcard query, so that the * occurs at the end. introduce special symbol $ to indicate end of string! pro*cent becomes cent$pro* How does this help when matching queries with the index? Jörg Tiedemann 21/55 Jörg Tiedemann 22/55 Permuterm index Permuterm index & term mapping all rotated entries map to the same string... Add all rotations to B-tree: HELLO! hello$, ello$h, llo$he, lo$hel o$hell For X, look up X$ For X*, look up X* For *X, look up X$* For *X*, look up X* For X*Y, look up Y$X* Example: For hel*o, look up o$hel* Jörg Tiedemann 23/55 Jörg Tiedemann 24/55

7 Processing a lookup in the permuterm index Alternative: Bigram (k-gram) indexes To sum up: Rotate query wildcard to the right Use B-tree lookup as before What is the problem with this structure? Permuterm more than quadruples the size of the dictionary compared to a regular B-tree. (empirical observation for English) Enumerate all k-grams (sequence of k characters) occurring in a term! more space-efficient than permuterm index Example: from April is the cruelest month we get the 2-grams (bigrams): $a ap pr ri il l$ $i is s$ $t th he e$ $c cr ru ue el le es st t$ $m mo on nt h$ $ is a special word boundary symbol, as before. Maintain a second inverted index from bigrams to the dictionary terms that contain the bigram Jörg Tiedemann 25/55 Jörg Tiedemann 26/55 Postings list in a 3-gram inverted index Processing wildcarded terms in a bigram index... etr - BEETROOT - METRC - PETRFY - RETREVAL retrieve all postings of matching k-grams intersect all lists as usual Query mon* can now be run as: $m AND mo AND on Gets us all terms with the prefix mon, but also many false positives like MOON.! Must post-filter these terms against query. Surviving terms are then looked up in the term-document inverted index.! k-gram are still fast and more space efficient than permuterm indexes. Jörg Tiedemann 27/55 Jörg Tiedemann 28/55

8 Processing wildcard queries Tolerant Retrieval: Spälling Korrection As before, we must potentially execute a large number of Boolean queries Most straightforward semantics: Conjunction of disjunctions Recall the query: gen* AND universit* (geneva AND university) OR (geneva AND université) OR (genève AND university) OR (genève AND université) OR (general AND universities) OR...! Very expensive!! Requires query optimization! Do we need to support wildcard queries? Users are lazy! f wildcards are allowed! Users will love it (?) Does Google allow wildcard queries? (And other engines?) Two general uses of spelling correction: Correcting documents being indexed Correcting user queries (more common) Two different methods: solated word spelling correction Check each word on its own for misspelling Will not catch typos resulting in correctly spelled words, e.g., an asteroid that fell form the sky Context-sensitive spelling correction Look at surrounding words Can correct form/from error above Jörg Tiedemann 29/55 Jörg Tiedemann 30/55 Correcting documents Correcting queries primarily for OCR ed documents tuned for OCR mistakes (trained classifiers) may use domain-specific knowledge confusion between O and D etc... but also: correct typos in web pages and low quality documents! fewer misspellings in our dictionary and better matching general R philosophy: don t change the documents more common in R than document correction typos in queries are common people are in a hurry users often look for things they don t know much about example: al quajda strategies: (also) retrieve documents with the correct spelling return alternative query suggestions ( Did you mean...? ) Jörg Tiedemann 31/55 Jörg Tiedemann 32/55

9 solated word correction solated word correction Premise 1: There is a list of correct words from which the correct spellings come. Premise 2: We have a way of computing the distance between a misspelled word and a correct word. Simple spelling correction algorithm: return the correct word that has the smallest distance to the misspelled word. Example: informaton! information Two choices: use a standard lexicon Webster s, OED etc.... industry-specific dictionary (for domain-specific R) advantage: correct entries only! vocabulary of the inverted index all words in the collection! better coverage but: include all misspellings! compute weights for all terms (based on frequencies) Jörg Tiedemann 33/55 Jörg Tiedemann 34/55 solated word correction Edit distance Task: Return lexicon entry that is closest to a given character sequence Q What is closest? Several alternatives: 1. Edit distance and Levenshtein distance 2. Weighted edit distance 3. k-gram overlap The edit distance between string s 1 and string s 2 is the minimum number of basic operations that convert s 1 to s 2. Levenshtein distance: basic operations = insert, delete, and replace Levenshtein distance dog-do: 1 Levenshtein distance cat-cart: 1 Levenshtein distance cat-cut: 1 Levenshtein distance cat-act: 2 Damerau-Levenshtein: additional operation = transpose Damerau-Levenshtein distance cat-act: 1 Jörg Tiedemann 35/55 Jörg Tiedemann 36/55

10 Levenshtein distance: Computation Recursive definition & dynamic programming: start with upper-left table cell fill table with edit costs f a s t c a t s Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Jörg Tiedemann 37/55 Jörg Tiedemann 38/55 Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Jörg Tiedemann 39/55 Jörg Tiedemann 40/55

11 Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Levenshtein distance: algorithm LEVENSHTENDSTANCE(s 1, s 2 ) 1 for i 0 to s 1 2 do m[i, 0] =i 3 for j 0 to s 2 4 do m[0, j] =j 5 for i 1 to s 1 6 do for j 1 to s 2 7 do if s 1 [i] =s 2 [j] 8 then m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]} 9 else m[i, j] =min{m[i 1, j]+1, m[i, j 1]+1, m[i 1, j 1]+1} 10 return m[ s 1, s 2 ] Operations: insert, delete, replace, copy Jörg Tiedemann 41/55 Jörg Tiedemann 42/55 Each cell of Levenshtein matrix Levenshtein distance: Example f a s t cost of getting here from my upper left neighbor (copy or replace) cost of getting here from my left neighbor (insert) cost of getting here from my upper neighbor (delete) the minimum of the three possible moves ; the cheapest way of getting here c a t s Jörg Tiedemann 43/55 Jörg Tiedemann 44/55

12 Weighted edit distance Using edit distance As above, but weight of an operation depends on the characters involved. Meant to capture keyboard errors, e.g., m more likely to be mistyped as n than as q. Therefore, replacing m by n is a smaller edit distance than by q. We now require a weight matrix as input. Modify dynamic programming to handle weights. given a query: get all sequences within a fixed edit distance intersect this list with the list of correct words return spelling suggestions Alternatively: use all corrections to retrieve doc s! slow! (and accepted by user?) use single best correction for retrieval Jörg Tiedemann 45/55 Jörg Tiedemann 46/55 Using edit distance Problems: lot s of possible strings even with few edit operations intersection with dictionary is slow Possible solution: use N-gram overlap can replace edit distance for spelling correction k-gram indexes for spelling correction Get all k-grams in the query term Use the k-gram index to retrieve correct words that match query term k-grams (recall wildcard search) Threshold by number of matching k-grams (e.g., only terms that differ by at most 3 k-grams) or use Jaccard coefficient > threshold Example: Jaccard(A, B) = A \ B A [ B Bigram index, misspelled word bordroom Bigrams: bo, or, rd, dr, ro, oo, om Jörg Tiedemann 47/55 Jörg Tiedemann 48/55

13 k-gram indexes for spelling correction: bordroom Context-sensitive spelling correction BO - aboard - about -boardroom - border OR - border - lord - morbid - sordid RD - aboard - ardent -boardroom - border One approach: break phrase query into conjunction of biwords look for biwords that need only one term corrected get phrase matches and rank them......! boardroom exists in 6 out of 7 lists! Jaccard = 6/( ) =6/ Jörg Tiedemann 49/55 Jörg Tiedemann 50/55 Context-sensitive spelling correction Another approach: Hit-based spelling correction Example: flew form munich Try all phrases with possible corrections: Try query flea form munich Try query flew from munich Try query flew form munch The correct query flew from munich has most hits. Many alternatives!! Not very efficient! try to correct only if few hits returned tweaking with query logs and expected hits Tolerant Search: Soundex Find phonetic alternatives. Example: chebyshev / tchebyscheff Soundex = class of heuristics (invented in 1918) 1. Retain the first letter of the term. 2. Replace all letters [AEOUHWY] to 0 3. Replace letters to digits: B, F, P, V to 1 C, G, J, K, Q, S, X, Z to 2 D,T to 3 L to 4 M, N to 5 R to 6 4. remove consecutive identical digits 5. remove all zeros and return first 4 characters Jörg Tiedemann 51/55 HERMAN! H655 Will HERMANN generate the same code? Jörg Tiedemann 52/55

14 To sum up Resources Using a positional inverted index with skip pointers efficient dictionary storage wild-card index spelling correction Soundex We can quickly run a query like (SPELL(moriset) /3 tor*to) OR SOUNDEX(chaikofski) Chapter 3 of R Resources at Soundex demo Levenshtein distance demo Levenshtein distance slides Peter Norvig s spelling corrector Jörg Tiedemann 53/55 Jörg Tiedemann 54/55 Next time Ranked Retrieval: efficient scoring & retrieval Putting it all together in a search system lab on boolean and ranked retrieval Later: text extraction automatic summarization Jörg Tiedemann 55/55