Crunching Numbers to Match Integer Sequences Hieu Nguyen

Transcription

1 Slide 1 of 22 Head Bites Tail: Crunching Numbers to Match Integer Sequences Hieu Nguyen Rowan University Math Department Colloquium December 8, 2011 ABSTRACT: In this talk I will discuss an algorithm to experimentally match integer sequences as part of an ongoing project to mine the Online Encyclopedia of Integer Sequences for new identities. In particular, a similarity measure called head-bites-tail overlap will be introduced and shown how to compute distance between two finite sequences and calculate a match probability. Examples of some experimental conjectures found using a Mathematica implementation of this algorithm will be presented. This talk is highly accessible to students: those having a background in high school algebra should be able to understand most of this talk and those with a background in discrete math and introductory computer programming should fully appreciate this talk.

2 2 Head Bites Tail.nb ü Searchable online database - ü Contains almost 200,000 integer sequences ü Created by Neil Sloane (AT & T Bell Labs) ü Maintained by OEIS Foundation ü Example: F n = 0, 1, 1, 2, 3, 5, 8, 13, 21,... Slide 2 22 Online Encyclopedia of Integer Sequences (OEIS)

3 Head Bites Tail.nb 3 Slide 3 of 22 Mining the OEIS ü Data Mining (Large Scale Pattern Recognition) Process of extracting patterns from large datasets using computer science, mathematics, and statistics. ü Mine OEIS for Integer Sequence Identities ü Enlarge OEIS database to include sequence transformations ü Find matches between integer sequences (experimental conjectures) ü Prove experimental conjectures that are interesting to obtain new identities ü GOAL: Discover interesting connections between different areas of mathematics

4 4 Head Bites Tail.nb Slide 4 of 22 Experimental Pattern Matching ü Example 1 ü A : Fibonacci sequence F n = 0, 1, 1, 2, 3, 5, 8, 13, 21,, (39 terms); n ³ 0 A000045S1T3: Sums of Squares Transformation n 2 F k = 0, 1, 2, 6, 15, 40, 104,, ; n 0 k=0 A000045S1T8: Product of Consecutive Terms Transformation F n F n+1 = 0, 1, 2, 6, 15, 40, 104,, ; n 0 EXPERIMENTAL CONJECTURE: n k=0 F 2 k = F n F n+1

5 Head Bites Tail.nb 5 ü Example 2 Slide 5 of 22 ü A131524: Number of possible palindromic rows in an n X n crossword puzzle a n = 0, 0, 1, 1, 2, 2, 4, 4, 7, 7, 12,..., ; n ³ 1 (50 terms) A131524S2T4: Binomial Transform of a 2 n (pad a 0 = 0L: n k=0 H-1L k n k a 2 k = 0, 0, 1, 1, 2, 3, 5, 8, 13, 4181; n 0 ü A018910S1T4: Pisot sequence L(4,5) b n = 4, 5, 7, 10, 15, 23, 36, 57,..., n ³ 0 (39 terms) A018910S1T4: Binomial Transform of b n : n k=0 H-1L k n k b k = 4, -1, 1, 0, 1, 1, 2, 3, 5, 8, 13,, 4181,..., Hn 0L n EXPERIMENTAL CONJECTURE: k=0 H-1L n n k a 2 k = F n-1 = n+2 k=0 H-1L n+2 n + 2 k b k Hn 1L

6 6 Head Bites Tail.nb Slide 6 of 22 Hunting for Identities ü Classical Approach ü Modern Approach Small-scale (human) versus large-scale (computer) Data Mining Algorithm for Integer Sequences Database of Sequences 8aHnL< Out[108]= Generate Subsequences Apply Transformations Generate Transformed Sequences THaHn k LL Mine Database 8THaHn k LL< for Patterns

7 Head Bites Tail.nb 7 Slide 7 of 22 Pattern Matching Algorithm for Integer Sequences ahnl bhnl Out[129]= T 1 HaHn k LL T 2 HbHm k LL Compute distance d between T 1 HaHn k LL and T 2 HbHm k LL If d d max, match found: T 1 HaHn k LL=T 2 HbHn k LL If d > d max, match not found

8 8 Head Bites Tail.nb Slide 8 of 22 Database of Sequence Transformations ü Source Data - OEIS ü Set of Transformations LABEL TRANSFORMATION FORMULA T1 Identity ahnl n T2 Partial Sums k=0 ahkl n T3 Partial Sums of Squares k=0 a HkL 2 n T4 Binomial Transform k=0 H-1L k K n k O ahkl T5 Self - Convolution n k=0 ahkl ahn - kl T6 Linear Weighted Partial Sums n k=1 k ahkl n T7 Binomial Weighted Partial Sums k=0 K n k O ahkl T8 Product of Consecutive Elements a HnL a Hn + 1L T9 Cassini a Hn - 1L a Hn + 1L - a HnL 2 T10 First Stirling n k=0 shn, kl ahkl T11 Second Stirling n k=0 SHn, kl ahkl ü Create MySQL Database of Sequence Transformations Acknowledgement: Doug Taggart (Undergraduate Research Assistant)

9 Head Bites Tail.nb 9 ID Label Subsequence Transformation Position Entry1 Entry2 Entry3 1 A000045S1T A000045S1T A000045S1T A000045S1T A000045S1T A000045S1T Null 39 A000045S1T Null Null

10 10 Head Bites Tail.nb Slide 9 of 22 Matching Integer Sequences ü Exercise: Consider the finite sequence 8aHnL< = 81, 1, 2, 3, 5, 8, 13, 21, 34, 55<. Compare ahnl with each of the four finite sequences below, which are similar to ahnl but do not match exactly. Is there a way to measure how close each sequence matches with ahnl in the sense that both are likely to be subsets of the same infinite sequence (namely the Fibonacci sequence)? If so, then which sequence matches best with ahnl? 1. {1, 1, 2, 3, 5, 8, 13, 21, 47, 55} 2. {55, 89, 144, 233, 377, 610} 3. {3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377} 4. {2, 3, 5, 8, 13, 21, 34} 5. {1, 0, 1, 1, 2, 3, 5, 8, 13} ü Mathematical Model: Determine an appropriate distance function (or similarity measure) to match two sequences that are similar, but not exactly the same.

11 Head Bites Tail.nb 11 Slide 10 of 22 Overlap ü Main Assumption: Perfect data set - no errors in the values of each integer sequence ü Overlapping Run 1. {1, 1, 2, 3, 5, 8, 13, 21, 47, 55} 8aHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} NO MATCH (Worst) 2. {55, 89, 144, 233, 377, 610} 8aHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} MATCH 3. {3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377} 8aHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} MATCH 4. {2, 3, 5, 8, 13, 21, 34} 8aHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} MATCH (Best)

12 12 Head Bites Tail.nb Slide 11 of 22 Ouroboros (Bites Tail) ü What qualifies as a match between two finite sequences? :bh1l, bh2l,..., bhm - 1L, bhml> Head Tail Head Tail :ah1l, ah2l,..., ahn - 1L, ahnl> We will say that two sequences likely match or are similar (in the sense that there is a chance that both finite sequences are part of the same infinite sequence) if the head (beginning) of one sequence bites (overlaps with) the tail (end) of the other sequence.

13 Head Bites Tail.nb 13 Slide 12 of 22 Head-Bites-Tail Overlap INFORMAL DEFINITION: We say that two finite sequences contain a head-bites-tail (HBT) overlap if there is an overlapping run which starts at the beginning of one sequence and stops at the end of either sequence. Let L denote the length of an HBT overlap. There are four cases to consider: CASE 1a: L = N - n ah1l,ah2l,... ahn 0 L,...,aHNL bh1l,..., bhll,...bhml CASE 1b: L = M ah1l,ah2l,... ahn 0 L,...,aHn 0 +M-1L,...,aHNL bh1l,..., bhml CASE 2a: L = M - m ah1l,..., ahll,...ahnl bh1l,bh2l,... bhm 0 L,...,bHML CASE 2b: L = N ah1l,..., ahnl bh1l,bh2l,... bhm 0 L,...,bHm 0 +N-1L,...,bHML

14 14 Head Bites Tail.nb Slide 13 of 22 Maximum HBT Overlap N M Let 8a HnL< n=1 and 8bHmL< m=1 be two finite sequences. DEFINITION: We say that ahnl and bhnl contain a head-bites-tail (HBT) overlap of length L if one of the following two conditions hold: 1. ahn - L + kl = bhkl for all k = 1,..., L or ahn 0 + k - 1L = bhkl for a fixed positive integer n 0 and all k = 1,..., L. 2. ahkl = bhm - L + kl for all k = 1,..., L or ahkl = bhm 0 + k - 1L for a fixed positive integer m 0 and all k = 1,..., L. DEFINITION: We define L max to be the maximum HBT overlap, i.e. the length of the longest HBT overlap, between ahnl and bhnl. If no HTB overlap exists, then we set L max = 0. ü Examples 1. 8aHnL< ={1, 1, 2, 3, 5, 2, 3, 5, 2, 3, 5} 8bHnL< = {2, 3, 5, 2, 3, 5} L = aHnL< ={1, 1, 2, 3, 5, 2, 3, 5, 2, 3, 5} 8bHnL< = {2, 3, 5, 2, 3, 5} L max = 6

15 Head Bites Tail.nb 15 Slide 14 of 22 HBT Distance DEFINITION: We define the head-bites-tail (HBT) distance d between ahnl and bhnl to be where L max is the maximum HBT overlap between ahnl and bhnl. d := dhahnl, bhnll = N + M - 2 L max NOTE: d can also be thought of as specifying the number of remaining elements in ahnl and bhnl that DO NOT overlap. ü Examples 1. 8aHnL< = {55, 89, 144, 233, 377, 610} 8bHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} d = H1L = aHnL< = {3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377} 8bHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} d = H7L = aHnL< = {2, 3, 5, 8, 13, 21} 8bHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 55, 81} d = H6L = aHnL< = {2, 3} 8bHnL< = {1, 1, 2, 3, 5, 8} d = H2L = 4

16 16 Head Bites Tail.nb Slide 15 of 22 Relative HBT Distance DEFINITION: We define the relative HBT distance r between ahnl and bhnl to be NOTE: 0 r 1 d r := rhahnl, bhnll = DEFINITION: We define the HBT probability of match p between ahnl and bhnl to be ü Examples 1. 8aHnL< = {55, 89, 144, 233, 377, 610} 8bHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} H1L d r = = 14 = aHnL< = {3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377} 8bHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 34, 55} H7L d r = = 7 = aHnL< = {2, 3, 5, 8, 13, 21} 8bHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 55, 81} H6L d r = = 4 = aHnL< = {2, 3} 8bHnL< = {1, 1, 2, 3, 5, 8} H2L d r = = 4 = d = N+M-2 L = 1-2 L N+M N+M N+M p := phahnl, bhnll = 1 - r = 2 L N+M

17 Head Bites Tail.nb 17 Slide 16 of 22 HBT Conjecture HBT CONJECTURE: dhÿ, ÿl is a distance function, i.e. d satisfies the three properties: I. Positive-definiteness: dhahnl, bhnll 0 and dhahnl, bhnll = 0 iff ahnl = bhnl II. Symmetry: dhahnl, bhnll = dhbhnl, ahnll III. Triangle inequality: dhahnl, bhnll dhahnl, chnll + dhchnl, bhnll NOTE: Evidence suggests that HBT Conjecture is true for the space of monotone sequences. ü Example: Triangle Inequality 8aHnL< = {1, 1, 2, 3, 5, 8, 13, 21, 47, 55} 8bHnL< = {55, 89, 144, 233, 377, 610} 8cHnL< = {3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377} dhahnl, bhnll = H1L = 14 = 9 left + 5 right dhahnl, chnll = H7L = 7 = 3 left + 4 right dhchnl, bhnll = H5L = 7 = 6 left + 1 right d = N + M - 2 L max \ dhahnl, bhnll dhahnl, chnll + dhchnl, bhnll

18 18 Head Bites Tail.nb Slide 17 of 22 Mathematica Implementation of Maximum HBT Distance In[131]:= ü Algorithm for finding L max (maximum HBT distance) 8u HnL< N n=1 = 81, 1, 2, 3, 5, 8, 13, 21, 47, 55< 8v HmL< M m=1 = 83, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377< K 1. Take last element uhnl and find its occurrences in 8vHmL<. Denote the positions of these occurrences by 8p k < k=1 (decreasing order). 2. Loop through k = 1,..., K: If 8uHN - p k + 1L, uhn - p k + 2L,..., uhnl} = 8vH1L, vh2l,..., vhp k L<, then uhnl and vhnl have an HBT overlap of length p k. 3. Repeat steps 1 and 2, but switch roles of uhnl and vhnl. 4. Set L max equal to the length of the longest HBT overlap obtained from steps 1-3. ü Mathematica Module Clear@HBTdistanceD; HBTdistance@u_,v_D:=Module@8lengthu,lengthv,positionlastuinv,positionlastvinu, match,distance,rdistance,i,p,overlap1,overlap2,overlaptemp<, lengthu=length@ud; lengthv=length@vd; positionlastuinv=flatten@position@v,u@@lengthudddd; positionlastvinu=flatten@position@u,v@@lengthvdddd; Print@"N = ",lengthu," ; ","M = ",lengthvd; match=0; overlap1=0; If@positionlastuinv!=8<, i=1; While@match==0&&i<=Length@positionlastuinvD, p=positionlastuinv@@-idd; overlaptemp=min@lengthu,pd; If@Take@u,-overlaptempD==Take@v,8p-overlaptemp+1,p<D, match=1;overlap1=overlaptemp, i++ D D D; match=0; overlap2=0; If@positionlastvinu!=8<, i=1;

19 Head Bites Tail.nb 19 D; match=1;overlap2=overlaptemp, i++ D D If@overlap1>overlap2, distance=hlengthu+lengthv-2*overlap1l; rdistance=distanceêhlengthu+lengthvl, distance=hlengthu+lengthv-2*overlap2l; rdistance=distanceêhlengthu+lengthvl D; Print@"N+M = ",lengthu+lengthv," ; ","L max = ", Max@overlap1,overlap2DD; Print@"d = ",distance," ; ","d r = ",rdistance," ; ", "p = ",1-rdistanceD; D; ü Examples In[133]:= HBTdistance@81, 1, 2, 3, 5, 8, 13, 21, 34, 55<, 81, 1, 2, 3, 5, 8, 13, 21, 34, 55<D N = 10 ; M = 10 N+M = 20 ; L max = 10 d = 0 ; d r = 0 ; p = 1 In[134]:= HBTdistance@81, 1, 2, 3, 5, 8, 13, 21, 34, 55<, 81, 1, 2, 3, 5, 8, 13, 21, 47, 55<D N = 10 ; M = 10 N+M = 20 ; L max = 0 d = 20 ; d r = 1 ; p = 0 In[135]:= HBTdistance@81, 1, 2, 3, 5, 8, 13, 21, 34, 55<, 855, 89, 144, 233, 377, 610<D N = 10 ; M = 6 N+M = 16 ; L max = 1 d = 14 ; d r = 7 8 ; p = 1 8 In[136]:= HBTdistance@81, 1, 2, 3, 5, 8, 13, 21, 34, 55<, 83, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377<D

20 20 Head Bites Tail.nb N = 10 ; M = 11 N+M = 21 ; L max = 7 d = 7 ; d r = 1 3 ; p = 2 3 In[137]:= HBTdistance@81, 1, 2, 3, 5, 8, 13, 21, 34, 55<, 82, 3, 5, 8, 13, 21, 34<D N = 10 ; M = 7 N+M = 17 ; L max = 7 d = 3 ; d r = 3 17 ; p = 14 17

21 Head Bites Tail.nb 21 Slide 18 of 22 EUREKA Project ü Database ü Over one million sequence transformations (T1-T11) have been calculated (A A170000) ü MySQL database of transformed sequences contains over 77 million rows (each row stores a window of 3 terms of a sequence) - 5 GB file ü Search Results ü Over 300,000 matches found so far (d r 1 ê 2, L max ³ 4) ü Preliminary analysis shows: - Most matches are trivial or already mentioned in OEIS (> 99%) - Small fraction of false positives (> 0.9%)

22 22 Head Bites Tail.nb Slide 19 of 22 Ten Experimental Conjectures ü EUREKA Database Website : A000129S1T3 = A041011S1T : A000240S1T7 = A006882S1T : A000241S1T8 = A028723S1T : A000295S1T9 = A031878S1T : A001076S1T3 = A041143S1T : A014445S1T3 = A001076S1T : A041041S1T3 = A162671S1T : A108099S1T7 = A132344S1T : A120580S1T2 = A024493S1T9 (Hankel Transform) A161937S1T : A161937S1T7 = A087299S1T8

23 Head Bites Tail.nb 23 Slide 20 of 22 Next Steps ü Scale up processing power and memory ü Perform search on a cluster of computers ª ü Implement parallel/distributed computing (Rowan s 3-node CC cluster) ü Improve sequence matching algorithms ü Reduce search-times ª ü Reduce trivial matches and false positives ü Expand Scope of Search ü Enlarge collection of sequence transformations ª ü Composition of sequence transformations ü Extend search to 2-D sequences (e.g. Pascal s triangle) and rational sequences (e.g. Bernoulli numbers)

24 24 Head Bites Tail.nb Slide 21 of 22 ü Disseminate Work ü Create database website ª ü Make database website accessible to the public (collaborate with OEIS) ü Graph Network Visualization of Identities A001911A S1T4 S2T1 A S1T4 A S1T4 A S1T4 A S3T4 Search Query Search Query A S1T4 Search Query Search Query A S2T4 Search Query Search Query A S1T4 A S3T1 Search Query Search Query A Search S1T4 Query A Search S1T4 Query Search Query A S1T4 Search Query A S1T4 Search Query A S1T7 Search Query A S1T4 Search A QueryS2T A S1T Search A QueryS3T1 Search A QueryS3T1 Search A Query S2T1 A Search Query S3T1 A Search Query S2T1 A SearchS3T1 Query A Search S3T1 Query Search A Query S3T1 A S2T1 A S3T1 Search Query A S2T4 Search Query A S2T1 A S1T4 Search Query A081662A S1T4 S2T1 Search Query Search Query Search Query Search Query Search Query Search Query A Search S2T1 Query ü Publish new interesting (non-trivial) EUREKA s experimental conjectures ü Seek Help ü Need good programmers (recruit students! ª ) ü Need collaborators to analyze and prove EUREKA s experimental conjectures (suitable as student research projects)

25 Head Bites Tail.nb 25 Slide 22 of 22 The End Thank you