132-avoiding two-stack sortable permutations, Fibonacci numbers, and Pell numbers

Transcription

1 Discrete Applied Mathematics 143 (004) avoiding two-stack sortable permutations, Fibonacci numbers, Pell numbers Eric S. Egge a, Touk Mansour b a Department of Mathematics, Gettysburg College, Gettysburg, PA 1735, USA b Department of Mathematics, Haifa University, Haifa, Israel Received 3 May 00; received inrevised form 7 November 003; accepted 4 December 003 Abstract We describe the recursive structures of the set of two-stack sortable permutations which avoid 13 the set of two-stack sortable permutations which contain 13 exactly once. Using these results stard generating function techniques, we enumerate two-stack sortable permutations which avoid (or contain exactly once) 13 which avoid (or contain exactly once) an arbitrary permutation. In most cases the number of such permutations is given by a simple formula involving Fibonacci or Pell numbers. c 004 Elsevier B.V. All rights reserved. MSC: Primary 05A15 Keywords: Two-stack sortable permutation; Restricted permutation; Pattern-avoiding permutation; Forbidden subsequence; Fibonacci number; Pell number 1. Introduction notation In[1] West introduced a stack-sorting function s on the set of nite sequences of distinct positive integers. This function may be dened recursively by s( ) = s(n) = s()s()n, where are sequences n is the largest element of the sequence n. Let S n denote the set of permutations of {1; ;:::;n}, written in one-line notation. West studied permutations which satisfy s t () =1:::n, which he called t-stack sortable permutations. He showed that the one-stack sortable permutations are counted by the well-known Catalan numbers conjectured that the number of two-stack sortable permutations in S n is (=n + 1)(n +1) ( ) 3n n. This conjecture was proved analytically by Zeilberger [3] combinatorially by Dulucq et al. [13]. More recently, Bona proved [9] the surprising fact that if W t(n; k) isthe number of t-stack sortable permutations in S n with k descents then W t(n; k)=w t(n; n 1 k). In this paper we study two-stack sortable permutations from the perspective of pattern avoidance. To dene patternavoiding permutations, suppose S n S k.wesay avoids whenever contains no subsequence with all of the same pairwise comparisons as. For example, the permutation avoids , but it has 586 as a subsequence so it does not avoid 143. In this setting is called a pattern or a forbidden subsequence is called a restricted permutation or a pattern-avoiding permutation. Pattern avoidance has already been found to be closely connected with stack sortability. For instance, West has shown [1,] that a permutation is one-stack sortable if only if it avoids 31, that a permutation is two-stack sortable if only if it avoids , except that the latter pattern is allowed when it is contained in a subsequence of type Pattern avoidance has also turned out to be connected with a variety of other seemingly unrelated areas, including addresses: eggee@member.ams.org (E.S. Egge), touk@math.haifa.ac.il (T. Mansour) X/$ - see front matter c 004 Elsevier B.V. All rights reserved. doi: /j.dam

2 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) the theory of Kazhdan Lusztig polynomials [], singularities of Schubert varieties [4 8,17], Chebyshev polynomials of the second kind [1,16,19], rook polynomials for a rectangular board [18], various other sorting algorithms [10,0,,3]. In this paper we add to the list of topics connected with pattern-avoiding permutations by describing connections between two-stack sortable permutations the Fibonacci Pell numbers. We begin by describing the recursive structures of the set of two-stack sortable permutations which avoid 13 the set of two-stack sortable permutations which contain 13 exactly once. We then combine these results with stard generating function techniques to enumerate, for various S k, two-stack sortable permutations which avoid 13, two-stack sortable permutations which avoid 13 contain exactly once, two-stack sortable permutations which contain 13 exactly once avoid, two-stack sortable permutations which contain 13 exactly once. In most of these cases the number of such permutations is given by a simple formula involving the Fibonacci or Pell numbers. For example, we show that the number of two-stack sortable permutations which avoid 13 is the Pell number p n, the number of two-stack sortable permutations which avoid is F n+4 n, the number of two-stack sortable permutations which avoid 13 contain 5134 exactly once is F n+ ( ) n+1. We conclude this section by setting some notation. We write P n to denote the set of two-stack sortable permutations in S n. For any set R of permutations, we write S n(r) to denote the set of permutations in S n which avoid every patternin R, we write P n(r) to denote the set of permutations in P n which avoid every patterninr, we write P R(x) to denote the generating function given by P R(x)= P n(r) x n : We write F 0;F 1;::: to denote the sequence of Fibonacci numbers, which are given by F 0 =0, F 1 =1, F n =F n 1 +F n for n. We observe that the generating function for the Fibonacci numbers is given by F nx n x = 1 x x : (1) We also observe that F n may be interpreted combinatorially as the number of tilings of a 1 (n 1) rectangle with tiles of size We write p 0;p 1;::: to denote the sequence of Pell numbers, which are given by p 0 =0, p 1 =1, p n =p n 1 + p n for n. We observe that the generating function for the Pell numbers is given by p nx n x = 1 x x : () We also observe that p n may be interpreted combinatorially as the number of tilings of a 1 (n 1) rectangle with tiles of size 1 11, where each 1 1 tile canbe red or blue. For more informationonthe Pell numbers, see [1,3,14,15].. Two-stack sortable permutations which avoid 13 another pattern In this section, we describe the recursive structure of the set of two-stack sortable permutations which avoid 13. We then use this descriptionto enumerate P n(13;) for various P k (13). Throughout we use the fact that since 3541 contains a subsequence of type 13 (namely, 354), for any set R of permutations we have P n(13;r)=s n(13; 341; 341;R). We begin with an observation concerning the structure of the permutations in P n(13). Proposition.1. Fix n 3 suppose P n(13). Then the following hold. (i) 1 (n)=1, 1 (n)=, or 1 (n)=n. (ii) The map from P n 1(13) to P n(13) given by n; is a bijection between P n 1(13) the set of permutations in P n(13) which begin with n. (iii) The map from P n (13) to P n(13) given by n 1;n; is a bijection between P n (13) the set of permutations in P n(13) whose second entry is n.

3 74 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) 7 83 (iv) The map from P n 1(13) to P n(13) given by ; n is a bijection between P n 1(13) the set of permutations in P n(13) which end with n. Proof. (i) Suppose by way of contradiction that 1 (n) n. Since avoids 13, the elements to the left of n in are all greater thanevery element to the right of n. Since there are at least two elements to the left of n at least one element to the right of n, there must be a patternof type 341 or 341 in inwhich n plays the role of the 4. This contradicts our assumption that P n(13). (ii) Since the given map is clearly injective, it is sucient to show that P n 1(13) if only if n; P n(13). Since P n(13) = S n(13; 341; 341), it is clear that if n; P n(13) then P n 1(13). To show the converse, suppose P n 1(13). If n; contains a pattern of type 13 then the n cannot be involved in the pattern, since it is the largest element of n;. Then contains a pattern of type 13, a contradiction. By similar arguments for we nd that n; P n(13), as desired. (iii) (iv) These are similar to the proof of (ii). Theorem.. For all n 1, P n(13) = p n: (3) Proof. For notational convenience, we abbreviate P(x) =P 13(x). By Proposition.1(i), when n 3 the elements of P n(13) may be partitioned into three sets: those which begin with n, those whose second entry is n, those whose last entry is n. By Proposition.1(ii), the generating function for those elements which begin with n is x(p(x) 1 x). By Proposition.1(iii), the generating function for those elements whose second entry is n is x (P(x) 1). By Proposition.1(iv), the generating function for those elements which end with n is x(p(x) 1 x). Combine these observations to obtain P(x)=1+x +x + x(p(x) 1 x)+x (P(x) 1) + x(p(x) 1 x): Solve this equationfor P(x) compare the result with () to complete the proof. We also give a combinatorial proof of (3). Theorem.3. For all n 1, there exists a constructive bijection between P n(13) the set of tilings of a 1 (n 1) rectangle with tiles of size 1 1 1, where each 1 1 tile can be red or blue. Proof. Suppose we are given such a tiling; we construct the corresponding permutation as follows. Proceed from right to left, placing one number in each box of a tile. If the rightmost empty tile is a 1 tile thenll it with the two smallest remaining numbers, in increasing order. If the rightmost empty tile is a red 1 1 tile thenll it with the largest remaining number. If the rightmost empty tile is a blue 1 1 tile then ll it with the smallest remaining number. When all tiles have been lled, one number will remain. Place this number in the leftmost position in the permutation. To obtain a permutationinp n(13), take the (group theoretic) inverse of the permutation constructed by the process above. It is routine to construct the inverse of the map described above, thus to verify it is a bijection. Though elementary, Proposition.1 enables us to easily nd P 13;(x) for various. For instance, we have the following result involving = 1:::d. Theorem.4. (i) P 13;1(x)=1. (ii) P 13;1(x)=1=(1 x). (iii) For all d 3, P 13;1:::d (x)= (1 x) d 3 r=0 (1 x x ) r+1 x d 3 r + x d (1 x)(1 x x ) d : (4) Proof. (i) Observe that only the empty permutation avoids 1. (ii) Observe that for all n 0, the only permutation in S n which avoids 1 is n; n 1;:::;; 1.

4 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) (iii) We argue by induction on d. By Proposition.1(i), for n 3 the elements of P n(13; 1 :::d) may be partitioned into three sets: those which begin with n, those whose second entry is n, those whose last entry is n. Since avoids 1 :::d if only if n; does, if only if n 1;n; does, it follows from Proposition.1(ii) (iii) that the generating functions for the rst two sets are x(p 13;1:::d (x) 1 x) x (P 13;1:::d (x) 1), respectively. Now observe that a permutation ; n avoids 1 :::d if only if avoids 1 :::(d 1). Therefore, inview of Proposition.1(iii), the generating function for the last set is x (P 13;1:::(d 1) (x) 1 x), we have P 13;1:::d (x)=1+x +x + x(p 13;1:::d (x) 1 x)+x (P 13;1:::d (x) 1) + x(p 13;1:::(d 1) (x) 1 x): Solve this equationfor P 13;1:::d (x) to nd that for all d 3, x P 13;1:::d (x)=1+ 1 x x P 13;1:::(d 1)(x): (5) Set d = 3 in(5), use (ii) to eliminate P 13;1(x), simplify the result to obtain P (1 x)(1 x 13;13(x)= x )+x : (1 x)(1 x x ) Therefore, (4) holds for d = 3. Moreover, if (4) holds for d then it is routine using (5) to verify that (4) holds for d +1. Corollary.5. For all n 1, P n(13; 13) = F n+ 1 (6) P n(13; 134) =1 8 5 n +1 Fn+1 + (F n+3 + F n+1): (7) 5 Proof. To prove (6), rst set d = 3 in(4) simplify the result to obtain P 1 13;13(x)=1 1 x + x +1 1 x x : Compare this last line with (1) to obtain(6). To prove (7), rst set d = 4 in(4) simplify the result to obtain P 1 13;134(x)=1+ 1 x 1 x x + 1 (1 x x ) : It is routine to verify that 1 (1 x x ) = (( 1 5 n + 1 ) ( ) ) 3 F n n +1 F n+1 x n ; (7) follows. Next, we consider P 13;d1:::(d 1) (x). Theorem.6. (i) P 13;1(x)=1=(1 x). (ii) P 13;31(x) = ((1 x x )(1 x)+x(1 + x))=(1 x). (iii) For all d 4, P 13;d1:::(d 1) (x)= (1 x)(1 x x ) d +(1 x ) d 4 r=0 (1 x x ) r+1 x d 3 r + x d (1 + x) (1 x) (1 x x ) d 3 : (8) Proof. (i) Observe that for all n 0, the only permutation in S n which avoids 1 is 1; ;:::;n 1;n. (ii), (iii) Using Proposition.1, we nd that for all d 3, P 13;d1:::(d 1) (x)=1+x +x + x(p 13;1:::(d 1) (x) 1 x)+x (P 13;1:::(d 1) (x) 1) + x(p 13;d1:::(d 1) (x) 1 x): Solve this equationfor P 13;d1:::(d 1) (x) use Theorems.4(ii).4(iii) to eliminate the factor P 13;1:::(d 1) (x), obtaining (8).

5 76 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) 7 83 Corollary.7. We have P n(13; 31) =n (n ) (9) P n(13; 413) = F n+4 n (n 0): (10) Proof. To prove (9), rst observe from Theorem.6(ii) that P 13;31(x)=3+x + (1 x) 4 1 x : Now (9) follows from the binomial theorem. The proof of (10) is similar to the proof of (9). We conclude this section by describing a recursive method for computing P 13;(x) for any permutation P k (13). Observe that this allows us to compute P 13;(x) for any permutation S k, since P 13;(x) =P 13(x) if S k P k (13). Theorem.8. Fix k 3 P k (13). Observe by Proposition.1 that exactly one of the following holds: (a) There exists P k 1 (13) such that = k;. (b) There exists P k (13) such that = k 1;k;. (c) There exists P k 1 (13) such that = ;k. Then the following also hold: (i) If (a) holds then P 1 x 13;(x)= x + x(1 + x)p 13; (x) : (11) 1 x (ii) If (b) holds then P 1 x 13;(x)= x + x P 13; (x) : (1) 1 x (iii) If (c) holds then P xp 13; (x) 13;(x)=1+ 1 x x : (13) Proof. The proofs of (i) (iii) are similar to the proofs of Theorems.4(iii).6(iii). Corollary.9. We have P n(13; 341) =3 n 1 (n ); (14) P n(13; 4513) =3 n 1 F n+3 +1 (n 1) (15) P n(13; 56134) =3 n n +1 Fn+1 (F n+4 + F n+) (n 1): (16) 5 Proof. To prove (14), rst use Theorems.8(ii).4(ii) to nd that P 5 13;341(x)= x 1 1 x x : Now (14) is immediate. The proofs of (15) (16) are similar to the proof of (14).

6 Proposition.10. For all d, E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) P 13;d:::1 (x)= (1 x x ) d 3 i=0 x i (1 + x) i (1 x) d i + x d (1 + x) d (1 x) d 1 : (17) Proof. We argue by induction on d. When d =, line (17) reduces to Theorem.6(ii). If (17) holds for a given d then it is routine using Theorem.8(i) to show that (17) holds for d +1. Corollary.11. We have P n(13; 31) =n (n ); (18) P n(13; 431) =n 8n +11 (n 3); (19) P n(13; 5431) = 4 3 n3 1n n 5 (n 4): (0) Proof. To prove (18), rst set d = 3 in(17) simplify the result to nd P 13;31(x)=3+x 4 1 x + (1 x) : Now (18) follows from the binomial theorem. The proofs of (19) (0) are similar to the proof of (18). 3. Two-stack sortable permutations which avoid 13 contain another pattern Fix d 1 set P(13) = n 0 P n(13): Inspired by results such as those found in [11], in this section we study the generating function for permutations in P(13) according to the number of patterns of type 1 :::d, d 1 :::d 1, or dd 1 :::1 they contain. We begin by setting some notation. Denition 3.1. For any permutation P k (13) any r 1, let b n;r which contain exactly r subsequences of type. Thenwe write B(x)= r b n;rx n : denote the number of permutations in P n(13) (1) We now consider the case in which =1:::d. Denition 3.. For any permutation any d 1, we write 1 :::d() to denote the number of subsequences of type 1 :::d in we write to denote the length of. Theorem 3.3. Let x 1;x ;::: denote indeterminates. Then we have ( n ) x 1:::d() d =1+ j 1 x j j ( P(13) d 1 n 1 1 ( m 1 ) j 1 x j 1 ( m 1 ) j 1 x j 1 x n m=1 j j ( m 1 j 1 j+1 )) : () Proof. For notational convenience, set A(x 1;x ;:::)= x 1:::d() d : P(13) d 1

7 78 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) 7 83 Using Proposition.1 we nd that A(x 1;x ;:::)=1+x 1 + x1x + x1 + x 1(A(x 1;x ;:::) 1 x 1) + x1x (A(x 1;x ;:::) 1) + x 1(A(x 1x ;x x 3;:::) 1 x 1x ): Solve this equationfor A(x 1;x ;:::) to obtain A(x x1a(x1x;xx3;:::) 1;x ;:::)=1+ : 1 x 1 x1 x Iterate this recurrence relationto obtain(). For a given d 1, we canuse () to obtain the generating function for P(13) according to length number of subsequences of type 1 :::d. Proposition 3.4. For all d 3, B1:::d(x)= 1 x d (1 x) (1 x x ) : d (3) Proof. In(), set x 1 = x, x d = y, x i = 1 for i 1;d. Exp the resulting expression in powers of y; we wish to nd the coecient of y. Observe that only the terms in the sum for which n = d 1orn = d contribute to this coecient. These terms are x d 1 (1 x x ) d (1 x x y) x d y (1 x x ) d (1 x x y)(1 xy x y ) ; respectively, (3) follows. Corollary3.5. For all n 0, the number of permutations in P n(13) which contain exactly one subsequence of type 13 is F n+ n 1. Proof. Set d = 3 in(3) simplify the result to nd B13(x)= 1 x +1 1 x x 1 (1 x) : Now the result follows from (1) the binomial theorem. As another application of (), we now nd the generating function for P(13) according to length number of right to left maxima. To do this, we rst nd the number of right to left maxima in a given permutation interms of 1 :::d(). Denition 3.6. For any S n, we write r max() to denote the number of right to left maxima in. Proposition 3.7. For all S n(13), we have n r max()= ( 1) d+1 1 :::d(): d=1 (4) Proof. Fix S n(13) x i, 16 i 6 n. We consider the contribution of those increasing subsequences of which beginat (i) to the sum onthe right side of (4). If (i) is a right to left maxima, then the only increasing subsequence which begins at (i) has length one. Therefore each right to left maxima contributes one to the sum. If (i) is not a right to left maxima thenwe observe that because avoids 13, the elements to the right of (i) which are larger than (i) are in increasing order. Therefore, the contribution of (i) to the sum is ( ) k k ( 1) i =0; i i=0

8 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) where k is the number of elements in larger than (i) to the right of (i). Combine these observations to obtain(4). Proposition 3.8. We have x y r max() xy(1 x x ) =1+ (1 xy x y)(1 x x ) : (5) P(13) Proof. In(), set x 1 = xy x i = y ( 1)i+1 ( ) n n ( 1) i = n1 i i=0 n=1 x n (1 x x ) = x(1 x x ) n 1 1 x x to simplify the result. for i. Use the facts Corollary3.9. For all r 1, let a r;n denote the number of permutations in P n(13) with exactly r right to left maxima. Then a r;nx n = xr (1 + x) r 1 (1 x x ) : 1 x x (6) In particular, a 1;n = p n 1 (n ); (7) a ;n = p n + p n 3 (n 4) (8) a 3;n =p n 3 (n 6): (9) Proof. To obtain(6), rst observe that 1 1 xy x y = x n (x +1) n y n : Combine this with (5) to nd that x y r max() = x(1 x x ) x n y n+1 (1 + x) n : 1 x x P(13) Take the coecient of y r inthis last line to obtain(6). To obtain(7), set r = 1 in(6) compare the result with (). The proofs of (8) (9) are similar to the proof of (7). We remark that (7) an d(8) canalso be obtained directly from Proposition.1 Theorem.. For instance, if P n(13) has exactly one right to left maxima then it must end in n. By Proposition.1(iii) Theorem., there are exactly p n 1 such permutations in P n(13), (7) follows. A similar but slightly more involved argument proves (8). Next we nd B 1 d1:::d 1(x). Theorem (i) B31(x)= 1 x 3 (1 x) : (30) (ii) For all d 4, Bd1:::d 1(x)= 1 x d : (31) (1 x) 3 (1 x x ) d 4

9 80 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) 7 83 Proof. (i) Fix P n(13) such that contains exactly one subsequence of type 31. We consider the three cases of Proposition.1. If(1) = n then n 3 = n; n ;n 1;n 3;n 4;:::;; 1. If () = n (1) = n 1 then does not contain exactly one subsequence of type 31. If = ;n then P n 1(13) contains exactly one subsequence of type 31. Combine these observations to nd B31(x)= 1 x3 (1 x) + xb1 31(x): Solve this equationfor B31(x) 1 to obtain(30). (ii) Fix P n(13) such that contains exactly one subsequence of type d1 :::(d 1). We consider the three cases of Proposition.1. If(1) = n then (n)=n 1, since avoids 13, 341, contains exactly one subsequence of type d1 :::(d 1). Moreover, it is routine to show that the map n; ;n 1 is a bijectionbetweenthose permutations in P n (13) which contain exactly one subsequence of type 1 :::(d ) those permutations in P n(13) which contain exactly one subsequence of type d1 :::(d 1). If () = n (1) = n 1 then does not contain exactly one subsequence of type d1 :::(d 1). If (n) =n then = ;n, where P n 1(13) contains exactly one subsequence of type d1 :::(d 1). Combine these observations to nd B 1 d1:::(d 1)(x)=x B 1 1:::(d )(x)+xb 1 d1:::(d 1)(x): Now use (3) to eliminate B 1 1:::(d )(x) solve for B 1 d1:::(d 1)(x) to obtain(31). Corollary3.11. For all n 1, the number of permutations in P n(13) which contain exactly one subsequence of type 5134 is ( ) n +1 F n+ : Proof. Set d = 5 in(31) simplify the result to nd B5134(x)= 1 x +1 1 x x 3 1 x + (1 x) 1 (1 x) : 3 Now the result follows from (1) the binomial theorem. We conclude this section by nding B 1 d:::1(x). Theorem 3.1. For all d 1, B 1 d:::1(x)= x d (1 x) d 1 : (3) Proof. This is similar to the proof of Theorem Two-stack sortable permutations which contain 13 exactlyonce We now turn our attention to the set of permutations in P n which contain 13 exactly once. Denition 4.1. For all n 0, we write Q n to denote the set of permutations in P n which contain exactly one subsequence of type 13. Using the methods of the previous section, one can obtain generating functions for those permutations in Q n which avoid, or contain exactly once, any permutation S k. In this section, we illustrate these derivations with examples in which the resulting generating functions are given in terms of the generating functions for the Pell or Fibonacci numbers. We beginwith ananalogue of Proposition.1. Proposition 4.. Fix n 3 suppose Q n. Then the following hold. (i) (1) = n, () = n, (n)=n, or = 314.

10 (ii) The map from Q n 1 to Q n given by n; E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) is a bijection between Q n 1 the set of permutations in Q n which begin with n. (iii) The map from Q n to Q n given by n 1;n; is a bijection between Q n the set of permutations in Q n whose second entry is n in which n is not an element of the subsequence of type 13. (iv) The map from Q n 1 to Q n given by ; n is a bijection between Q n 1 the set of permutations in Q n which end with n. (v) The map from P n to Q n given by n ;n; ; where is the permutation of 1; ;:::;n 3;n 1 obtained by replacing n with n 1 in, is a bijection between P n the set of permutations in Q n whose second entry is n in which n is an element of the subsequence of type 13. Proof. (i) Suppose 1 (n) n. We consider two cases: n is an element of the subsequence of type 13 or n is not an element of the subsequence of type 13. If n is not an element of the subsequence of type 13 then the elements to the left of n are all greater thanevery element to the right of n. Since there are at least two elements to the left of n at least one element to the right of n, there must be a patternof type 341 or 341 inwhich n plays the role of the 4. This contradicts our assumption that Q n P n: Now suppose the subsequence of type 13 in is a; n; b. Since contains no other subsequence of type 13, all elements to the left of n other than a are greater than b all elements to the right of n other than b are less than a. Observe that if there are additional elements both to the right left of n then one of these elements on each side of n, together with a n, form a pattern of type 341 or 341. Therefore, there can only be additional elements on one side of n. Since we have assumed 1 (n) n, there must be at least one additional element to the left of n no additional elements to the right of n. If there are two (or more) additional elements to the left of n thenthey combine with n b to form a 341 or a 341 pattern, so there must be exactly one additional element to the left of n. Now the only possibilities are = 314 = 134. But 134 has more than one subsequence of type 13, so we must have = 314, as desired. (ii) (v) These are similar to the proof of Proposition.1(ii). Using Proposition 4., we now nd the generating function for Q n. Theorem 4.3. We have Q n x n = x3 (1 + x x x 3 ) : (1 x x ) Moreover, (33) Q n = 1 (51npn 145pn 1npn+1 +60pn+1) (n 3): (34) 4 Proof. For notational convenience, set Q(x)= Q n x n : To obtain(33), observe that by Proposition 4. we have Q(x)=xQ(x)+x Q(x)+x P 13(x)+x 4 : Use (3) () to eliminate P 13(x) solve the resulting equation for Q(x) to obtain(33).

11 8 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) 7 83 To obtain(34), rst observe that Q(x)= x 87x 36 49x 51 +4x 15 + (1 x x ) 1 x x : Now observe that 1 (1 x x ) = 1 4 ((n +1)pn +(3n +4)pn+1)xn : Combine these observations with () to obtain(34). Proposition 4. enables us to nd the generating function for those permutations in Q n which avoid, or contain exactly once, any permutation. We illustrate how this is done by considering those permutations in which = 1:::d appears exactly once. Denition 4.4. For any permutation S k, let c n denote the number of permutations in Q n which contain exactly one subsequence of type. We write D(x) 1 to denote the generating function given by D(x)= 1 c nx n : Theorem 4.5. For all d, D 1 1:::d(x)= (d )x d+ (1 x) (1 x x ) d 1 : (35) Proof. The case d = is immediate, so we assume d argue by induction on d. By Proposition 4., wend D1:::d(x)=xD 1 1:::d(x)+x 1 D1:::d(x)+xD 1 1:::(d 1)(x)+x 1 B1:::d(x): 1 Use (3) to eliminate B1:::d(x) 1 solve the resulting equation for D1:::d(x), 1 obtaining ( ) D1:::d(x)= 1 xd1:::(d 1)(x)+ 1 : 1 1 x x Use induction to eliminate D 1 1:::(d 1)(x) (35) follows. x d+ (1 x) (1 x x ) d Corollary4.6. The number of permutations in Q n which contain exactly one subsequence of type 13 is n 5 (Fn+1 + Fn 1) (5Fn+1 Fn 1)+n + (n 4): 5 Proof. This is similar to the proof of (7). 5. Directions for future work In this section we present several directions in which this work may be generalized, using similar techniques. 1. InSection we enumerated two-stack sortable permutations which avoid 13 one additional pattern. Using the same techniques, one ought to be able to enumerate two-stack sortable permutations which avoid 13 two or more additional patterns. For instance, for any d, it should be possible using our techniques to nd the generating function P 13;13:::d;13:::d (x). Moreover, we expect that this generating function will be given in terms of the generating function for the Fibonacci numbers.. InSection3 we found generating functions for two-stack sortable permutations with respect to various statistics on permutations. Using similar techniques, one ought to be able to nd generating functions for two-stack sortable permutations with respect to additional statistics, such as rises, descents, right to left maxima, right to left minima, left to right maxima, left to right minima. 3. InSection4 we enumerated two-stack sortable permutations which contain exactly one subsequence of type 13 avoid (or contain exactly once) another pattern. One ought to be able to use similar techniques to enumerate permutations which contain exactly r subsequences of type 13, for a given r.

12 E.S. Egge, T. Mansour / Discrete Applied Mathematics 143 (004) Acknowledgements The authors thank the anonymous referee who provided several helpful comments suggestions. References [1] G. Alexerson, Elementary problems solutions, problem B-10, Fibonacci Quart. 4 (1966) 373. [] D. Beck, The combinatorics of symmetric functions permutation enumeration of the hyperoctahedral group, Discrete Math. 163 (1997) [3] A.H. Beiler, Recreations in the Theory of Numbers, nd Edition, Dover Publications Inc., New York, [4] S. Billey, Pattern avoidance rational smoothness of Schubert varieties, Adv. Math. 139 (1998) [5] S. Billey, W. Jockusch, R. Stanley, Some combinatorial properties of Schubert polynomials, J. Algebraic Combin. (4) (1993) [6] S. Billey, V. Lakshmibai, On the singular locus of a Schubert variety, J. Ramanujan Math. Soc. 15 (3) (000) [7] S. Billey, T. Lam, Vexillary elements in the hyperoctahedral group, J. Algebraic Combin. 8 () (1998) [8] S. Billey, G. Warrington, Kazhdan-lusztig polynomials for 31-hexagon-avoiding permutations, J. Algebraic Combin. 13 () (001) [9] M. Bona, Symmetry unimodality in t-stack sortable permutations, J. Combin. Theory. Ser. A 98 (1) (00) [10] M. Bousquet-Melou, Multi-statistic enumeration of two-stack sortable permutations, Electron. J. Combin. 5 (1) (1998) R1. [11] P. Bren, A. Claesson, E. Steingrmsson, Catalan continued fractions increasing subsequences in permutations, Discrete Math. 58 (00) [1] T. Chow, J. West, Forbidden subsequences Chebyshev polynomials, Discrete Math. 04 (1 3) (1999) [13] S. Dulucq, S. Gire, O. Guibert, A combinatorial proof of J. West s conjecture, Discrete Math. 187 (1998) [14] E.I. Emerson, Recurrent sequences in the Pell equations, Fibonacci Quart. 7 (1969) [15] A.F. Horadam, Pell identities, Fibonacci Quart. 9 (1971) 45 5, 63. [16] C. Krattenthaler, Permutations with restricted patterns Dyck paths, Adv. Appl. Math. 7 (/3) (001) [17] V. Lakshmibai, M. Song, A criterion for smoothness of Schubert varieties in Sp n =B, J. Algebra 189 () (1997) [18] T. Mansour, A. Vainshtein, Avoiding maximal parabolic subgroups of S k, Discrete Math. Theor. Comput. Sci. 4 (1) (000) [19] T. Mansour, A. Vainshtein, Restricted 13-avoiding permutations, Adv. Appl. Math. 6 (3) (001) [0] R. Tarjan, Sorting using networks of queues stacks, J. Assoc. Comput. Mach. 19 (197) [1] J. West, Permutations with forbidden subsequences stack-sortable permutations, Ph.D. Thesis, M. I. T., [] J. West, Sorting twice through a stack, Theoret. Comput. Sci. 117 (1993) [3] D. Zeilberger, A proof of Julian West s conjecture that the number of two-stack sortable permutations of length n is (3n)!=((n + 1)!(n + 1)!), Discrete Math. 10 (199)