The Mean and Variance of the Numbers of r-pronged Nodes and r-caterpillars in Yule-Generated Genealogical

Transcription

1 Annals of Combinatoics 10 ( /06/ DOI /s c Bikhäuse Velag, Basel, 2006 Annals of Combinatoics The Mean and Vaiance of the Numbes of -Ponged Nodes and -Catepillas in Yule-Geneated Genealogical Tees Noah A. Rosenbeg Depatment of Human Genetics and Bioinfomatics Pogam, Univesity of Michigan, 2017 Palme Commons, 100 Washtenaw Ave, Ann Abo, MI , USA noah@umich.edu Received August 23, 2004 AMS Subject Classification: 05C05, 92D15 Abstact. The Yule model is a fequently-used evolutionay model that can be utilized to geneate andom genealogical tees. Unde this model, using a backwads counting method diffeing fom the appoach peviously employed by Head (Evolution 46: , fo a genealogical tee of n lineages, the mean numbe of nodes with exactly descendants is computed (2 n 1. The vaiance of the numbe of -ponged nodes is also obtained, as ae the mean and vaiance of the numbe of -catepillas. These esults genealize computations of McKenzie and Steel fo the case of 2 (Math. Biosci. 164: 81-92, Fo a given n, the two means ae lagest at 2, equaling 2n/3 fo n 5. Howeve, fo n 9, the vaiances ae lagest at 3, equaling 23n/420 fo n 7. As n, the faction of intenal nodes that ae -catepillas fo some appoaches (e 2 5/ Keywods: binay seach tee, cheies, coalescent, genealogy, labeled topology, pectinate 1. Intoduction In many contexts in evolutionay biology, it is of inteest to investigate the pobability distibutions of vaious attibutes of genealogical tees. Fist, pedictions about these distibutions can be made using models of the evolutionay pocess that poduces the tees. These pedictions can then be helpful in undestanding which scenaios ae possible o likely outcomes of evolution [2,5,16,22,26,29]. By compaison with estimates made fom biological data, they can also povide insight into the natue of the pocesses that geneate the data [9, 19, 20, 23, 34]. Pehaps the simplest evolutionay model fom which pedictions about genealogical tees can be made is the Yule o Yule-Hading model [4, 12, 31, 35, 36, 39]. Unde this model, beginning with an ancestal lineage, the genealogical tee fo n lineages is fomed by successive binay banching events, so that at any point in time, all lineages have equal pobability of being the next to banch into two. Equivalently, looking 129

2 130 N.A. Rosenbeg backwads fom n lineages in the pesent, at any time point, all pais of lineages have equal pobability of being the next to coalesce into one. This etospective viewpoint is typically adopted in population genetics, whee the Yule model is used with each lineage coesponding to a distinct copy of a paticula genetic site (taken fom a set of such copies in a population of individuals. In this context, when combined with a specific model fo the times at which coalescences occu, the Yule model is temed the coalescent model [14, 15, 24, 38]. Hee, using a backwads counting appoach, we extend known popeties of genealogical tees unde the Yule model to obtain the mean and vaiance of the numbe of nodes with exactly descendants among the n lineages (2 n 1. These quantities then enable computation of the mean and vaiance of the numbe of -catepillas in genealogical tees. 2. Definitions The definitions used hee ae lagely based on those of Semple and Steel [32]; Figue 1 illustates many of the key concepts. A genealogical tee o genealogy G (G, X, ψ, t fo n leaves is a ooted binay tee G fo which (1 ψ is a bijection that associates each leaf of G with a label in a label set X, and (2 t is a map that associates each point p of G (that is, each vetex and each point lying on an edge with a nonnegative eal numbe t(p, such that (i fo any two distinct points p 1, p 2, if the path fom p 1 to the oot of G includes p 2, then t(p 1 < t(p 2 ; (ii fo any two distinct inteio vetices v 1, v 2, t(v 1 t(v 2 ; (iii fo a point p, t(p 0 if and only if p is a leaf of G. If the path fom a point p 1 to the oot includes p 2, then p 1 is descended fom p 2, which, in tun, is ancestal to p 1. Tivially, a point is both ancestal to and descended fom itself (howeve, the point itself is not included when counting its numbe of descendants. Fo convenience the label set X is taken to be {x 1, x 2,..., x n }. Inteio vetices ae altenately temed intenal nodes (it is assumed in a genealogy that each intenal node has exactly two descendants. The value of t(p fo a point p is the time of p. Thus, the leaves of genealogies ae viewed as existing in the pesent, with time inceasing into the past. The lineage of a point p at time u t(p is the unique point that is both ancestal to p and has time u; this time is usually 0 in uses of the tem. The most ecent common ancesto (MRCA of a subtee G of G (o a subset X X is the node with the smallest time among the collection of nodes that ae ancestal to all elements of G (o to the collection of leaves with label set X. Let t(g be the set of values taken by t ove all intenal nodes of G. Let h be the unique bijection fom t(g into {1, 2,..., n 1} with the popety that fo any two vetices v 1, v 2, if t(v 1 < t(v 2, then h(v 1 < h(v 2. The coalescence sequence o labeled histoy of G is the sequence of patitions π 0, π 1,..., π n 1 of X such that π 0 (G {{x 1 }, {x 2 },..., {x n }} and fo i 1, 2,..., n 1, π i (G is fomed fom π i 1 (G by combining the two blocks in π i 1 (G containing leaves descended fom the vetex h 1 (i into the same block in π i (G. The labeled histoy of G epesents the sequence of events that educe the n leaves to thei MRCA; h 1 (i coesponds to the ith coalescence o coalescent event. The k-tuncated coalescence sequence o k-tuncated labeled histoy of G (1 k n is the sequence π 0, π 1,..., π n k.

3 -Ponged Nodes and -Catepillas in Genealogical Tees 131 i ii iii iv v vi Time Figue 1: Example genealogies with the label set {1, 2, 3, 4, 5, 6}. Genealogies (i (v all have the same unlabeled topology. In compaison with (i, (ii has the same unlabeled histoy and labeled topology, but a diffeent labeled histoy; (iii has the same labeled topology, but diffeent labeled and unlabeled histoies; (iv has the same unlabeled histoy, but a diffeent labeled histoy and labeled topology; (v has diffeent labeled and unlabeled histoies and a diffeent labeled topology. Genealogy (vi has diffeent labeled and unlabeled histoies and topologies fom the emaining genealogies. It is a 6-catepilla and is the only one of the genealogies to contain a pitchfok. Genealogies (i (v all contain a symmetic 4-ponged node; the 4-ponged node in (vi is not symmetic. Thee ae thee cheies in each of (i (v and one chey in (vi. The unlabeled histoy of G is a paticula sequence of patitions π 0, π 1,..., π n 1 of X obtained in the following manne. We begin with a set of available labels, A {1, 2,..., n}, and we equate B A and π 0 π 0. Sequentially, fo i 1, if one of the blocks in π i (G containing leaves descended fom h 1 (i includes only one leaf, we then eassign the label of that leaf as the label b with the smallest value among those in B, and eplace B with B \ {b} (if both blocks each include only one leaf, eassign the labels of both leaves the ode in which the two eassignments ae made is unimpotant. We epeat this pocedue until each leaf v has been assigned a label γ(v fom A. The unlabeled histoy of G is then the labeled histoy of G (G, A, γ, t. The labeled topology of G is the tee G with label set X and labeling γ, ignoing the time function t. The unlabeled topology of G is the tee G, ignoing the label set X, the labeling γ and the time t. The subgenealogy G v of G induced by an intenal node v is the genealogy (G v, X v, ψ G v, t G v, whee G v is the subtee of G containing all leaves that descend fom v, X v is the label set fo G v, and ψ G v and t G v denote the estictions of ψ and t to G v, espectively. An intenal node v of G o its induced subgenealogy is -ponged (2 n if the subgenealogy contains exactly descendants among the leaves of G; the node (o subgenealogy is an -catepilla if G v contains exactly descendants among the leaves of G, and if the intenal node with the smallest value of t among all intenal nodes in G v is descended fom all othe intenal nodes in G v. A genealogy whose oot is an -catepilla is temed pectinate. As special cases, 2-ponged and 3-ponged intenal nodes (o thei induced subgenealogies ae temed cheies and pitchfoks, espectively. Both cheies and pitchfoks ae necessaily catepillas. Fo a genealogy G, d (G is the numbe of -ponged nodes in G. An intenal node with moe than two descendants in X is symmetic if the two subgenealogies induced by its two immediate descendants have the same unlabeled topology. Fo a genealogy G, s(g is its numbe of symmetic intenal nodes. The functions d and s can be applied also to labeled o unlabeled histoies o topologies.

4 132 N.A. Rosenbeg 3. The Yule Model This section eviews popeties of genealogical tees geneated by the Yule model (Yulegeneated genealogies. A Yule-geneated genealogy has the popety that at any time, each pai of lineages is equally likely to be the next to coalesce (o equivalently, evesing the diection of time, each lineage is equally likely to be the next to divide. It follows diectly fom this popety that the pobability distibution of the labeled histoy of such a genealogy is unifom. In this section and in those that follow, G (G, X, ψ, t is teated as a andom n- leaved Yule-geneated genealogy, and genealogy implicitly efes to the Yulegeneated genealogy G. Hencefoth, a 1, a 2, a 3, i, k, n, n 1, n 2, n 3, and ae assumed to be positive integes with n 3, 2, and except whee othewise specified, i n 1 and k n. It is assumed in each of (3.1 (3.5 that T is chosen fom the appopiate set of objects to which the esult applies (fo example, in Theoem 3.3, the collection of possible labeled topologies fo the label set X. Except fo (3.3 and (3.4, which ae included fo completeness, the esults in this section ae used in poving the new esults in the following sections; othe than (3.8 and (3.9, they utilize the unifom distibution of labeled histoies fo Yule-geneated genealogies. Theoem 3.1. [4, 30] The pobability I(T that a genealogy fo n lineages has k- tuncated labeled histoy T is I(T 1 2n k k!(k 1!. I n,k n!(n 1! Coollay 3.2. [8,21] The pobability H(T that a genealogy fo n lineages has labeled histoy T is H(T 1 H n 2n 1 n!(n 1!. Theoem 3.3. [1, 4, 35] The pobability L(T that a genealogy fo n lineages has labeled topology T is 2 n 1 L(T n! n 3 ( 1d (T. Theoem 3.4. [25, 38] The pobability U(T that a genealogy fo n lineages has unlabeled histoy T is U(T 2n 1 d 2(T (n 1!. Theoem 3.5. [4, 36] The pobability Q(T that a genealogy fo n lineages has unlabeled topology T is Q(T 2n 1 d 2(T s(t n 3 ( 1d (T.

5 -Ponged Nodes and -Catepillas in Genealogical Tees 133 Coollay 3.6. [33, 37] The pobability Q(T that a genealogy fo n lineages has unlabeled topology T, whee T is pectinate, is Q(T 2n 2 (n 1!. Theoem 3.7. [34, 38] In a genealogy fo n lineages, the pobability C n that the two nodes immediately descended fom the oot have i and n i descendants is C 1n 1/(n 1, if i n/2 and n is even, C 2n 2/(n 1, if i n/2 and i {1, 2,..., n 1}. Theoem 3.8. [4, 30] The numbe W a1,a 2 of (n a 1 a 2 -tuncated coalescence sequences possible fo a fixed collection of n n 1 +n 2 labels (with n 1 a 1, n 2 a 2, such that each coalescence sequence contains two specified subsequences, one of length a 1 that coalesces n 1 lineages to n 1 a 1 and anothe of length a 2 that coalesces the emaining n 2 lineages to n 2 a 2, is the binomial coefficient ( a1 + a 2 W a1,a 2. Theoem 3.9. [30] The numbe W a1,a 2,a 3 of (n a 1 a 2 a 3 -tuncated coalescence sequences possible fo a fixed collection of n n 1 + n 2 + n 3 labels (with n 1 a 1, n 2 a 2, n 3 a 3, such that each coalescence sequence contains thee specified subsequences, one of length a 1 that coalesces n 1 lineages to n 1 a 1, one of length a 2 that coalesces n 2 lineages to n 2 a 2, and the thid of length a 3 that coalesces the emaining n 3 lineages to n 3 a 3, is the tinomial coefficient ( a1 + a 2 + a 3 W a1,a 2,a 3. a 1, a 2, a 3 Remak A coespondence exists between Yule-geneated genealogies with n leaves and the entities geneated duing constuction of andom binay seach tees with n 1 vetices (leaves plus intenal nodes. A andom binay seach tee is obtained via sequential addition of descendant vetices to a ooted binay tee so that each vetex with one descendant has anothe slot in which a descendant can be added, and each vetex with no descendants has two slots in which descendants can be added (inteio vetices in binay seach tees ae allowed to have eithe 1 o 2 descendants. At any time (evesing the diection of time, all potential slots fo insetion of vetices ae equally likely to be the next to have a vetex added [17, pp ]. The coesponding binay seach tee fo a given Yule-geneated genealogy G (G, X, ψ, t is the pai (G b, t G b, whee G b is the subtee of G containing only the intenal nodes of G and the edges that connect them, and t G b is the estiction of t to G b. 4. The Numbe of -Ponged Nodes Afte poving thee combinatoial identities as Lemmas 4.1, 4.2, and 4.3, the mean and vaiance of the numbe of -ponged nodes in a Yule-geneated genealogy ae obtained a 1

6 134 N.A. Rosenbeg in Theoem 4.4, and the popeties of the vaiance function ae then examined in Theoem 4.8. The mean had peviously been obtained in [13] using a Polya un method, and the mean and vaiance wee both calculated fo the case of 2 in [18]. Related poblems have also been consideed in [10]. Because of the coespondence between Yule-geneated genealogies and andom binay seach tees, many esults deived in the context of binay seach tees [6, 7, 17] can be intepeted as statements about Yule-geneated genealogies. Fo example, using the fact that a node with descendant nodes in a binay seach tee has +2 descendants among the leaves of the coesponding Yule-geneated genealogy, Theoem 4.4 (i and the < n/2 case of Theoem 4.4 (ii ae demonstated (via diffeent poofs fom the ones hee in [6, Theoem 5], which futhe obtains a limiting distibution in n fo the numbe of -ponged nodes (see also [3]. Lemma 4.1. Fo positive integes n and with n 3, 2, n, ( ( ( k + 1 n k 2 n n k1 Poof. The identity follows fom [11, Identity 3.3], a statement staightfowad to pove using induction on m, m s lq ( l q ( m l s ( m + 1, q + s + 1 substituting k + 1 fo l, 2 fo q, 2 fo s, and n 1 fo m. Lemma 4.2. Fo positive integes n,, and i, with n 2 1, i, 2, n 2 j1 ( ( ( ( j + 1 i + j + 1 n 2 i j n i j 2 j 2 i (i + 2[(i + 3n (i + 12 (i 1] 2(n 2 1 ( ( 2 i 2 n. ( Poof. Set w n 2 and denote the atio of the summand in Equation 4.1 to the ight-hand side by F(w, j: F(w, j ( j+1 ( i+ j+1 ( 2+w 2 i j ( +w i j 2 j 2 i ( 2 i 2 ( 2+w Togethe with the poof cetificate R(w, j 2(2 + w + 1 (i + 2[4 + (i + 3w (i 1]. ( j 1(2 + w 1 i j(4 + 2w 2i + 2 j + 4 j + 3 jw + i jw, ( j + 1(w + 1 j(2 + w + 1(4 + 3w iw F(w, j satisfies the hypotheses of the Wilf-Zeilbege automatic summation theoem [27, Theoem 7.1.1], fom which it follows that w j1 F(w, j is not dependent on w. Because F(w, j 1 when w 1, the esult follows. w j1

7 -Ponged Nodes and -Catepillas in Genealogical Tees 135 Lemma 4.3. Fo positive integes n and with n 3, 2, n, ( (i + 2[(i + 3n (i + 12 (i 1] 2 i 2 2(n i n + 8n 2 4( + 1(2 1(2 + 1 ( n ( ( n. ( Poof. Adding tems fo i 0 and i 1 to both sides of Equation 4.2 and setting u i, the statement we wish to pove is equivalent to ( ( ( u + c u + c u + c α 1 u 2 + α 2 u + α ( + 5n + 7n , u0 u u u whee c 2, α 1 n 2 1, α n 2n +1, and α 3 ( +2( n + n + 1. But this identity follows by setting c 2 in the following thee identities concening nonnegative integes and c: u0 u u0 ( u + c u ( u + c u ( u + c u 2 u0 u ( + c + 1 (c + 1 c + 2 ( + c + 1 (c + 1(2 + c + 1 (c + 2(c + 3 ( + c + 1 Each of equations is staightfowad to pove by induction on. Theoem 4.4. In a genealogy fo n 3 lineages, if 2 n 1, (i [13] the mean numbe of -ponged nodes, M(n,, is 2n ( + 1, (ii the vaiance of the numbe of -ponged nodes, V(n,, is (4.3 (4.4. (4.5 V 1 (n, 2(42 3 4( 1n ( + 1 2, if < n/2, (4.6 (2 1(2 + 1 V 2 (n, (5 7( 1n ( + 1 2, if n/2, (4.7 (2 1 V 3 (n, 2(2 + 2nn 2 ( + 1 2, if > n/2. (4.8 Poof. Enumeate the ( n subsets of the label set that contain elements, denoting the bth such subset by S b. Let Z b be the indicato vaiable fo whethe thee is some intenal

8 136 N.A. Rosenbeg node of the genealogy fo which S b is the label set of its induced subgenealogy. The numbe of -ponged nodes in the genealogy is Z ( n b1 Z b. (i The mean numbe of -ponged nodes in the genealogy is M(n, E[Z] ( n b1 E[Z b ] ( n E[Z 1 ] ( n P[Z 1 1]. P[Z 1 1] can be detemined by counting the faction of coalescence sequences fo which S 1 has MRCA at an -ponged node (Figue 2. In such a sequence, at the time just less than that of the -ponged node, the lineages with labels in S 1 have coalesced to 2 lineages and the n lineages with labels in X \ S 1 have coalesced to k lineages (1 k n. The next coalescence event is the -ponged node. The emaining k +1 lineages coalesce to the MRCA fo X. Thus, applying Theoem 3.1, Coollay 3.2, and Theoem 3.8, the numbe of coalescence sequences in which S 1 has MRCA at an -ponged node is k1 n I,2I n,k W 2,n k H k+1. Consequently, using Lemma 4.1, M(n, ( n 1 n H n k1 2(n 1!( 1! (n 1! I,2 I n,k W 2,n k H k+1 n k1 ( ( k + 1 n k n ( + 1. (4.9 (ii The vaiance of the numbe of -ponged nodes is [( ( n V(n, E [ ( n E 2 ] Z b E b1 [ ( n ] Z b E b1 [ ( n ] 2 Z b b1 ] 2 Z b + E b1 [ ( n Z b Z b b,b b b [ ( n ] M(n, M(n, 2 + E Z b Z b. (4.10 b,b b b Case 1. < n/2. If S b S b φ, then E[Z b Z b ] 0. Fo all disjoint S b and S b, E[Z b Z b ] P[Z b Z b 1] has the same value. The numbe of odeed pais (S b, S b fo which S b, S b X and S b S b φ is 2 ( ( n n. Theefoe, supposing that (S1, S 2 is such a pai, [ ( n ] ( ( n n E Z b Z b 2 P[Z 1 Z 2 1]. (4.11 b,b b b ]

9 -Ponged Nodes and -Catepillas in Genealogical Tees k Time n- S X \ S 1 1 Figue 2: Counting the faction of coalescence sequences fo which S 1 has MRCA at an -ponged node. Using Theoem 3.1, the numbe of ways that lineages can coalesce to 2 lineages is I,2, and fo 1 k n, the numbe of ways that n lineages can coalesce to k lineages is I n,k. By Theoem 3.8, the numbe of ways of inteweaving these sequences of 2 and n k coalescences is W 2,n k. By Coollay 3.2, the numbe of ways that k + 1 lineages can coalesce to 1 lineage is H k+1. Consequently, the total numbe of coalescence sequences fo which S 1 has MRCA at an -ponged node is k1 n I,2I n,k W 2,n k H k+1. By Coollay 3.2, the total numbe of coalescence sequences fo n lineages is H n. Thus, the desied quantity is (1/H n k1 n I,2I n,k W 2,n k H k+1. 2 l i 2 j Time n-2 S S X \ {S U S } Figue 3: Counting the faction of coalescence sequences fo which both S 1 and S 2 ae label sets fo subgenealogies, with the time of the MRCA of S 1 smalle than that of S 2. Using Theoem 3.1, the numbe of ways that lineages can coalesce to 2 lineages is I,2 ; fo 2 i, the numbe of ways that lineages can coalesce to i lineages is I,i ; fo 1 j n 2, the numbe of ways that n 2 lineages can coalesce to j lineages is I n 2, j. By Theoem 3.9, the numbe of ways of inteweaving these sequences of 2, i and n 2 j coalescences is W 2, i,n 2 j. Counting the numbe of ways that the emaining lineages can coalesce so that S 2 is the label set fo an induced subgenealogy follows the same agument as in Figue 2.

10 138 N.A. Rosenbeg In ode to have Z 1 Z 2 1, both S 1 and S 2 must be label sets fo subgenealogies. Without loss of geneality, suppose that the time of the MRCA of S 1 is less than that of S 2. P[Z 1 Z 2 1] can be detemined by counting the faction of coalescence sequences fo which S 1 and S 2 ae the label sets fo induced subgenealogies (Figue 3. At the time just less than that of the MRCA of S 1, the lineages with labels in S 2 have coalesced to i lineages (2 i, and the n 2 lineages with labels in X \ {S 1 S 2 } have coalesced to j lineages (1 j n 2. The next coalescence poduces the -ponged node fo which S 1 is the label set of the induced subgenealogy. At the time just less than that of the MRCA of S 2, the emaining i lineages ancestal to S 2 have coalesced to 2 lineages, and the emaining j + 1 lineages ancestal to X \ S 2 have coalesced to l lineages (1 l j. The next coalescence poduces the -ponged node fo which S 2 is the induced subgenealogy. Finally, the emaining l + 1 lineages ancestal to X coalesce to 1 lineage. The total numbe of possible coalescence sequences fo all n lineages with labels in X is H n. Using Theoem 3.1, Coollay 3.2 and Theoems 3.8 and 3.9, P[Z 1 Z 2 1] i2 n 2 j1 Simplifying this expession and using Equation 4.11, [ ( n ] E Z b Z b b,b b b 8( 1! 2 (n 2 1! i2 j+1 I,2 I,i I n 2, j W 2, i,n 2 j I i,2 I j+1,l W i 2, j+1 l H l+1 l1. H n n 2 j1 j+1 l1 ( j+1 2 (n 1! ( l+1 2 ( i+ j l 1 i 2 ( n 2 i j 2 ( n i j i Sequentially applying Lemmas 4.1, 4.2, and 4.3 to sum ove indices l, j and i, [ ( n ] E Z b Z b 2( n + 8n 2 n b,b 2 ( (2 1(2 + 1, b b and the esult follows fom Equation Case 2. n/2. In this case, a genealogy can have eithe zeo o two -ponged nodes, so that Z b 1 fo eithe zeo o two values of b. Consequently, the sum in Equation 4.10 can have eithe no tems equal to 1, o exactly two tems equal to 1: [ ( n ] [ ( n ] E Z b Z b 2P Z b Z b 2. b,b b b,b b b b A genealogy can have two distinct sets each with n/2 labels and each with MRCA at an -ponged node if and only if the two sets ae disjoint and both -ponged nodes.

11 -Ponged Nodes and -Catepillas in Genealogical Tees 139 Vaiance of the numbe of -ponged nodes n Numbe of descendants ( Figue 4: The vaiance V(n, of the numbe of -ponged nodes fo n 50. ae immediately descended fom the oot. By Theoem 3.7, the pobability of having two such nodes is 1/(n 1. Theefoe, using Equation 4.10, V(n, and the esult follows using n 2. 2n ( + 1 ( + 1 2n + 2 ( + 1 n 1, Case 3. > n/2. In this case, at most one set of labels can be the label set of an induced subgenealogy, so that fo any distinct b, b, Z b Z b 0. The esult then follows fom Equation Coollay 4.5. [18] In a genealogy fo n 5 lineages, the mean numbe of cheies is n/3 and the vaiance of the numbe of cheies is 2n/45. Coollay 4.6. In a genealogy fo n 7 lineages, the mean numbe of pitchfoks is n/6 and the vaiance of the numbe of pitchfoks is 23n/420. Remak 4.7. Fo a given n, the mean numbe of -ponged nodes is lagest fo 2, and declines monotonically as inceases. The vaiance, howeve, exhibits a moe complicated patten. Figue 4 displays a typical example, namely n 50. The global maximum of V(50, occus at 3 athe than at 2, and the decline afte 3 is inteupted by a small peak at 50/2 25. The following theoem summaizes the geneal shape of V (n, as a function of.

12 140 N.A. Rosenbeg Theoem 4.8. Fo a fixed n 15 as well as fo n 11, 13, as anges ove integes fom 2 to n 1, (i the fou highest values of V (n, ae, fom geatest to smallest, at 3, 4, 2, and 5; (ii fo 3 n 2, V(n, > V(n, + 1, unless n is even and + 1 n/2, in which case V(n, < V(n, + 1. Poof. We will need the following six inequalities, each of which is staightfowad to pove fom Equations using elementay methods. (a Fo integes n, with n 3 and 2 n 1, V 2 (n, > V 1 (n, > V 3 (n,. (b Fo integes n, with n 3 and 3 n 1, V 1 (n, > V 1 (n, + 1. (c Fo positive n, V 1 (n, 3 > V 1 (n, 4 > V 1 (n, 2 > V 1 (n, 5. (d Fo even integes n 12, V 1 (n, 5 > V 2 (n, n/2. (e Fo integes n, with n 10 and n/2 < n 1, V 3 (n, > V 3 (n, + 1. (f Fo even integes n 16, V 2 (n, n/2 > V 1 (n, n/2 1. Using (b and (c, V 1 has its fou highest values at 3, 4, 2, and 5, espectively. Fo n 11, V(n, 5 V 1 (n, 5. By (a and (b, V 1 (n, 5 > V 3 (n, fo all integes n, with n 11, n/2 < < n. Applying (d, it follows that fo n 11, (i holds. Using (a and (b, V 1 (n, n/2 1 >V 3 (n, n/2+1. Fo n 3, V(n, V 1 (n, fo 2 n/2 1 and V(n, V 3 (n, fo n/2 + 1 n 1. Applying (a, (b, and (e, (ii holds fo odd n > 10. Fo even n it must also be veified that V(n, < V(n, + 1 fo + 1 n/2; this follows fom (f, but only fo n 16. Remak 4.9. The theoem shows that fo lage enough n, the vaiance follows a paticula patten as a function of. Fo vey lage, the vaiance appoaches 2n/ 2, povided n emains lage than 2 (2.5n/ 2 if n 2. Fo lage n, stating with 2, V(n, /n follows the sequence 2/45, 23/420, 8/175, 2/55, 610/21021, 171/ At small n (n 14, howeve, the behavio of the vaiance function can diffe fom that specified by the theoem (Figue The Numbe of -Catepillas The mean and vaiance of the numbe of -ponged nodes can be used to obtain the mean and vaiance of the numbe of subgenealogies with any given unlabeled topology. Theoem 5.1 gives the geneal fomulas fo an abitay unlabeled topology T ; the case of a pectinate T is consideed in Coollay 5.2, and the popeties of the vaiance function ae then exploed in Theoem 5.4. Theoem 4.4, Theoem 5.1, and Coollay 5.2 can all be consideed genealizations of the fomulas of [18] fo the mean and vaiance of the numbe of cheies. Theoem 5.1. In a genealogy with n 3 lineages, if 2 n 1 and T is an unlabeled topology with leaves, (i the mean numbe of nodes that have induced subgenealogy T, M T (n,, is Q(T M(n,, and (ii the vaiance of the numbe of nodes that have induced subgenealogy T, V T (n,, is V T (n, Q(T [1 Q(T ]M(n, + Q(T 2 V(n,.

13 -Ponged Nodes and -Catepillas in Genealogical Tees 141 Vaiance of the numbe of -ponged nodes n16 n15 n14 n13 n12 n11 n10 n9 n8 n7 n6 n5 n Numbe of descendants ( Figue 5: The vaiance V(n, of the numbe of -ponged nodes fo n 4 though 16. Fo n 3, V(3, 2 0. Poof. Let Y b be the indicato vaiable fo whethe thee is some intenal node of the genealogy fo which both (a S b is the label set of its induced subgenealogy, and (b this induced subgenealogy has unlabeled topology T. The numbe of nodes whose induced subgenealogies have unlabeled topology T is ( n Y Y b. b1 Note that if Z b 0, then Y b 0. Applying Theoem 3.5, if Z b 1, then Y b 1 with pobability Q(T. (i The mean numbe of nodes that have induced subgenealogy T is M T (n, E[Y ] ( n b1 ( n b1 P[Z b 0]E[Y b Z b 0] + P[Z b 1]E[Y b Z b 1] Q(T P[Z b 1] Q(T M(n,. (ii Using the conditional vaiance fomula [28, p. 138] with the fact that a Benoulli

14 142 N.A. Rosenbeg 3 Vaiance of the numbe of -catepillas 2.7 n Numbe of descendants ( Figue 6: The vaiance V cat (n, of the numbe of -catepillas fo n 50. andom vaiable with paamete Q(T has vaiance Q(T [1 Q(T ], V T (n, Va[Y] E[Va[Y Z]] +Va[E[Y Z]] E[Q(T [1 Q(T ]Z] +Va[Q(T Z] Q(T [1 Q(T ]M(n, + Q(T 2 V(n,. Coollay 5.2. In a genealogy with n 3 lineages, if 2 n 1, then (i the mean numbe of -catepillas, M cat (n,, is 2 1 n ( + 1!, and (ii the vaiance of the numbe of -catepillas, V cat (n,, is V 1cat (n, 2 1 n[(2 1(2 + 1( + 1! ( ] ( + 1! 2, if < n/2, (2 1(2 + 1 V 2cat (n, 2 1 n[(2 1( + 1! ( ] ( + 1! 2, if n/2, (2 1 V 3cat (n, 2 1 n[( + 1! 2 1 n] ( + 1! 2, if > n/2.

15 -Ponged Nodes and -Catepillas in Genealogical Tees Vaiance of the numbe of -catepillas n16 n15 n14 n13 n12 n11 n10 n9 n8 n7 n6 n5 n Numbe of descendants ( Figue 7: The vaiance V cat (n, of the numbe of -catepillas fo n 4 though 16. Fo n 3, V(3, 2 0. Poof. Applying Coollay 3.6, Q(T 2 2 /( 1! if T is pectinate. Inseting this quantity along with the values of M(n, and V(n, fom Theoem 4.4 into Theoem 5.1, the esult follows. Remak 5.3. Similaly to the case of -ponged nodes, the mean numbe of -catepillas is lagest fo 2, and the vaiance is lagest at 3. Figue 6 displays V cat (50, as a function of. In the case of catepillas, unlike the -ponged node case, the monotonic decline in vaiance afte 3 is not inteupted at n/2, as is demonstated in the following theoem. Theoem 5.4. Fo a fixed n 11 as well as fo n 6, 7, 9, 10, as anges ove integes fom 2 to n 1, (i the fou highest values of V cat (n, ae fom geatest to smallest, at 3, 2, 4, and 5; (ii fo 3 n 2, V cat (n, > V cat (n, + 1. Poof. The cases of n 6, 7, 9, 10 can be veified fom Coollay 5.2 (ii by diect computation. Fo n 11 the poof follows fom the following six inequalities: (a, (b, and (e ae easily poven fom the coesponding statements in the poof of Theoem 5.4, and (c, (d and (f ae staightfowad using elementay methods. (a Fo integes n, with n 3 and 2 n 1, V 2cat (n, > V 1cat (n, > V 3cat (n,, (b Fo integes n, with n 3 and 3 n 1, V 1cat (n, > V 1cat (n, + 1, (c Fo positive n, V 1cat (n, 3 > V 1cat (n, 2 > V 1cat (n, 4 > V 1cat (n, 5, (d Fo even integes n 12, V 1cat (n, 5 > V 2cat (n, n/2,

16 144 N.A. Rosenbeg (e Fo integes n, with n 10 and n/2 < n 1, V 3cat (n, > V 3cat (n, + 1, (f Fo even integes n 8, V 2cat (n, n/2 < V 1cat (n, n/2 1. The poof of (i uses (a, (b, (c, and (d, and follows that of Theoem 4.8 (i, except that the positions of V 1cat (2, and V 1cat (4, ae evesed. The poof of (ii uses (a, (b, (e and (f and follows that of Theoem 4.8 (ii, except that the diection of (f guaantees V cat (n, > V cat (n, + 1 at + 1 n/2 fo even n. Remak 5.5. Fo lage elative to n, a genealogy is likely to contain at most one - catepilla, and V cat appoaches M cat. Fo lage n, with 2, V cat (n, /n follows the sequence 2/45, 23/420, 67/1575, 364/19305, 28466/ , 823/ At small n (n 8, howeve, as in the case of -ponged nodes, the behavio of the vaiance can diffe fom that specified by the theoem (Figue 7. It is inteesting to compae the mean and vaiance of the numbe of -ponged nodes with those of the numbe of -catepillas. Fo 2 and 3, an -ponged node is necessaily a catepilla, and the numbes of -ponged nodes and -catepillas in a genealogy ae equal. Fo 4, an -ponged node need not be an -catepilla, and consequently, the mean numbe of -catepillas is stictly less than the mean numbe of -ponged nodes. Fo 4, as can be veified by elementay compaison of the fomulas in Theoem 4.4 and Coollay 5.2, the vaiance of the numbe of -catepillas is stictly less than the vaiance of the numbe of -ponged nodes, with two exceptions: fo (n, (6, 4, both vaiances equal 6/25, and fo (n, (7, 4, the vaiance of the numbe of catepillas equals 56/225, while the vaiance of the numbe of -ponged nodes equals only 21/100. Summing ove all, each intenal node is -ponged fo one value of, and the mean total numbe of -ponged nodes (not counting the oot is 2 n 1 2n/[( + 1] n 2, as it should. The mean numbe of catepillas intenal nodes (not counting the oot that ae -catepillas fo some is 2 n n/( + 1!. As a faction of n, this sum has a lage-n limit of (e 2 5/ ; thus, fo lage Yule-geneated genealogies, on aveage 60% of subgenealogies ae pectinate. Acknowledgments. I thank the efeees fo caeful eadings of the manuscipt. This wok was suppoted by a National Science Foundation Postdoctoal Fellowship in Biological Infomatics and by a Buoughs Wellcome Fund Caee Awad in the Biomedical Sciences. Refeences 1. D. Aldous, Pobability distibutions on cladogams, In: Discete Random Stuctues, D. Aldous and R. Pemantle, Eds., Spinge-Velag, New Yok, (1996 pp D.J. Aldous, Stochastic models and desciptive statistics fo phylogenetic tees, fom Yule to today, Statist. Sci. 16 ( M.G.B. Blum and O. Fançois, On statistical tests of phylogenetic tee imbalance: the Sackin and othe indices evisited, Math. Biosci. 195 ( J.K.M. Bown, Pobabilities of evolutionay tees, Syst. Biol. 43 ( J.H. Degnan and L.A. Salte, Gene tee distibutions unde the coalescent pocess, Evolution 59 (

17 -Ponged Nodes and -Catepillas in Genealogical Tees L. Devoye, Limit laws fo local countes in andom binay seach tees, Random Stuctues Algoithms 2 ( L. Devoye, Limit laws fo sums of functions of subtees of andom binay seach tees, SIAM J. Comput. 32 ( A.W.F. Edwads, Estimation of the banch points of a banching diffusion pocess, J. Roy. Statist. Soc. Se. B 32 ( J. Felsenstein. Infeing Phylogenies, Sinaue, Sundeland, MA, Y.X. Fu, Statistical popeties of segegating sites, Theo. Pop. Biol. 48 ( H.W. Gould, Combinatoial Identities, Gould Publications, Mogantown, WV, E.F. Hading, The pobabilities of ooted tee-shapes geneated by andom bifucation, Adv. Appl. Pobab. 3 ( S.B. Head, Pattens in tee balance among cladistic, phenetic, and andomly geneated phylogenetic tees, Evolution 46 ( R.R. Hudson, Gene genealogies and the coalescent pocess, Oxf. Suv. Evol. Biol. 7 ( J.F.C. Kingman, On the genealogy of lage populations, J. Appl. Pobab. 19A ( W.P. Maddison and M. Slatkin, Null models fo the numbe of evolutionay steps in a chaacte on a phylogenetic tee, Evolution 45 ( H.M. Mahmoud, Evolution of Random Seach Tees, Wiley, New Yok, A. McKenzie and M. Steel, Distibutions of cheies fo two models of tees, Math. Biosci. 164 ( A.O. Mooes and S.B. Head, Evolutionay pocess fom phylogenetic tee shape, Quat. Rev. Biol. 72 ( A.O. Mooes and S.B. Head, Using tee shape, Syst. Biol. 51 ( F. Mutagh, Counting dendogams: a suvey, Discete Appl. Math. 7 ( J.E. Neigel and J.C. Avise, Phylogenetic elationships of mitochondial DNA unde vaious demogaphic models of speciation, In: Evolutionay Pocesses and Theoy, S. Kalin and E. Nevo, Eds., Academic Pess, New Yok, (1986 pp M. Nodbog, On the pobability of Neandethal ancesty, Ame. J. Hum. Genetics 63 ( M. Nodbog, Coalescent theoy, In: Handbook of Statistical Genetics, Chapte 7, D.J. Balding, M. Bishop, and C. Cannings, Eds., Wiley, Chicheste, UK, (2001 pp R.D.M. Page, Random dendogams and null hypotheses in cladistic biogeogaphy, Syst. Zool. 40 ( P. Pamilo and M. Nei, Relationships between gene tees and species tees, Mol. Biol. Evol. 5 ( M. Petkovšek, H.S. Wilf, and D. Zeilbege, A B, Petes, Wellesley, MA, J.A. Rice, Mathematical Statistics and Data Analysis, 2nd edition, Duxbuy Pess, Belmont, CA, N.A. Rosenbeg, The pobability of topological concodance of gene tees and species tees, Theoet. Popul. Biol. 61 ( N.A. Rosenbeg, The shapes of neutal gene genealogies in two species: pobabilities of monophyly, paaphyly, and polyphyly in a coalescent model, Evolution 57 ( N.A. Rosenbeg, Gene genealogies, In: Evolutionay Genetics: Concepts and Case Studies, C.W. Fox and J.B. Wolf, Eds., Oxfod Univesity Pess, Oxfod, 2006.

18 146 N.A. Rosenbeg 32. C. Semple and M. Steel, Phylogenetics, Oxfod Univesity Pess, Oxfod, J.B. Slowinski, Pobabilities of n-tees unde two models: a demonstation that asymmetical inteio nodes ae not impobable, Syst. Zool. 39 ( J.B. Slowinski and C. Guye, Testing the stochasticity of pattens of oganismal divesity: an impoved null model, Ame. Natualist 134 ( M. Steel and A. McKenzie, Popeties of phylogenetic tees geneated by Yule-type speciation models, Math. Biosci. 170 ( J. Stone and J. Repka, Using a nonecusive fomula to detemine cladogam pobabilities, Syst. Biol. 47 ( J.R. Stone, Pobabilities fo completely pectinate and symmetic cladogams, Cladistics 19 ( F. Tajima, Evolutionay elationship of DNA sequences in finite populations, Genetics 105 ( G.U. Yule, A mathematical theoy of evolution based on the conclusions of D. J.C. Willis, F.R.S., Philos. Tans. Roy. Soc. Lond. Se. B 213 (