Week 3 Classical Probability, Part I

Week 3 Classical Probability, Part I

Week 3 Objectives Proper understanding of common statistical practices such as confidence intervals and hypothesis testing requires some familiarity with probability theory. We start with classical probability, which arose from games of chance such as rolling dice or dealing cards. In such experiments, where all outcomes are equally likely, the probability of an event is determined by enumerating the outcomes making up the event. After reviewing basic set operations, the necessary counting techniques are presented. The transition to experiments where the outcomes are not equally likely can be seamless. The axioms and properties of probability in general experiments are presented, and the important notion of a probability mass function is introduced.

1 2 3

Overview Outline The central role probability plays in statistics is the reason why statistics classes typically cover this subject. Classical probability, which arose from games of chance, includes combinatorics and the concepts of conditional probability and independence. Probability evolved to deal with modeling the randomness of phenomena such as the number of earthquakes, the amount of rainfall, the life time of a given electrical component, or the relation between education level and income, etc. Such probability models will be discussed in Chapters 3 and 4.

Sample Spaces The set of all possible outcomes of a random experiment is called the sample space of the experiment, and will be denoted by S. some examples follow. 1) The sample space of the experiment which selects two fuses and classifies each as non-defective or defective is S 1 = {(0, 0), (0, 1), (1, 0), (1, 1)}, where 0 and 1 stand for non-defective and defective, respectively. 2) The sample space of the experiment which selects two fuses and records how many are defective is S 2 = {0, 1, 2}. 3) The sample space of the experiment which records the number of fuses inspected until the second defective is found is S 3 = {2, 3, 4,...}.

4) Three undergraduate students are selected and their opinions about expanding the use of solar energy are recorded on a scale from 1 to 10. Give the sample space of this experiment. What is the size of this sample space? Solution: The set of all possible outcomes consist of the triplets (x 1, x 2, x 3 ), where x 1, x 2 and x 3 denote the response of the 1st, 2nd and 3rd student, respectively. Thus, S 4 = {(x 1, x 2, x 3 ) : x 1 = 1, 2,..., 10, x 2 = 1, 2,..., 10, x 3 = 1, 2,..., 10}. There are 10 10 10 = 1000 possible outcomes.

5) Only the average rating, provided by the three undergraduate students of the previous example, is recorded. Give the sample space, S 5, of this experiment. What is the size of this sample space? Solution: S 5 is the collection of all distinct averages (x 1 + x 2 + x 3 )/3 formed from the 1000 triplets of S 4. The word distinct is emphasized because, for example, (5, 6, 7) and (4, 6, 8) both yield an average of 6. The size of S 5 is determined, most easily, in R: S4=expand.grid(x1=1:10,x2=1:10,x3=1:10) # all triplets in the sample space S 4 length(table(rowmeans(s4))) # returns 28 for the number of different averages

Events Outline In experiments with many possible outcomes, investigators often classify individual outcomes into distinct categories. Opinion ratings may be classified into low (L = {0, 1, 2, 3}), medium (M = {4, 5, 6}) and high (H = {7, 8, 9, 10}). Opinion ratings of three students can be classified according to the 28 distinct average values. Definition Collections of individual outcomes are called events. An event consisting of only one outcome is called a simple event. Events are denoted by letters such as A, B, C, etc.

We say that a particular event A has occurred if the outcome of the experiment is a member of (i.e., contained in) A. If the result of 5 coin tosses is two heads and three tails, the event A = {at most 3 heads in five tosses of a coin} has occurred. The sample space of an experiment is an event which always occurs when the experiment is performed.

Set Operations The union, A B, of events A and B, is the event consisting of all outcomes that are in A or in B or in both. The intersection, A B, of A and B, is the event consisting of all outcomes that are in both A and B. The complement, A or A c, of A is the event consisting of all outcomes that are not in A. The events A and B are said to be mutually exclusive or disjoint if they have no outcomes in common. That is, if A B =, where denotes the empty set. The difference A B is defined as A B c. A is a subset of B, A B, if e A implies e B. Two sets are equal, A = B, if A B and B A.

Union of A and B A B Intersection of A and B A B A B A B Figure: Venn diagrams for union and intersection

The complement of A A c The difference operation A B A A B Figure: Venn diagrams for complement and difference

A A B B Figure: Venn diagram illustrations of A, B disjoint, and A B

Commutative Laws: a) A B = B A, b) A B = B A Associative Laws: a) (A B) C = A (B C) b) (A B) C = A (B C) Distributive Laws: a) (A B) C = (A C) (B C), b) (A B) C = (A C) (B C) De Morgan s Laws: a) (A B) c = A c B c b) (A B) c = A c B c

Example The following table classifies 100 cell phone calls according to their duration and number of handovers a it undergoes. Number of Handovers Duration 0 1 > 1 3 10 20 10 < 3 40 15 5 Let A = {call undergoes 1 handover}, B = {call lasts < 3 min}. (a) How many of the 100 calls are in each of A B and A B? (b) Give word descriptions of (A B) c and A c B c. Are these sets equal? Which property is confirmed? a changes of the source cell, as when the phone is moving into the area covered by another source

Example (Continued) Solution: (a) There are 80 calls that undergo 1 handover or last < 3 min (or both). There are 15 calls that undergo 1 handover and last < 3 min. (b) (A B) c consists of the 20 calls that are not in A B, i.e., the calls that last 3 min and undergo either 0 or > 1 handovers. A c B c consists of the calls that do not undergo 1 handover and do not last < 3 min. Thus, (A B) c = A c B c confirming the fist of De Morgan s laws. Read also the examples in p. 57

Definition of Probability The probability of an event E, denoted by P(E), is used to quantify the likelihood of occurrence of E by assigning a number from the interval [0, 1]. Higher numbers indicate that the event is more likely to occur. A probability of 1 indicates that the event will occur with certainty, while a probability of 0 indicates that the event will not occur. The likelihood of occurrence of an event is also quantified as a percent or in terms of the odds; read Section 2.3.1., p. 60.

Assignment of Probabilities It is simplest to introduce probability in experiments with a finite number of equally likely outcomes, such as those used in games of chance, or simple random sampling. Probability for equally likely outcomes If the sample space consists of N outcomes which are equally likely to occur, then the probability of each outcome is 1/N.

Population Proportions as Probabilities Let a unit be selected by s.r. sampling from a finite statistical population of a categorical variable. If category i has N i units, then the probability the selected unit came from category i is p i = N i /N. where N is the total number of units (so N = N i ). Thus, (a) In rolling a die, the probability of a three is p = 1/6. (b) If 160 out of 500 tin plates have one scratch, and one tin plate is selected at random, the probability that the selected plate has one scratch is p = 160/500.

Efron s Dice Outline Die A: four 4s and two 0s Die B: six 3s Die C: four 2s and two 6s Die D: three 5 s and three 1 s Specify the events A > B, B > C, C > D, D > A. Find the probabilities that A > B, B > C, C > D, D > A. Hint: When two dice are rolled, the 36 possible outcomes are equally likely.

Outline 1 2 3

Even when the population units are equally likely, the values of the random variable recorded may not be equally likely. When die is rolled twice, each of the 36 possible outcomes are equally likely. But if we record the sum of the two rolls, these outcomes are not equally likely. Definition The probability mass function, or pmf, of a discrete random variable X, is a list of the probabilities p(x) for each value x of the sample space S X of X.

Example Roll a die twice. Find the pmf of X = the sum of the two die rolls. Solution: The sample space of X is S X = {2, 3,..., 12}. The pmf can be found with the R commands: options(digits = 2); S=expand.grid(X1=1:6,X2=1:6); table(s$x1+s$x2)/length(s$x1) The probabilities these commands return are show in the following table: x 2 3 4 5 6 7 8 9 10 11 12 p(x) 0.028 0.056 0.083 0.111 0.139 0.167 0.139 0.111 0.083 0.056 0.028 Try also which(s$x1+s$x2==7); S[which(S$X1+S$X2==7),].

Sample Space Population It is useful to think of a random experiment as random (but not simple random) sampling from the sample space population. 1 Sampling a tin plate and recording the number of scratches (recall there can be either 0, 1 or 2 scratches) can be thought of as random (but not simple random) sampling from S = {0, 1, 2}. 2 Sampling a US voter and recording his/her opinion, on a scale from 1 to 10, regarding expanding the use of solar energy can be thought of as random (but not simple random!) sampling from S = {0, 1,..., 10}.

Definition When a sample space is thought of as the set from which we sample, we refer to it as sample space population. The random sampling from the sample space population (which is need not be simple random sampling) is called probability sampling, or sampling from a pmf. This idea makes it possible to think of different experiments as sampling from the same population. For example Inspecting 50 products and recording the number of defectives, and Interviewing 50 people and recording if they read New York Times can both be thought as probability sampling from their common sample space S = {0, 1, 2,..., 50}.

Simulations Outline The concepts of pmf and sample space population and probability sampling are useful for conducting simulations, i.e., for generating a large number of repetitions of an experiment. Such simulations are used to better understand aspects of complex populations. Example (Simulating an Experiment with R) Use the pmf of X = sum of two die rolls to simulate 1000 repetitions of the experiment which records the sum of two die rolls. Take their mean and use it to guess the population mean. The R commands are S=expand.grid(X1=1:6,X2=1:6) ; pmf=table(s$x1+s$x2)/36 mean(sample(2:12, size=1000, replace=t, prob=pmf))

Outline 1 2 3

Why Count? Outline In classical probability counting is used for calculating probabilities. if the sample space consists of a finite number of equally likely outcomes, then for the probability of an event A we need to know the number of outcomes in A, N(A), and the total number of outcomes, N(S), because P(A) = N(A) N(S) Some counting questions are difficult (e.g., how many different five-card hands are possible from a deck of 52 cards?). Thus we need specialized counting techniques.

Combinations The number of samples of size n that can be formed from N units is called the number of combinations of n objects selected from N, is denoted by ( N n), and equals ( ) N = n N!, where k! = 1 2 k. n!(n n)! ( ) 52 For example, there are = 52! = 2, 598, 960 5 5!47! different five-card hands from a deck of N = 52 cards. Knowing the above, we can calculate the probability of the hand with 4 aces and the king of hearts, as well as of A = {the hand has 4 aces}. What are these probabilities?

Other counting questions can also be thought of as the number of different samples, of a certain size, that can be taken from a population. How many different n-long sequences consisting of k 1s and n k 0s can be formed? The answer is the number of combinations ( n k). (Why?) When inspecting n items as they come off the assembly line, the probability of the event E = {k of the n inspected items are defective} is calculated using the concept of independence and from knowing the number of n-long sequences consisting of k 1s (for defective) and n k 0s (for non defective).

In what follows we will justify the formula for ( n k). In the process we will introduce the notion of permutations.

The Product Rules The Simple Product Rule: Suppose a task can be completed in two stages. If stage 1 has n 1 outcomes, and if stage 2 has n 2 outcomes regardless of the outcome in stage 1, then the task has n 1 n 2 outcomes. How many five-card hands with four aces are there? Solution: Think of the task of forming a hand with four aces from a deck of 52 cards. This task can be completed in two stages: First select the 4 aces, and then select one additional card. Here n 1 = 1 and n 2 = 48 (why?). Thus, there are 1 48 = 48 such hands.

Example (1) 1 In how many ways can we select the 1st and 2nd place winners from the four finalists Niki, George, Sophia and Martha? Answer: 4 3 = 12. (Why?) 2 In how many ways can we select two from Niki, George, Sophia and Martha? Answer: 12 2 = 6. (Why?) Note: 6 = # of combinations = ( ) 4. 2

The General Product Rule: If a task can be completed in k stages and stage i has n i outcomes, regardless of the outcomes the previous stages, then the task has n 1 n 2 n k outcomes How many binary sequences of length 10 (i.e., a 10-long sequence of 0s and 1s) are there? Answer: Think of the task of forming a binary sequence of length 10. This task consists of 10 stages and each stage has two outcomes (i.e., either 0 or 1) regardless of the outcomes the previous stages. Thus, there are 2 10 = 1024 different sequences. Read also Example 2.3-4, p. 63.

Example (2) 1 In how many ways can we select a 1st, 2nd and 3rd place winners from Niki, George, Sophia and Martha? Answer: 4 3 2 = 24. (Why?) 2 In how many ways can we select three from Niki, George, Sophia and Martha? Answer: 24 6 = 4. (Why?) Note: 4 = # of combinations = ( ) 4. 3

Permutations Outline The answer to Example (1), part 1, i.e. 12, is the number of permutations of 2 items selected from 4. The answer to Example (2), part 1, i.e. 24, is the number of permutations of 3 items selected from 4. Definition The number of ordered selections (i.e. when we keep track of the order of selection) of k items from n is called the number of permutations of k items selected from n, it is denoted by P k,n, and equals P k,n = n (n 1)... (n k + 1) = n! (n k)!

Combinations In the answer to Example (1), part 2, i.e. 12 2, the 2 in the denominator is the number of permutations of 2 items selected from 2 (P 2,2 = 2 1). In the answer to Example (2), part 2, i.e. 24 6, the 6 in the denominator is the number of permutations of 3 items selected from 3 (P 3,3 = 3 2 1). Extending the reasoning used to obtain these answers, we have The number of combinations of k items selected from a group of n is (n ) k = P k,n k! = n! k!(n k)!

Binomial Coefficients The numbers ( n k) are called binomial coefficients because of the Binomial Theorem: (a + b) n = n k=0 ( ) n a k b n k. k

Example (a) How many paths going from the lower left corner of a 4 3 grid to its upper right corner are there? Assume one is allowed to move either to the right or upwards. (b) How many of the paths in (a) pass through the (2, 2) point of the grid? (c) What is the probability that a randomly selected path will pass through the (2, 2) point? Hints: (a) A path can be represented by a 7-long binary sequence with four 1s (1 denotes a step to the right) and three 0s (steps upwards). (b) The # of paths from the lower left corner to the (2, 2) point times the # of paths from (2, 2) to the upper right corner.

Example An order comes in for 5 palettes of low grade shingles. In the warehouse there are 10 palettes of high grade, 15 of medium grade, and 20 of low grade shingles. An inexperienced shipping clerk is unaware of the distinction in grades of asphalt shingles and he ships 5 randomly selected palettes. 1 How many different groups of 5 palettes are there? ( 45 5 ) = 1, 221, 759. 2 What is the probability that all of the shipped palettes are low grade? ( 20 5 ) / ( 45 5 ) = 15, 504/1, 221, 759 = 0.0127. 3 What is the probability that 2 of the shipped palettes are of medium [ grade and 3 are from low grade? (15 )( 20 ) ] 2 3 / ( ) 45 5 = (105 2280)/1, 221, 759 = 0.0127 = 0.1959.

Example A communication system consists of 15 indistinguishable antennas arranged in a line. The system functions as long as no two non-functioning antennas are next to each other. Suppose six antennas stop functioning. a) How many different arrangements of the six non-functioning antennas result in the system being functional? (Hint: The 9 functioning antennas, lined up among themselves, define 10 possible locations for the 6 non-functioning antennas so the system functions.) b) If the arrangement of the 15 antennas is random, what is the probability the system is functioning?

Example What is the probability that 5 randomly dealt cards form a full house? Solution: First, the number of all 5-card hands is ( 52 ) 52! 5 = = 2, 598, 960. Next, think of the task of forming a 5!47! full house as consisting of two stages. In Stage 1 choose two cards of the same kind, and in stage 2 choose three cards of the same kind. Since there are 13 kinds of cards, stage 1 can be completed in ( 13 1 )( 4 2) = (13)(6) = 78 ways (why?). For each outcome of stage 1, the task of stage 2 becomes that of selecting three of a kind from one of the remaining 12 kinds. This can be completed in ( 12 1 )( 4 3) = 48 ways. Thus there are (78)(48) = 3, 744 possible full houses, and the desired probability is 0.0014.

Multinomial Coefficients The number of ways n units can be divide in r groups of specified sizes n 1,..., n r is given by ( ) n n! = n 1, n 2,..., n r n 1!n 2! n r! These numbers are called multinomial coefficients because of the Multinomial Theorem. In how many ways can 8 engineers be assigned to work on projects A, B, and C, so that 3 work on project A, 2 work on B, and 3 work on C? Answer: 560

Read Examples 2.3-8, 2.3-11, pp. 66, 67

The axioms governing any assignment of probabilities are: Axiom 1: P(E) 0, for all events E Axiom 2: P(S) = 1 Axiom 3: If E 1, E 2,... are disjoint P(E 1 E 2...) = P(E i ) i=1

Properties of Probability The following properties follow from the three axioms: 1) P( ) = 0 2) If E 1,..., E m are disjoint, then P(E 1 E m ) = P(E 1 ) + + P(E m ) 3) If A B then P(A) P(B) 4) P(A) = 1 P(A c ), for any event A

Two additional properties deal with the probability of the union of events that are not disjoint: 5) P(A B) = P(A) + P(B) P(A B) 6) P(A B C) = P(A) + P(B) + P(C) P(A B) P(A C) P(B C) + P(A B C) The formula for property 6 follows the so-called inclusion - exclusion principle and extends to the union of more than three events.

Example The probability that a firm will open a branch office in Toronto is 0.7, that it will open one in Mexico City is 0.4, and that it will open an office in at least one of the cities is 0.8. Find the probabilities that the firm will open an office in: 1 neither of the cities (Answer: 1 0.8 = 0.2) 2 both cities (Answer: 0.7 + 0.4 0.8 = 0.3) 3 exactly one of the cities (Answer: (0.7 0.3) + (0.4 0.3) = 0.5, or 0.8 0.3 = 0.5)

Example The R commands attach(expand.grid(x1=0:1,x2=0:1, X3=0:1,X4=0:1)); table(x1+x2+x3+x4)/length(x1) yields the following pmf for the random variable X = number of heads in four flips of a coin: x 0 1 2 3 4 p(x) 0.0625 0.25 0.375 0.25 0.0625. (a) What do Axiom 2 and property 2 say about the sum of all probabilities? (Answer: They sum to 1) (b) What is P(X 2)? (Answer: 0.375 + 0.25 + 0.0625 = 0.6875) Read also Examples 2.4-2, 2.4-3, pp. 75, 76