CS1800: Intro to Probability. Professor Kevin Gold

CS1800: Intro to Probability Professor Kevin Gold

Probability Deals Rationally With an Uncertain World Using probabilities is the only rational way to deal with uncertainty De Finetti: If you disagree, there is always some unwise bet you should be willing to make, which will lose money in the long term Though programs can be deterministic, the world they interact with is full of uncertainty. This means probability is important in the fields that deal with real data and real sensors - AI and machine learning. Random users will buy random things from your website Data mining The self-driving car may be unsure of the depth or identity of an object Robotics The interpretation of a sentence or speech recording may be ambiguous, but some interpretations are more likely Natural Language Processing A diagnosis may rely on several symptoms, each of which only tends to coincide with some underlying diseases Probabilistic Expert Systems

Probability Also Plays a Role in the Design of Everyday Algorithms We may want to Analyze the expected running time Asymptotic Analysis Behave randomly to avoid bias Randomized Algorithms Create algorithms that are difficult for an adversary to predict and exploit Cryptography/Security Design programs that have novel behaviors or generate random content Procedural Content Generation

Events, Experiments, and Probabilities An event is something that might happen - for example, flipping a coin and finding the result is heads, or rolling an even number on a six-sided die. An experiment or observation refers to a situation with one or more possible outcomes. Flipping a coin is an experiment with two possible outcomes: heads or tails. Rolling a six-sided die has six outcomes. An event may describe multiple outcomes: roll an even number A probability is real number in the range [0,1] that describes how much we should believe that an experiment will result in an event happening - because the outcome fits the event description. 0 means the event does not happen. 1 means it definitely does. 0.5 means we think both possibilities are equally likely.

Probability Notation Short for probability ; sometimes just P() Pr(heads) = 0.5 the probability brief description of event

Probability Notation A random variable representing a value determined by the experiment Pr(C = H ) = 0.5 the specific value that matches the event we re interested in the probability We ll discuss random variables more when we discuss expectations

Counting to Calculate Probabilities The sample space, which we ll call S, is the set of all possible outcomes of an experiment. If we think all outcomes are equally likely, we can count outcomes to determine the probability of an event. The probability is Pr(event) = E / S where E is the set of outcomes in which the event is true, and S is the total set of all outcomes. Roll of a 6-sided die S E 1 2 3 4 5 6 Pr(roll 5 or 6) = 2/6 = 1/3

Counting to Determine Probabilities - Examples Flipping a single coin S = {H, T} Pr(heads) = 1/2 (E = {H}) Rolling a 6-sided die S = {1,2,3,4,5,6} Pr(roll a 1) = 1/6 (E = {1}) Pr(roll an even number) = 3/6 = 1/2 (E = {2, 4, 6})

Counting to Determine Probabilities - Examples Rolling two six-sided dice S = {(1,1), (1,2), (1,3),, (2,1), (2,2), (6,5),(6,6)} ( S = 6*6 = 36 possible combinations) Chance of doubles (matching values) E = {(1,1), (2,2), (3,3), (4,4), (5,5), (6,6)} Pr(doubles) = 6/36 = 1/6 Chance of rolling a total of 6 E = {(1,5), (2,4),(3,3),(4,2),(5,1)} Pr(roll a 6) = 5/36

More Sophisticated Counting to Calculate Probabilities Flipping 4 coins S = { TTTT, TTTH, TTHT, TTHH,, HHHH } (2 4 = 16 possibilities) Pr (exactly two heads) E = # of strings with exactly 2 H characters = C(4,2) = 4*3/2 = 6 Pr (exactly two heads) = 6/16 = 3/8 Pr(at least one head) E = strings with at least one H = 16 - strings with no H s = 16 - { TTTT } = 15 Pr(at least one head) = 15/16 We can encode outcomes as strings, then count strings

Pascal s Triangle and Coin-Flipping If I want to know the probabilities of exactly 0, 1, 2, 3, 4, or 5 heads when flipping 5 coins, I need to know how many strings like TTTTT have 0, 1, 2, 3, 4, or 5 H s These numbers are C(5,0), C(5,1), C(5,2), C(5,3), C(5,4), and C(5,5) Which can be looked up as a row of Pascal s triangle: 1, 5, 10, 10, 5, 1 The relevant probabilities are then 1/2 5, 5/2 5, 10/2 5, 10/2 5, 5/2 5, and 1/2 5, or about 0.03, 0.16, 0.31, 0.31, 0.16, 0.03.

Counting Produces Your Weather Forecast Ever wonder where those percentages come from in the weather forecast ( 60% chance of rain )? These are typically produced by running many simulations of the current weather, propagating the present into the future And using random values to perturb the simulation The final forecast percentages are produced by counting the outcomes in which the event in question took place

Fallacies to Avoid: Assuming Equal Likelihood of Outcomes We could start to go around thinking everything has a 50% chance of happening because there are two outcomes - either it does happen, or it doesn t. 50% chance of being struck by lightning before the end of the day! 50% chance of winning the lottery today! Counting only produces reasonable probabilities when the counted outcomes are equally likely. That s why many examples use coins, cards, or other scenarios where this is obviously true. In many real-world cases, there is something we could count to get a reasonable probability. For the lottery, we could count equally random ticket numbers For lightning strikes, we could model the problem as picking a day and person at random from our records, and ask whether that person was struck on that day

Card-Drawing Problems Recall that a deck of playing cards has 52 possibilities, encoded as [rank] of [suit], such as Jack of Hearts or Two of Spades 13 ranks: 2, 3, 4,, 10, Jack (J), Queen (Q), King (K), Ace (A) 4 suits: Clubs (C), Diamonds (D), Spades (S), Hearts (H) Decisions in games such as Poker hinge on the probability of drawing particular subsets of these cards, such as a flush with 5 cards of the same suit. The order in the hand doesn t matter and the cards are unique, so a subset is a good model. A hand such as {2C, 2D, 10H, 10D, KC} is a subset of cards - the deal chose this many cards from the deck. The sample space consists of C(52,5) possibilities for a 5 card hand (it s not a permutation because the order that the cards were dealt doesn t matter).

Sample Card-Drawing Problem: Pocket Aces Pr (chance of 2 aces when dealt 2 cards) S = set of all 2 card hands = {{2H,2D}, {2H,2C},,{AC,AS}} S = C(52,2) E = set of all 2 ace hands = {{AC, AD}, {AC,AS},{AC,AH},{AD,AS},{AD,AH}, {AS,AH}} E = C(4,2) = 6 (choosing 2 of 4 aces in deck) Pr(2 aces) = 6/C(52,2) = 6/(52*51/2) = 0.0045 [which we could describe as 0.45% or 1 in 221]

Second Card Drawing Problem: The Full House A full house is a set of 3 cards of the same rank, along with a set of 2 cards of a different rank (eg, 3 kings, 2 aces). What s the probability of a full house in 5 cards? S = Total 5 card hand possibilities: C(52,5) = 2598960 The rest is counting E, ways to get a full house Pick the first rank: 13 Pick the second rank: 12 Pick which of the 4 suits is missing from first rank: 4 Pick which two cards of the second rank were chosen: C(4,2) = 6 13*12*4*6 = 3744 Prob of full house = 3744/2598960 = 0.0014 [or 0.14%]

Drawing With Replacement vs Without Replacement In probability problems involving cards or drawing colored balls out of urns (weird but classic), the instructions may say the draws are either: Without replacement - When an item is drawn, it can t be drawn again on a subsequent draw. Generally true of card games unless you shuffle your card back in on every draw. For these, you can treat the whole experiment as drawing a subset of the items, and use combinations. With replacement - When an item is drawn, it is put back where you got it (replaced) and can be drawn again. You can thus treat each big experiment as repetitions of the same little experiment (or as independent events, which we ll get to)

With/Without Replacement Example 20 balls in an urn: 18 red, 2 green. Draw 3 balls with replacement We can think of our items as {red 1, red 2,, red 18, green 1, green 2 } With replacement - we throw back the ball each time Valid draws look like (red 1, red 1, green 1 ) - all 20 possibilities for each ball, order matters, repetition allowed Sample space size is 20*20*20 = 8000 Chance of 0 green: 18 3 red-red-red possibilities, 18 3 /8000 = 0.729 Chance of exactly 1 green: 18 2 *2*3 sequences with one green (choosing red IDs, green ID, green loc), 18 2 *2*3/8000= 0.243 Chance of exactly 2 green: 18*2 2 *3 sequences (red ID, green IDs, loc of red), 18*2 2 *3/8000 = 0.027 Chance of all 3 green: 2 3 /8000 = 8/8000 = 0.001

With/Without Replacement Example 20 balls in an urn: 18 red, 2 green. Draw 3 balls without replacement We can think of our items as {red 1, red 2,, red 18, green 1, green 2 } Without replacement - we keep the ball out of the urn after drawing it Valid draws look like {red 1, red 2, green 1 } - all 3-element subsets of the set are equally likely Sample space size is C(20,3) = 1140 (fewer possibilities and we re ignoring order in both numerator and denominator of the probability now) Chance of 0 green: C(18,3) ways to draw 3 different reds, C(18,3)/1140 = 0.716 (was 0.719 with replacement) Chance of exactly 1 green: C(18,2)*2 choices of 2 red, 1 green; C(18,2)*2/1140 = 0.27 (was 0.243 with replacement) Chance of exactly 2 green: sets differ only in choice of red ball; 18/1140 = 0.016 (was 0.027) Chance of all 3 green: 0 (only 2 green balls exist!)

Fallacies to Avoid: The Sample Space vs Reality Recalling that there are 52 cards in the deck: Let s say I deal you a card, then I deal Alice a card at random (both face down, ie in secret). What is the chance you are holding the Ace of Spades? Consider this change to the question. What if I dealt a single card to you, then burned the rest of the deck? What is the chance that you are holding the Ace of Spades?

Fallacies to Avoid: The Sample Space vs Reality The sample space consists of the possibilities that are still open to you, from your point of view. We could burn random cards from the deck, and unless you know what those cards were, it doesn t change your view that there are 52 equally likely possibilities when you draw from the top of the deck. Because you don t know what those cards were. But if we learn information that rules out possibilities for what happened, that changes probabilities. If Alice said hey, the 3 of clubs! that s one fewer card in the sample space, and the probability of the Ace of Spades becomes 1/51.

In-Class Exercises (you can leave these unsimplified) Rolling 3 6-sided dice, what is the probability that they sum to 18? What about the probability that they sum to 17? The probability that the roll consists of a 3, a 2, and a 1, not necessarily in that order? If I m dealt 2 cards, what is the probability that both are in the range 2-10?

In-Class Exercises (you can leave these unsimplified) Rolling 3 6-sided dice, what is the probability that they sum to 18? E = {(6,6,6)} 1/6 3 = 1/216 What about the probability that they sum to 17? The probability that the roll consists of a 3, a 2, and a 1, not necessarily in that order? If I m dealt 2 cards, what is the probability that both are in the range 2-10?

In-Class Exercises (you can leave these unsimplified) Rolling 3 6-sided dice, what is the probability that they sum to 18? E = {(6,6,6)} 1/6 3 = 1/216 What about the probability that they sum to 17? E = {(5,6,6), (6,5,6),(6,6,5)} 3/6 3 = 3/216 = 1/72 The probability that the roll consists of a 3, a 2, and a 1, not necessarily in that order? If I m dealt 2 cards, what is the probability that both are in the range 2-10?

In-Class Exercises (you can leave these unsimplified) Rolling 3 6-sided dice, what is the probability that they sum to 18? E = {(6,6,6)} 1/6 3 = 1/216 What about the probability that they sum to 17? E = {(5,6,6), (6,5,6),(6,6,5)} 3/6 3 = 3/216 = 1/72 The probability that the roll consists of a 3, a 2, and a 1, not necessarily in that order? E = {(1,2,3), (1,3,2), } E = 3! = 6 6/6 3 = 1/36 If I m dealt 2 cards, what is the probability that both are in the range 2-10?

In-Class Exercises (you can leave these unsimplified) Rolling 3 6-sided dice, what is the probability that they sum to 18? E = {(6,6,6)} 1/6 3 = 1/216 What about the probability that they sum to 17? E = {(5,6,6), (6,5,6),(6,6,5)} 3/6 3 = 3/216 = 1/72 The probability that the roll consists of a 3, a 2, and a 1, not necessarily in that order? E = {(1,2,3), (1,3,2), } E = 3! = 6 6/6 3 = 1/36 If I m dealt 2 cards, what is the probability that both are in the range 2-10? E = {{2D,2H},,{10S,10C}} # cards = 9 ranks* 4 suits = 36 E = C(36,2) E / S = C(36,2)/C(52,2) = 0.475

More Counting-Based Probability Examples A deck has 7 cards, A,B,C,1,2,3,4. What is the probability that, after shuffling, the letters are in order in the deck and the numbers are in order, but they aren t necessarily contiguous? Examples: AB12C34, 123A4BC Number of orderings of deck: 7! (7 factorial) Number of orderings with success (hint: once we choose where letters go, nothing left to choose; LLNNLNN = AB12C34):

Axioms of Probability A valid probability is in the range [0,1] The sum of the probabilities of all outcomes ( simple /disjoint events) must be 1. (We should be certain that something will happen.) Corollary: Pr( A) = 1 - Pr(A) Pr (A v B) = Pr(A) + Pr(B) - Pr(A ^ B) This is just an application of inclusion-exclusion to probability - if events overlap in outcomes, we shouldn t double-count

The Birthday Paradox How many people do we need in a room to think it s more likely than not that 2 people share a birthday? The pigeonhole principle tells us we need 367 people (taking into account leap years) to be absolutely certain. But we need far fewer to think it s more likely than not that 2 people share a birthday. This is because after a while, it becomes unlikely that we manage to avoid all collisions repeatedly. Every new birthday has to randomly not match the birthdays that came before. The CS application is that sometimes two pieces of data will want to be randomly stored in the same place, needing extra attention - we may care how often that happens

The Birthday Paradox Grace bday Eve Chuck Alice Bob bday bday bday bday Jan Jan Jan Jan Jan Jan Jan Dec Dec 1 2 3 4 5 6 7 30 31 At what point is it more likely than not that two balls land in the same bin?

The Birthday Paradox N people, want to figure what N should be so that Pr(shared birthday) > 50% Pr (shared birthday) = 1 - Pr(no shared birthday) Pr(no shared birthday) = [ways to have N distinct birthdays] [ways to have N birthdays] = P(365,N)/365 N = 365*364* *(365-N+1)/365 N P(365,N) here is an r-permutation, not a probability This makes the chance of a shared birthday 1 - P(365,N)/365 N Some experimentation can reveal that when N = 23, this crosses the threshold to Pr(shared) = 1 - P(365,23)/365 23 = 0.5073 And when N = 70, we get P(shared) = about 0.999

Independent Events When events intuitively can have no bearing on each other - 2 flips of a coin, 2 rolls of a die - the events are said to be independent. When events are independent, Pr(A ^ B) = Pr(A)Pr(B). The chance of both events happening is equal to the product of the two probabilities. Chance of flipping heads twice: (1/2)(1/2) = 1/4. Chance of rolling a 2, then a 6 on a 6-sided die: (1/6)(1/6) = 1/36 Chance of flipping heads 5 times out of 5: (1/2) 5 = 1/32 The equation above in bold is the mathematical definition of independence. If it s not clear whether events are independent, we check whether that equation holds. If events are not independent, Pr(A ^ B) = Pr(A)Pr(B) is not true. You ll need to use conditional probability in that case (described later).

Independent Events In the Real World Independent Separate die rolls Card draws with replacement Whether it s a hot day on Mars and Venus Birthdays of people who aren t related Not Independent Whether a 6-sider roll is even and whether it s > 3 (2 vs 4,6) Draws without replacement Whether it s a hot day on two consecutive days Whether two stocks went up on the same day True independence can be difficult to find in the real world, but it s a good approximation until it isn t (2008 financial crisis)

Checking Indepdendence Suppose we roll 2 six-sided dice just once - both A and B below refer to the same roll. A = event of rolling doubles B = event of rolling at least one 6 Are these events independent? Unclear, so let s check Pr(A) = 6/36 = 1/6 (because six possible doubles rolls) Pr(B) = 1 - Pr(no 6 s) = 1-5 2 /36 = 11/36 Pr(A ^ B) = Pr(rolled double 6 s) = 1/36 Pr(A ^ B)!= Pr(A)*Pr(B). These events aren t independent. If I told you that the roll was doubles, that should affect your belief that at least one six was rolled (it goes down because of all the mixed rolls with exactly one six).

Checking Indepdendence Suppose we roll 2 six-sided dice just once - both A and B below refer to the same roll. A = event of rolling doubles B = event of rolling a 6 on die #1 Are these events independent? Unclear, so let s check Pr(A) = 6/36 = 1/6 (because six possible doubles rolls) Pr(B) = 1/6 Pr(A ^ B) = Pr(rolled double 6 s) = 1/36 Pr(A ^ B) = Pr(A)*Pr(B). These events are independent. Intuitively, rolling a second die in secret and telling you whether it matches the first doesn t really tell you anything about what the first die was.

Birthday Paradox Revisited With Independence Jan Jan Jan Jan Jan Jan Jan Dec Dec 1 2 3 4 5 6 7 30 31 We can think of each possible collision as an independent event The chance of avoiding all collisions is 365/365 * 364/365 * 363/365 * * (365-N+1)/365 This results in an identical equation to our countingbased approach earlier (Pr(collision) = 1 - P(365,N)/365 N )

Doing a Monopoly Problem Using Independence In Monopoly, you have 3 chances to get out of jail without paying a fine, by rolling doubles (on 2 6-sided dice). What is the chance of at least one doubles in 3 rolls? Pr (at least one doubles on 3 rolls) = 1 - Pr(no doubles on 3 rolls) = 1 - Pr(no doubles on 1 roll) 3 Using Independence = 1 - (1 - Pr(doubles on 1 roll)) 3 = 1 - (1-1/6) 3 = 1 - (5/6) 3 = 1-125/216 = 91/216 = 0.42

Multiplying Independent Probabilities Isn t Strictly Better If you are too attached to computing probabilities of complex events by multiplying probabilities, some problems may cause headaches An example we already saw was the problem of the probability of 7 cards A,B,C,1,2,3,4 keeping both letters and numbers in order. This is much cleaner as a problem of counting combinations than of thinking of it as 7 events of card drawing - events that aren t even independent

Summary Probability is a measure of how much we should believe an event will happen, in the range [0,1]. With equally likely outcomes we can count outcomes to determine probabilities - using all our counting techniques ( E / S ) When drawing without replacement we generally can use combinations to count subsets, but drawing with replacement is more like just counting strings With independent events Pr(A ^ B) = Pr(A)Pr(B) but truly unrelated events can be rare in the wild