CSE 151 Machine Learning. Instructor: Kamalika Chaudhuri

Transcription

1 CSE 151 Machine Learning Instructor: Kamalika Chaudhuri

2 Probability Review Probabilistic Events and Outcomes Example: Sample space: set of all possible outcomes of an experiment Event: subspace of a sample space Toss a coin two times. Sample space is {HH, HT, TH, TT} A = {HH, HT} is the event that toss 1 is a H Measure the temperature. Sample space is (, ) A = [32, 100] is the event that the temperature lies between 32 and 100 degrees F

3 Probability We assign a number P(A) to all events A in a sample space Ω s.t. Axiom 1. Axiom 2. Axiom 3. P (A) 0, for all A P (Ω) =1 If A 1,A 2,...are disjoint, then P ( i A i )= i P (A i ) Such a function P(A) is called a probability distribution Example: Two tosses of a coin. P(HH) = P(HT) = P(TH) = P(TT) = 0.25

4 Independence and Conditional Probability Two events A and B are independent if P (A B) =P (A)P (B) A set of events A1,..,Ak are independent if k P ( k i=1a i )= P (A i ) If P(B) > 0 then the conditional probability of event A given B is P (A B) P (A B) = P (B) i=1 If A, B are independent, what is P(A B)?

5 Random Variables A random variable X is a mapping that assigns a real number to each outcome ω in a probability space X(ω) X Example: Flip a coin two times, and let X be the number of heads Measure the temperature two times and let X be the sum of the measurements We will look at discrete and continuous random variables

6 Cumulative Distribution Functions A cumulative distribution function (cdf) of a random variable X is a function FX defined by: F X (x) =P (X x) Example: X is the number of heads in two tosses of a fair coin 1 cdf

7 Probability Mass Functions The probability mass function (pmf) of a discrete random variable X is a function fx defined by: f X (x) =P (X = x) Example: X is the number of heads in two tosses of a fair coin What is fx? fx(0) = 0.25, fx(1) = 0.5, fx(2) = 0.25, fx(i) = 0, otherwise

8 Examples of pmfs Binomial distribution: Bin(n, p) n f X (k) = p k (1 p) n k, 0 k n k f X (k) =0, otherwise Exercise: How will you verify this is a pmf? Exercise: How will you calculate the cdf?

9 Examples of pmfs Geometric distribution: Geom(p) f X (k) =p(1 p) k 1,k=1, 2, 3,... f X (k) =0, otherwise Exercise: How will you verify this is a pmf? Exercise: How will you calculate the cdf?

10 Probability Density Functions The probability density function (pdf) of a continuous random variable X is a function fx such that: 1. f X (x) 0, for all x f X (x)dx =1 For all a b, P (a X b) = b a f X (x)dx For a continuous random variable X, the cdf FX(x) is: F X (x) = x f X (t)dt

11 Examples of pdfs Uniform(a,b): f X (x) = 1 b a,a x b f X (x) =0, otherwise Exercise: Verify this is a pdf Exercise: How will you calculate the cdf?

12 Examples of pdfs Normal: N(µ, σ 2 ) f X (x) = 1 σ 2 2π e (x µ) /2σ 2 Exercise: Verify this is a pdf Standard normal: normal distribution with mean 0 and stdev 1 Φ(z) : cdf for a standard normal Properties: 1. If X is N(m, s 2 ), then Z = (X - m)/s is a standard normal 2. If Xi is N(mi, si 2 ), and Xis are independent, then Z = X Xk is N(m1+..+mk, s sk 2 ) 3. If Z is N(0,1) then m + sz is N(m, s 2 )

13 Examples of pdfs Exponential: Exp(β) f X (x) = 1 β e x/β,x>0 f X (x) =0, otherwise Exercise: Verify this is a pdf

14 Bivariate Distributions In the discrete case, a function f(x, y) is a joint mass function for random variables (X, Y) if: f(x, y) = Pr(X = x, Y = y)

15 Bivariate Distributions In the continuous case, a function f(x, y) is a joint density function for random variables (X, Y) if: 1. f(x, y) 0, for all x, y f(x, y)dxdy =1 For any set A, P ((X, Y ) A) = A f(x, y)dxdy

16 Examples of joint pdfs Uniform: f(x, y) =1, 0 x 1, 0 y 1 f(x, y) =0, otherwise Exercise: Verify this is a pdf Exercise: What is P(X > 0.5, Y < 0.5)?

17 Examples of joint pdfs Multivariate Normal: f(x, µ, Σ) = 1 (2π) d/2 det(σ) 1/2 exp( 1 2 (x µ) Σ 1 (x µ)) Properties: 1. Standard multivariate normal has mean 0d and covariance Id 2. If Z is standard multivariate normal, then X = m + S 1/2 Z is N(m, S) More properties as exercise

18 Marginal Mass & Density Functions In the discrete case, if random variables (X, Y) have a joint mass function fxy, then the marginal mass function for X is: f X (x) =P (X = x) = y P (X = x, Y = y) = y f XY (x, y) What is the marginal mass function for Y? In the continuous case, if random variables (X, Y) have a joint mass function fxy, then the marginal density function for X is: f X (x) = f XY (x, y)dy What is the marginal density function for Y? y

19 Conditional Mass & Density Functions In the discrete case, the conditional mass function for X is: if fy(y) > 0 f X Y (x y) =P (X = x Y = y) = P (X = x, Y = y) P (Y = y) = f XY (x, y) f Y (y) In the continuous case, the conditional density function for X is: if fy(y) > 0 f X Y (x y) = f XY (x, y) f Y (y)

20

21 Expectation For a discrete random variable X, the expectation is defined as: E[X] = x xf X (x) For a continuous random variable X, the expectation is defined as: E[X] = xf X (x)dx Let Y = r(x). Then E[Y] can be computed as: E[Y ]= r(x)f X (x)dx E[Y ]= r(x)f X (x) x continuous discrete

22 Variance For a random variable X, the variance is defined as: Var(X) =E[(X E[X]) 2 ] Property: E[(X E[X]) 2 ]=E[X 2 ] (E[X]) 2 Exercise: Prove the property

23 Independence of random variables, Covariance Random variables X and Y are independent if for any two sets A and B, P (X A, Y B) =P (X A)P (Y B) For two random variables X, Y, cov(x, Y) = E( (X - E[X]) (Y - E[Y]) ) cov(x, Y) can also be written as: cov(x, Y) = E(XY) - E(X) E(Y) Property: If X and Y are independent, then Var(X + Y) = Var(X) + Var(Y) Cov(X, Y) = 0 Does the converse hold?