Summary of Radom Variable Cocepts April 9, 2000 his is a list of importat cocepts we have covered, rather tha a review that derives or explais them. he first ad primary viewpoit: A radom process is a idexed collectio of radom variables. hat is, a radom process is {X(t): t I}, where I is a ifiite collectio of times (or time idices). For each time t i I, there is a radom variable deoted X(t). Sometimes we write X t istead of X(t). he radom process itself is deoted {X(t): t I}, or by oe of the shorthads: {X(t)}, X(t), X or ay of the previous with X t replacig X(t). Discrete ad Cotiuous ime Radom Processes Discrete-time radom processes have I = {0,,2,...}, I = {,2,3,... } or I = {...,-2,-,0,,2,... }. For discrete-time processes we ofte use the followig otatios {X()}, {X } or replace by some other letter such as i,j,k,m that suggests a iteger. Cotiuous-time radom processes have I = [a,b], where - a<b. Probability Distributio of the Radom Process {X(t)} o kow the probability distributio of a radom process is to kow the joit distributio of every fiite collectio of its radom variables; i.e. to kow the joit distributio of X(t ), X(t 2 ),...,X(t ) for every ad t,...,t I. For example, it suffices to kow the joit cdf, pdf or cmf of X(t ), X(t 2 ),...,X(t ). Kowig the probability distributio of a radom process, we ca compute the probability of ay evet ivolvig the radom process, or ay coditioal probability, or ay expected value. Partial Characterizatios of the Probability Distributio of a Radom Process {X(t)} he probability distributio of a radom process is a awful lot to have to kow or to specify. Cosequetly, we ofte work with oe or more of the followig partial descriptios of its probability distributio.. First-order distributio: his cosists of the margial distributios of every idividual radom variable X(t), t I, as specified, for example, by cdf, pdf or pmf of every idividual radom variable. 2. Secod-order distributio: his cosists of the joit distributio of every pair of radom variables (X(t),X(s)) t,s I, as specified, for example by the joit cdf, pdf or pmf of every pair of radom variables. 3. th-order distributio: his cosists of the joit distributio of every collectio of radom variables (X(t ),...,X(t )), t,...,t I, as specified, for example by specifyig the joit cdf, pdf or pmf of collectio of radom variables. 4. mea fuctio: m X (t) = E X(t) 5. power fuctio: P X (t) = E X 2 (t) 6. autocorrelatio fuctio: R X (t,t 2 ) = correlatio betwee X(t ) ad X(t 2 ) = E X(t ) X(t 2 ) EECS 40 Witer 2000
7. autocovariace fuctio: C X (t,t 2 ) = covariace of X(t ) ad X(t 2 ) = E [(X(t )-m X (t ))(X(t 2 )-m X (t 2 ))] = R X (t,t 2 ) - m X (t )m X (t 2 ) (we derived this formula i class) Some Discrete-ime Examples. IID Radom Process (Idepedet ad Idetically Distributed): Like the ame suggests, the radom variables are idepedet ad idetically distributed. For IID radom processes the complete probability distributio is determied from the probability distributio of just oe radom variable. 2. Beroulli Radom Process: A biary IID radom process. 3. Movig Average (MA) Radom Process: M- Y = bi X i i=0 where {X } is a IID process, M 2 is a iteger, ad b 0,...,b M- are parameters. 4. Autoregressive Radom (AR) Process: N Y = ai Y -i + X N i= where {X } is a IID process, X is idepedet of Y -,Y -2,..., N is a iteger, ad a 0,...,a N are parameters. 5. Autoregressive Radom Movig Average (ARMA) Process: N M- Y = ai Y -i + bi X i i= i=0 where {X } is a IID process, X is idepedet of Y -,Y -2,..., N is a iteger, a 0,...,a N are parameters, M 2 is a iteger, ad b 0,...,b M- are parameters. 6. Biomial Radom Process: Y = Xi i= where {X } is a Beroulli process, with X = 0 or. 7. Gaussia Radom Process: his meas that every fiite collectio of the radom variables is joitly Gaussia. Some Cotiuous-ime Examples. Siusoidal radom process: X(t) = A cos(ωt + Θ) where A ad Θ are idepedet radom variables, ad Θ is uiformly distributed o [0,2π). EECS 40 2 Witer 2000
2. Poisso Coutig Process: Let, 2,... be a IID radom process where each is expoetially distributed with mea /a. is the th iterarrival time. Let S = + 2 +...+. S is the th arrival time. Y(t) = if S t ad S + > t, i.e. if the th evet has occurred by time t but ot the +'th. 3. Radom Process with Fiite Number of Sample Fuctios: Sample fuctios: X(t,) =, X(t,2) = -2, X(t,3) = si πt, X(t,4) = cos πt. hese have probabilities P(), P(2), P(3), P(4) 4. Gaussia Radom Process: his meas that every fiite collectio of the radom variables is joitly Gaussia. 5. White Noise Ra\dom Process: his is a WSS radom process with power spectral desity that is costat with frequecy. he Secod, i.e. Alterative, View of a Radom Process A radom process is a radomly chose sample fuctio. More specifically, a radom process is {X(t,s): t I, s S}, where I is a set of time-idices (as before) ad S is the sample space of some uderlyig radom experimet with probability law P. For each t, X(t,s) is a radom variable. (Recall that model of a radom variable as a fuctio of a uderlyig radom experimet. Note that the value of this model is that all radom variables are viewed as fuctios of the same uderlyig experimet.) For each s S, X(t,s) is a fuctio of t called a sample fuctio. Oe may thik of the radom process as beig geerated i the followig way: At the begiig of time, the uderlyig experimet is performed (whose probability law is P) resultig i a outcome s. he radom process produces the sample fuctio X(t,s) (for this particular s). his is called the sample-fuctio viewpoit. I priciple, oe ca derive the probability distributio of the radom process (as eeded i the first viewpoit) by kowig the fuctio X(t,s) ad the probability law P of the uderlyig experimet. Statioarity A radom process {X(t): t I} is (strictly) statioary if the probability distributio of X(t +τ),x(t 2 +τ),...,x(t +τ) does ot deped o τ for every choice of ad t,...,t. hat is, F X(t +τ),x(t 2 +τ),...,x(t +τ) (x,...,x ) does ot deped o τ ad if the radom process has joit pdf's or pmf's, the same holds for them. he basic idea is that for a statioary r.p. the probability distributios of radom variables (ad vectors) do ot chage with time shifts. he probability of somethig happeig at time t is the same as the probability of it happeig at ay other time. he followig are some of the cosequeces of statioarity: f X(t) (x) = fx(s)(x) all t,s,x f X(t)X(t+τ) (x,x2) = fx(s)x(s+τ)(x,x2) all t, s, τ, x,x2 µ X (t) is the same for all t R X (t,t+τ) does ot deped o t. EECS 40 3 Witer 2000
Wide-sese Statioarity A radom process {X(t): t } is wide-sese statioary (WSS) if m X (t) ad R X (t,t+τ) do ot deped o t. Statioarity wide-sese statioary. he coverse is false. Wide-sese statioarity is a weak kid of statioarity that is easier to check ad to work with, sice it oly depeds o the mea ad autocorrelatio fuctios. Properties of the Autocorrelatio Fuctio of Wide-Sese Statioary (or Statioary) Radom Processes. Symmetry: R X (-τ) = RX(τ) 2. R X (0) R X (τ) for all τ 3. R X (τ) = R decay (τ) + R m (τ) + R periodic (τ), where R decay (τ) is a fuctio such that R decay (τ) 0 as τ, R m (τ) = (m X ) 2 is the term due to the mea of the radom process, ad R periodic (τ) is a periodice fuctio that is itself due to a periodic compoet of the sample fuctios. Ergodicity (this will ot be covered o the exam, but is icluded here for completeess) Recall the law of large umbers. Does it hold for radom processes other tha IID? Sometimes yes, sometimes o. Processes for which it does are called ergodic. Defiitio: (ot the stadard mathematical defiitio) A discrete-time statioary radom process {X(): =,2,...} is (strict-sese) ergodic if i= g(x(i+),,x(i+m) E g(x(),...,x(m)) almost surely as for ay m ad ay fuctio g(x,...,xm) such that E g(x,,xm) is well-defied. A cotiuous-time statioary radom process {X(t): t [0,)} is (strict-sese) ergodic if 0 g(x(t+τi), X(t+τm)) E g(x(τ),...,x(τm)) almost surely as for ay m, τ,...,τm ad ay fuctio g(x,...,xm) such that E g(x(τ),,x(τm)) is welldefied. For "two-sided" discrete- ad cotiuous-time radom processes, the above averages are replaced by 2+ i=- ad 2 -, respectively. he basic idea is that for ergodic processes, time averages coverge to expected values. As examples, the followig are cosequeces of ergodicity. 2. i= X(i) EX, 0 X(t) dt EX i= 2 2 X (i) EX, 0 X 2 (t) dt EX 2 EECS 40 4 Witer 2000
3. 4. i= X(i)X(i+) R X(), 0 X(t)X(t+τ) dt RX(τ) i= X(i)X(i+m) R X(m), 5. A N P(A) where A is ay evet ad A is the umber of times A occurs i X(),...,X() For statioary processes that are ot ergodic, time averages such as those above coverge, but ot to the expected value. Istead, all that we ca say is E g(x(i+),,x(i+m)) i= E g(x(),...,x(m)) as E 0 g(x(t+τi), X(t+τm)) E g(x(τ),...,x(τm)) as Radom Processes ito Liear Filters Here we focus oly o cotiuous-time radom processes ad filters. he situatio is basically the same for discrete-time radom processes ad filters, but we have ot had the time to discuss it. If the wide-sese statioary radom process {X(t)} with mea m X ad autocorrelatio fuctio R X (τ) is the iput to a liear filter with impulse respose h(t) ad frequecy respose H(f), the the output of the filter is a wide-sese statioary radom process {Y(t)} with m Y = m X h(t) dt = m X H(0) - R X (τ) = R X (τ) * h(τ) * h(-τ) recall the def of covolutio: x(t)*y(t) = x(u) y(t-u) du - Importat Fact: If the iput to a liear filter is a Gaussia radom process, the the output is a Gaussia radom process. Power Spectral Desity he power spectral desity of a WSS radom process {X(t)} is S X (f) = F{R X (τ)} = R X (τ) e -j2πfτ dτ = Fourier trasform of R X (τ) - Properties of the power spectral desity:. S X (f) 0. 2. S X (-f) = S X (f). 3. S X (f) dx = power i {X(t)} - EECS 40 5 Witer 2000
f 2 4. 2 S X (f) dx = power i {X(t)} i the frequecy bad [f,f 2 ]. f his is why it is called a power spectral desity; i.e. oe itegrates it over a frequecy bad to obtai the power i that bad. 5. If {X(t)} is the iput to a liear filter with frequecy respose H(f), the the output radom process {Y(t)} has power spectral desity S Y (f) = S X (f) H(f) H(-f) = S X (f) H(f) 2 6. Alterate formula for the power spectral desity (more difficult to work with) S X (f) = lim 2 E F{X (t)} 2, where X (t) = X(t), - t 0, else 7. Power spectral desity of some simple processes: a. X(t) = A, where A is a radom variable S X (f) = E A 2 δ(f). b. X(t) = A cos (2πft+Θ) where A ad Θ are idepedet radom variables ad Θ is uiformly distributed o [0,2π). he S X (f) = 2 E A2 [δ(f-f o ) + δ(f+f o )] c. If the iput to a liear filter with frequecy respose H(f) is white oise X(t) with S Y (f) = c, the the output has power spectral desity S Y (f) = c H(f) 2 EECS 40 6 Witer 2000