ADSP ADSP ADSP ADSP. Advanced Digital Signal Processing (18-792) Spring Fall Semester, Department of Electrical and Computer Engineering

Transcription

1 ADSP ADSP ADSP ADSP Advaced Digital Sigal Processig (8-79) Sprig Fall Semester, 7 Departmet of Electrical ad Computer Egieerig OTES O RADOM PROCESSES I. Itroductio Radom processes are at the heart of most techiques developed i ADSP. They are a atural extesio of the cocept of radom variables i probability theory. I these otes we briefly ad superficially review some of the cocepts discussed i the lectures. These topics are treated i greater detail i Appedix A of Oppeheim, Schafer, Yoder, ad Padgett (, hereiafter referred to as OYSP), ad i somewhat greater detail i Chapter 8 of Oppeheim ad Schafer (975), which is available o the course web site. There are, of course, may complete text ad referece books o radom processes. II. Brief review of probability ad radom variables Probability theory is used to describe some thigs that are difficult to develop usig classical techiques ad for which we oly care about characterizig outcomes i useful fashio. For example, tossig a coi is actually othig more tha a very complicated problem i ewtoia mechaics: if we kew the positio ad velocity of a coi whe we throw it up i the air, ad we kow the locatio ad mechaical reflectace of all the objects i a room, we should be able to predict whether it will lad heads or tails. But it is hard to work the problem that way, ad the truth is that we do t really care. A more useful characterizatio is to simply state that each side comes up about half the time, ad that successive tosses are idepedet. We could make a similar argumet with dice, roulette wheels, etc. The importat thig is that probability is just a mathematical characterizatio that is both cosistet ad useful. A. Probabilities of evets May discussio of probability theory begi with a discussio of probabilities of evets. Evets A i that are mutually exclusive ad assiged probabilities PA i that are umbers betwee ad that sum to over all possible evets. We also talk about the joit probabilities of two evets PAB as well as the coditioal probability of oe evet give the other, PAB. These are related accordig to the equatio Sice PAB PAB PB

2 8-79 otes o Radom Processes -- Fall, 7 PAB PAB PB PBA PA, we ca write the well kow Bayes rule, PBA PAB PB PA B. Radom variables May iterestig thigs that are modeled by probability theory ca be described as radom experimets of which the outcome is a umber. These outcomes are referred to as radom variables. For example, the radom variable x may represet the temperature outside the frot door of my home at 9 am today. This umber (which could i priciple be ay real umber but is much more restricted i practice) is ukow a priori but we have some idea of what it might be like statistically, give the time of year. (The temperature i September is likely to be differet from the temperature i Jauary!) Aother example of a radom variable is the iteger umber represetig the umber of passegers that are i the first bus that stops at the corer of Fifth ad Highlad Aveues, ot far from my home. We will use the probability desity fuctio (pdf) to represet the radom variable x. ote that the ame of the radom variable is idetified by the subscript of p, ad the dummy variable represets possible values that x may take o. The probability desity fuctio is related to probabilistic evets by the equatio Pa x b p x d ote that p x caot be egative, ad its itegral over all values of must be. We ca defie coditioal probability desity fuctios p xa A ad joit probability desity fuctios p xy i similar fashio. ote that the margial probability desity ca be obtaied by itegratig the joit probability desity fuctio p xy over all possible values of. C. Some other useful relatioships b a p x p x You should have covered may useful relatioships ad properties of radom variables i your prior courses i probability theory. For example, two radom variables x ad y are statistically idepedet if their joit pdf ca be factored ito two margial pdfs: p xy p x p y The expected value (or statistical average) of ay fuctio of x, Egx gp x d gx ca be obtaied by Certai statistical averages will be used widely i our discussios. For example, the mea of x is Copyright 7, Richard M. Ster

3 8-79 otes o Radom Processes -3- Fall, 7 m x Ex, the variace of x is x Ex m x Ex m x, the stadard deviatio of x is x, ad the covariace of two radom variables x ad y is xy Ex m x y m y. III. Itroductio to radom processes The key thig to keep i mid is that the theory of stochastic processes ca be cosidered to be a extesio of the theory of probability, except that i stochastic processes the outcome of the radom experimet is a time fuctio x (called a sample fuctio ) that i priciple exteds over all time. I cotrast, the outcome of the radom experimet i probability theory is a umber. The value of a sample fuctio at a particular time x, which will also be otated from time to time as x, is a radom variable described by the pdf p x or p x. (Lametably, either of these otatioal covetios is very elegat!) Examples of sample fuctios of a radom process are show i Figs.,, ad 3, which depict broadbad oise, speech, ad a radom-phase cosie wave, respectively. ote that i all cases the sample fuctios actually exted from to i time, ad oly a brief portio is depicted of ecessity. (I other words, we ca oly observe oe sample fuctio of ay particular radom process i our lifetime!) ote also that i the case of the radom-phase cosie the sample fuctios appear to be determiistic. evertheless, they are radom because the phase of each cosie is a radom parameter. x[] x[] x[] Figure. Sample fuctios of a radom process. Copyright 7, Richard M. Ster

4 8-79 otes o Radom Processes -4- Fall, 7.5 x[] x[] x[] Figure. Sample fuctios of a radom speech waveform. x[] x[] x[] Figure 3. Sample fuctios of radom-phase cosie waves. Copyright 7, Richard M. Ster

5 8-79 otes o Radom Processes -5- Fall, 7 A. First-order ad secod-order esemble averages Esemble averages (commoly kow as expected values) describe the average values of radom variables (icludig the values of radom processes at specific times) averaged over the esemble. Most of the importat characteristics of radom processes that we care about, such as their meas, variaces, covariaces are actually esemble averages. I geeral, the expected value of ay fuctio g of a radom variable x is defied as Egx gp x d Some importat statistical averages based o x iclude Mea m x Ex p x d Autocorrelatio xx, Ex x p x x dd Crosscorrelatio Commets: xx xx m x m x Autocovariace xx, E x m x x m x m x mx p x x dd xy Ex y p x y Crosscovariace xy, E x m x y m y m x my p x y dd d d The variace of x is x xx B. Basic radom process properties Copyright 7, Richard M. Ster

6 8-79 otes o Radom Processes -6- Fall, 7 Ex + y Ex + Ey always, because of the liearity of the expectatio operator Eax aex agai because of the liearity of expectatio Varx + y Varx + Vary oly if the radom processes x ad y are statistically idepedet C. Statioarity Figure 4. Sample fuctios of a statioary radom process. Cosider the three radom processes with sample fuctios show i Figs. 4, 5, ad 6. The sample fuctios i Fig. 4 appear to be more or less the same over all time, ad for these sample fuctios the time origi (i.e. ) could be shifted without visibly affectig the properties of the sample fuctios. I cotrast, the sample fuctios i Fig. 5 appear to be driftig upward after about Sample 5, while the sample fuctios i Fig. 6 appear to become more variable after about Sample 5, ad for these processes the temporal origi is meaigful (ad is about 5 samples before the chages i the sample fuctios are observed). We characterize the property that the first process has ad the latter two do ot have as statioarity. We typically defie two types of statioarity, wide-sese ad strict-sese statioarity. A radom process is strict-sese statioary (SSS) if. Ex Ex m x. xx depeds oly o, so we ca write xx xx 3. For ay collectio of times 3,... etc., ay joit momet Ex x x 3 etc etc, Copyright 7, Richard M. Ster

7 8-79 otes o Radom Processes -7- Fall, 7 Figure 5. Sample fuctios of a radom process with icreasig mea. Figure 6. Sample fuctios of a radom process with icreasig variace. Copyright 7, Richard M. Ster

8 8-79 otes o Radom Processes -8- Fall, 7 the result depeds oly o the various differeces betwee the times i questio. A radom process Commets: xt is wide-sese statioary (WSS) if oly the first two coditios above are satisfied. Statioarity i geeral (i.e. strict-sese statioarity ) requires that similar relatios hold true for higher order momets of the radom process as well. It is usually quite hard to prove. Kowig that a radom process is WSS will be good eough for us almost all of the time. Coceptually, statioarity meas that the locatio of the time origi does ot affect the statistics of the process, as oted above. Some properties of statioary radom processes: xx m xx m m x xx Ex xx x, the variace of the radom process lim xx m m x m lim xx m m xx m xx m for xx m xx D. Time averages While esemble averages are well defied ad mathematically cosistet, they are ot very helpful i practice because we ever ca look at multiple sample fuctios, as metioed above. Istead, we estimate esemble averages by their correspodig time averages, which are meaigful oly if a radom process is statioary. Specifically, for a statioary radom process, we defie the time average of a arbitrary fuctio g of the radom process x as gx lim gx + Some importat time averages iclude Mea x lim x + Copyright 7, Richard M. Ster

9 8-79 otes o Radom Processes -9- Fall, 7 Autocorrelatio x x + m lim x + x + m ad so o for other time averages Cosiderig the diagrams of the sample fuctios we have bee showig, it is useful to thik of esemble averages as vertical, fixig the time (or times) ad averagig (vertically) over the esemble of sample fuctios. I cotrast, time averages are obtaied by averagig a particular fixed sample fuctio horizotally over time. E. Ergodicity Sice the theory of radom processes is based o esemble averages but we must estimate the esemble averages by measurig time averages, it is desirable that the correspodig time ad esemble averages are all equal. I other words, we would like it if This property is called ergodicity. A radom process is ergodic if its correspodig time ad esemble averages are all equal. Ergodicity is geerally hard to prove. Coceptually, ergodicity meas that the statistical attributes of each sample fuctio is reicarated i each of the other sample fuctios sooer or later. Most statioary radom processes are also ergodic, but there are some spectacular exceptios to this rule. F. Examples: Egx gx lim gx + Cosider the example of the radom phase cotiuous-time cosie xt cost +. Calculate the esemble average mea ad autocorrelatio: Ext xt p x d This is much more easily doe usig the parametric defiitio: Ext E[x t p d cost d Similarly, Ext xt -- cost where t R xx t,t R xx t t Copyright 7, Richard M. Ster

10 8-79 otes o Radom Processes -- Fall, 7 [you should be able to show this easily] ow do a similar example but with time averages. They come out the same, so we have ergodicity, at least i the secod-order sese. Secod example: ow cosider the very simple discrete-time radom process x k, where k is i tur a radom variable, for example a Gaussia with mea zero ad variace. Show that x is a statioary RP. But ote that the time average mea of each of the sample fuctios are all differet (ad equal to the value of the sample fuctio, of course). But the time average meas are geerally ot the same as the esemble average mea which would be zero for the case that I suggested, so the process is statioary but ot ergodic. This is uusual, but there are a few other similar examples (mostly early as trivial) that ca be produced. IV. Gaussia radom processes A radom process is Gaussia if the radom variables formed by samplig the radom process at arbitrary istats of time all are joitly Gaussia radom variables. Although this may seem like a circular defiitio, it really is t... to test for Gaussiaity, it is oly ecessary to show (or be told) that the pdf of the correspodig radom variables are really Gaussia. I other words, Idividual radom variables are Gaussia if p x m x exp x x For Gaussia radom vectors we geerally use the matrix for of the pdf: p x exp T -- m C x Cx m x x Commets: A Gaussia radom process is ot ecessarily statioary If a Gaussia radom process is wide-sese statioary, it is also strict-sese statioary. (This is because higher-order momets of Gaussia radom variables ca always be factored ito first-order ad secod-order momets.) V. Power Spectral Desity Fuctios The power spectral desity fuctio is the mathematical represetatio that eable us to characterize the frequecy distributio of a radom process. As you will recall, the total eergy of a discrete-time fuctio may be represeted as Eergy x Copyright 7, Richard M. Ster

11 8-79 otes o Radom Processes -- Fall, 7 The problem with the use of eergy is that all statioary radom processes have ifiite eergy. This is because statioarity implies that E x R xx is a costat that is idepedet of, so whe we sum that quatity over all time the result becomes ifiite. (Determiistic periodic time fuctios also have ifiite eergy for basically the same reasos.) A more appropriate measure is power, which refers to eergy divided by time. We ca defie the power associated with a sample fuctio of a radom process x as Power lim x ote that this is also the time average of the quatity x, so if the radom process is ergodic, the total power will also be equal to the esemble average E x, or Ex if the radom process has real sample fuctios. While the total power is obtaied from this average, we will cosider ow the very importat power spectral desity fuctio, which describes how the total power is distributed over the costituet frequecy compoets. To provide greatest isight, we will cosider three approaches to the defiitio of power spectral desity fuctios. x H BP (e jω ) Δ H BP (e jω ) Δ ω π ω ω π Physical defiitio. Cosider the system i the figure above. Assume that the iput to the filter x is a statioary radom process with a particular distributio of power respect to frequecy, ad with a total power equal to x. The filter is a ideal badpass filter with badwidth ad ceter frequecy. As becomes small, the amout of power i the sigal that emerges from the output of the filter is equal by defiitio to x P xx I the expressio above, the fuctio P xx represets the power spectral desity fuctio of the radom process x at the frequecy. The factor of i the umerator represets the fact that both positive ad egative frequecies are passed through the filter. The factor of is eeded because the total Copyright 7, Richard M. Ster

12 8-79 otes o Radom Processes -- Fall, 7 power i x is obtaied by itegratig P xx over all frequecies. Total Power P xx d I other words, the power spectral desity fuctio is proportioal to the amout of power i a radom process that is preset at a give frequecy. Fourier trasform defiitio. Recall that we defied the total power as Power lim x From Parseval s theorem we ca also derive the power spectral desity fuctio as what we get whe we compute the magitude squared of the DTFT of the sample fuctios widowed over a fiite duratio, divided by the duratio over which we are lookig at (because we are computig power spectral desity, ot eergy spectral desity). We the compute the expected value of the result over all of the sample fuctios of a radom process, ad the takig the limit as the duratio goes to ifiity. Hece we ca defie the power spectral desity fuctio P xx of a real radom process x as P xx lim E x e j Autocorrelatio fuctio defiitio. Let us ow assume that the radom process x is real. By expadig the magitude squared expressio, we obtai P xx lim E x + e j lim E + x e j x l e jl l P xx lim E x + x l e j l l Lettig m l we ca rewrite this as P xx lim E x + x m + e jm l m xx me jm m Replacig the variable m by m we obtai Copyright 7, Richard M. Ster

13 8-79 otes o Radom Processes -3- Fall, 7 P xx xx me jm m xx me jm m The latter equality is valid because, agai, the radom process x is assumed to be real. I other words, the power spectral desity fuctio of a real radom process is the DTFT of the correspodig autocorrelatio fuctio. This relatioship is kow as the Wieer-Khichie theorem, ad it is typically the easiest way to evaluate power spectral desity fuctios. A radom process is said to be white if it has a power spectral desity fuctio that is a costat over all frequecy, i.e. This implies that the autocorrelatio fuctio of a white radom process is a impulse: Sice a white radom process must also be zero mea (otherwise there would be a ipulse i the power spectral desity fuctio at ), this implies that the values of a sample fuctio of a white radom process at differet times are all statistically idepedet. VI. Radom processes ad liear filters Itroductio: P xx x xx m xm x h[] y Let us ow cosider what happes to the statistics of a radom process as it is passed through a LSI system. Specifically, let us assume that a wide-sese statioary radom process x is iput to a liear filter with uit sample respose h producig the output process x. As i the case of determiistic sigals, the iput ad output are related by the covolutio sum: y hk x k k The mea of the output is easily obtaied as Copyright 7, Richard M. Ster

14 8-79 otes o Radom Processes -4- Fall, 7 m y Ey E hkx k hk Ex k hk m x hk k k k k The autocorrelatio of the output is straightforward but requires a little more algebra: yy m Ey y + m E hkx k hl x + m l k l or yy m hk hl xx + m l k l ow let r l k or l r+ k. Substitutig, we obtai yy m xx m r hk hr + k r k The ier sum, hk hr + k ca be cosidered to be the covolutio of the uit sample respose h k with itself time-reversed, which is actually the (uormalized) autocorrelatio fuctio of the determiistic h. Recogizig this, we ca rewrite the equatio for the output autocorrelatio as yy m xx m r hh r xx m hh r r xx mhmh m So, the autocorrelatio fuctio of the output radom process is obtaied by covolvig the autocorrelatio fuctio of the iput covolved with the uit sample respose ad covolved agai with the uit sample respose time reversed. Fially, otig the Fourier trasform pairs hm He j h m He j xx m P xx we ca obtai a relatio betwee the iput ad output power spectral desity fuctios: yy m xx mhmh m P xx He j He j P xx He j I other words, the output PSD equals the iput PSD multiplied by the magitude squared of the trasfer fuctio of the filter. It ca also be show quite easily that the cross-correlatio fuctio betwee the iput ad the output ca be writte as Copyright 7, Richard M. Ster

15 8-79 otes o Radom Processes -5- Fall, 7 xy m Ex y + m xx mhm ad P xy P xx He j Copyright 7, Richard M. Ster