Online Determination of Noise Level in Weather Radars

Transcription

1 Online Determination of oise Level in Weather Radars Igor R. Ivić,, and Sebastián M. Torres, Cooperative Institute for Mesoscale Meteorological Studies (CIMMS), The Universit of Oklahoma, 0 David L. Boren Blvd., orman, OK 7307, USA OAA/OAR ational Severe Storms Laborator, orman, Oklahoma, 0 David L. Boren Blvd., orman, OK 7307, USA Igor Ivić. Introduction All receivers detect signals that are at some level above the limit imposed b the random voltage (i.e., the noise) inherent in ever electronic device. Consequentl, proper measurement of noise power is of paramount importance for the estimation and censoring of the weather radar data, which, in turn, is essential for the correct operation of automated algorithms and accurate forecasts derived from such data. Incorrect noise power measurements ma lead to reduction of coverage in cases where noise power is overestimated or to radar data images cluttered b noise speckles if the noise power is underestimated. Consequentl, the correct noise measurement is essential for proper operation of censoring techniques in both single and dual-polarized radars (Ivić and Torres 009, Ivić et al. 009). Moreover, when an erroneous noise power is used at low signal-to-noise ratios (SR), estimators usuall produce biased meteorological variables such as in the case of reflectivit and spectrum width. Tpicall, the noise in weather radars can be measured in several was. For instance, on the ational Weather Surveillance Radar 988 Doppler (WSR-88D) the noise is measured as part of the sstem online calibrations performed after each volume scan. Such measurement takes place at a high antenna elevation angle and the result is adjusted for other antenna elevations. In sstems that do not have the capabilit to perform online calibrations (e.g., the ational Weather Radar Testbed Phased-Arra Radar) the noise must be measured offline. Clearl the downside of such approach is that it does not capture the temporal variations of noise power. Moreover, the nature of the noise sources in radar data is such that noise can have angular dependence in both azimuth and elevation (e.g., noise from cosmic radiation and from the oxgen and water vapor molecules) (Doviak and Zrnić 993). Consequentl, the benefit of noise measurements at each antenna position becomes obvious. The onl wa such measurement can be performed operationall is in parallel with data collection. Thus, an efficient approach that estimates noise power from measurements that contain both signal and noise is needed. Ideall, noise power estimates should be computed for ever sampling volume, for example, b using spectral noise estimation methods. In the past several methods have been proposed. Hildebrand and Sekhon (974) describe a method that subjects the Fourier coefficients to a series of tests whereb coefficients are recursivel discarded until statistical conditions suggest onl noise samples remain. Urkowitz and espor (99) used the Kolmogorov-Smirnov (K-S) test applied to the periodogram b successivel discarding the Fourier spectral lines until the criterion for the noise hpothesis is satisfied. Siggia and Passarelli (004) used rank order statistics on power spectral densit estimate to dnamicall determine the noise level. Common to all these approaches is discarding excess Fourier coefficients until the remaining ones satisf conditions for noise. Inevitabl, each approach introduces bias in noise level determination even when no signal is present and particularl for radar volumes with weather signals that have wide spectrum widths or if using a small number of samples in the dwell time. This can not be avoided and the onl question is how significant the bias is. Hence, a need arises for a more precise and continuous sstem noise power calibration that is robust and feasible for real-time implementation on weather radars. In this paper, we propose a novel method to estimate the sstem noise power dnamicall from the in-phase and quadrature data for ever antenna position (radial). The technique uses a novel criterion to detect radar volumes that do not contain significant weather signals and uses those to estimate the sstem noise power. The proposed method overcomes the limitations of the previous work in cases when the numbers of samples at each range position is small because it does not use the Fourier coefficients. Moreover, this is usuall the case in the surveillance mode where the unambiguous range is long and the number of range positions devoid of signal is more than sufficient for qualit noise estimation resulting in the algorithm performing ver well. This technique is evaluated using a time-series collected with the ational Weather Radar Testbed Phased-Arra Radar (WRT) and the research WSR-88D KOU radar, both located in orman, OK. Results show that the proposed technique produces noise power estimates that are closel matched to the ones obtained from manuall identified, signal-free radar volumes at far ranges from the radar; thus, providing empirical validation. A real-time

2 implementation of this technique is expected to significantl improve the data qualit of operational weather radars which often rel on accurate noise power estimates.. oise estimation algorithm Unlike the previousl discussed approaches to noise estimation, the algorithm presented in this paper does not produce a noise power estimate for each radar volume. Rather, it attempts to discard all samples at range locations where the presence of signal is detected. The assumption is that there are enough range bins devoid of signal to ield noise estimates with satisfactor accurac. This is almost alwas true when using long pulse repetition times (PRT), which result in unambiguous ranges in excess of 300 km. On the other hand, when the PRT ields shorter unambiguous ranges, it is possible that the majorit of samples contain signal as storms span or exceed the entire unambiguous range. In such cases, the algorithm is unable to produce reliable noise estimates. For dual PRT scans (i.e., those using a long PRT for range coverage and a short PRT for Doppler velocit measurements), if the noise power at a given antenna position can not be estimated from the short PRT data, it is usuall readil available from the long PRT. To illustrate the steps of the noise estimation algorithm we will use data collected with the ational Weather Radar Testbed Phased-Arra Radar (WRT PAR) in orman, OK. This particular set of data was collected with a long PRT with unambiguous range of km. Radar echoes are oversampled b a factor of 4, where samples are 60 m apart and the transmitted pulse is roughl 40 m long. The number of samples in the dwell time is 5. This radar does not perform online calibrations so the default noise level is ascertained b offline measurements. The test power profile is shown in FIG.. B visual inspection, we see that there should be no signal beond the 5000-th sample. B averaging the power of all samples beond 5000, 5500, and 6000 we get 8.06, 7.64 and 7.6. Thus, we assume that the true noise power is 7.6 (.46 db) in this case Initial noise level FIG. Received power as a function of range at the elevation angle of 0.5 deg. The number of samples at each range position is 5 and the range sample spacing is 60 m. The initial and true noise values are indicated with a red and green line, respectivel. This data was collected using the WRT PAR in orman, OK. The first step in the algorithm requires an initial knowledge of the noise power (herein referred to as the initial noise level for the purposes of the algorithm) which should be a rough estimate in the vicinit of the true noise power (e.g., within ± db). This initial noise level is used to obtain the coherenc-based threshold (CBT) (Ivić and Torres 009) b simpl multipling it with the value chosen from the look-up table (where the number of samples is used as an entr into the table). Then, all samples at range positions classified to contain signal-like returns are discarded. In this particular case the threshold produces the false alarm rate of ; hence, if the initial noise value were the true noise value, such threshold would produce about 44 false signal detections in 0 million. Because the true noise power is lower than the initial one, the number of false signal detections is even smaller. On the other hand, if the initial noise value were lower than the true one, the number of false detections would be higher. Here, we are interested in those samples that are classified as noise (i.e., those that are censored b the CBT). Thus, it is important that the number of false detections is not too high to prevent too man noise samples being classified as signals (hence not used for noise estimation) which can potentiall bias the noise estimate. Choosing a CBT threshold that ields low false detection rate ensures this does not occur for a wide range of initial noise values. In this case,

3 however, the default absolute noise power for the WRT PAR is 9.5 (.9 db); hence, the initial noise power is overestimated b the offline procedure. The second step refines the outcome of the first one b using the autocorrelation coefficient (ACF) at lag. In this step, the lag- ACF is calculated at each range position and those for which it is larger than the predetermined threshold are deemed to contain signal and related samples are consequentl discarded. The ACF threshold is set to pass 99% of the noise samples so it does not bias the noise estimate. The threshold value depends on the number of samples in the dwell time (M) and in this particular case it is Because the autocorrelation coefficient is not dependent on the noise and signal powers, this step discards samples at range positions containing highl correlated signals. After appling the first two steps, the remaining noise-like range bins are shown in FIG.. It is obvious that a significant amount of range positions still contain signal-like returns. This is reflected b the mean power of this data set which is 3.0 (3.6 db); thus, it is still well above the far range or true noise level. The next step is to appl a range persistence filter. The filter finds 0 or more consecutive power values that are larger than the median power in the set and discards them along with 0 samples on either side. The rationale for choosing 0 consecutive samples is as follows. The probabilit that one power sample is larger than the median is 0.5; hence, the probabilit that 0 randoml chosen independent samples are larger than the median is = Consequentl, this filter should detect and remove larger sample powers (evident of signal-like returns) that exhibit some continuit in range while leaving those in predominantl in noise areas. After appling the range persistence filter to our sample data set, we obtain FIG. (b). The mean power of the resulting set is 9.69 (or.94 db), which is still higher than the true noise power for this case (b) FIG. Power of range bins after the first two steps of the algorithm. (b) Power of range bins after the third step of the algorithm. In the fourth step, the matrix of samples (range time vs. sample time) is reshaped into a vector where the samples from to M belong to the first column of the matrix, samples M+ to M belong to the second column of the matrix and so on. Then, a running average of K samples is performed as RAVG m K ( ) = V( m k), for m K, () K k= 0 where V(k) are the elements of the reshaped samples vector. Because m K, the first K elements are alwas discarded. K is chosen to be 750. FIG. 3 shows the results of this step on the sample data set. ote that this makes the part of the range profile where signal is still present more evident. In a noise-onl case the probabilit that one averaged point is larger than the mean times D is (Ivić and Zrnić 009) ( ) K K K K p e p dp =Γinc ( D K, K ) K!. () D

4 When D is. (or 0% of the mean noise power) and K is 750 this probabilit is 0.38%. The mean is found from the data after the range persistence filter. Averaged points that are larger b more than 0% of the so found mean (herein referred to as outliers ) are detected and all samples that went into the averaged points are discarded. This is repeated while the number of discarded samples is larger than times the total number of samples or up to a maximum of five iterations. In this particular case the outlier filtering is performed twice. The results are shown in FIG. 3 (b). The mean power of this data set is Finall, instead just using plain average, the mean power is obtained using rank ordered statistics as described in Appendix A. In this particular case this ields a noise power estimate of 7.67, which is in almost perfect agreement with the true or far range noise level. The steps of the algorithm are summarized below: ) Censor using coherenc based thresholding (Ivić and Torres 009) with the initial noise level. ) Censor using the autocorrelation coefficient (ACF) with the threshold set to pass 99% of noise samples based on the number of pulses per dwell (M). 3) Run range persistence filter that detects 0 or more consecutive samples with power larger than the median and discards them plus 0 surrounding samples on each side. 4) Reshape all samples into a one dimensional arra. 5) Obtain mean power. 6) Perform running average of 750 points. 7) Discard samples used to obtain outlier averaged points (i.e., those larger than. times the mean power from 5). 8) If the number of discarded samples is smaller than times the total number of samples or the number of iterations of steps 5, 6, and 7 is 5, proceed to step 9. Otherwise, go back to step 5. 9) Estimate the noise power using rank ordered statistics as described in 0) ) Appendix A x 0 4 x 0 4 (b) FIG. 3. Power of range bins after the first iteration of the fourth step of the algorithm. (b) Power of range bins after the second iteration of the fourth step of the algorithm. 3. Performance examples In this section, examples of the algorithm performance on time-series are presented. The first set of data is collected with the ational Weather Radar Testbed Phased-Arra Radar (WRT). This radar does not perform online calibrations thus if noise is not estimated from data the initial value of 9.5 (obtained b offline measurements) is used for product generation. The presented data is from a dual PRT tilt with a long PRT of 3.04 ms and a short PRT of ms, at an elevation of 0.5 deg.

5 ABSOLUTE OISE POWER Estimated noise Default noise AZIMUTH (deg) FIG. 4. oise estimates compared to the initial (default) and the far range ( true ) noise, and (b) reflectivit field with additional detections obtained using estimated noise highlighted in white. The algorithm is set so that if fewer than 000 samples remain in step 8, the estimator reports it is unable to produce an estimate. B imposing the requirement that the estimates are made from at least 000 samples, the algorithm is prevented from producing results when the majorit of samples contain signal. Additionall, if exactl 000 samples are used for estimation, the estimate is within ±0% of the true mean with probabilit. In this particular case, the algorithm fails to produce results from the short PRT data rather often. Consequentl, if the short PRT estimate is not available, the one from the long PRT is used. In this example however, even in the cases when the algorithm produces results from the short PRT data, the samples are usuall heavil inundated with signal so the noise power is frequentl overestimated. Consequentl, the short-prt and long-prt estimates are compared and the two results are combined onl if the are within 0% of each other. Otherwise, onl the long PRT estimate is used. In FIG. 4, the estimated noise is plotted with the one obtained from the distant ranges free of visible signals (the true noise). This shows that the proposed noise power estimator is reliable and robust as it produces values close to the true noise power. FIG. 4 (b) shows the reflectivit field obtained using the estimated noise power and using CBT for detection (Ivić and Torres 009). The additional detections obtained using the estimated noise power as opposed to the default one are highlighted in white. Another example shows the surveillance scan data collected b the dual polarized KOU WSR-88D research radar in orman, OK. The PRT used for this scan was 3. ms with 7 samples (M = 7) per dwell. This radar performs online calibrations which attains the initial noise value. The comparison between the far range and the estimated noise is given in FIG. 5. As in the previous case, it is apparent that the estimated and the true noises agree ver well. Moreover, the noise estimation procedure filters out the jumps in the far range noise (most likel caused b point target interferences). FIG. 5 (b) shows the reflectivit field obtained using the coherenc based censoring technique for dual-polarized radars (Ivić et al. 009). Additional detections resulted from the use of the noise estimation algorithm are highlighted in white. In this particular case, the noise obtained b the online calibration is about db larger than the one produced b the dnamic estimation. 4. Summar A method to estimate noise power dnamicall from data was presented. Through a set of steps, the algorithm classifies samples as containing signal or not. The approach requires an initial rough guess on the noise power. In sstems that use online calibrations such value is alread available. Other possibilit is to have it measured offline. First, coherenc based detection using the initial noise guess is applied to the data. Then, the range positions which exhibit strong correlations along sample-time are discarded using the measured autocorrelation coefficient. Range positions with powers that are consecutivel higher than the median one are disposed of next. At this point all the remaining samples are rearranged into a long vector and the running average is performed to make the remaining weak signal areas more visible. To dispense with the remaining signal, all samples associated with the averaged points, larger than 0% of the mean power, are discarded. The last step is repeated until the ratio of the number of the discarded samples and the total ones falls under the probabilit that the averaged point exceeds. times the mean power value (i.e., ). The noise estimate is produced from the remaining samples using rank ordered statistics. In the implementation presented in this paper, the minimum number of the remaining samples required to produce

6 reliable estimate is set to be 000. The algorithm accurac was verified b comparing its results to the noise powers obtained from the data at the far range positions devoid of visible signals. Such comparison shows that the technique produces noise powers with improved accurac as opposed to offline and online calibrations with minimal bias. ABSOLUTE OISE POWER 4.4 x H far range noise H estimated noise V far range noise V estimated noise AZIMUTH (deg) FIG. 5. oise estimates compared to the far range noise. (b) Reflectivit field with additional detections, resulting from the use of the noise estimation algorithm, highlighted in white. Acknowledgment Funding for part of this research was provided b OAA/Office of Oceanic and Atmospheric Research under OAA-Universit of Oklahoma Cooperative Agreement #A7RJ7, U.S. Department of Commerce. The statements, findings, conclusions, and recommendations are those of the author(s) and do not necessaril reflect the views of OAA or the U.S. Department of Commerce. References Doviak, R. J., and D. S. Zrnić, 993: Doppler radar and weather observations. Academic Press, 56 pp. Hildebrand P. H., and R. S. Sekhon, 974: Objective Determination of the oise Level in Doppler Spectra, Journal of Applied Meteorolog, 3, Ivić I. R., and S. M. Torres, 009: Using Signal Coherenc to Improve Detection on Weather Radars, 34th Conference on Radar Meteorolog, Williamsburg, VA, AMS. Ivić I. R., D. S. Zrnić, and T. Yu, 009: Use of Coherenc to Improve Signal Detection in Dual-Polarization Weather Radars, Journal of Atmospheric and Oceanic Technolog, 6, Urkowitz H., and J. D. espor, 99: Obtaining Spectral Moments b Discrete Fourier Transform with oise Removal in Radar Meteorolog, Int. Geoscience and Remote Sensing Smp., Houston, TX, IGARSS, 5 7. Siggia A. D., and R.E. Passarelli, 004: Gaussian Model Adaptive Processing (GMAP) for Improved Ground Clutter Cancellation and Moment Calculation, ERAD 004, Visb, Sweden, Appendix A In this appendix, we derive the rank ordered statistics power estimator which is used in the last step of the algorithm. If n independent samples that are exponentiall distributed with mean power are arranged in ascending order this results in a distribution function that can be associated with each Y i (i is the position in the ascending vector) viewed as random variable of the form

7 ( ) ( ) ( ) ( ) j= 0 i ( n i ) n! fy ( ) = e e e i i! n i! n! = i! n i! j! i j! ( i )! ( ) ( )( i j ) ( i j ) ( n i+ e e ) i ( ) ( ) ( ) ( i j n n L n i+ i ) ( n j ) = e. j= 0 j! ( i j)! The maximum likelihood value at each position in the ascending arra can be obtained from the following expression: ( )! ( ) i ( ) i df Y n i i n i+ n i+ ( n i+ ) = e e e e d ( i )!( n i)! (A.) i n! ( n i+ ) = e e ( n i ) ne + +. ( i )!( n i)! Setting the previous expression to 0, one can solve for as : dfy i ( ) n = 0 = ln d n i+. (A.3) Having an arra of ascending powers values, the mean power can be found as one that minimizes the mean square error between the measured powers and the maximum likelihood values. That is, R n n = i ln, i= 0 n i+ n R n n = i ln ln i= 0 n i n i + + (A.4) n n n n = i ln + ln i= 0 n i+ i= 0 n i+ = 0. So, the mean power can be estimated as n n i ln ˆ i= 0 n i+ =. (A.5) n n ln i= 0 n i+ (A.)