Approaches for Compression of Super-Resolution WSR-88D Data

Transcription

1 IEEE Geoscience and Remote Sensing Letters 1 Approaches for Compression of Super-Resolution WSR-88D Data S. McCarroll, M. Yeary, D. Hougen, V. Lakshmanan, and S. Smith Abstract Weather radar products from the United States National Weather Service (NWS) are used by the government and private sectors. Very high resolution radar data are increasingly being utilized in real time. However, the bandwidth needed to transmit this data (termed Level II super resolution data) from the radar to the destination site is a limiting issue. General purpose compression programs are not tuned to the properties of weather radar data. As the NWS continues to upgrade the capabilities of the radar network, the amount of data will continue to increase. As a result, compression is of vital interest to keep down maintenance, storage, and transmission costs. A method for lossless compression of this data on a radialby-radial basis focusing on the delta (difference) between range bins of super resolution radar data is presented and is called super resolution delta compression (SRDC). There are several specialized aspects of SRDC that are based on the properties of weather radar data. SRDC was tested on Level II reflectivity product data from several S-band Doppler weather radars in the NWS network, and was compared with two general purpose compression programs and a different weather-radar-specific compression approach. The results show that the newly developed SRDC yield is approximately 15% better than the next best approach and approximately 47% better than only preprocessed radials. Index Terms radar signal processing, compression, highresolution data, Doppler radar W I. INTRODUCTION EATHER Surveillance Radar 1988-Doppler (WSR- 88D) radars have several operating modes and scanning strategies to optimize data recovery. Normal resolution weather data products have 360 radials sampled at 1 degree azimuths. Each reflectivity radial is sampled at every quarter kilometer for 460 km, and then is averaged once for every kilometer. This produces 460 range bins for each of the 360 radials in an elevation cut [1]. For super resolution data, This work was paritially supported by NSF-HRD S. McCarroll was with NOAA s Radar Operation Center, Norman, OK USA and is currently with National Security Agency. M. Yeary is with the School of Electrical & Computer Engineering, and the Atmospheric Radar Research Center, University of Oklahoma, Norman, OK USA. D. Hougen is with the School of Computer Science, University of Oklahoma, Norman, OK USA. V. Lakshmanan is with the Cooperative Institute of Mesoscale Meteorological Studies, a collaborative institute between the University of Oklahoma and the National Severe Storms Laboratory, Norman, Oklahoma. S. Smith is with NOAA s Radar Operations Center, Norman, OK USA. azimuth sampling is increased from 1 degree to 0.5 degrees, doubling the number of radials per elevation cut. Reflectivity sampling is no longer averaged every kilometer, meaning each radial is recorded four times as often, up to 460 km. Because every radial is now sampled four times as often, and because there are twice as many radials, there will be eight times as many range bins that sampled data is stored.. According to current Radar Operations Center (ROC) projections, throughput will be increased approximately 2.3 times, with a max 160 Kbps per radar, and a network max of 7.3 Mbps in the Level I data stream between a Radar Data Acquisition (RDA) unit and a Radar Product Generator (RPG) [2]. All NWS Doppler radars have now have a capability to switch to super resolution format from normal resolution; therefore, this research focuses on super resolution Level II data. Level II super resolution data are packaged one radial at a time, along with its necessary header information, for transmission from the instrument to a radar product generator (RPG). Similarly, 120 of these packages are concatenated together to be transmitted to the end-users from the RPG. Because of this, each radial of data will be encoded one at a time to be packaged for transmission. Ideally this compression would occur in the RDA and not be decompressed until it has been received by the end-user. Since the Collaborative Radar Acquisition Field Test (CRAFT) became operational in 2004, weather radars have been distributing data in real-time [3], [4]. Many uses for multi-radar data have been devised which require this realtime access to the radar data [5], [6]. The transmission of super resolution data is contingent on the availability of funds to cope with the increased communications bandwidth required on a radar-by-radar case [7]. More data are requested from NWS radars during periods of severe weather than at other times. Periods of severe weather are also when the highest resolution radar data is needed to ensure the safety of the affected population. Due to the high traffic during these times, communication channels could get congested so that customers cannot obtain desired information. This is a real concern considering the price and size of communication lines and the expected future size upgrades to weather radar data. For example, a forthcoming upgrade to the radars is to add polarization capability, which will increase the number of raw moments transmitted from the instrument to the RPG. In this paper, we concentrate on radar reflectivity products, but similar compression techniques will have to be devised for the other raw moments -- both the ones currently produced (velocity, spectrum width) and the

2 IEEE Geoscience and Remote Sensing Letters 2 ones that are forthcoming (differential reflectivity, correlation coefficient). For these reasons, lossless compression of super-resolution radar data is important to the NWS for both storage and data transmission/communication. Data compression has been studied for several decades now, yet its application to weather radar data is relatively new. There are a number of standard compression methods that are useful to consider including run-length encoding, bit variance and bit packing, differencing, and pre-processing techniques. Run Length Encoding: Run-length encoding (RLE) replaces runs (consecutive repeated characters) with a single character and the length of the run. For instance, AAAAAAAAAAAAAAA requires 15 bytes to store. After RLE, only two bytes (15A) are required. As of 2001, the image data for most WSR-88D base products and radialformat derived products were packed in a 4-bit RLE format [8]. Applying a simple RLE on the radar data can expand the size of the radial due to small number and size of runs of valid data. There are some radials where a simple RLE should not be used. Bit Variance and Bit Packing: Bit variance is a method where different blocks of data are represented with different numbers of bits. Some data can be represented in only a few bits, while other data might require a byte. The optimal number of bits to use to represent the data blocks can vary from file to file. Bit packing is a method of combining the bit stream into full 8-bit bytes for transmission/storage, but the bytes themselves have no intrinsic meaning. Once bit variance has been applied to the data, bit packing (and unpacking) is how the actual savings is accrued. Huffman Encoding [9] and other entropy encoding techniques use frequency analysis of the symbols to determine how many bits are used to represent each symbol. The range of symbol data is important with bit variance while the frequency of occurrence is important to entropy encoding. Differencing: Differencing subtracts the previous value from the current value and stores the difference instead of the raw value. In general, a differential set of data can be represented with a smaller dynamic range if there are no large changes between adjacent values. In a byte, data can range between 0 and 255, but the range for differencing is -128 to 127. A difference value of -25 means that the current value is 25 less than the previous one and 25 means 25 is added to the previous value. Differencing is best used when there are no large changes that must be represented outside of the difference byte range. Pre-processing: In the SRDC method proposed in this paper, there is a pre-processing step at the beginning that removes a number of trailing zeros from a radial. This technique is lossless since only the size of a radial must be known to fully reconstruct the data. The technique is specific to weather radar data because most streams of data do not have large numbers of trailing zeros. II. PROPOSED APPROACH General purpose compression strategies are easy to apply and work well for generic compression, but they do not take into account the structure of the weather data. The characteristics of the radar data in question include a large number of missing-data values which are encoded as zeros. As an example, Figure 1 shows the total number of zeros, which is comprised to two components for the 720 radials in the first elevation cut of the KBIS radar on November 11th, 2008 at 12:30pm CST: the number of trailing zeros, which are those at the end of a radial, and the number of non-trailing zeros. Fig 0. Zero statistics for an example elevation cut. A. Compression The proposed super resolution delta compression (SRDC) scheme begins by executing a pre-processing step of removing the trailing zeros, which are present at the end of almost all radials. The size of a radial is known by the decoder so, by zero-filling on decompression, no data are lost. In addition to trailing zeros, data may be missing elsewhere in each radial. The zero value of this missing data is re-encoded as the median of the values of valid symbols found adjacent to missing data. Any range bin values that are equal to the median are shifted so only the missing data hold that value and is not an actual data value. The decoding process knows the median value and reconstructs the radial without any loss of information. In the range bin byte there is always at least one value that is not used by virtue of the range folding field not being used in reflectivity data. This process results in less of a difference between the missing data and the real data, and in some cases helps to shorten the difference from one valid symbol to another. Additional details are in [10]. Figure 2 shows the frequency histogram of the raw level-codes for the 852 range bins that were not trailing zeros on radial 185 of KBIS elevation cut 1. Not shown here (so that the graph would remain focused on the valid data) are the 23.12% of 852 range bins where missing-data are represented as zero. In this data, 70.07% of the symbols are between the values of 67 and 109. Once the zeros are taken into account, 93.19% of the data can be represented by 43 symbols. It is noted that the term missing data here refers to data that was sampled beyond the threshold for 8-bit reflectivity level-codes. Or the term can be used when the corresponding range bin is in an area of the atmosphere that weather does not exist, though that can be called beyond threshold as well. In

3 IEEE Geoscience and Remote Sensing Letters 3 Fig 2. Histogram of range bin values for KBIS radial 185. general, SRDC is a lossless compression algorithm and does not alter the range bin values of the original radial or when the radial is decoded at the destination. The missing data range bins are reverted back to their zero value during the decoding process, and all other bins are adjusted accordingly so that there is no difference between the original and resulting radials. The missing data encoding of the median and the differencing is only used to reduce the size of the transmitted structure and does not alter the resulting radial. Figure 3 is the frequency histogram for the difference values for radial 185 of the same KBIS data. The difference method is used to calculate the difference values. In this histogram, 26.53% of the 852 differenced bins are zero. This represents instances when the range bin is the same value as the previous range bin. This primarily happens due to runs of missing-data zeros in the raw range bin, but also occur with runs of valid data. As shown in Figure 3, 94% of the difference values are between -10 and 10, and would efficiently be represented by 21 symbols (missing data values included). The characteristics of the range bin values show that except for a few large drops or peaks, there are small changes between one range bin and the next, so a differencing approach is appropriate for SRDC. Next, RLE is used to calculate how often there is a run of each symbol, including the symbol for missing data. Because missing data runs were typically much longer than valid data runs, separate RLE values were recorded for missing data. Since there are so few runs in the data, the output can be reduced even further by restricting the number of bits used for the run bins. Bit variance is used on the differenced values and their respective run-lengths to find the best length for the data in the bit stream. This allows for the majority of the data to be represented in a relatively small number of bits and a method of continuation to encode outliers. To represent difference outliers caused by drops or peaks, a continue method was introduced. Because the run-length of a RLE will never legitimately be zero, that value is encoded as continue. Any time a zero is registered in the run-length bin, the difference value is appended until the run-length is nonzero. Any time that the difference is larger or smaller than the range allows, continue bins are used to encode the actual difference value. This allows for the main range of values to Fig 3. Histogram of difference values for radial 185. be represented in the most efficient number of bits while still retaining the capability of representing larger difference values. When there is a continuance, there is at least the max-range number added to or subtracted from the difference. In the regular representation the max-range is recorded in the difference bin. If there is a large change and the max range divides the difference value many times, those max ranges are represented with a run-difference pair. An advanced continue method keeps track of the number of maximum ranges representing the differences. The last difference bin always contains the remainder of the difference value and it determines the sign of the entire difference. Notice that the difference bins that represent the number of max-range bins are never negative, they are only counters. However, an issue arises if the difference is a multiple of the maximum range. In that case, both unique instances will map to the same value, 0. A sign-bit (or sign flag) was introduced in the difference bin to represent the signed direction of the difference. Because of the sign-bit, the ranges for 3-bit difference values are from -3 to 3 instead of -3 to 4, for example. There must always be at least one bit used for the run bin to show whether there is a continuance or if it is actual data. If only one bit is used for the run-length and there is a repetition of a symbol, then several run-difference pairs must be used to represent the number of occurrences. The number of bits used for the run bin and the number of bits used for the difference bins do not have to be the same. Since the two types of bins come in pairs, as long as the number of bits used for both the run and difference bins are known, the encoded data can be recovered precisely. Because weather radar data differs from cut to cut and from radial to radial, the optimal number of bits in one set of data is not necessarily the same as in the next set. It is useful to vary the number of bits used for the bin sizes to yield the best compression results for a specific data set, so SRDC was run several times using different bin-block length values. However, because some bit combinations enlarge the size consistently those combinations do not need to be calculated in the future. Also, while varying the bits for each of the bins yields the best compression, it may be preferred to use a set number of bits for both run-length and difference bins for the sake of computation time.

4 IEEE Geoscience and Remote Sensing Letters 4 The numbers of bits to represent the differenced values are still dependent on the minimum and maximum of the difference range. This is why the missing-data symbol (the most frequently encountered symbol) is encoded in such a way as to reduce the size of changes from missing data to valid data by using the mid-range difference method. Since the only values that affect this difference are the bins that are on the rising or falling edges of missing data, the method begins by storing the rising and falling values. The median of this stored data is set as the encoding of the missing data. Any range-bin value that is less than or equal to this median encoding is shifted down by one. Additionally, the first range bin of the radial is differenced using the median value, which helps with the problem of the first bin always being large because it is the actual level code and not differenced. The median difference method has the chance of bridging the gap between two valid bins, splitting the jump into two smaller parts rather than one large leap. A representation was developed to provide an RLE of exclusively the runs of missing data. This is possible because there would normally be no combination where the run bin and the difference bin were both 0. When this sequence is met it signals a zero-bin encoding. Because this encoding can add to the size of the data it was not applied to runs of valid data because the majority of valid data runs were not large. The zero-bin contains the number of sequential missing data values, up to the max range allowed by the number of bits used to represent the zero-bin. Similar to what was done with the continue method, and considering the fact that the zero-bin will never be 0, the scheme was expanded to additively represent the larger runs of missing-data. If the run of missing data values is larger than the maximum range that the zero-bin bits supports, a 0 is recorded in the zero-bin until the remaining missing data values can be represented within the range. When decoding this representation it is important to expect to read zero-bins until there is a non-zero value. Once a non-zero value is recorded, the size of the run of missing data is known and the next two bins expected are a run-bin and a difference-bin respectively. B. Decompression A radial is decompressed from the encoded data, the missing-data encoded value, and the number of bits used to represent symbol differences and run-lengths. These values can be included in the header of each radial without significant overhead (i.e., several more bits are needed to represent this). The data are decoded section by section until it is represented as the differenced symbols and their run-lengths. Using the median encoding, the symbols are returned to the original data level codes of the radar. The runs are re-added to the symbol data and the radial is zero-filled to return the original radial. C. Bin Configurations SRDC introduces several different encoding schemes to reduce radial size. Figure 4 shows each of the four formats in which the data can be encoded. In the figure the lowercase letters under the boxes indicate the bin type: r corresponds to a run bin, d is a difference bin, b is a block continue bin, and z is a zero bin. The d and b bins are the same size. The capital letters in the boxes represent an actual value. R corresponds to R r D b 0 B 0 B R D r d z r b r b r d 0 0 Z Z r d z z z Fig 4. Bin configurations. a run-length value, D is a difference value, B is the number of times the maximum range goes into the difference value, and Z is the run-length number of missing data. The figure is broken down into four sections, 1, 2, 3 and 4 and these cases cover all the options necessary to encode a radial of data without loss. These cases are: 1) The base case, where the difference value can be recorded in the number of bits used for the difference bin. 2) The continue case, where a 0 in an r bin indicates the next bin is a block bin b. A subtotal difference is calculated: (sum of Bs) (sign of D) (max range of b). If the R value is anything other than 0 then it indicates that a D is to be added to the subtotal to produce the range bin s difference value. 3) The missing data run-length case, where both r and d are 0 and indicates that there is a run of length Z of missing data. This case is used when the run of missing data is within the max range allowed by the number of bits used to represent z. 4) The advanced missing data run-length case, where the run is larger than the max range of the bits of z. Every time a 0 is in z the count of missing data goes up by the max range of z and the next bin expected is still a zero bin. Once there is a non-zero value in z, the number Z is appended to the running total and the final run-length is known. The next bin expected is now a run bin. III. TESTING AND RESULTS The generic, off-the-shelf compression programs Bzip2 [11] and Gzip [12] and a weather-specific compression scheme called Linear Prediction (LP) [13] were compared with SRDC on compression size. Bzip2 is a block-compression technique [14] while Gzip2 is based on Lempel-Ziv [15] and Huffman coding [9]. For all compression schemes tested, trailing zeros were removed as the first step. This was done in order to provide a more challenging comparison for SRDC and means that the savings reported for SRDC are from other aspects of the method, not just zero-stripping and filling. The data tested are Level II Super Resolution reflectivity files from the National Climatic Data Center. From this archive, 30 datasets were randomly selected by day, hour, and station ID. Similarly, a random volume file from the hour of scanning and a random cut number were generated. 30 cuts from 30 different radar stations were retrieved using these

5 IEEE Geoscience and Remote Sensing Letters 5 TABLE I RADIAL COMPRESSION RESULTS (IN UNITS OF BYTES) Method Mean Std Dev Ave CR Best of 128 PP SRDC LP Gzip Bzip Best of all randomly generated values. From each of these cuts, four radials were selected at random, except for the KTWX radar where 12 radials were sampled, for a total of 128 radials. Table I shows the mean and standard deviation of the radial set after each program was run. A radial with no compression has a constant size of 1840 bytes, one byte for each of the 1840 range bins. Table 1 also shows the average compression ratio (CR) for the radial set (a larger value indicates better compression). The pre-processing (PP) result is the radial after the pre-processing step of removing the trailing zeros. SRDC is the compression scheme proposed in this paper with the best number of bits used to represent the run bins, difference bins, and zero bins. Best of all shows the data set when all programs were run and the best results were selected. Out of the 128 tests compressing by radial, SRDC showed the best compression 119 times. Gzip compressed the best 3 out of the 119 times. LP compressed best 6 times, and Bzip2 never compressed best. One-tailed, paired t-tests were implemented on the results from radial compression and has a p-value of less than 0.01, except when comparing SRDC to best of all. Table 2 shows the percentage savings comparing the average size of the 128 radials after running SRDC to the size after computing the pre-processing technique of removing trailing zeros (PP), LP, Gzip, and Bzip2 respectively. TABLE II PERCENT COMPRESSION SAVINGS OF SRDC PP LP Gzip Bzip2 SRDC 46.30% 37.71% 16.90% 21.22% IV. DISCUSSION Bzip2 may be discounted from further compression consideration because its mean of radial compression sizes is consistently larger than that of Gzip. Additionally, when comparing the sizes of the 128 radials after compression, Bzip2 never out-performed any of the other methods. LP showed the second highest number of best compressed radials; however, the variation between values is quite large. SRDC had the best compression results on 119 out of the 128 radials and it also demonstrated close to 15% savings over the next best compression approach. Complete SRDC compression is also about 47% smaller than preprocessing alone. SRDC finds the best compression while varying the zero bins from 2 bits to 7 bits. The difference between SRDC and Gzip is approximately 17% in compression size. Some form of compression should be used since all of the schemes surpass the 10% threshold on an uncompressed radial. We suggest the use of SRDC specifically since it is smaller than the next-best compression scheme (Gzip) by approximately 17%. V. CONCLUSIONS The results show that SRDC should be used to compress Level II super resolution reflectivity data on a radial by radial basis. Since the RDA transfers the Level II reflectivity data to the RPG by individual radials, the SRDC scheme should be used to yield higher compression. Further, the LDM in the RPG packages weather data 120 radials at a time, but each radial is split by header messages. Higher compression would be achieved if the LDM compressed each radial block using SRDC while packaging 120 radials together for transmission. In conclusion, the NWS has a need for compressing weather radar data one radial at a time, and SRDC appears to be the best program to satisfy this need. REFERENCES [1] National Weather Service, Radar Operations Center: Interface Control Document for the RPG to Class 1 User, , April 15, [2] M. Istok, WSR-88D Build 10 Changes Affecting Users of Level 2 Data., Retrieved September 2008, from [3] K. Droegemeier, K. Kelleher, T. Crum, J. Levit, S. Del Greco, L. Miller, C. Sinclair, M. Bennar, D. Fulker, and H. Edmon, Project CRAFT: A test bed for demonstrating the real time acquisition and archival of WSR-88D base (Level II) data, AMS Conf. on IIPS, pp. 1-4, [4] K. Kelleher, K. Droegemeier, J. Levit, C. Sinclair, D. Jahn, S. Hill, L. Mueller, G. Qualley, T. Crum, S. Smith, S. Del Greco, S. Lakshmivarahan, L. Miller, M. Ramamurhty, B. Domenico, and D. Fulker, Project CRAFT: A real-time delivery system for NEXRAD level II data via the Internet, Bulletin of the American Meteorological Society, vol. 88, no. 7, pp , July [5] K. Droegemeier, Real-time Acquisition and Archival of WSR-88D Base Data. University Corporation for Atmospheric Research (UCAR) Quarterly, Fall [6] Lakshmanan, V., Smith, T., Hondl, K., Stumpf, G., & Witt, A. A realtime, three dimensional, rapidly updating, heterogeneous radar merger technique for reflectivity, velocity and derived products, Weather and Forecasting, vol. 21, no. 5, pp , [7] R. Vogt, Modifications to WSR-88D Level II Data Stream and Format, Retrieved May 5, 2008, from news/radnews html. [8] National Weather Service, AWIPS Application Integration Framework Manual, July [9] D. Huffman, "A Method for the Construction of Minimum-Redundancy Codes", Proc. of the I.R.E., pp , September [10] S. McCarroll, thesis: Compression strategies for super resolution WSR- 88D radar data, May 2009, pp University of Oklahoma. [11] J. Seward, retrieved September, 2008 [12] J. Gailly and M. Adler, retrieved November, 2008 [13] V. Lakshmanan, Lossless coding and compression of radar reflectivity data, 30th International Conference on Radar Meteorology, pp , American Meteorological Society, (Munich), July [14] M. Burrows and D. Wheeler: "A block-sorting lossless data compression algorithm", Digital SRC Research Report 124. ftp://ftp.digital.com/pub/dec/src/research-reports/src-124.ps.gz [15] Jacob Ziv and Abraham Lempel; A Universal Algorithm for Sequential Data Compression, IEEE Transactions on Information Theory, 23(3), pp , May 1977.