Spectral Line Bandpass Removal Using a Median Filter Travis McIntyre The University of New Mexico December 2013 Abstract For spectral line observations, an alternative to the position switching observation strategy is to take one observation on source and subtract the median filter of the resulting bandpass from itself. The median filtering technique has two great advantages over position switching in that it is at least twice as efficient with telescope time and it does not subtract extended emission from itself. However, median filtering can create a reduced spectrum with a flat baseline only if the general frequency of the signal is known (i.e. within tens of widths of the signal) and narrow compared to any structure in the bandpass (e.g. standing waves caused by a blocked aperture). Median filtering an entire bandpass is not an effective strategy because the signal must be in a local maximum in order to recover its flux and profile. Therefore, at least a first order fit of the bandpass around the signal must be removed before applying a median filter. A radio recombination line from an observation with the Arecibo Radio Telescope is used to test this method. Its flux and line width are recovered compared to position switching. 1. Introduction Removing a median filter from a bandpass is a desirable method of processing a spectral line signal if source emission is too extended for position switching to work properly. It is also useful in that it takes less than half as much telescope time to reach the same sensitivity. The following sections will describe the median filtering technique and analyze its major properties, benefits, and limitations. 2. Process The first step is to make at least a linear fit of the bandpass around the suspected location of the spectral line signal. The next step is to create a running median of the fitted bandpass with a window that is more than twice the width of the signal and less than twice the width of any bandpass structure present. The final step is to subtract this median bandpass from the fitted bandpass in order to create a reduced spectrum with a flat baseline. Below is a more detailed analysis of the effects of these processes on the quality of the data. The H164α radio recombination line shown in Figure 1 is taken from an observation of the HII region, S257, made with the 305m Arecibo Radio Telescope and is used as an example throughout this paper. 3. Analysis A median filter has certain features that should be understood before using it as a data reduction technique. At least a linear fit of the bandpass around the spectral line signal must be removed before applying a median filter or it will remove a significant amount of the signal. A median filter will always subtract some flux from a signal, but the exact amount is well understood if the baseline is flat and the noise is Gaussian. The wider the filter window, the less flux it will remove from a signal. A median filter will not remove any structure (e.g. standing waves) in the bandpass less than half the width of the filter window, which puts an upper limit on the width of the window when significant bandpass structure is present.
Figure 1. Unprocessed spectrum, in receiver units per channel, of an observation centered on the H164α radio recombination line of the HII region S257. This data is one polarization of a 5 minute integration using the Arecibo Radio Telescope with the Mock Spectrometer backend. Each channel is 21 khz. To process this data we will remove a linear fit and then a median filter. 3.1. Removing a Linear Fit At least a linear fit of the bandpass around the signal must be removed before applying a median filter because any slope in the baseline will significantly harm the quality of the median filter. For a bright signal with a channel width, w, a median filter will miss mw of its flux per channel, where m is the slope of the baseline. This will subtract (mw)w = mw 2 from the signal's integrated flux in the reduced spectrum. When the H164α line in Figure 1 is bandpass corrected with a median filter without first removing a linear fit, its integrated flux is 1/4 of the value given by the traditional position switching (ON- OFF) data reduction method as shown in Figure 2. The integrated flux of H164α is 16000 Receiver Units x Channels using the position switched method and 4000 Receiver Units x Channels using the median filter method when a linear fit is not removed. The median filter window is 36 channels, 4 times wider than the signal. Figure 2. Left Panel. Spectrum of H164α using a median filter without first removing a linear fit from the bandpass. Right Panel. H164α using the position switching data reduction method, (ON-OFF)/OFF. The peak flux of H164α is cut in half and the integrated flux is reduced by 3/4 when a linear fit is not removed. If a linear fit of the bandpass is removed, then the flux of the signal is recovered. Figure 3 shows the spectrum of H164α when a median filter is applied after a linear fit is removed. The width of the filter window used is 36 channels and the width of the signal is 9 channels. The integrated flux of H164α in Figure 3 is 14500 Receiver Units x Channels, within 10% of the integrated flux from the position switched method.
Figure 3. Spectrum of H164α using a median filter after a linear fit is removed from the bandpass. The integrated flux of the source is 14500 Receiver Units x Channels compared to 16000 from the position switched spectrum in Figure 2. 3.2. The Effect of Gaussian Noise on a Median Filter Once a first order fit around the signal is removed, a median filter can be applied. In the idl programming language, the simplest way to do this is to subtract the median array, Median(Data,Window), from the Data array, where Window is the number of channels to take the median of around each channel. The median window has to be more than twice the width of the signal to prevent subtracting the signal from itself. A median filter will always subtract flux from a signal if it is surrounded by Gaussian noise, even if the baseline of the noise is perfectly flat. However, the amount of flux subtracted is based off of the distribution of Gaussian noise and so is well known and the subtracted flux can be recovered statistically. The distribution of Gaussian noise is the integral, P(N) = 1 N 2π e x2 /2 dx, where P is the fractional percent of noise values, x, above the noise value, N (in standard deviations, σ). There is not a normal solution for the indefinite integral. A numerical solution of the definite integral is inverted and plotted in Figure 4 to show how the noise values are sorted from high to low before the median is taken. Figure 4. The fractional distribution of Gaussian noise sorted from high to low noise values. Any median filter will include this distribution of values in its window due to Gaussian noise adjacent to the signal. For instance, the median of a window 3 times wider than a bright signal is the noise value above 0.25.
The larger a median window is, the less flux it will subtract from the signal. For a signal greater than 3σ that is a step function of width, w, and is surrounded by a flat baseline of Gaussian noise, the median of a window, c times the width of the signal, is the value of the Gaussian noise distribution at (c-2)/(2(c-1)). In the example of the H164α line from Figure 3, the median filter window is 4 times wider than the signal, so the median across the signal is the value above 0.33 in Figure 4, or ~ 0.4σ per channel. This flux is subtracted from the signal when the median filter is removed from the bandpass. This subtracts 0.4 x 400 Receiver Units x 9 channels = 1440 Receiver Units x Channels of integrated flux from the H164α line, where the spectrum s RMS = 400 Receiver Units. Assigning this subtracted flux back to the H164α line raises its integrated flux to 14500 + 1440 = 15990 Receiver Units x Channels, within less than 1% of the integrated flux measured from the position switched spectrum. A small table of chosen noise values subtracted by a median filter in this way is given in Table 1. Flux Subtracted from Signal Filter Window Width / Signal Width 2σ 2.1 1σ 2.5 0.5σ 3.5 0.25σ 5 0.1σ 12.5 Table 1. Noise values subtracted as flux per channel from the signal due to a median filter with a window of width c times wider than the signal. This flux is subtracted from the signal and from the spectrum w(c/2 1) on either side of the signal. A median filter subtracts the same flux from every channel across the signal, as well as the spectrum within w(c/2 1) on either side of the signal. For small windows, the integrated flux subtracted from the signal can be recovered statistically by multiplying the noise subtracted by the width of the signal. However, for windows much wider than the signal, the baseline around the signal will be lowered by the same value and so the relative signal above the noise will not be missing any integrated flux as illustrated in Figure 5. For instance, a median filter with a window 10 times the width of the signal will subtract 0.125σ per channel from both the signal, and the baseline 4 signal widths on either side of the signal. The major assumption being made here is that the baseline of the noise is flat. If that is not the case, then we must take into account the effect of structure in the baseline. Figure 5. A diagram of two different. theoretical median fits around an idealized, step-function signal in a flat baseline of Gaussian noise using a filter window c times wider than the signal width. For a small window, c = 2, a median filter will subtract flux from the signal that has to be recovered statistically. For a larger window, c = 4, a median filter will subtract the same flux from the baseline around the signal, making it possible to recover the signal flux without statistical arguments.
3.3. Non-Linear Features in the Bandpass There are many causes of non-linear structure in a bandpass: sensitivity fall-off at the edges, receiver (and backend) response per frequency, standing waves from telescopes with partially blocked apertures. Typically, the narrowest of these structures, and thus the least responsive to a median filter, are the standing waves. A partially blocked telescope aperture can reflect some incoming radiation through the telescope support structure and this radiation will reach the receiver out of phase with the original wavefront. The Fourier transform of the time delay in this phase shift results in a standing wave in the bandpass. The structure of these waves depends on the telescope design, the time of day, and the zenith and azimuth angle of the observation. These structures can be removed with a median filter, which is a nice advantage as they are complicated superpositions of wave functions that cannot be fit with a polynomial, but only if the filter window is small compared to the width of the structure. A median filter is blind to any structure that is smaller than half the width of the filter window. This puts an upper limit on the width of the window that can be used for the median filter. A simple standing wave with some amplitude A and wavelength, λ, will create a structure, Asin(2πn/λ), in the bandpass across channels, n. A median filter window of width, W, is blind to this standing wave if W > λ. In fact, a median filter is blind to any part of the wave that is narrower than half the width of the window, as shown in Figure 6. Figure 6. Any median filter of width, W, will not remove any structure with width narrower than W/2. The filled in region of the curve above will not be fit by a median filter of width W. This is true for any bandpass feature narrower than half the width of the filter window. A median filter will be blind to the wave inside where A sin ( 2π λ n) = A sin [2π λ (n + W 2 )]. Solving for n gives n = π λ 2π arctan * sin ( λ W) 1 cos ( π λ W)+, meaning that the amplitude of the wave function will be reduced by
sin ( π sin (arctan * λ W) 1 cos ( π λ W)+), and the fraction of the wave successfully fit by the median filter is 1 - W/λ. For instance, if the window of the median filter is W = λ/2, applying the technique will reduce the amplitude of the standing wave by 70% and remove 1/2 of the wave. For Arecibo, the brightest standing waves have a frequency of 1 MHz, which will not be removed a median filter with a window wider than 250 km/s in L-band. Therefore, the median filter technique will fail when observing galaxies in HI (50 500 km/s wide), but it is quite effective for observing emission line sources coming from discrete regions in the Milky Way, which are of the order 20 30 km/s wide. Median filter windows that are too wide can fail to remove standing waves in the bandpass, so the effectiveness of the technique can be improved if standing waves can be removed all together. This is theoretically possible by clipping the features, corresponding with the frequency of the waves, in a Fourier transform of the bandpass (Briggs et al., 1997). A Fourier transform of the S257 observation is shown in Figure 7. The bandpass has 8192 channels spanning 172 MHz. The bright, 1 MHz standing wave at Arecibo should show up around u = 172, but this area is drowned out by ringing between the Milky Way and a spike in the center of the Mock Spectrometer bandpass. Clipping these bright sources effectively in the frequency domain before applying the Fourier transform may help in locating the standing waves in the time domain. It may also help to remove the steep sides of the bandpass in order to mitigate the sinc function centered on u = 0. These tactics should be looked into going forward as removing standing waves in the bandpass will relax the upper limits on the width of the median filter window, thus improving its effectiveness. Figure 7. Fourier transform of the bandpass from the S257 observation covering 8192 channels and 172 MHz bandwidth. 4. Conclusion The median filtering technique is superior to position switching in telescope efficiency and for
extended source observations. It is inferior for blind redshift observations and when the signal is wide compared to standing waves in the bandpass. The technique requires knowing, apriori, the observed frequency of the spectral line and that its width is small compared to standing structure in the bandpass. A median filter will always remove flux from a signal, but this flux is a function of the Gaussian noise distribution and is well understood. The integrated flux of a signal can be recovered using the median filter technique, and one example, using real data, is given here. The technique is best suited for narrow spectral line sources, but it can be applied to wider sources if standing waves in the bandpass can be removed effectively. References: Briggs, F. H.; Sorar, E.; Kraan-Korteweg, R. C.; van Driel, W., 1997, PASA, 14, 37.