Democracy in Action. Quantization, Saturation, and Compressive Sensing!"#$%&'"#("

Transcription

1 Democracy in Action Quantization, Saturation, and Compressive Sensing!"#$%&'"#("

2 Collaborators Petros Boufounos )"*(&+",-%.$*/ 0123"*4&5"*"%16(

3 Background If we could first know where we are, and whither we are tending, we could then better judge what to do, and how to do it. -Abraham Lincoln

4 Sparsity / Compressibility Images K << N large N pixels wavelet coefficients Chirps N wideband samples K << N large Gabor coefficients (time-freq)

5 Sparsity / Compressibility power time Wireless spectrum N wideband samples sparse in (time-frequency) most of spectrum is not occupied frequency band PLM, Amateur, others: MHz TV 2-6, RC: MHz Air traffic Control, Aero Nav: MHz Fixed Mobile, Amateur, others: MHz TV 7-13: MHz Maritime Mobile, Amateur, others: MHz Fixed Mobile, Aero, others: MHz Amateur, Fixed, Mobile, Radiolocation, MHz TV 14-20: MHz TV 21-36: MHz TV 37-51: MHz TV 52-69: MHz Cell phone and SMR: MHz Unlicensed: MHz Paging, SMS, Fixed, BX Aux, and FMS: MHz IFF, TACAN, GPS, others: MHz Amateur: MHz Aero Radar, Military: MHz Space/Satellite, Fixed Mobile, Telemetry: MHz Mobile Satellite, GPS, Meteorologicial: MHz Fixed, Fixed Mobile: MHz PCS, Asyn, Iso: MHz TV Aux: MHz Common Carriers, Private, MDS: MHz Space Operation, Fixed: MHz Amateur, WCS, DARS: MHz Telemetry: MHz U-PCS, ISM (Unlicensed): MHz ITFS, MMDS: MHz Surveillance Radar: MHz Measured Spectrum Occupancy in Chicago and New York City 0% 50% spectrum occupancy 0.0% 25.0% 50.0% 75.0% 100.0% K << N Spectrum Occupancy Chicago New York City 100% % of time frequency (MHz) large Gabor coefficients (time-freq) [data by Shared Spectrum Company] [

6 Signal Models Sparse nonzero signal coefficients Compressible signal coefficients decay according to e.g. man-made signals such as MSK, QPSK, sparse in time-frequency approximate by largest coefficients coefficients K=5 e.g. wavelet coefficients of natural images are in range: [DeVore, Jawerth, Lucier] sorted coefficients K-term approximation power law decay

7 Compressive Sensing signal (length-n) measurements (length-m) signal estimate (linear) (M<<N) bounded noise reconstruction (non-linear) Hallmarks non-adaptive linear measurements Notation integrates sensing, compression, processing measurements are democratic, contain a similar amount of information progressive reconstruction

8 Measurement Model measurements sparse signal nonzero entries is K-sparse or compressible is random

9 Matrix Requirements Restricted Isometry Property (RIP) for all K-sparse RIP of order 2K implies: for all K-sparse, for and stable embedding K-planes

10 RIP Matrices measurements sparse signal nonzero entries If elements of are drawn from subgaussian distribution, then has RIP with high probability for

11 Reconstruction measurements sparse signal nonzero entries Optimization Bounded measurement noise: [Candes, Romberg, Tao; Donoho] Greedy Algorithms (Orthogonal) Matching Pursuit (OMP) Iterative Hard Thresholding (IHT) [Tropp, Gilbert] [Blumensath, Davies] Compressive Sampling Matching Pursuit (CoSaMP) [Needell, Tropp]

12 Reconstruction Guarantees signal (length-n) measurements (length-m) signal estimate (linear) reconstruction error (optimization) (M<<N) bounded noise reconstruction (non-linear) best K-term approximation [Candes, Romberg] error due to measurement noise error due to K-term approximation similar guarantees for greedy reconstruction approaches

13 CS ADC Random Demodulator for Fourier-sparse signals tradeoff complexity of ADC with complexity of randomized hardware maps analog signal to discrete measurements /##%!"#$%&'()%&* +$*,#' -#)#'(.&' low rate ADC 1-1 Nyquist rate chipping sequence [Tropp, Laska, Romberg, Duarte, Baraniuk; Laska, Kirolos, Duarte, Ragheb, Baraniuk, Massoud]

14 CS Imaging Single Pixel Camera digital micromirror device (DMD) displays random 0/1 patterns photodiode computes optical inner product photodiode voltage is quantized scene photodiode quantizer DMD processing [Duarte, Davenport, Takhar, Laska, Sun, Kelly, Baraniuk ]

15 Quantization Quantization quantization interval error per measurement bounded: Finite Dynamic Range Quantization saturation level (dynamic range) bit-rate (bits per sample) quantization interval is measurements above saturate saturation error is unbounded all practical quantizers have finite dynamic range

16 Quantization Error ()*+,-).)/ expected error/n!"!&!"!%!"!$! 1!!"# $ $"# % %"# & saturation ' level error is dominated by saturation finite 1 dynamic range infinite 141( dynamic range saturation rate is effectively zero Message: avoid saturation (for gaussian samples) error is dominated by quantization 1

17 Avoiding Saturation Scale down signal such that little or no saturation occurs achieved with automatic gain control (AGC) Typical rule 6 saturation events per million Too conservative? quantization error increases as gain decreases saturation events may be rare

18 CS with Quantization signal (length-n) measurements (length-m) quantizer signal estimate R(. ) reconstruction (linear) (M<<N) noise (non-linear) Problem error is unbounded due to saturation events CS results assume bounded errors Conventional Approach avoid saturation scale down measurements can we do better?

19 Signal Recovery from saturated MEasurements All power tends to corrupt and absolute power corrupts absolutely. -Lord Acton

20 Saturated Measurements measurements saturated measurements measurements below saturation level saturated measurements sparse signal nonzero entries Democratic Measurements (before quantization and saturation) each measurement contains roughly the same amount of information Saturated Measurements easy to detect have magnitude greater than G

21 Saturation Rejection measurements discard saturated measurements sparse signal nonzero entries Rejection Approach simply discard saturated measurements

22 Saturation Rejection measurements discard saturated measurements discard rows of sparse signal nonzero entries Rejection Approach simply discard saturated measurements discard corresponding rows of

23 Saturation Rejection measurements sparse signal nonzero entries Rejection Approach simply discard saturated measurements discard corresponding rows of apply standard reconstruction algorithms to, ex:

24 Saturation Rejection measurements sparse signal Hallmarks any out-of-the-box CS reconstruction algorithm can be used quantization error on remaining measurements is bounded exploits democracy of measurements can be used for other types of processing (such as compressive matched filtering for detection) nonzero entries but... we are throwing away information

25 Saturation Consistency measurements saturated measurements sparse signal Consistent Approach modify recovery algorithms so that the solution is consistent with saturation level add saturation constraint: nonzero entries saturated measurements : vector of ones

26 Saturation Consistency sparse signal : rows corresponding to positive saturated measurements nonzero entries : rows corresponding to negative saturated measurements new matrix:

27 Saturation Consistency Consistent Approach (optimization) measurement error term (quantization) saturation consistency constraint : vector of ones

28 Saturation Consistency Consistent Approach (optimization) cont... exploits democracy of measurements alternative measurement error terms can be used [Jacques, Hammond, Fadili] Consistent Approach (greedy) often faster than optimization we introduce saturation consistent CoSaMP (SC-CoSaMP) Overview find new supports update support set estimate coefficients prune result

29 Saturation Consistent-CoSaMP SC-CoSaMP (greedy) denotes iteration while (not converged) new! compute proxy: merge support: union of support of largest support of ( selects positive elements of vector) coefficients of new! estimate coefficients: prune: keep largest coefficients of end

30 Three Approaches Conventional Approach avoid saturation, scale down measurements new! Rejection Approach simply discard saturated measurements ex: new! Consistent Approach use all measurements, put constraint on saturated ones (optimization) (greedy) SC-CoSaMP, SC-IHT

31 Random Measurements and Democracy Information is the currency of democracy. -Thomas Jefferson (attributed)

32 Democracy Setup matrix with entries denotes the rows of indexed by the set Definition We call -democratic if we have, for every with and for all -sparse Notes stronger than RIP for is an instance of matrix for signal dependent

33 Democracy Theorem is where -democratic with probability exceeding we can pick, such that for a fixed,

34 The Democracy of Gaussian Measurements new! Proof sketch (that is -democratic) concentration of measure for for any in, draw a random (in this result, elements of are ) with high probability restricted isometry of ( standard procedure) fix K-dimensional subspace pick set of points such that for any we have apply concentration of measure to to bound apply union bound over all K-dimensional subspaces K-plane

35 Concentration of Measure for : Employ Order Statistics Concentration of measure: detail only need to consider two bounding cases keep rows corresponding to smallest magnitude measurements keep rows corresponding to largest magnitude measurements example: lower bound are Gaussian Definition draw i.i.d. variables, the -th order statistic is the -th largest element let select smallest elements of if given -th order statistic of,, then solve -th order statistic of (for ) (for ) distributed as truncated Gaussian, use Markov inequality to obtain bound PDF of -th order statistic of

36 Simulations No man s knowledge here can go beyond his experience. -John Locke

37 Simulation Setup Signal models K-sparse: coefficients are i.i.d. Gaussian, random locations p-compressible: coefficient magnitudes chosen as assign random signs and positions signals are scaled to have unit norm Measurement matrix elements are i.i.d. Gaussian with variance 1/M Reconstruction metric Signal-to-Noise ratio (SNR)

38 Approach Recap Three approaches conventional approach reconstruct with saturated measurements rejection approach discard saturated measurements consistent approach use saturated measurements as constraint

39 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] M/N = 2/16 consistent approach performs only slightly better SNR (few measurements) Conventional Reject Consistent Avg Saturation Rate Saturation Rate Saturation Level (G) rejection approach performs worse

40 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] M/N = 6/16 both approaches perform better for a fixed G too much saturation for rejection approach SNR dB Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) 1 Saturation Rate

41 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] M/N = 6/16 optimal performance occurs at significantly nonzero saturation rate SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) 1 Saturation Rate 0.09

42 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] 60 34dB M/N = 15/16 (many measurements) 1 SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) Saturation Rate

43 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] 60 M/N = 15/16 (many measurements) 1 SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) Saturation Rate 0.14

44 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] SNR M/N = 2/16 M/N = 6/16 M/N = 15/16 Conventional Reject Consistent Avg Saturation Rate Saturation Level (G) Saturation Rate SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) 1 Saturation Rate SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) 1 Saturation Rate similar performance for all approaches saturation rate should be close to zero rejection, consistent approaches achieve higher optimal SNR best performance at nonzero saturation rate

45 K-sparse Signals Fixed parameters N =1024 K =20 B=4 [algorithm: cvx] SNR M/N = 2/16 M/N = 6/16 M/N = 15/16 Conventional Reject Consistent Avg Saturation Rate Saturation Level (G) Saturation Rate SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) 1 Saturation Rate SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) 1 Saturation Rate performance gain increases as a function of M how do optimal performances compare?

46 SC-CoSaMP Fixed parameters N =1024 K =20 B=4 [algorithm: sc-cosamp] rejection approach best SNR is 20dB higher than conventional approach consistent approach best SNR is 23dB higher than conventional approach rejection approach SNR is only 3dB lower than consistent approach SNR what does consistency buy us? Maximum SNR(dB) optimal performance 3dB 20dB 20 Conventional Reject Consistent M/N too few measurements

47 Saturation Robustness Fixed parameters N =1024 M/N=6/16 K =20 B=4 [algorithm: sc-cosamp] Saturation robustness range of saturation rates such that SNR of each approach is as good as best conventional approach SNR )*+ (! '! &! %! $! #!! saturation rate ;12<0< =5>- ; ,73 3!!"#!"$!"%!"& ),-./, : Saturation Rate

48 Saturation Robustness Fixed parameters N =1024 M/N=6/16 K =20 B=4 [algorithm: sc-cosamp] Saturation robustness saturation rate range of saturation rates such that SNR of each approach is as good as best conventional approach SNR )*+ (! '! &! %! $! #!! rejection approach robustness ;12<0< =5>- ; ,73 3!!"#!"$!"%!"& ),-./, : Saturation Rate

49 Saturation Robustness Fixed parameters N =1024 M/N=6/16 K =20 B=4 [algorithm: sc-cosamp] Saturation robustness saturation rate range of saturation rates such that SNR of each approach is as good as best conventional approach SNR consistent approach robustness )*+ (! '! &! %! $! #!! rejection approach robustness ;12<0< =5>- ; ,73 3!!"#!"$!"%!"& ),-./, : Saturation Rate

50 Saturation Robustness Fixed parameters N =1024 M/N=6/16 K =20 B=4 [algorithm: sc-cosamp] Saturation robustness range of saturation rates such that SNR of each approach is as good as best conventional approach SNR consistent approach is more robust to saturation events +,-./,-0123/,-43/,254 '!"&!"%!"$!"# !!!"# 912:0:-42-!"$!"% ()*!"& '

51 Compressible Signals Fixed parameters N =1024 M/N=6/16 B=4 [algorithm: cvx] SNR p=0.4 p=0.8 p=1 Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) Saturation Rate SNR Avg Saturation Rate Conventional Reject Consistent Saturation Level (G) Saturation Rate SNR Conventional Reject Consistent Avg Saturation Rate Saturation Level (G) Saturation Rate rejection, consistent approaches achieve higher optimal SNR best performance at nonzero saturation rate performance gain decreases as function of p

52 Simulation Summary Simulation results both rejection and consistent approaches can boost reconstruction performance (if enough measurements to spare) best performance occurs at significantly nonzero saturation rate best performance of both approaches is similar consistent approach is more robust to saturation events Similar results for other measurement systems Rademacher (similar is used in CS camera) random demodulator random sampling Nyquist sampling of Fourier-sparse signals, CS reconstruction

53 Extensions Bureaucracy is not an obstacle to democracy but an inevitable compliment to it. -Joseph Schumpeter

54 Automatic Gain Control Conventional AGC scale down for no saturation requires complicated hardware and signal heuristics often not sensitive to drop in signal strength CS AGC significantly nonzero saturation rate uses saturation rate only to determine gain [Goldberg] Setup process in time-blocks (indexed by ) each block determines the gain for the next

55 Saturation-based AGC Quantizer AGC + + Compute gain desired saturation rate measured saturation rate previous block iteration gain feedback parameter (controls responsiveness) saturation rate error system is based on saturation rate only

56 AGC in Action Measurements Gain 2 0! signal strength drops by 90% range of quantizer [-1,1] Scaled measurements Saturation rate 2 0! AGC iteration set block-size: 32 measurements uses saturation rate only: sensitive to decrease in signal strength

57 Future Work What is not yet done is only what we have not yet attempted to do. -Alexis de Tocqueville

58 Joint-Source-Channel Coding Fountain Codes [Luby] sends N packets, as long as at least M are received, the original data can be reconstructed robust to erasures Multiple Description Coding (MDC) partition data into sets (ex: odd, even samples) compress each set independently reconstruct via interpolation progressive and robust to erasures and errors Democracy CS reconstruction CS measurements are robust to erasures progressive

59 Conclusions However strong the general case for democracy, it is not an ultimate or absolute value, and must be judged by what it will achieve. It is probably the best method of achieving certain ends, but not an end in itself. -F. A. von Hayek

60 Conclusions rethink the approach to saturation mitigation Democracy can use any subset of CS measurements (if subset is large enough) Two New Approaches discard saturated measurements include saturated measurements as constraint Hallmarks improved reconstruction performance improved robustness to saturation nonzero saturation rates are encouraged Extensions simple AGC based only on saturation rate dsp.rice.edu/cs