Super-resolution of Multispectral Images R. Molina, J. Mateos, M. Vega, Universidad de Granada, Granada, Spain. A. K. Katsaggelos Northwestern University, Evanston (IL). Erice, April 2007 Data Analysis in Astronomy 1
Outline I. Super resolution in Remote Sensing II. Bayesian Problem Formulation III. Global Bayesian Modeling and Inference IV. Local Bayesian Modeling and Inference V. Examples VI. Conclusions Erice, April 2007 Data Analysis in Astronomy 2
I Super resolution in Remote Sensing With an ideal sensor we would have high resolution multispectral images. Unfortunately due to spectral and spatial decimation we have: Spectral decimator Spatial decimator Erice, April 2007 Data Analysis in Astronomy 3
A band (y b )of the high resolution multispectral image (y) we want to estimate Spectral response of panchromatic sensor Observed high resolution panchromatic image (x) A band (b) of the observed low resolution multispecral image (Y b ) High resolution multispectral image (y) we want to estimate Upper case: low resolution Lower case: high resolution Y(i)=(Y 1 (i),,y K (i)) T y(j)=(y 1 (j),,y L (j)) T Erice, April 2007 Data Analysis in Astronomy 4 NOTATION
II. Bayesian Problem Formulation y = multispectral image we would observe under ideal conditions with a high resolution sensor. It has B bands To be estimated y =[y t 1, y t 2,...,y t B ]t, each band is of size p=mxn pixels We observe a low resolution multispectral image Y with B bands Available Y =[Y t 1, Y t 2,...,Y t B ]t, each band is of size P=MxN pixels with M<m and N<n. Erice, April 2007 Data Analysis in Astronomy 5
The sensor also provides us with: panchromatic image x Available of size p=mxn, obtained by spectrally averaging the unknown high resolution images y b Bayesian Goals Modeling (based on building) p(ω M, y, Y, x) =p(ω M )p(y Ω M )p(y, x y, Ω M ), Inference (based on calculating, approximating, ) p(ω M, y Y, x) =p(ω M, y, Y, x)/p(y, x) Ω M denotes the set of hyperparameters needed to describe the required probability density functions Erice, April 2007 Data Analysis in Astronomy 6
III Global Bayesian Modeling and Inference We assume that Y and x for a given y and a set of parameters Ω M are independent p(y, x y, Ω M )=p(y y, Ω M )p(x y, Ω M ). observed unknown Each band, Y b is related to its corresponding high resolution image by Y b = DHy b + n b, b =1,,B, H is a pxp blurring matrix, D is a Pxp decimation operator and n b is the capture noise, assumed to be Gaussian with zero mean and variance 1/β b. So p(y y, Ω M )= BY p(y b y b, β b ) BY β P/2 b exp ½ 12 ¾ β b ky b DHy b k 2. b=1 b=1 Erice, April 2007 Data Analysis in Astronomy 7
Graphically: (i, j) denotes low resolution pixel. This low resolution pixel consists of four high resolution pixels (u,v) with (u,v) Є H ij ={(2i,2j), (2i+1,2j), (2i,2j+1), (2i+1,2j+1)}. (2i,2j) (2i+1,2j) (2i,2j+1) (2i+1,2j+1) Low resolution pixel (i,j) Y b ( i, 1 j) = y 4 ( u, v) H ij b ( u, v) = ( Hy b )( i, Low resolution band from its corresponding high resolution band j) Erice, April 2007 Data Analysis in Astronomy 8
The panchromatic image is formed as a linear combination of the high resolution hypercube bands plus additive noise: x(u, v) = λ b b y (u, v) + ε(u, v) λ b 0 are known quantities weighting the contribution of each high resolution band we want to estimate to the high resolution panchromatic image. Mathematically, p(x y, Ω M ) γ p/2 exp There is work to be done on the estimation of these weights. Blind deconvolution techniques? Erice, April 2007 Data Analysis in Astronomy 9 b ( 1 2 γ k x B X b=1 λ b y b k 2 ).
Lansat ETM+ Spectral Response Color White Red Green Blue LANDSAT ETM+ band 1 (0.45 µm to 0.515 µm) 2 (0.525 µm to 0.605 µm) 3 (0.63 µm to 0.69 µm) 4 (0.75 µm to 0.9 µm) Color LANDSAT ETM+ band Yellow 5 (1.55 µm to 1.75 µm) Not shown 6 (10.4 µm to 12.5 µm) Cyan 7 (2.08 µm to 2.35 µm) Magenta Pan (0.51 µm to 0.9 µm) Erice, April 2007 Data Analysis in Astronomy 10
In our global image model we assume a Conditional Auto- Regressive (CAR) model. We will BY ½ p G (y Ω G ) ᾱ p 2 b exp 1 ¾ y t improve it b Cy b, b=1 later 2ᾱb where C is the Laplacian operator (with 8 neighbours). Set of unknown for our global model Parameters of the prior Parameter of the panchromatic image formation model Parameters of the low resolution image formation model Hyperspectral super-resolution image ᾱ =(ᾱ 1,...,ᾱ B ) Erice, April 2007 Data Analysis in Astronomy 11 γ β =(β 1,...,β B ) y
Results on the global model: Assuming that the parameters are known: R. Molina, J. Mateos, and A.K. Katsaggelos, Super resolution reconstruction of multispectral images in Virtual observatories: Plate Content Digitization, Archive Mining and Image Sequence processing, April 2005. R. Molina, J. Mateos, A.K. Katsaggelos, and R. Zurita-Milla, A new super resolution Bayesian method for pansharpening Landsat ETM+ imagery in 9th International Symposium on Physical Measurements and Signatures in Remote Sensing (ISPMSRS), 2006. Assuming that the parameters are unknown: R. Molina, M. Vega, J. Mateos, and A.K. Katsaggelos, Hierarchical Bayesian Super Resolution Reconstructiuon of Multispectral Images in 2006 European Signal Processing Conference (EUSIPCO 2006) R. Molina, M. Vega, J. Mateos, and A.K. Katsaggelos, Parameter Estimation in Bayesian Reconstruction of Multispectral Images Using Super Resolution Techniques in 2006 International Conference on Image Processing (ICIP 2006), Erice, April 2007 Data Analysis in Astronomy 12
Estimation of the posterior distribution of image and parameters R. Molina, M. Vega, J. Mateos, and A.K. Katsaggelos, Variational Posterior Distribution Approximation in Bayesian Super Resolution Reconstruction of Multispectral images, Applied and Computational Harmonic Analysis, special issue on Mathematical Imaging, 2007 to appear. A.K. Katsaggelos, R. Molina, and J. Mateos, Super resolution of images and video, Synthesis Lectures on Image, Video, and Multimedia Processing, Morgan & Claypool, 2007 (to appear). Some examples http://decsai.ugr.es/~rms Erice, April 2007 Data Analysis in Astronomy 13
panchromatic Low resolution bands 1 to 4 Erice, April 2007 Data Analysis in Astronomy 14
Example: Band 1 Low resolution bilinearly interpolated Super resolution band Erice, April 2007 Data Analysis in Astronomy 15
Low resolution bilinearly interpolated R = band 3 G = band 2 B = band 1 Super resolution restoration Erice, April 2007 Data Analysis in Astronomy 16
R = band 3 G = band 4 B = band 2 Low resolution bilinearly interpolated Super resolution restoration Erice, April 2007 Data Analysis in Astronomy 17
Can we improve the global model? Does it make sense to assume the same variance over each band of the whole super-resolution multispectral image? Erice, April 2007 Data Analysis in Astronomy 18
III Local Bayesian Modelling and Inference We can assume that we already have from the global model: A good estimate of the parameter of the panchromatic image formation Good estimates of the parameters of the low resolution image formation we fix it we fix them γ β =(β 1,...,β B ) Some information on the parameters of the prior from the estimated SR image ᾱ =(ᾱ 1,...,ᾱ B ) we use this information, How?, we ll see now We want to improve the image estimate by improving the image model Erice, April 2007 Data Analysis in Astronomy 19
Let us consider again the global image model p G (y Ω G ) BY ᾱ p 2 b exp b=1 which can be rewritten as ½ 1 ¾ y t b Cy b, 2ᾱb i5 i4 α b (i,i4) i3 α b (i,i3) α b (i,i2) i i2 α b (i,i1) i1 p(y Ω G )= BY p(y b ᾱ b ) b=1 BY py b=1i=1 l=1 4Y ½ ᾱ 1 8 b exp 1 2 ᾱb i6 1 8 [y b(i) y b (il)] 2 ¾, i7 i8 We can make the prior locally adaptive by writing p(y Ω L )= BY p(y b α b ) b=1 BY py b=1i=1 l=1 We are using an approximation of the partition function 4Y α 1 8 b (i, il)exp ½ 12 α b(i, il) 18 ¾ [y b(i) y b (il)] 2, Erice, April 2007 Data Analysis in Astronomy 20
α b (i, il) controls, for the b-band, the smoothness of the restoration between pixels i and il α b =(α b (i, il) i =1,...,p, l =1,...,4). and Ω L =(α 1,...,α B ) We have to estimate Ω L and y To introduce information on Ω L we write BY p(ω L )= py b=1 i=1 l=1 4Y p(α b (i, il) a o b,c o b), where for ω εω L and u ω >0 and v ω >0 we have with p(ω u ω,v ω ) ω u ω 1 exp[ v ω ω], E[ω] =u ω /v ω, var[ω] =u ω /v 2 ω. Here we can introduce information from the global model Erice, April 2007 Data Analysis in Astronomy 21
Finally we have the local modeling p(ω L, y, Y, x) =p(ω L )p(y Ω L )p(y y)p(x y), Their parameters have been estimated using the global model How do we estimate image and parameters? We approximate p(ω L, y Y, x) by q(ω L, y). where q(ω L, y) =arg min C KL (s(ω L, y) p(ω L, y Y, x)) s(ω L,y) with C KL (s(ω L, y) p(ω L, y Y, x)) = That is, the criterion used to find q(ω L,y) is the minimization of the Kullback-Leibler divergence = Z µ s(ωl, y) s(ω L, y)log dω M dy p(ω L, y Y, x) Z µ s(ωl, y) s(ω L, y)log dω L dy +const, p(ω L, y, Y, x) Erice, April 2007 Data Analysis in Astronomy 22
We choose to approximate the posterior distribution p(ω L,y Y, x) by q(ω L, y) =q(ω L )q D (y), where q(ω L ) denotes a distribution on Ω L q D (y) denotes a degenerate distribution on y. How do we estimate these two distributions? Erice, April 2007 Data Analysis in Astronomy 23
Given y K the current estimate of the image where q D (y) is degenerate then where 1 E[α b (i, il)] q k+1 (Ω L ) q k+1 (Ω L )= = μ b c o b a o b BY b=1 i=1 l=1 q k+1 (α b (i, il)) = p α b (i, il) a o b + 1 8, 1 2 Estimated from the global model Erice, April 2007 Data Analysis in Astronomy 24 py E[α b (i, il)] q k+1 (Ω L ) = a o b + 1 8 c o b + 1 2 μ b = 1 8 [yk b (i) yk b 4Y q k+1 (α b (i, il)), +(1 μ b ) 1 2 [yk b (i) y k b (il)] 2, ao b a o b + 1 8 1 8 [yk b (i) yk b (il)]2 + c o b. = αk+1 b (i, il). (il)]2 MLE estimate of 1/α b (i, il) Fidelity to the distribution of the parameters. How do we estimate it?
Given q k+1 (Ω L ) the current estimate of the distribution of Ω L o y k+1 = argmin n E[log p(ω L, y, Y, x)] y q k+1 (Ω L ). The method alternates between the estimation of the image and the distribution of the parameters Erice, April 2007 Data Analysis in Astronomy 25
V. Examples Original image Observed LR image Observed panchromatic image Erice, April 2007 Data Analysis in Astronomy 26
Bicubic interpolation Global model Local model The spatial improvement of the reconstructed image has been assessed by means of the correlation of the high frequency components (COR) which measures the spatial similarity between each reconstructed multispectral image band and the panchromatic image. We have also used the peak signal-to-noise ratio (PSNR) between the reconstructed and original multispectral image bands. Erice, April 2007 Data Analysis in Astronomy 27
PSNR COR Band 1 2 3 1 2 3 Bicubic interpolation 12.7 12.7 12.7 0.50 0.50 0.51 Using the global image model 13.5 13.4 13.4 0.68 0.68 0.68 Using the local image model 18.9 19.0 18.9 0.99 0.99 0.99 Erice, April 2007 Data Analysis in Astronomy 28
Observed LR multispectral image Observed panchromatic image Global reconstruction Local reconstruction Erice, April 2007 Data Analysis in Astronomy 29
V. Conclusions Global and Local super resolution methods in Remote Sensing have been described. Local models seem to take into account local variability quite successfully. How do we estimate the fidelity parameters? Can the local image model be improved? Erice, April 2007 Data Analysis in Astronomy 30