Block-based Video Compressive Sensing with Exploration of Local Sparsity

Block-based Video Compressive Sensing with Exploration of Local Sparsity Akintunde Famodimu 1, Suxia Cui 2, Yonghui Wang 3, Cajetan M. Akujuobi 4 1 Chaparral Energy, Oklahoma City, OK, USA 2 ECE Department, Prairie View A&M University, Prairie View, TX, USA 3 ET Department, Prairie View A&M University, Prairie View, TX, USA 4 College of CSMT, Alabama State University, Montgomery, AL, USA Abstract Nowadays, compressive sensing is becoming an increasingly popular method for signal processing. The fact that it supersedes the Shannon-Nyquist criterion for sampling makes it very attractive for processing image/video signals in the increasingly media rich environment. Features individual measurement package in transmission, compressive sensing based coding system is more robust to channel noise. The block-based approach lends itself to a parallel processing and makes it capable for applications such as surveillance camera. The purpose of this paper is to improve the pioneering work done in compressive sensing by developing and implementing a block sparsity detection system. Furthermore, different transform based local sparsity decision methods are applied to the above mentioned system to compare the effectiveness of sparsity detection. Experimental results show that the Gini index calculated in Discrete Wavelet Transform domain is a promising method for detecting local sparsity. Keywords: Compressive Sensing, Sparsity, Gini Index, Discrete Wavelet Transform 1 Introduction It is an undisputed fact that multimedia contents in the form of image, audio, and video are increasing exponentially in our society recently. How to efficiently transmit multimedia information through network is critical. Traditional image and video coding standards (JPEG, MPEG, H.26x) are very successful in signal decorrelation which is the main contribution of data compression. They have achieved decent compression ratio in order to save transmission bits. But the limited resiliency to channel errors makes them not applicable under noisy wireless network [1]. Also the dominated video coding standards feature a more complicated encoder compares to decoder, this paradigm is suitable for broadcasting or downlink nature of traditional video applications [2]. Since the encoder incorporates motion estimation motion compensation (MEMC) and all decoding procedures, it is very time and energy consuming. This scheme does not fit for the emerging uplink applications where the encoder resides in a more mobile device like cell phone. The capturing and transmitting of multimedia signals are still supported at the simple encoder side which demands low-power and less computational complexity. Obviously the traditional hybrid video encoder needs a technique revolution. In recent years, people show interests in compressive sensing (CS), where signals can be sampled at a sub- Nyquist rate on a random basis and could be perfectly reconstructed with an assumption of sparsity [3-8]. The compressive sensed signal is transmitted and reconstructed by each random measurement. The dependency on critical packages for decoding has been reduced. The signal reconstruction quality related to the percentage of measurements received, not specific ones. As a result, CS based codec is more robust [7] to noisy channel condition. To introduce a CS based video coding scheme, this paper is organized as follows. Following the introduction of section I, section II provides background information about CS, sparsity, and Gini index. Section III presents the proposed coding system. Experimental results and conclusions are shown in section IV and V respectively. 2 Background It is quite evident that if applied appropriately, CS has a large variety of potential applications. Examples range from single pixel camera [9] and Magnetic Resonance Imaging [10], to wireless sensor networks [11] and facial recognition [12]. Among them, CS-based image processing is investigated. In [6], a block-based random image sampling coupled with a projection-based reconstruction provides a smooth reconstruction as well as making the parallel processing possible in future hardware implementation. Beyond the success of CS-based image application, several trials have been explored to extend it to video processing. Stankovic and et. al proposed a block-based compressive video sensing algorithm which utilizes the DCT domain sparsity as a guidance to code a sequence of video frames [13]. 2.1 Compressive Sensing (CS) CS allows sampling to occur at a sub-nyquist rate onto a random basis and could also be reconstructed to the original signal with the condition of sparsity. If the N- dimensional signal x is of K-sparse (only K none zero coefficients) with respect to some transform matrixψ.

The sampling of x is then a linear transformation by a matrix Φ to a vector y=φx. Let Φ be an matrix where K<M<<N. The usual choice for the measurement basis Φ is a random matrix. We further assume that Φ is orthonormal such that ΦΦ T =I, so y has M elements; we call each element of y as a measurement of x. The decoder recovers the signal x from y knowing ψ and Φ. The measurement process is non adaptive in that Φ does not depend in any way on prior knowledge of the signal x. It is a random matrix. The magic of compressive sampling is that Φ can be designed such that a sparse/compressible x can be perfectly reconstructed from the measurements y. Sparsity relates to the fact that a signal or data set may contain more information or detail in some parts than others. The reason behind CS holds that it is possible to save storage space and still acquire an accurate representation of the image by sampling the more detailed areas at a higher sampling rate and the less detailed areas at a lower sampling rate. This reduces the number of samples taken and thus, is more efficient in utilizing the storage space and transmitting the signal. CS exploits the fact that many natural signals are sparse or compressible in the sense that they have concise representations when expressed in the proper basisψ. This is achieved by using an appropriate transform to convert the data into that basis. This means that to find out the sparsity distribution of a signal, we need to convert it into an appropriate form using a suitable transform. Conventional sampling follows the following scheme: The full N-sample signal x is acquired. In transform coding, the complete set of transform coefficients {s i } is computed via S=ψ T x; where ψ is a transform matrix that can represent signal x in a more sparsed domain. The K largest coefficients are kept and the rest (N K) coefficients are discarded [8]. Finally, the K values and locations of the largest coefficients are encoded. This traditional approach of compression suffers from over sampling and transform of all N samples, while CS takes care of the above drawbacks and is achieved by a recursive reconstruction algorithm of the original signal, x. 2.2 Sparsity Compressive sensing makes sub-nyquist rate sampling possible by utilizing the concept of sparsity. It is common knowledge that some parts of an image contain less detail than others. We consider that those regions are more Sparse. With the concept of sparsity in hand, it is reasonable to assume that such image can be accurately represented, or reconstructed using data that does not have to be evenly sampled. For example, one could sample the objects with detailed information at a higher sampling rate and the background at a lower rate. This scheme would afford a number of benefits, some of which are less memory space taken, faster processing time for sampling at encoder side. It is important to note that the scheme employed in this research work divides the image to be sampled into square blocks for easy processing. This leads to decisions concerning sparsity of a region to be made in a discrete manner. Furthermore, such information about the sparsity can be sent earlier to the decoder. And each block can be sampled and transmitted without having to wait for the whole image to be sampled as in traditional video coding. Image x is divided into equal sized blocks for easy processing. We sample x compressively by multiplying it with a suitable random matrixφ, also known as the measurement matrix. It is of the dimensions. The purpose of Φ is to scan the image to obtain measurements of the signal. After CS random projection, each measurement becomes an M dimension vector. If the sparsity of each block is different, we expect the M value for each block should be different. The differences are determined by the sparsity test administered to determine which blocks are sparse or not from the key frames. After multiplying x by varying dimension ofφ, we obtain the result of the encoder, ready for transmission to the decoder side of the system. 2.3 DWT Gini Index Sparsity decision need to be made for each block before apply the random measurement. There are several different methods to make the sparsity decision. In [13], DCT domain coefficients were chosen. Of course another popular transform domain Discrete Wavelet Transform (DWT) is also a possible choice. In our approach, we choose Gini Index from DWT domain to discuss the sparsity. Gini index is a measure of income equality or disparity in a population. By its definition, it is easy to relate the Gini index to sparsity, since a Gini value of 0 shows that income is concentrated in the hands of one individual or that the data is sparse [14]. Fig. 1 shows how the Gini index is calculated graphically using the Lorentz curve: 1 y x Fig. 1 Graphical Illustration of Gini Index Calculation [15] The Gini index is equal to: B/A A B L(x) 1

Using a formula, it is calculated by: G = 1 2 L( x) dx, where L(x) is a function that represents the Lorentz curve. To further explore the sparsity decision process, a DWT Gini Index based algorithm is proposed. This is an improvement on the ordinary DWT algorithm. It is implemented by three steps: The input frame is transformed into the DWT domain. The diagonal subband is chosen to calculate the Gini index value. A predetermined threshold is applied to the chosen DWT high band coefficients to determine sparsity. As the high band DWT coefficients contain detail information, the edge information is extracted from the image and resides in high band. Calculate Gini index from high band can represent the texture of the image more accurately than low band or spatial coefficients. Therefore, the sparsity decision is more precise. Also in Gini index method, it is possible to focus on only one parameter to determine the threshold to reduce the decision complexity. 3 Proposed Compressive Sensing Video Coding System Fig. 2 gives the system diagram of the proposed compressive video sampling system. In order to illustrates the efficiency of proposed DWT Gini Index method, other sparsity decision methods: DCT, and spatial Gini are also shown in the processing box along with DWT Gini Index, but in the real system, only one method is chosen. And we ve proved in later context that DWT Gini Index is a better choice. The video signal is divided by reference frames and non-reference frames. Only reference frames are used to calculate the sparsity. Non-reference frames use the decision obtained from reference frames to guide the sampling. Here compressive sampling applies to sparse blocks where Φ matrix is of dimension. The output coefficients reduce to a size vector. The sampling ratio is determined by. In a sparse case, this ratio is chosen among 10%, 20%, or 30%. Conventional sampling applies to non-sparse blocks where the value in the Φ matrix can be increased up to the same value of, where the signal will be 100% sampled. 4 Experimental Results The simulations to obtain data concerning the sparsity decisions were performed on the MATLAB platform. Two popular QCIF sequences were used. One is Akiyo, 1 0 Non-ref Frame Reference Frame Divide into Blocks Switch (drives by sparsity decision) Sparsity Test Divide into Blocks Fig. 2 System Diagram Conventional Sampling Compressive Sampling DCT /Spatial Gini /DWT Gini on Each Block (a) Akiyo (b) Hall Monitor Fig. 3 Test Video Sequences Sparsity Decision as shown in Fig. 3 (a), a video conference type. The other is Hall-monitor, Fig. 3 (b), a surveillance camera feed. For each sequence, we choose the first frame as the reference frame and the rest as non-reference frames. There are several parameters need to be modified to optimize the system. They are: Block subsample rate for sparse blocks and for non-sparse blocks): The nature of this coding system is to have dramatically lower than. Both rates directly relate to the Φ matrix dimension of. To implement this scheme, two different values of will be chosen according to the sparsity decision of each block. This sparsity decision will be sent to the decoder as a binary mask with the reference frame. Sample rate ( ): The overall sampling rate for the whole image includes both sparse blocks and non-sparse blocks. This is calculated by: / where is the number of sparsed blocks, and is the number of non-sparse blocks. Block Size (B): This is the size of the blocks that an original image is divided into before the test for sparsity is carried out. The images came in the QCIF format, so block size is set to 16 by 16 experimentally. T and C: In DCT based sparsity decision method [13], if the number of DCT coefficients in the block whose

absolute value is less than C is larger than T, the block is a sparse block. Threshold (TH): This is the threshold to determine sparsity in Gini Index related methods. If the Gini coefficient value calculated from a block is lower than this threshold, the block is treated as sparse. It should also be noted that different sparsity decisions lead to different reconstruction results since the decoder can only receive substantially reduced samples for sparse blocks. Fig. 4 to Fig. 6 compare three different methods in the choice of sparsity blocks. The line across the graph is the sparsity decision for each method. The blocks above the line are sparse while the blocks below the line are non-sparse. For the DCT based method, our parameters included T = 200 and C=4 and 16 as an example shown in Figs. 4-6 give the spatial Gini and DWT Gini coefficients respectively. The TH is selected as 0.15 for spatial Gini, and 0.5 for DWT Gini. This also shows the DWT Gini coefficient values are more spread out and easy for thresholding. 300 DCT Sparsity Decision DWT Gini 0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 DWT Gini Sparsity Decision 0.35 0 10 20 30 40 50 60 70 80 90 100 No. of blocks Fig.6 Distribution Plot Showing DWT Gini Coefficient Values, While Threshold TH = 0.5. Fig. 7 illustrates the sparsity decision of one video frame of Akiyo where the sparse blocks are marked by squares. This result gives an accurate representation of sparsity in the image being sampled. The blocks with a square boundary will be sparsely sampled at a rate of, which is 10%, 20% or 30%. 250 200 DCT 150 100 50 0 0 10 20 30 40 50 60 70 80 90 100 No. of Blocks Fig.4 Distribution Plot Showing Coefficient Values for T = 200, C=4 and 16 for the DCT-based Sparsity Decision. Fig. 7 Image Shows Sparsity Decision Using DWT Gini Index Method Spatial Gini 0.4 0.35 0.3 0.25 0.2 0.15 Spatial Gini Sparsity Decision The reconstruction technique of each video frame utilizes the BCS-SPL-DDWT approach introduced in [6]. The results of the simulation are shown below. Fig. 8 gives a decoded video frame of Akiyo. Fig. 9 and 10 provide reconstructed PSNRs from Akiyo and Hall Monitor of 30% and 10% subsample rate for sparse block respectively. 0.1 0.05 0 0 10 20 30 40 50 60 70 80 90 100 No. of block Fig.5 Distribution Plot Showing Spatial Gini Coefficient Values, While Threshold TH =0.15.

a v e r a g e P S N R Fig. 8 Reconstructed Image Fig. 9 PSNR vs. Sample Rate for Akiyo with Subsample Rate for Sparse Block to be 30% average PSNR 35.5 35 34.5 34 33.5 23.2 23 22.8 22.6 22.4 22.2 22 21.8 DCT Spatial Gini Index DWT Gini Index Fig. 10 PSNR vs. Sample Rate for Hall Monitor with Subsample Rate for Sparse Block to be 10%. It is observed that DWT Gini Index outperforms other methods for most of the range of Hall-monitor sequence. It also provides better reconstruction quality for nearly half range of the Akiyo sequence. 5 Conclusion akiyo q cif, subrate = 0.3 0.49 0.5 0.51 0.52 0.53 0.54 0.55 0.56 sample rate DCT Spatial Gini Index DWT Gini Index hall-monitor, subrate = 0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 sample rate The intent of this research work is to determine an effective technique of detecting local sparsity in a video sequence and thereby guide block based compressive sampling. It improves the sparsity decision algorithm from [13]. In conclusion, DWT Gini Index can locate sparsity more accurately as the results shown in Fig. 9 and Fig. 10 that use DWT Gini Index to determine the sparsity blocks can get better result and the reconstructed frame PSNR outperforms other methods. The experimental results point out that the DWT domain coefficients coupled with Gini index calculation is a promising method of determining sparsity. Also, it proves that the DWT Gini method is less complex and is comparable in performance with the popular DCT based method. In specific situation as a security camera sensed video sequence, the hardware can be designed according to the fixed sparse background to reach low sampling rate and omit the sparse decision process. 6 Future Work This work did not explore the area of motion compensation/estimation. For a more robust system, research could be carried out to build an efficient motion estimation algorithm at the decoder side to better utilize temporal redundancy. In this vein, exploration could be made to find out if placing the reference frame iteratively at strategic points/intervals in the video stream will improve compressive sensing performance. It is also a good idea to explore a more efficient decoder to reduce the burden on the encoder. Acknowledgements The authors would also like to acknowledge Dr. James Fowler and Dr. Sunkwang Mung for providing the Block Compressed Sampling-Smooth Projected Landweber (BCS-SPL) reconstruction algorithm packages. References [1] S. Pudlewski, and T. Melodia, On the Perforemance of Compressive Video Streaming for Wireless Multimedia Sensor Networks, 2010 IEEE InternationalConference on Communications (ICC), pp. 1-5, Cape Town, South Africa, May 23-27, 2010. [2] G. Hua, C. W. Chen, Distributed Vidoe Coding with Zero Motion Skip and Efficient DCT Coefficient Encoding, IEEE International Conference on Multimedia,and Expo, pp. 777-780, year 2008. [3] E. Candes and Tao, Near-optima signal recovery from random projections: Universal encoding strategies? IEEE Transactions on Information Theory, vol. 52, no. 12, pp. 5406 5425, December2006. [4] D.L.Donoho,Compressed sensing, IEEE Transactions on Information Theory, vol. 52, no. 4, pp. 1289 1306, April 2006 [5] E.J.Candes and M.B.Wakin, An introduction to compressive sampling, IEEE Signal Processing Magazine, vol. 25, no.2,pp.21 30, March 2008. [6] S. Mun and J.E.Fowler, Block compressed sensing of images using directional transforms, Proceedings of the International Conference on Image Processing, Cairo, Egypt, pp. 3021-3024, November 2009.

[7] S. Schnell, J. N. Laska, C. Hegde,M. F. Duarte, M. A. Davenport, R. G. Baraniuk, Texas Hold em Algorithms for Distributed Compressive Sensing, in Proceedings of the 2010 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.2886-2889, Dallas, TX. Mar. 14-19, 2010. [8] R. Baraniuk, Compressive Sensing, IEEE Signal Processing Magazine, Vol. 24 No. 4, pages 118-121, July 2007. [9] M. F. Duarte, M..A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, R. G.Baraniuk, Single Imaging via Compressive Sampling, IEEE signal processing magazine, pages 83-89, March 2008. [10] J. P. Haldar, D. Hernando, Z. Liang, Compressed Sensing MRI with random encoding, IEEE transactions on Medical Imaging, vol. 30, no. 4, pp. 893-903, Oct. 2011. [11] W. Bajwa, J. Haupt, A. Sayeed and R. Nowak, Joint source-channel communication for distributed estimation in sensor networks, IEEE Transactions on Information Theory, vol. 53, no. 10 pp. 3629-3653, October 2007. [12] M. H. Mahoor, M. Zhou, K. Veon, S. M. Mavadati and J. Cohn, Facial Action Unit Recognition with Sparse Representation, 2011 IEEE International Conference on Automatic Face & Gesture Recognition, pp.336-342, March 2011. [13] V. Stankovic, L. Stankovic, and S. Cheng, Compressive video sampling European Signal Processing Conf. (EUSIPCO), Lausanne, Switzerland, August 2008 pages 1-5. [14] O. O. Odejide,C. M. Akujuobi, A. A. Annamalai, G. L Fudge. Determination of Best Sparsifying Basis for Compressive Sampling, International Review of Electrical Engineering, vol 4, No 4, pages 327-344, 2009. [15] L. Gonzalez, and et al., The Similarity between the Square of the Coeficient of Variation and the Gini Index of a General Random Variable, Journal of Quantitative Methods for Economics and Business Administration, 2010 vol. 10, pp. 5 18. ISSN 1886-516X.