L1-Optimized Linear Prediction for Light Field Image Compression

Transcription

1 L-Optimized Linear Prediction for Light Field Image Compression Rui Zhong, Shizheng Wang 2, Bruno Cornelis, Yuanjin Zheng 2, Junsong Yuan 2, Adrian Munteanu Department of Electronics and Informatics (ETRO) Vrije Universiteit Brussel, Brussels, Belgium School of Electrical and Electronic Engineering 2 Nanyang Technological University, Singapore shizheng.wang@foxmail.com, yjzheng@ntu.edu.sg, jsyuan@ntu.edu.sg Abstract The advent of consumer-level plenoptic cameras has sparkled the interest towards the design of efficient compression techniques for light field images. State-of-the-art compression systems such as prove to be inefficient when directly applied on this type of data due to the inherent spatial discontinuities among neighboring microlens images. In this paper, a novel light field image compression system is proposed. The disk-shaped pixel clusters corresponding to each microlens in the light field image are efficiently predicted based on the neighboring disks. In this context, an optimized linear prediction design based on L minimization of the residuals is proposed. K- means clustering is employed on training data in order to determine the optimized set of predictors. The experimental results on an extensive set of light field images demonstrate that the proposed coding scheme yields an average of 2.93 db and 3.22 db gain in, and 52.67% and 57.27% average rate savings compared to and 2 respectively. I. INTRODUCTION Arun Gershun introduced the concept of light fields in [], referring to the amount of light traveling in every direction through every point in space. The high dimensional light field data provides sufficient information to enable a broad range of applications in image-based rendering [2], computer graphics [3][2], free-viewpoint video [4], and many more. Cheap sensor technology and technological advances in micro-optics have led to the commercialization of the first plenoptic cameras. Lytro, which is the first plenoptic camera addressing the consumer market, combines microlens arrays with high-resolution image sensors to capture the 4D light field data [5].The camera allows for digital re-focusing after light field acquisition and 3D effects [6]. The Lytro micro-lenses have a disk shape in the sensor plane (see the example of Fig. ); the disk-shaped cluster of pixels is sometimes called a superpixel see e.g. [7]. The actual shape of a super-pixel is approximately a disk with 99 pixels, as depicted in Fig.. The light field image resolution depends on the number of microlenses and the pixel count in each disk. In Lytro II cameras, there are approximately 2, micro-lenses, leading to huge resolution light field images [7]. To enable world-scale deployment and use of such devices, storing, processing and transmitting light field image data needs to be efficient and user-friendly. Despite these needs, an efficient compression system for light field image data still needs to be developed. The conventional standard [8] is a well-known, yet rudimentary candidate to compress light field image data. A powerful alternative is given by the intra-codec from the standard [9]. is the state-of-the-art in video coding, employing very efficient coding tools and substantially improving compression performance over all of its predecessors. However, the existing encoder is designed with the assumption of local spatial and temporal continuities in video. Light field images are characterized by systematic spatial discontinuities between microlenses (see Fig. ), which leads to inefficient intra-prediction, poor residuals and high bitrates. Specific designs for light field image compression include the approach of B. Girod et. al. [9] which is based on a disparity-compensated lifting scheme, solving the transform irreversibility problem in wavelet-based light field coding and offering various forms of scalability. In [], a novel selfsimilarity compensated prediction method was proposed, yielding high coding efficiency for 3D holoscopic video. The patch-match based compensated prediction identifies for a given block the most similar block from the neighboring reconstructed regions []. This compensated prediction method works well for the considered holoscopic images involving a regular grid of rectangular super-pixels. However, these methods are not compatible with the disk-shaped super-pixels produced by Lytro cameras. The lenslet data can be converted into conventional 2D images by collecting from each disk the pixels with the same spatial coordinates; this yields a conventional 2D image corresponding to a particular viewing angle. The pixels in such an image are spatially correlated, indicating that pixels with the same spatial coordinates within the disks are also correlated. This observation motivates our approach, in which we explore the disk-based plenoptic redundancy in order to achieve efficient compression. Essentially, we propose a linear prediction method to predict the disks in the lenslet image based on the decoded neighboring disks. The major lines of reasoning in the proposed method are as follows: (i) establish the disk as the coding unit instead of using conventional block-based structures as in ; (ii) determine optimized weights for each neighboring disk by minimizing the L distortion between the prediction and the original light field; (iii) perform clustering of the weights obtained in a training phase and determine optimized intraprediction modes; (iv) use these intra-modes at run time and adapt the -based coding tools to encode the residuals.

2 Figure 3. Spatial configuration of the disks corresponding to neighboring micro-lenses in the light field image: the reference disks R, R, R are located around target disk T. 2 3 Figure. Cropped light field image I_bike with a magnified super-pixel (or disk). Figure 2. The offline to online processing framework of the proposed method. (a) Original weight parameter sets obtained by minimizing the L-norm. The remainder of the paper is organized as follows. The proposed framework for light field image coding is given in section II, while the L-norm residual formulation and optimization algorithm are presented in section III. In section IV, we report and analyze the experimental results. Section V draws the conclusions of this work. II. PROPOSED LIGHT FIELD COMPRESSION SYSTEM USING TRAINED WEIGHTED PREDICTION The proposed compression system consists of two phases, as shown in Fig. 2. In an offline phase, the weights used to express the linear relationship between the current (i.e. target) disk and its neighboring (i.e. reference) disks are determined. In a subsequent, online phase, the trained weights are used for prediction and the sparse residual is encoded. A more detailed description of the full framework is given below. A. Offline Training The target disk is modelled as a linear combination of its three nearest neighboring reference disks. The target disk and its reference disks are depicted in Fig. 3. The weights expressing the linear relation between the target and reference disks are obtained by solving the following problem: 3 T min, subject to,, 2, 3 wp W w w w p T W R () (b) Trained weighted parameter sets after K-means clustering ( K 32 ). Figure 4. Trained weighted parameter sets before and after K- means clustering. where R ( R, R2, R3 ) denote the reference disks surrounding the target disk T (T is a P matrix, W is a 3 matrix, R is a 3 P matrix, and P 99 is the number of pixels in a disk, as illustrated in Figure 3). Variables w, w2 and w 3 are the weights that need to be found as solution of the optimization problem (), while satisfying the obvious constraint that they sum to. We choose an L-norm to minimize the residual between the target pixel and its linear reconstruction, for its robustness

3 against outliers in the data and in order to increase the efficiency of the subsequent online residual encoding step (consisting of a DCT transform and quantization). The optimization problem () is solved using a standard L constrained-optimization toolbox [8]. Due to the vast amount of disks (dictated by the resolution of the lenslet image), it would be inefficient to encode the weights W for each disk T. Therefore, the original weights W obtained as the solution of () are clustered by means of a K-means clustering step [2], which allows for an efficient indexing of the weights by each cluster center. Note that there is no need to transmit the whole weight set to the decoder; the training and the derivation of the centroids can be done offline (see Fig. 2), and the results can be simply stored in a look-up table, known at both the encoder and decoder sides. Hence, in practice, only the index of the representative cluster center is encoded for each disk T. In the experimental section, we set the number of clusters K to 32. In Fig. 4, there are the trained weight sets before and after K-means clustering. B. Online Encoding with Weighted Neighbors Prediction In our coding approach, we regard a disk as the basic encoding unit. This lies in contrast to conventional coding paradigms that rely on block-based coding structures and tools. The proposed encoding framework consists of 4 parts, namely, disk-based intra prediction, transformation, quantization, and entropy coding. Intra-prediction incorporates a disk skip mode, and the K linear prediction modes, as introduced in section II. It is important to note that none of the original s intraprediction modes, except for the skip mode, are being used. During disk-based intra prediction, one chooses the nearest three reconstructed disks as references to generate the prediction of the target disk T based on the trained prediction weights. In this respect, we traverse all the K prediction modes, determined as described in section II, in order to compute the best linear prediction mode, as follows: T W arg min T W R, (2) best W W, W2,, W K where wk, k K is the set of trained prediction weights corresponding to intra-mode k. One uses log K 2 bits to index the best linear prediction mode wbest for each target disk T ; as mentioned, in the experiments one choses K 32. After intra prediction, the proposed method uses s discrete cosine transformation (DCT), quantization, and contextadaptive binary arithmetic coding (CABAC) to further encode the residuals. Following the closed-loop coding paradigm, entropy decoding, inverse quantization, and inverse transformation are all part of the encoder to generate the reference reconstructed disks. One notes that entropy coding and decoding are also part of the loop for debugging purposes, that is, to guarantee that, even if the processing unit is changed from a block to a disk, correct encoding is performed by matching the coder with the corresponding decoder. The compressed stream makes use of s syntax elements of intra prediction and residual information [9]; we adapt them to the proposed codec and design the corresponding syntax elements for the proposed intra prediction modes. In intra prediction, the original syntax elements of consist of the CU_skip_flag, the PartMode indicating whether the current Coding Tree Unit (CTU) is partitioned into smaller size, the most possible intra prediction modes (Planar, DC and Angular modes), and block residual coefficients for intra prediction [3]. In the disk-based prediction scheme, the coding unit is fixed to a disk, which makes PartMode unnecessary. Thus, the syntax elements are the skip flag and the index of the best linear prediction mode. The latter is indicated by modifying the intra mode syntax elements in ; and prev_intra_luma_pred_flag, rem_intra_luma_pred_mode, and mpm_idx are used to code the index of the best linear prediction mode for each disk. The value of rem_intra_luma_pred_mode is replaced by the index of the best weight set, while mpm_idx is substituted by the most possible weight sets index. The residual coefficients are conventionally encoded using CABAC. III. EXPERIMENTAL RESULTS AND ANALYSIS A. Experimental setup and evaluation The proposed light field image compression method is implemented based on the HM reference software [5]. Thorough experiments are carried out on raw light field image dataset provided by EPFL [4]. After demosaicing and devignetting, the lenslet images contain 3 RGB channels and weight channel. The proposed method is applied only on the 3 colour channels. The EPFL light field image dataset includes 2 light field images captured by the Lytro camera [4]. The resolution of the raw bit images is 7728x5368 requiring bytes. The raw images are demosaiced, devignetted, clipped from bit to 8 bit, and then converted to the YCbCr4:2: format for encoding. To evaluate the compression performance, we determine the and cost in bytes of the encoded light field images against three conventional coding systems, namely, [8] and 2 [9] which serve as reference codec for the consumer market, and the state-of-the-art [3]. There is only one frame to be encoded, so is set to operate in intra prediction mode; the CTU size is set to 32x32, which can be partitioned into 6x6 blocks. The linear predictors are determined based on only one of the test images from the EPFL dataset (i.e. I_Bike ) which serves as training data. The resulting weights are subsequently used in the coding experiments for all the other light field images. To comprehensively evaluate the coding performance, we set 4 basic QPs, namely 22, 27, 32 and 37. After all the disks are compressed by the proposed method, the decoded disks form the decoded light field image. The of the decoded light field image is computed relative to the raw 8 bit image as: log, with MSE, MSE M 2 ( Lp Lp) (3) N p where, M is the number of disks in a frame, and N is the image resolution ( M , N P for Lytro images, P 99 is the number of pixels in a disk unit); Lp and L p are the reconstructed and original sets of pixels in disk p. As expressed by (3), the MSE accumulates the squared differences between decoded and original disks in the lenslet image.

4 I Bike I5 Vespa I7 Desktop I_Color_Chart_ Figure 5. Rate-distortion performance comparison between the proposed method, conventional, 2 and. B. Experimental results In Fig. 5, the rate-distortion curves are shown for light field images I_Bike, I5 Vespa, I7_Desktop and I_Color Chart from the EPFL dataset. The compression ratios obtained relative to the original raw light field data are reported in Table I. TABLE I. COMPRESSION RATIOS FOR THE PROPOSED METHOD RELATIVE TO UNCOMPRESSED RAW DATA Image Compression ratios for proposed method QP I_Bikes I2_Danger I3_Flowers I4_Stone_Pillars I5_Vespa I6_Ankylosaurus I7_Desktop I8_Magnets_ I9_Fountain I_Friends_ I_Color_Chart_ I2_ISO_Chart_ Overall, at low rates, the is the same or slightly lower than that of ; however, the reaches also unappealing ranges for this type of data (below 35 db). On the other hand, at medium and high rates, the proposed method reaches much better compression performance compared to. Table II reports the BD- and BD-BR computed using Bjontegaard s evaluation tools [6]. The results demonstrate that the proposed compression method achieves very high rate savings compared to the state-of-the-art. Overall, the average gain is 2.93 db, 3.22dB and 9.6 db against, 2, and respectively, corresponding to 52.67%, 57.57%, and 73.69% rate savings respectively. Maximum gains in rate relative to go as high as 79.82%. These very large rate savings prove that the proposed disk-based linear prediction approach is particularly efficient on this type of data. The complexity of the proposed method is evaluated by the average encoding time. Specifically, the average encoding time costs are , 27.72, , and seconds for 4 QPs (22, 27, 32, and 37). As for each disk, the coding time is nearly.97 millisecond. In, the coding time costs are , 74., , seconds respectively, which are approximate 25 times slower than the proposed method. IV. CONCLUSIONS In this paper, we have proposed a novel compression system for light field image data. The proposed approach exploits the spatial correlation amongst neighboring disk-shaped pixel clusters corresponding to each microlens in the light field image. With this respect, we capture these correlations by assuming a linear dependency model between a target disk and its neighboring reference disks in the lenslet image. The efficiency of state-of-the-art compression schemes, such as, on this type of data is low due to the presence of the

5 TABLE II. BD- OF THE PROPOSED METHOD COMPARED TO, 2, AND. Image Vs Vs 2 Vs I_Bikes I2_Danger I3_Flowers I4_Stone_Pillars I5_Vespa I6_Ankylosaurus I7_Desktop I8_Magnets_ I9_Fountain I_Friends_ I_Color_Chart_ I2_ISO_Chart_ Average inherent spatial discontinuities in light field data and due to the use of block-based coding structures which do not match well the microlens pattern. In our approach, we propose a disk-shaped coding unit and choose an L-norm to minimize the residual between the target disk and its linear reconstruction. These choices significantly increase the efficiency of the subsequent residual encoding step performed using an adapted. The experimental results demonstrate that the proposed coding method achieves significantly higher and particularly high rate savings compared to. ACKNOWLEDGEMENT The work in this paper has been supported by FWO, the 3DLicornea project funded by the Brussels region (Innoviris), and by the Singapore Government under grant No. NRF-CRP REFERENCES [] A. Gershun, The light field, Journal of Mathematics and Physics XVIII, pp.5 5, 936. [2] M. Levoy, P. Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 3-42, 996. [3] G. Wetzstein, D. Lanman, W. Heidrich, and R. Raskar, Layered 3D: tomographic image synthesis for attenuationbased light field and high dynamic range displays, ACM Transactions on Graphics, vol. 3, no. 4, July 2. [4] J. Carranza, C. Theobalt, M.A. Magnor, H.-P. Seidel, Free-viewpoint video of human actors, ACM SIGGRAPH 23, pp , 23. [5] R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, Light field photography with a hand-held plenoptic camera, Computer Science Technical Report CSTR, 2(), pp.-, 2. [6] T. Georgiev, Z. Yu, A. Lumsdaine, and S. Goma, Lytro camera technology: theory, algorithms, performance analysis, Procceedings of SPIE, vol. 8667, 23. [7] G. Wetzstein, I. Ihrke, W. Heidrich, On plenoptic multiplexing and reconstruction, International Journal on Computer Vision, vol., no. 2, pp , 23. [8] K. G. Wallace, The still picture compression standard, Communications of the ACM, pp. 3-44, 99. [9] A. Skodras, C. Christopoulos, and T. Ebrahimi, The 2 still image compression standard, IEEE Signal processing magazine, vol. 8, no. 5, pp , 2. [] G.J., Sullivan, J.-R. Ohm, W. Han, and T. Wiegand, Overview of the High Efficiency Video Coding () standard, IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 2, pp , 22. [] C. Conti, L.-D. Soares, and P. Nunes, -based 3D holoscopic video coding using self-similarity compensated prediction, Signal Processing: Image Communication, vol. 42, pp , 26. [2] Hartigan, J. A.; Wong, M. A. (979). "Algorithm AS 36: A K-Means Clustering Algorithm". Journal of the Royal Statistical Society, Series C 28 (): 8. [3] High Efficiency Video Coding, Recommendation ITU-T H.265/International Standard ISO/IEC 238-2, 25. [4] M. Rerabek, L. Yuan, L. Authier, and T. Ebramini, EPFL light-field image dataset, ISO/IEC JTC/SC 29/WG 69th Meeting, e- dataset, 25. [5] Joint Collaborative Team on Video Coding (JCT-VC), reference software, HM version 6.8, [6] G. Bjontegaard, Calculation of average differences between RD-curves, VCEG Contribution VCEG-M 33, Austin, April 2. [7] G. Wetzstein, I2. : Invited Paper: On the duality of compressive light field imaging and display, SID Symposium Digest of Technical Papers, vol. 46. no [8] J. Duchi, L-norm: Methods for Convex-Cardinality Problems, Stanford Technical Report, 28. [9] B. Girod, C.-L. Chang, P. Ramanathan, and X. Zhu, Light field compression using disparity-compensated lifting, IEEE International Conference on Multimedia and Expo, ICME 23, vol., pp. I-373, 23. [2] K. Changil, K. Subr, K. Mitchell, A. Sorkine-Hornung, and M. Gross, Online view sampling for estimating depth from light fields, IEEE International Conference on Image Processing, ICIP 25, pp , 25.