Perceptual Uniform Simulation for Tang Tomb Mural Inapinting by Camera Array

Transcription

1 Percetual Uniform Simulation for Tang Tomb Mural Inainting by Camera Array Meng Wu 1,2,*, Huiqin Wang 2 and Wenzong Yang 3 1 School of Management, Xi'an University of Architecture and Technology, Xi'an, Shaanxi, China, School of Information and Control Engineering, Xi'an University of Architecture and Technology, Xi an, Shaanxi, China, Shaanxi History Museum, Xi'an, Shaanxi, China, Abstract Image inainting is an effective restoration technology for damaged images used to fill the lost or deteriorated arts of art treasures. Tang tomb mural uses camera array to get high resolution digital images. It is colorful and has lenty of construction information, so it is hard to make a ercetual uniform simulation. In order to simulate the rocess of filling the damaged ixels as the restorers do, this aer studies an exemlar based inainting technique. There are some difficulties to inaint directly: i) the original exemlar-based image inainting algorithm relied on a global traversal of all ixels to find the best matching exemlar, it is a big time wasting rocess; ii) the source of the exemlar using only a single lens caturing image, there is a limit on the exemlars to get the final best effects; iii) the mural is colorful, but the revious algorithm always used RGB color model to inaint the missing atch individually in three channels which exerimentally can easily make a mistake in color dislay. To solve these roblems, in this aer the exemlars source are chosen from camera array, and reduce redundancy aim to save rocessing time. In order to simulate a visual ercetual uniform rocess, it takes the mural s color information into account by adding Lab color arameter to the data term. This hels to calculate the riority value and change the order of the filling strategy. The simulation result shows that the imroved algorithm can quickly restore the Tang tomb mural better and reduce the error accumulation. Keywords - Inainting, Mura, camera array, riority I. INTRODUCTION Tang dynasty lay great role in Chinese history. Tang tomb murals which are from the imerial tombs give so many information to hel audience to know the scene 2000 years ago [1]. There are aroximately 500 ieces of murals collected in Shaanxi History Museum. Most of them are damaged by the underground water, the bacterium, the sedimentation etc. so they have some cracked, flaked, faded, disruted, detached, owdered, mildewed disease [2]. Image inainting is a novel technique which imitates the mural restorations by filling the damaged arts according to the remaining arts based on PDE (Partial Differential Equation) or Texture synthetic algorithm. This technology has received more and more attentions in digitalized relics rotection and virtual restoration. There are some novelty rogresses on ancient murals in recent years. In 2004, Pei et al. [3] studied on Chinese ancient aintings and murals color restoration by modeling with Markov random field model to estimate the stains, cracks, and artificial damage of the image. In 2008, in order to obtain more correlation of adjacent scale wavelet coefficients, George Paandreou et al. [4] added the hidden Markov tree model of image sarse reresentation model which can adjust the multi-dimension correlation of wavelet coefficient to imrove the accuracy of image reconstruction. They use this method to reair Greece rehistoric mural. In 2012, Purkait et al. [5] develoed an exemlar based coherent texture synthesis techniques combined with a novel high-frequency generating technique that can enhance line or brush strokes. In 2013, Ghorai et al. [6] roose a novel algorithm by detecting straight and curve line to enhance the deteriorated mural images. In 2014, Kumar et al. [7] roosed an integrated algorithm to virtually enhance the mural images by taking the weighted average of original image with the mean image. In China Lu dongming s team in Zhejiang university and Wang Shuwen s team in northwest university for nationalities [8~11] their research focuses on Dunhuang grottoes murals mural restoration. However, Dunhuang murals are exosed in the oen environment while Tang tomb murals are all underground. They have different eroded models. Dunhuang grottoes murals is under the blazing sun for quite a long time, so the color fading mode is research focus oint. The Tang tomb mural is underground, for 2000 years the left information is valid and colorful, it can be the exemlars source to fill the cracked arts using the exemlar-based inainting technology. The Tang tomb mural original digital images are collected by Swiss Sinar P2 large format cameras with digital back sinar 75 LV. By using camera array to catch lots of multi-lens HD digital ictures as the exemlars source to inaint the mural image. In order to get a both efficiency and effectiveness result, this aer firstly, exand the exemlar source from one single icture to the camera array icture; secondly, reduce the redundancy by counting the similarity between the exemlars; finally, imrove the riority formula by adding the Lab color arameters that can make a ercetual uniform Tang tomb mural inainted result. DOI /IJSSST.a ISSN: x online, rint

2 II. KEY OBSERVATIONS ON EXEMPLAR-BASED IMAGE INPAINTING Exemlar-based image inainting algorithm is roosed by Criminisi [12] in It can rebuild the missing information both textures and structures simultaneously. It can imitate the stes how the ancient ainter draw the mural. It is rocessing the mural image which is left by using texture synthesis methods along the bright isohote. So the filling order which exemlar should inainted first and the filling object which exemlar is most similar to the missing art are very imortant to the results. The essential model is: Figure 1. The image of essential inainting model ˆ is the area of ˆ, and n is a unit vector orthogonal to the front in the oint. The riority is comuted for every exemlar on the boundary, with distinct exemlars for each ixel on the boundary of the target region. We comute directional similarity between the normal comonent of intensity gradient I, where the suerscrit reresents the normal comonent, and normal vector n at ixel. The rogram stes are: 1) To find the exemlar that has the highest riority in the edge of damaged area. There are two main arameters in the riority P( : ) one is the confidence term C( ) and the other is the data term D( ), and they are defined as follows: P( ) C( ) D( ) (1) Cq ( ) q ( I) C( ) (2) I n D( ) (3) 2) To find the best matching exemlar in the source region and fill it to the corresonding ositions where to be inainted. Assumed that ˆ is the highest riority exemlar that needs to be inainted, and find the best matching exemlar qˆ from the exemlars source region. Where the distance d( ˆ, qˆ) between two generic exemlars ˆ and qˆ is simly defined as the sum of squared differences (SSD) of the already filled ixels in the two exemlars. qˆ arg min d( ˆ, qˆ ) (4) q Having found the source exemlar qˆ, ' is coied from its corresonding osition from the area of qˆ. 3) To udate the confidence term After filling the exemlar, the new filled ixels become the known region from unknown region, so we need to udate all the confidence of each ixel on the edge in the inainting exemlar. C( ˆ') C( qˆ), ' ˆ (5) 4) Reeat the above stes until the entire image restoration is comlete. From the model we noticed that: 1) The essential model just uses the gray isohote as the structures data. The Tang tomb mural is underground and not affected by ultraviolet, so it has lenty color when they were excavated. If using the model inaint the color image directly it maybe not suitable. It needs to add some color information to the riority P( ) formula. This aer uses different color models to searate the color channel. And below are the inainting exeriment use RGB and Lab model. (a) Damaged image (b)rgb inainted (c)lab inainted Figure 2. The results of inainted in different color channel Picture (a) is damaged image whose missing region includes four different colors. Picture (b) is Red, Green, and Blue (RGB) color model to fill the missing art in three channels individually [13]. But RGB three channels have weak correlation, so there be some ixels get the wrong color results. Picture (c) choose L for lightness and a and b for the color-oonent dimensions (Lab) color model to fill the region in three channels individually. Lab color model can insure the exemlars to get color information correct [14]. But if just inainted Lab in three channels, there will be loose some construction information wrong such as the figure (c). So it is necessary to add the Lab arameters into the riority comutational rocess to make a ercetual uniform result. 2) The single lens just catches a art of the mural image. The whole mural image needs a camera array to get the all of the information. There is a contradiction between the efficiency and effect. If the exemlar source is just from one DOI /IJSSST.a ISSN: x online, rint

3 single lens, it limits to offer a fitful ˆq to instead of ˆ ; if the exemlar source is from the entire camera array, there may be a running time waste. So it is needed to find a suitable exemlar source to get a good balance between the efficiency and effect. MSE). If distributional difference between the adjacent ixel exemlar is small, then MSE (0 ~ 255) will aroach to zero. When the MSE between the exemlar in a continuous region are close to zero, the search rocessing in the region will have a greater redundancy, as in the figure below. III. MURAL NOVEL INPAINTING BY CAMERA ARRAY A. Exand exemlar source Usually the inainting model uses the left exemlars that just from one lens to imitate the missing art. But the Tang tomb mural is bigger than normal otions. For examle, the "Polo Play Paint" is three meters high and seven meters long. Figure 3. The Tang tomb mural Polo Play Paint It was collected by camera array to the get high definition image by taking the two adjacent ictures overla 40% -50%. So the total camera array makes a big data size, as follows. TABLE I. THE CAMERA ARRAY DATA SIZE Blocks B-1 B-2 B-3 B-4 B-5 cameras Data(G) The table I shows exemlar-based image inainting algorithm is a global traversals-based otimization model which gives the comuter CPU busy tasks if it stich the ictures at first. It will make the CPU running out, so this aer use the method that inaint the image by searate lens but the exemlars source come from the camera array. Figure 4. The camera array ictures of HD Tang tomb mural In order to reduce the redundancy and the running time, this aer use to. It defines the similarity between the exemlars use the mean square error (Mean Squared Error, Figure 5. The redundancy in continuous exemlars The image discretized with the 5 5 exemlar exemlares, and we define two adjacent exemlars that set i, j as the center for the exemlar i, j and set i 1, j1 as the center for the exemlar i 1, j, i 1 2, j 2 as the center for the adjacent exemlar i2, j resectively, the mean square 2 error between them is defined by the following formula: N N 2 2 ( i, j ( n) i, j ( n)) / N (6) n1n 1 In this aer, chose 7, it is according to the exerience of heritage restoration standard, which is suitable for texture effects of Tang tomb murals, and the redundant is controlled within reasonable limits. The inainting rocess is as follows: It discrete all the camera array images of tomb murals as I 1, I 2,, I m and set all exemlars 5 5 i, j( i, j I 1, I 2,, I m ) to make a exemlarss library. 1) To detect the damaged region edges from the I 1 which is just from one camera cature icture; 2) To calculate the riorities of each oint P ( ), ( Î dw. ) 3) To find the highest riority exemlar ˆ from the original image I 1 ; 4) To discretize the original image I 1 and the other camera array images I 2, I 3,, I m, and find the best suitable DOI /IJSSST.a ISSN: x online, rint

4 exemlar ˆq to match ˆ from the obtained exemlars library whose MSE is large than, and filled it into osition ˆ ; 5) Udated the confidence termc( ˆ '); 6) Reeat the above stes until the entire image inaiting is comlete. Exeriment results are shown as below: (a) (b) (c) (d) Figure 6. Inainting results by camera array The icture (a) is the sliced damaged mural image; the icture (b) is the marked damaged mural image; the icture (c)is the sliced inainted mural image; the icture (d)is the inainted mural image with camera array. The icture (d) shows that inainting by camera array restores the mural in single lens one by one, and uses the camera array source to fill the marked regions, then sliced the inainted ictures later. It shows a better result than (c) which sliced the ictures first and inainted later. The imroved algorithm can effectively solve the roblem between the less information in a single hoto can't get enough exemlars and sliced the multi-lens ictures together in the traversal model cause a heavy time waste. So overall, it makes a better inainted effect and obviously shortened rocessing time. We choose four different inainting methods to test the rocess time consuming. They are as follows: Figure 7. The time-consuming comarison There are four different methods to inaint the same mural 'Tang olo'. ' ' shows the time when we inaint each single-lens hotos individually and sliced all the ictures finally; 'ο' shows the time when we sliced the ictures first and then inaint the whole mural; ' 'shows the time when we sliced the ictures first and then inaint the whole mural without redundancy; ' ' shows the time when we inaint each single-lens hotos but the exemlars from the all of the camera array ictures without redundancy. So just change the exemlars source it will ensure the effect and save the running time. B. Otimize the riority by adding color arameter However, there are some exemlars which are not suitable to exress the construction information in local arts. It's because that the essential exemlar-based image inainting model just use gray-gradient to calculate the highest riority in different channels for color image inainting. In this aer art two figure 2, it shows that the inainted results by three channels individually in Lab color sace is better than in RGB color sace. This is some local details as bellows: (a) (b) (c) (d) (e) Figure 8. Inainting in different color sace. Picture (a) is the damaged local image, icture (b) is the mask image, icture (c) is the marked local image, (d) the RGB three channels inained result[13], icture (e)is the Lab three channels inained result[14]. It shows that choose Lab color sace can kee the color consistent better than DOI /IJSSST.a ISSN: x online, rint

5 RGB sace, but there are some construction error in (e). The main two reasons for this henomenon as following: 1) The riority is decided by the result of confidence term and data term. If one of them is very small or near to 0, the exemlar can be filled incorrect. If inainting is in different channels individually, the filling rocess may be chosen different exemlars in different channels. It made a worse result when the different channel reconstructed. 2) The confidence term in the texture art is less stable than data term, when the low frequency texture regions have lower riority than it. That makes the image inainting will along with the direction of lower texture exemlars and lead to the error accumulation. In [15] it shows when the inainting goes on and on, there are few original ixels in the exemlar ˆ, the exemlar's confidence value will be reduced quickly lead the riority value decent along with it as follows: Figure 9. The tendency of the riority arameters It means if a data value is very small, the exemlar can t be timely filling no matter how much confidence value is, even if the atch window is only a few ixels is unknown. So we should increase the roortion of stable data term. For the mural images, image structure information is very comlex, and the image generally has continuous exemlars of similar color. Exemlars in the objects edge lead the construction continuity, so they should get the bigger riorities. So the value of D( ) in these exemlars is more imortance. In this aer it refine the riority comutation formula, and adds the Lab color arameter into the data term directly instead of inainting L,a,b three channels searately, and changes the riority algorithm formula as flows: P( ) C( ) W ( ) W ( ) D( ) S( ) ( L) ( a) ( b) S( ) (10) C( ) is the confidence term, W( ) is the data term, it include the original data term D( ) and the color term S( ), and is the adjusting arameter.the is the mean-square, it include L, ab, three elements instead of using Lab sace channel individually; S( ) is the normalize color distribution in, is the exectation of exemlars in. It adatively changed when C( ) 0. 7& 60, then 1, 0 ; when C( ) ,then 0, 1 ; other circumstances, when C( ) 0 & 0, , It means that when the data term is small or aroach to 0, the exemlars whose color changes slowly and confidence is bigger will be inainted firstly; But when the confidence term is small or aroach to 0, it will make the exemlar s riority quickly reduce to 0, then the exemlars will be later reaired. Between these two situations it can chose the golden section oint, so that make a good balance on both C( ) and D ( ) in Lab sace. The otimized riority formula takes the mural color elements into full consideration and instead of using three channel images to inaint the color mural individually. C. Redefining filling strategy In the exemlar-based inainting riority udate model in[15], udate the filling edge once after inainting each damaged exemlar in the front region reeatedly by the riority value formula, until the all of the exemlars on it inainted then go to the next layer, it named "eeling onion skins". In this method, we noticed that the high texture areas along the construction direction may inainted on and on, cause the error accumulation caused "garbage exemlars". Meanwhile the other arts of the fronts have not changed anymore in this reeating of udating. So in this aer we roose a new udating method that not udate until all exemlars on the filling edge are inainted, it inaints all of the filling edge exemlars one by one, then udate once. It can greatly reduce the running time of mural inianting obviously, and can fill the exemlars in a ring filling order from the outside to the inside; we called it "eel a boiled egg". It efficiently de duce the "garbage exemalrs" and save considerable running time. IV. THE EXPERIMENTAL RESULTS The exeriments is on the Windows latform seven and the demo use Microsoft Visual C++ oen-source comuter vision library in 2010 (Oen Source Comuter Vision Library,OenCV2.3.1).The rocessor is Inter (R) Xeon (R) X5670@2.93GHz, and the memory is 64G. In order to see the details of the efficiency of the Tang tomb mural exeriments, we choose one single icture inainted effects to comare the different among Aujol's algorithm [13] and Suny's algorithm [14],Sai's algorithm [15]. The results are shown as follows: DOI /IJSSST.a ISSN: x online, rint

6 (a1) (a2) (a3) (b1) (b2) (b3) (c1) (c2) (c3) (d1) (d2) (d3) running time with a better effect. First we exand the exemlars source from camera array instead of one hoto, and reduce the redundancy by counting the similarity of all the exemlars, that means we increase the efficient exemlars; second we add Lab color factor to the riority formula to avoid the confidence value descending quickly and the essential formula just include gray factor, that means we increase the color information of murals; at last we change the riority udate strategy in the filling edge, that decrease the times of udating and ensure the inainting effect. This method is roer to HD multi-lens Tang tomb mural image which cature by camera array. Inainting in each hotos using lenty of camera array source then sliced all the hotos later can ensure the whole Tang tomb mural be efficient inainted and the imrovement of riority method can make the ercetual uniform effect in both color and construction. When we do the exeriments in this research, we noticed that there are something should be imroved in future. It is needed to build an exemlars library including all the disease which make the information missing. So the Tang tomb mural can be inainted by different scale in different missing style. ACKNOWLEDGMENT (e1) (e2) (e3) Figure 10. Tang tomb mural inainting details comarison Picture (a1~a3) is the original mural; icture (b1~b3)) is results of the algorithm mentioned in [13]; icture (c1~c3)) is the results of algorithm mentioned in [14]; icture (d1~d3)) is the results of algorithm mentioned in [15]; icture (e1~e3)) is the results of our algorithm. TABLE II. THE TIME CONSUMING(S) Methods [13] [14] [15] Our a b c The table II shows that our algorithm uses the shortest time consuming to make the damaged region be inainted. The exemlars source exanding and color arameter adding ensure the better effect, and the similar exemlar redundancy and the filling strategy ensure the efficiency. V. CONCLUSION This aer rooses a novel image inainting algorithm, whose aim is to solve the roblem that Tang tomb mural is too large to inaint entirely. We take three strategies to imrove the exemlar-based algorithm, and it makes the inianting rocess could be imlemented and reduce the This aer is suorted by Scientific Research Foundation for Returned Scholars, Ministry of Education of China (K05055), Natural Science Foundation of Shaanxi Province of China (2017JM6435), Secial fund of the Education Deartment of Shaanxi Province (17JK1426), Science and Technology Program of Beilin District (GX1605). REFERENCES [1] Zhou Manxing. The status of the tomb murals in the art history of China [J]. Shaanxi education (high version). 2012(09):31-33 [2] Duan Ping. Tang tomb murals reserved rotection and restoration by the current situation and methods [J]. China's cultural relics scientific research. 2007(04),56-59 [3] S. C. Pei, Y. C. Zeng, C. H. Chang. Virtual restoration of ancient Chinese aintings using color contrast enhancement and lacuna texture synthesis [J]. Image Processing, IEEE Transactions on,2004,13(3): [4] George Paandreou, Petros Maragos. Image Inainting with A wavelet Domain Hideen Markov Tree Model [J].Acoustics, Seech and Signal Processing, 2008.IEEE International Conference.: [5] Purkait, Pulak. Digital restoration of damaged mural images[c].acm International Conference Proceeding Series, 8th Indian Conference on Comuter Vision, Grahics and Image Processing, ICVGIP 2012, December 16, 2012:33-38 [6] Ghorai, M. Kolkata. A robust faint line detection and enhancement algorithm for mural images [C]: Comuter Vision, Pattern Recognition, Image Processing and Grahics (NCVPRIPG), 2013 Fourth National Conference on, 2013:1-4 [7] Kumar, K., Mayank. Digital restoration of deteriorated mural images[c] Digital restoration of deteriorated mural images. ICSIP 2014, 5th International Conference on Signal and Image Processing, 2014: [8] Yuan, Qingshu, Lu, Dongming; Wu, Qi; Liu, Gang. Large area DOI /IJSSST.a ISSN: x online, rint

7 interactive browsing for high resolution digitized Dunhuang murals [J].Lecture Notes in Comuter Science, v 5940 LNCS, 2009: [9] Yang Xiaoing, Wang Shuwen. Based on markov samling of dunhuang fresco restoration [J]. Journal of comuter alications, 2010,(07): [10] Yang Xiaoing, Wang Shuwen. Imroved algorithm based on riority of dunhuang murals comlex damaged area reair [J]. Journal of comuter-aided design and grahics, 2011,23(2): [11] Wang Shuwen Yang Xiao equality. Dunhuang mural image restoration method based on image decomosition [J]. Journal of shandong university,2010(02):11-17 [12] A.Criminisi,P.Perez,K.Toyama.Region filling and Object removal by exemlar-based image inainting[j].ieee Transactions on Image Processing,2004,13(9): [13] Aujol J F, Ladjal S, Masnou S. Exemlar-based inainting from a variational oint of view [J]. SIAM Journal on Mathematical Analysis, 2010, 42(3): [14] Suny A H, Mithila N H. A shadow detection and removal from a single image using lab color sace [J]. IJCSI International Journal of Comuter Science Issues, 2013, 10(4). [15] Sai Hareesh, A. Chandrasekaran, V. A fast and simle gradient function guided filling order rioritization for exemlar-based color image Inainting[C] IEEE International Conference on Image Processing, ICIP. 2010: DOI /IJSSST.a ISSN: x online, rint