Aesthetic Visual Style Assessment on Dunhuang Murals

Transcription

1 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): DOI: 0.007/s y Aesthetic Visual Style Assessment on Dunhuang Murals YANG Bing ( ), XU Duan-qing ( ), TANG Da-wei ( ) YANG Xin 2 ( ), ZHAO Lei ( ) (. College of Computer Science, Zhejiang University, Hangzhou 30027, China; 2. College of Computer Science, Dalian University of Technology, Dalian 6024, Liaoning, China) Shanghai Jiaotong University and Springer-Verlag Berlin Heidelberg 204 Abstract: Dunhuang murals are gems of Chinese traditional art. This paper demonstrates a simple, yet powerful method to automatically identify the aesthetic visual style that lies in Dunhuang murals. Based on the art knowledge on Dunhuang murals, the method explicitly predicts some of possible image attributes that a human might use to understand the aesthetic visual style of a mural. These cues fall into three broad types: composition attributes related to mural layout or configuration; color attributes related to color types depicted; brightness attributes related to bright conditions. We show that a classifier trained on these attributes can provide an efficient way to predict the aesthetic visual style of Dunhuang murals. Key words: Dunhuang murals, aesthetic visual style, feature descriptors CLC number: TP 39.4 Document code: A 0 Introduction Aesthetic visual style assessment is to evaluate which school one painting belongs to. Aesthetic visual style as a whole can be complex, which includes not only the contents of the visual object itself in the paintings, but also the presentation technique or the feeling the painting wants to tell audiences. Automatic assessment on aesthetic visual style of pictures is a very challenging and far from solved problem. The visual objects to be evaluated in this paper are especially Dunhuang murals, aesthetic visual style research on them could help people fully understand the unique Dunhuang murals. Comparing with other Chinese paintings, Dunhuang murals have their own characteristics. From a special viewpoint, Dunhuang murals are the fusion of Eastern and Western culture that affected by the Western-style, Tibetan life and painting style from the Central Plains region. Moreover, these murals painted in different dynasties almost have their aesthetic styles [-2]. Even in the adjacent era, or even in a short dynasty, Dunhuang murals also have their unique characteristics. In these murals, more charts are used to portray religious beliefs, religious activities or Jataka stories, and the incoherence of space is taken to express the changes of Received date: Foundation item: the National Basic Research Program (973) of China (No. 202CB725305) and the National Key Technology R&D Program of China (No. 202BAH03F02) xdq@zju.edu.cn time. In this paper, we aim at getting more information to assess the aesthetic visual style of Dunhuang murals. We try to build a connection between human perception on Dunhuang murals and computational visual features extracted from these murals. Throughout comprehensive studies on literatures [-2] that related to Dunhuang murals and Chinese art history, we have found that the aesthetic visual style of Dunhuang murals contains mainly three components: depiction by line, emotional expression by color, and the light and dark of brightness. Therefore, in our method, composition, color and brightness information are combined together to describe the aesthetic visual style of Dunhuang murals. Related Work For an image, including its points, lines, composition, light, or colors, all components are integrated together to represent the aesthetic visual style, while the aesthetic visual style could be reflected by each component respectively. There are a large number of works related to analysis of the aesthetic visual style. Based on Naïve Bayes classifier, Keren [3] proposed an algorithm under the estimation that for one artist, his works had the same art style. Li and Wang [4] applied 2D multiresolution hidden Markov models combined with multi-levels Debauchies wavelet coefficient feature to identify the authors of Chinese ancient paintings.

2 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): Like aesthetic visual style assessment, aesthetic visual quality assessment also belongs to the category of subjective problem. The two topics both need to transfer human cognition knowledge to mathematics or computer models. Therefore, we give a brief review on aesthetic visual quality assessment. There are many frameworks for aesthetic quality assessment. Ke et al. [5] found three distinguishing factors that lie between professional photos and snapshots (e.g, simplicity, realism and basic photographic technique) anddesignedhighlevelsemantic features to measure the perceptual differences. Datta et al. [6] extracted 56 visual features of photographic images to discriminate between aesthetically pleasing and displeasing images. Moorthy et al. [7] proposed and evaluated a set of low level features that are combined in a hierarchical way in order to construct a computational model of aesthetic appeal of consumer videos. Inthispaper,wetrytoconstructaschemetobridge the human vision with the computer. We just consider global features in our method. This is because, according to Heihrich Wolfflin s comments [8],theaesthetic visual style is in general represented by the whole image. Furthermore, local features may be inaccurate to describe, so these features are ignored in this paper. Visual characteristics such as color, brightness and composition concepts constitute the main body of expert analysis in this paper. 2 Describable Attributes Before designing features to assess a mural s aesthetic visual style, we have studied a lot of research papers related to Dunhuang murals [9-]. Finally, we found that three distinguishing factors play important roles in determining the aesthetic visual style of murals: composition attributes, color attributes and brightness attributes. In general, we propose 20 features which are concatenated together to construct the feature set F ea = {f i, f j i =, 2,, 5, 9, 0,, 20,j =6, 7, 8} that used in aesthetic visual style assessment. Note that, these features are not randomly selected, but are proposed based on knowledge and experiences in Dunhuang murals and human perception. 2. Composition Attributes The composition of Dunhuang murals forms a unique aesthetic visual style that is characterized by dynastic changes. Our compositional attributes address questions related to the arrangement of locations and relationship of objects in a mural where artist organizes individual local components to represent his ideological contents. 2.. Blur Effect There is inevitably certain extent damage to Millennium due to natural or man-made destruction during over one thousand years. The most direct appearance is the blur effect of the mural itself. For one mural, the ambiguity may reduce its own ornamental quality, when considering a series of murals created in different dynasties, and the blur effect could be seen as style changing cues along with the era. To evaluate the blur effect in Dunhuang murals, we apply the method proposed by Brainard et al. [2],which we had found to achieve best results. We model a blurred mural I b as the result of a Gaussian smoothing filter G σ applied to an otherwise sharp mural I s,as I b = G σ I s,wherethesymbol means convolution. The parameter σ of Gaussian filter and the sharp mural I s are both unknown. We assume that the frequency distribution for all sharp murals I s is approximately equal, we then have the parameter σ of Gaussian filter to represent the degree of blurring. We can estimate the maximum frequency of the mural I b by taking its two dimensional Fourier transforms and looking for the highest frequency whose power is greater than some threshold as: F fft = F (I b ), C = {(x, y) F fft (x, y) >θ}, wheref means Fourier transform and θ is set to be 4 in our experiments, x and y denote the coordinates in pixel. If the highest frequency is small, it can be considered to be blurred by a large σ. So the blurring feature is inverse-proportioned to the smoothing parameter σ, which can be measured as { 2 ( W ) f =max w, 2 ( H ) } h W 2 H 2 σ, () where w and h are variables, W and H represent the width and height of the image Standard Deviation In probability and statistics theory, standard deviation is taken to measure the variability or diversity, and to measure the degree of the data deviate from the arithmetic mean value. We use the standard deviation to probe the amount of variation in shading with each region because it may reveal painter-specific shade variations, as following: f 2 = W H (I(x, y) Ī)2, (2) x= y= where I(x, y) isintensityvalueatpixel(x, y), and Ī is the mean value of the mural Rule of Thirds The rule of thirds [3] is a composition rule in visual art such as painting and photography. The rule means that, using imaginary lines to cut the image horizontally and vertically each into three parts, there are nine parts with same area and four intersection points in the image after cutting. Consequently, the intersection points may be found as the important parts of the composition instead of the center point. To some extent,

3 30 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): the rule of thirds can be considered as an approximation to the golden ration. In this paper, we combine the rule of thirds with following edge distribution to represent the composition of images, aiming at detecting the compositional style of Dunhuang murals developed in different dynasties Edge Distribution The stroke style of Dunhuang murals has been constantly changing in the long period. For example, in the Northern Zhou Dynasty murals, the Buddhist and Jataka usually used white as grounding color, and lines were drawn with a smooth outline. When it came to the Tang Dynasty murals, there appeared the Eighteen Descriptions and the Bump Law [2]. From the above discussion, it is clear that the edge distribution could be selected as a feature since we don t consider the brushwork in this paper. Edge distribution To measure the spatial distribution of edges, we calculate the ratio of area that the rule of thirds cuts. One issue is that, artists may not limit in accord with the four intersections point the rule of thirds cuts. To overcome this problem, we choose ( 5 two region thresholds: 2 W 3 4 W, 5 ) 2 H 3 4 H and ( 4 W 3 4 W, ) 4 H 3 4 H. Then the image energy of the chosen region is computed as: f 3 = f 4 = x=5w/2 y=5h/2 x=w/4 y=h/4 I(x, y), (3) I(x, y). (4) Edge distribution 2 For one mural, we calculate the area of the smallest bounding box that encloses a certain ratio of the edge energy. Through trials on the training set, the ratio is selected to be 8% (90% in each direction). So the second feature for edge distribution is to calculate the area ratio of the bounding box over the area of the whole image, i.e., f 5 = W b H b, (5) where W b and H b are the width and height of the bounding box, respectively Wavelet-Based Texture The use of texture is a composition attribute in paintings. In this paper we use Daubechies wavelet transform to measure the spatial smoothness in the images. We perform a wavelet transform on all three color bands I h, I s, I v, the corresponding feature are formed as f 6, f 7 and f 8,whereI h, I s and I v mean the h, s and v vectors in HSV color space respectively. 2.2 Color Attributes As long as the substitution of dynasties and the change of social environment, the color of Dunhuang murals alters continuously, color murals were increasingly more abundant, color were represented lively, the figures were full life-like, and the strokes were delicate Spatial Variations of Color This feature tries to identify the differences in color palette used by murals in different dynasties. To measure the spatial variation of color, we apply the following method which is similar to Florin s method [4]. For an input mural, its R, G and B channels are normalized by division by mural intensity. At each pixel, we could determine the orientation of the plane that best fits a 5 5 neighborhood centered on the pixel of interest in the R, G and B domains respectively. Thus, at each pixel, we obtains three normals: n R, n G and n B. The average of the areas constructing facets of the pyramid determined by these normals is taken as a measurement of the spatial variation of color around the pixel. f 9 = S(I), (6) where S( ) is the function of the above processes, and I is the vector of mural image. It is intuition that for murals created in different dynasties, the color palate is distinguishable. Therefore, the spatial variation of color is different from each other Average Value of H ue and S HSV color model template corresponds to the color palette of the artists. The artists obtain different colors from a solid color by changing the concentration and depth of the color relative to the solid color. So, using HSV color model could clearly represent the artists inner idea about color. We choose the average value of H ue and S channels as two features: f 0 = f = W x= y= W x= y= H I h (x, y), (7) H I s (x, y), (8) where I h (x, y) andi s (x, y) arethevaluesatpixel(x, y) respectively The Proportion of Warm Color and Cold Color Artists in the Tang Dynasty were good at using of the harmonic of contrasting colors, mainly through the contrast that a small amount of red soil, azurite and malachite green colors were included to make the color of murals more intensity and quite dynamic. Thus, we select the ratio of warm color to cold color as one feature to depict characteristics of dynasty styles. In RGB color space, there s no explicit definition that whether each color in the natural world belongs to the

4 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): warm color or cold color. To solve this problem, in this paper, we define a distance that evaluates the difference between the color of one pixel and the color of red, yellow, blue and green. If the minimum distance arises in the color of red or yellow case, then the color of this pixel is treated as the warm color, vice versa: f 2 = I(x, y) c, (9) where c means the value of red, yellow, blue, or green color. Throughout all pixels of the mural, we calculate the number of warm color pixels and cold color ones respectively, and then the ratio of warm color pixels to cold color ones is chosen as one measurement of aesthetic visual style. 2.3 Brightness Attributes It is intuitively rational that brightness affects people s impression on a mural. Artists use a series of techniques to represent bright condition of a scene Average Brightness Value The average brightness value is computed according to: W H f 3 = I v (x, y), (0) x= y= where I v (x, y) is the intensity value at pixel (x, y) ofv vector in HSV colour space Uniformity of the Luminance To capture the brightness, we measure the uniformity of illumination of the mural. For one input image I, I is the vector of mural image, we do the followings step by step. () Get the log of I the logged image is I. (2) Undertake fast Fourier transformation (FFT) of I I fft. (3) Generate a binary image I bin, the same size as the FFT matrix, which is 0 all over, except for identical 8pixel 8 pixel squares in each corner. (4) Multiply I bin by the FFT image I fft the masked spectral image I masked. (5) Apply the inverse Fourier transform to I masked I ifft ; (6) Transfer exp(i ifft ) I illu. f 4 = I illu (I), () where after a series of processes I illu, we could attain a value I illu and assign it to f 4. Finally, we get the uniformity of luminance of the mural according to Eq. () Arithmetic Average Brightness The arithmetic average brightness is calculated as follows: f 5 = W x= y= H B ri (x, y), (2) where B ri (x, y) is the value at pixel (x, y) ofb, B = (I R + I G + I B )/3, the I R, I G, I B are the R, G, B channel vectors of the mural image, and I R (x, y), I G (x, y), I B (x, y) arethevaluesatpixel(x, y) ofi R, I G and I B respectively Brightness Contrast Without brightness contrast, it would be difficult to discriminate the exact place. We add the brightness contrast feature as follows. First, we compute the brightness difference of one pixel in image as: f 6 = B ri (x, y) B ri (x,y ), (3) j N(j) where B ri (x, y) B ri (x,y ) represents the absolute difference between the B ri (x, y) for pixel (x, y) and the B ri (x,y ) for its neighboring pixel (x,y ), denote (x,y )asj, andn(j) is the neighborhood around pixel j Logarithmic Average Brightness To some extent, the logarithmic average brightness could describe the brightness of the mural. It is calculated as: f 7 = 255 W H x= y= ( Bri (x, y) ) lg + ε, (4) 255 where ε is a small positive constant to prevent from computinglg Lab Color Descriptor A Lab color space [4] is a color-opponent space with dimension L for lightness, and dimensions a and b for the color-dimensions, based on nonlinearly compressed commission international de I eclairage (CIE) XY Z color space coordinates. The nonlinear relations for L, a, b are intended to mimic the nonlinear response of the eye. The Lab color descriptor is computed as: f 8 = f 9,20 = 80() W 200() x= y= H I L (x, y), (5) W x= y= Q (a, b), H I Q (x, y)+80, (6) where I L (x, y), I Q (x, y) arethevaluesatpixel(x, y) of vectors I L and I Q, I L is the vector of L channel, I Q is vector of a, b channel respectively. In summary, 20 features are extracted from a mural to represent its aesthetic visual style globally. These features are based on the analysis of Dunhuang morals, including some art rules, and they were evaluated through experiments in next section.

5 32 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): Experimental Results Aesthetic visual style assessment is a subjective problem. Therefore, how to construct a mathematics and computer model to describe the aesthetic style visual of one mural is a non-trivial task. In this section, we will detail our experimental results. 3. Dataset To evaluate the classification performance, we chosen three dynasties Dunhuang murals as typical aesthetic visual styles to be evaluated: Northern Zhou, Tang and Yuan. The reason why we selected these dynasties is that, in Dunhuang mural history, art researchers always divided murals into three categories: the early, middle and final types based on dynastic style of mural. Consequently, we took these three dynasties murals as the representative of each period. Although the number of Dunhuang murals is large, digital murals are not easy to obtain, thus the actual samples in our experiment are not so big. We have constructed a dataset consisting of 450 murals with 50 murals for each dynasty. To objectively and correctly measure our method, the content of murals included: joss, plant, mountain and water, building and others. We adopted the leave-n-out cross validation method for experiment. We repeated the algorithm for 0 times: for each class, N (N = 30, 40, or 50) murals were randomly selected as train murals and the rest are test murals. Each time we perform an independent experiment for training and testing. 3.2 Classification Performance In this section, we will show our experimental results accordingto classificationperformance. Table demonstrates the classification results and we could find that for the classification of Tang from Northern Zhou, our method works very well and the classification accuracy is high. But for the classification of Tang from Yuan and Northern Zhou from Yuan, the accuracy is a little lower. The reason for this can be explained as follows. TheNorthernZhoudynastymurals emphasized the potential of air movement toward the formation of flow lines which shows a kind of beautiful emotion, while in the Tang dynasty, colors were lively used and the brushwork line was thick. When it came to the Yuan dynasty, it almost reached the end of the Dunhuang mural history. Artists tended to use uniform, energetic and flexible curves, and don t give re-color. Therefore the murals in this period looked like somewhat solid. In a word, in the Northern Zhou period the rhythm of curve was stressed, murals belonged to freehand images rather than realistic ones. Murals in Tang showed a rich, soft, vibrant artistic charm. However, in the Yuan dynasty, the characteristics style of former two periods had been absorbed. The style in the Yuan dynasty had both the Tang dynasty s round, i.e., a sense of flexibility, and the characteristics of the Northern Zhou dynasty. So, compared with the difference between Tang and Northern, the differences between Tang and Yuan or Northern Zhou and Yuan are not so clear. As seen in Table, this results in a little lower classification accuracy in our experiment. Note that, the number in the last line of Table shows multi-label classification accuracy of three typical dynasties Dunhuang murals. In order to comprehensively evaluate the benefits of our proposed method, we also test the method [5] on our established dataset. Table 2 shows the classification results when using that method. It is easy to conclude that our method is superior to method proposed in Ref. [4] on Dunhuang murals dataset. Table Dynasties Classification performances for Dunhuang murals of different dynasties Accuracy N =30 N =40 N =50 Tang, Northern Zhou ± ± ± Tang, Yuan ± ± ± Northern Zhou, Yuan ± ± ± Tang, Northern Zhou, Yuan ± ± ± Table 2 Classification performances for Dunhuang murals using the method in Ref. [5] Dynasties Accuracy N= 30 N= 40 N= 50 Tang, Northern Zhou ± ± ± Tang, Yuan ± ± ± Northern Zhou, Yuan ± ± ± Tang, Northern Zhou, Yuan ± ± ±0.08 4

6 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): Table 3 lists the classification performances on the multi-label case (i.e., Tang, Northern Zhou and Yuan) based on different categories of features. From Table 2, the experimental results show that, among the three types attributes, e.g., composition attributes, color attributes and brightness attributes, the composition features perform better than others, and color and brightness features seem as competitors with each other. Furthermore, the improvement by combining all features proves that our proposed features are complementary, as shown in the last line of Table 2. We also tested the performance of each individual feature by using support vector machine (SVM). The most five distinctive features are listed below: {f 2,f 9,f 5,f 8,f 20 }. These results help us understand more about which feature is more powerful to describe aesthetic visual style of Dunhuang murals. We try to explain these results in the following ways based on some art knowledge. () f 2, standard deviation: a measurement of the dispersion of a set of data from its mean. If an image is supposed to be uniform throughout, the standard deviation should be small that indicates that the pixel intensities do not stray very far from the mean. The murals in different dynasties may relate to different standard deviation values, as shown in Fig., the vertical coordinate is the standard deviation value, and horizontal coordinate corresponds to index of test image. (2) f 9, spatial variation of color: this feature tries to describe the color palette of Dunhuang murals. It is intuitively rational that the variation of the whole image color affects people s impression on a painting. (3) f 5, arithmetic average brightness: the most popular brightness editing algorithm is based on arithmetic mean model. This brightness measure has the biggest difference with luminance. Figure 2 depicts the histograms of the arithmetic average brightness value for three dynasties, and the difference among three dynasties is obvious, the vertical coordinate is the arithmetic average brightness value, and horizontal coordinate corresponds to index of test image. (4) f 8,f 20,L, b average values in Lab color space: in Table 3 Classification performances on the multi-label case using different categories of features Dynasties Accuracy N= 30 N= 40 N= 50 Composition attributes ± ± ± Color attributes ± ± ± Bright attributes ± ± ± All attributes ± ± ± Fig. The histograms of the standard deviation value for three dynasties Fig. 2 The histograms of the arithmetic average brightness value for three dynasties

7 34 J. Shanghai Jiaotong Univ. (Sci.), 204, 9(): Lab color space, its b component aspires to perceptual uniformity, and its L component closely matches human perception of lightness. As mentioned above, we have pointed that, the relations for L, a, b are intended to mimic the response of the eye and also used for identify the aesthetic visual style among murals of different dynasties. However, there are some misclassified murals as shown in Fig. 3. From the Fig. 3, we argue that, some murals are so atypical in their aesthetic visual style that they are not easy to be distinguished even by the human eye. Thus, it is a non-trivial work for computer to assess the aesthetic visual style of these murals. Fig. 3 4 Conclusion The misclassified murals in the experiment The aesthetic visual style assessment is a problem of subjective cognition. As one of important art works, Dunhuang murals have their sole characteristics; especially the aesthetic visual style develops with the dynasties going by. To solve this problem, we extract a group of perception-related global feature to construct a framework to describe the aesthetic visual style of Dunhuang murals. With the analysis of on Dunhuang murals, the aesthetic visual style of these murals is represented according to three components: composition attributes, color attributes, and brightness attributes. These attributes are then all integrated together to evaluate which dynastic aesthetic visual style the mural belongs to. Finally, we use support vector machine as the classifier. The experimental results show that, the proposed method produces good classification accuracy when using the extracted features. The performance of individual extracted feature is also evaluated, and this will help us learn more about intrinsic art knowledge of Dunhuang murals. There is great room for future work in several directions. First, the same aesthetic visual styles should have somewhat similarity. We could try to find the similarity measurement between same styles. Second, more art-related analysis should be explored by the future research. Third, search a more efficient classification method to improve accuracy of our method. Finally, the best paper [5] of 20 International Conference on Computer Vision (ICCV 20) inspires us to use the idea of relative attributes to recognize different aesthetic visual styles, and this work is in process. References [] Guo Xiao-li. The enlightenment of Dunhuang fresco color research in Sui-Tang Dynasties to modern color design [D]. Beijing, China: Beijing Institute of Fashion Technology, 200 (in Chinese). [2] Huang Jun. The studies for Dunhuang fresco character style in the Tang Dynasty [D]. Beijing, China: China Academy of Art, 2007 (in Chinese). [3] Keren D. Recognizing image style and activities in video using local features and naive Bayes [J]. Pattern Recognition Letters, 2003, 24(6): [4] Li J, Wang J Z. Studying digital imagery of ancient paintings by mixtures of stochastic models [J]. IEEE Transactions on Image Processing, 2004, 3(3): [5] Ke Y, Tang X, Jing F. The design of high-level features for photo quality assessment [C]//2006 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 06). NewYork,USA:IEEE Computer Society, 2006: [6] Datta R, Joshi D, Li J, et al. Studying aesthetics in photographic images using a computational approach [C]//The 9th European Conference on Computer Vision (ECCV 06). Graz, Austria: Springer, 2006: [7] Moorthy A K, Obrador P, Oliver N. Towards computational models of visual aesthetic appeal of consumer videos [C]//The th European Conference on Computer Vision (ECCV 0). Crete, Greece: Springer, 200: -4. [8] Heihrich W. Principles of art history [M]. Pan Yaochang trans. Shenyang, China: Liaoning People s Publishing House, 987. [9] Liu Xing. Rendering 3D mountain and rock models in Chinese painting style [D]. Shanghai, China: School of Software, Shanghai Jiao Tong University, 2007 (in Chinese). [0] Wang Xiang-hai, Qin Xiao-bin, Xin Ling. Advances in non-photorealistic rendering [J]. Computer Science, 200, 37(9): (in Chinese). [] Qian Xiao-yan. Study on non-photorealistic rendering theory and method of artistic style image [D]. Nanjing, China: Nanjing University of Science and Technology, 2007 (in Chinese). [2] Brainard D H. Color appearance and color difference specification [M]. Washington D C, USA: Optical Society of America, 2003: [3] Dhar S, Ordonez V, Berg T L. High level describable attributes for predicting aesthetic and interestingness [C]//20 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR ). Colorado Springs, USA: IEEE Computer Society, 20: [4] Gehler P, Nowozin S. On feature combination for multiclass object classification [C]//2009 IEEE International Conference on Computer Vision (ICCV 09). Kyoto, Japan: IEEE, 2009: [5] Parikh D, Grauman K. Relative attributes [C]//20 IEEE International Conference on Computer Vision (ICCV ). Barcelona, Spain: IEEE, 20: