Chinese civilization has accumulated

Color Restoration and Image Retrieval for Dunhuang Fresco Preservation Xiangyang Li, Dongming Lu, and Yunhe Pan Zhejiang University, China Chinese civilization has accumulated many heritage sites over thousands of years. Dunhuang Mogao Cave, located in Gansu province, is typical. Well preserved to date, the caves contain more than 555 grottoes, 2,000 painted statues, and 5,000 square meters of wall paintings. However, they face serious damage from many different factors. Scientists and artists have worked for a long time on how to preserve and restore the murals s, but still have not found satisfactory methods. Discovering how to use advanced computer techniques to preserve, restore, and reuse Dunhuang fresco art information is a very interesting and complex task. In our project, we integrate many techniques, such as image processing, informaton retrieval, knowledge representation, and intelligence reasoning, to deal with these problems. Restoration of the frescoes is the basic focus of our project. Because of human activity, most of the frescoes fade day by day, with some seriously damaged. Here we propose a new, intelligent approach for restoration. Each restored fresco will be preserved in the fresco database along with the original. For many of the Dunhuang frescoes, the key to preservation is to find an effective method for organizing, storing, managing, and retrieving the fresco data. We adopt content- and semantics-based image retrieval methods to solve these problems. At the same time, our fresco database will support Dunhuang pattern creation and virtual cave travel, reusing the fresco art information from the database. In this article, we mainly focus on the restoration and retrieval techniques we use for fresco preservation. Color restoration techniques Color fading is common in the Dunhuang caves almost every has faded to some extent. Color shifts result when a pigment s original bright, intense hue becomes gloomy and vague. The causes that lead to fresco spoilage divide into internal and external. The internal factor causing change is that the pigments contain lead. The external factors are very complicated. They involve physical, chemical, and biological reasons, such as sunshine, oxidization, temperature, humidity, mold, bacteria, and so on. Sunshine, temperature, and humidity are the most crucial factors. While a cave s location and size determine the temperature and humidity range inside, frescoes closer to the sunshine are more seriously spoiled than those further back. The extent of changes in a lead pigment differs under different external conditions. Based on this analysis of mural-spoiling factors and pigment -changing patterns, our main process to restore a spoiled fresco follows: Acquire a fresco s structure by clustering and separate the fresco image into several layers so that each represents one pure pigment. Determine the original pigment used for each layer using pigment domain knowledge. Restore every layer of the mural image according to the fade and change rules over time of pigments under different light, temperature, and moisture. Combine the restored levels into one image so as to obtain the -restored mural image. Figure 1 describes the whole process. Two important types of domain knowledge benefit fresco restoration: knowledge from Dunhuang artists and restoration knowledge from preservation experts. Color knowledge underlies the Dunhuang artists design, specifying the use of pigments at different parts of a fresco. Different frescoes have different sets. We set up some typical templates to process the complex relationship between s and pigments. Environmental factors play an important role in the change of the pigment component. Based

mural Restored mural Frame-based reasoning Figure 2. Frame- and rule-based reasoning. Selecting Region restoration Region merge Operating Matching Attribute descriptions Rule-based reasoning Color template layer N Analysis of fading factors Color restoration layers layer N Color restoring domain knowledge Figure 1. The system architecture behind mural restoration. on experimental observation and measurement, Dunhuang preservers have drawn some conclusions about changes in the pigment components under the influence of external conditions. We use a hybrid structure of frames and rules to organize and represent the preserver s knowledge. Frame-based reasoning serves as the main reasoning mechanism in determining. Our method compares descriptions of a fresco s spoiled condition with the frames in the knowledge base and retrieves the frame that best matches the condition. When the process requires the attributes in slots in a frame, the control turns to rule-based reasoning. Here, it uses template knowlege to judge the restored (see Figure 2). Visually, we assume that regions, which have similar properties, belong to the same class, even if spatially disconnected. Here, we adopted the image -clustering method to divide s into a few dominant s. The pixels with similar to a dominant belong to the same layer. Edge detection and region growing approaches combined find large, well-defined segments for coarse segmentation. Segments can grow or expand based on several criteria for fine segmentation. The criteria can be defined from local, regional, or global considerations according to a local analysis of neighborhood pixels belonging to the region under study. So we can restore the of a given region on a layer using frame knowledge. The style and painting skill characteristic of Dunhuang are rich and ful. Techniques include using demi-tint painting to stress the stereo effect on human faces, hybrid pigments to depict human skin, stratum layers to show the rough and uneven textures in an entity s surface, integration of line drawing and ation, and so on. We use design knowledge to restore the painting effects. Semantics- and content-based retrieval To manage the huge amounts of fresco data, our system s key functionality involves efficiently indexing and retrieving images from the Dunhuang fresco database. Currently, the content-based image indexing and retrieval seems a promising alternative to relying on text retrieval. However, painters created the Dunhuang frescoes during different dynasties. The frescoes differ from more general images in that they contain semantic information about the intention behind their creation and about Buddhist culture. We combine the visual image content with the semantic information to retrieve the Dunhuang art information. Style analysis of the Dunhuang frescoes We can describe the style of the Dunhuang frescoes from four aspects: theme, layout, pattern elements, and ing. Nearly every piece of the Dunhuang frescoes reflects a Buddhist art theme according to the design background specific to the time. The fresco theme determines the content, April June 2000 1070-986X/00/$10.00 2000 IEEE 39

and the fresco content determines the design style. The fresco layout sets the framework, which depends on the theme of the fresco design. Therefore the different contents of Dunhuang frescoes have many layout forms. The pattern elements are the basic units of a Dunhuang Figure 3. A group of ellipses can approach the contour fresco. Each kind of of the letter F. Dunhuang element has some unique charactristics. In a sense, Dunhuang elements represent a fresco style; for example, Apsarase fresco describes the typical representation of Dunhuang imagery. We can categorize all these elements into classes, and each class divides into several subclasses. Over time, Dunhuang fresco s evolved into two appearances. The first displays the ing style used when ancient artists designed the frescoes, reflected in those frescoes whose upper layers have recently flaked off. The styles share a common property hybrid s, which result from the mixing of the dominant s from each dynasty. Since the s chosen depend on the specific object painted, most objects possess several hybrid s. The second appearance is the seen by people today, much of it faded. (a) (b) Retrieval methods Based on this analysis, we can organize and retrieve Dunhuang frescoes according to Dunhuang fresco style. In our system, a semantic network holding features of objects and relationships between objects represents fresco content. The network nodes represent the objects, and the relationships between objects are represented by edges between such nodes. Both nodes and edges have labels of attributes corresponding to the features of objects and relationships, respectively. The similarity of two images equals the weighted average of similarity of their nodes and edges. Here, nodes include semantics-based retrieval, -based retrieval, and shape-based pattern element retrieval. Edges represent the fresco layout relationship. Semantics-based fresco retrieval. In our system, users can input semantic information to find (c) Figure 4. The restoration of skin, beginning with a faded fresco (a), the region extraction on the skin layer (b), and the restoration based on the system s pigment fading knowledge (c). 40 1070-986X/00/$10.00 2000 IEEE

the fresco they want. Semantic information includes the cave identifier, the fresco s location in a cave, the fresco theme, the dynasty, keywords about the story told in the fresco, and so on. For semantic information retrieval, we use traditional database matching technology. Color-based fresco retrieval. We provide two methods for -based image retrieval. For the first one we use layers to represent information based on clustering. For the layer similarity between two images, we proposed a new matching method called Color Layer Matching (CLM). 1 For the second method we use the composition information to retrieve images from the artist s point of view. A matching method called Color Composition Matching (CCM) measures the similarity of composition of two images. 1 (a) Element shape-based fresco retrieval. The Dunhuang frescoes contain all sorts of shapes, which reflect the pattern creation style current during the different dynasties and characterize the frescoes. We first normalize a shape to keep its representation invariant to its transformation during panning, size changes, orientation, and symmetric reflection. We proposed a new idea that uses a group of ellipses to approach the closed contour of a shape (see Figure 3). The ellipse features of a shape serve to measure the similarity of two shapes. 1 Object layout-based fresco retrieval. Layoutbased image retrieval needs an abstract representation of direction as well as topology between image objects. We adopt 2D-PIR (Project Interval Representation), 2 defined as a triple <R, X, Y>. Here, R represents a topological relationship, X the interval relationship along the x-axis, and Y the interval relationship along the y-axis. The similarity between <R 1, X 1, Y 1 > and <R 2, X 2, Y 2 > depends on the degree of similarity between R 1 and R 2, X 1 and X 2, and Y 1 and Y 2. (b) Experimental results We developed a prototype of a restoration and fresco database system using VC++ 6.0 on WinNT 4.0. We proved its effectiveness by experimenting with typical faded murals. Figure 4 demonstrates some of our experiments. We populated the fresco database with 100 images of different types. Figure 5 shows the retrieval results. For multifeature retrieval, relevance feedback adjusts the weight value. The user carries out the (c) Figure 5.Content-based retrieval results: -based retrieval (a), shape-based retrieval (b), and spatial relationship retrieval (c). April June 2000 1070-986X/00/$10.00 2000 IEEE 41

resulting evaluation. Based on our own experiments, we have found our method very promising for retrieval and restoration of the Dunhuang frescoes. MM Reference 1. X. Li, The Study of Content Based Image Retrieval and Image Database Modeling, PhD thesis, Dept. of Computer Science and Engineering, Zhejiang University, Hangzhou, China, 1999. 2. M. Nabil et al., Picture Similarity Retrieval Using the 2D Project Interval Representation, IEEE Trans. on Knowledge and Data Engineering, Vol. 8, No. 4, Aug. 1996, pp. 533-539. Xiangyang Li is at Lucent Technologies, Bell-Labs Innovation, in Beijing, China. His research interests include content-based image retrieval, multimedia databases, multimedia collaborative computing, and multimedia communication. Li received the BS degree and MS degree in the Department of Mining System Engineering from China University of Mining Technology in 1993 and 1996, respectively. He received his PhD degree in the Department of Computer Science and Engineering from Zhejiang University, China in 1999. Dongming Lu is an assistant professor in the Institute of Artificial Intelligence, Zhejiang University, China. His research interests include multimedia databases, artificial intelligence, and computer network applications. Lu received his BS degree, MS degree, and PhD degree in the Department of Computer Science and Engineering at Zhejiang University, China in 1989, 1993, and 1996, respectively. Yunhe Pan is a full professor in the Department of Computer Science and Engineering at Zhejiang University, China. He has been President of Zhejiang University since 1994 and is an Academician of China Engineering Academy. His research interests include artificial intelligence, cognitive science, imagery thinking, computer-aided design and computer graphics, computer art, and geographical information systems. Pan has published in the fields of artificial intelligence, cognitive science, intelligent CAD, and computer arts, and has written three books. He is an associate editor of Journal of Chinese Science, Journal of Computers, and Journal of Electronics. Readers may contact Li at Lucent Technologies, Bell-Labs Innovations, 3/F Aero Space Great Wall Building, No. 30, Hai Dian Nan Lu, Beijing, China, 100080, e-mail xli17@lucent.com. Three-Dimensional Modeling for Virtual Relic Restoration Ichiroh Kanaya and Qian Chen Wakayama University, Japan Yuko Kanemoto and Kunihiro Chihara Nara Institute of Science and Technology, Japan Relics change in quality when excavated from ruins and exposed to air and/or sunlight. This is one reason why we developed a new computerbased method to record and preserve archaeological properties. Here we propose a new method of 3D modeling of a relic shard that achieves sufficient accuracy and efficiency. With this approach, a single relic shard is scanned twice with a laser range finder, once for the front face and another for the back face. After that, the two shape data of the front and the back face are integrated using physical constraints of the shard shape. 42 1070-986X/00/$10.00 2000 IEEE