Intelligent Interactive Applications for Museum Visits

Size: px

Start display at page:

Download "Intelligent Interactive Applications for Museum Visits"

Hollie Austin
5 years ago
Views:

1 Intelligent Interactive Applications for Museum Visits Kemal Egemen Ozden 1, Devrim Unay 2, Huseyin Inan 1, Bahtiyar Kaba 1, Ovgu Ozturk Ergun 1 1 Computer Engineering, Bahçeşehir University, Istanbul, Turkey 2 Electrical & Electronics Engineering, Bahçeşehir University, Istanbul, Turkey {kemalegemen.ozden, devrim.unay, ovgu.ozturk}@eng.bahcesehir.edu.tr {huseyin.inan, bahtiyar.kaba}@stu.bahcesehir.edu.tr Abstract. This paper presents user interaction modules of a recent digital cultural heritage conservation and exploration project concerning two important museums: Istanbul Archeology Museum and the Museum of Dolmabahçe Palace Collections in Istanbul. The project aims at enriching the visitor experience through modern digital technologies. Main modules include 3D scanning of the artifacts, information screen and mobile interaction with Augmented Reality.3D scanning task includes capturing the accurate 3D geometry of the target artifacts. Information screen enables users to browse the museum catalogue with multimedia content which results in easier and enjoyable learning while keeping a stronger visitor engagement. Mobile module relies on visual computing applied to camera images of a mobile device to locate the artifacts in the scene and lay artificial tags/markers on the screen. The result is a more natural and flexible method to interact with historical artifacts. Keywords: 3D scanning, mobile and touch-screen applications, object recognition, augmented reality, interactive museum visits, information retrieval 1 Introduction The need for preservation and flexible exploration of historical artifacts generated increasing interest in using digital technologies in the cultural heritage context. Technological advances typically result in a fresh novel viewing angle to the problem and eventually new applications arise. Especially recent advances in computer and interaction technologies, namely touch-screens, smart mobile devices, realistic 3D rendering with consumer level computers and easy multimedia content generation and consumption tools, create new perspectives on the issue. Turkey, and its former capital city Istanbul, harbor a great potential for such applications. Being on the crossroads of eastern and western civilizations, the country has a colossal stock of historical artifacts which is both a challenge and an opportunity for digital conservation society. In that regards, our project explored modern ways of interaction with two important museums of Istanbul. Istanbul Archeology Museum is the oldest museum in Istanbul with almost 1 million artifacts. Dolmabahçe Palace Collections is relatively smaller and is dedicated to mostly palace

life of late Ottoman period. The project aims better user engagement and experience by utilizing modern digital technologies in those two traditional museums of Istanbul. Fig. 1.

2 life of late Ottoman period. The project aims better user engagement and experience by utilizing modern digital technologies in those two traditional museums of Istanbul. Fig. 1. Project Organization and Overview of the Project Overview of the Project: The project is a sub-project of a bigger work conducted by a consortium of Bahcesehir Univ., Tokyo Univ. and aforementioned museums that is funded by Istanbul Development Agency. Functionally similar but content-wise different work packages have been executed parallel for both museums. 3D content generation has been achieved by utilizing 3D scanners. Touch-screen application, Information Screen is a PC powered large touch-screen interface that is typically located in the entry of target showroom or building. This application enables an easy browsing experience of 3D models and other text/multimedia material related to chosen artefacts. Rather than being a complete catalogue it more serves as an interest booster. Mobile Augmented Reality browser application is targeted towards easy and natural interaction with the artefacts. By using a mobile device s camera and local visual computing techniques, objects in the scene are located which help in laying artificial tags and markers on mobile device s screen over real images from the camera. Similar Projects in the Literature: With recent advances in digital visualization and information technologies, the opportunities to nurture cultural data with digital technologies for preservation of cultural heritage, digital visualization of artifacts and enhanced museum experiences have increased. There have been several works[5] in many countries all around the world on these topics. However, the work presented in this paper is a pioneering project in Istanbul including 3D reconstruction, tabletsupported interaction and touch-screen technologies. One of the important examples of interactive digital museum projects is a collaborative project between Musee du Louvre in France and DNP Museum Lab in Japan [1]. It is a comprehensive project including tablets, touch-screen devices and virtual reality rooms, which were first installed in Tokyo in In 2010, a selection of the displays has been moved to Musee du Louvre in Paris. Another significant project is a world-wide renowned Bayon Digital Archieve Project and it is connected project Virtual Asukakyo [2]. The group in Ikeuchi Laboratory started the project in 2003 to reconstruct Angkor Thom in Cambodia (160m X 140m X 45m) by using various scanners and sensors. Following this project, the same group reconstructed Asuka Village in Japan and developed Mixed Reality

tools to visualize lost buildings of Asukakyo with computer graphics and synthesize them with the real landscape of Asuka Village [3]. 3D Murale project has been supported by IST program of EU [4].

archaeological ruins in Sagalassos, Turkey. There are three major tools in the project.

2 3D Reconstruction of Museum Artifacts Reconstruction of 3D digital models of museum artifacts is extremely important in order to represent artifacts in digital world and develop information

First, the scanner should not give harm to the material.

3 tools to visualize lost buildings of Asukakyo with computer graphics and synthesize them with the real landscape of Asuka Village [3]. 3D Murale project has been supported by IST program of EU [4]. An international team of multimedia content creators and researchers have been led by Brunel University to develop 3D multimedia tools for measurement, reconstruction and visualization of archaeological ruins in Sagalassos, Turkey. There are three major tools in the project. Strat Tool is developed and used for processing 3D data in one environment, Museum Visualization Tool is used to display data in 3D and WebCAME is produced to visualize the content via Internet. 2 3D Reconstruction of Museum Artifacts Reconstruction of 3D digital models of museum artifacts is extremely important in order to represent artifacts in digital world and develop information technologies for various applications. Most of the time laser scanners are utilized to have highresolution 3D models. However, there are some critical concerns while scanning artifacts. First, the scanner should not give harm to the material. Next, the artifacts have a wide range of sizes, from tiny coin size to very large sculpture size; the scanner should be moved flexibly in order to capture hidden details from right angles. Third, both texture and geometric shape should be captured with high level of detail. Artifact King Mari Bowl Tablet Duck Lion Place Mes * Mes Mes Mes Ana Date 1894 B.C. 1200B.C. 700 B.C. Size 165cm Ht Small Small 41x24cm 140x192x46cm Diorite Marble Basalt Basalt Texture S S S S S 3D model Artifact Vase Plate Table Place DP * DP DP Date 1839 A.C A.C. Size Medium Small Big Texture M S M 3D model *Mes stands for Mesopotamia, Ana stands for Anatolia *DP stands for Dolmabahce Palace, S: single color, M: multi-color texture Fig 2. Example 3D models of Scanned Museum Artifacts In this project, Konica VIVID 910 scanner is used to scan museum artifacts. A group of artifacts with different properties (size, date, material, texture, complexity) are chosen from two museums as given in Fig. 2. The most challenging part is scanning

4 items in their exhibition places. Removing items from the museum or from the stands is not allowed due to security reasons or big sizes. This brings challenges such as: non-uniform lighting conditions, hidden parts between items and exhibition walls, varying reflectance coefficients due to varying material aging. To overcome these issues, extra processing is required on scanned raw data by means of customized or commonly used 3D processing software. Here, we see exciting research problems to tackle, in order to develop methods to achieve better automatic scanning processes. 3 Interactive Information Screen for Museum Artifacts Museums are increasingly augmented with digital technology for better engagement of the visitors via improved interaction with the artifacts. Here we present touchscreen based systems developed for interactive presentation of artifacts exhibited in the Istanbul Archeology Museums and the Museum of Dolmabahçe Palace Collections. The system is composed of a 40-inch touch-screen (Vestel AIO Touch), and an in-house developed C# application (with an MS Access Database storing textual, visual and 3D scan data of the artifacts) running on a desktop PC (Intel Pentium i7 Processor 3.40GHz, 8GB Ram, NVIDIA GeForce GT 640 graphic card). The application supports both Turkish and English languages. The systems are installed to both museums, and are up and running since end The entry screens of both systems can be seen in Fig. 3. Fig. 3. Entry screens of the systems installed to the Museum of the Ancient Orient (left) and the Museum of Dolmabahçe Palace Collections. Istanbul Archeology Museums Museum of the Ancient Orient: An appealing application should present the museum artifacts in a well planned and designed scenario to better attract and retain the attention of the visitors. Accordingly, the applications we developed for both museums present the artifacts in a scenario designed in collaboration with the archeologists. Accordingly, a subset of the artefacts from the collection of the Museum of the Ancient Orient are selected and grouped based on their geographical origin and the presence of a form of cuneiform script. Digital images of each selected artifact are then acquired and post-processed via a professional system (examples can be seen in Fig. 4). In our application the user may select the group(s) of artifacts for display. If the selected group is of Anatolia, Mesopotamia or Egypt, the artifacts therein are displayed on an animated map (Fig. 5). Otherwise information about the selected cuneiform document is displayed directly (Fig. 6). In addition to textual descriptions and high resolution images, as seen in Fig. 6 our application allows real time

interaction and visualization of 3D laser scans for some of the artifacts (indicated by a

Exemplary artifact images from the Museum of the Ancient Orient.

group(s) of artifacts for display. Fig. 6.

g. The Code of Hammurabi) on the left, and allowing real time visualization and

Museum of Dolmabahçe Palace Collections: Following discussions with the archeologists, we

scenario based on Sultan Abdül Hamid II, and selected four artifacts: Banquet Table,

5 interaction and visualization of 3D laser scans for some of the artifacts (indicated by a 3D sign on the animated map display - Fig. 5). Fig. 4. Exemplary artifact images from the Museum of the Ancient Orient. Left-to-right: Treaty of Kadesh (Anatolia), Statue of Lugaldalu (Mesopotamia), Book of the Dead (Egypt). Fig. 5. Screenshot of the application displaying all the selected artifacts of the Museum of the Ancient Orient on an animated map. The bottom left pane allows the user to select the language (Turkish or English) and the group(s) of artifacts for display. Fig. 6. Screenshots of the application displaying information on a selected cuneiform document (e.g. The Code of Hammurabi) on the left, and allowing real time visualization and interaction of 3D scan of a selected artifact (e.g. Oblation Bowl) on the right. Museum of Dolmabahçe Palace Collections: Following discussions with the archeologists, we agreed upon presenting the artifacts of the Museum of Dolmabahçe Palace Collections in a scenario based on Sultan Abdül Hamid II, and selected four artifacts: Banquet Table, Drawer, Dressing Table, and Vases produced at the Imperial Yıldız Ceramic Plant. In the developed application the user touches one of the artifacts or the picture of Sultan Abdül Hamid II on the main screen (Fig. 7), and the corresponding details are displayed where the user can scroll through textual information and image gallery (Fig. 8).

Fig. 7. Main screen of the application developed for the Museum of Dolmabahçe Palace Collections. Fig. 8. Screenshots displaying textual and visual details of exemplary artifacts.

multimedia forms about historical artifacts, an actual visit to a brick-and-mortar museum is still a popular activity.

Typically information is provided by limited sized text plates, in one or few popular languages, in small numbers and font sizes due to space constraints.

6 Fig. 7. Main screen of the application developed for the Museum of Dolmabahçe Palace Collections. Fig. 8. Screenshots displaying textual and visual details of exemplary artifacts. 4 Augmented Reality Based Mobile Interaction with Historical Artifacts Even though today s ICT technology provides many mechanisms to reach tremendous amount of information in many languages and multimedia forms about historical artifacts, an actual visit to a brick-and-mortar museum is still a popular activity. However during such visits, tourists typically lack the proper channels to access basic information about the artifacts, leave aside the wealth of multimedia material already available on web. Typically information is provided by limited sized text plates, in one or few popular languages, in small numbers and font sizes due to space constraints. A popular and more flexible schema is to provide visitors a headphone unit where the user can enter an id number for an artifact and listen to the provided information in different languages. Various museums also try to add touch-screen computers, similar to previous discussion, to various locations in the museum. Considering the increased use of mobile smart devices with camera units, we developed an innovative way to interact with the artifacts (hence learn more about it) in the museum. Being in the vein of Augmented Reality applications, our system relies on automatic localization of the artifacts on the scene and laying interaction elements on the screen of the mobile device next to the image of the artifacts. An example is in Fig. 9. The target exhibition stage consists of ancient cuneiform clay tablets, which contain extremely intriguing texts (e.g. first love letter) however due to concerns of space and aesthetics, the exhibitors cannot put too much informative text (including the translation) next to the artifacts.

Fig. 9. AR browsing application takes the picture of the scene, automatically locates the artifacts and puts an interaction button on each of them.

After localization, it lays interaction buttons on top of each artifact.

(pictures, audio, and video) about the artifact. Our system results in a very flexible on-site digital browsing experience.

System does not need any extra marker in contrast to many AR systems. It uses natural scene features to localize the artifacts.

7 Fig. 9. AR browsing application takes the picture of the scene, automatically locates the artifacts and puts an interaction button on each of them. In our application, the user directs the camera of the mobile device to the target scene, takes a photo and system tries to localize the elements that were registered before. After localization, it lays interaction buttons on top of each artifact. As the user clicks on the buttons, the name of the artifact is written on a text box, and if the user is interested he can touch further on the name to access the text and multimedia material (pictures, audio, and video) about the artifact. Our system results in a very flexible on-site digital browsing experience. Thanks to being on-site, the user never loses the feeling of immersion and reality. Access to information is natural and fast without touching the real artifacts and cluttering the scene. System does not need any extra marker in contrast to many AR systems. It uses natural scene features to localize the artifacts. The system utilizes modern image recognition techniques which are described next. Artifact Recognition: The artifact localization algorithm works in two steps, offline and online stage. In the offline stage the target scene is modeled from multiple images. 2-8 images are enough depending on the complexity and depth variation of the scene. The scene in Fig. 10 is modeled from 5 images. 2-3 pictures are enough for planar scenes. Fig. 10. In the offline stage, a scene is modeled from multiple images In the core of our system two techniques exist, local image features and image transformation estimation. Former are classical techniques in computer vision which became especially popular after the publication of Lowe [6] on SIFT features. In our case we used royalty-free open-source ORB [7] features. These can be summarized as areas in an image which can be detected in a scene from various vantage points. They are robust to certain types of rotations and scale changes. Hence they give us the ability to automatically register images of the same scene or the objects to each other. First step in offline stage is to detect those features on the input images. Modeling the transformation of the scene is an important part of many computer vision systems. One important decision is whether 3D nature of the scene will be modeled or not. 2D modeling is simpler but may not model the depth variation of the scene very well.

8 After various trials we opted for simpler 2D transformation estimations and to handle increased error we use many pictures to model the scene. Hence rather than seeing the scene as a single 3D entity, we modeled it as multiple 2D images. After detecting local feature points, second step in offline stage is to chain the images and their features by matching the local features. Hence we create tracks of local features on all the input images. Third stage is the meta-information tagging stage where the user put tags (put virtual artifact ids) on input images which are later to be used to associate that location with artifact information. In the online stage, mobile device camera is directed towards the target scene and a picture is taken. In the second step, local image features are detected in the scene, as in offline stage. In the third step, these features are matched against feature tracks of the scene model that was computed in offline stage. Then after, the best model image is searched which transforms best to the taken picture. This is done by projective transformation (homography) estimation [8]. In the next step, the computed homography matrix is used to transfer the virtual artifact positions to user screen and renders interaction buttons. The rest is straightforward digital browsing experience. 5 Conclusion We presented the results of a project about utilizing modern commodity hardware to create novel interaction experience for museum visitors. To create a 3D rendering experience, accurate geometric representations are generated by 3D scanners. Selected artifacts and related 3D/multimedia information is presented through information screen application. Natural and easy interaction with real artifacts is achieved through usage of smart mobile devices and visual computing techniques to locate the objects and lay augmented meta-data over them. Acknowledgments. This project was supported by Istanbul Development Agency, Global Tourism Center Istanbul Program with project number, ISTKA/2012/KTM- 31. References 1. Louvre-DNP Museum Lab, 2. Bayon Digital Archival Project, 3. Virtual Asukakyo, D Murale, 5. Bruno, F., Bruno, S., De Sensi, G., Luchi, M. -., Mancuso, S., & Muzzupappa, M.: From 3D reconstruction to virtual reality: A complete methodology for digital archaeological exhibition. Journal of Cultural Heritage, 11(1), (2010) 6. Lowe, D.G.: Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vision, 60 (2), (2004) 7. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.R.: ORB: An efficient alternative to SIFT or SURF. In: International Conference on Computer Vision, pp (2011) 8. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, New York (2003)

The presentation based on AR technologies

The presentation based on AR technologies Building Virtual and Augmented Reality Museum Exhibitions Web3D '04 M09051 선정욱 2009. 05. 13 Abstract Museums to build and manage Virtual and Augmented Reality exhibitions 3D models of artifacts is presented