Knowledge Acquisition and Representation in Facility Management

2016 International Conference on Computational Science and Computational Intelligence Knowledge Acquisition and Representation in Facility Management Facility Management with Semantic Technologies and Augmented Reality Prof. Dr.-Ing. habil. Christian-Andreas Schumann Eric Forkel Frank Otto Institute for Management and Information West Saxon University of Applied Sciences Zwickau Zwickau, Germany Prof. Dr.-Ing. Egon Müller Dr. Michael Clauß Department of Factory Planning and Factory Management Chemnitz University of Technology Chemnitz, Germany Dr. André Lanka Dr. Lanka & Partner Software Developers Chemnitz, Germany Abstract One more and more upcoming challenge for IT systems used in the Facility Management domain is the automated acquisition and processing of distributed heterogeneous data. Especially for approval or maintenance processes the mobile access to necessary data constantly gains significance. But because of their limited size particularly mobile devices require new ways for representing the needed amount of information. This conceptual paper discusses new approaches of knowledge acquisition and representation, which enable the efficient support of facility managers in their daily tasks. The underlying research project develops concepts and solutions for the semantic modelling and linking of distributed data as well as its context-based representation on a mobile device via Augmented Reality. Keywords Semantic Technologies, Augmented reality, Knowledge Acquisition, Knowlwdge Representation, Facility management I. INTRODUCTION Facility Management (FM) nowadays faces a wide range of different tasks: Planning, execution and optimization of support processes have to secure a company s primary processes. Especially the approval of facilities and their maintenance tasks are important fields for FM-activities. Over a facility s lifetime a multiplicity of different data arises in different situations and phases. These data often are heterogeneous, which means the data have different formats and are stored on several media. Due to the fact that several companies are only involved in specific phases of a facility s lifecycle the data often get lost and thereby are not accessible to all companies involved in this lifecycle. These are the problems that are addressed within the research project Facility Management with Semantic Technologies and Augmented Reality (FMstar). This project, which is funded by the German Federal Ministry for Economic Affairs and Energy, consists of partners from three universities and four industrial companies. With their comprehensive mix of competencies in factory (Chemnitz University of Technology), facility (University of Technology and Economy Berlin, UAS) and information management (West Saxon University of Applied Sciences Zwickau), the university partners provided the theoretical background and core of the methodological input while two of the partners from industry offered real-world insights for an adequate problem scope and useful solution definition. This specifically means that the companies offered access to state-of-the-art planning, management and maintenance tools and processes in facility management, so that the prospected system architecture could comply with the most important practical restrictions. A third industrial partner provided specific capabilities for the conceptualization and implementation of the semantic database, while the last industry-partner had the task to support the programming of the software prototype. [1] 978-1-5090-5510-4/16 $31.00 2016 IEEE DOI 10.1109/CSCI.2016.191 1001 1005

II. CONCEPTUAL APPROACH As already described the most important obstacle for providing a facility manager with all needed data for a certain task are the heterogeneous and distributed data. Therefore the intention of the research project FMstar was to reduce the lack of information by providing possibilities to link several information resources via a semantic database. In the next step the data, which are needed to fulfil a certain task, should be filtered out of the whole amount of information by an automated recognition of the worker s contexts, e.g. his personnel properties, his current task, his location or the current time. This context-based amount of information then should be displayed on a mobile device like a smartphone or a tablet computer via an Augmented Reality representation by overlaying the physical facility with a corresponding threedimensional model. Figure 1 shows these intentions. implicit recognize the semantics and the data structure by using inference will be needed. [1] Fig. 2. Alternative strategies for integrated data access [1] Fig. 1. The intentions of the research project FMstar [2] A. Semantic Linking The semantic linking bases on an ontology with FMrelevant keywords and their relations. This ontology builds the basic concept for integrating the existing information systems used by the FM-companies. For this purpose there are different strategies, which are shown in Figure 2. One strategy focuses on the usage of identical standards, patterns and vocabulary for all application systems that are used by the different partners. Due to the fact that this won t work in practice, it can be stated that the usage of individual standards should be regarded as the common basis for data integration. Nowadays the mapping of data from different information systems commonly is a manual task which requires knowledge about data structures and semantics. In future the increasing number of information sources (due to e. g. higher automation ratio and intelligent sensors) has to lead to approaches with automated importplugins. As a requirement for automated mapping either explicit descriptions of the semantics and the data structure of the corresponding IT systems or algorithms that are able to B. Context Adaption Context adaption is the approach to solve the problem how a facility manager can be provided with only the relevant amount of information. The question whether a piece of information is relevant for the user s current situation or not is defined by the Relevance Theory, especially by a function of the processing effort on the one hand and a positive cognitive effect on the other one. The processing effort in this case means the energy which a user needs to recognize, comprehend and use a certain piece of information. If the user s demand for information is satisfied by receiving important information based on the user s situation, a positive cognitive effect will be achieved. The lower the processing effort and the higher the positive cognitive effect, the higher is the relevance of a certain piece of information [3]. Thereby the relevance of information depends on the user s situation. According to Dey the information that can describe an entity s situation is defined as its context [4]. Therefore context adaption describes a system s ability to adapt its behavior on the basis of certain context information [5]. C. Augmented Reality After having the data integrated and adapted to the user s context it has to be visualized to him. Due to the fact that mobile devices are omnipresent nowadays a tablet computer is an intuitive approach to solve this problem. In the FMstar project a tablet computer is used to determine the user s context (e.g. his location or his current task) as well as the representation of the needed information. These information are fixed to a three-dimensional model of the corresponding machine or the machine s components. The 3D model and the digital information thereby are superimposed over the device s camera view (which shows the real world behind the tablet) in an Augmented Reality representation. III. DEVELOPMENTAL PROCESS For this prototype the following restrictions were stated: The prototype should provide a semantic database which should connect all needed information systems automatically via an FM-ontology. 1002 1006

The facility manager should get an intuitive access to the provided information by using a standard mobile device. Hereby the facility manager s interaction with the system should be realized by a virtual 3D-model which should be displayed as an Augmented Reality representation. The user then should interact with the components of this 3D-model to see their respective properties and tasks. The 3D-model should be provided in a standard data type, which different construction software tools could export to. Within this project the Industrial Foundation Classes standard (IFC) was defined as suitable. Based on these restrictions it was decided to build up the prototype in the form of a client-server-architecture. The server component should contain the semantic database, the 3Dmodels and additional server-side functionalities. The client s software should consist of an Android-Application. As a suitable device for this application the Google Nexus 10 tablet was used, because of its good performance and little price compared to other mobile devices at the time the project started. The project development consisted of the parts Defining use cases, processes and structures from the facility management domain Modelling and Setting up the semantic database Setting up the server-side functionality Developing an Android application for the representation of and the interaction with the 3Dmodel Testing, evaluating and improving the prototype Due to the research project s experimental character with only little experience available for a foresightful planning, the developmental process followed the approaches of agile project management and software development. As far as possible the previously described parts of the project development therefor were split into atomic tasks. Figure 3 shows the developmental process along general lines. Basing on the project s industrial partners needs the project s modelling team started modelling the corresponding use cases, processes and structures in a formally way with Unified Modelling Language (UML) and ARIS, a business process modelling tool. After that, the results were discussed with and verified by the industrial partners. Finally, based on these models, an ontology for the facility management domain has been modelled using the modelling software Protégé (see protege.stanford.edu). This ontology defined basic FM concepts, terms and properties as well as their relationships. Later on this ontology was used for structuring the central semantic database. Fig. 3. The development process chosen in the project [1] When the work on the ontology was finished, the software prototype s server component had to be expanded to provide the functionality that was needed to use the ontology to connect to the industrial partner s IT systems. For the ontology s integration into the server component the Apache Jena Framework was used (see jena.apache.org). Originally it was intended that these connections should work automatically by combining the ontology s terms with corresponding terms in the IT systems, but due to the fact that the ontology only contained basic FM terms this automatic connection didn t work, because the object notation in the IT systems the industrial partners were using differed from this basic standard. So for each information system a function to map this special system to the ontology had to be implemented in the prototype s server component. After adapting the different IT systems to the semantic database all facility s data that were created in its lifetime were integrated in this database and could be provided by the prototype. This besides others included the facility s composition, its properties, documentations, manuals, maintenance plans and historical data. The challenge now was to provide the maintenance worker with all information he could need to fulfil his tasks, but not to overwhelm him with too many unnecessary information. To pass this challenge the worker s situation had to be analyzed. Therefor different contexts were defined: the time, the user and his role, his location and viewing direction, his current task and the related objects. The concrete status of each context should either be determined after some user s action (e.g. the user is known after he logged in, the task and the object are known after the user selected them) or automatically (e.g. the user s locating and viewing direction, the time of the tablet). After the contexts determination the available amount of information could be reduced to the needed one. This allowed the worker to have an intuitive overview over all relevant information. The worker additionally was able to create new data by writing notes, taking pictures and marking something on them and writing some remarks to these pictures or by recording some voice messages. These data were automatically linked to the previously selected facility s objects. Another requirement the prototype had to fulfil was about displaying the information the maintenance worker would need to complete a certain task as an Augmented Reality (AR) representation. For such an AR representation it is necessary that the real and the virtual world are displayed on one screen 1003 1007

matching each other. To ensure this, it is necessary to know the worker s location and viewing direction. To locate a worker there are many approaches that can be used, e. g. location via GPS or Wi-Fi triangulation. But approaches like these have some disadvantages. Positioning via GPS maybe won t work in buildings because of the missing connection to the satellite system. For using a Wi-Fi triangulation or similar systems like ibeacons it is necessary to have such a system installed at the company s workshop and even if such a system exists it cannot be guaranteed that the calculated position is as exact as it has to be. So, for getting a sufficient positioning the project team had to choose another approach. It was decided to use QR-Codes as markers for an initial positioning. For enabling the tablet to read these QR-Codes an already existing Android module was included into the prototype. This module then was extended to provide the ability to not only read the code s content, but also to calculate the worker s position (related to the QR-Code) by recognizing the code s size and distortion. After the worker has located himself by scanning a QR-Code the intended further way was to have the ability to freely move around. The tablet should recognize this movement and adjust the 3D-model according to this movement. Therefor two different approaches were tested: First one used the tablet s inbuilt sensors (gyroscope and accelerometer) to recognize the movement and adjust the 3D-model accordingly. This approach worked quite well for rotational movements but didn t recognize translational movements. So the second approach was to recognize movements via feature tracking. This method uses the device s camera. It takes pictures in defined time intervals and then tries to find out similarities on two pictures. If some similar features were found the algorithm can calculate their deferral and adjust the 3D-model accordingly. This approach also worked, but was only able to calculate two deferrals per second, which is too less to have a smooth 3D-model s adjustment. In the end the solution for locating the worker was to always have QR-Codes in the worker s field of view. After being able to calculate the worker s location and viewing direction another functionality was implemented into the client, which calculated the visible part of the 3D-model and only displayed the according components and thereby improved the application s loading time once more. For testing purposes of this locating functionality some of the 3D-model s components were printed out with a 3D-printer and equipped with the corresponding QR-Codes. Due to the fact that not every company allows Wi-Fi or mobile connections in their workshop another challenge was to give the worker the ability to work with the tablet without having a connection to the server component. Therefore every information was cached on the tablet, linked to the corresponding 3D-model. When starting the application on the tablet with an established server-connection it was checked if there was anything updated whether on the server or the client side while no connection was available and in this case these information were updated on the corresponding prototype s component. One last challenge was the optimization of the facility manager s tasks in a manner of self-organization. One case this approach is used in is the order of the tasks that are done by a certain facility manager. When having done these tasks several times it becomes possible to suggest the most efficient order to fulfil these tasks to the facility worker and thereby minimize the time the worker needs to get his tasks done. Fig. 4 shows the prototype s architecture in an overview. Fig. 4. Overview over the prototype's architecture [1] IV. HOW THE INITIATIVE WAS RECEIVED The prototype with its status at the respective time was presented at fairs like the CeBIT (2016), the Hanover Fair (2014 and 2015) or the facilitymanagenent in Frankfurt/Main (also in 2014 and 2015) as well as on conferences like the IN- TECH in Leiria (Portugal, 2014) or the European Facility Management Conference in Berlin (2014). Most of the interested visitors were enthusiastic about the project s ideas and the prototype s development. Mostly all of them said, that this could be a promising approach to meet the upcoming challenges within the Facility Management domain. Some of the visitors remarked that the usage of a tablet would not fit their needs, because their workers need their hands free to fulfil their tasks. For them an approach with smart glasses would suit better. Others stated that the locating functionality won t work in their workshop, because they can t or won t equip their facilities with QR-Codes. Due to the fact the development process concentrated on the functional implementation, the prototype s design wasn t that eye-catching than it could be. Some visitors also criticized this. Nonetheless the most critiques came from the circumstance that the prototype needs 3D-models as a central element. Interested visitors remarked that 3D-models are not state-of-the-art, yet, in the Facility Management domain. Facility managers nowadays have 3Dmodels for only 15 to 20 percent of all their facilities, but almost all of them are sure that this percentage will increase rapidly within in the next years. Further feedback came from presentations to potential buyers and a questionnaire several companies from the Facility Management field took part at. This feedback conformed widely to the feedback the project team got at the fairs and conferences. 1004 1008

V. CONCLUSION AND PLANS TO FURTHER DEVELOP THE INITIATIVE First of all it has to be stated that all of the project s main goals were achieved, although some of them not in the originally intended quality. The built up ontology is able to link the different IT systems, but only in a manually way. The problem in this case was that the ontology only contains basic terms which wasn t sufficient to automatically link the IT systems. With more time for building up the ontology more terms could be included, which would lead to better results in this case. The 3D-model can be displayed on the tablet and the facility worker can interact with it as well as its components, properties and tasks. The worker is able to locate himself to adjust the 3D-model according to his location. A marker-less solution would be a more comfortable way. Modern tablets with more powerful components would generate better results when using the feature tracking functionality. Another approach for this case would be to try to use other apps like for example layar that promise fast and reliable feature tracking functionality. The recognition of different user s contexts works as intended. Beside the improvements that were discussed in the previous section some more steps have to be done to enlarge the prototype s benefit. As since the project start some time has passed, new promising technologies arise on the market. One of them is the Google Tango Project. Its included technology promises to create 3D-models by simply scanning the environment when walking around with a special Google Tango Tablet that is held in front of the body. Such tablets can increase the number of exiting 3D-models with only little afford. Another new technology are smart glasses. By using such a wearable device instead of the tablet the worker would have his hands free to do his actual work. The most important information to fulfil his tasks could be displayed on the glasses, so that these information remain, however, in his field of view. For potential users who want to work with the tablet furthermore, a redesign of the Android application s surface would possibly support a higher intuitiveness to the facility workers, even if the surface complies with its requirements, so far. Additionally, the further developmental process could implement some more use cases, e.g. such as navigating a worker through a large workshop to his destination or giving remote-access to another distant worker. REFERENCES [1] M. Clauß, J. Götze, E. Müller, C.-A. Schumann, Real-Time Data Access Through Semantic Technologies and Augmented Reality in Facility Management Processes, Iternational Conference on Innovative Technologies IN-TECH, pp. 45-48, September 2014. [2] J. Götze et al., FM Knowledge Modelling and Management by Means of Context Awareness and Augmented Reality, European Facility Management Network, EuroFM Journal Advancing Knowledge in FM, pp. 235-243, March 2014. [3] D. Sperber, D. Wilson, Relevance Theory, L. Horn, G. L. Ward, Handbook of Pragmatics, Blackwell, Oxford, p 251, 2002. [4] A. K. Dey, Providing Architectural Support for Building Context- Aware Applications, Georgia Institute of Technology, Atlanta, 2000. [5] W. O. Sitou, Requirements Enginering kontextsensitiver Anwendungen, Institut für Informatik der Technischen Universität München, München, 2009. 1005 1009