Augmented reality for machinery systems design and development

Published in: J. Pokojski et al. (eds.), New World Situation: New Directions in Concurrent Engineering, Springer-Verlag London, 2010, pp. 79-86 Augmented reality for machinery systems design and development Marcin Januszka a,1, Wojciech Moczulski a a Silesian University of Technology, Department of Fundamentals of Machinery Design Abstract. The paper presents the application of augmented reality to support product design and development. The paper concentrates on a new engineering tool. By this reason design and its issues are presented in a very limited way. The goal of the research was to improve design process and reduce product development time. The presented system bases on CAD software which is integrated with an expert system and augmented reality techniques as a human-computer communication interface. The authors present possibilities of using augmented reality technology in CAD design with the hope that maybe someday it would become integral part of a standard design process of more reliable and durable machinery systems. Keywords. augmented reality, human-cumputer interface, CAD, reliability, computer-aided decision making 1 Introduction A technical evolution causes that people design more and more complex technical objects with the use of methods and tools from various disciplines. A computer technology allows to improve efficiency of design and development process. Some problems (e.g. concerning on reliability of machinery systems) without computeraided systems could be non-solved [12]. CAD systems are very useful tools to make allowance for reliability of products. The designer which makes some modifications (e.g. in power transmission system) is able to evaluate an influence of these modifications on reliability of the designed machinery system [12]. In addition to CAD more and more virtual and mixed reality (see [2] [3] [10] [14] [16]) techniques are used to improve and optimize the design and development process. Digital (virtual) prototypes are built at the beginning of a product development process. These virtual prototypes should be changed and modified easly to optimize the development process. As always, to make a proper decision by the designer during design process (e.g. conceptual design) and evaluate reliability of a future product, it is important 1 Silesian University of Technology, Department of Fundamentals of Machinery Design, 18A Konarskiego str., 44-100 Gliwice, Poland, e-mail: marcin.januszka@polsl.pl

2 M. Januszka, W. Moczulski to adequate present (visualize) the product [3] [6] [16]. Thanks to modern visualization systems it is possible to maximize legibility of product models (e.g. possibility of visualization in 1:1 or higher scale). For that purpose the designer can use innovative tools of visualization: 3D monitors, virtual caves, head-mounted glasses etc. Over the last few years an innovative technology called augmented reality (AR) may be used for aiding designers in efficient way [2] [3] [6]. Augmented reality creates and environment where virtual (most often 3D) are inserted in a predominantly real world enironment (fig. 1). Fig. 1 Reality-virtuality continuum [6] AR technology can enhance a user s perception of the real world with information that is not actually part of the scene but is relevant to the user s present activity [1]. It provides a natural and intuitive means for user to work efficiently in the real worlds environment. Augmented reality systems allow to aid the user in his/her operation thanks to combing in one environment information about virtual and real world. Using AR techniques, the designer wearing HMD devices can move about in a phisical 3D space to view interactive 3D models that are being displayed from different perspectives. Adapting AR to practical uses is connected with various problems (see reports included in [11]), especially concerning hardware technologies. There are still some problems to solve, but for industrial partners to be interested in investing into this technology its possible benefit and its integration into the whole company has to be visible. There are several reports of trying to apply AR in industry. In [13] the authors present research, development and deployment of AR systems in the automotive, aviation and astronautics industries. In [9] the author summarizes research realized by industrial AR (IAR) consortium which supports the augmented reality for development, production, and servicing in industry (especially in automobile industry). In this paper the authors describe results of research on augmented reality technology to aid machinery systems design and development. The paper concentrates on a new engineering tool. The design process and its issues are presented in a limited way. The paper presents the results which are a continuation of the research initiated by the authors in 2005 [8]. 2 Research background Our research concentrate on improving CAD and development of machinery systems with the use of modern techniques of visualization. The new AR systems (as presented in this paper) may be considered in the support of concurrent engineering and remote and group collaboration methodologies. The goal of research was to elaborate a method and system for supporting CAD. The method

Augmented reality for machinery systems design and development 3 should allow to improve efficiency (e.g. shortening of development time) of a design process and in effect design more reliable and durable machinery systems. At the beginning the authors decided to elaborate some mechanisms of searching, collecting, processing of data (especially 3D models) necessary for the designer during designing of a new product. A method for knowledge/data representation which are necessary for the designer was one of the main challenge of the authors. These data should be concerned on existing constructional solutions. These data could be used by the user to design new products or improve existing constructional solutions in order to increase reliability. To solve the problem a simple expert system and a database were elaborated. Design knowledge is acquired from: domain experts, a professional/scientific literature or results of research (exploitation, laboratory etc.) collected in databases [7]. The expert system delivers to the designer knowledge about successive steps of the design process of a machinery system and practical solutions of realized constructional problems. Data adequate to solved problem (concerning on algorithm of design process) are displayed in a real environment (with the help of AR system) around the user while he/she is going to make some decisions. An elaboration of a system for presenting knowledge and data to the user was the next research problem. The designer should be able to use data and knowledge from the system during a design process [15]. For that purpose AR as a tool to control a dialogue between the designer and the CAD system and the system aiding decision making could be used. The system should allow visualization of data about: existing constructional solutions, symptoms of failures and inefficiencies (e.g. exceeded limit of vibrations, noise, temperature, pressure etc.) in a previous version of a product, algorithms of design processes of selected group of machinery systems (e.g. mobile robots) or machinery parts and elements, critical points of the design process etc. Knowledge for the designer should be presented in form of: drawings, pictures, 3D models, schemes, tables, text or audio specification. In case of 3D models (not only, but especially) should be able to display them in any scale (especially 1:1 scale) with possibility of view from any perspective in a very intuitive way. Using AR techniques in CAD causes some advantages. First of all is that the user has possibility to preview data (especially 3D models) in real environment, directly around users at any place, from any perspective (each of designers have own perspective), in any scale. AR also gives possibility of total interacting with displayed data by manipulation of position and orientation in a space around the user. It was proved in [14] that perception of 3D designs in better when changing views by observer movement than by model rotation (e.g. with the use of the standard monitor). AR mode for changing views of data - especially 3D models - allows the user to understand the prospective system in a more comprehensive way, thus making design process more efficient than the one supported by conventional present-day CAD systems [4] [5]. The designer has possibility for fast and efficient verification of designed products in order to increase reliability. Advantages of AR techniques cause rationalization of the design process: shortening development time, reducing development costs, improving utilization of existing constructional solutions and utilization of existing exploitation data [4] [5].

4 M. Januszka, W. Moczulski 3 Overview of the proposed prototype system The authors carried out an implementation of the prototype system for a design process of selected group of machinery systems mobile robots. Although, it is also possible to build the system for any machinery systems. 3.1 Hardware and software components of the system The basic component of the implemented system is a computer with MS- Windows XP system installed. The computer runs Dassault Systemes CATIA V5R19 (modeling software) and a special elaborated AR application. The HMD with a small USB camera attached is connected to a video card of the computer. The video camera captures video of the real world and sends it to the computer. HMD allows to see data from the computer (e.g. 3D models, text etc.) in real environment surrounding the user. The user wears the HMD with the video camera attached, so that when she/he looks at the tracking card with a special marker through the HMD a virtual object is seen on it (fig. 2). Fig. 2. System for aiding the designer of machinery systems (components) Very important and difficult part of AR system proposed by the authors is software. The presented system consists of: ARToolKit tracking libraries, VRML (Virtual Reality Modeling Language) parser, expert system (system aiding decision making) with database with 3D models (in CATPart or VRML formats), Dassault Systemes CATIA V5R19 (modeling software), a main application to integrate all software components and to realize system functions. The system base on the public-domain augmented reality tracking library called ARToolKit (from HIT Lab [17]) with LibVRML97 parser for reading and viewing VRML files. ARToolKit is a software library that uses computer vision techniques to precisely overlay VRML models (3D models, text, pictures etc.) onto the real world. For that purpose software uses markers. Each marker shows a different digitally-encoded pattern on it, so that unique identification of each marker is possible. In presented conception

Augmented reality for machinery systems design and development 5 the markers are printed on the cards. We can compute the user s head location as soon as the given marker is tracked by the optical tracking system. Finally the main application allows to display data superimposed on the real world (exactly on the card with the marker position and orientation). Thanks to an elaborated graphical user interface (GUI) users have possibility to e.g. (fig. 3): update knowledge in a knowledgebase (by a Knowledge Engineer), view instructions about stages of a design process of a mobile robot (the system aids the designer in a design process), preview data from a database (tables, drawings, pictures, text, voice or video information), use (by export to CAD system) existing 3D models to improve them, view results of designer's work (e.g. 3D models, results of simulations: kinematics, structural strength, thermal strength), in AR mode etc. All displayed information (especially 3D models) could be viewed in the real environment in full 3D mode and 1:1 scale. 3.2 Aiding decision-making The main goal of the implemented system was to aid the designer of machinery systems for designing more reliable mobile robots. Knowledge (necessary during design process) to the knowledgebase is acquired e.g. from experts and inscribed in knowledgebase thanks to an electronic form. Knowledge (represented in procedural form) and data is used in the design process to eliminate causes of failures and inefficiencies in future products. Data is collected in the database. The designer can take advantage of data (3D models, drawing documentation, diagrams, BOM s, catalogue data etc.) or knowledge as often as needed (fig. 3c,d). In the database the data is collected in form of text, graphics, drawings, sounds, 3D models. The presented prototype system guides the designer during successive stages of the design process of a mobile robot. Fig. 3. Main functions of the prototype system: a) b) viewing results of designer s work, c) d) previewing data from a database, e) viewing instructions or procedures concerning design process (AR system aiding decision making)

6 M. Januszka, W. Moczulski The system aids the designer in decision making with the use of a modern AR communication human-computer interface (fig. 3e). The expert system aids the designer only in qualitative way, nevertheless future research are going to aid the designer also in quantitative analysis. Prompts inferred by the expert system after user acceptance are automatically applied in designed product 3.3 Aiding the designer - functioning of the system In the presented conception of the system the user with HMD on the head sits in front of a computer. The user looks in the direction of the card with marker(s) through HMD and virtual objects are seen onto this card. As virtual objects the text information, 3D models, pictures, catalogue tables, drawings from the database or instructions from the knowledgebase could be displayed (fig. 3). Information displayed with the use of AR system are very helpful for the designer. In the same time the user can design the mobile robot in CATIA CAD system. The user can view models of existing robots or components of robots in AR mode (in 1:1 scale, from any perspective). It is possible to export/import these parts between modeling software CATIA V5R19 and AR system. The designer in the CATIA s workplane can see the part and simultaneously this part could be seen in AR mode (fig. 4). When the design process is accomplished the user can preview results of his/her work in AR environement. Fig. 4. Previewing results of designer s work and manually manipulating the model for an inspection The designer can manually manipulate the model for an inspection. It is also possible to export the models with results of analysis e.g. strength analysis and the others. Other designers if sit around a table with a marker and examine projects of a robot can also see a three-dimensional virtual image of the 3D model. The image is exactly aligned over the real world so the designers are free to move around the table and examine it (collaborative work) from any viewpoint. Each person has their own viewpoint to the model. They are also free to interact with the model in real time. All the helpful information from AR system and models of machinery

Augmented reality for machinery systems design and development 7 systems are viewed by the user in the real environment from any perspective in a very efficient and intuitive way. 3.4 Verification The verification of the system was realized only in an easy way because of early stage of the system development. Within a framework of a verification of research, some experiments depend on realization of a complete design process of a mobile robot were carried out. Experiments were realized with the use of presented in this article system and without. Some experienced and non-experienced designers were selected to take part in the experiment. Efficiency of the system was confirmed, thanks to shortening development time of an elaborated product (mobile robot). In the case of experienced designers the development time was shortened about 10%, while in case of non-experienced designers the time was shortened about 20%. As well, utility of the subsystem for aiding decision making during a design more reliable mobile robots was confirmed. The system allows for the designer to choose a better solution of the problem e.g. application of caterpillar drive instead of wheel drive, which was worse in previous version of a similar mobile robot to inspect ruins. Non-experienced designers have more efficient access to data and knowledge, so they are able to finish the design process faster than designers which don t use the presented system. The system evaluation was carried out only on the base of special forms filled by designers after an experiment. Detailed verifications will be realized during further development of the system, and the authors are going to present the results in successive publications. 4 Conclusions A development of AR technology in design (but not only, also in maintenance and diagnostics) has been initiated by the author and research team from Department of Fundamentals of Machinery Design few years ago [8]. The results of research confirm advantages from application of AR techniques in a design domain. AR mode for visualizing data and completely understanding the model content is more efficient, intuitive and clear than the traditional one (e.g. with the use of a monitor). By this reason the AR technology, as a kind of new user interface, introduces completely new perspective for the computer aided design systems. Results of research confirm other potential advantages derived from the presented system using AR techniques, especially: extended efficiency of direct access to data and knowledge which is necessary during a design process, possibility of aiding decision making and delivering detailed design algorithms to improve reliability of designer machinery systems, an improved mechanism of making full use of existing knowledge and data (possibility of importing/exporting data between a knowledgebase/database and CAD system),

8 M. Januszka, W. Moczulski possibility of presenting data necessary during a design process (e.g. tables, results of research, schemes, 3D models) and results of this process (e.g. final 3D models) in efficient way, with possibility of interaction with these data and viewing from any perspective and also in any scale (also 1:1 scale). The presented system basing on the method of computer-aided design of reliable machinery systems and using AR techniques is in an early stage of development. However, advantages of systems based on AR confirm possibility of future introduce of augmented reality technology in design engineering enterprises. 5 References [1] Azuma RT. A Survey of Augmented Reality. In Presence: Teleoperators and Virtual Environments 6, 4 (August 1997); pp. 355-385 [2] Dunston PS, Wang X, Bilinghurst M, Hampson B. Mixed reality benefits for design perception. ISARC 2002,Gaithersburg, 2002; pp. 191-196. [3] Dunston PS, Bilinghurst M, Luo Y, Hampson B. Virtual visualization for the mechanical trade. ISARC 2000, National Taiwan University, 2000; pp. 1131-1136. [4] Januszka M, Moczulski W. Machinery design aided by augmented reality technology. Computer Assisted Mechanics and Engineering Sciences, 2007, 14: 621-630. [5] Januszka M, Moczulski W. Collaborative augmented reality in CAD design. Machine Dynamics Problems 2006, 30(3): 124-131. [6] Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: a class of displays on the reality-virtuality continuum. SPIE Telemanipulator and Telepresence Technologies, 2351, 1994. [7] Moczulski W. Methods of Knowledge Acquisition for the needs of machinery diagnostics (Monograph, in Polish). Mechanika, vol. 130, Gliwice, 1997. [8] Moczulski W, Panfil W, Januszka M, Mikulski G. Applications of augmented reality in machinery design, maintenance and diagnostics. In: R. Jablonski, M. Turkowski, R. Szewczyk, eds., Recent Advantages in Mechatronics, Berlin Heidelberg: Springer- Verlag, 2007; pp. 52-56. [9] Navab N. Developing Killer Apps for Industrial Augmented Reality. IEEE Computer Graphics and Applications, 2004; pp. 16-20. [10] Nölle S, Klinker G. Augmented Reality as a Comparison Tool in Automotive Industry. Proc. of IEEE/ACM ISMAR 2006, IEEE Computer Society, 2006; pp. 249-250. [11] Ong SK, Nee AYC. Virtual and Augmented Reality Applications in Manufacturing. Springer, 2004. [12] Oprzędkiewicz J., Computer-aiding in reliability of machines (in Polish). WNT, Warszawa, 1993. [13] Regenbrecht H, Baratoff G, Wilke W. Augmented Reality Projects in the Automotive and Aerospace Industries. IEEE Computer Graphics and Applications, 2005; pp. 48-56 [14] Shin DH, Dunston PS, Wang X. View changes in Augmented Reality Computer- Aided-Drawing. ACM Transactions on Applied Perceptions, Vol. 2(1), 2005; pp. 1-14 [15] Skarka W. Methodology of knowledge-based engineering (Monograph, in Polish). Publishing house of Silesian University of Technology, Gliwice, 2007. [16] Wang X, Dunston PS. Potential of Augmented Reality as an Assistant Viewer for Computer-Aided Drawing. Journal of computing in civil engineering (November/December), ASCE 2006; pp. 437-441. [17] ARToolKit Documentation - HITLab at the University of Washington. Available at: <http://www.hitl.washington.edu/artoolkit/documentation/>. Accessed on: Feb 16 th 2010.