iwindow Concept of an intelligent window for machine tools using augmented reality

iwindow Concept of an intelligent window for machine tools using augmented reality Sommer, P.; Atmosudiro, A.; Schlechtendahl, J.; Lechler, A.; Verl, A. Institute for Control Engineering of Machine Tools and Manufacturing Units, University of Stuttgart, Seidenstr. 36, Stuttgart, 70174 Germany Philipp.Sommer, Agus.Atmosudiro, Jan.Schlechtendahl, Armin.Lechler, Alexander.Verl@isw.uni-stuttgart.de Abstract: Most of today s machine tools are equipped with windows, allowing the operator to observe the current process. These windows need to fulfill high safety requirements, but provide no further functionality. In this paper, we will introduce the concept for an intelligent machine window, which will replace current windows in machine tools, to allow the observation of the machine interior at any time and provide enhanced functionality to the operator. To achieve this, the machine interior must be visualized, for example by 3D-reprojection using multiple cameras or by a real-time computer model and enriched with further context-related information using augmented reality. Different use cases for providing additional contextrelated information to the machine operator by augmented reality will also be discussed in this paper. To give the operator an immersive feeling, head-tracking will be used to provide a correct perspective view. To evaluate the visualization by a real-time computer model and the head-tracking, a small demo was set up. Keywords: Augmented Reality, Visualization, Machine Tools 1. INTRODUCTION Today, most machine tools are equipped with a window, allowing the operator to observe the current process. On the other side, these windows prevent the operator from dangers from within the machine and need to fulfill high safety requirements. Therefore, in the field of laser cutting machines and traditional machine tools such as milling, these windows are very cost-intensive. Still, because of cooling lubricant mist, dust and chippings, it is often not possible to observe the tool and the work piece during the process. Therefore, despite high costs these windows provide only small use. For start-up, operation and maintenance of machine tools, additional supporting systems are used. These systems are for example HMIs for interaction with the machine, smart devices like tablets as additional displays for information and diagnosis and simulation systems for virtual start-up of the machine and for process validation. The use of all these different systems, simultaneous or one after another, could overwhelm the

operator. Therefore, these systems need to be merged and a context-dependent relation between them needs to be created. As we can see from this, there is a need to replace traditional windows in machine tools with a new system to face the high costs due to safety requirements and provide enhanced functionality to the operator. The observation of the machine interior, especially machine tool and work piece, should be possible at any time. To achieve this, the machine interior must be visualized and enriched with further context-related information using augmented reality. To reduce the amount of different systems used by the operator, they need to be merged into one single system. In our current research project iwindow we try to develop an intelligent machine window, providing these functionalities und improvements. 2. VIRTUAL AND AUGMENTED REALITY IN PRODUCTION Recently, virtual and augmented reality have more and more found their way into production and manufacturing environments. Scientific researches as in [Ong et al., 2008; van Krevelen and Poelman, 2010; Ma et al., 2011] identify the potential of augmented reality technologies to optimize industrial processes. Most of these applications address assistive or instructive tasks. But in the field of laser cutting machines and traditional machine tools such as milling, there are only a few known augmented reality research projects. In [Zhang et al., 2006] and [Zhang et al., 2010], the authors present an implementation of machine simulation in a real machining environment based on augmented reality. Though these papers focused on the research of the simulation and not to provide added value to the operator. [Olwal et al., 2008] presents the usage of augmented reality to overlay a real machine tool with process data. The process data, such as forces, power, speed override, feed and RPM, is projected on a transparent holographic optical element in front of the machine window. This display covers only a small area of the work space and only textual and very simple content can be shown on the screen. Also during process, the machine interior cannot be observed due to cooling lubricant mist, dust and chippings. 3. USE CASE DESCRIPTION AND CONCEPTION The main disadvantage of previous work to augmented reality in production is the lack of added value for the operator. To provide added value to the operator using an iwindow, we deduced different use cases. These use cases and their conception are presented in the following chapters. 1.1. Visualization of work piece contour on leftover material During manual production on laser cutting machines, the operator often uses leftover material and manually places the work piece contour on it. To achieve a better material utilization the component placement should be optimized. An approach to this is to

manually place work piece contours on the material on a screen showing a camera image of the machine interior. Yet this can be achieved by using augmented reality on external screens and with a single camera. To provide additional perspectives, a full three-dimensional observation of the machine interior is required. Also, the operator should be able to position the work piece contour on the leftover material either on the machine window or on a second screen (e.g. a tablet). During positioning, it might be necessary to rotate and zoom the current view. The work piece contour will be extracted from the NC program and converted to line segments. These segments will be represented as three-dimensional objects and can therefore be correctly observed from every direction. Also possible occlusion by other objects will be shown correctly. To freely position the contour on the leftover material, touch input will be used. Figure 1: positioning of a work piece contour on leftover material. Source: TRUMPF 1.2. Visualization of tool paths iwindow should allow the operator to visualize tool paths. This can help to detect unintended behavior and collisions. Also, the operator has the possibility to see the tool path in regions where he normally would not see it (e.g. behind the work piece). During the process, it should be recorded whether the tool is moved through air or material. This can be evaluated in three possible ways: by a material removal simulation, optical through camera images or by evaluating the power signals of spindle and feed drives. The results of these recordings can be used to improve the machine program in regard of paths, which are programmed as material removal (G1) but are running through air and can therefore be covered at rapid traverse (G0). To visualize tool paths before running the real machine, a simulation of the machine is needed. This could also help to detect possible collisions in advance and prevent damage.

1.3. Evaluation of process results In the field of machine tools, as in production in general, close tolerances have to be kept. Small violations against this tolerances are often not easily visible to the eye, but can be observed with sensors. Also, in the field of laser cutting machines, the quality of laser cuts can be evaluated based on sensor signals. The main disadvantage today is the missing reference to the position of a detected failure. To improve this, iwindow could augment a work piece with information about the process results. As we regard both laser cutting machines and traditional machine tools such as milling, this use case is separated in two sub scenarios. The first scenario should validate the quality of the cut in laser cutting machines. In the second scenario a vibration sensor should be used to draw conclusions about the surface quality of the work piece in milling machines. In both scenarios, the sensor signals need to be evaluated in real-time and the results graphically presented to the operator. This could probably look like the illustration in figure 2. If a direct reference to the machine program can be generated, the operator has the possibility to correct the associated lines of code. Figure 2: Visualization of quality of laser cuts. Source: TRUMPF 1.4. Operating the machine on the machine window To reduce the amount of additional supporting systems, which could overwhelm the operator, these systems need to be merged on the iwindow. Context-sensitive data for information and operation should be displayed. During process this should be for example the remaining time, tool life warnings, state of machine aggregates (e.g. cooling lubricant pressure and temperature), state of machine periphery (e.g. feed drive temperature) and NC/PLC errors. Also, it should at least be possible to control the following mechanisms to operate the machine and the iwindow directly on the window: open and close the machine door, manual movement and positioning of feed drives, control the spindle clamping, changing and zooming the perspective, managing added value services and controlling the simulation. All operating actions shall be supported by control paradigms known from the consumer sector as for example gesture control and multi-touch control. The biggest challenge here is to find a proper control concept to facilitate the operator s tasks.

1.5. Machine simulation In this use case, behavior and movements of machine axes and periphery will be simulated either by a real-time online, a predictive online or an offline simulation. Both online simulations are connected to the machine control. The real-time simulation simulates detailed actual values for axes and periphery, based on the controls target values during operation. Through this simulation additional process data for diagnosis and prediction, not available in the machine control, can be provided to other components and added value services. This could for example be power consumption data or a material removal simulation. The predictive simulation uses look-ahead data from the control to predict collisions and stop the control before any damage could occur. It is intended to always run this simulations parallel to the real machine components. The offline simulation provides the possibility of a dry run of machine programs in advance of testing them on the real machine. The aim of a dry run of control programs is to detect possible collisions, errors und problems early and without damage. The offline simulation could be executed on the iwindow as well as on a standard computer and will use a software-based control. It will execute much faster than the real process to save development time for machine programmers. A structural overview of the interaction of machine control, simulations, real components and added value services is given in figure 3. Figure 3: Structural overview of the interaction of machine, simulation and other iwindow components 4. SYSTEM OVERVIEW The intelligent machine window iwindow will merge real and virtual world by augmenting the machine interior with computer generated content. Therefor we defined three information layers, shown in figure 4.

The first layer is represented by the machine interior. To allow the observation of the machine interior, especially machine tool and work piece, at any time, the machine interior must be visualized and enriched with further context-related information using augmented reality. A traditional machine window will be replaced by either an ordinary opaque or a semi-transparent screen of the same size. We will evaluate and compare both types of displays. In case of a semi-transparent screen, the machine interior will be visible directly. By using an opaque screen, ordinary computer displays or TV screens can be used. Because in this case the machine interior is not directly visible, it will be observed by cameras, reconstructed in 3D and displayed on the screen. But, as said before, because of cooling lubricant mist, dust and chippings, it is often not possible to observe the tool and the work piece during the process. To provide an alternative representation during such processes a real-time computer model will be used. To give the operator an immersive feeling, head-tracking will be used for both visualization methods to provide a correct perspective view. The second layer can be used by added value services to enrich the machine interior with augmented reality objects. Objects in this layer can be created, modified, moved and deleted. These objects can be both three-dimensional and two-dimensional. Also, they can be placed at a position either in the three-dimensional interior space or in the twodimensional screen space. The third layer is the HMI layer. The HMI will be similar to a state-of-the-art HMI but extended by functionality to control the iwindow itself. The window should be equipped with a multi touch input for the operator to interact with it. Figure 4: Information layers Corresponding to these information layers, three main software components can be defined. A rough system overview is presented in figure 5. The first component is the visualization of the machine interior. It uses camera data as input to reconstruct a three-dimensional representation of the machine interior. If instead the computer model is used, it will be animated according to the current machine state based on control data. The second component is in fact a variety of different components. Added value services can be added to the iwindow, in the same manner as apps on smartphones. Therefor interfaces for in- and output data, such as control data, operator inputs and image data, will be defined. Added value services can be installed, activated, deactivated and

uninstalled at any time. The third main component is the HMI. Additionally to the functionality of a state-of-the-art HMI, it is capable of controlling the iwindow itself. This includes for example switching between visualization via camera data and computer model and managing and controlling added value services. Additionally, to provide these main components access to control data, a control connection component is needed. Also, the augmented reality visualizer will merge all graphical content created by the visualization, the added value services and the HMI into image data to display on the screen. As said before, head-tracking data will be used by this component to provide a correct perspective view. Figure 5: Rough system overview 5. DEMO To evaluate our concepts, such as the visualization by a real-time computer model, a small demo was set up. Beckhoff TwinCAT is used as machine control, because it is running on a standard computer. For visualization a simplified CAD model of the machine is loaded into the game engine Unity, see figure 6. Figure 6: Machine visualization in game engine Unity

To animate the machine correctly, a connection between machine control and visualization was established via ADS-Protocol (Automation Device Specification). The visualization periodically requests actual axis positions from the control and translates the corresponding model parts to represent the correct actual machine state. To provide a correct perspective view, even if the operator is moving in front of the screen, a Microsoft Kinect evaluates the users head position and sends a position vector to the visualization. 6. CONCLUSION To replace traditional windows on machine tools, we introduced the concept of the intelligent machine window iwindow. Recent research and publications to augmented reality in production lack the ability to provide added value to the operator and to reduce the amount of different system he needs to use. To provide enhanced functionality and added value to the operator we deduced use cases for iwindow. Based on this use cases, a rough system overview was designed. To evaluate the concept a first demo, consisting of machine control, visualization and head-tracking, was set up. Acknowledgements The authors are grateful to the Federal Ministry of Education and Research (BMBF) and German Aerospace Center (DLR) for funding the presented work in the project 'iwindow' under the funding code: 01IM14003c. REFERENCES [Ma et al., 2011] Ma, D.; Gausemeier, J.; Fan, X.; Grafe, M.: Virtual Reality & Augmented Reality in Industry. 2nd Sino-German Workshop. Springer, Heidelberg. [Olwal et al., 2008] Olwal, A.; Gustafsson, J.; Lindfors, C.: Spatial augmented reality on industrial CNC-machines; In: Proceedings of SPIE 2008 Electronic Imaging, 6804. [Ong et al., 2008] Ong, S.K.; Yuan, M.L.; Nee, A.Y.C.: Augmented reality applications in manufacturing: a survey; In: International Journal of Production Research 46, 10, pp. 2707 2742. [van Krevelen and Poelman, 2010] van Krevelen, D.W.F.; Poelman, R.: A Survey of Augmented Reality Technologies, Applications and Limitations; In: The International Journal of Virtual Reality 9(2), 1. [Zhang et al., 2006] Zhang, J.; Ong, S.K.; Nee, A.Y.C.: A Volumetric Model-Based CNC Simulation and Monitoring System in Augmented Environments; In: International Conference on Cyberworlds 2006, pp. 33 42. [Zhang et al., 2010] Zhang, J.; Ong, S.K.; Nee, A.Y.C.: Development of an AR system achieving in situ machining simulation on a 3-axis CNC machine; In: Computer Animation and Virtual Worlds 21, 2, pp. 103 115.