vis IT international 2006 Digital Situation Table Information Fusion Environmental Applications Quality Monitoring Dynamic Business Processes

Transcription

1 vis IT international 2006 Digital Situation Table Information Fusion Environmental Applications Quality Monitoring Dynamic Business Processes Agent-based Diagnosis ISSN

2 Editorial Notes Content Publisher Prof. Dr.-Ing. Jürgen Beyerer Editor Sibylle Wirth Layout Christine Spalek Sibylle Wirth English Edition Barbara Villringer Harthill Printing Engelhard & Bauer Karlsruhe Editorial Address Fraunhofer-Institut Informations- und Datenverarbeitung IITB Fraunhoferstr Karlsruhe Germany Phone: +49 (0) 7 21 / Fax: +49 (0) 7 21 / presse@iitb.fraunhofer.de Fraunhofer IITB Karlsruhe 2006 Ein Institut der Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e. V. München ISSN Page 4 Page 6 Page 8 Page 10 Page 12 The Digital Situation Table: Unrestricted Overview and Detail Ralf Eck Ontology-based Information Fusion for Interactive Image Rainer Schönbein Information Fusion in Environmental Applications Thomas Usländer Robot-based Quality Monitoring using Imaging Sensors Chen-Ko Sung Basis for Dynamic Business Processes in Production Michael Baumann Photo acknowledgements Title page Ingram Image Library Sibylle Wirth page 14 MEV über PR-Netzwerk der Fraunhofer Gesellschaft page 15 imagelibrary Page 14 adrian: Agent-based Diagnosis in Realtime Industrial Automation Networks Christian Frey Reproduction of any material is subject to editorial authorization. international 2

3 Editorial Dear friends of the IITB, A principal core competence of the Fraunhofer-Institut für Informations- und Datenverarbeitung IITB (Fraunhofer-Institute for Information and Data Processing) is systems theory and systems engineering. This means the ability of analyzing, modeling, synthesizing, implementing and commissioning very complex systems, adding value by exploiting data, information, and knowledge technically. Regularly, such tasks require a broad range of capabilities in different disciplines as well as professional skills concerning information on all abstraction levels. At the IITB we have a well balanced mixture of experts on different scientific fields (computer science, electrical engineering, mechanical engineering, physics, mathematics) and also technical staff for establishing interdisciplinary systems. To provide our partners and customers with profitable solutions for difficult tasks, it is frequently helpful and sometimes indispensable to have long duration stamina beyond the range of the cycle time of doctoral theses. Our recipe for success lies in a sane combination of the liquid creativity of young scientists with crystalline intelligence cumulated in many successful projects. Jürgen Beyerer Additionally, in order to satisfy the needs of our customers in a sustainable manner we need a profound knowledge about the fields of application. It is obvious, that such an expertise cannot be acquired in a day. Here again, staying power is required to finally fuse our primary competences with deep specific knowledge about the application domain into an optimal solution. Professional project and quality management round off our palette of competencies, resulting in high repeat customer rates for the IITB of > 5 orders / customer. The papers in this issue of visit give an idea typical systems the IITB has developed to solve complex tasks according to the discussion above. Karlsruhe, June 2006 Jürgen Beyerer 3 international

4 Themes All Information at a Glance The Digital Situation Table: Unrestricted Overview and De Ill. 1: Cooperative decision-making at the digital situation table. CONTACT Dipl.-Geophys. Ralf Eck Interactive Analysis and Diagnosis Fraunhofer IITB Karlsruhe Phone: 07 21/ ralf.eck@iitb.fraunhofer.de Map-based searches and navigation are integral components of environmental information systems. The information stored in these systems is mainly spatial, and interpretation of the information often requires visualizing it in a geographic context, e.g., topographic maps or digital environmental models. The different visual displays of the relevant data bases must be interpreted cooperatively by a variety of people under great time pressure. Using an integrated cartographic representation, various areas of interest are chosen and compiled into variable visual displays leading to a unified decision-making process. Both continuously-available overviews as well as highresolution displays are essential in this process. The current desktop-oriented, monitorbased work stations provide the required detail resolution, but they impede the overview and are not suitable for cooperative work by larger teams. Large-screen monitors and projectors, on the other hand, allow the team the necessary overview but the detail resolution suffers. Visualization System A prototype digital situation table was created which can, for example, be applied to facilitate the situational evaluation of major disasters. A team of experts can thus have not only a largescale overview of the situation but also can focus in as necessary on a detailed view. This view is provided by Fovea- Tabletts which permit individualized interaction with the Geographic Information System and which control location-specific secondary Information. international 4

5 tail Ill. 2: Fovea-Tablett with high resolution and MC-MXT marker for determination of position. The digital situation table is a work station with a horizontal table-like work surface and a vertical monitor-like display screen. The horizontal component serves as a map table and work space for an overview of the situation using back projection technology; the vertical monitor allows visualization of side views. The local resolution on the horizontal work space is suitable only for an overview presentation because of its low pixel density. To compensate for this, the Fovea-Tablett (FT) technology developed at the IITB is used. One or several small, portable display units such as a tablet PC with high local resolution are simply placed on the map table. A measuring device determines its position and rotation with regard to the map table. These data are then transmitted (wireless) to the FT which displays the view in such a way that the viewers have the impression of looking through the FT at the overall map, but with a much higher resolution at that particular position. The FTs can be moved around the work table at will. Each expert in the team can use his/her own FT freely and quickly focus on a specific area of interest. Fovea Tablet Tracking Each FT is marked underneath with an identification mark and an integrated number code. A camera underneath the digital situation table tracks the marks of all the FTs on the surface of the table. The number code identifies the FT, and the identification mark enables an exact position measurement. To avoid disturbances, the measuring camera operates within the near infrared. The Multi-Cursor-MarkerXtrackT (MC-MXT) process developed at the IITB is used for identification and scanning processes. The MC-MXT-Tracker-Server works in an independent camera-pc unit as an embedded system with TCP/IP socket interface with XML-protocol. Interactive operation of the system during an ongoing operation is not necessary. The image is saved on the FT and the relevant view is calculated and displayed. If data management is to be restricted to the server, transmission to the FT must be cable-free, and, more importantly, at a high transmission rate. Areas of application include border and ground monitoring, regional and city development, disaster management, environmental monitoring, and also visualization in production control systems, to name just a few. 5 international

6 Themes All Data at a Glance Ontology-based Information Fusion for Interactive Image and reference information from various sources, a unified presentation must be achieved through the appropriate fusion of data and information. Data Fusion in Time and Space CONTACT Dr. Rainer Schönbein Interoperability and Assistance Systems Fraunhofer IITB Karlsruhe Phone: 07 21/ rainer.schoenbein@iitb.fraunhofer.de Ill. 1: Space and time-related data and information fusion. Introduction For interactive image interpretation, user interfaces must handle the various processing phases as well as support the varying roles of the image interpreter. Of vital importance in the preparation of a new task is the ability to comprehensively and comfortably research collateral data and reference information. It is, however, easy for the user to be overwhelmed by the amount of information. To alleviate this problem, adaptation of the user interface to any given research task is crucial. To this end, information models and search mechanisms must be developed which provide not only a high hit accuracy but which also guarantee that no relevant information is overlooked (high rate of hits). To aid in the visualization of the retrieved sensor data, collateral data, The central component in the fusion of data is the image presentation which aids not only in the visualization of the images but also serves as background and reference surface to enter location-specific coordinates, e.g. for spatial queries or for measurement of detected objects (length, width, surface, volume, number, direction, distance, etc.). Another required task of the image presentation is the overlaying of images provided by different sensors, from different time frames, or from different viewing angles. The overlay with maps and models must support both grid and vector graphics. In addition to the space- related data fusion using a uniform coordinate system, time-related information categorization is fundamental. The space and time-related data fusion provides the technical basis for semantic information fusion using ontologies. Thematic Information Fusion Based on Ontologies An ontology represents a description of concepts (terms) and relationships within a certain domain which the user can share with a software system and other users. Ontologies combine human understanding of symbols with the processing capabilities of machines. With regard to space and time-related information, corresponding concepts and relations must be implemented international 6

7 Interpretation Ill. 2: Structure of the ontology including layers and views. in support of varying space and time data. Both absolute terms related to space and time as well as relative terms such as»north of«or»later than«must be taken into consideration. Domain ontologies constitute an essential component in information fusion. Interpretation-relevant objects are modeled according to their properties and combined using user and task-specific contexts (»views«). Illustration 3 represents a section of the»land vehicle«aspect from the domain ontology in the»looks like«view. When using the»looks like«view, the standard view for an image interpreter, an ontology-based search for information on land vehicles in space and time areas of interest, therefore, also presents information on, for example, detected»leopard 2«or»trucks«. This information can then be used for the automatic annotation of the sensor image or the map representation (ill.4). Application The ontology-based information fusion described here is applied as a component of a software agent-based cooperative information system for intelligence and reconnaissance. The ongoing development is part of the»isva«(intelligenter Sensor-Verbund Aufklärung) research project sponsored by the BMVg (Federal Ministry of Defense) and BWB (Federal Office of Defense Technology). The goal of the ISVA ontology is the support of an application to simplify the search for people, information, and services in the area of intelligence, surveillance and reconnaissance. Thematic Fusion of information with space and time variability represents a basic function in interactive image interpretation in this area of application. Ill. 3: Section of the domain model from the area of»and vehicles«in the»looks like«view. Ill. 4: Section of an SAR image (Synthetic Aperture Radar) with vehicles and added annotation. 7 international

8 Themes Illustrative Data Visualisation Facilitates Decisi Information Fusion in Environmental Applications In the field of environmental software applications, information fusion is used in many facets. On the one hand, decision makers in the environmental field require information of varying specificity; on the other hand a meaningful data base even with incomplete measurement data must be made available. There are a multitude of reasons for missing measurement data. Environmental sensors such as water-monitoring locations, radar monitoring stations, on-site cameras or cameras on satellites always measure only one aspect of the environmental phenomena: e.g. optical satellite images can detect snow-covered surfaces but cannot provide information as to the consistency of the snow, which is, however, essential for the prediction of flooding. For this, radar images in combination with land-use maps are required. The costs for installing and operating measurement networks increase proportionally with the density of the monitoring locations. As a result, available data from environmental sensors frequently do not have sufficient time and space resolution because of gaps in the measurement network. To make up for this data deficit, generation of additional information using fused data sources open up promising possibilities. Here are a few examples: Aggregation and Classification The implementation of the European Water Framework Directive (WFD) requires extensive work to fuse measurement data. Using chemical, physical, and biological data measured regularly on site as well as the assessment of hydro-morphology structure, a statistical aggregation of measurement values leads to a classification of natural water bodies into quality classes. The goal of the directive is to achieve comparable»good«water quality for all water bodies by To make a comparison Baden-Württemberg nitrate classification CONTACT Dipl.-Inform. Thomas Usländer Information Management Fraunhofer IITB Karlsruhe Phone: 07 21/ thomas.uslaender@iitb.fraunhofer.de Landuse Geology Ill. 1: Generation of a nitrate map for groundwater in Baden-Württemberg using SIMIK+. international 8

9 ons terrain model 30 x 30 Meter possible, thematic data fusion processes which have been agreed on by all member states are necessary. land use 30 x 30 Meter WFD information systems such as the product suite WaterFrame of the Fraunhofer IITB, already in use for the implementation of the WFD in Baden- Württemberg, Thuringia and Bavaria, support these processes with suitable data models as well as integrated selection and interpretation services across all thematic domains. (water) storage capacity of the soils digital river network, positions of the water gauge and climate stations Geostatistical Interpolation Description of groundwater quality is usually based on point measurements taken at selected sites. To get an overview of the quality, regionalized maps for each individual quality parameter may be created using the interpolation process SIMIK+ (Simple Updating and Indicator Kriging based on Additional Information). A special feature of SIMIK+ is that it takes into account land use and geology of the ground water aquifers as additional information for the determination of the sphere of influence of the existing point measurement values. This form of information fusion was used by the Environmental State Agency of Baden-Württemberg (LUBW) to generate a groundwater quality atlas of Baden-Württemberg, providing a statewide overview of the ground-water quality for 55 chemical parameters. LARSIM Model For the entire area of Baden-Württemberg (approx km2), water balance models exist which are based on networking of the grid planes Ill. 3: Fusion of several information levels in the LARSIM model. the program system LARSIM (Large Area Runoff Simulation Model, M. Bremicker). The model takes into consideration climate data, evaporation, snow cover, snow property and snowmelt information as well as ground water storage. LARSIM calculates detailed spatial and temporal data regarding soil moisture and run-off; e.g. it is used to predict flooding for the Flood Forecasting Centre of the LUBW. Time-based Models The EU project i-marq has developed a fusion and modeling engine for biological and chemical processes in sea water. Water quality is evaluated on the basis of the interdependent parameters of turbidity and concentrations of oxygen, nitrate, plankton or e.coli bacteria, respectively. These quality measurements are taken selectively at intervals on the one hand, but they are also extrapolated in space and time using stochastic modeling and a Kalman filter. In this way, predictions as to water quality under different conditions are generated (e.g. water run-off). Over the next view years, IITB plans to work systematically in the area of information fusion and its application in environmental information systems. 9 international

10 Themes Robot-based Quality Monitoring Using picture-giving Sensors In industrial production, two significant trends can currently be observed, both favor the use of robots equipped with intelligent picture-giving sensors. On one hand, product series are increasingly characterized by frequent changes in type of product, shorter product cycles, and a smaller number of components, making classic automation concepts with their multitude of sensorless, rigidly-programmed industrial robots too inflexible. On the other hand, there has been a steep drop in prices for high-quality intelligent picture-giving sensors, enabling completely new, flexible robot sensor applications in production at reasonable cost. As a result of this development, robots with multiple, intelligent sensors are increasingly used for demanding production and Fehler Prüfzonen CONTACT Dipl.-Ing. Chen-Ko Sung Autonomous Systems and Machine Vision Fraunhofer IITB Karlsruhe Phone: 07 21/ chen-ko.sung@iitb.fraunhofer.de Ill. 1: Short-range and wide-range testing of fuselages. international 10

11 assembly tasks. An especially attractive area of application is the inline quality moni-toring of complex, large-area production parts such as the fuselage of aircraft or parts of bodies of road and rail vehicles (ill. 1). As part of the ROBOSENS project, the IITB developed and tested a new threestep concept for multiple sensor quality monitoring. The robot-based system uses an array of test-specific short-range and wide-range sensors which make the inspection process more flexible and problem-specific. In the first step, the exact position of a production piece is determined by wide-range picturegiving sensors. (Movement of production pieces is carried out by monorail conveyors which do not allow precise positioning). In the same step, problem zones are defined for detailed inspection. In the second step, an up-dated inspection flow chart is generated, which is then carried out using various inspection-specific short-range sensors (e.g. cameras, feelers, etc.). Ill. 2: Development and demonstration platform. In the third step, the robot uses those sensors which are necessary for a given inspection flow chart and guides them along an optimal motion trajectory into the previously-identified areas for detailed inspection. In these areas a qualitative comparison of the observed actual topography with the modeled target topography is made using intelligent image-processing methods. In addition, quantitative scanning and measurement of selected production parameters can be carried out. For a fuselage, for example, it can be determined if construction elements are missing and/or if certain bore diameters are true to size. An intelligent, sensor-based distance-control concept (Visual-Servoing-Principle) accurately controls the robot s movement with regard to the part and prevents possible collisions with unexpected obstacles. For the navigation and position control of the robotic movement with regard to the imprecisely-guided production object as well as for the comparison of the observed actual topography with the target topography, reference models are required. These models are scanned prior to the automated inspection flow chart using suitable short-range and wide-range sensors (e.g. 3-D split-beam sensors). By using the proposed, robotbased concepts of multiple sensor quality monitoring, the customary use of expensive 3-D-CAD models of the test objects for high-precision CNC controlled machine tools or coordinate inspection machines becomes, for the most part, unnecessary. To test this innovative inline qualitymonitoring concept and to adapt it to customized tasks, a development and demonstration platform was created. It consists of an industrial robot with various sensor ports a so-called»sensor train station«with various task-specific, interchangeable sensors and a flexible transport system (ill.2). The primary goal of the development and demonstration platform is to investigate, optimize and demonstrate to potential cooperation partners how the system can be applied to reduce effort and increase flexibility. For example, it can be used in the robot-based coordination of short and wide-range monitoring, for the introduction of learningcapable evaluation processes, as a tool for visualizing results and for user interaction, as well as for the flexible networking and integration of various wide and short-range sensors. 11 international

12 Themes Adaptive Production Control Basis for Dynamic Business Processes in Production Modeling of Business Processes Production enterprises require integrated business processes, which operate from the business level to the machine level, to react instantly to market changes or changes in client needs. In the past, the field of IT was characterized by a multitude of systems, which were often islands unto themselves. Production-related IT systems, in particular, are highly heterogeneous, and many client-specific solutions exist. Each of these solutions has its own technology, which is usually not adaptable to systems of other suppliers. This problem could be solved and integration of software systems of different manufacturers could be simplified through a unified, enterprise-spanning modeling of production sites and processes. Based on this need, the American National Standards Institute (ANSI) and the Instrumentation, Systems and Automation Society (ISA) created the ISA S95 Initiative in The ISA S95 provides a functional description of an enterprise and has become the de facto standard for enterprise control systems. It provides data models and terminology for a general, unified description of production systems and processes. This modeling can be used throughout the entire enterprise by different software systems, accelerating and simplifying the introduction of new systems. The focus of the ISA S95 is on vertical integration of systems for production management, now often referred to as MES systems (Manufacturing Execution System) and the overlying ERP systems (Enterprise Resource Planning). CONTACT Dr.-Ing. Michael Baumann Control systems Fraunhofer IITB Karlsruhe Phone: 07 21/ michael.baumann@iitb.fraunhofer.de Ill. 1: Vertical integration of the enterprise levels. international 12

13 requires a person belonging to class»welder«with the properties»class-1- certified«and at a»training level > apprentice««. MES Control Loop An essential task at the MES level consists of detailed planning and production control based on the actual production process. In this control loop, detailed planning as the final controling element plays an essential role in production control. Detailed planning represents a very complex optimization problem and depends to a large degree on enterprise-specific conditions. The currentlyavailable algorithms to solve detailed planning tasks are characterized by problem-inherent knowledge. If, for example, a new condition is added to the planning task, the algorithm has to either be adjusted with great effort or a completely new one must be developed. The effort, and, therefore, cost to the enterprise is enormous a model to describe planning problems. The ISA S95 can serve as the basis for such a standardized detailed planning function. The models must be flexible so that the detailed planning can be adapted to changes in products and processes through formulation of constraints, for example, without the need for costly new or changed programming of the planning algorithms. Examples of such constraints include universal definitions of dependencies such as»start process B after process A is finished«, or through a universal resource specification as in»process A Based on this constraint-driven modeling of the planning problem, the detailed planning determines the optimal time frames for the carrying out of the various jobs, taking into consideration all constraints in the utilization of resources and dependence on machines, workers, production facilities, and materials. Such standardized detailed planning provides the user with the following advantages: Simplified planning Reduction in planning time Short-term reaction to deviations and failures Optimization of the production process Unified modeling without breaks and, therefore, shorter start-up times. Standardized Detailed Planning The challenge is to create a universally applicable optimization function which can be flexibly adapted to different enterprises and frameworks by means of Ill. 2: MES control loop for reactive production control. 13 international

14 Themes Zero Downtime Zero downtime automation systems require a continuous, system-wide comprehensive diagnosis of the physical process and communication network instead of selective on-demand diagnosis as practiced at present. Within the joint project ADRIAN, funded by the German Federal Ministry of Education and Research (BMBF, Research Initiative»Software Engineering 2006«), the Fraunhofer IITB and the Softing AG developed a generic and highly flexible software agent-based diagnostic concept for industrial automation networks. Diagnostic Concept CONTACT Dipl.-Ing. Christian Frey Systems for Measurement, Control, and Diagnosis Fraunhofer IITB Karlsruhe Phone: 07 21/ christian.frey@iitb.fraunhofer.de Starting in the mid eighties, the development of industrial Fieldbus communication set the foundation for distributed automation systems. Fieldbus and Industrial Ethernet communication systems are the backbone of distributed automation systems and, therefore, the loss of communication causes a loss of system functionality. The Fieldbus itself offers only low-level diagnostic functionality. If a failure occurs, usually human interaction is required for fault localization and isolation by using specialized tools in conjunction with asset-specific expertise. To establish an integral, continuous diagnostic functionality, the communication network is an ideal access point for retrieving the essential information. The field devices of the automation system are connected over the network and broadcast their messages periodically. By analyzing the messages with respect to the carried I/O-data and the communication characteristics, e.g. response time, compressed diagnostic information of the components can be retrieved. This is utilized for diagnosing the physical behavior of the underlying process and the communicational behavior of the network. international 14

15 Considering the heterogeneous platforms as well as the varying number of network devices in distributed automation systems, a key implementation issue is the effective coordination and cooperation of the distributed software modules as well as the management of their interactions. With respect to these requirements, software agent technology is applied to establish the proposed diagnostic functionality. Diagnostic Agent To acquire the diagnostic information, every device in the automation network is monitored by a so-called diagnostic agent. The task of this software entity is to»just-in-time«pro-actively retrieve, maintain and compress the diagnostic information from the broadcast messages. Particularly with regard to the developed diagnostic concept for Fieldbus-based automation systems, the diagnostic agent transforms quantitative information, e.g. the response time and the I/O-data of the corresponding field device, into a qualitative distribution-based representation. Diagnostic Manager In order to perform an overall diagnosis of the automation system, the socalled Diagnostic Manager periodically queries the compressed spatial diagnostic information brokered by the diagnostic agents. In terms of analyzing the communicational functionality of the network, the distribution-based features are interpreted as fuzzy member functions which are evaluated by a succeeding fuzzy classification module. To diagnose the physical behavior of the underlying process, the developed concept utilizes SOM, a self-organizing, map-based methodology. As a nonlinear projection method for highdimensional data, the SOM is used to induce a data-driven model of the process based on the observed I/Odata without any explicit analytical process knowledge. By analyzing the quantification error of the SOM when presenting a state vector composed of the device s I/O-data, malfunctions of the underlying process are diagnosed. The developed agent-based diagnostic concept was successfully tested in a demonstration plant and is currently applied to several industrial batch processes international

16 Karlsruhe Fraunhofer-Institut für Informations- und Datenverarbeitung IITB Fraunhoferstraße Karlsruhe Phone: +49 (0) 7 21 / Fax: +49 (0) 7 21 / info@iitb.fraunhofer.de Ilmenau Fraunhofer-Anwendungszentrum Systemtechnik AST Am Vogelherd Ilmenau Phone: +49 (0) / Fax: +49 (0) / sabine.bartnik@ast.iitb.fraunhofer.de Beijing Representative for Production and Information Technologies Unit 0610, Landmark Tower 2 8 North Dongsanhuan Road Chaoyang District Beijing, PR China Telefon: Fax: muh@fraunhofer.cn