Geo-Monitoring By High-Resolution Optical Sensors

Transcription

1 1 Geo-Monitoring By High-Resolution Optical Sensors Andreas WAGNER, Efstratios STYLIANIDIS, Konstantinos SMAGAS, Jiri TRDLICKA, Gerhard PAAR, Ben HUBER, Christoph REITH, Alexander REITERER Abstract In the international research project DE-MONTES 1, which is funded by the European Commission under the 7th Framework Programme (FP7), a fully-automated high-resolution measurement and analysis system for geo-monitoring is being developed. The system is implemented and tested by the use of concrete applications posed by four participating small and medium-sized enterprises (SMEs). Thus, a subsequent integration into existing industrial systems is ensured. The development is realised primarily by three research institutions operating in the fields of geodesy, photogrammetry, computer vision and software development. The paper describes the main strategies followed during the project, technical implications over various fields of expertise, an overview on the scientific and technical approach, as well as the results achieved so far. 1 Introduction The necessity for monitoring geo-risk areas such as rock slides is growing due to the increasing probability of such events caused by environmental change. Europe is leader in surveying technology, due to well-established providers of measurement systems and frameworks. In Europe, rock slides cause increasing damage particularly in alpine areas. DE-MONTES (Deformation Monitoring by High Resolution Terrestrial Long Range Sensing) provides an efficient, highly automated, high-resolution, terrestrial, long range sensing measurement and analysis system which is able to monitor geo-risk (and related) objects by means of non-signalized natural target points - which is a key to such systems due to the lack of accessibility and the required distance-to-the-object. Even novel sensor concepts such as terrestrial laser scanning can only cover a subset of the requirements (cf. WUNDERLICH ET AL. 2010). They limit the distance of application to 1-2 km and do not provide the ability to track individual surface points in high resolution. This would be an important feature to detect regions of motion early enough for taking measures of protection, warning inhabitants, closing infrastructure or evacuation. It is mostly SMEs that run and offer services in this area, reacting with a case-by-case strategy, using conventional technology. There is no well-established market yet on this family of applications, although the occurrence of dozens of events per year indicates that a mature observation system such as DE-MONTES is overdue. With the help of the project research partners, DE-MONTES project is building and testing a productive prototype system for short-term exploitation of the SMEs involved. After project termination SMEs immediately can use the system in their service portfolio, gain new business fields by exporting the system to related application areas and can also use components 1

2 2 A. Wagner et al. of the system in related in-house R&D such as construction, archaeological site survey, industrial inspection or geo-monitoring. The paper gives an overview of the project including its consortium (Section 2) and contractual framework, the requirements of SMEs (Section 3) as well as the measurement and analysis concept (Section 4 & 5). Applications and preliminary results (Section 6) demonstrate the system s functionalities and a conclusion (Section 7) completes the paper. 2 DE-MONTES project overview 2.1 Funding Platform: Research for the benefit of SMEs DE-MONTES is co-funded by the European Commission under the Research for SMEs (R4SMES 2 ) funding scheme for the period This scheme is a bottom-up scheme: the co-funded projects may address any research topic across the entire field of science and technology. R4SMES is dedicated to support small groups of innovative SMEs in solving technological problems and acquiring technological know-how using RTD (Research and Technological Development) performers. In fact, this programme is supporting SMEs to outsource research, increase their research efforts, extend their networks, exploit research results better and acquire technological know-how, bridging the gap between research and innovation. The SMEs are the direct beneficiaries of the project and SMEs are outsourcing (subcontracting) most of the project s research and demonstration activities to RTD performers. In return, they receive the technological know-how in order to develop new or improve existing products, systems, processes or services. The participating SMEs and the participating RTD performers under R4SMES programme are connected via a customer-seller relationship. Actually, the SMEs are buying knowledge from RTD performers, who sell their expertise and work. R&D activities undertaken by the SMEs themselves investing their own resources are essentially focussed on initial specifications and, later, on validation and testing of the acquired knowledge. The intellectual property rights and knowledge developed during the project belong to the SMEs. 2.2 DE-MONTES goals The main objective of DE-MONTES is to provide the participating SMEs with a new kind of measurement system for long-range terrestrial deformation measurements, mainly based on high precision visual measurements via Image Assisted Total Stations (IATS) which can be combined with comprehensive lower resolution Terrestrial Laser Scanners (TLS) and Digital Photogrammetry. The main application field of the system is natural geo-hazard monitoring such as rock falls, landslides and other natural disasters caused mainly from mass movements (PAAR ET AL. 2012). Based on the individual core business of the participating SMEs the results are also 2

3 Geo-monitoring by high-resolution optical sensors 3 aiming to support monitoring and reconstruction of man-made objects, e.g. tunnels, dams, bridges etc., as well as of cultural heritage objects. The main DE-MONTES strategies are as follows: The strategic & marketing approach of the DE-MONTES SMEs is to exploit new sensor technology for the growing field of geo-risk monitoring, setting a new standard and generating a completely new market. IATS, TLS and Digital Photogrammetry are to be used as available technologies by the RTD performers. The existing technologies will be enhanced properly and integrated into a prototype system for geo-risk monitoring but also for other applications which is available for commercial use by the DE-MONTES SMEs. Proper point identification & matching, sensor calibration, measurement correction and orientation, vision data fusion and vision systems integration are key features. 2.3 Project partners The DE-MONTES project consists of seven partners (cf. Figure 1) coming from five different European countries, namely Austria, Cyprus, Czech Republic, Germany and Norway. In the following we will shortly introduce the involved four SME and three scientific companies: Figure 1: DE-MONTES logo and project partners Cautus Geo AS (CAU), located in Norway, is a company focusing on development, installation and maintenance of different survey systems for geo-monitoring, automatic monitoring of stability, deformation and the environment. The company is involved in the whole process from early project/system design to system maintenance and early warning. Cautus Geo has many years of experience from complex geo-monitoring systems, mapping and surveying, geotechnical and hydrological sensors, GIS, data management with different presentations and development of hardware and software solutions.

4 4 A. Wagner et al. Dibit Messtechnik Austria Inc. (DMG), located in Innsbruck, is a worldwide operating engineering office, which provides surveying services, equipment renting and customized Dibit solutions. For data acquisition photogrammetry, state-of-the-art scanner technologies ("stopand-go" and kinematic laser scanner, hybrid systems) are in use. Their own research and development department optimizes and improves these solutions constantly, so projects are able to be carried out with state-of-the-art technologies. Renowned clients assign Dibit Messtechnik Austria Inc. and associated Dibit Measuring Technique USA Inc. for various projects within Europe, Asia and USA. GeoImaging Ltd (GEO) from Cyprus offers scientific and professional solutions in Geoinformatics and IT. It is also active in Research and Development (R&D) programmes, in EU & national funded projects and provides consultations services as well. The products and services span state of the art knowledge and technology from spatial/image information acquisition, processing, analysis to representation and visualisation. The area of projects that GeoImaging is acting concerns informatics in the broader extent, i.e. Photogrammetry, Imaging, 3D modelling, GIS, IT/Software Development & Databases, Cartography, Remote Sensing, Surveying and GPS. Neovision s.r.o. (NEO) is a Czech company which offers research, custom development, design, implementation, installation and solutions in the areas of optical quality assurance/inspection, precise online 2-D and 3-D non-contact measurement, image processing and visualization, robot navigation and software development. Neovision s.r.o. has experience in development, installation and service of turn-key systems such as industrial stations, robotics systems, vision measurement or image processing components. Their customers can be found among manufacturers of digital cameras, electronic components, jewellery, steel, textile industry and among universities and research institutes. The Centre for Machine Perception at the Department of Cybernetics of the Czech Technical University (CTU) in Prague, Czech Republic, is a computer vision, pattern recognition and mathematics of uncertainty research unit. The expertise of the CMP includes fundamentals as well as applications of computer vision, pattern recognition and machine learning including calibrating multi-camera optical systems, stereo reconstruction, omnidirectional and panoramic cameras based on mirrors and lenses, image matching and object recognition in images. Joanneum Research GmbH (JR) situated in Graz, Austria, is co-ordinator of the DE-MON- TES project, fulfilling their role as link between research and industry. The Research Group for Remote Sensing and Geoinformation has broad experience in the evaluation and presentation of data from remote sensing and terrestrial monitoring systems. The range extends from optical, thermal and video data through to laser scan and SAR data. Solutions and products are offered as value-added 2D/3D data products for use in environmental monitoring, safety and security surveillance, mobile data acquisition and tunnel surveying as well as for robotic systems used in extra-terrestrial applications. The research services are complemented by expertise in prototype development, project management and consulting. The Chair of Geodesy of the Technische Universität München (TUM) in Germany is a key player in the field of image-assisted measurement systems, their calibration and application development. The institute is mainly involved in research activities concerning the application of geodesy to engineering and other natural sciences. Typical fields of research are: cal-

5 Geo-monitoring by high-resolution optical sensors 5 ibration and instrument testing, metrology, quality management, structural monitoring, deformation measurements, monitoring of alpine mass movements, video-tachometry and terrestrial laser scanning. The research group cooperates on an international level with research institutes, governmental offices and industry. 2.4 Tools Since the project consortium is distributed throughout Europe, in addition to meetings and video conferences various project tools are used to ensure the best possible cooperation. The most important of these are briefly described below. Content Management System (CMS) Within the project an internal document store (CMS) has been made available. For this purpose a www-server based on Plone 3 is used. It allows all project partners to download, archive and exchange project related data during the whole project duration. This project related internal communication platform is only accessible to members of the project consortium. Data exchange In order to facilitate consortium needs to exchange large data-sets, an FTP server was implemented. Data is stored in a predefined and documented structure. Uploaded data can be cross referenced to the CMS. Issue tracker Experience shows that open issues occur during the project work and it is very important to keep track of them and to make them accessible for all partners involved in this task. For a proper handling of open issues an Issue Tracking system was implemented for DE-MON- TES. Open issues are regularly revealed during Project Management Board (PMB) Meetings. These are in the operational phase of the project mostly software development issues. The action list of each PMB Meeting contains a list of open issues which are also transferred to the issue tracker to ensure a single entry point. The Issue Tracker implements a natural work flow of responsibility timeline priority status paradigm and is self-explanatory. Each DE-MONTES partner has been granted access via a password protected account. Jenkins One of the main challenges in software engineering with multiple programmers is to prevent integration problems when merging all developer workspaces. In DE-MONTES an opensource continuous integration (CI) server Jenkins 4 is used. This software originally developed as the Hudson project enables to build and test software projects continuously after

6 6 A. Wagner et al. every commit and reports the results to the developers. Whereas the RTDs upload and maintain the source code the SMEs are able to download and test the build libraries and executable files. 3 SMEs requirements The first stage of the project has been dedicated mainly to discuss and define SMEs requirements and expectations. As each of the SMEs is working in a different field of structure monitoring and scanning, it was very important to find a common understanding of their requirements and needs. For example the technical aspects were the scanning distance, the scanning sensors and devices, monitored scenario and data processing, etc. Other important issues were more general requirements and priorities of the SMEs such as system setup time and price, data pre-processing, setting tools, readiness for future technologies, etc. The negotiation has been based mainly on two consortium meetings and on several bilateral meetings mainly with the project coordinator. Considering the project complexity and the number of possible variations, the negotiation lasted approximately three months. The process of the requirements specification has been supervised by the RTDs in terms of problem feasibility and time consumption. Sixteen use cases have been defined during the requirements specification as concrete examples for a better understanding to the SMEs requirements. The use cases describe concrete applications scenarios with their parameters, conditions, workflow and required outputs. Unfortunately, more detailed information cannot be discussed in this paper because of the confidential status of the Requirements & Test Specification Document. 3.1 Requirements Some examples of the most important requirements of the SMEs are given in Table 1: Table 1: Requirements Most important requirements of SMEs Readiness of the system for future technology development such as increasing camera resolution, parallel computing, mosaicking, laser scanners etc. Individual functions are provided as libraries. Components are provided in English language only. Modular structure to enable future adjustments and modifications. Short time to market of system and system components is very important. Newly developed source code should be independent from OS in terms of 3rd party components (e.g. MSFC). Special cases need to be reported / negotiated with SMEs. System final results will be compliable in 64bit environment, Windows 7 / Windows XP. Core functions that are newly developed are implemented in C/ C++. System shall be able to generate (dense) point clouds as output. Image comparisons shall be able to express changes in sub-pixel resolution. IATS Control SW and TLS Control SW shall provide hardware abstraction layers.

7 Geo-monitoring by high-resolution optical sensors 7 System should automatically detect signalized points in the area based on approximate sensor orientations and predefined pattern templates. The system components should be able to operate automatically in the field and be controlled remotely. 3.2 Licensing issue There is a reasonable requirement of the SMEs to get information about all licensing issues of 3rd party components and software before the development. In case of additional licensing cost the trade-off between developments costs, license costs and losing some functionality need to be negotiated. 3.3 Acceptance tests The acceptance criteria and tests have been defined at the beginning of the project. They specify the testing procedures and potential problem solutions as follows: Verification - No tuning by the user required; standard environments - Compilation, execution, successful run and result as expected - Done within 2 weeks Acceptance on SME s own data close to the demo data - Done within 1 Month Acceptance on similar data in industrial environment or real word application - Potential problems are solved within 1 Month or in separate contract 4 Sensor and measurement concept The DE-MONTES project integrates methods and techniques from standard surveying, computer vision, photogrammetry, mechatronics, software engineering and geological sciences. DE-MONTES measurements are based on the use of two synchronized IATS (see Figure 2) to provide high-accuracy 3D point measurements in different epochs and in combination with TLS measurements giving medium-accuracy information over the full area of interest. The IATS concept is based on forward intersection of (automatically detected) corresponding points in the geo-referenced images of both IATS as shown in WAGNER ET AL. (2012). The basic principle of this measuring method was conceived in a former research project i-meas (REITERER ET AL. 2009), while the necessary calibration procedures for IATS systems have been described and implemented according to WASMEIER (2009).

8 8 A. Wagner et al. Figure 2: Technical concept of IATS measurements In addition, oriented image data acquired by traditional digital photogrammetry can be integrated which gives added value in terms of modularity towards a low-cost medium accuracy system while using the same processing modules as for the IATS images. Data fusion with digital photogrammetry is an evident extension as the same processing modules can be used for data captured by the IATS, and for medium-accuracy requirements, they can act as standalone low-cost solution. The DE-MONTES environment consists of several modules as depicted in Figure 3. The Data Acquisition Module is responsible for data gathering, data storage and to some extent monitoring and management of the available devices. A task scheduler assigns data acquisition tasks to the Data Acquisition Module which in turn issues acquisition commands to the appropriate device control module or initiates data import by the Image & 3D Import Module. For long-term access to all gathered data the results are stored into a spatial-temporal data base. The Data Acquisition Module is technically implemented using a client-server architecture and provides unified access to all sensors and imported data. It is accomplished with the help of the DE-MONTES hardware driver concept. This concept is based on the SMEs need to create hardware drivers that can be applied for multiple different sensors. In respect to TLS for example multiple different Laser Scanner models from different manufacturers should be usable from an overlying application without knowing about the device specific implementations underneath or in the best case without even knowing what kind of device is present. In a more general view the SMEs also expressed the wish to control other sensor classes (such as total stations, pan-tilt units or distometers) using the same driver concept. A full implementation of those requirements calls for an abstract, hierarchical, multilayer driver model which was realized for the IATS as well as for TLS control module as shown in Figure 4.

9 Geo-monitoring by high-resolution optical sensors 9 Figure 3: DE-MONTES modules and interfaces between modules 5 Analysis concept After data recording (oriented images as described above), the analysis chain is launched. A first step is 3D target selection by means of computer vision algorithms which have to be robust against changing illumination conditions (cast shadows, direct sunlight, ambient light, etc.). Our first approach was to use the SURF operator (BAY ET AL. 2008) but due to patent constraints it comes into conflict with the requirements of the SMEs (see Section 3.2). Currently different alternatives are under test. With 3D selected targets, 3D coordinates can be calculated by stereo reconstruction or by spatial forward intersection with the IATS angle values. These coordinates can be used to generate a 3D point cloud or/and digital elevation model (DEM). If targets can be tracked through different epochs, 3D deformation vectors can be directly calculated, see also Figure 2.

10 10 A. Wagner et al. Figure 4: DE-MONTES Hierarchical Driver Model With exception of the traditional digital photogrammetry method, stable points outside of the monitoring area can be measured by standard theodolite or TLS methods. These points are used for classical deformation analysis and information to statistical significance can be calculated as e.g. described in NIEMEIER (1985). Based on this data a reliable alerting system could be developed. This is an issue beyond the DE-MONTES project scope, but still being of SMEs interest. As another result the volumetric change will be calculated. The whole area can be split into different tiles and for each segment the deformation is described by transformation parameters. The Iterative Closest Point (ICP) algorithm is therefore used on the point clouds, starting with initial values of corresponding tracked points within the particular tile. Visualisation is also part of the analysis chain. Based on the surface reconstruction (DEM) the images can be mapped and an orthophoto is generated. The representation of deformation will be in form of displacement vectors, colour coded point clouds and a combined method of both ways, similar to RAUTEK ET AL. (2006). 6 First results At current project stage the middle of the project the exact requirements and use cases of the SMEs are defined. The control module for the laser scanners and the IATS devices (including graphical user interfaces - GUI) are developed and demonstrated to the SMEs in a project meeting. Also first tests of the processing chain are performed. Due to the extensive development of the above mentioned software modules and the required calibration of the IATS, only analysis examples of the classic digital photogrammetry data can be shown. A meeting of all RTD partners to test devices, control modules and the processing chain will take place end of February 2013 in Graz.

11 Geo-monitoring by high-resolution optical sensors 11 Figure 5 illustrates a reconstruction of a rock wall in Norway taken from DE-MONTES SME partner Cautus. In this case, fourteen images from different locations along the base of the wall where imported and processed (three of these images is shown in Figure 5, top left). With the help of different software modules, e.g. JANCOSEK & PAJDLA (2011), bundled in one library the images have been matched (top right), a surface mesh (bottom left) was generated, textured and also output as orthophoto (bottom right). The results are also delivered as VRML (Virtual Reality Modelling Language) files and can be displayed in an interactive 3D viewer. Figure 5: Reconstruction of Rock Wall by means of photogrammetry, Norway 7 Summary and Conclusion The paper presents the European research project DE-MONTES which aims to provide the participating SMEs with a new kind of measurement system for long-range terrestrial deformation measurements as well as reconstruction software of man-made objects. The project including the founding, the goals and the partners were presented together with some aspects of technical development and the underlying strategies. In order to ensure the most efficient processing various project control tools are used and discussed as well. The measuring and analysis concept aims to gather 3D points from 2D measurements (images) with high resolution. The implementation will be delivered to the participating SMEs as libraries and executable files which will be able to control different optical sensor devices, automatically measure different epochs of predefined monitoring areas and calculate deformation vectors, volume changes and graphical output to fulfil the requirements of the SMEs measuring tasks. Through the modular concept it is possible to incorporate the whole software structure as well as only parts of the software into existing in-house SMEs solutions. Furthermore, the later addition of extra sensors (e.g. GBInSAR) is thus possible.

12 12 A. Wagner et al. Acknowledgement The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/ ) under grant agreement n DE- MONTES. Bibliography Bay H., Ess A., Tuytelaars T., Van Gool L. (2008): SURF: Speeded Up Robust Features, Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp Jancosek M., Pajdla T. (2011): Multi-View Reconstruction Preserving Weakly-Supported Surfaces. In IEEE Conference on Computer Vision and Pattern Recognition pp Niemeier, W. (1985): Deformationsanalyse. In: Pelzer H. (ed.): Geodätische Netze in Landes- und Ingenieurvermessung II. Konrad Wittwer, Stuttgart, pp Paar G., Huber N-B., Bauer A., Avian M., Reiterer A. (2012): Vision-Based Terrestrial Surface Monitoring. In Pradhan B., Buchroithner M. (eds.) Terrigenous Mass Movements. DOI / Springer Berlin-Heidelberg, pp Rautek P., Reiterer A., Gröller M. (2007): Caricaturistic Visualization of Deformation Data Based on High Density Point Clouds. In: "Optical 3-D Measurement Techniques VIII", A. Grün, H. Kahmen (eds); Volume II, pp Reiterer A., Lehmann M., Miljanovic M., Ali H., Paar G., Egly U., Eiter T., Kahmen H. (2009): A 3D Optical Deformation Measurement System Supported by Knowledge-Based and Learning Techniques. Journal of Applied Geodesy, Vol. 3, pp Wagner A., Reiterer A., Wasmeier P., Rieke-Zapp D., Wunderlich Th. (2012): Vision-Based Geo-Monitoring A New Approach For An Automated System. EGU General Assembly 2012, Vienna (conference poster). Wasmeier P. (2009): Grundlagen der Deformationsbestimmung mit Messdaten bildgebender Tachymeter, PhD Thesis, Technische Universität München Wunderlich Th., Wasmeier P., Reith Ch. (2010): Potentials and Limits of Geodetic Systems for Landslide Monitoring. In: Proceedings of GeoDarmstadt, Congress DGG und GV, Darmstadt.