Augmented Reality as part of a man-machine interface in e-navigation

Transcription

1 Augmented Reality as part of a man-machine interface in e-navigation Stephan Procee 1 and Michael Baldauf 2 1 Maritiem Instituut Willem Barentsz, Terschelling, the Netherlands - partner in the ACCSEAS project ( 2 World Maritime University, MaRiSa Research Group, Malmö, Sweden and Institute of Innovative Ship Simulation and Maritime Systems (ISSIMS), Hochschule Wismar, Germany, Abstract Head-Up Display (HUD) Information on the Bridge: Goggles worn by the mariner showing the visible seascape with an overlaid computer generated image enabling the mariner to increase their situational awareness. This provides visual clues to the mariner such as alerts of impending collision or grounding. For example, a visual cue could be overlaid synthetically on the real-world view to indicate the direction of an imminent collision risk. 1 INTRODUCTION Mariners are traditionally focused on visual recognition and identification of targets. The Collision Regulations are based as well on visual recognition of a target and its relative course and speed. Therefore the strategy and action of the Watch Officer to avoid collision is well trained and, apart from low visibility situations, is always based on visual observation and the Watch Officer s experience. Although much effort is taken to minimize the risk of collision, accidents still happen. Accident investigation shows that human error plays by far the most important role in the cause of accidents (i.a. [5]). Once the Watch Officer is distracted from his task of watch keeping he will no longer effectively react according to the Collision Regulations. Although ARPA/AIS can generate an audible alarm as collision Warning, Mariners who are distracted from their task will have difficulty to identify the dangerous target and cannot timely act in order to avoid collision in the little time between the alarm and the critical CPA. On the other hand, until today there are no commonly agreed thresholds for CPA and TCPA limits to trigger the warnings. Moreover, the provided algorithm to trigger those warnings is insufficient in a way that the thresholds apply to all situations while the OOW applies different criteria for different encounter situations. Augmented Reality can act as an Expert Support system. The principle structure and functional layout of such a "Marine Navigation Augmented Reality System" (MNARS) are given in figure 1 below. By superposing a virtual image on the outside world the reality is enhanced by visual clues. These clues can represent dangerous targets, however, in the e- Navigation domain the functionality need not be limited to collision avoidance alone. The ACCSEAS project (see A1), in which this Augmented Reality study is embedded, has 1

2 developed a prototype portfolio of services and solutions for mariners and related traffic management professionals. Figure 1: MNARS functional layout Amongst these services are No-Go Area Service (A2) and Tactical Route Exchange and Route Suggestion service (A3), to name just two. Both these services can provide crucial safety related dynamic input to the navigator. When this information is presented to the mariner through a Head Up Display (HUD) showing visual clues in their geo-referenced direction on top of the real world objects, the effectivity of the information processing will be enhanced, thus speeding up the process of decision making and consequential safe manoeuvring. An example is shown in fig.4 in one of the following chapters where the red projected box points to a critical target, and a red projected fence shows the area to be avoided and a route suggestion is shown by the projection of green runway lighting. This projected information is corrected for the ship s attitude and motion and also takes into account the position of the observer in order to avoid parallax errors. In the ACCSEAS project a testbed is set up to do a proof of concept in an existing ships bridge simulator operated by the Maritiem Instituut Willem Barentsz at Terschelling. 2 TESTBED OUTLINE Work has currently started to set up and build an AR application on one of the bridge simulators of MIWB. A sketch of the projected bridge layout can be seen in fig. 2. Initially, an offer from the industry to use one or more transparent monitors for portraying the augmentation information was accepted over the alternative to project information on the bridge window. 2

3 Figure 2: Initial sketch of bridge layout with goggles (port side position) and transparent monitors in a cocoon like configuration (starboard side) The latter option would need a certain percentage of opaqueness of the bridge window which was considered unacceptable because a clear outside view is compulsory. It later turned out that the industry was not ready for large scale production of transparent see-through monitors, so an alternative was found in goggles (see fig 3). Figure 3: Augmented Reality Goggle 3

4 As the projected ACCSEAS services were in the development phase initial focus was put on collision avoidance. So the proof of concept was aimed at portraying AIS targets in the right perspective in the augmented reality goggle. An application was written to parse AIS messages from the coded AIVDM format to ASCII readable text and select relevant information from it like the target s time, position, course and speed. From the time, position course and speed of the own ship the relative movement of the target can be calculated on the basis of the transmitted AIS information and thus providing a calculated CPA and TCPA of the target and a relative position, i.a. direction and distance, to the Watch Officer. This information is projected in the augmented reality goggle initially as a marker and text. Further development aims at a classed colour coding of the marker on the basis of alarm level. Alarm levels can be classified based on CPA and TPA and the sector, relative direction and distance, in which the target is located. The testbed comprises of a Norcontrol ships bridge simulator with an own ship and an area with targets transmitting AIS messages. The stream of AIS messages is read through a COM port by the laptop on which the Augmented Reality application is running. Filtering is done to overcome the discontinuity in time of transmitted AIS messages because all targets send their messages in different time slots and in different rates depending on their status, speed and turn rate. The goggle s position and direction of view is measured by a Tracker. This device consists of a motion sensor and a system that locks at a set of visual markers, i.a. locators, on the bridge. The aim of this tracker is to provide information of the observers view direction in order to calculate the relative position of the projected information, e.g. the red box in which the dangerous target is shown. The functionality of AR is to point directly visually in the direction of the dangerous target, thus induce an immediate focus of the WO on the dangerous target (see fig. 4). Figure 4: Sketch of augmented reality, showing dangerous target, area to be avoided and route suggestion 4

5 3 ALARM LEVELS AND SHIP S SAFETY ZONE On basis of comprehensive task analysis IMO s Performance standards for Integrated Navigation Systems (MSC.252(83) provides a essential navigational tasks that needs to be supported by an INS. One of which is collision avoidance and another is alert management. In global terms alert management shall contribute to harmonization of priorities and classification and is to enhance the handling, distribution and presentation of alerts. IMO defines three levels of alerts first is Caution (lowest level; as a kind of a signal that there is a situation with certain deviation from usual (safe) conditions) secondly Warning (requiring immediate attention of the bridge team) and finally, highest level of an alert Alarm (requiring immediate action to avoid any hazard to navigation. In respect to the task of 'collision avoidance' there is similarity to the different stages of an encounter situation with risk of collision. Cockcroft/Lameijer suggested and discussed those stages in [1] in respect to the obligations of a stand-on vessel in case of an encounter situation of two engine driven vessels on crossing courses according to rule 17. From this basic discussion, a more detailed and sophisticated model for situation assessment has been derived [2]. This model takes into account further rules of COLREGs for other types of encounter situations under conditions of good and restricted visibility. Furthermore this enhanced model concentrates on recommendations for action to be taken derived from COLREGs but also provides suggestions for the quantification of limit values for the situation dependent safe passing distances as a CPA threshold as well as for TCPA limits when to take those actions. Furthermore, taking into account the prevailing circumstances of a specific situation (data of environmental conditions as well as ship dimension and status (especially in respect to the actual manoeuvring characteristics) it is possible to even harmonize the situation-dependent definition of safety zones around a ship (expressed by a harmonized situation dependent CPA) and, in case of a potential violation of it, to prioritize collision alerts into the given levels caution, warning and alarm by prioritizing the targets in respect to the remaining time to take action to avoid a collision. For this purpose the TCAS concept, used for collision avoidance in aviation, can be transferred and adapted [2]. In TCAS i.a. a so called Resolution Advisory requiring the pilot to follow a climb or descent instruction is implemented as a last line of defence. Applying this to collision avoidance in open sea, for instance a target, violating the safety zone of the own ship should generate a collision alarm, when it comes close to the lower manoeuvring limit, at which the own ship by its own manoeuvre alone is able to avoid a collision. This manoeuvre can be determined by using dynamic prediction of the own ship evasive manoeuvre using Fast time simulation [4]. 4 FIRST RESULTS AND PRELIMINARY FINDINGS One of the results is the demonstration of the tracker and AIS superimposed image on the augmented reality goggle. The first test was done in the ACCSEAS project group (see fig. 5) 5

6 where a logfile of AIS messages was projected in the goggle where the observer was located in a classroom without simulator targets present. The meaning of this demonstration was to show the feasibility of the goggle and it s intended use to the ACCSEAS expert group. It was agreed that the augmented reality has enourmous potential for practically every domain, but in the navigation domain where visual reconnaissance is still considered dominant it certainly will. Further findings are that training is needed to get accustomed to the overlay of a dynamic image over the real outside view. Personal observation learned that focusing on the dynamic virtual image was not very difficult when the real background was kept steady, however, focus was immediately shifted to the background when this background got dynamic e.g. by moving your head in another direction. 5 DISCUSSION AMD FURTHER INVESTIGATIONS The application has two functions, one is to alert the mariner by means of an audible signal together with a visual clue pointing towards the dangerous target through the Head-up Display, the other is a display of operational information. Operational information can be considered in the widest sense. It is certainly not limited to collision avoidance. As outlined in the ACCSEAS prototype description, anti grounding and Marine Safety information a.o. are equally important. The effective management of safety critical information is a key factor in safe navigation. Depending on the situation this sometimes leads to an information overload for the watch officer in which he is just limited assisted by the integrated bridge systems presently on board. Moreover, the high number of alarm signals generated by each individual system on a modern ships bridge might even contribute to the state of information overload. Thus, helping the mariner in his decision making on the basis of relevant and validated operational information should be one of the key elements to enhance safety. Augmented Reality aims at increasing situational awareness by showing visual clues in the real world direction, i.a. Head-up, and acts as an information filter and expert support decision system. The selection and validation of available information is traditionally done by the human on the bridge. Although this task will stay there as long as humans operate ships, it can be foreseen that an increase in the amount of available information, as already shown today by the introduction of multiple ways of communication (e.g. satellite and AIS), leads to a harder task of managing information than before. Thus any support in effective information management must be investigated. The role of Augmented Reality can be multifold in this regard. Validating information is done by the WO more effective when the information is filtered and portrayed in geographical referenced direction. The quality of system (re- )generated information is effectively checked by visualizing the information as overlay on the visual outside real world, thus enhancing the integrity of the system as a whole and increase the user appreciation if the functionality of the system. As already pointed out in section three a functional alarm algorithm is of paramount importance for user acceptance. Users, confronted with either false alarms or lack of alarms, lose their faith in the system completely. Setting the alarm threshold too wide, in order to 6

7 have more time to react, is considered disadvantageous because it might generate unwanted alarms. It also happens that alarms are switched off because of the numerous false alarms. The present testing is aimed to check the technical feasibility of an AR application, i.a. a proof of concept in the simulated environment. Secondarily testing is needed to investigate what the human factors implication is for the watch keeper. It will be very interesting for example to find out whether WO s tend to feel themselves protected by a last resort alarm as described and act accordingly or whether it has no affect at all. At the Maritime Simulator Training Center (MSTC), situated near the Maritiem Instituut Willem Barentsz at Terschelling, each year over 1000 students and licensed mariners are trained, so a great number of test persons with a wide variety of experience can be found for testing. The ACCSEAS prototype E-navigation services like Route Exchange and grounding alarm (NoGo area) a.o. are foreseen to be projected as an overlay on ECDIS. However, the functionality of AR might increase the effectiveness or the acceptance of these services. This should be tested in a simulated Ships Bridge environment. Development is ongoing, as is discussion on a variety of issues in relation to Augmented Reality and E-navigation. To name a few issues, there are: 1. Input sensors, what is the relation of the nav-sensors in use to the accuracy of the calculated CPA/TCPA and the reliability of alarm 2. Decisions on alarm threshold and target classification. exchange alarm threshold with other ships VTS 3. Display unit, is there a user acceptance to wear goggles during watch, will alternative monitors be developed as time and foreseen use progresses 4. Multiple users on the bridge, what about the use of Augmented Reality when a bridge team of Captain, Pilot, Watch Officer and Cadet is working on a difficult high-traffic landfall during limited visibility. 6 Conclusion and Acknowledgement Augmented reality has great potential as man machine interface in the navigation domain. However this is unknown territory in a great many aspects. Much work is to be done to test this innovative concept in the broad field of human factors and the technical aspects mentioned previously. Although research and development on this topic is carried out in a few institutes around the globe, any interest and effort in bringing this promising development further is welcomed. Te authors like to express their gratitude towards the European Union which enables innovative projects like ACCSEAS by co-funding. Without this EU interregional initiative creating an e-navigation prototype testbed would not happen at this scale. 7

8 7 Literature [1] Cockroft A. N. and Lameijer J.N.F. (2012). A Guide to the Collision Avoidance Rules: International Regulations for Preventing Collisions at Sea. 7th Edition, Elsevier Butterworth-Heinemann, Oxford. [2] Baldauf, M.; Benedict, K.; Fischer, S.; Motz, F.; Schröder-Hinrichs, J.-U. (2011), Collision avoidance systems in air and maritime traffic. Proc. IMechE Part O: Journal of Risk and Reliability. Vol. 225 (3) [3] Hilgert, H., Baldauf, M. (1997), A common risk model for the assessment of encounter situations onboard ships. Ocean Dynamics, 1997, 49(4), , England. [4] M. Baldauf, S. Klaes, K. Benedict, S. Fischer, M Gluch, M. Kirchhoff, M. Schaub: (2012) Application of e-navigation for Ship Operation Support in Emergency and Routine Situations. European Journal of Navigation; Volume 10: 2; 4-13 [5] Jens-Uwe Schröder-Hinrichs, Erik Hollnagel, Michael Baldauf, Sarah Hofmann & Aditi Kataria (2013): Maritime human factors and IMO policy, Maritime Policy & Management: The flagship journal of international shipping and port research, 40:3, Appendix- the ACCSEAS project and selected services proposed for test trails A1 ACCSEAS Accessibility for Shipping, Efficiency Advantages and Sustainability It is a demanding job navigating a vessel and mariners receive a lot of information at one time through various sources. ACCSEAS (Accessibility for Shipping, Efficiency Advantages and Sustainability) is a three year project that believes e-navigation will eventually make a mariner s job easier by taking information from several different systems and platforms and displaying it in an easy to use and integrated way, giving the mariner an effective tool to help them take decisions more quickly and effectively. The North Sea Region (NSR) hosts some of the busiest shipping lanes in the world. Yet the amount of navigable space available to shipping is set to decrease as more demands are placed on this area of water. Demands are created not only by the trend towards larger vessels, and growing vessel traffic, but also by the areas of sea already dedicated to, and planned for, energy extraction (for example wind farms and oil platforms). Increasing vessel traffic added to decreasing sea space could involve a significant navigational safety risk, and may greatly impact shipping efficiency. The ACCSEAS project developed a prototype portfolio of services and solutions for mariners and related traffic management professionals. The intention of these services is to make navigating the North Sea safer, simpler and more efficient. RESEARCH AND DATA GATHERING The ACCSEAS Project has published a Baseline and Priorities report One of the initial tasks for ACCSEAS was to create a comprehensive report aimed at assessing and prioritizing potential e-navigation solutions to resolve recognized issues in the North Sea Region. In the past year beneficiaries have worked together to identify the issues and trends relating to North Sea Region maritime transport routes, access to ports and peripheral maritime areas reliant on these routes. Led by Thomas Porathe, Chalmers University, the 8

9 Baseline and Priorities Report identifies existing navigation services provision and their planned evolution. Visit: to read the report. Please contact a member of the ACCSEAS project for more information, or visit A2 - NO-GO AREA SERVICE Today vessels are sailing using charts with static depth curves, which mean mariners need to manually calculate or obtain tidal information and apply wanted under-keel clearance (UKC). No-Go Area is an on-board service that would provide vessels a live picture of where it cannot safely go along its intended route. It would appear as an option on the ECDIS screen on the bridge of a ship. Using: the dimensions of the ship real-time tidal data water depth Enabling the Mariner, this service will: Create a visualization, unique for each vessel Indicate areas on the vessel s route where there is a possibility of grounding Could be turned on or off on demand Integrate with other services such as Route Suggestion so that an intended route would take into account any no-go areas. It saves mariners from making complicated calculations or navigating difficult areas by sight. A3 TACTICAL ROUTE EXCHANGE AND ROUTE SUGGESTION This service allows mariners to communicate their intended routes with each other and Vessel Traffic Services. It will also allow VTS centres to suggest the most efficient/safe routes to the vessel. Route Exchange in which a vessel s projected future route could be exchanged over appropriate communication links ship-to-ship, ship-to-shore or shore-to-ship. A vessel could transmit its intended route to advise other vessels in its vicinity of its intentions. Similarly, shore-based services could transmit a route suggestion to the ship, as a decision aid for the mariner, to advise potential alternative routings and their benefits. This service will: Be a harmonized way of communicating routes between vessels and VTS Be visualized as an option on the ECDIS/VTS screen, with only pertinent routes displayed Increase spatial awareness for the mariner and VTS centre Allow VTS centres to share routes of other vessels. The exchange of this information should lead to better situational awareness for the mariner as they can see the intention of nearby vessels, and take appropriate action if necessary. 9