Real-time map projection in virtual reality using WebVR

Size: px

Start display at page:

Download "Real-time map projection in virtual reality using WebVR"

Augusta Charlotte Nelson
5 years ago
Views:

1 Real-time map projection in virtual reality using WebVR Marko Letić*, Kosa Nenadić ** and Lazar Nikolić* * Faculty of Technical Sciences, University of Novi Sad, Novi Sad, Serbia ** Schneider Electric DMS NS LLC Novi Sad, Novi Sad, Serbia marko.letic@uns.ac.rs, kosa.nenadic@schneider-electric-dms.com, lazar.nikolic@uns.ac.rs Abstract - In this paper, we present an implementation of a real-time web-based solution that projects pre-rendered map tiles onto a virtual reality plane. An interaction layer, that enables different ways of map manipulation including its surroundings, is developed. The implementation of the projection is described in detail. The main goal of our work is to show the advantages of the web-based map applications over the maps that currently exist and are being developed for native hardware and software support in virtual reality. We measure and analyze their performances and discuss possible future integrations of the already existing platforms with the proposed solution. The implementation leverages JavaScript with WebGL by extending the possibilities of the A-Frame framework based on WebVR. The results of the performance evaluation proved a high usability and interaction level of the proposed solution with already existing and widely available hardware and software support. Keywords - map; projection; virtual; reality; WebVR; A- Frame; WebGL I. INTRODUCTION Today, there is an abundance of hardware and software solutions available on the market that provide some type of virtual reality (VR) experience to their users [1], [2]. The largest companies in the industry, such as Facebook, HTC, Sony, Google and others, are entering themselves in this expanding market [3]. Almost all of their solutions, e.g. Facebook s Oculus Rift, HTC s Vive, Sony s Playstation VR, Google s Daydream, use different software platform to run their VR experience. Consequently, this approach prevents developers to create VR enabled solutions that are available to a wide number of users. Indeed, smartphones based on Android and ios are making VR accessible to a wider audience more than ever before, although the hardware on these devices currently does not match the performance of the devices such as Oculus Rift or HTC Vive [4]. Nevertheless, consumer devices may perform on a par with expensive specialized hardware for certain tasks [5]. Also, one of the existing problems is the lack of a crossplatform solution for Android and ios. Therefore, a separate development for each platform is required to approach most of the users. This could be changed with the development and use of WebVR. Various incidents are happening around the world at any given moment, such as earthquakes, forest fires, terrorist attacks, car accidents, to name a few. On the other hand, population data, crime data, congestion of traffic routes, weather forecasts and their statistics varies depending on the time of the year and the time of the day. If we look at any of them, a need to associate these types of information to their location and visualize them, is required for any further analysis and presentation. Progress has been made in this field in previous years with solutions like MapBox, Google maps, OpenLayers and others. Also, a VR Geographic Information System (GIS) analysis platform is proposed in [6], which serves 3D digital city and supports integrated VR GIS functions offering 3D analysis and visualization of the concerned city s massive information. The main idea behind this paper is to build a webenabled real-time map that works in virtual reality. The motivation was the exploration of new possibilities for geospatial data analysis opened by introducing a new VR dimension. The solution is developed in response to the expansion of big data and the growing need to visualize large volumes of geospatial data on the web. The solution that is presented in this paper explains how we enabled real-time map projection in VR using A- Frame framework that is based on WebGL and WebVR integration. The paper is organized as follows. In section II, related work is presented. Section III describes how a scene was created in VR. Section IV introduces a map, while in Section V its projection is presented. Section VI describes the implementation of the interaction layer. A performance analysis of the proposed solution is done in section VII. The last section concludes the paper. II. RELATED WORK A modular web-based VR framework is presented in [1] that extends video conferencing capabilities with virtual and mixed reality functionality. One of the modules, that is in charge of displaying content in VR, utilizes the A- Frame framework linking the WebVR and Three.js. In this way, a VR enabled head-mounted display can present VR content in the Firefox web browser. In the paper [7], the authors explore the use of the WebVR and the Google Cardboard for low-cost immersive analytics. They focus on recreating known physical visualization designs in VR, exploring the performance trade-offs between fast and realistic rendering and stating that WebVR shows promising results. An analysis of several Map APIs and evaluation of the most suitable API for interactive images regarding its usability, accessibility and performance is conducted in MIPRO 2018/SSE 1669

2 [8]. The paper [9] identifies and compares the most capable libraries suitable for foundation of a web GIS client. The applied criteria include their GIS feature coverage and some quality metrics. In paper [10] GPU power is used to accelerate data discovery and visual analytics in MapD solution, where the power of MapD database is leveraged in MapD's Immerse visualization platform for complex data visualizations (including detailed GIS representations). A. WebVR WebVR is an open specification that enables VR experience in the web browser [11]. Its goal is to facilitate VR experience to users, regardless of a device they have. The only two things that are required by WebVR are a headset and a compatible browser. The easiest way to get started is with a basic headset like Google Cardboard and a suitable web browser installed on the users smartphone. WebVR can also be used on advanced headsets with a browser that supports this specification and can run on the chosen headset. Its development is led by major browsers engineers, and some independent developers. Some of the web browsers that currently support this specification are Chrome for Android, Chromium, Firefox, Microsoft Edge, Oculus Browser and Safari [12], [13]. There are also WebVR emulators available that enable the API testing without any use of external hardware headsets. WebVR has two base concepts. The first one sends stereoscopic images to both handset lenses. The other one receives positional data from the sensors that track a head movement. The API enables access to these data and their integration with already existing 3D solutions in web browsers. Certainly, one of the most used libraries for displaying 3D computer graphics in a browser is Three.js. This library allows for the creation of GPU-accelerated 3D animations using the JavaScript language with WebGL API. The projects running on Three.js can be made VR-ready by adapting them to use WebVR. In this way, many existing projects can be ported to VR without much additional work. B. A-Frame A-Frame is a web framework for building virtual reality experience. Originally developed by Mozilla, A- Frame was intended to be an easy but powerful way to develop VR content. A-Frame uses Custom Elements (a W3C s draft recommendation) to act merely as data containers and it does not trigger the layout engine. Hence, it overcomes 2D layout which is a primary performance concern for normal web applications. It uses in-memory 3D operations with minimal overhead which are rendered with WebGL leveraging OpenGL or Direct3D. III. CREATING A SCENE IN VIRTUAL REALITY A-Frame framework comes with a prepared set of components to use to order to create a simple 3D scene. A-Frame can be developed from a plain HTML file without having to install anything. All that is needed is to include the library in the <script> tag and to add a valid scene with <a-scene> tag which is shown in Listing 1. Components that are rendered in this scene are later used as a placeholder for map projection. <html> <head> <script src=./aframe.min.js"></script> </head> <body> <a-scene> <a-box top color="#000"></a-box> <a-box left color="#000"></a-box> <a-box bottom color="#000"></a-box> <a-box right color="#000"></a-box> <a-entity position=" "> <a-camera></a-camera> </a-entity> </a-scene> </body> </html> Listing 1. HTML used to render basic A-Frame scene This HTML will render a VR-enabled 3D scene in the browser that is presented in Figure 1. Figure 1. Basic A-Frame VR scene created as a placeholder As it can be seen in Figure 1, A-Frame has rendered a scene that contains a frame in which a plane object is injected. The image in the bottom right corner, that represents glasses, designates that the scene is VR enabled in the browser. Although the previous scene can be created in a simple way, integration of a map that can be interacted with, is not an easy task. A-Frame supports a custom component creation and this approach should be used to add any custom object to a scene. IV. CREATING A MAP ENTITY It would be a straightforward process to add a map to the scene, if the A-Frame allowed adding standard HTML elements. As this is not the case, the decision was to add our map as a canvas and then to project it as a texture onto a plane that will be added to the scene afterwards. Since the subject of this paper is the projection of a map and not the creation of one, authors decided to use MapBox GL JavaScript library that renders maps from raster and vector tiles using WebGL. MapBox GL uses tileset sorting and ordering mechanism to properly load and cache tiles that are to be displayed. A tileset is a collection of raster or vector data broken up into a uniform grid of square tiles at 22 percent zoom levels MIPRO 2018/SSE

3 MapBox GL library internally determines which tiles to load and show, based on different conditions such as the width and height of the container, zoom level and user actions such as panning, rotating or tilting. The map navigation controls were employed to create an interaction layer to enable the map manipulation in virtual space as there are no mouse or keyboard that could be used. Although this solution relies on MapBox GL to render the map properly, this layer can be easily replaced with a similar library. As the A-Frame is based on entity-component-system pattern, we created an entity, the element, that was added to the HTML as an A-Frame element. In the A-Frame Custom primitives (i.e. elements) are registered using AFRAME.registerPrimitive(name, definition) function. Parameter name is a string and it represents the name of a new element, while definition parameter is a JavaScript object that defines defaultcomponents and mappings properties. We named our element a-mapgl and registrated it with A-Frame as shown in Listing 2. This element represents an entity in the entity-component-system pattern. AFRAME.registerPrimitive('a-mapgl', extenddeep({, meshmixin, { defaultcomponents: { mapgl: {, geometry: { primitive: 'plane', material: { color: '#FFF', shader: 'flat', side: 'double', transparent: true,, mappings: { height: 'geometry.height', width: 'geometry.width' )); Listing 2. Registering a-mapgl entity This enables adding of <a-mapgl> element to the main HTML file, but it still does not have any logic implemented for it to project the map onto the plane. In order to do this we created a component named mapgl that is referenced in the defaultcomponents property in the a-mapgl entity. V. MAP PROJECTION To register a component that can be used with the entity we used AFRAME.registerComponent(name, definition) function. The main functions that we redefined for this component are init and update. A. Init function Init function is used to set up initial state and instantiate variables. It is called once when the component is initialized. This function is used to dynamically create a container that MapBox GL uses to inject the generated canvas element into. The main problem with generating this container is that MapBox sets the width and height of the canvas based on the offset of the container. Because this canvas is hidden, MapBox GL is not able to properly render the map. One solution is to set the position of the container to be fixed and move it out of the visible part of the view in order that the map can be properly rendered and invisible for a user as shown in Listing 3. element = document.createelement('div'); element.setattribute('id', 'mapboxcontainer'); element.style.position = 'fixed'; element.style.left = document.width; document.body.appendchild(element); Listing 3. Creating a hidden map canvas container This element can now be referenced with the MapBox GL map constructor as a container for the canvas as show in Listing 4. var map = new mapboxgl.map({ container: 'mapboxcontainer', style: 'mapbox://styles/mapbox/streets-v9',... ); this.mapinstance = map; Listing 4. Generating a map instance After the constructor initializes the map variable, the canvas is added to the container and tiles are loaded. This variable is then exposed globally as a service instance that is used to build an interaction layer and update the map if a change occurs. Adding a texture that represents the canvas is now based on adding a source attribute that references the id of the canvas element. This id can be dynamically added after the map finishes loading as shown in Listing 5. this.el.setattribute('src', #canvasid'); Listing 5. Adding the canvas element as a source of the element texture A-Frame uses meters in a 1:1 ratio, since WebVR is also using meters. This means that 1 unit in A-Frame is equal to 1 meter in real world. Because the plane element requires these units to define width and height of the containers, a mapping needs to be implemented so that the rendered map can scale its width and height defined in pixels to fit onto the texture of a plane shown in Listing 6. We defined a multiplication scale factor. It is recommended that this calculation result be a power of two since older graphic cards process textures in this way. mapwidth = geometry.data.width * this.data.multiplyscalefactor; mapheight = geometry.data.height * this.data.multiplyscalefactor; Listing 6. Calculating map size based on a scale factor If we include the a-mapgl element and place around it a frame-shaped object we get a VR scene as illustrated in Figure 2. MIPRO 2018/SSE 1671

Figure 2. Real-time map projection in VR B. Update function Update function is triggered when at least one of the parameters passed with component registration has been changed.

schema { //The desired center (LngLatLike) center: {default: 0, //The desired zoom level (number) zoom: {default: 0, //The desired bearing, in degrees.

4 Figure 2. Real-time map projection in VR B. Update function Update function is triggered when at least one of the parameters passed with component registration has been changed. Since we can choose which parameters to include we decided to define similar parameters map instance uses to navigate the map shown in Listing 7. schema { //The desired center (LngLatLike) center: {default: 0, //The desired zoom level (number) zoom: {default: 0, //The desired bearing, in degrees. The bearing is the compass direction that is "up"; //for example, a bearing of 90 orients the map so that east is up. (number) //pitch The desired pitch, in degrees. (number) bearing: {default: 0, //If zoom is specified, around determines the point //around which the zoom is centered. (LngLatLike) around: {default: 0, Listing 7 - Registration component parameters These parameters can be exposed outside of the a- mapgl element and used to control the map. For this to work, the update function will refresh the map with the new values. VI. INTERACTION LAYER In a similar way that we created a-mapgl element (entity) and added it to the solution, we have created an interaction element that hooks to the parameters that can update the map. As headsets do not have mouse or keyboard input, a gaze-based (fuse-based) cursor has been implemented to call the event handlers attached to this element. If we set a cursor to be gaze-based, the cursor will triggers a click when the user gazes at an entity for a predefined amount of time. It can be imagined as a pointing device, the direction of the user s sight is a straight line going from the camera position to an infinite target. When you gaze at the entity the line intersects with it. The disadvantage of gaze-based interactions is that they require the user to turn their head significantly. Figure 3. shows the component s appearance. Figure 3. Interaction layer component An example of an event handler attached to the interaction is presented in Listing 8. var setcomponentproperty = window.aframe.utils.entity.setcomponentproperty; entity.ongaze = function(handletype) { if (handletype === 'zoomin') { setcomponentproperty(mapel, 'map.zoom', zoomlvl+1); else if (handletype === 'zoomout') { setcomponentproperty(mapel, 'map.zoom', zoomlvl-1); Listing 8. Interaction layer event handler VII. PERFORMANCE ANALYSIS The performance testing of this real-time map projection in virtual reality based on A-Frame and WebVR was conducted on a computer with an Intel Core i5 at 2.7 GHz CPU with 8GB of RAM and 128 GB SSD running macos High Sierra and a VR emulator in Chrome for macos version 64. The testing and comparison was done in four different scenarios. The parameters that were monitored were frames-per-second (FPS) and the size of the JavaScript heap size. The results are presented in Table 1. Scenarios that were analyzed include: Scenario 1 - Map rendering onto the canvas in 2D with zoom interaction for 20 zoom levels; Scenario 2 - Rendering a basic A-Frame scene described in Section II; Scenario 3 - Rendering the A-Frame scene with the included a-mapgl element described in Sections IV - V without the interaction layer. We are using WebVR API to change the camera position and navigate through the VR space; Scenario 4 - Scenario 3 with the interaction layer that is used to zoom the map for 20 zoom levels. Tools used to measure these performances are: Stats.js JavaScript Performance Monitor developed by Ricardo Cabello, the author of Three.js (used to measure FPS) and Performance profiler from Chrome developer tools. Main reason why FPS meter from Chrome developer tools was not used is because FPS meter does not measure FPS performance properly when a canvas is used MIPRO 2018/SSE

5 Profiling time for each test was set to 30 seconds, as this was a very intensive operation for the device on which the analysis was running. In Table 1. a range of FPS and memory values are presented for the given timeframe. The results of the tests are displayed in Table 1. Scenario Number FPS (higher is better) Table 1. Performance analysis test results JS heap size (MB) As the range from 30 to 60 FPS is considered optimal for the human eye [14] it is noticeable that Scenarios 1 and 4 behave suboptimal in some cases. As Scenario 1 only contains a map rendered by MapBox GL library, there was nothing we could have done to optimize this performance. The main drop in FPS happens when the library starts loading and rendering new map tiles. As our solution depends on this rendering, it is clear that FPS rate cannot be higher. A drop below 19 FPS in Scenario 4 happens when the user starts changing the position of the camera while the loading of new tiles is still in progress. CONCLUSION The results displayed in the performance analysis suggest that interactions with the map in virtual reality can be bellow optimal level if a user decides to move around its surroundings (by changing camera position) while the map tiles are loading. A solution that disables camera movement until the tile loading is completed can be implemented to prevent drop in the performance. If we remove this scenario from our use case, frame rates are ranging from 31 to 39 fps which is considered optimal for the human eye to process [13]. The results of the performance evaluation proved a high usability and interaction level of the proposed solution with already existing and widely available hardware and software support. The solution described in this paper showed a possible future of the map-based web applications using a component that can be easily integrated into already existing web 3D projects. The next step in the development of this solution is its modification and adjustment to work with augmented reality so it can interact with the real world. A new interaction layer can be created to use new types of controllers such as Leap Motion that supports hand and finger motions as input, but requires no hand contact or touch. and QoE assessment of HTC Vive and Oculus Rift for pick-andplace tasks in VR, th Int. Conf. Qual. Multimed. Exp. QoMEX 2017, pp. 3 5, [4] N. M. Papachristos, I. Vrellis, and T. A. Mikropoulos, A Comparison between Oculus Rift and a Low-Cost Smartphone VR Headset: Immersive User Experience and Learning, Proc. - IEEE 17th Int. Conf. Adv. Learn. Technol. ICALT 2017, pp , [5] M. K. Young, G. B. Gaylor, S. M. Andrus, and B. Bodenheimer, A Comparison of Two Cost-differentiated Virtual Reality Systems for Perception and Action Tasks, in Proceedings of the ACM Symposium on Applied Perception, 2014, pp [6] W. Wang et al., Spatial query based virtual reality GIS analysis platform, Neurocomputing, vol. 274, no. 2017, pp , [7] P. W. S. Butcher, J. C. Roberts, and P. D. Ritsos, Immersive Analytics with WebVR and Google Cardboard, in Posters presented at the IEEE Conference on Visualization (IEEE VIS 2016), Baltimore, MD, USA, [8] G. Velkoski, M. Gusev, and S. Ristov, Analysis of interactive image technologies, 24th Telecommun. Forum, TELFOR 2016, [9] G. Farkas, Applicability of open-source web mapping libraries for building massive Web GIS clients, J. Geogr. Syst., vol. 19, no. 3, pp , [10] T. Mostak, Using GPUs to accelerate data discovery and visual analytics, FTC Proc. Futur. Technol. Conf., no. December, pp , [11] WebVR info page, [Online]. Available: [Accessed: 30-Jan-2018]. [12] A. Deveria, Up-to-date browser support tables for WebVR, GitHub, [Online]. Available: #search=webvr. [Accessed: 30-Jan-2018]. [13] Oculus, Oculus Browser, Oculus VR, LLC [14] K. Debattista, K. Bugeja, S. Spina, T. Bashford-Rogers, and V. Hulusic, Frame Rate vs Resolution: A Subjective Evaluation of Spatiotemporal Perceived Quality Under Varying Computational Budgets, Comput. Graph. Forum, p. n/a--n/a. REFERENCES [1] S. Gunkel, M. Prins, H. Stokking, and O. Niamut, WebVR meets WebRTC: Towards 360-degree social VR experiences, Proc. - IEEE Virtual Real., pp , [2] A. Fineschi and A. Pozzebon, A 3D virtual tour of the Santa Maria della Scala Museum Complex in Siena, Italy, based on the use of Oculus Rift HMD, 2015 Int. Conf. 3D Imaging, IC3D Proc., [3] M. Suznjevic, M. Mandurov, and M. Matijasevic, Performance MIPRO 2018/SSE 1673

WebVR: Building for the Immersive Web. Tony Parisi Head of VR/AR, Unity Technologies

WebVR: Building for the Immersive Web Tony Parisi Head of VR/AR, Unity Technologies About me Co-creator, VRML, X3D, gltf Head of VR/AR, Unity tonyp@unity3d.com Advisory http://www.uploadvr.com http://www.highfidelity.io