International Global Navigation Satellite Systems Society IGNSS Symposium 2009 Holiday Inn Surfers Paradise, Qld, Australia 1 3 December, 2009 Trials of commercial Wi-Fi positioning systems for indoor and urban canyons Thomas GALLAGHER School of Surveying & Spatial Information Systems, University of New South Wales, Sydney, Australia Phone: +61 (2) 9385 4189 Fax: +61 (2) 9313 7493 Email: gallagth@gmail.com Binghao LI School of Surveying & Spatial Information Systems, University of New South Wales, Sydney, Australia Phone: +61 (2) 9385 4189 Fax: +61 (2) 9313 7493 Email: binghao.li@unsw.edu.au Allison KEALY Department of Geomatics, The University of Melbourne, Melbourne, Australia Phone: +61 (3) 8344 6804 Fax: +61 (3) 9347 2916 Email: akealy@unimelb.edu.au Andrew G DEMPSTER School of Surveying & Spatial Information Systems, University of New South Wales, Sydney, Australia Phone: +61 (2) 9385 6890 Fax: +61 (2) 9313 7493 Email: a.dempster@unsw.edu.au ABSTRACT It is now widely accepted that GPS meets, under ideal operational conditions, all attributes of a ubiquitous positioning system, i.e. accuracy, reliability and availability. However, its performance quickly deteriorates in certain environments, such as indoors or in urban canyons. In such environments, the demand for location based services (LBS) is growing exponentially, mainly because of the rapid expansion of the smart-phone market. In this context, 802.11-based positioning systems can be used as they are accurate enough for most current LBS and most importantly because they do not require any specialised hardware or additional infrastructure. In this growing market, two main companies, Skyhook and Ekahau, have emerged, proposing two different Wi-Fi positioning solutions, both relying on the fingerprinting technique. The purpose of this paper is to test these two different 802.11 positioning systems in environments where one cannot rely on GPS alone to obtain a position, i.e. indoors and in urban canyon environments. First, the results obtained indoors are detailed, and then those obtained in urban canyons. An attempt was also made to link the positioning error with observable parameters of the wireless network such as the number of access points scanned. Static and mobile tests were conducted, both indoors and outdoors, and with different types of hardware. The results show that the Ekahau system performs well indoors with position errors less than 10 metres most of the time, and that Skyhook has accuracies up to 10 metres outdoors, but is very dependent on the environment.
KEYWORDS: Wi-Fi positioning, indoor positioning, urban canyon positioning, Skyhook, Ekahau. 1. INTRODUCTION These last few years, the demand for Location Based Services (LBS) has grown exponentially, especially given the rapid expansion of the smart phone market. Phones such as Apple iphone, or HTC Dream, are now small terminals which often embed GPS and Wi-Fi chips, in addition to their basic phone capabilities. Nowadays, the most important source of location information is the Global Positioning System (GPS). However, it is well known that GPS doesn t perform well in areas such as indoors or in urban canyons, where tall buildings block the satellites signals. It is especially in these types of areas that the demand is growing the fastest. To answer this growing demand, and given the massive utilization of Wi-Fi signals for other purposes, Wi-Fi positioning technology has attracted much attention from both researchers and private companies, first because the user doesn t need any specialized hardware as most of the mobile devices are equipped with a Wi-Fi chip. Second, there is no need to deploy an extra dedicated network as radio signals from at least a few access points (APs) can be detected in the majority of areas of interest, due to the proliferation of wireless networks. Finally, the accuracy that can be achieved when using Wi-Fi positioning technology is usually good enough to answer the needs of most users, such as guiding people towards the closest bus or train station, or directions to a nearby shop or area of interest. Two positioning techniques relying on Wi-Fi signals have been developed. The first uses a signal propagation model and a geometric model of the environment. Using these, it can deduce distances to an AP from signal strength (SS) measurements. Trilateration can then deduce the position of the user (Li et al., 2006). This technique is easy to implement, but suffers from the difficulty of building sufficiently realistic propagation models. Moreover, it needs the exact location of the AP (which is usually unavailable) to use trilateration. The second technique is called fingerprinting. This technique has received more attention recently as it doesn t need propagation or environment models. It first requires the building of a database of SS from different AP at some reference points in the desired positioning area. The location of the user is then obtained by measuring the SS at the user location, and comparing it with the different elements of the database (Ladd et al., 2002) (Li et al., 2005). The main disadvantage of this method is the labor required to build and maintain the database. For instance, if a major building is built in the area, the database needs to be updated. The purpose of this paper is to test two different Wi-Fi positioning systems, in poor GPS environments, typically indoors and in urban canyons. The first, the Ekahau Positioning Engine (EPE), is commercialized by Ekahau, a Finnish company. EPE is based on the fingerprinting technique described earlier. The database must be built by the user, according to the area requiring coverage. It requires the user to walk around the entire area to be covered to record the signal strengths. The second system, called Skyhook Wireless Positioning System (WPS), is commercialized by Skyhook, based in Boston, MA. As EPE, WPS relies on the fingerprinting method, but it doesn t require the user to perform the survey before positioning, as the database is built and maintained by Skyhook. In fact, Skyhook hires drivers to do their surveying work. They deploy a fleet of data collection vehicles to conduct a comprehensive AP survey within the targeted coverage areas. Skyhook s architecture is
designed so that all the calculations are made on Skyhook s servers. When a user requires its location using Skyhook, the software scans the nearby AP and sends the results to Skyhook s servers, so an Internet connection is required to use Skyhook. The server then processes this data and sends back the computed position to the user. This request/answer process takes less than two seconds most of the time. This paper also makes an attempt to relate the positioning error to different observable parameters such as the number of AP scanned, or the variance of SS. Position error estimation is mainly valuable in three ways: first it notifies the user about the trust one can have in the position requested. For instance, a user will discard a position with an error estimation he judges too important for the service he requires. Second, it can be used by inference algorithms to give more or less priority to a position depending on the error made, and third, it can be used in order to optimize the performances of the system, for instance by installing more AP in high error zones. The paper is divided in two parts: the first details the results of the indoor tests for both systems, the second one details the results of the tests made in urban canyons in the CBD of Melbourne and Sydney. Finally, Section 4 concludes the paper and makes some suggestions about future work. 2. INDOOR POSITIONING 2.1 Ekahau Positioning Engine (EPE) This section presents the practical results obtained from testing a commercially available system named Ekahau Positioning Engine (EPE). The testings took place on the 4 th floor of the University of New South Wales (UNSW) Electrical Engineering building, in Sydney, on the 4 th of May 2009. For the purpose of the trials, and as recommended in Ekahau s guidelines (Ekahau, 2008), a WLAN network was deployed, using a unique identifier, and using only channels 1, 6 and 11 to avoid interference between channels. The next step was to build the database used in the fingerprinting technique. This was done using Ekahau Site Survey software, which records the radio field strength pattern across the covered area. Ekahau s software is proprietary, so precise details cannot be given here as to how they build the database. However, the accuracy of the site survey determines the accuracy of the results. It requires walking around the entire area to be covered with a laptop and a Wi-Fi adapter in order to record signal strengths. This task is quite time consuming and often impractical for large areas (outdoors for instance). Figure 1 shows the site survey used in the indoor testing of Ekahau. The tracks with arrows represent the path followed by the rover during the survey. Once the survey is made, the Ekahau client software must be installed on the devices used for tracking. This software is compatible with the majority of Wi-Fi adapters embedded in laptops and with nearly all mobile phone operating systems. Ekahau also commercialises dedicated small Wi-Fi tags which can easily be tracked by EPE. Ekahau also provides an extensive Java API to manage incoming positioning messages, which was used to write a data logging application which recorded several relevant information such as x and y coordinates, number and MAC addresses of AP scanned, and SS in an XML file format.
Figure 1. View of Ekahau Site Survey software. The colors represent the quality of the Wi-Fi coverage (green = one AP over -55 dbm and two ones over -75 dbm; red = less than 3 AP over -75 dbm) Figure 2 shows the positioning results for 13 test points equally distributed along the main corridor of the floor, between access points A and C. EPE works with a local coordinate system, depending on the map uploaded. The coordinates are given in pixels. So the true coordinates were obtained using a scale to convert pixels to metres, and a reference point that was plotted accurately on the map. Two different mobile phones were used, at two different times, in order to test the repeatability of the results: Figure 2. Results of the corridor tests: HTC phone (top), O2 phone (bottom)
Here the Root Mean Square (RMS) value was chosen because it is more relevant than the Standard Deviation when discussing distance errors. Indeed 95% of the samples are expected to be under 2*RMS. As you can see, the results are satisfying with an average positioning error of 7 metres and 95% of the time, accuracy is expected to be greater than 16.5 metres. Also, the repeatability of the results is very good, especially when considering the extreme variability of Wi-Fi signals. The biggest difference in means observed at the same point is of 6.5 metres. Table 1 summarises the results of two other tests conducted on the same floor. A Dell laptop was first placed in a busy area and then moved into a quieter one, and sent its position every 5 seconds for a period of 2 hours at each location: Average error 2 * RMS Busy area 12.84 m 16.52 m Quiet area 2.67 m 8.75 m Table 1. Results of Ekahau static tests As expected, the results are better in the quiet area. Indeed, as the environment is more stable (fewer people coming in and out), the values of SS for the AP will vary less than in the busy area. Therefore, when requesting a location, the values recorded and compared to the database values will be closer to the ones recorded earlier. An attempt was also made to link the position error with observable parameters of the wireless network. In Ekahau s case, the latter are the number of AP scanned, and their SS value. Figure 3 shows the position error versus the number of AP scanned for all the samples collected, both in the corridor and during the static tests. It also shows the proportion of weak access points, i.e. those whose SS are just above the devices detection threshold. Figure 3. Position error vs. number of AP scanned for Ekahau indoors
As can be seen on the left part of the graph, it appears that the more AP used in positioning, the better the results will be. However, it also seems that when the number of weak APs increases, the performance degrades. So it seems better to keep only AP with strong SS than to use all those the device can detect. As the Ekahau algorithm is proprietary, it wasn t possible to check this fact by not taking into account of weak APs. But it appears to be a valid hypothesis and future work using a positioning algorithm developed at UNSW will check the impact of weak AP on accuracy. Another parameter studied was the variability of the SS. If the SS variance for a given AP at the same location is high, it is easy to see that the probability that the algorithm selects the wrong fingerprint is higher than when the SS are steady. To produce Figure 4, the variance of the SSs for each of the scanned AP was calculated, and these variances were then averaged in order to get an overall SS variance value. This operation was repeated for the 13 points located along the corridor, and for the two static test points. Then, the mean positioning error at the test point was computed to see whether or not a bigger SS variance will produce a bigger positioning error: Figure 4. Mean positioning error vs. Overall SS variance for Ekahau indoors There is no obvious correlation between the positioning error and the overall SS variance. However, more investigation is needed, to focus on the impact of the SS variance for each AP, and especially for the strongest ones, as the hypothesis that it should have a bigger impact on the positioning error needs to be more thoroughly tested. As a conclusion to this part, EPE performances indoors are satisfying when an accurate survey has been conducted, although being a little less than what Ekahau claims (average accuracy of 3 metres). With such accuracies, a lot of new LBS can be imagined, such as guiding people in a shopping centre or through an important subway system. EPE has already been adopted by many businesses such as retail, logistics, hospitals or manufacturing, and is used to track accurately assets or people. However, an extensive survey must be conducted, and the configuration of the system requires time and knowledge. An attempt was also made to link the positioning error with observable parameters. Encouraging results were obtained,
but further investigation in this topic will be made in comparison with an algorithm developed at UNSW. 2.2 Skyhook Wireless Positioning System (WPS) This section presents the practical results of the indoor tests of Skyhook WPS. The tests took place on the 4 th floor of the University of New South Wales (UNSW) Electrical Engineering building, in Sydney, on the 4 th of May 2009. Further testings were conducted in various indoor locations in the CBD area of Sydney, on the 7 th of July 2009. In order to use Skyhook WPS, one needs to register to Skyhook in order to get a username and a password to use the system. Registration for research or private purposes is free, but payment is required if a large number of requests are made (more than 1000 a day). Once registered, Skyhook has developed APIs for nearly any operating system both on laptops and cell phones. These APIs are always very simple and contain fewer than 10 basic functions, which manage the scan of the Wi-Fi networks and the request to Skyhook s servers. This is a good thing as it allows developers to add location functionalities to their applications very easily. Figure 5 and Figure 6 show the results of the first testings, which took place at the same 13 points located along the corridor used for Ekahau, in Electrical Engineering building in UNSW. The phone used was a HTC TyTN II. As can be seen in Figure 6, the system worked with excellent availability (more than 95% at each test points), and with a reasonable accuracy. Moreover, it seems that Skyhook always outputted roughly the same position. This is probably caused by the way the area was surveyed by Skyhook. Figure 5. Satellite view of Skyhook indoor results at UNSW, for test points 1 and 13
Figure 6. Skyhook indoor results at UNSW Further trials were conducted in various indoor locations around the CBD area. Typically, these locations were shops or commercial galleries, and at different height levels from underground level to the 2 nd floor of a typical skyscraper (10-15 metres high). Google Earth was used to spot the true coordinates of the test points. Google Earth s accuracy was tested using the survey marks located across the area. The maximum error believed to be committed when spotting the test points is of 6 or 7 metres. Table 2 details the results of these testings: TP number Availability Average error Error std Description of TP 1 0 % Retail shop, ground level 2 0 % Same as TP 1, 1 st floor 3 96.6 % 46.9 m 2.7 m Walking bridge 4 93.7 % 57.9 m 4.4 m Walking bridge 5 93.5 % 24.1 m 4.3 m Retail shop, 1 st floor 6 87.1 % 43.9 m 16.4 m Retail shop, ground level 7 70 % 32.2 m 7.9 m Commercial gallery, ground level 8 6.2 % 28.5 m 1 m Commercial gallery, 1 st floor 9 77.4 % 74.9 m 4.3 m Retail shop, ground level 10 90.3 % 98 m 24.7 m Commercial gallery, ground level 11 93.3 % 58.3 m 1.7 m Commercial gallery, 2 nd floor 12 25 % 63.3 m 9.5 m Food court, underground 13 0 % Food court, underground 14 83.9 % 34 m 2.1 m Commercial gallery, 2 nd floor 15 90.6 % 36 m 2.2 m Retail shop, ground level 16 0 % Supermarket, underground 17 58.6 % 216 m 154.1 m Retail shop, ground level Global 56.8 % 62.6 m 18.1 m Table 2. Skyhook indoor results in Sydney CBD
As can be seen, the accuracy reached in these indoor environments is quite impressive. Some of the test points used were located underground, far away from the road (more than 50 metres), and the system was still working, even though with degraded accuracy. However, Skyhook doesn t appear to be accurate enough to be used to guide people inside a building for instance. As we will see in the next section, WPS has been more designed to provide efficient positioning information at a larger scale. 3. URBAN CANYON POSITIONING 3.1 Ekahau Positioning Engine (EPE) The second test of EPE was conducted outdoors, in the CBD area of Sydney. We chose an area where several survey points can be found. These points can be easily located on the streets, and their coordinates are known very precisely, often at centimetre level. In the same way as for indoors, a survey of the area was made prior of testing the engine. This was a really time consuming task because it required walking all around the area holding the laptop for one and a half hours. Figure 7 shows the result of this survey: Figure 7. Ekahau city survey. A: pm147489 B: pm147479 C: pm150399. As can be seen, different types of areas were surveyed. In the north of the map is a wide open area called Martin Place. In such areas, results are expected to be less accurate than in the streets as fewer access points are usually seen by the user, and they are more likely to be weak ones. Three points were chosen to evaluate Ekahau on this map, two in Martin Place, and one at the corner of George St. and King St., located in the South-West of the area surveyed. Table 3 and Figure 8 show the results obtained: As can be seen, the further the device is from the buildings, the worse the results are. The worst point is pm150399, which is located in an open area. The best one, pm147479, is located in a quite narrow street, closer to the buildings.
Distances in metres pm147479 pm147489 pm150399 Maximum error 16.22 70.23 124.83 Average X error 1.17 7.57 26.34 Average Y error 14.28 18.76 26.88 Average error 14.33 20.53 40.31 Standard deviation 2.33 23.55 33.48 90% error 15.90 66.26 102.7 Table 3. Ekahau results in Sydney s CBD Figure 8. Outdoor test of Ekahau. The thick lines intersection is the true position. The thin lines intersection is the mean estimated position. The dashed lines represent the standard deviation of the estimated position. The little crosses are all the samples. The three colors are for the three test points. As a conclusion to this part, and as expected, EPE is less efficient outdoors than indoors. This is probably due to the fact that outdoors, the AP are located further away from the device, therefore the SS recorded will be a lot weaker than the ones which can be recorded indoors. Moreover, the environment outdoors is much more variable than indoors as they are many people and cars moving around the device. Finally, surveying a large outdoor area is quite a challenge, as it requires walking slowly all over the area, carrying a laptop. 3.2 Skyhook Wireless Positioning System (WPS) Outdoor tests of Skyhook WPS were made in the CBD areas of Sydney and Melbourne. The first test took place in the CBD of Sydney. It consisted in going to the same survey marks used in Ekahau s test, and request locations to Skyhook servers. Three tests, at different dates and different times, were made to test the repeatability of the results. Tests were carried out using different phones to test once again the repeatability of the results. Table 4 details the result of these tests, and Figure 9 shows where the test points are located. The two first
columns in Table 4 show the results collected with a HTC Dream phone, at two different dates, and the third column shows the results collected with an older HTC phone running Windows Mobile 5, at another time. As can be seen, the performance is good enough to provide basic LBS, such as giving directions to a nearby bus station for instance. Also, the repeatability of the results is good, especially when using the same phone. That tends to prove that certain characteristics of Wi- Fi signals are somehow repeatable and can be modeled functionally based on their signal strengths. When changing phones, the results tend to vary a little bit more. Indeed, the quality of the measurements is based on the properties of the Wi-Fi antenna, which varies from one model to another. Another test took place in the city of Melbourne. It was a mobile test. The HTC Dream phone was fixed under the windshield of a car which was driven around the CBD area of Melbourne. Figure 8 shows the path the car followed. The car drove through different type of areas: from a metropolitan environment (residential and commercial buildings varying in height and footprint) in the north of the area to an urban canyon environment in the south of the area. Distances in metres pm40209 pm147009 pm77469 Maximum error 96 134 112 224 203 156 72 62 81 Average error 43 35 57 50 43 32 49 45 47 Standard deviation 17 32 23 44 37 51 7 5 6 90% error 65 72 84 107 87 102 58 52 58 pm147489 pm147017 pm147017 Maximum error 93 332 153 196 117 319 65 152 151 Average error 23 52 35 67 59 80 36 39 44 Standard deviation 17 82 36 30 19 55 8 12 21 90% error 42 201 81 113 79 146 41 41 61 pm147479 pm147477 pm150124 Maximum error 176 89 548 148 105 386 98 75 229 Average error 50 49 131 73 43 119 56 25 56 Standard deviation 29 17 124 43 31 103 32 20 43 90% error 85 65 370 141 98 335 92 52 101 pm150399 Global Maximum error 165 160 166 133 143 230 Average error 105 122 104 55 51 84 Standard deviation 47 47 49 27 30 51 90% error 156 156 157 90 90 149 Table 4. Skyhook results in Sydney s CBD. For each test point, results for each of three tests are shown.
Figure 9. Location of the Skyhook test points Figure 10. Path followed during the driving test in Melbourne
Another interesting feature of Skyhook WPS is that when a request is made to Skyhook s server, the answer contains a value of error in metres, which according to Skyhook, gives the user the radius of a circle in which the user is located 95% of the time. Although it is not known how this error value is computed, it is interesting to compare it with the true error committed by Skyhook, and see how it performs (Figure 11). For this test, the true position was obtained using a high sensitivity GNSS receiver. Even in urban canyon environment, it was accurate enough to use it as a reference position for Skyhook (less than 10 metres errors). Once again, two rounds were made in order to test the repeatability of the results. As you can see there is some correlation between the two errors, with the estimated error constantly being more important than the real one. This means that we can rely on Skyhook s given value of error to have an idea about it, knowing that the true value of error will probably be smaller. Once again, the performance in terms of position error is acceptable, especially in regards to the relatively rough conditions of this test, as the experiment was conducted in a moving car. The repeatability of the results is also satisfying, with close average positioning errors for both rounds. Figure 11. True error (blue) vs. Skyhook error (green) for both rounds
Finally, an attempt was made to relate the error made by WPS with observable parameters of the network. Given the information returned by Skyhook s servers, the only parameter available is the number of AP scanned by the device. Normally, the greater the number of AP scanned, the better the results should be. Indeed, increasing the number of AP will average the pseudo-random error caused by the differences between the SS returned by the device then scanning the network, and the values recorded by Skyhook in their database. Figure 12 shows the true error made by Skyhook versus the number of AP scanned, for about 3000 location requests in various locations: Figure 12. Skyhook positioning error vs. number of AP scanned As can be seen, Figure 12 seems to indicate that the more AP are scanned, the better the results are, but further investigation is needed to confirm this result. Also, a lot of additional parameters influence the performance such as the SS of the AP, or the surroundings of the device. To conclude this part, we can say that the performances of Skyhook WPS in urban canyons are good. First, it provides a positioning solution whose accuracy is good enough for most of the LBS. Moreover, Skyhook sent to the user an error estimate which although not being exact, is most of the time conservative. 3. CONCLUSIONS Two commercially available Wi-Fi positioning systems were tested, both indoors and in urban canyons, the two main type of environment where GPS cannot be used as a primary source of positioning. In these environments, both systems have shown accuracy, availability and a great accessibility to users as no new equipment is required. Ekahau Positioning Engine is quite accurate indoors, but its deployment requires time and knowledge. Skyhook, on the contrary, is less accurate, but offers coverage in major cities all over the world, both indoors and outdoors. An attempt was also made to try to relate positioning accuracy with observable
parameters of the Wi-Fi network. No strong relation could be shown in this paper between the positioning error and the number of AP scanned, although it appeared that more gave better accuracy, or the overall variability of the SS. However, these appear to be valid hypotheses and further specific investigations will be conducted using a positioning algorithm developed at UNSW. Further work will also include integration of these existing Wi-Fi positioning technologies with GPS when GPS only cannot deliver a pin point position (only 2 or 3 satellites in view). REFERENCES Ekahau Inc. Professional Services (2008), Ekahau RTLS Guide: Requirements and Best Practices for Ekahau RTLS 4.3 Ladd AM, Bekris KE, Rudys A, Marceau G, Kavraki LE, Dan S (2002) Robotics-based location sensing using wireless Ethernet, Eighth ACM International Conference on Mobile Computing & Networking, Atlanta, Georgia, USA, 23-28 September, pp. 227-238 Li B, Wang Y, Lee HK, Dempster A, Rizos C (2005) A new method for yielding a database of location fingerprints in WLAN, IEEE Proceedings Communications, vol. 152, no. 5, pp. 580-586. Li B, Salter J, Dempster A, Rizos C (2006) Indoor Positioning Technique Based on Wireless LAN, First IEEE International Conference on Wireless Broadband and Ultra Wideband Communications, Sydney, Australia, 13-16 March, paper 113