This is the peer reviewed author accepted manuscript (post print) version of a published work that appeared in final form in:

Transcription

1 A study in virtual navigation cues for forklift operators This is the peer reviewed author accepted manuscript (post print) version of a published work that appeared in final form in: Pereira, Alexandre, Lee, Gun A, Almeida, Edson & Billinghurst, Mark 2016 'A study in virtual navigation cues for forklift operators' Proceedings of the 18th Symposium on Virtual and Augmented Reality, SVR 2016, article no , pp This un-copyedited output may not exactly replicate the final published authoritative version for which the publisher owns copyright. It is not the copy of record. This output may be used for noncommercial purposes. The final definitive published version (version of record) is available at: Persistent link to the Research Outputs Repository record: General Rights: Copyright and moral rights for the publications made accessible in the Research Outputs Repository are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognize and abide by the legal requirements associated with these rights. Users may download and print one copy for the purpose of private study or research. You may not further distribute the material or use it for any profit-making activity or commercial gain You may freely distribute the persistent link identifying the publication in the Research Outputs Repository If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

2 A Study in Virtual Navigation Cues for Forklift Operators Alexandre Pereira * IFCE Fortaleza Brazil Gun A. Lee HIT Lab NZ University of Canterbury Christchurch, New Zealand Mark Billinghurst School of ITMS University of South Australia Adelaide, Australia ABSTRACT Augmented Reality (AR) is a technology that can overlap virtual elements over the real world in real time. This research focuses on studying how different AR elements can help forklift operators locate pallets as quickly as possible in a warehouse environment. We have developed a simulated AR environment to test Egocentric or Exocentric virtual navigation cues. The virtual elements were displayed to the user in a HUD (head-up display) on the forklift windshield, fixed place in front of the user operator, or in a HMD (head-mounted display), where the virtual cues are attached to the head of the user. A user study found that the Egocentric AR view was preferred over the Exocentric condition and performed better while the HUD and HMD viewing methods produce no difference in performance. Keywords: Augmented Reality, logistics, forklift, navigation. Index Terms: H.5.2 User Interfaces; H.5.1 Multimedia Information Systems - Artificial, augmented, and virtual realities. 1 INTRODUCTION Augmented Reality (AR) is technology that aims to seamlessly blend virtual information with the real world [1]. Researchers have shown that AR can be applied in numerous applications in education, engineering and entertainment, and other domains. For example, doctors can use AR to see medical data inside the patient body [8] and architects can see unfinished buildings [13]. In this paper, we explore how AR could be used to improve the performance of forklift operators in a warehouse environment. Researchers have previously shown that AR can improve objectpicking performance [10], vehicle navigation [6], and training tasks [17]. Based on this research, AR should be able to help forklift operators by assisting with the following: 1.Pallet location: AR cues could visually identify a target pallet location in a warehouse. 2.Navigation: AR cues could overlay directional instructions to guide the driver through the warehouse. 3.Slot location: AR cues could highlight an empty slot in the warehouse where boxes could be placed. Figure 1 shows the initial concept for how Augmented Reality could be used to accomplish each of these three functions. For example, virtual boxes could be used to highlight pallet pick-up location or empty slots, while navigation arrows could appear on the ground to guide the operator to a task location. * pereira.alexandremagno@gmail.com gun.lee@hitlabnz.org mark.billinghurst@unisa.edu.au a) Pallet Location b) Navigation c) Slot Location Figure 1: Artists Concept of AR use in a Warehouse. 2 RELATED WORK There has been previous research in using wearable computers and AR for stock picking and AR for vehicle navigation that is relevant. In this section, we review key work in these areas. 2.1 Order Picking Driving a forklift around a warehouse and moving goods is part of the order picking process. Depending on the industry, logistics costs amount to 5 to 8 % of revenue [12] and order picking accounts for 55% to 65% of the total operational costs of a warehouse [2]. Therefore reducing picking costs and improving performance could result in significantly lower warehouse costs. The order picking process consists of two navigation phases: a) finding a path to the right shelf and b) picking an object out of the shelf location [14]. It is estimated that 50% of a picker s time is spent traveling from one storage rack to the next, and 35% of time locating and picking from the correct bin [11]. As shown in figure 1, AR could be used to support navigation to the shelf location and then indicating which object on a shelf should be picked. Improvements in task time for either of these areas could result in significant performance improvement. 2.2 AR For Stock Picking A number of research groups have explored if AR and wearable technology can benefit stock picking operations. In [4] researchers evaluated order picking assisted by four approaches: heads-up display (HUD); cart-mounted display (CMD); pick-by-light; and paper pick list. They found that using the HUD and CMD both produced significantly faster pick times than using either the paper-based system (by 20%) or the Pick-by-Light method (by 40%). There was also an 80% reduction in errors using HUD and CMD compared to the other two methods. They report that pickby-hud and pick by-cmd were superior on all metrics to the current practices of pick-by-paper and Pick-by-Light, but there was no significant difference between the two. In a similar experiment [16] the researchers tested the use of a head-mounted display (HMD) based picking chart over a traditional text-based pick list, a paper-based graphical pick chart, and a mobile Pick-by-Voice system. The HMD condition was significantly faster (by 10% - 40%) than the average time per task when using any of the other methods. Iben et al. [5] compared a text-based paper pick list to a pick list rendered on a HMD, using additional context information. They found a similar performance benefit, with the HMD users taking an average of 5.3 seconds per object pick and the paper user requiring 6.3 seconds, and the HMD users making 40% fewer errors than the paper users.

3 Overall, these results show that using an HMD with a 2D graphical interface can produce a significant improvement in stock picking applications. This is because the bin picking information is always in view and the operator s hands are free to perform the picking task. It seems that AR interfaces in a HMD may not provide a significant benefit due to the difficulty of locating target bins on a small field of view display and making the virtual cues appear precisely overlaid on the real world. 2.3 AR for Vehicle Navigation A final area of related work is using AR cues to improve vehicle navigation. Although most of this research has been conducted on cars or trucks driving outdoors, the lessons learned could be applied to forklifts driving in an indoor warehouse. Research has been done on using the windshield as a HUD for vehicles, specifically to provide car navigation cues. For example, Tonnis et al. [15] created an interface that projects AR cues on the windshield that align with the road surface. Narzt et al. [7] go beyond this to not only show navigation arrows, but also cues highlighting freeway exits, points of interest such as petrol stations, and alerts to potential hazards such as pedestrians. Some research has compared AR to non-ar interfaces for car navigation and safety. For example, Park et al. [9] compared driver performance when using an AR HUD cues showing arrows on the road, to 2D icons on the HUD. They found that drivers had a significantly faster response time to lane changing information when shown with the AR cues. The University of California conducted a study with the windshield as a display and using AR elements to convey an alert when the driver exceeded the speed limit [3]. They also found that users had a faster reaction time using the HUD than when using an in-car display, due to the alert being easily visible in their field of view. 2.4 Summary In summary, previous research has shown that 2D interfaces on wearable displays can improve performance in stock picking, but AR interfaces on HMDs may not be as effective. However, using a vehicle HUD to show AR cues could improve navigation and responsiveness to alert information. This research implies that virtual cues presented in a HUD in a forklift could improve operator performance. However, there has been no previous work conducted to explore this, and especially not comparing performance with an AR HUD to an AR HMD interface. 3 PROTOTYPE DESIGN In order explore the use of AR for forklift operations we decided to develop a simulated AR interface in an immersive virtual environment. This was because of the difficulty of implementing an AR system on a real forklift due to technical challenges (e.g. limitations in vehicle tracking, AR display, etc) and being able to get access to a large group of forklift operators for testing. To develop a simulated AR experience, five key technological components were needed: (1) a simulated AR interface, (2) a 3D warehouse model, (3) support for a display device, (4) support for input devices and (5) vehicle control code. The simulation was developed in the Unity3D game engine 1, using a 3D model of a warehouse, a forklift and a virtual human operator, purchased from the TurboSquid website 2 (see figure 2). Figure 2: 3D models used in the simulation. We used the Unity NGUI plug-in 3 to enable easy development of user interface elements, and the NGUI HUD Text extension 4 to add text to the screen. Work had to be done to add shelving, lighting and other features to the simulated virtual warehouse. The main goal was to create a simulated AR interface attached to the virtual forklift model in the warehouse environment. This was done by adding a virtual camera view in the Unity3D simulation that was attached to the user s viewpoint and then including AR interface objects into this view. Figure 3a shows the Exocentric AR view from the operator's position and Figure 3b shows the Egocentric AR view. The key interface elements are: The Warehouse Map: a semitransparent map that shows a view from above the forklift and that follows it. Orientation Arrow: Virtual cue overlaid on the warehouse floor showing the path to a destination. Orientation Label: Indicates the box number to find in the warehouse (a) Exocentric Map View (b) Egocentric Arrow View Figure 3: Simulated AR interfaces. In the Egocentric view, the AR elements are directly displayed to the subject from a first person perspective. In the Exocentric view, the virtual navigation element is a map that shows a top down view from above of the forklift, including the arrow. The simulated AR interface was viewed in an Oculus Rift DK1 head mounted display that provided a 3D view and a fully immersive virtual reality experience. User input was captured from a Ferrari GT Thrustmaster steering wheel and pair of pedals and a software component was developed to simulate driving a forklift in response to the user input (see figure 4). Figure 4: A user operating the simulator

4 4 EXPERIMENTAL EVALUATION We conducted an experiment to compare using different AR elements for navigation inside a warehouse; comparing an AR HUD to an AR HMD interface in the forklift, and the use of an egocentric versus exocentric display for map information. 4.1 Experimental Design In the user study, the subject is supposed to find three boxes inside a warehouse using a different set of conditions without touching any of the boxes spread on the ground. The dependent variables in this experiment were: Box Time: The time needed to reach each target box Total Time: The sum of the time to reach all boxes Collisions: The number of times that the user collided with random boxes spread on the floor. The subjects completed the task in the following conditions: HUD: The AR display simulation is attached to the forklift windshield. If the user moves his head, the virtual cues will stay in the same position (figure 5). HMD: The AR elements are attached to the head of the subject. If the user operator rotates his head, the virtual cues will also rotate (figure 6). Egocentric: The AR elements are directly displayed to the subject from a first person view (figure 5 and 6). Exocentric: The AR elements are available inside a map that shows a view from above the forklift (figure 7). Figure 5: HUD with Egocentric cue condition. Figure 6: HMD with Egocentric cue condition. Figure 7: HMD with Exocentric cue condition. There were four test scenarios where three target boxes were placed at different locations in the warehouse, with a similar distance between them. The experimental task was to locate the three boxes in order using the AR navigation cues. With the four test scenarios and four interface conditions, there were sixteen combinations of these elements. In order to avoid the bias of learning effect, the order of the conditions for each subject was varied using a Latin Squares technique. To start, subjects were given a consent form and a demographics questionnaire. Next, they were told the experiment tasks and the differences between each test condition. Subjects were then placed in the virtual forklift simulator and taught how to drive until they were comfortable with the controls and could find a highlighted box and touched it with the forklift forks. Once the subjects understood how to control the vehicle, they were allowed to begin the experiment. When the experiment started, the subject had to follow the AR navigation cues to locate a highlighted box and touch it with at least one fork of the forklift. When the subject touched the first highlighted box, the AR navigation cues pointed to the second highlighted box. Then, when it was touched by one of the forks, the navigation cues pointed to the third highlighted box. Finally, when the user reached the third box, the time needed to reach each box, the total performance time, and the number of times that the subject hit boxes on the ground were recorded. Each subject had to do this task four times, once for each AR display condition, each time in a scenario where the target boxes were placed in different places. After the subjects finished each trial, they filled out a questionnaire about the usability and efficiency of the tested simulation condition. When the subjects finished all four tests, they were asked to make a ranking of the conditions and explain why they chose this particular order of preference. 4.2 Results In the results the Egocentric condition is referred as the letter G, Exocentric condition as the letter X, HUD condition is shown as U and HMD condition is referred to as M. There are four combinations of conditions that were tested in the experiment: MX: HMD interface with an Exocentric view. MG: HMD interface with an Egocentric view. UX: HUD interface with an Exocentric view. UG: HUD interface with an Egocentric view. There were 16 participants, 8 men and 8 women, with an average age of 23.4 years old User performance From the performance quantitative data we found: - There was no performance time difference between the different AR display conditions (HUD vs. HMD). - Users navigated significantly faster in the egocentric cue condition than the exocentric condition. - There was no significant difference in the number of boxes hit on the ground between the different conditions. Table 1 shows the task completion time in seconds for the subjects to complete navigating to all three boxes. A two-way repeated measure ANOVA test showed that there was no significant main effect of the HUD vs. HMD factor (F(1,15)=0.03, p=.87), but there was a significant main effect on the Egocentric vs. Exocentric factor (F(1,15)=4.98, p<.05). No significant interaction between the factors was found (F(1,15)=0.07, p=.80). Descriptive statistics show that participants took less time to complete the task in conditions with Egocentric condition (Mean= secs) compared to those with an Exocentric condition (Mean= secs). Table 1: Task completion time. Condition Mean Time (s) Std. Dev. MX MG UX UG

5 Table 2 shows the number of times that users collided with boxes on the ground as an indication of how accurately they were driving. A two-way repeated measure ANOVA showed that there was no significant main effect of the HUD vs. HMD factor (F(1,15)=1.15, p=.30), and no significant main effect of the Egocentric vs. Exocentric (F(1,15)=0.01, p=.93). No significant interaction between the factors was found (F(1,15)= 0.32, p=.58). Table 2: Number of hits on boxes on the ground. Condition Mean Std. Dev. MX MG UX UG Questionnaire: Rating After each condition, a questionnaire with the following ten rating questions was answered on a Likert scale from 1 (Strongly Disagree) to 7 (Strongly Agree): Q1. It was easy to navigate to the target boxes Q2. It was easy to learn how to use the virtual cues Q3. The navigation cues were useful to the task Q4. The navigation cue was intuitive Q5. The navigation cue was natural Q6. The navigation cue was effective Q7. The navigation cue was mentally stressful Q8. The navigation cue was physically stressful Q9. The navigation cue deviated my attention from the boxes on the ground Q10. I was able to drive the forklift well Table 3 shows the average results of the rating questions across all the conditions. The first 4 questions are about ease of use showed the mean values being much higher than the expected average (4.0), indicating that the subjects found that the navigation cues were actually helping them to complete the experimental tasks. Questions 7 and 8 ask about the stress of using the interface, with a mean much lower than the average, showing that the participants experienced little stress while doing the experiment. Table 3: Likert scale rating results with the Friedman test p-value. Quest. MX MG UX UG p-val While descriptive data suggests that the Egocentric conditions (UG and MG) had a better rating in most questions (e.g. the UG condition was the best rated in 9 out of 10 questions), a nonparametric Friedman test was conducted for each question but no significant difference was found Questionnaire: Ranking After participants finished all four conditions, they were asked to rank the order of preference of the conditions using the values from 1 (best condition) to 4 (worst condition). From the ranking quantitative data the following results were found: - Users prefer the Egocentric over the Exocentric condition. - There was no preference difference between the different AR display conditions (HUD vs. HMD). Table 4 shows the mean rank for each of the conditions. Friedman indicated there was a significant difference between the conditions with a Chi-square value of and p=.006. Descriptive statistics show participants ranked Egocentric conditions higher (Mean=1.91) compared to those with an Exocentric condition (Mean=3.09). Table 4: Ranking results. Ranking Condition Mean Rank MX 3.00 MG 1.94 UX 3.19 UG 1.88 In order to compare each combination of conditions in the experiment, post hoc tests were performed with Wilcoxon signedrank test with a Bonferroni correction applied. The results showed there was a significant difference between Egocentric and Exocentric conditions while no significant difference was found between the other possible conditions in the experiment. Table 5 shows the post hoc test results and the associated Z values. The test for Ego vs. Exo had a significant result, and UX vs. UG conditions also showed significant difference with the Bonferroni corrected a significant level set at p <.008. Other post hoc tests did not have a significant result. Table 5: Post hoc tests for ranking results. Test Condition Z-Value p-val. Ego vs. Exo * HMD vs. HUD MX vs. MG MX vs. UX MG vs. UG MG vs. UX UX vs. UG * 4.3 Qualitative Feedback Participants were asked to give qualitative feedback with the reasons for their rankings that they made at the end of the experiment. One of the main reasons for the Egocentric conditions being ranked the best was because the virtual arrows on the ground were bigger, clearer, and centered in front of the subjects view. One of the subjects summarized the predominant impression of the participants: "The arrows on the ground were the best due to being bigger and clearer so easy to follow. The map felt slightly disconnected from the task and took additional mental processing to read." The Exocentric conditions were ranked lower because they required additional mental processes to understand how to reach the highlighted boxes. Users had to mentally map their position and orientation in the virtual map to what they were seeing in front of them. The map view also blocked part of the user s view.

6 Participants tended to make more errors in the Exocentric conditions. Many times the participants entered a wrong aisle in the warehouse because they were concerned about not hitting boxes on the ground. When they tried to re-orient themselves, often they had already missed the best path and had to go another way. In the Egocentric conditions, this did not happen as they could see the virtual orientation arrows all the time. Between two Exocentric conditions, the participants preferred the one with the head-mounted display because they could continue to see the map even if they turned their heads. One of the participants said: "The HMD map, which was moved along with the head was better than a fixed map because I was able to see it even when I was looking somewhere else." 5 DISCUSSION Overall, the HUD interface with an Egocentric view (UG condition) had the best results. The HMD interface with an Egocentric view (MG condition) also performed well, but slightly below the UG condition results. For most of the measures, the HUD interface with an Exocentric view (UX condition) had the worst results in both task completion time and questionnaire results. The HMD interface with an Egocentric view (MX condition) also had poor results, but slightly better than the UX condition. The results clearly show that the Egocentric conditions had the best results in both the quantitative and qualitative data, compared to the Exocentric conditions. Many participants said that the Egocentric scenarios were better because they could see the virtual cues clearly and require less mental effort. Between the Egocentric combinations, the participants preferred the HUD condition. This may be due to the fact that having an AR cue that is always in front of the point of view of the subjects no matter the direction the user participants are looking could be a little distracting. There was no significant difference in the HUD and HMD condition in both task performance and questionnaire results. This may be related to the fact that users spent most of their time looking straight ahead, making these conditions very similar. 6 CONCLUSIONS AND FUTURE WORK In this project we developed a simulated AR interface for a forklift operator in a warehouse, performing stock picking and movement tasks. The main lessons learned are that users felt that AR cues helped their performance and that Egocentric cues were more useful than Exocentric cues. We did not see a significant difference between the HMD and HUD display conditions, but this was probably due to the nature of the tasks and is something that could be explored further. There are a number of ways that this research could be extended in the future. First, adding moving forklifts to the scene would enable exploring how AR could be used to help with situational awareness of the driver s surroundings. Next, it would be good to test the AR simulator with real forklift drivers, and investigate how AR technology can enhance existing skills, or enable novice operators to acquire skills in a short time. Finally, we should make a real version of the AR interface that could be installed in an actual forklift. Although valuable lessons can be learned from the AR simulation, there is also important feedback that can only be collected from people in their real work environment. [2] Bartholdi, J., and Hackmann, S. (2009) Warehouse and distribution science release 0.89; Georgia Institute of Technology, Tech. Rep., January Available Online: [3] Doshi, A., Cheng, S., and Trivedi, M. (2009). A novel active headsup display for driver assistance. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 39(1), [4] Guo, A., Raghu, S., Xie, X., Ismail, S., Luo, X., Simoneau, J.,... & Starner, T. (2014). A comparison of order picking assisted by headup display (HUD), cart-mounted display (CMD), light, and paper pick list. In Proceedings of the 2014 ACM International Symposium on Wearable Computers (pp ). ACM. [5] Iben, H., Baumann, H., Starner, T., Ruthenbeck, C., and Klug, T. (2009) Visual based picking supported by context awareness: Comparing picking performance using paper-based lists versus lists presented on a head mounted display with contextual support. In ICMI-MLMI, Cambridge,MA, USA, November ACM. [6] Kim, S., & Dey, A. K. (2009). Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp ). ACM. [7] Narzt, W., Pomberger, G., Ferscha, A., Kolb, D., Muller, R., Wieghardt, J., Hortner, H., and Lindinger, C. (2003) Pervasive information acquisition for mobile ar-navigation systems. In Fifth IEEE Workshop on Mobile Computing Systems and Applications, [8] Navab, N., Feuerstein, M., &Bichlmeier, C. (2007). Laparoscopic virtual mirror new interaction paradigm for monitor based augmented reality. In Virtual Reality Conference, VR'07. IEEE (pp ). [9] Park, K. S., Cho, I. H., Hong, G. B., Nam, T. J., Park, J., Cho, S. I., & Joo, I. H. (2007). Disposition of information entities and adequate level of information presentation in an in-car augmented reality navigation system. In Human Interface and the Management of Information. Interacting in Information Environments (pp ). Springer Berlin Heidelberg. [10] Reif, R., & Günthner, W. A. (2009). Pick-by-Vision: An Augmented Reality supported Picking System. [11] Schwerdtfeger, B., & Klinker, G. (2008). Supporting order picking with augmented reality. In Proceedings of the 7th IEEE/ACM international Symposium on Mixed and Augmented Reality (pp ). IEEE Computer Society. [12] Straube, F.; Pfohl, H.-C.: (2008) Trends und Strategien in der Logistik 2008: Globale Netzwerke im Wandel, DVV, 2008 Bremen [13] Thomas, B., Piekarski, W., & Gunther, B. (1999). Using augmented reality to visualise architecture designs in an outdoor environment. International Journal of Design Computing: Special Issue on Design Computing on the Net (dcnet'99), 2. [14] Tompkins, J., White, J., Bozer, Y., and Tanchoco, J. (2003) Facilities Planning, [15] Tonnis, M., Lange, C., and Klinker, G. (2007) Visual longitudinal and lateral driving assistance in the head-up display of cars. In Proceedings of the 6th International Symposium on Mixed and Augmented Reality (ISMAR), pages 91 94, [16] Weaver, K. A., Baumann, H., Starner, T., Iben, H., & Lawo, M. (2010). An empirical task analysis of warehouse order picking using head-mounted displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp ). ACM. [17] Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), REFERENCES [1] Azuma, R. (1997) A Survey of Augmented Reality Presence: Teleoperators and Virtual Environments, pp , August 1997.