Virtual Podium with HTC Vive

Transcription

1 1 CS294W Final Paper Team Soapbox Jesse Min(jesikmin, ), JeongWoo Ha (jwha, ), Min Kim(tomas76, ) Virtual Podium with HTC Vive Abstract Public speaking is a difficult task for many people because speakers can be intimidated by speaking before large groups and speaking in new environments. To improve speaking ability, speakers would benefit from getting feedback, but it is not feasible for most people to receive constant feedback. Our project tries to solve the problem by creating a Virtual Reality application with diverse virtual presenting environments with augmented features such as custom powerpoint slides and speaker notes. To give meaningful speech feedback, Hound SDK and Fitbit API are incorporated to provide automatic feedback of verbal tics and heart rates. I. Introduction In order to make speech practicing more effective, we have created a virtual presentation application with which a speaker can practice: Virtual Podium with HTC Vive. In this VR environment, speakers can experience the circumstances in which they are going to give a speech in advance and thereby become easily be accustomed to the setting and familiarized with a large audience as well. In addition, our VR app provides personalized speech feedback such as report on heart rate and verbal tics using Natural Language Processing and third party APIs. With Hound SDK, we are able to convert voice into text and search the verbal tics such as like, and you know. With Fitbit API, we can retrieve the heart rate of a user during mock presentations. This paper is organized as follows. Firstly, some of related works that are relevant will be reviewed. Secondly, we will provide high level and detailed description of our application. Thirdly, the

2 2 process and results of user study will be discussed. Fourthly, conclusion and possible future work will be provided. Lastly, we will discuss the things we have learned throughout the project. II. Related Work There are a number of web/mobile/standalone speech practice applications. Some mobile applications like Articulation Station are aimed at comparing pronunciation of the users to the standard pronunciation and provides analysis on in which part the user should correct his or her pronunciation. Most relevant apps are aimed at people with special needs- patients who particularly need speech therapy for stroke, aphasia, and autism. In addition, most of the existing speech practice or analysis applications provide users intensive feedback, such as how long they have arbitrarily halted during their speech and overall pace of the speech. In the field of virtual reality, there has been applications that provide users extensive and realistic vicarious experience, but more, there are VR applications that make people feel as if they are in the Star Wars movie or as if they are standing in the middle of the Sahara.. However, there has not been any substantial effort to combine these two fields of technology to provide an app for public speaker who would like to practice and improve their ability as well as receive feedback. We were motivated by the idea that if we utilize open APIs such as Hound SDK and Fitbit API, we could give people speech analysis and rich feedback as existing commercial speech practice apps do. We can also render virtual conference room or 360-video recorded environments in virtual reality (more precisely on HTC Vive HMD). As a result, we decided to integrate those two state-of-art technologies to offer users with new paradigm of speech practice in VR machine.

3 3 III. High-level Project Description Each execution thread of Virtual Podium consists of three major parts: entry stage, practice stage, and analysis stage. The below Figure 1 describes the three stages in more detail. Figure 1. Program execution stages In the entry stage, after a user runs the program, he first needs to sign into the app so that the program can retrieve all the personalized presets and basic personal data from the web server. Then, the user can select between two practice modes virtual conference room mode and real room mode rendered by 360-video. We currently have rough mockups for these but still need to complete the work in the future. In the practice stage, if the user chose to practice in 360-video mode, he can make a choice among many options such as small classroom setting and large auditorium setting. The user can actually begin practicing his or her speech from this point. On the HTC Vive head mounted display (or HMD), the user will be able to view simple statistics including time and heart rate on real-time basis. In addition, the user can provide a typed script and powerpoint slides to the program beforehand so that he can view the script and slides in the HMD while practicing and move onto the next sentence or slide by giving a

4 4 trigger with two HTC controllers. The user can also remove the text script, if he wants to, by holding his head up and make the text appear again by bowing his head slightly. During the analysis stage, as soon as the user finishes practicing, the program utilizes Fitbit API to analyze the overall heart rate. It also employs Hound SDK to report how many and often the user unconsciously exhibited verbal tics such as like and you know during the practice. After fetching HTTP response from those APIs, Virtual Podium displays a neatly organized report onto its web dashboard personalized for each user. IV. Detailed Project Description Vizard was used to build the main virtual reality application. The advantage of using Vizard is that many Python libraries as well as Python scripts can be incorporated into our application. To model 3D conference room, we used predefined models in Sketchup and modified them according to our needs. Our main feature of the program consists of practicing speech in the modeled virtual environment with 3D rendered powerpoints and script. To control powerpoint slides and script, we utilized two HTC Vive controllers. Vizard automatically parses user provided powerpoint slides and script to render to the virtual environment. When the user clicks the right controller, the powerpoint slide will move to the next slide. Similarly, the next line of the speaker note will be rendered if the left controller is clicked. In addition, we rendered a 3D timer in the virtual environment to show the user how long the time elapsed. The below two figures illustrate how the rendered powerpoint slides and speaker notes look like in virtual environment.

5 5 Figure 2. Screenshot of custom speaker notes in virtual modeled environment. Figure 2. Screenshot of powerpoint slides in virtual modeled environment. To provide the user with useful feedback, we incorporated Fitbit API and Hound API. The Fitbit API is used to provide the user with heart rates recorded during the presentation, and Hound API is used to run NLP algorithms. On the desktop that runs the Vizard application, a set of GUI elements is displayed to enable call to Hound and Fitbit API. When the user clicks a Hound button, a Hound API

6 6 will be activated to record user s speech into text. If the Fitbit button is clicked, the heart rate from the presentation start time to presentation end time will be recorded. To avoid interference with the main virtual reality program, Fitbit API and Hound API calls are made through multithreading. Appropriate tokens and authorizations are provided through REST API calls to access personal data from the server. We discovered that single thread calls will freeze the program. The program will exit when the analysis is done. The analysis result is stored in the MySQL database to be used by the Dashboard web application. Dashboard web application is built with RoR, and it will be hosted through Heroku with Postgres database. We enable user signup and user signin to access personalized analysis data. D3.js is used to display the Fitbit and Hound analysis data into the web dashboard. We allow the user a few different modes with different room sizes. We filmed a 360 video of a large classroom and small classrooms, both filled with students to use as our virtual reality application (a CS106A review session, a CS161 class, and a CS294S class were filmed with the consent of instructors). The videos were converted to sphere format and rendered in HTC virtual environment. A user can choose which room he or she wants to present in and start presenting in the virtual environment selected. The limitation of this approach is that the users are not able to walk around in the virtual environment while the 3D modeled virtual environment allows people to move in the virtual space.

7 7 Figure 3. Application multi threading execution flow V. User Research Our user research strategy consisted of two different parts first, each user study participant was interviewed on overall impression toward the application and completed a survey (Appendix B) and second, we analyzed every user s heart rate and voice recorded. We conducted the user study on 8 potential users, five recruited and three of us. Each subject of study of five recruited users was a Stanford undergraduate student from different majors and years. The participants of the study were debriefed of instructions such as how to calibrate the focus of HMD and how to use HTC controllers. Every one of them practiced their own speech in three different modes: virtual conference room mode, 360-angle small lecture room, and 360-angle large lecture hall. We then asked users to complete the survey form, conducted interview on general comment on our app, and analyzed on collected heart rate and verbal tick data. Figure 4 displays heart rate change with different virtual environment mode for each user. Six out of eight participants or 75% of participants showed tendency of increasing heart rate as the

8 8 environment was shifted from virtual conference room mode to 360-video small lecture room and from 360-video small lecture room to 360-video large lecture hall mode. Except subject 4 and 7, a participant s heartbeat became faster or, in other words, a participant felt more anxiety as room changed from a virtual room to a more realistic setting and a small room to a spacious lecture hall. This heart rate data collected by Fitbit on a real-time basis was coherent with the user's explicit remark during interview and survey that virtual room was interesting yet unrealistic, while 360-video setting, particularly 360-video large lecture hall mode, was surprisingly realistic, compelling, and effective not only because of the size of the room but also due to the larger size of audience. Figure 4. Average heart rates of subjects according to virtual environments Besides the heartbeat, we converted voice data of testers into text on a real-time basis using Hound SDK. Such processed text data was linearly searched to find certain verbal ticks including like, well, and you know. The example statistic for subject 6 is shown below:

9 9 Figure 5. Verbal tics for subject 6 Users evaluated this verbal-tick-specific speech feedback as one of the most useful features in this application as most of them were not aware of their occasional habits such as like while they were speaking. However, there was still a limitation as well. As our tester recruiting was done during the week 9, one of the busiest weeks for every Stanford student, we could not force testers to prepare for 5 or 10 minutes long speech. Because users could give only 2 to 3 minutes speech at most, capturing verbal tick in such short speech sometimes did not result in meaningful result. Nonetheless, according to the survey, users were satisfied the feature. In addition, many users provided with several suggestions for future work in their survey response. Some people suggested of employing more realistic audio sound along with the current realistic visual environment for instance, users are able to listen to their sound with appropriate echo/ambient sound through earphones. Others suggested for a high-resolution 360 video. From our experience, 1080K resolutions video, which is usually considered as a fairly high-resolution video, was not enough for 360 setting where sphere mapping onto HMD is required. Our Samsung 360 Gear was acceptable for our prototype but not sufficient for a complete application in the future.

10 10 VI. Conclusion VR technologies have become more widely approachable to the general public. There are much more ways the VR technologies could be used besides gaming and entertainment. There are already some implementation of VR technology in the medical field - to cure the users with specific phobias. Although public speaking is one of the many skills that is important and helpful for many people, many people have difficulties acquiring the desired skills, VR programs could help them reduce public speaking anxiety and to help them practice before an actual speech. Leveraging various existing technologies, such as Hound SDK and Fitbit API, we have and created a virtual environment by which users can practice public speaking skills anywhere at anytime. According to our user research, we have discovered that people are indeed interested in using such a technology and they do feel that virtual environments can partially replace the actual presentation environments for practice. This project is just a first step in what could potentially become a very powerful application. What we could make out of it really depends on how we create new demands and meet the future needs. Virtual reality could change behaviors of human beings, and this powerful methods could and should be used wisely to assist the users to change for the better. As we will be able to incorporate other high-end technologies such as haptic sensories, it would be possible to create more realistic virtual speech-practicing environment to relieve public speaking anxiety. VII. Future Work In this section, we will discuss the limitations of our current technology and four possible future improvements that can make our application more feasible and effective. Firstly, more realistic rendering of the real-world environment where the users would actually give a speech would be crucial. Although we have created virtual 3D environments using SketchUp and 360 videos to make them look real, our app still has some limits. One crucial addition would be to enable user to interact with the virtual audience in HTC Vive. What makes speakers in real life get really

11 11 nervous is the intimidati audience. If we could program the virtual environment to have audience who react depending on the user s speech, volume, gestures, and etc., it would be more helpful for the users to get a good better of what it is like to give a pitch in front of the anticipating audience. Secondly, another powerful addition would be the more accurate integration of various feedback features. Our current integration of Hound and Fitbit is just a start of what we could provide through the virtual speech practice. We could implement gestures trackers through Leap Motion to observe the user s hand gestures and walking patterns on the stage; we could implement eye gaze trackers to analyze how much time the user spends to engage with the each audience in the room; we could also record the user practicing through the VR machine so that the user would see himself. Thirdly, enabling multiple users to join the same virtual environment to practice public speaking would be another key addition to this project. Multiple users in the same virtual environment could help people who are not physically together to practice ahead of time with the anticipated speech environment. Lastly, we could potentially have a platform where people can share virtual speech environment that they created to filmed. The more options the users have, the better for them to experience diverse environments with dynamic audiences. VIII. Discussion The VR industry has made some great progress in the recent years, and we currently have many technologies that are implementable with the VR techs in many ways. There are powerful software tools such as Unity, Unreal Engine, and Vizard to create high quality virtual 3D environments. Furthermore, 360-video technology is available and now being used widely in various fields. Integrating real-world videos of various conference rooms, lecture halls, auditoriums helped the users be familiar with the actual environment much better than the 3D modeled room. Accessing these real environments through

12 12 virtual reality machines helped many testers to feel comfortable when they actually gave speeches in the real environments. We were able to integrate other helpful features that could help the users to get fruitful feedbacks after the practice. Using Hound SDK, we were able to detect how many times the users used the filler words such as like, um, and ah. Providing with the users at which point of their speech they used the filler words helped them to locate where they were feeling more anxious and uncomfortable about the contents. Another technology we implemented was Fitbit - analyzing the user s heart rate aside with their speech helped them focus which part of their speech to focus on. In addition, we could potentially use the heart rate data to analyze how we could help the users prepare in certain speech environments. Another key aspect of the project was the user testing and user research. We wanted to get comprehensive feedback from people with different academic backgrounds. We also wanted to analyze the psychological effects of this project. There were many useful feedback such as dual controller being too burdensome as they would not be holding the controllers in a real speech environment. Nevertheless, many subjects really liked the options of choosing the pre-filmed real world 360-video mode and the virtual 3D model mode. Some of them said that multiple practices with the virtual environment would actually help them reduce anxiety about public speaking. However, more accurate measures of heart rate and other scientific measurements are needed to prove that this application would actually help reduce public speaking anxiety. IX. Team Member Contributions All of our members equally contributed to the project for the entire quarter. We all worked together to tackle major issues regarding Vizard IDE, sphere mapping of 360 video, integration of Hound SDK / Fitbit API with the HTC Vive application, and so on. Each member still had some particular portions of the project more dedicated than others. Jesse Min put most of his time in sphere-mapping 360-video onto the HMD and implementing Fitbit API and Hound SDK for measuring

13 13 heart rate and speech analysis. JeongWoo Ha was dedicated to setting up the Sketchup environment and rendering it smoothly onto the HTC HMD through Vizard. He put a great deal of effort into fine-tuning visual details in the VR environment, such as coordinating and display of text script/slides. Min Kim committed most of his time to implement motion sensing part of VR application and sending trigger to the program with HTC controller. He also worked a great deal on streamlining the VR application with the analysis stage (Fitbit API / Hound SDK) including the web dashboard.

14 14 X. Bibliography 1. Hound SDK Documentation: 2. Houndify Python Github Example: 3. Fitbit API Documentation: 4. Fitbit API Blog Tutorial: -api/ Vizard 5.0 Documentation: 5. PythonMulti-threading: 6. Samsung GEAR 360 Camera How-to: XI. Appendix A. JSON Response of Hound SDK and Fitbit API

15 15 B. User Research Survey User Test Questionnaire Name Gender Age Personal Speech Experience Q1. Do you usually feel nervous giving a speech in front of the public? Q2. What part of the public speaking are you most uncomfortable with? Q3. Have you completed Stanford PWR2 course? No Yes Q3-1. If yes, how did you practice your final presentation? Q3-2. If yes, do you think the app might have helped your PWR2 presentations? Q3-3. If No, do you think this app will help your future PWR2 presentations? Virtual Reality Experience Q4. How real was the virtual conference room? Q5. How real was the small 360-video rendered room? Q6. How real was the large 360-video rendered room? Q7. Did the large 360-video rendered room make you more nervous than the small room? Q8. Would you use the 360-video rendered room or the 3D virtual conference room for practicing?

16 16 Overall Experience Q9. Will you use this app if this app is officially released in the future? Q10. What were some advantages of this app? Q11. What were some drawbacks of this app? Q12. Please briefly summarize and comment on overall impression after using this VR speech practicing application. (Any suggestion is also welcomed.) Q13. What additional features would you like to see in this application?