Virtual reality and Immersive Media

Size: px

Start display at page:

Download "Virtual reality and Immersive Media"

Erik Tate
5 years ago
Views:

Jingfei Lin (Jade) Phase 2 Paper Data Visualization In The Community November 8, 2017 Virtual reality and Immersive Media Visualization and understanding of how

1 Jingfei Lin (Jade) Phase 2 Paper Data Visualization In The Community November 8, 2017 Virtual reality and Immersive Media Visualization and understanding of how immersive experiences like virtual reality activates physical and physiological stimuli, with community partner Associated Press and Multimer. Photo courtesy of Associated Press 1

2 Introduction Virtual reality has already been booming in games, entertainment, and media, journalists around the world are also embracing this technology as a new way of storytelling. The most intriguing potential of VR in journalism is the immersive experience that VR provides, which has the ability to take the audience into the location itself, witness an event and see augmented information at the same time. The traditional journalism makes people easy to hear a terrible piece of news and simply write it off, and thus some news lose their power when the viewer is just watching them on a comfortable couch. However, both the physical and physiological experience would change when watching topics like war, natural disaster, political change in full 360 immersion. Associated Press (AP) is trying to investigate how virtual reality devices and content affect the viewer's physical and physiological stimuli, by conducting a study with neuroscientists at Multimer that measured participants engagement and brain activity when consuming immersive media content. This paper and the final data visualization will focus on these issues by looking at the datasets from the study and pinpoint emerging information. Collecting the data AP conducted the study on August 12, 2017, at Black Box Theater in New York University's MAGNET (Media and Game Network), which provided the motion capture system as well as different types of VR headsets. AP also teamed up with the neuroscientists at Multimer to track participants' mental states. There were 12 participants experiencing four VR stories in three types of VR devices. The stories included three 360 video stories which featured the fight against ISIS in Iraq (House To House: The Battle For Mosul), a cultural parade in New Orleans (The Second Line: A parade against violence), an exploration of the ivory trade in Thailand (Thai Elephants: Surviving in the 21st century), and an underwater experience exploring biodiversity and marine species (The Ocean Is Scary! TheBlu). The participants experienced each story wearing three different VR headsets (Google cardboard, Samsung Gear and HTC Vive) and also the motion capture suits which collected their upper body movement data. Meanwhile, the EEG brainwave sensors and heart-rate monitors would be recorded as well. 2

3 After experiencing each story, the participants were also asked to fill out surveys in the Multimer data app about how the story affected their emotions and the sparkling moments they could recall from the story. Cleaning the data The three datasets (EEG, heart rate and motion) were cleaned, parsed, and analyzed using several Python analysis libraries by the scientists at Multimer and there are two available datasets to use. One is the motion capture data which is exported to both.csv and.fbx file, the other one is the Multimer data in.csv format. The motion capture data has the x, y, z-axis positions of many different joints where the motion sensors were attached to the participant's body parts, and the MoCap data were exported in 15 FPS. And the Multimer data contains the participant's standing position, heart rate, EEG signal which falls into four categories (powerful, stimulating, fascinating and restorative). For now, the two datasets are not yet combined together regarding each participant's motion and psychology states synchronously. But later they would be integrated into one file for each participant using the same timestamp. Another issue with the motion capture dataset is that there are some unlabeled joints data that need to be determined which body parts and/or which participant they belong to, since each individual does not have the identical number of joints data recorded. Understanding the data For measurement of the psychology effects caused by VR, Multimer s data scientists used custom algorithms to identify four kinds of key sentiments by looking for specific balances of the brain waves associated with attention and relaxation, as well as analyzing them against heart rate and verbal feedback. The four key sentiments are: Open-mindedness: Associated with an individual being attracted to a topic, but not alarmed. Fascination: Associated with a relaxed interest in a topic. Stimulation: Associated with an individual being more attentive than they are relaxed. Power/Intensity: Associated with the lasting impact of the experience. 3

They have found out that regular VR headsets drove higher levels of stimulation and intensity, and the stories involved in war and conflict could evoke higher engagement compare to stories with

4 They have found out that regular VR headsets drove higher levels of stimulation and intensity, and the stories involved in war and conflict could evoke higher engagement compare to stories with discovery and scientific topics. Also, experiencing a roomscale VR story could drive higher levels of open-mindedness and fascination compared to other VR headsets which people could not move around quite much. However, they haven't studied how these VR devices and contents affect physical motions, and also the comprehensive understanding of both the physical and psychological states when viewing the VR stories. There was also a prototype data visualization created by our fellow classmate Pan Pan, who has taken this class last year. But from the communications with the AP specialists, they think it looks cool but it causes a lot of confusion like what the color of the traces means, how are the temporal and spatial changes of motions presented, etc. They are looking for some visualizations that could be more intuitive and easier to understand. Pan Pan s prototype of visualizing the data 4

Historical precedents case study In order to study how to visualize both the motion and emotional states data synchronously, and represent them in a spatio-temporal structure, I found the case Child

5 Historical precedents case study In order to study how to visualize both the motion and emotional states data synchronously, and represent them in a spatio-temporal structure, I found the case Child development: HouseFly and WordScape quite inspirational [1]. Professors Deb Roy and Rupal Patel began an investigation in their own family with the birth of their first child to study child development as it occurs in the home. They installed a camera and microphone in the ceiling of every room of their house and recorded the majority of their child s waking experience for the first three years of life, resulting in a dataset of 80,000 hours of video and 120,000 hours of audio. Between 2009 and 2010 Philip DeCamp developed the a software tool to visualize and browse this massive dataset called HouseFly, in collaboration with Deb Roy, director of the Cognitive Machines group at the MIT Media Lab. Instead of displaying each stream of video separately, HouseFly combines them to create a dynamic, 3D model of the house. The user can navigate to any location in the 5

6 house at any time and get a better sense of what they would have seen and heard if people had actually been there. The application allows one to move in space and through time to examine the 80,000 hours of video. At the bottom, the timeline offers another way to navigate the content, including the ability to add notations in time about words of interest in the transcripts of the speech environment of the child. The above image shows twenty minutes of motion by the child (red) and the caregiver (green) which are represented as traces rendered in space. To examine the temporal dimension of the motion, one can switch the view to the side and the traces will be ordered vertically, with earlier times at the bottom, allowing a chronological view of interactions (bottom). Ideas identification of visualizing the data 6

7 As Munzner illustrates in Visualization analysis & design, there are four nested levels of design: domain situation, task and data abstraction, visual encoding and interaction idiom, and algorithm [2]. The task and data abstraction level addresses the why and what questions, and the idiom level addresses the question of how each of these levels has different threats to validity, so it s a good idea to choose the validation method with these levels in mind. To answer the why and what questions, reflections on what AP needs to understand is crucial for the design. During our skype meeting, they mentioned showing three frames of participants' data using various VR devices at the same time, to see how people moved differently watching the same story. In this case, a traditional clean line graph or bar chart that sum up everything would not be helpful, since these forms could not show human motions intuitively. Besides the devices themselves, they would also like to see the variations between different stories and how each individual's familiarity with VR devices affect their overall experiences. To combine all of these aspects, an overall statistical graph and several interactive spatio-temporal structures would be a better solution. For representing the motions of the participants, I am considering about two possible ideas. One is importing the data into motion builder and unreal, then export them to several videos and add Multimer data visualizations as upper layers; the other one is visualizing them using three.js by importing the.bvh files, in this way, I have more freedom to manipulate the visuals that could be combined with Multimer data at the same time, however due to the massive amount of data, the performance of the animations could not be promised. AP specialists also mentioned sharing the work during production so that they could check out conveniently online like google docs. In this case, visualization in python jupyter notebook or big chunks of videos won't be a good solution. While web browsers continue to get faster and improve in functionality [3], an interactive web application that enables the community partner to view and play with would be helpful. 7

Visualizations, Rockport Publishers, 2006. ProQuest Ebook Central [2] Munzner, Tamara.

8 Possible sketches Timeline Works Cited [1] Meirelles, Isabel. Design for Information : An Introduction to the Histories, Theories, and Best Practices Behind Effective Information Visualizations, Rockport Publishers, ProQuest Ebook Central [2] Munzner, Tamara. Visualization Analysis and Design, CRC Press, ProQuest Ebook Central [3] Yau, Nathan. Visualize This : The FlowingData Guide to Design, Visualization, and Statistics, John Wiley & Sons, Incorporated, ProQuest Ebook Central 8

Bring Imagination to Life with Virtual Reality: Everything You Need to Know About VR for Events

Bring Imagination to Life with Virtual Reality: Everything You Need to Know About VR for Events 2017 Freeman. All Rights Reserved. 2 The explosive development of virtual reality (VR) technology in recent