Playing with sound and gesture in digital audio games

Similar documents
Playing with sound and gesture in digital audio games. From prototype design to evaluation.

Adapting Videogame Interfaces for the Visually Impaired: A Case Study of Audio Game Hub

Access Invaders: Developing a Universally Accessible Action Game

Gaming Development Fundamentals

Xdigit: An Arithmetic Kinect Game to Enhance Math Learning Experiences

CompuScholar, Inc. Alignment to Utah Game Development Fundamentals 2 Standards

"From Dots To Shapes": an auditory haptic game platform for teaching geometry to blind pupils. Patrick Roth, Lori Petrucci, Thierry Pun

Individual Test Item Specifications

CompuScholar, Inc. Alignment to Utah Game Development Fundamentals Standards

Video Games and Interfaces: Past, Present and Future Class #2: Intro to Video Game User Interfaces

Game Designers. Understanding Design Computing and Cognition (DECO1006)

Online Gaming Support for Parents (source YHGFL) JE

MELODIOUS WALKABOUT: IMPLICIT NAVIGATION WITH CONTEXTUALIZED PERSONAL AUDIO CONTENTS

Waves Nx VIRTUAL REALITY AUDIO

An Introduction to ScratchJr

Interactive Simulation: UCF EIN5255. VR Software. Audio Output. Page 4-1

Interactive Exploration of City Maps with Auditory Torches

Human Factors. We take a closer look at the human factors that affect how people interact with computers and software:

Arcaid: Addressing Situation Awareness and Simulator Sickness in a Virtual Reality Pac-Man Game

Technology offer. Aerial obstacle detection software for the visually impaired

Interface in Games. UNM Spring Topics in Game Development ECE 495/595; CS 491/591

LCC 3710 Principles of Interaction Design. Readings. Sound in Interfaces. Speech Interfaces. Speech Applications. Motivation for Speech Interfaces

Controlling vehicle functions with natural body language

E90 Project Proposal. 6 December 2006 Paul Azunre Thomas Murray David Wright

Digital Media & Computer Games 3/24/09. Digital Media & Games

A Cross-platform Game for Learning Physics

Perception of room size and the ability of self localization in a virtual environment. Loudspeaker experiment

GAME DEVELOPMENT ESSENTIALS An Introduction (3 rd Edition) Jeannie Novak

Discussion on Different Types of Game User Interface

6 Ubiquitous User Interfaces

Fitur YAMAHA ELS-02C. An improved and superbly expressive STAGEA. AWM Tone Generator. Super Articulation Voices

ModaDJ. Development and evaluation of a multimodal user interface. Institute of Computer Science University of Bern

Rethinking Prototyping for Audio Games: On Different Modalities in the Prototyping Process

TrampTroller. Using a trampoline as an input device.

Physical Affordances of Check-in Stations for Museum Exhibits

ADVANCED WHACK A MOLE VR

Blindstation : a Game Platform Adapted to Visually Impaired Children

Leading the Agenda. Everyday technology: A focus group with children, young people and their carers

Design and Evaluation of Tactile Number Reading Methods on Smartphones

Individual Test Item Specifications

While there are lots of different kinds of pitches, there are two that are especially useful for young designers:

Level 3 Extended Diploma Unit 22 Developing Computer Games

COMS 465: Computer Mediated Communication

Blind Hero: Enabling Guitar Hero for the Visually Impaired

Introduction to HCI. CS4HC3 / SE4HC3/ SE6DO3 Fall Instructor: Kevin Browne

Level 3 Extended Diploma Unit 22 Developing Computer Games

Getting Started with Coding Awbie. Updated

Game Design. Review of Jesper Juul s Half-Real: Video Games between Real Rules and Fictional Worlds. Simon Cutajar

Multisensory Virtual Environment for Supporting Blind Persons' Acquisition of Spatial Cognitive Mapping a Case Study

KATHERINE ISBISTER (2016) HOW GAMES MOVE US: EMOTION BY DESIGN. CAMBRIDGE: MIT PRESS. ISBN: Julian Beimel

Designing an Auditory W-LAN based Game

A contemporary interactive computer game for visually impaired teens

DEVELOPMENT ESSENTIALS:

Proposal Accessible Arthur Games

Virtual Reality Based Scalable Framework for Travel Planning and Training

Getting Started with Osmo Coding. Updated

Exploring Surround Haptics Displays

Have you ever been playing a video game and thought, I would have

User Guidelines for Downloading Calibre Books on Android with Talkback Enabled

Students: Bar Uliel, Moran Nisan,Sapir Mordoch Supervisors: Yaron Honen,Boaz Sternfeld

TOKYO GAME SHOW 2018 Visitors Survey Report

How Representation of Game Information Affects Player Performance

Gameplay. Topics in Game Development UNM Spring 2008 ECE 495/595; CS 491/591

Official Design Document

Project Multimodal FooBilliard

Introduction. Overview

Parents Guide to Fortnite

interactive laboratory

Experimenting with Sound Immersion in an Arts and Crafts Museum

Language: Domains & Levels Listening: Developing Speaking: Beginning Reading: Developing Writing: Expanding

Multi-User Interaction in Virtual Audio Spaces

Should all future kids contents be interactive? What are the best games and TV coproduction models?

Designing an Audio System for Effective Use in Mixed Reality

Important Information

EDUCATING AND ENGAGING CHILDREN AND GUARDIANS ON THE BENEFITS OF GOOD POSTURE

INVESTIGATING BINAURAL LOCALISATION ABILITIES FOR PROPOSING A STANDARDISED TESTING ENVIRONMENT FOR BINAURAL SYSTEMS

Blue-Bot TEACHER GUIDE

Mage Arena will be aimed at casual gamers within the demographic.

Who plays mobile games? Player insights to help developers win

My first experience with sound for theatre was in eighth grade. I was only playing cheesy

Spatial Interfaces and Interactive 3D Environments for Immersive Musical Performances

MECHANICAL DESIGN LEARNING ENVIRONMENTS BASED ON VIRTUAL REALITY TECHNOLOGIES

HOW TO CREATE A SERIOUS GAME?

Orbital Delivery Service

ACCESSIBYTE ARCADE BY ACCESSIBYTE LLC 2016 ACCESSIBYTE LLC. All information contained is 2016 Accessibyte LLC 1

The Deep Sound of a Global Tweet: Sonic Window #1

Fortune Run: A Mobile Game Showcasing Cultural Celebration in Malaysia

VIDEOGAMES IN EUROPE:

Why visualize RL data in SL?

Gerbilcide Project Sacks, Nottingham, Albert, Miller, Kong Gerbilcide Game Design Document

Ad Specifications July, 2016

Browser Streaming and Mobile-App Instructions for ios and Android tm

Multisensory virtual environment for supporting blind persons acquisition of spatial cognitive mapping, orientation, and mobility skills

Introduction. Review. Keywords: 3D sound; audio games; blindness; navigation; virtual reality.

Executive Summary: Game Overviews: Evaluation Criteria: 15 March 2012 TCO Multimedia

SNGH s Not Guitar Hero

Procedural Level Generation for a 2D Platformer

The Next Generation of Gaming Consoles

A Study on the Navigation System for User s Effective Spatial Cognition

VIRTUAL REALITY. Mete CINAR - Merve KAYA - Gonul KANBAY - Umit VATANSEVER. Course Professor Rushan ZIATDINOV FATIH UNIVERSITY

Transcription:

A. Weisbecker, M. Burmester & A. Schmidt (Hrsg.): Mensch und Computer 2015 Workshopband, Stuttgart: Oldenbourg Wissenschaftsverlag, 2015, S. 423-429. Playing with sound and gesture in digital audio games Dr. Sonia Fizek 1, Dr. Julie Woletz 2, Jarosław Beksa 3 Gamification Lab, Centre for Digital Cultures, Leuphana University Lüneburg 1 Media Uselab, Centre for Digital Cultures, Leuphana University Lüneburg 2 School of Computer and Mathematical Sciences, Auckland University of Technology 3 Abstract This article introduces the genre of a digital audio game and discusses selected play interaction solutions implemented in the Audio Game Hub, a prototype designed and evaluated in the years 2014 and 2015 at the Gamification Lab at Leuphana University Lüneburg. 1 The Audio Game Hub constitutes a set of familiar playful activities (aiming at a target, reflex-based reacting to sound signals, labyrinth exploration) and casual games (e.g. Tetris, Memory) adapted to the digital medium and converted into the audio sphere, where the player is guided predominantly or solely by sound. The authors will discuss the design questions raised at early stages of the project, and confront them with the results of user experience testing performed on two groups of sighted and one group of visually impaired gamers. 1 Introducing digital audio games Have you been mesmerized in your childhood by playing the blind man's buff? Or maybe you always wondered why the fables read out by your parents were far more magical than their counterparts watched on the TV screen? After all, sounds have limitless potential to create beautifully immersive sceneries without the help of a single paintbrush or animated pixel. For centuries human beings have willfully got lost in the imaginary worlds of sound, from oral stories told by the fire, through music to more recent inventions, such as radio drama and audio books. With the introduction of the first computers and the ever-growing popularity of electronic entertainment, it was only a matter of time before the blind man's 1 The following team members participated in the project: Jarosław Beksa (lead concept design, sound design,production), Sonia Fizek (concept design support, in-game scripts), Enrique Perez (production support), Laleh Torabi (graphic design), Julie Woletz (UX testing and evaluation), Silke Laub (testing support), Sophie Jent (programming), David Scheele (programming), Marvin Töllner (programming support).

424 Playing with sound and gesture in digital audio games Sonia Fizek, Julie Woletz, Jarosław Beksa 2 buff and sound-based approaches in general would be adapted to the digital medium. Contrary to popular video games, digital audio games exist primarily or solely in the world of sounds and [ ] do not rely on visuals. They generally can be played without graphics by the visually impaired (Collins 2013, 24). It is the sonic layer that becomes the basis for gameplay mechanics and interface design. The first attempts to build electronic audio games may be tracked to late 1970 s, when Atari designed Touch Me, a rhythm game on a handheld device, which became an inspiration for the popular Simon (1978). Although sounds played a crucial role, it was not possible to play the game and operate the device with no visuals. The player had to memorize and reproduce the sequence of four buttons lighting up with particular sounds attached to them. In 1996 another sound-based handheld playful device appeared on the market Bop It. This time the interaction pattern was designed solely with audio in mind. The device featured a button, a lever, and a handle. The player s task was to listen to the commands (Bop it, Twist it, or Pull it) and interact with respective parts of the electronic console. Interestingly, point values were also represented by various tones. The latter example, although not designed with visually impaired players in mind, could have been easily operated by them, due to its intuitive design that did not require the engagement of vision. The potential of solely sound-based mechanics was also explored in story-driven audio games, such as Real Sound Kaze No Regret (1999) created for Sega Dreamcast and Sega Saturn consoles. In 2001 Shades of Doom, the first Windows-based adventure title fully accessible to the visually impaired, was introduced by the GMA Games. 2 Inspired by the graphical game Doom (1993), it adapted the first-person shooter (FPS) genre to the world of sounds. The players oriented themselves in the soundscape by the echo of footsteps, the wind howling through the passages, and the subtle sounds of nearby props. In the recent years, digital audio games have been gradually gaining popularity, also amongst visual players. Several games based on moving through space have been created, predominantly for tablets and smartphones: Papa Sangre (2010), Sound Swallower (2011), Audio Defence: Zombie Arena (2014), and Blindside (2012) for PC, amongst others. An experimental playful audio experience was developed by a Copenhagen Game Collective, using the existing commercial game controllers. In Dark Room Sex Game (2008) two players bring their invisible avatars to orgasm by shaking the Nintendo Wii motion controls (Oldenburg 2013). Among the games specifically developed for visually impaired people, Finger Dance is an original audio-based rhythm-action game (Miller et.al 2007). With the popularization of touchscreen portable devices (smartphones and tablets), the ubiquity of mobile game platforms (e.g. App Store, Google Play, Windows Store), and the proliferation of independent game scene ( indie ), audio games seem to be entering a yet to be explored territory. More importantly, they gain a wider recognition and, as we will demonstrate, have the capacity to attract not only visually impaired but also visual players. 2 The titles designed by the GMA Games in the past decade are exclusively tailored to the visually impaired community and known only in relatively small circles.

Playing with sound and gesture in digital audio games 4253 2 The Audio Game Hub prototype In the Audio Game Hub (2014) multiple casual games and cross-culturally familiar playful experiences have been transferred into the audio sphere. The players may draw their audio bows in Archery and Hunting, improve their memory in Animal Farm, test reflexes in the multiplayer Samurai, or find the way out of the fearsome audio Labyrinth, amongst others. The Audio Game Hub is a practice-led research project and a work-in-progress. A collection of eight various mini-games constitutes a perfect playground sample, as it allows for the evaluation of various interface solutions in a homogeneous interactive environment. Each of the games requires a different type of interaction pattern and audio gameplay mechanics. Another crucial aspect of the audio gameplay experience is the platform. The game has been developed for mobile touchscreen devices (ios operating system), with an interface combining gesture-recognition with auditory instructions. The Audio Game Hub interface uses a set of gestures and corner buttons, which allows the player to navigate through the available options and menu elements. A minimalistic graphical interface was implemented to reduce the entry barrier for sighted players. It can be disabled and enabled at any time. To proof that audio-only games may provide new and exciting gaming experiences for all kinds of players, this feature has been used during the user experience testing one group of sighted players started the game with visual aids off, and another with visual aids on. The main focus has been laid on the audio interface and different types of sounds (Friberg & Gärdenfors, 2004, 4). Each user action is represented by an interactive sound. Each game starts with an audio tutorial explaining the rules and game mechanics. An array of background, ambient sounds and various in-game character voices have been implemented to represent each of the game worlds and enhance the immersion of the player. Selected interactive sounds used in the game s menu and in individual games have been assigned to the same actions and events. This points to another crucial aspect of interactive sounds - their repeatability, which [ ] establishes an expectation that we will hear the same sound as a reaction to the same action (Collins 2013, 33). 2.1 Main menu interface The interface solutions have been implemented on the level of the main menu interface, and in-game interfaces. The Audio Game Hub interface constitutes the main menu of the application, which allows the player to navigate through the available options and menu (fig. 1). The options are assigned to four functional corner buttons as follows: The upper left corner: Return (to previous menu) The upper right corner: Information (additional cues corresponding to the selected menu squares or in-game content) The bottom left corner: Visual mode (visual aids on and off; for visual players)

426 Playing with sound and gesture in digital audio games Sonia Fizek, Julie Woletz, Jarosław Beksa 4 The bottom right corner: Settings (such as voice, sound effects, ambience, and music volumes) Fig. 1: Audio Game Hub main menu interface with supporting visuals The menu elements and corner buttons are activated by the double tap gesture. The UI elements are scalable to any screen size. When the player double taps any of the buttons, the audio description is activated. Such a feedback loop provides auditory cues, which further guide the player s actions. Each corner is also assigned a unique descriptive audio icon, which adds another level of a sonic signifier to a given option. The gesture-based audio menu interface has been designed specifically with visually impaired in mind. Image-based cues are provided for the visual players, but they may be turned off by double tapping the Visual mode button. The menu elements may be browsed by swiping left or right. The sounds of the audio descriptions of the consecutive menu elements are placed in stereo panorama corresponding to their position in the menu. In all the games subtle whispered audio cues are available, informing about reaching a corner button or the edge of the screen. The sounds of the menu interface corner buttons have been recorded in the binaural 3D technique, which provides spatial audio representation corresponding to their physical location on the screen. When such stimuli are presented over headphones, there is a realistic perception of auditory space [ ] (Wenzel et al. 1993, 111). 2.2 In-game interfaces The in-game audio interfaces are discussed here with reference to two exemplary minigames - Animal Farm and Labyrinth. The first example is a digital audio adaptation of Memory, an analogue card party game, also known as Match Match or Pairs, amongst others. The player s task is to find matching pairs of farm animals stored in boxes. To locate the boxes, the player moves their finger on the screen (searching for a wooden box sound).

Playing with sound and gesture in digital audio games 4275 Each box is opened by a double tap gesture, which triggers the sound of a selected animal. Matching pairs of animals are removed until there are no boxes left and the game is solved. Fig. 2: Animal Farm in-game interface start Fig. 3: Animal Farm in-game interface playing Labyrinth is a classic arcade maze. The player s goal is to find the exit by following the guiding sound. When getting closer to the exit, the guiding sound becomes louder. The game consists of randomly generated rooms, whose borders are drawn by sounds signifying the player stepping against a wall. Fig. 4: Labyrinth in-game interface 3 User experience testing and evaluation During the development process (using Agile method) each of the mini-games and the main user interface were preliminarily tested with sighted as well as visually impaired players.

428 Playing with sound and gesture in digital audio games Sonia Fizek, Julie Woletz, Jarosław Beksa 6 Gathered feedback allowed the team to improve the user interface and game mechanics. At a later design stage the Audio Game Hub was introduced to and tested on two different groups of sighted gamers (group one of n=12 heavy gamers, playing at least 4 hours per week on different devices; group two of n=12 casual gamers, playing max. 3 hours per week mostly on mobile devices) and a third group consisting of n=5 visually impaired players with gaming experience. Thus, a total of 29 participants ranging from 19 to 52 years of age tested the Audio Game Hub during individual face-to-face interviews of 60 minutes each. To gain insights on various players needs and interests, participants played in different modes: Group one of n=12 heavy gamers: one half of the group played with visual aids on, the other half played with visual mode off Group two of n=12 casual gamers: one half of the group played with visual aids on, the other half played with visual mode off Group three of n=5 visually impaired persons played without being able to see the screen at all The main goal of the testing was to identify usability problems with this relatively new kind of audio interface and interactive sounds, and to evaluate how well the different types of interaction and gameplay mechanics fitted the needs and interests of the various groups of gamers. Further more, the overall user experience of each game in the Audio Game Hub was evaluated in a much broader sense. During the 60 minutes test interviews, participants were observed while playing and could give detailed feedback regarding aspects like quality of spoken explanations and sound clues as well as attractiveness of in-game character voices and ambient sounds, and perceived immersion and joy-of-use of each game. Test results revealed that the majority of sighted gamers would prefer playing with visual mode off and enjoyed the unusual audio-only gaming experience. To continue playing heavy gamers expect additional levels with increasing difficulties, which are not yet implemented in the Audio Game Hub. Although some usability aspects still need to be improved, visually impaired gamers found their needs and interests well met and praised the perceived joy-ofuse. Furthermore, testing showed that sound centered gameplay mechanics like acoustic orientation in the Labyrinth, aiming with sound clues in Archery, or using sound memory in Animal Farm are perceived as challenging especially for sighted players, but received best feedback. All user groups emphasized the importance of good ambient sounds and voice design and its influence on the overall gaming experience. 3 4 Conclusion We need to realize that the potential of audio games is not a stable indicator and depends on various factors, surpassing the gameplay design itself. The interactive digital entertainment 3 A full report of the user experience testing report is forthcoming.

Playing with sound and gesture in digital audio games 4297 has matured over the past few decades. An independent scene focused on the development of experimental projects is also contributing to the growing popularity of audio games. The potential of audio gaming is also increasing due to the constant improvements in technology. Since sound-based games do not need to rely on visual elements and big screens, mobile devices seem to be perfect solutions. The invention of a smartphone in the late 2000 s opened new possibilities for audio game designers. The ease of access to mobile digital content platforms, such as App Store, Google Play, and Windows Store should also be acknowledged. On the other hand, visually impaired players, thanks to more ubiquitous and intuitive digital technologies and devices, seem to be entering a lower usability barrier than ever before. References Collins, K. (2013). Playing with sound: a theory of interacting with sound and music in video games. Cambridge, MA: The MIT Press. Friberg, J., Gärdenfors, D. (2004). Audio Games: New perspectives on game audio. In: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology. Jumanji, Singapore 2004, 148-154. GMA Games. Available at: http://www.gmagames.com/ourgames.shtml. Accessed 17 June 2015. Miller, D., Parecki, A. & Douglas, S. A. (2007). Finger dance: a sound game for blind people. In: Assets 07: Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, New York, NY, USA: ACM, 2007, 253 254. Oldenburg, A. (2013). Sonic Mechanics: Audio as Gameplay. In Game Studies.org, 13(1). Available at: http://gamestudies.org/1301/articles/oldenburg_sonic_mechanics. Accessed 17 June 2015. Wenzel, E. M., Arruda, M., Kistler, D. J., & Wightman, F. L. (1993). Localization using non-individualized head-related transfer functions. Journal of the Acoustical Society of America, 94(1), 111 123. Contact Dr. Sonia Fizek Gamification Lab, Centre for Digital Cultures (CDC), Leuphana Universität Lüneburg Scharnhorststraße 1, 21335 Lüneburg, Germany Soniafizek@soniafizek.com Dr. Julie Woletz Media Uselab, Centre for Digital Cultures (CDC), Leuphana Universität Lüneburg Scharnhorststraße 1, 21335 Lüneburg, Germany Woletz@inkubator.leuphana.de Jarosław Beksa School of Computer and Mathematical Sciences, Auckland University of Technology Wakefield St 2-14, 1010 Auckland, New Zealand jarek.beksa@gmail.com