Mr Beam. magazine. The Soundscapes concept and the Personics sensor system. Feature

Transcription

1 Feature Mr Beam The Soundscapes concept and the Personics sensor system Tony Brooks Soundscapes We are surrounded by the ever-widening use of technology, and the benefits and opportunities this technology affords are particularly apparent in the area of special needs. Tony Brooks has carried out groundbreaking work in this (and other) area(s) for the past 15 years. His Soundscapes concept and its current blueprint, the Personics sensor interface system (see shaded box on p4), both testify to Brooks focus on the body as transmitter and receiver, and his drive to develop an optimum interface that all people even the severely disabled have creative access to. In this article, Tony Brooks concentrates on the relevance of the technology he has developed for the special needs field. He starts by given a short history of his interest in the area, and recaptures how early dreams have resulted in more recent concrete achievements. History - the timeline! I was exposed to disability from early on in life through contact with my disabled uncle. I started playing music as a teenager, and during my many hours of practicising my bass I would sometimes give my uncle a basic expression pedal he could put pressure on, pressure that affected the sound produced. My uncle was unable to speak, but through this interaction we developed our own form of communication which resulted in much teasing, and laughter, and magical moments. Perhaps this was my first artless attempt at the creation of a play environment for persons with a disability, which I would later develop into the Soundscapes concept. In the mid-eighties I was exposed to computers and became inspired by their potential use for the manipulation of musical synthesizers and visuals.as a result I began to envision an environment that would give everybody (irrespective of ability, age or creed) access to the manipulation of audio-visuals. And I started to work towards the creation of such a system, accessible to all, that would enable emotional expression and sensual exploration, and would thereby improve people s quality of life in different ways, depending on their needs and desires. The system would use technology to translate various kinds of human input into multimedia response or feedback (both auditory and visual). Over the next few years my research into this led me to visit a wide range of institutes and centres (from Stanford University, California, to the Veterans Rehabilitation Center in Palo Alto, California, to the local Handicapped Activity Center in Aarhus, Denmark!) where, in dialogue with various special needs groups, I tried to test the feasibility of my concept. I concluded that the optimum input interface for my purposes would need to be able to capture information from the human without any interference. The captured information would need to be of a sufficiently high resolution to feed into a computer and give high-quality results for smooth feedback. I envisioned that such an interface would be of benefit in many ways to many people, including, but not restricted to, people with special needs. The interface I aimed to develop had to meet a number of challenging requirements. Wires that would encumber the body and restrict freedom of movement were not acceptable. The interface had to be non-tangible, invisible, multi-dimensional (i.e. not just linear) and capable of capturing finite information from 2 mms up to approximately 15 metres. It also had to be silent, and non-dependent on room shape and decoration. Moreover, the interface had to be able to give high definition data without interference and cross talk, and to communicate the captured information data to distant systems for remote consultation via the World Wide Web. It was also to be user-friendly, that is, considerate of users who may not be too technologically aware (such as helpers and therapists), as well as easy-to-use, fun, and human-centred! The resulting concept is called Soundscapes, which started as a research project in the mid-eighties and has culminated in the design and production of the Personics sensor system.the Personics system is the leader in a new generation of sensor systems that has been prioritised so as to make the user interface as easy as possible to understand and operate. The 2

2 Snapshots from a Personics interactive disco: great legwork and the ultimate air guitar.the Personics system was set up at a local activity centre for disabled people, a rhythm was played and people at the centre were invited to join in. Upon entering the space they found that they could affect the rhythm s tempo and pitch by moving their bodies through various interactive spaces. Image 1 shows one of the stars of the day, a woman who kicked through various percussive sound areas. Image 2 shows Bo from the local Activity Centre for disabled people, who discovered a specific interactive area which enabled him to play a Hendrix-style guitar lick simply by moving his hands. sensors detect (even the smallest) body movements in invisible light; the various output responses are both visual and sonic, and include sounds, images, colour, vibration and lights, as well as physical movement from intelligent robotic units (which are often an amplified riposte to the human input). The system is calibrated for each participant by a specially-trained supervisor, and the software, which is viewed and controlled from a basic PC computer, remembers which participants are working/playing and what their preferences and capabilities are. The current prototype has a three-head array of small but powerful sensors that do not exhibit cross talk and operate at the ranges mentioned above (from 2 mms to 15 metre).these sensors are mounted on flexible thin goosenecks that can be positioned to capture muscle movements much smaller than most other systems on the market are able to do. The first prototype of the Personics system was launched in 1999 at the Pappagallo Production House in Aarhus, Denmark.The most recent version was presented this year at the 5th Scientific Congress of the Paralympic Games in Sydney (which had as its theme Pushing the Limits: Optimising Potential Through Science and Technology). A commercial model of the system will be available next year. Why and who for? What makes the Personics system special? Briefly, I would suggest that there are three main factors. First, the data capture it affords: all it takes to affect a visual, sonic or robotic transformation is the flicker of an eyelid, the twitch of a muscle or small movement of the mouth. Second, the two-way communication between sensor and easy GUI (Graphical User Interface) on the control unit (home PC), which enables calibration and optimum interaction for each individual; in other words, the 3

3 A sketch of the space for the Personics interactive dance space. People enter not knowing how to participate, but understand quickly when they put their hand (or make another kind of movement) in any of the interactive spaces.they can either enjoy their own exploration or interact with others.the zones may give single media feedback or multiple feedback (at the points where the zones overlap). The Personics system The Personics system converts various kinds of sensor input, such as physical movement, into selected multimedia feedback.the system is used for entertainment, rehabilitation, disabilities training, art installations, and group interaction. The individual decides on the sensor input Each sensor can have its own-programmed characteristics, and the individual using the system determines what kind of input to use, depending on his or her needs, goals, capabilities and preferences for feedback,. Signal processing combines the sensor input The core of the system is a package of sophisticated electronics specially designed to read and combine various sensory signals into useful data, which are interpreted in turn by the Personics software system or transmitted to other digital interfaces (such as MIDI). Much effort is put into ensuring the flexibility of the intelligent electronics, so that anyone can use the product according to his or her own capabilities. Feedback software creates a unique multimedia show. The signals received from the special electronics drive sound, graphics, lights and other devices, resulting in the creation of a multimedia show. The basic package offers choice from a set of graphics and sounds that can be played on a PC. Larger systems can be used to drive a myriad of devices such as intelligent lights, vibration platforms or mechanical devices. New possibilities are continually being developed. The internet becomes an integral tool to expand the potential of every aspect of the system. Personics.net can be used to review data for rehabilitation therapy, find new graphics, sounds, program updates or the latest news on Personics applications, and communicate and play with other people using the system worldwide. 4

4 system can be tailored to suit individual prerequisites and preferences. And thirdly, the fact that the system enables most other existing sensor systems to plug & play i.e. communicate within the same environment. This built-in interconnectivity makes it possible to enhance existing systems (often purchased at a high price) which can then be controlled from a standard PC. All this increases the system s user-friendliness. In addition, the control software is designed to be intuitive and easy-to-learn: it takes the user step-bystep through setting up parameters for each session, and is capable of retrieving information and preferences relative to each individual. This way it eliminates unnecessary repetitive tasks, and ensures that the data is meaningful for progressive analysis. A simple touch screen, similar to those used at ticket and information kiosks, was selected as the input control surface. Why use this technology in special needs cases? Imagine for a moment a person whose only conduit for expression and communication is one infinitesimal physical gesture. The technology concerned offers new ways to translate such a gesture into meaningful data that can be mapped according to the needs and preferences of the individual. I envisioned the technology as a home system that would minimize visits to a therapist 1, through the unit s ability to send data for remote consultation via the World Wide Web ( tele-medicine ). The implementation of the World Wide Web elements allows for, amongst other things, downloadable upgrades, exchange of information, sounds, images, and so on, and access to a professional forum of therapists and users. In short, the system can be used as a home training aid at a remote location, as well as a recreational and communication tool. In the everyday home environment, family members of a disabled person can encourage exercises and play with the person and the system. The advantages include less stress, fewer expenses and environmental benefits. Tony Brooks teaching a therapist to control the position, colour and shape of some pseudo-olympic rings before leaving for Sydney Virtual reality and human-centred technology in an ideological digital dreamland At a conference on Virtual Reality and Persons with Disabilities, held at California State in 1992, Jaron Lanier, one of the most prominent spokespeople on Virtual Reality, referred to the not improbable future use of a form of computer clothing. Such clothing, he said, would be worn over the major sense organs located at the surface of the skin (the ears, eye, nose, mouth and skin itself) and would be able to provide a stimulus (e.g. tactile or force feedback) received from an alternative environment. The input information for that alternative environment would optimally be captured and transmitted through a low interference interface, to allow maximum experiential participation and free communication and expression. Lanier concluded that A little known fact is that in the history of virtual reality development, the community of researchers building virtual reality machines and virtual reality software has been, in many cases, almost the same community as the people working on tools for the disabled.there s been an incredible overlap between the two communities, and I think the reason for that is very obvious: our goals are almost the same. The goal is to see how you can use technology and mould it to a person, instead of asking the person to come to the technology. Again, how do you make things HUMAN-Centred? And furthermore, when you work 5

5 with virtual reality you discover that the amount of individual variation between people and the way that their senses work is pretty high; and so in the future, virtual reality systems will be individualized anyway. So, in our field, I don t know if there ever will be a distinction between what work is for people with disabilities and what isn t. I think it s really all part of the same thing. I often wonder (when drifting into my own ideological digital dreamland!) if one day the air surrounding my body will be capable of procuring, transmitting and receiving the necessary highresolution data information without any impeding attachment. I speculate that such a thing may be made possible by involving neural, physically modelled (sonic and visual) and other elements relating to synesthesia, within an environment with interactive high-definition volumetric images. This would include not only screens and projections, but also a VIS (Virtual Interactive Space ) of interactive neural ions programmed to be intelligent enough to transmit and receive data relative to the body sense with which each ion is in contact. In addition, the system I imagine would have the power, memory and analytical capacity (amongst other things) to enable calibration of all relevant parameters to each individual participant, and act upon that data so as to optimise the system automatically. Lanier s computer clothing becomes Brooks ion skin! We haven t reached that stage yet. But the current Personics prototype sensor system already allows for non-tactile navigation of virtual reality environments through small gestures or body movements, and enables participants to paint, play music, create audio-visual environments and play games through input as minimal as the flicker of an eyelid, the twitch of a muscle, or even a breath. Research continues into the use of the Personics system in education and in learning-aid environments. I also continue my own ongoing exploration of synesthesia relative to the new technologies. Much work remains to be done. But on the basis of my own experiences and research so far, I strongly believe that many people stand to gain from the use of this technology, both within and beyond the field of special needs. 1 You may already have noticed that there is no mention of the T word ( therapy ) in this article.this is deliberate: the whole concept of Soundscapes was built around the notion of play rather than therapy, because we observed that in some sessions the mere mention of the word therapy would lower the energy and hence the input of some (verbally sensitive) participants. This was no longer the case if we used the word play. Tony Brooks is a visionary performing and conceptual artist and researcher. He has presented his work at numerous international conferences and events during the past 15 years, and lectures widely at universities, colleges and institutes worldwide. Brooks work is currently being researched at the Center for Brain Injury, Copenhagen, and at the Center for Advanced Visualization and Interactivity at Aarhus University, Denmark. Brooks is involved in two European projects that use his system. He has just been awarded a grant of 12 M DKK (1,6 M Euro) for further research at the Brain Injury Institute, Copenhagen University. The first Personics Sensor System is scheduled for limited release in February For more information on the Personics system: Tony Brooks will be demonstrating the Personics system at his workshop at the i3 Spring Days on 24 April 2001 in Porto. For more information, see spring01/w11-info.html 6