Do You Feel What I Hear?

1 Do You Feel What I Hear? Patrick Roth 1, Hesham Kamel 2, Lori Petrucci 1, Thierry Pun 1 1 Computer Science Department CUI, University of Geneva CH - 1211 Geneva 4, Switzerland Patrick.Roth@cui.unige.ch 2 The Group for User Interface Research EECS, UC Berkeley Berkeley, Ca. 94720, USA

2 Abstract In this research we implemented three different methods for presenting scientific graphs to blind and visually impaired people. Each rendering method employed either audition, kinesthetic or a combination of those two modalities. In order to allow for distance learning, we have used low cost portable devices for the output graph rendering. The three modes of representation were then compared by three separate groups of blind and visually impaired computer users. Each group consisted of four participants. Results reveal that the combination of both audio and kinesthetic modalities can be a promising representation medium of common scientific graphs for visually challenged people. KEYWORDS Blind, non-visual and non-tactile graph representation, audio and kinesthetic rendering.

3 Introduction The representation of scientific graphs is among the teaching concepts that are difficult to electronically transform onto computers for blind and visually impaired people. Most often tactile channel (e.g., hard copies produced via Braille printers) seems to be privileged by specialists to convey such type of graphical information. However, this method in its nature does not provide an adequate solution when presenting graphical information via distance learning. Hard copies are static in nature (i.e., they cannot be modified) and tactile devices are expensive as well as hard to carry (Kamel & Landay, 2000). Over the past several years, triggered by the advent of virtual reality devices, various investigations have been made by researchers regarding the presentation of non-visual scientific data using either audio or kinesthetic modality (Flowers & Hauer, 1995; Fritz & Barner, 1999; Mansur, Blattner, and Joy, 1985; Ramloll, Yu, Brewster, Riedel, Burton, and Dimigen, 2000; Sahyun & Gardner, 1997). However, very little work has been undertaken in comparing these two non-visual rendering techniques. In addition, most often researchers conduct experiments using high cost devices (e.g., the PHANToM Haptic interface). These devices present a problem to most ordinary blind users since they cannot afford them. Addressing the problems mentioned above, we have designed three methods of non-visual graph representation, making use of two different modalities. In the first rendering method, we used auditory feedback (non-speech sounds). In the second method, we utilized a kinesthetic interaction through the use of a Haptic mouse. As for the third rendering method, we combined the first and the second modalities. In order to fit with the distance-learning concept and to address the mobility issues, we rendered the graphical output by using low cost mobile devices (i.e., a force feedback mouse).

4 Audio and Kinesthetic Representation The three methods that we developed to represent scientific graphs provide two different categories of graphical information. The first category conveys information regarding the overall shape of the graph, whereas the second category provides extra information regarding several aspects of the graph: its maximum and minimum values; all the point(s) where the graph crosses either the horizontal or the vertical axis; using the kinesthetic rendering, the user recognizes a starting or an ending position when the Force Feedback (FF) device is pointing at either one of them. Audio Encoding For the audio rendering of the entire graph, we have implemented a traditional approach. We presented the graph by a single tone where its pitch represented the Y coordinate and the duration represented the X coordinate (Flowers & Hauer, 1995; Mansur, Blattner, and Joy, 1985; Sahyun, 1999). In addition, to improve the rendering, we immerged the tone into a 2D sound plane in which the stereo effect allowed to represent the X coordinate and the elevation effect the Y coordinate (Roth, Petrucci, Assimacopoulos, and Pun, 2000). This plane is vertically oriented and virtually located in front of the blind listener. For instance, if the rendered graph is a linear curve (i.e., y=a*x+b, where a and b are constants and a is superior to zero), the user hears a tone ascending obliquely from the bottom left to the top right (see Figure 1). When the shape to be rendered is ascending, the tone increases in frequency.

5 Figure 1. 2D sound plane rendering. For the rendering of the extra information of the graph, each output is characterized by a different non-speech sound. These tones differ so they can be easily recognized if they are played at the same time (Brewster, 1995). Also, the tones are short so that the blind users can quickly interpret them. Kinesthetic Encoding The haptic rendering of the graph is based on a virtual fixture (Rosenberg, 1993). In this method, the blind user by holding the mouse trails the edge of the graph following its path. Concretely, we determine a point on the graph (P min ), and the distance between this point and the cursor position (c) forms the minimum distance (d) (see Figure 1.a). If the distance is below a given threshold, we activate the force (F) that attracts the mouse towards the graph (see Figure 1.b). Therefore, when the mouse moves away from the graph, it is automatically attracted back to it.

6 Distance threshold curve x max x max x max x min P min c d x min P min F p c a x min P min c fixture region Figure 2.a. Distance setting. Figure 2.b. Attraction force. Figure 2.c. Audio rendering. All the extra information of the graph is produced by vibrations. These vibrations are generated by a periodical effect that repetitively moves back and forth in a very short distance and with a relatively high frequency. We can thus differentiate between different vibrations by either varying their frequency, their direction, or both. Audio-Kinesthetic Encoding This encoding method combines the audio and the kinesthetic modalities. In this method, the information of the entire graph is rendered by the virtual fixture (explained section Kinesthetic Encoding ). As for the extra information, we have replaced the vibration effects by non-speech sounds. In addition, we encoded the slope of the graph by two different tones (i.e., one tone for each ascending or descending slopes). This tone also varies in frequency proportionally to the slope itself (see Figure 3). Figure 3. Auditory coding of the slope.

7 During the development of the virtual fixture, we have noted several problems that concerned the push back force limitation. This limitation, due to the nature the haptic device, does not allow blind users to follow the path of the graph without deviating. To solve this problem, we developed a feature that we called auditory fixture. In this feature, we use sound as an alarm to inform the user that he or she leaves the graph. In this case, when the blind user moves the mouse (cursor) away from the curve fixture region an auditory signal is automatically played (see Figure 2.c). Hardware Configuration The evaluation of the system was made on a Pentium PII computer. The audio rendering was obtained by using the DirectSound3D 1 library on a SoundBlaster Live 2 sound card, and a Sennheiser HD580 3 headphone. We choose to work with a SoundBlaster Live and a Sennheiser II because they provide together an effective 2D sound rendering. As for the kinesthetic interaction, we worked with the Logitech WingMan Force Feedback 4 mouse. This mouse is much more affordable than the PHANToM 5 device. To obtain a more detailed comparison between the two force feedback systems, please refer to the report written by Sjöström (1999) and (Roth, Giess, Petrucci and Pun, in press). System Evaluation We have conducted a series of experiments in order to study the efficiency of each of the three methods mentioned above. The three methods of representation were experiment by three groups (G1, G2 and G3) each consisted of four participants aged between 20-45 years old. We assigned one method to each of the three groups; in each group, two of the participants were congenitally 1 library included on windows 98 or recent systems. 2 available on regular computer stores or at www.soudblaster.com for about $200. 3 available on music stores or at http://www.sennheiser.com/ for about $200. 4 available on regular computer stores or at www.logitech.com for about $50. 5 available at www.sensable.com for about $13500.

8 blind and the two others were visually impaired. G1 participated in recognizing scientific graphs via the auditory encoding method (see section Audio Encoding ). G2 examined the graphs using the kinesthetic encoding (see section Kinesthetic Encoding ). Finally, G3 obtained the scientific graphs using the audio-kinesthetic encoding (see section Audio-Kinesthetic Encoding ). During the experiment, the visually impaired participants were not blindfolded, however they did not have access to the computer screen. All of the participants had received some type of mathematical education background (viz., knowledge of scientific graphs.). We gave each of the participants the same amount of time (20 minutes) to get familiarized with the method they were testing. During this time, we briefly explained the method of representation they were going to experiment and we let them use the program on a simple graph (i.e., y = abs(x), 0 x 4). Tasks and Scenarios Each of the participants was given three different scientific graphs one at a time for evaluation: a linear function: y = 1-x, -2 x 2 (see Figure 4.a); a parabolic curve: y = x 2, -2 x 2 (see Figure 4.b); a periodical sine wave: y = sin(x), -4 x 4 (see Figure 4.c); Figure 4.a. y=1-x Figure 4.b. y=x 2 Figure 4.c. y=sin(x) For each graph, after interacting with it for a period that did not exceeds five minutes, each of the participants was asked to reproduce it. The blind participants reproduced it using a tactile medium, where the visually impaired used a pencil and a sheet of paper. The success of reproduction was based on two complementary criteria (C1, C2). The first criterion C1 is

9 subjective and relative to the level of general similarity between the real graph and the one that as been reproduced by the participant. As for the second criterion C2, we evaluated more specific similarities of the two graphs such as curvatures or symmetries. In addition, we asked all participants while drawing to place the horizontal and vertical axis. Our goal was to check the efficiency of the extra information rendered by the encoding method that was being tested. In this task, the criterion of success was based on the precision in which the participant placed the horizontal axis (criterion C3) respectively the vertical axis (criterion C4) on the graph. After completing each task, we asked all participants several questions regarding the characteristic of the graph rendered (e.g., the periodicity, minimum and maximum values, location, and continuity). Finally, after conducting the experiment, we then asked each of the three groups to evaluate the system they had tested. Results and Discussion For the graph that represented the linear function, the results shown in Figure 5 reveal that the group G1 using the audio encoding had the most difficulties in reproducing the linear function (25% of success for C1 and C2). As it is shown in Figure 6.a, the majority of G1 participants represented the linear function as a parabola. y=1-x Percentage of Success 120% 100% 80% 60% 40% 20% 0% C1 C2 C3 C4 Criteria G1 G2 G3 Figure 5. For the linear function, percentage of participants of groups G1-G3 that correctly performed the criteria C1-C4.

10 All the participants of G1 agreed that the auditory rendering did not have sufficient information concerning the linearity. However, this problem did not occur to the groups G2 (75% of success for C1 and C2) and G3 (100% of success for C1-C4) which used the kinesthetic sense as principal modality. Figure 6.a. Linear function reproduced by participant from G3. Figure 6.b. Linear function reproduced by participant from G1. As Figure 7 shows, no participant in any of the three groups had any difficulty to recognize then reproduce the parabolic graph (100% of success for C1). However, we noted some imprecision y=x 2 120% Percentage of Success 100% 80% 60% 40% 20% 0% C1 C2 C3 C4 G1 G2 G3 Criteria Figure 7. For the parabolic function, percentage of participants of groups G1-G3 that correctly performed the criteria C1-C4.

11 for the three groups and more for the group G3 (50% of success) in reproducing the exact symmetry of the parabolic (see Figure 8.a). The participants of G2 were not able to recreate their mental perception of the horizontal axis (0% of success for C3) (see Figure 8.b). We believe the reason is that at the origin there were two different vibrations (the first was for the x axis and the second was for the y axis). This group expressed that they could not distinguish between them. One possible solution to this problem would be to replace these vibrations occurring at the same time by their consecutive output. However, such approach may require the participants to spend more time to recognize the graphs. Figure 8.a. Parabolic function reproduced by participant from G3. Figure 8.b. Parabolic function reproduced by participant from G2. As shown in Figure 9, G1, G2 and G3 were capable to reproduce a recognizable overview of the y=sin(x) Percentage of Success 120% 100% 80% 60% 40% 20% 0% C1 C2 C3 C4 G1 G2 G3 Criteria Figure 9. For the periodical function, percentage of participants of groups G1-G3 that correctly performed the criteria C1-C4.

12 periodical graph (75% of success for C1). They however had difficulties in precisely reproducing the two curvatures at -π/2 and π/2 (see Figure 10.a,b,c). The audio rendering was not sufficient enough to precisely encode such kind of information for G1 (25% of success for C2). Concerning G2 and G3 participants (0% of success for C2), the principal drawback is due to the fact that the workspace of the force feedback device was small. Another problem is that the nature of this device required hand grasping, as opposed to finger grasping, which did not allow for high precision feedback (Cutkosky, 1990; Sjöström, 1999). Figure 10.a. Periodical graph reproduced by participant from G1. Figure 10.b. Periodical graph reproduced by participant from G2. Figure 10.c. Periodical graph reproduced by participant from G3. Therefore, we concluded that by replacing hand grasping (see Figure 11.a) with finger grasping and by increasing the size of the workspace as well as the force of feedback, the recognition would be facilitated. To this effect, we designed a modified device allowing finger grasping (see Figure 11.b). We are currently investigating the use of this modified device.

13 Figure 11.a. Force feedback device used for the experiment. Figure 11.b. Modified device. Most of the participants felt that the three systems would be of a great assistance for providing access to scientific graphs via computer systems. The participants who are used to Braille preferred interacting with the tactile medium over the kinesthetic rendering, but it should be noted that not all blind people are Braille readers. Conclusion In this paper we have presented the design of three methods of non-visual graph representation as well as their user s evaluations. Our results show that in general the participants were capable to reproduce an overview of three scientific graphs they were testing. In some case however they had difficulties in reproducing curvatures and symmetries. In our point of view, by taking into account all the possible modifications (i.e., finger grasping, workspace and size increased), the combination of the kinesthetic and audio renderings seems to be the most promising method. As an extension to this work, we plan the development of a framework on the Web where blind users will be able to access scientific graphs on-line. This work will be done in order to promote the possibility of using such technique of representation in the domain of distance education.

14 Acknowledgments This project is partly financed by the Association pour le Bien des Aveugles, Geneva, Switzerland. The authors are grateful to the Eingliederungsstelle für Sehbehinderte institution, Basel, Switzerland, and particularly to Denis Rossel, Denis Page and, Alain Barrillier for their help in the design and evaluation of the prototype. References Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1995). Parallel earcons: Reducing the length of audio messages. International Journal of Human-Computer Studies, 43(2), 153-175. Cutkosky, M.R., Howe, R.D. (1990). Human Grasp Choice and Robotic Grasp Analysis. In Dextrous Robot Hands, S.T. Venkataraman and T. Iberall, (Ed.), Springer-Verlag, New York, 1990. Flowers, J. H., and Hauer, T. A. (1995). Musical versus visual graphs: Cross-modal equivalence in perception of time series data. Human Factors, 37, 553-569. Fritz, J. P., Barner, K. E. (1999). Design of a Haptic Visualization System for People with Visual Impairments. IEEE Transactions on rehabilitation engineering, vol 7, No 3, 372-384. Kamel, H., Landay, J. (2000). A Study of Blind Drawing Practice: Creating Graphical Information Without the Visual Channel. The Fourth International Conference on Assistive Technologies, 2000, ACM Press, Arlington, VA. 17-25. Mansur, D.L., Blattner, M.M., and Joy, K.I. (1985). Sound graphs: a numerical data analysis method for the blind. Journal of Medical Systems, 9(3), 63-174. Ramloll, R., Yu, W., Brewster, S.A., Riedel, B., Burton, A.M. and Dimigen, G. (2000). Constructing Sonified Haptic Line Graphs for the Blind Student: First Steps. The Fourth International Conference on Assistive Technologies, 2000, ACM Press, Arlington, VA. 17-25. Rosenberg, L. B. (1993). Virtual fixtures: perceptual tools for telerobotic manipulation. IEEE Virtual Reality Annual International Symposium, 1993, Seattle, WA. 76-82.

15 Roth, P., Petrucci, L., Assimacopoulos, A., Pun, T. (2000). From Dots to Shapes : an auditory haptic game platform for teaching geometry to blind pupils. Computer Helping People with Special Needs, 2000, Austrian Computer Society, Karlsruhe, Germany. 603-610. Roth, P., Giess, C., Petrucci, L., Pun, T. (in press). Adapting Haptic Game Devices for non-visual Graph Rendering. First International Conference on Universal Access in Human-Computer Interaction, 2001, New Orleans, LA. S. C. Sahyun, Gardner, J. A. (1997). Testing the Equivalence of Visual and Auditory Graphs. International Conference on Auditory Display, 1997, Palo Alto, CA. Available WWW: http://www.physics.orst.edu/~sahyun/icad97/poster.html Sjöström, C. (1999). The IT Potential of Haptics touch access for people with disabilities. licentiate thesis. Center for Rehabilitation Engineering Research. Available WWW: http://www.certec.lth.se/doc/touchaccess/