Article A comparison of three nonvisual methods for presenting scientific graphs ROTH, Patrick, et al. Abstract This study implemented three different methods for presenting scientific graphs to visually impaired people: audition, kinesthetics, or a combination of the two. The results indicate that the combination of both audio and kinesthetic modalities can be a promising representation medium of common scientific graphs for people who are visually impaired. Reference ROTH, Patrick, et al. A comparison of three nonvisual methods for presenting scientific graphs. Journal of Visual Impairment and Blindness, 2002, vol. 96, no. 6, p. 420-428 Available at: http://archive-ouverte.unige.ch/unige:47498 Disclaimer: layout of this document may differ from the published version.
1 Do You Feel What I Hear? Patrick Roth 1, Hesham Kamel 2, Lori Petrucci 1, Thierry Pun 1 1 Computer Science Department CUI, University of Geneva CH - 1211 Geneva 4, Switzerland Patrick.Roth@cui.unige.ch 2 The Group for User Interface Research EECS, UC Berkeley Berkeley, Ca. 94720, USA
2 Abstract In this research we implemented three different methods for presenting scientific graphs to blind and visually impaired people. Each rendering method employed either audition, kinesthetic or a combination of those two modalities. In order to allow for distance learning, we have used low cost portable devices for the output graph rendering. The three modes of representation were then compared by three separate groups of blind and visually impaired computer users. Each group consisted of four participants. Results reveal that the combination of both audio and kinesthetic modalities can be a promising representation medium of common scientific graphs for visually challenged people. KEYWORDS Blind, non-visual Computer Science Departmenand non-tactile graph representation, audio and kinesthetic rendering.
3 Introduction The representation of scientific graphs is among the teaching concepts that are difficult to electronically transform onto computers for blind and visually impaired people. Most often tactile channel (e.g., hard copies produced via Braille printers) seems to be privileged by specialists to convey such type of graphical information. However, this method in its nature does not provide an adequate solution when presenting graphical information via distance learning. Hard copies are static in nature (i.e., they cannot be modified) and tactile devices are expensive as well as hard to carry (Kamel & Landay, 2000). Therefore, stationary machines limit the autonomy that is provided by the mobility to blind users (Cartwright, 1994). Over the past several years, triggered by the advent of the virtual reality, various investigations have been made by researchers regarding the presentation of non-visual scientific data using either audio or kinesthetic modality (Flowers & Hauer, 1995; Fritz & Barner, 1999; Mansur, Blattner, and Joy, 1985; Ramloll, Yu, Brewster, Riedel, Burton, and Dimigen, 2000; Sahyun & Gardner, 1997). However, very little work has been undertaking in comparing these two nonvisual rendering techniques. In addition, most often researchers conduct experiments using high cost devices (e.g., the PHANToM Haptic interface). These devices present a problem to most ordinary blind users since they cannot afford them. Addressing the problems mentioned above, we have designed three methods of non-visual graph representation, making use of two different modalities. In the first rendering method, we used auditory feedback (non-speech sounds). In the second method, we utilized a kinesthetic interaction (the use of a Haptic mouse). As for the third rendering method, we combined the first and the second modalities. In order to fit with the distance-learning concept and to address the mobility issues, we rendered the graphical output by using low cost mobile devices (i.e., a force feedback mouse).
4 Audio and Kinesthetic Representation The three methods that we developed represent scientific graphs provide two different categories of graphical information. The first category conveys information regarding the overall shape of the graph. Where the second category provides extra information regarding several aspects of the graph: The maximum and minimum values. All the point(s) where the graph crosses either the horizontal or the vertical axis. Using the kinesthetic rendering, the user recognizes a starting or an ending position when the Force Feedback (FF) device is pointing at either one of them. Audio Encoding For the audio rendering of the entire graph, we have implemented a traditional approach. We presented the graph by a single tone where its pitch represented the Y coordinate and the duration represented the X coordinate (Flowers & Hauer, 19..; Mansur, Blattner, and Joy, 1985; Sahyun, 1999). In addition, to improve the rendering, we immerged the tone into a 2D sound plane (Roth, Petrucci, Assimacopoulos, and Pun, 2000) in which the stereo effect clearly mapped the X coordinate and the elevation effect enhanced the Y coordinate. This plane is vertically oriented and virtually located in front of the blind listener.
5 Figure 1. 2D sound plane rendering For instance, if the rendered graph is a linear curve (i.e., y=a*x+b, where a and b are constants and a is superior to zero), the user hears a tone ascending obliquely from the bottom left back to the top right channel (see Figure 1). When the shape rendering is ascending, the tone increases in frequency. For the rendering of the extra information of the graph, each output is characterized by a different non-speech sound. These tones differ so they can be easily recognized if they are played in at the same time (Brewster, 1995). Also, the tones are short where the blind users can quickly interpret them. Kinesthetic Encoding The haptic rendering of the graph is based on a virtual fixture (Rosenberg, 1993). In this method, the blind user by holding the mouse trails the edge of the graph following its path. Concretely, we determine a point on the graph (P min ), and the distance between this point and the cursor position (c) forms the minimum distance (d) (see Figure 1.a). If the distance is below a given threshold, we activate the force (F) that attracts the mouse to the graph (see Figure 1.b). Therefore, when the mouse moves away from the graph, it automatically pushes back to it.
6 Distance threshold curve x max x max x max P min d P min F P min fixture region x min c x min c x min c Figure 2.a. Distance setting. Figure 2.b. Attraction force. Figure 2.c. Audio rendering. All the extra information of the graph is produced by vibrations. These vibrations are generated by a periodical effect that repetitively moves back and forth in a very short distance and with a relatively high frequency. We can thus differentiate between different vibrations by either varying their frequency, their direction, or both. Audio-Kinesthetic Encoding This encoding method combines the audio and the kinesthetic modalities. In this method, the information of the entire graph is rendered by the virtual fixture (explained in section 2.2). As for the extra information, we have replaced the vibration effects by non-speech sounds. In addition, we encoded the slope of the graph by two different tones (i.e., one tone each for each ascending or descending slope). This tone also varies in frequency proportionally to the slope incline (see Figure 3). Figure 3. Auditory coding of the slope
7 During the development of the virtual fixture, we have noted several problems that concerned the push back force limitation. This limitation due to the nature the haptic device, doesn t allow blind users to follow the path of the graph without deviating. To solve this problem, we developed a feature that we called auditory fixture. In this feature, we use sound as an alarm to inform the user that he leaves the graph. In this case, when the blind user moves the mouse (cursor) away from the curve fixture region an auditory signal is automatically played (see Figure 1.c). Hardware Configuration The evaluation of the system was made on a Pentium PII computer. The audio rendering was obtained by using the DirectSound3D 2 library on a SoundBlaster Live 3 sound card, and a Sennheiser HD580 4 headphone. We choose to work with a SoundBlaster Live and a Sennheiser II because they provide together an affective 2D sound rendering. As for the kinesthetic interaction, we worked with the Logitech WingMan Force Feedback 5 mouse. This mouse is much more affordable than using the PHANToM 6 device. To obtain a more detailed comparison between the two FF systems, please refer to the report written by Sjöström (1999) and (Roth, Giess, Petrucci and Pun, in press). System Evaluation We have conducted a series of experiments in order to study the efficiency of each of the three methods mentioned above. The three methods of representation were experiment by three groups (G1, G2 and G3) each consisted of four participants aged between 20-45 years old. We assigned one method to each of the three groups with two of the participants were congenitally blind and 2 library included on windows 98 or recent systems 3 available on regular computer stores or at www.soudblaster.com for about $200 4 available on music stores or at http://www.sennheiser.com/ for about $200 5 available on regular computer stores or at www.logitech.com for about $50 6 available at www.sensable.com for about $13500
8 the two others were visually impaired. G1 participated in recognizing scientific graphs via the auditory encoding method (see section 2.1). G2 examined the graphs using the kinesthetic encoding (see section 2.2). Finally, G3 obtained the scientific graphs using the audio-kinesthetic encoding (see section 2.3). During the experiment, the visually impaired participants were not blindfolded, however they did not have access to the computer screen. All of the participants had received some type of mathematical education background (viz., knowledge of scientific graphs.). We gave each of the participants the same amount of time (20 minutes) to get familiarized with the method they were testing. During this time, we briefly explained the method of representation they were going to experiment and we let them used on a simple graph (i.e., y = abs(x), 0 x 4). Tasks and Scenarios Each of the participants was given three different scientific graphs one at a time for evaluation: a linear function: y = 1-x, -2 x 2 (see Figure 4.a) a parabolic curve: y = x 2, -2 x 2 (see Figure 4.b) a periodical sine wave: y = sin(x), -4 x 4 (see Figure 4.c) Figure 4.a. y=1-x Figure 4.b. y=x 2 Figure 4.c. y=sin(x) For each graph, after interacting with it for a period that did not exceeds five minutes, each of the participants was asked to reproduce it. The blind participants reproduced it using a tactile medium, where the visually impaired used a pencil and a sheet of paper. The success of reproduction was based on two complementary criteria (C1, C2). The first criterion C1 is
9 subjective and relative to the level of general similarity between the real graph and the one that as been reproduced by the participant. As for the second criterion C2, we evaluated more specific similarities of the two graphs such as curvatures or symmetries. In addition, we asked all participants while drawing to place the horizontal and vertical axis. Our goal was to check the efficiency of the extra information rendered by the encoding method that was being tested. In this task, the criterion of success was based on the precision in which the participant placed the horizontal axis (criterion C3) respectively the vertical axis (criterion C4) on the graph. After completing each task, we asked all participants several questions regarding the characteristic of the graph rendered (e.g., the periodicity, minimum and maximum values, location, and continuity). Finally, after conducting the experiment, we then asked each of the three groups to evaluate the corresponded system they had tested. Results and Discussion The graph that represented the linear function, the results shown in Figure 5 reveal that the group G1 used the audio encoding had the most difficulties reproducing the linear function (25% of success for C1 and C2). As it is shown in Figure 6.a, the majority of G1 participants represented the linear function as a parabolic. y=1-x Percentage of Success 120% 100% 80% 60% 40% 20% 0% C1 C2 C3 C4 Criteria G1 G2 G3 Figure 5. For the linear function, percentage of participants of groups G1-G3 that correctly performed the criteria C1-C4
10 All the participants of G1 agreed that the auditory rendering did not have sufficient information concerning the linearity. However, this problem did not occur to the groups G2 (75% of success for C1 and C2) and G3 (100% of success for C1-C4) which used kinesthetic as principal modality. Figure 6.a. Linear function reproduced by participant from G3 Figure 6.b. Linear function reproduced by participant from G1 As Figure 7 show, no participants of the three groups had any difficulty to recognize then reproduce the parabolic graph (100% of success for C1). However, we noted some imprecision y=x 2 120% Percentage of Success 100% 80% 60% 40% 20% 0% C1 C2 C3 C4 G1 G2 G3 Criteria Figure 7. For the parabolic function, percentage of participants of groups G1-G3 that correctly performed the criteria C1-C4
11 for the three groups and more for the group G3 (100% of success for C1) in reproducing the exact symmetry of the parabolic (see Figure 8.a). The participants of G2 were not able to recreate their mental perception of the horizontal axis (0% of success for C3) (see Figure 8.b). We believe the reason is that at the origin there were two different vibrations (the first was for the x axis and the second was for the y axis). This group expressed that they could not distinguish between them. One possible solution to this problem would be to replace these vibrations occurring at the same time by their consecutive output. However, such approach may require the participants to spend more time to recognize the graphs. Figure 8.a. Parabolic function reproduced by participant from G3 Figure 8.b. Parabolic function reproduced by participant from G2. As shown in Figure 9, G1, G2 and G3 were capable to reproduce a recognizable overview of the y=sin(x) Percentage of Success 120% 100% 80% 60% 40% 20% 0% C1 C2 C3 C4 G1 G2 G3 Criteria Figure 9. For the periodical function, percentage of participants of groups G1-G3 that correctly performed the criteria C1-C4
12 periodical graph (75% of success for C1). They however had difficulties in reproducing precisely the two curvatures at -π/2 and π/2 (see Figure 10.a,b,c). The audio rendering was not sufficient enough to precisely encode such kind of information for G1 (25% of success for C2). Concerning G2 and G3 participants (0% of success for C2), the principal drawback is due to the fact that the workspace of the FF device was small. Another problem is that the nature of this device required hand grasping, as opposed to finger grasping, which did not allow for high precision feedback (Cutkosky, 1990; Sjöström, 1999). Figure 10.a. Periodical graph reproduced by participant from G1 Figure 10.b. Periodical graph reproduced by participant from G2 Figure 10.c. Periodical graph reproduced by participant from G3 Therefore, we assume that by replacing hand grasping with finger grasping and increasing the size of the workspace as well as the force of feedback (see Figure 11.a and 11.b), this device will improve his precision rendering.
13 Figure 11.a. Force feedback device used for the experiment Figure 11.b. Modified device Most of the participants expressed their feelings regarding the three systems would be of a great assistance to them providing access to scientific graphs via computer systems. Since all blind people are not Braille readers, the participants who are used to Braille preferred interacting with the tactile medium over the kinesthetic rendering. Conclusion In this paper we have presented the design of three methods of non-visual graph representations as well as their user s evaluations. Our results show that in general the participants were capable to reproduce an overview of three scientific graphs they were tested. In some case however they proved difficulties in reproducing curvatures and symmetries. In our point of view, by tacking into account all the possible modifications, the combination of the kinesthetic and audio renderings seems to be the most promising method. As an extension of this presented work, we plan the development of a framework on the Web where blind users will be able to access scientific graphs on-line. This work will be done in order to promote the possibility of such technique of representation in the domain of distance education.
14 Acknowledgments This project is partly financed by the Association pour le Bien des Aveugles, Geneva, Switzerland. The authors are grateful to the Eingliederungsstelle für Sehbehinderte institution, Basel, Switzerland, and particularly to Denis Rossel, Denis Page and, Alain Barrillier for their help in the design and evaluation of the prototype. References Brewster, S.A., Wright, P.C. & Edwards, A.D.N. (1995). Parallel earcons: Reducing the length of audio messages. International Journal of Human-Computer Studies, 43(2), 153-175. Cartwright, G. P. (1994). Distance learning : A different time, a different place, Change, 26 (4), 30-32. Cutkosky, M.R., Howe, R.D. (1990). Human Grasp Choice and Robotic Grasp Analysis. In Dextrous Robot Hands, S.T. Venkataraman and T. Iberall, (Ed.), Springer-Verlag, New York, 1990. Flowers, J. H., and Hauer, T. A. (1995). Musical versus visual graphs: Cross-modal equivalence in perception of time series data. Human Factors, 37, 553-569. Fritz, J. P., Barner, K. E. (1999). Design of a Haptic Visualization System for People with Visual Impairments. IEEE Transactions on rehabilitation engineering, vol 7, No 3, 372-384. Kamel, H., Landay, J. (2000). A Study of Blind Drawing Practice: Creating Graphical Information Without the Visual Channel. The Fourth International Conference on Assistive Technologies, 2000, ACM Press, Arlington, VA. 17-25. Mansur, D.L., Blattner, M.M., and Joy, K.I. (1985). Sound graphs: a numerical data analysis method for the blind. Journal of Medical Systems, 9(3), 63-174. Ramloll, R., Yu, W., Brewster, S.A., Riedel, B., Burton, A.M. and Dimigen, G. (2000). Constructing Sonified Haptic Line Graphs for the Blind Student: First Steps. The Fourth International Conference on Assistive Technologies, 2000, ACM Press, Arlington, VA. 17-25. Rosenberg, L. B. (1993). Virtual fixtures: perceptual tools for telerobotic manipulation. IEEE Virtual Reality Annual International Symposium, 1993, Seattle, WA. 76-82.
15 Roth, P., Petrucci, L., Assimacopoulos, A., Pun, T. (2000). From Dots to Shapes : an auditory haptic game platform for teaching geometry to blind pupils. Computer Helping People with Special Needs, 2000, Austrian Computer Society, Karlsruhe, Germany. 603-610. Roth, P., Giess, C., Petrucci, L., Pun, T. (in press). Adapting Haptic Game Devices for non-visual Graph Rendering. First International Conference on Universal Access in Human-Computer Interaction, 2001, New Orleans, LA. S. C. Sahyun, Gardner, J. A. (1997). Testing the Equivalence of Visual and Auditory Graphs. International Conference on Auditory Display, 1997, Palo Alto, CA. Available WWW: http://www.physics.orst.edu/~sahyun/icad97/poster.html Sjöström, C. (1999). The IT Potential of Haptics touch access for people with disabilities. licentiate thesis. Center for Rehabilitation Engineering Research. Available WWW: http://www.certec.lth.se/doc/touchaccess/