Automatic Online Haptic Graph Construction

Transcription

1 Automatic Online Haptic Graph Construction Wai Yu, Kenneth Cheung, Stephen Brewster Glasgow Interactive Systems Group, Department of Computing Science University of Glasgow, Glasgow, UK {rayu, Abstract Just accessing graphs is a difficult task for blind and partially-sighted people let alone constructing graphs by themselves. By using multimodal Virtual Reality technologies, graphs rendered in a computer can be explored by blind people relatively easily. This has been validated in our research on multimodal data visualisation systems. In this paper we introduce a new development of a Web-based graph construction tool. People can access it using a standard Web browser. It can automatically generate three types of graphs: line graphs, bar charts and pie charts based on data entered by the user. The constructed graphs are presented to the user through visual, auditory and haptic means. A Logitech WingMan Force Feedback Mouse is used to convey the haptic information. Evaluations have been conducted to verify the usability of the system. Experiment participants have provided positive feedback on the tool. Keywords: multimodal, haptics, assistive technology, graph visualisation, graph construction 1. Introduction This paper describes the development of an automatic online graph construction tool. This tool is implemented using a Java Applet, the Immersion TouchSense Plug-ins and MIDI sounds. It provides an opportunity for users to create graphs in their Web browser. Multiple sensory modalities, including vision, audio and touch, enable universal access. We developed this graph construction tool particularly for blind and partially-sighted people. Three types of graphs including line graphs, bar charts and pie charts can be generated based on the data set provided by the user. In this paper we will first introduce the background of this work, we will then describe the development of the graph construction tool and discuss its evaluation. 2. Background Traditionally, constructing line graphs, bar charts or pie charts is not a straightforward task for blind people. Accessing and exploring such graphs is also a burden. However, graphs and other visualisation techniques are vital tools for understanding data efficiently and effectively, so this poses a major problem for blind people. Without access to graphs a valuable data comprehension tool is lost to them. Sighted school children are taught to use and create graphs by themselves. Blind school children also need to learn these skills if they are to keep up with their sighted classmates, but having no visual feedback is a major obstacle. Mathematics is a compulsory subject in the school curriculum so blind students need the tools to be able to use graphs and other visualisations. Moreover, we encounter graphs in our daily life. In order to mix with sighted people in school and at work, blind people need to know graphs and their interpretation. Without the ability to create and use graphs many educational and job opportunities are denied to blind people [1]. Special tools and materials have been developed to help blind people access and create graphs through their senses of hearing and touch. To access graphs, blind people s sense of touch is mainly used. Tactile diagrams generated on swell paper (this generates lines that stand above the surface of special paper after heating) or other materials are commonly used. The way of constructing a graph on raised paper is quite laborious and involves visual to haptic conversion, printing and raising process, and verification of raised information. A sighted person is required to do these tasks. However, there are situations in which blind people need to create graphs by themselves, especially in the school environment. Blind school children need to learn how to plot a graph based on a given set of data. Pencils and rulers are inappropriate so special instruments and materials have been developed. Most commonly used in schools are pins and rubber bands on a wooden drawing board. For example, to create a line graph, blind school children would use pre-prepared paper with raised gridlines to plot the graph. They would first mark the data points by pushing in drawing pins. Then they would wrap a rubber band around the pins to form a basic line. They could then feel the information on this graph with their hands. There are some problems associated with this kind of graph construction method. Firstly, the pins are sharp objects and could injure the student by accident. Secondly, the rubber band may not provide a true line representation. It may not go through

2 all necessary pins (Figure 1). Thirdly, the tension on the stretched rubber band could cause it to snap. Finally, a pre-prepared paper with raised gridlines is required. Once such graphs have been created it is hard to change them removing a pin to reposition it can cause the rubber band to come off the other pins. graphical feedback to visualise the data while blind people can use the audio and haptic features to extract information. Graphs are generated automatically based on the data set supplied by users (in a similar way to a tool like Microsoft Excel) therefore blind people can work independently. The constructed graph can also be printed out and raised on swell paper if necessary. As a result, blind people can use it to present ideas and communicate with their sighted colleagues through a familiar medium. Moreover the initial investment is very low, users only need to buy a Logitech WingMan Force Feedback Mouse (~ 60) to receive haptic information. Figure 1. A line formed by pins and a rubber band (Left, correct wrapping; right, incorrect wrapping as the rubber band has slipped off P 2 and now forms a triangle). There are other ways to construct graphs by using special instruments such as thermal or water-based pens on specially treated paper which is sensitive to the heat/ moisture change. Alternatively, a deformable plastic film can be used. By writing on it, the pressure deforms the paper and produces a trace which can be perceived by user s fingertips. However, these kinds of instrument are more suitable for free hand sketching and practicing signing a signature than plotting formally structured graphs. Once a graph has been constructed there is no easy way to remove a line if a mistake has been made; the user must start again. To address the problems which blind people facing in constructing graphs, we have developed an automatic, online graph construction tool which utilises the advantages of Virtual Reality and information technologies. Figure 2. Logitech WingMan Force Feedback Mouse. The tool is embedded in the Web pages hosted on our Web-site ( A simple tree structure directory is used to classify the types of supported graphs. A main page lists the hyperlinks to the three different types of graphs currently supported: line graphs, bar charts and pie charts. Once users select one type of graph, they will be taken to the page which consists of the plotting area, data entry field and control buttons (Figure 3). 3. System Overview The graph construction tool is a Web-based Java Applet. It has automatic graph construction capability. A standard Web browser with Sun Microsystems s Java and Immersion TouchSense plug-ins is required to run the graph construction tool (both of these are free downloads). The constructed graphs provide three forms of output: graphical, auditory and haptic. In order to feel the haptic representation, a Logitech WingMan Force Feedback mouse is used (Figure 2). This tool provides computer-generated graphs which are flexible and easy to change. To modify the graphs, users just need to re-generate the graphs, therefore the cost is extremely low. Representations are provided in multiple sensory modalities so that the visualisation outputs of the tool are universally accessible. Sighted people can use the Figure 3. A sample line graph from the graph construction tool. The plotting area occupies most of the screen and is located in the top section. The data entry field resides below the plotting area. At this moment, we only provide 10 data entry boxes so that a maximum of 10 data points

3 can be plotted into a graph. In the future we will make the number of possible data points flexible so an arbitrary sized data set can be used. Two control buttons are placed right next to the data entry field. One button labelled as Random is used to generate a random set of data, mainly for demonstration purposes. The OK button is for the actual graph construction. Once the data is entered, users press the OK button to create the graph in the plotting area. 4. Graph Implementations The graphs generated by the tool contain graphical, auditory and haptic features. The graphical features follow the same style as for standard graphs. The implementations of haptic and audio features vary according to the types of graphs. They are built on the findings of our previous study [2] which is our first attempt to present multimodal graphs on the Web Line Graphs In the current implementation, we use straight lines to link the data points together when creating line graphs. There is no smoothing or curve fitting for the data points. This gives a true indication of data trend. To represent a line in haptics, the rectangular enclosure effect supported by the Immersion TouchSense plug-ins is used. An enclosure effect defines an area, either rectangular or elliptical, in which the mouse cursor movement is constrained. The mouse cursor is bound within the area unless users force the cursor through the bounding edges. On the line graph, only the top and bottom edges of the enclosure effect are activated, and the gap between these two edges is kept to about 1 pixel thick so that a narrow path is formed. By joining all the enclosure effects together, a data line is created. The audio is implemented using Java MIDI. The sound is played continuously and varies in pitch according to the mouse cursor position. High data value points are mapped to high pitched notes and vice versa [3]. Therefore, by moving the mouse along the line, various pitches will sound and inform users about the shape of the data. The sound is only played when the mouse cursor falls into the bounded area Bar Charts The bar charts are relatively easier to create. A simple rectangular effect without any modifications can be used to represent a bar. Users can feel the bar from its inner and outer edges. All the bars are located close to each other so the user can feel a set of bar shapes in a line across the screen. A discrete sound is used to present the bar value. The pitch-value mapping is used again; the higher the bar, higher the pitch. The sound is triggered when the mouse cursor entered into a bar Pie Charts A combination of rectangular and elliptical enclosure effects is used to construct a pie chart. An elliptical enclosure effect forms the circle of the pie. Rectangular enclosure effects with the same formatting as in the line graph assemble the divisions of the pie. Each rectangular effect is rotated in an angle that it projects from the centre of the pie to the edge. A discrete sound mapping is used again. The pitch is mapped to the proportion of the pie division. The same triggering mechanism is used to play the sound. 5. Evaluations A two-part evaluation has been conducted to investigate the usability of the graph construction tool. The first part of the evaluation involved testing the effectiveness of the audio and haptic representations. The experiment consists of audio only, haptic only and audio and haptic conditions. The experimental graphs are pie charts as experiments on line graphs and bar charts have been carried out in our previous studies [4 & 5]. The test participants were students in the Department of Computing Science at the University of Glasgow. They were blind-folded in the experiment. Using sighted participants first allows us to evaluate (and improve) the effectiveness of our designs before we test them with blind people (as our access to large numbers of blind participants is limited we want to ensure our designs are as good as possible before we test with them). Two questions were asked during the evaluation: Locate the largest & smallest divisions Locate the two most similar divisions in value The answer accuracy, task completion time and subjective workload were measured to assess users performance. In the second part of the evaluation we have run studies with blind people. Think aloud and user group discussion methods were used to assess the suitability and usefulness of the graph construction tools Part 1 Testing with sighted people Experiment Set-up The evaluation used a between-groups design. The main investigation issues of this evaluation were on the effectiveness of three different representation modes which were audio only, haptics only, and audio and haptics. A total of eighteen people were recruited and they were evenly divided into three groups. Each group of people attempted a set of sixteen graphs in each experimental condition. Each graph was associated with a specific question type. The first question type was asked on the first eight graphs while the next eight graphs used the second question type. The difficulty of each graph varied,

4 for example, some graphs contained predominantly larger or smallest portions. A three-minute time limit was placed on each graph. To measure the users performance, users answers and the time taken to acquire the answers were recorded. Moreover, the NASA workload index (TLX) was assessed on each user after the experiment [6]. Experimental Results The results of users answers are shown in Figure 4. The correct answers (in percentage) are plotted against the experimental conditions and separated in two groups of questions. Condition 1 (audio and haptics) maintains a high percentage of accuracy (over 70%) in two types of questions. Condition 2 (haptics only) received lowest accuracy (less than 20%). Overall, the users performed better with the first type of question than the second. The average task completion time with respect to two types of questions is shown in Figure 6. In the first questions, condition 2 (haptics only) obtained the longest time whereas condition 3 (audio only) received the longest time in the second questions. However the variation is very high in the condition 3 results. Time (Seconds) % 80% 0.0 Questions 1 to 8 Questions 9 to 16 Question Figure 6. Task completion time. Average Correct Answers (%) 60% 40% 20% 0% -20% Questions 1 to 8 Questions 9 to 16 Question Figure 4. Correct answers (Condition 1 audio with haptics; Condition 2 haptics only; Condition 3 audio only). Standard error bars are shown. Again the results of the first and second types of questions are combined and compared between experimental conditions (Figure 7). The overall results showed that the users took more time in the condition 2 (haptics only) and much less in the condition 1 (audio and haptics). The ANOVA and Tukey s HSD test revealed that there is a significant difference between condition 1 and 2. However, there is no significant difference between condition 1&3 and 2& Further analysis has been done on the overall results of the first and second questions (Figure 5). ANOVA and Post Hoc Tukey s HSD tests were performed to determine if the difference between the conditions were significant. The results showed that there was a significant difference between all conditions. Time (Seconds) % 20.0 Avg Correct Answers (%) 79% 80% 60% 40% 36% 20% 10% 0% Condition 1 Condition 2 Conditon 3-20% Condition Figure 5. Overall correct answers. 0.0 Condition Figure 7. Overall task completion time. The workload perceived by each user was calculated and an average figure for each condition was determined. Figure 8 shows the average task load of each condition. Condition 2 (haptics only) was perceived as the most difficult condition while condition 1 (audio and haptics) was the easiest one. The significance test shows that there is a significant difference between all conditions.

5 TLX Value Condition Figure 8. Overall task load index. Discussion & Improvement The experimental results confirm the findings of our previous evaluations on other types of graphs that the multimodal representation works better than the single modal. Users performed better in the audio and haptics condition and found it easy to use. Based on the findings of this part of the evaluation, some improvements have been made to the haptic and audio representations. A haptic groove has been added to the circumference of the pie so that users can trace the edge of the pie to get the most of the haptic information (Figure 9). The sound on the pie has been made shorter and crisper so that the redundant echo has been removed. Figure 9. Improved haptic pie modelling groove resistive walls In the first half of the experiment, users were asked to explore the Web page and enter data into the data entry fields. The purpose was to see whether they could successfully use the screen reader, JAWS (Henter-Joyce Inc.), to navigate the Web page as well as whether they could enter data easily to create graphs. During the experiment, users were encouraged to speak out their thoughts about the interface and the way of interaction. In the second half of the experiment, users were asked to perform the same task which sighted people did in the part one evaluation. They were given a set of six graphs to explore. The accuracy of their answers was noted but their comments on the graph representation were more important. The first user was visually impaired and usually used a screen magnifier to access computers. He found that the layout of the Web page is quite clear and easy to navigate although he was not familiar with JAWS and its audio feedback. Using the Tab key can cycle through the items on the Web page easily. Although JAWS repeats the labels on the data entry field, which could be confusing, he got used to it quite quickly. He found no problems in entering the data into the data entry fields. The Tab key was used to move the focus back and forth between the edit boxes. The screen reader provided adequate feedback to the data entry process. In exploring the graphs and locating the answers, he did not require too much time to pick up the skills to use the force feedback mouse and the features on the graphs. His answers matched with sighted people s, his answers locating the largest and smallest portions tended to be more accurate. He has found that audio and haptic feedback work well together. Audio is very helpful in judging the size of the portion so that he used audio to get most of the answers. Haptics also helped to determine how big things are. The force on the groove was not strong enough for him to stay on so it was quite easy to wonder off from the pie edge. The second user became blind in the later stages of his life and he relies on screen readers to access computers. Therefore he is very skilful in using JAWS. He had no difficulty in navigation on the Web page. He pointed out that it would be nice that the reading of the data entry labels can be turned off and only reads the values in the edit boxes. That would save him time in the data entry process Part 2 Testing with blind people During the exploration of the graph using the force feedback mouse, he found that moving around the pie In the second part of the evaluation visually impaired people were tested. Three different aspects of the investigation, which were navigation on the interface, data entry, information extraction, have been carried out. However, they were difficult to be evaluated in a quantitative test therefore a user group discussion and think aloud method was used to get users feedback on the improved tool. Due to the limited resources of blind people in the local area and slow response to our experiment advertisement, at the time of writing of this paper only two blind people took part in the evaluation. edge is definitely useful to get the information. Moreover, joints between sections on the pie are easy to spot. On the other hand, he suspected that the small workspace of the force feedback mouse would be difficult for older people as delicate manipulation is required. His answers to the questions are very accurate due to his ability of the use of the mouse and the audio feedback. 6. Conclusions The Web-based graph construction tool has been tested by sighted and blind people. Both groups of people found it

6 easy to use when audio and haptic feedback were available. Their performance in the experiment has confirmed the effectiveness of the multimodal representation. Moreover, their feedback indicates that the Web-based graph construction tool is useful. There are some problems when a screen reader is used on the Web pages such as the reading of redundant information. This can be improved in our future development of the tool. Our goal is to provide a flexible interactive tool for users to draw graphs on the Web pages. 7. Acknowledgements This research work is part of the Multivis project which is funded by EPSRC Grant GR/M44866, ONCE (Spain) and Virtual Presence Ltd. 8. References [1] Dimigen G., Scott F., Thackeray F., Pimm M., Roy A. W. N., Career expectations of British visually impaired students who are of school-leaving age, Journal of Visual Impairment and Blindness, 87, 1993, p [2] Yu, W., Reid, D. and Brewster, S.A. Web-Based Multimodal Graphs for Visually Impaired People. Accepted by 1st Cambridge Workshop on Universal Access and Assistive Technology (CWUAAT), Cambridge March [3] Mansur, D.L., Blattner, M. and Joy, K. Sound-Graphs: A numerical data analysis method for the blind. Journal of Medical Systems 9 (1985), [4] Yu, W., Ramloll, R., Brewster, S.A. and Riedel, B. Exploring computer-generated line graphs through virtual touch. Proceedings of The Sixth International Symposium on Signal Processing and Its Applications, vol. 1, pp , 2001 (Kuala-Lumpur, Malaysia), IEEE. [5] Yu, W., and Brewster, S.A. Comparing Two Haptic Interfaces for Multimodal Graph Rendering. Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (Haptics 2002), Florida, USA, IEEE, pp [6] Hart S. G. and Wicken S. C., Workload assessment and prediction, in MANPRINT, an approach to systems integration. 1990, Van Nostrand Reinhold: New York.