An Investigation of Search Behaviour in a Tactile Exploration Task for Sighted and Non-sighted Adults.

Transcription

1 An Investigation of Search Behaviour in a Tactile Exploration Task for Sighted and Non-sighted Adults. Luca Brayda Guido Rodriguez Istituto Italiano di Tecnologia Clinical Neurophysiology, Telerobotics and Applications dept. Department of Neurosciences, Via Morego 30 Ophthalmology and Genetics, Genova University of Genoa, Italy luca.brayda@iit.it guido@unige.it Claudio Campus Cristina Martinoli Istituto Italiano di Tecnologia Istituto David Chiossone onlus Telerobotics and Applications dept. Corso Armellini 11 Via Morego Genova Genova martinoli@chiossone.it claudio.campus@iit.it Ryad Chellali Istituto Italiano di Tecnologia Telerobotics and Applications dept. Via Morego Genova ryad.chellali@iit.it Copyright is held by the author/owner(s). CHI 2011, May 7 12, 2011, Vancouver, BC, Canada. ACM /11/05. Abstract In this work in progress we propose a new method for evaluating objectively the process of performing a tactile exploration with a visuo-tactile sensory substitution system. Both behavioral and neurophysiological cues are considered to evaluate the identification process of virtual objects and surrounding environments. Our experiments suggest that both sighted and visually impaired users integrated spatial information and developed similar behavioural and neurophysiological patterns. The proposed method could also serve as a tool to evaluate touch-based interfaces for application in orientation and mobility programs. Keywords Cognitive Maps, Disability Access, Performance Metrics, Assistive Technology, Tactile & Haptic UIs. ACM Classification Keywords H.5.2 Information Interfaces and Presentation: User interfaces, Evaluation/ methodology. General terms Human Factors, Measurements, Performance.

2 Introduction In several contexts it is difficult, if not impossible, for humans to use the visual channel to acquire geometrical information for exploration and navigation purposes. It is then necessary to find alternative means to display spatial information. A way to do that is through sensory substitution systems. For blind people the problem is well known, and several works giving alternative representations of geometrical or syntactical information through touch, speech and heat have been proposed, with contradictory results. The earliest sensory substitution devices for visually impaired users converted visual stimuli to tactile representations [1]. Since then, many advances have been made due to the exploitation of tactile displays developed for telerobotics, tele-presence and virtual reality fields [2]. However, many of these have been limited to prototypes, due to an unbalanced technological effort that is more dedicated to developing sensors to capture the world than to understanding how efficient the sensory feedback mechanism is [3]. Additionally, there is a need for objective means of evaluating the degree of sensory transfer (in this case: from tactile input to mental spatial representation). Current performance driven metrics are applied following an unlimited training session [4]. However, the underlying mental mechanisms which participate in building an active spatial imagination, and which are known in neuroscience, are not yet being evaluated with regards to computer-human interfaces. Dynamic changes in EEG, for example, were reported while processing visually/haptically presented spatial information for both blind and sighted people [5]. the DIGEYE (DIGital EYE) system, lets visually impaired users explore unknown virtual objects [6], also finding EEG patterns related to spatial imagination (blindfolded sighted users) [7]. Motivated by the initial success of our device, we compare here sighted users and visually impaired users directly in the learning phase, to address the following research questions: 1. Do sighted and visually impaired users develop identifiable strategies when reconstructing VR-based objects with touch? 2. If yes, how similar are these strategies? 3. Is there a neurophysiological measure, which could be a good candidate for describing the presence or absence of spatial processing in both populations? 4. Are strategies and neurophysiological measures related to each other in both populations? Before reporting on results and conclusions, we describe the DIGEYE system and the experimental protocol adopted. The DIGEYE system Figure 1. Block scheme of the DIGEYE system We have shown that a sensory substitution interface,

3 of metrics linked to the degree of visuo-tactile information transfer. Ideally, these metrics can also be used either to inform the user or, indirectly, to correct the stimuli and tune the interface. Figure 2. Interaction with the single taxel of the TAMO device: the stalk rises/lowers at transitions of H (height) values. The DIGEYE system is depicted in Fig. 1. It comprises three parts: first, a tactile device coupled with a sensing tablet, second, a software handling a Virtual reality Environment (VE) synchronized with the first part, third, a software module to record online and to process offline both user behaviour and neurphysiological status. Specifically, given some height function, H=f(X,Y), a TActile MOuse-shaped (TAMO) device (see Fig. 2) is able to display the height information H at any absolute position (X,Y) of the 210x297 mm tablet. While users move TAMO, the VE receives the current position and sends back the values of H wirelessly to the controller inside TAMO. The controller generates tactile feedback by activating a stepper motor, which raises a stalk proportionally to H, thereby bijectively generating a taxel for each pixel of the tablet. Users seek spatial information by freely moving the TAMO device and attempting to reconstruct spatial information about the object or environment. More details are available in [6]. Experimental Setup 10 volunteers, naïve to the aims of the study, participated in this preliminary experiment. 5 male, visually impaired (totally blind) participants (mean age 31 ± 7), were selected by the David Chiossone Rehabilitation Institute for Visually Impaired in Genoa. 5 other sighted subjects participated blindfolded (3 females, 2 males; mean age 38 ± 15). Both males and females were chosen because no significant gender related asymmetries for this kind of task were found in the literature. Approval by the local ethics research committee and the written informed consent according to the declaration of Helsinki were obtained. None of the participants had any medical factors which were believed to reasonably affect experimental results. The experimental protocol, depicted in Fig. 3, comprised 4 conditions of increasing complexity: a pure motor session with no tactile feedback, and three objects with one, two and four levels (the presentation order is not While using the sensing device, ElectroEncephaloGram (EEG), ElectroCardioGram (ECG), ElectroMyoGram (EMG) and ElectroOculoGram (EOG) biodata are recorded and synchronized with behavioral data (i.e. X,Y and H values). The output of the system is a series Figure 3. Experimental protocol to evaluate the visuo-tactile sensory transfer with the DIGEYE system.

4 randomized). Users were instructed to explore each object 10 times. Every exploration trial was triggered by a start and a stop sound and lasted 10s. Ten seconds of rest were given between trials, such that for each condition users were recorded for 200s (10*(10+10)). 2 minutes of rest were given between conditions, as well as at the beginning and at the end of the experiment. The processing of biodata was carried in collaboration with the Clinical Neurophysiology of the Genoa University, who also hosted the actual experiments. EEG was recorded with Ag/AgCl electrodes at 14 active scalp sites according to the international 10/20 system, sampled at 1024 Hz and synchronized with behavioral data. All EEG-based measures were normalized with respect to signals obtained during resting epochs. EEG was processed with EEGLab software. More details can be found in [7]. Note that abstract geometrical virtual objects (shown in Fig. 3) were chosen on purpose, to minimize the contribution of past experience of users, which could be a possible source of bias. Note also that constraining the exploration to 10 seconds has three advantages: first, users are stimulated to use their natural cognitive resources as much as possible; second, the reduced tactile information enhances the use of spatial memory, which is a natural condition in everyday orientation and mobility (O&M) tasks for visually impaired users; third, processing of brain signals by means of averaging is more practical. User strategies To address the question: Do sighted and visually impaired users develop identifiable strategies when reconstructing VR-based objects with touch? we observed the behavioural data. Fig. 4 depicts the 3D Figure 4. Strategy (lines) and Decision Points (circles) of one sighted subject (left) and one visually impaired subject (right) in the 4-step scenario. points explored by one sighted user (left side) and one visually impaired user (right side), while exploring the most complicated scenario (4-steps). Lines with the same color denote (X,Y) points with the same H value. Subjects appear mainly to develop two kinds of strategies. One is a back-and-forth "grid" strategy, more visible on the left, aimed at crossing the total length/width of the object. The second is a "Z-scan" strategy, more visible from the red line of the right sub-picture, aimed at detecting the direction of the virtual walls by repeatedly scanning and sampling the up/down transitions. The first strategy complies with results published by Lahav [4]. The second strategy was interesting for the involved O&M operators, since it mimics within this virtual scenario the movement of a white cane in the real world. This confirms a possible strong link, which still needs to be thoroughly explored, between visually impaired users performance in real and virtual environments. On average, both groups allowed us to identify their apparent strategy, so that the first question can be positively answered. Decision Points To identify how subjects reacted to zones with different information density, we wondered if a measurable way of representing a strategy could exist. This would tell us if the chosen strategy is random or if it follows a certain

5 pure motor 1-step 4-step sighted users a c e visually impaired users Figure 5. Decision Points depicted for both populations and for three conditions pattern. In Fig. 5 we sought for points (X,Y) where users decided to change the direction of the exploration. We defined these Decision Points (DP) as the upper 5% percentile of the distribution of curvatures, e.g. points where the radius of circles approximating the paths is small. The DPs (depicted as green circles, also visible in Fig. 4) can be observed for all sighted users as compared to visually impaired: in subfigures a and b a uniform distribution indicates absence of strategy (in fact no stimulation is provided and H=0 always), while in subfigures c and d clearly more DPs are found nearby borders, i.e. just before or after transitions of H. Visually impaired seem to explore more zones far apart from their body (which is close to the lower border of the tablet), but the behavior is very similar. More interesting is the observation of fig e and f: the zones hosting more DPs are almost the same. b d f Those clouds of DPs tend to form a cross along the most informative directions, with a higher density close to the central area of the environment (which has the maximum concentration of information). This is a hint that both sighted and visually impaired have their own exploratory strategies, and that they are integrating the spatial information logically. Concerning the question If yes, how similar are these strategies? we can preliminarily say that strategies are very similar. Neurophysiological hints To answer to the question is there a neurophysiological measure, which could be a good candidate for describing the presence or absence of spatial processing in both populations? we observed that the power spectral density of the overall EEG signals averaged across electrodes in the fronto-central region was reduced in the interval [8-32 Hz]: this is due to the pure movement of the mouse. This pattern is an expected result and is common to both populations. However, Fig. 6 shows that the power in the theta waves band [4-8Hz] significantly differs between pure motor and active explorations. It also shows that, while in pure motor sessions a decreased power is observable, in contrast and only when exploring an environment, both groups show an increase in the activity of mind in a brain area (fronto-central) and in a frequency band (theta) which are in literature related to the processing and to the mental reconstruction of spatial information. More statistical evidence is to be provided with an increased sample size in the next steps of our investigation. A metric related to the relative power in theta can be the candidate to answer to the research question. Finally the question Are strategies and neurophysiological measures related to each other in both populations?

6 Acknowledgements We thank F. Famà, C. Bruzzo and L. Lucagrossi for their precious technical support, as well as J. Jenvrin and S. Saliceti for engineering the tactile device, P. Milgram and S. Hennig for precious feedback and suggestions. Figure 6. Power variations in the theta [4-8 Hz] band, in the fronto-central zone, in function of condition and visual capability. is still an open issue in our study, but our preliminary results suggest that uniformly distributed DPs correspond to absence of strategy and to absence of theta activation, while geometrically distributed DP corresponds to presence of a strategy and to presence of theta activation. Conclusions and Future Work In this work we showed that a small sample of both sighted and visually impaired users explore and reconstruct virtual objects by tact in a similar way. Since according to experts in rehabilitation the users show similar behaviour than in real context, the DIGEYE system, once optimized, can serve as tool to objectively assess and possibly improve the orientation strategies in a virtual environment for Orientation and Mobility programs. Our metrics can also serve as tool for objective assessment of tactile interfaces conveying spatial information. References [1] Goldish L H., Taylor HE., "The Optacon: A Valuable Device for Blind Persons ", New Outlook for the Blind, 68, 2, 49-56, (1974) [2] Yamamoto A. et al, Electrostatic tactile display with thin film slider and its application to tactile telepresentation systems, Vis. and Computer Graphics, IEEE Trans. on, vol. 12, no. 2, pp , (2006) [3] Y. Visell. Tactile sensory substitution: Models for enaction in HCI, in Interacting with Computers, 21(1-2):38 53, [4] Lahav O. et al, "BlindAid: a Learning Environment for Eablig People who are Blind to Explore and Navigate throughunkown Real Spaces", IEEE Virtual Rehabilitation, [5] Caplan JB. et al, Human theta oscillations related to sensorimotor integration and spatial learning, J. of Neusc 23, no. 11, pp (2003) [6] Chellali R., Brayda L., Fontaine E., "How Taxelbased displaying devices can help blind people to navigate safely" - ICARA [7] Campus C., Brayda L. et al, Evaluating visuotactile sensory substitution for navigation in virtual worlds: preliminary neurophysiological assessment and results on a tactile-based interface, 9th IEEE-RAS Int. Conf. on Humanoid Robots, 2009 [8] Song, J;Yang,S, Touch your way: haptic sight for visually impaired people to walk with independence, in Proc of CHI 2010