Design of a Multi-Functional Module for Visually Impaired Persons

Transcription

1 Article Design of a Multi-Functional Module for Visually Impaired Persons Dong-Soo Choi 1, Tae-Heon Yang 2, Won-Cheol Bang 3 and Sang-Youn Kim 1, * 1 Advanced Technology Research Center, Korea University of Technology and Education, Cheonan-si, Korea; mycds88@koreatech.ac.kr 2 Department of Electronic Engineering, Korea National University of Transportation, Chungju-si, Chungbuk, Republic of Korea.; thyang@ut.ac.kr 3 Ultrasound R&D Group, Healthcare & Medical Equipment Business, Samsung Electronics Co., Ltd, Korea; wc.bang@samsung.com * Correspondence: sykim@koreatech.ac.kr; Tel.: Authors 1 and 2 contributed equally. Abstract: Measuring a distance through an ultrasonic sensor and creating haptic alert information by a vibrotactile actuator are two major functions in an electronic aid for a visually impaired person. It is quite challenging to combine and to efficiently execute these two functions with a single module. Thus, this study presents a new structure that measures the distance and generates haptic information with only one module. The design focus is to maximize the vibrotactile amplitude from the module and to minimize the measured distance error using the proposed module. In order to evaluate the performance of the proposed module, a test setup was constructed. Using the setup, the vibrotactile strength of the proposed module was investigated according to the input frequency, and then, the distance between the proposed module and a target object was measured. The results show that the proposed module effectively produces haptic information while measuring the distance well between the module and a target object. Keywords: Miniature sensing and actuation module, haptic, ultrasonic sensor, and vibration. 1. Introduction Due to the advancement of hardware technology, many efforts have been made to replace walking sticks and guide dogs for visually impaired people [1-13]. Most of these efforts have been developed in the form of a handheld device including sensors to detect obstacles or to measure a distance between the handheld device and a target object [3-13]. There are three major candidates (cameras, laser scanner sensor, and ultrasonic sensor) for detecting obstacles. In the case of a stereo camera, we acquire left and right images using two cameras and then compute the three-dimensional space coordinates (x, y, z) based on the two images. L. A. Johnson and C. M. Higgins developed a camera belt with two webcams and a tactor belt, consisting of 14 vibrating motors. The stereo camera belt finds obstacles using depth algorithm and sends corresponded signals to the tactor belt [1]. J. Zelek et al. developed a stereo-vision system that included a tactile unit for visually impaired people [2]. In their work, a Pixel-to-Pixel stereo algorithm was used to obtain a depth map. A laser sensor obtains the distance between a sensor and a target object using a pulse modulation method, a frequency modulation method, a phase shift method, or an interferometry method. J. M. Benjamin and N. A. Ali developed a laser cane that included three receivers and three different beams (a downward beam, a straight-ahead beam, and an upward-looking beam) to detect various obstacles such as drop-offs, straight-ahead, and head-height obstacles [5]. D. Yuan and R. Manduchi developed a handheld device consisting of a laser and a camera for a laser triangulation system for range sensing [6]. The infrared (IR) sensor, which is composed of a transmitter and a receiver, is also used for measuring the distance from the sensor to a target. Its operating principle is similar to the laser sensor by the author(s). Distributed under a Creative Commons CC BY license.

2 J. Villanueva and R. Farcy developed an active optical pathfinder using an IR detector [7]. A. A. Nada et al. introduced a smart sensor composed of two IR sensors for detecting horizontal and inclined obstacles [8]. Among the sensors for detecting an obstacle and measuring the distance between a sensor and an obstacle, the most common module that is used in hand-held devices for visually impaired persons is an ultrasonic sensor [9-13]. An ultrasonic sensor consists of two main parts; one creates an ultrasonic wave, and the other detects the ultrasonic wave, which is reflected back to the object. The distance between the ultrasonic sensor and the object can be obtained by computing the time from the moment the ultrasound is fired to the moment it returns. The presence of obstacles or the distance to obstacles detected by the ultrasonic sensor is transmitted to the visually impaired through an auditory channel. However, this information may not be conveyed to the visually impaired persons when they walk on a noisy street. Even if visually impaired persons are traveling on a street with just a little noise, they have to always pay attention to the voice from the portable device while trying to find their way. Therefore, there is a possibility that the visually impaired will be vulnerable to surrounding danger signals (for example, a car's horn or surrounding help sounds). In order not to face visually impaired persons with such a dangerous situation, most devices for blind people transmit tactile information in addition to auditory information. S. Gallo et al. developed a white cane grip consisting of an ultrasonic sensor and a vibrotactile module [10]. D. Dakopoulos and N. G. Bourbakis introduced an electronic travel aid including a tactile module for visually impaired persons and then conducted a comparative study for its evaluation [11]. Y. Wang and K. J. Kuchenbecker developed a haptic alerting module consisting of an eccentric motor and an ultrasonic sensor and attached the module to a traditional white cane [12]. C. Nguyen presented an obstacle detector, which distinctly provides different haptic sensations to blind people for each distance range [13]. In these systems, a piezo material is widely used to fabricate the transmitter of an ultrasonic sensor. The piezo material rapidly oscillates more than 20 khz to create an ultrasonic wave. In other words, there is a possibility that the transmission part of the ultrasonic sensor can be used as a vibration motor. However, it is not easy to increase the vibration strength of the transmitter of the ultrasonic sensor as in conventional vibration motors. The primary goals of this study were to construct a new single module that can generate a vibration strong enough to stimulate human skin while measuring the distance and to evaluate and assess its performance. The next section explains the design of the proposed module and its working principle. After describing the experimental setup, a series of tests were performed to show that the proposed module produces a strong enough vibration to stimulate the human skin while measuring the distance between the module and a target object as well. 2. Design and Fabrication of the Proposed Ultrasonic Sensor Integrated Haptic Actuator Module Generally, an ultrasonic sensor consists of a transmitter and a receiver. The ultrasonic transmitter, which is fabricated using a piezo material, generates ultrasonic waves by rapidly vibrating the piezo material at a frequency of 20 khz or higher. When a voltage is applied to a piezoelectric material, the size and the shape of the piezoelectric material are changed. As soon as the voltage is removed, the piezoelectric material is returned to its original shape. Due to this effect, applying an AC voltage to a piezoelectric material causes the material to vibrate. However, vibrations from the ultrasonic transmitter are too weak to be detected by humans. Therefore, in this paper, we propose a new structure that can be used as a haptic actuator and also as an ultrasonic transmitter. In order to provide various tactile sensations, we designed the module to produce a strong vibrational force (3 G of acceleration [14]) at a resonant frequency. Furthermore, to generate ultrasonic waves, the module should produce at least 20 khz of vibrations. Among the many types of materials, we decided to use a piezo material because it has a wide enough vibration frequency bandwidth, and it can easily be redesigned for various structures. Figure 1 shows the schematic view of the proposed miniature sensing and actuating module. The proposed system includes a top cover, a horn array, a permanent magnet, three bimorph piezo-

3 actuators, a bottom cover, and an elastic spring. To maximize the performance of the proposed module, it is necessary to consider a structure that can combine two or more energies. In our proposed system, an elastic spring was added to provide an additional elastic returning force. We attached two piezo actuators to one side of the elastic spring and attached one piezo actuator to its opposite side. The elastic spring with the piezo-actuators was fixed on the bottom cover after bonding the permanent magnet to the elastic spring. The fabrication process was done by putting the horn array on the top of the two piezo actuators and fitting the top cover in the body of the proposed module. Bottom cover Permanent magnet Elastic spring Piezo actuator Figure 1. Schematic illustration of the proposed module. Top cover Horn array Figure 2 shows the working principle of the proposed module. The top and bottom covers are made of pure iron. When a voltage input is applied to the bimorph piezo actuators, they are bent down, and this bending motion causes the spring to extend downward. The magnetic force is inversely proportional to the square of the distance between the permanent magnet and the iron. If the spring is stretched to the downward direction, the distance between the permanent magnet and the iron (the bottom cover) becomes closer, and the magnetic force suddenly increases. Due to this increased magnetic force, the permanent magnet moves to the bottom cover and is stuck on the cover, and the spring is extended downward. As a result, such motion further accelerates the bending motion of the piezo actuator. If we apply 0 V to the piezo actuator, the actuator returns to its original state. This returning force restores the spring to its original shape and makes the magnet fall off the bottom cover. Therefore, an AC voltage input makes the piezo actuator with the elastic spring vibrate. Horn array Displacement Elastic spring Permanent magnet Initial State Bottom cover Piezo actuator Final State

4 Figure 2. Working principle of the proposed module. As we discussed before, an elastic spring was used for maximizing the repulsive force. The simulation of the elastic spring was conducted using a 3D analysis tool (COSMOSXpress). Stainless steel was selected as the material for the elastic spring because it is light and has good elastic characteristics. Figure 3 shows the simulation result of the elastic spring. As shown in Figure 3, a thin plate spring design with a spiral pattern was selected in order to minimize the volume of the transducer while providing a sufficient restoring force. As the final material for the spring, Stainless Steel 304 was chosen because it is light, non-magnetic, and robust to rust, and furthermore, it has good elastic characteristics. The outer size of the spring was determined to be 20 mm 20 mm which takes into consideration the overall size of the transducer and the placement of the piezo actuator. In order to improve the performance of the proposed module, an elastic restoring force was initially stored by the attractive force of the permanent magnet. Figure 3 shows an example of the FEM analysis when the elastic spring is fully extended with a maximum deflection of 1.4 mm. Figure 3. FEM simulation result of the elastic spring. In order to determine the thickness of the spring so that the restoring force can be initially stored without any plastic deformation by the attraction force of the permanent magnet, a parametric study was performed using an FEM analysis tool. Figure 4 summarizes the simulation results and shows the elastic returning force of the spring at a maximum displacement of 1.4 mm with varying spring thicknesses. The required restoring force should be larger than the attraction force of the permanent magnet. Because the attractive force of the permanent magnet is about 5 N, the thickness of the spring was determined to be 0.35 mm based on the graph with the elastic returning force and the spring thickness (Figure 4). Figure 4. The simulation result of the elastic spring: the relationship between the thickness of the elastic spring and the elastic returning force. In order for the proposed structure to operate well as an ultrasonic transmitter, the piezo actuator attached to the elastic spring should be capable of producing displacements greater than 1.4

5 mm initially deformed. A FEM simulation was performed to verify that the piezo actuators mounted on the spring produces a displacement greater than the maximum displacement of 1.4 mm as shown in Figure 5. Because the piezo actuator generates a large force of 100 N or more, no simulation of the force was performed. The simulation results show that the piezo actuators can generate displacements up to about 3.5 mm in the spring of the proposed structure as shown in Figure 5. This result shows that the proposed ultrasonic transducer composed of piezo actuators, a spring, and a permanent magnet can move up and down sufficiently to generate the maximum acoustic radiation force. Figure 5. FEM simulation result of the displacement occurring when the piezoelectric actuators push each edge of the spring. Figure 6 shows the fabricated proposed module and its components. The proposed module consists of a permanent magnet, a horn array, a top cover, a bottom cover, an elastic spring, and piezo actuators. A permanent magnet is attached to the elastic spring, and then, the elastic spring with the piezo actuators is placed inside the bottom cover. The piezo actuators are bent, and this bending motion makes the elastic spring stretch. We put the horn array on the top of the piezo actuators in order to send the air in it forward, and we fitted the top cover in the body of the sensing and actuating module. The top cover and bottom cover were made of pure iron (KSC 2504, Carbon less than 0.8 %). The size of the proposed module ( mm 3 ) is considerably small enough to be embedded into hand-held devices. Horn array Top cover Bottom cover 6mm Elastic spring with piezo actuator (top) Elastic spring with piezo actuator (bottom) Figure 6. Proposed fabricated module with its components. 3. Performance Evaluation of the Proposed Sensing and Actuating Module A distance measurement experiment was conducted to investigate the possibility of an ultrasonic transmitter. Figure 7 shows the experimental environment consisting of the proposed module, an ultrasonic receiver, a microcontroller, a tape measure, and a PC. We placed the tape measure on the rail and fixed the proposed module to one end of the rail. The ultrasonic receiver was

6 Measured distance(m) Measured distance error (%) Preprints ( NOT PEER-REVIEWED Posted: 29 June 2018 attached to the other side. The distance between the proposed module and the ultrasonic receiver was measured as we changed the ultrasonic receiver from 0.5 m to 2 m at 0.25 m intervals. All tests were done in ten times. Microcontroller PC Tape measure Proposed module Sonar receiver Figure 7. Experimental environment for measuring the distance using the proposed module. Figure 8(a) shows the measured distance between the proposed module and the receiver. To accurately investigate the performance of the proposed module as an ultrasonic transmitter, we computed the distance error. The distance error was within 3% when the real distance was lower than 1.5 m, and it significantly increased as the real distance increased to greater than 1.75 m. When the real distance approached 2.5 m, the distance error was over 30 %. The proposed module can be used as an ultrasonic transmitter in the range of 0 ~ 1.5 m Distance between a transmitter and a target object (m) (a) Distance between a transmitter and a target object (m) (b) Figure 8. Experimental environment for measuring the distance using the proposed module. The purpose of this study was to develop a module that can not only create haptic information but also measure a distance. Therefore, we measured the distance between the proposed module to a target object, and then, we investigated whether the proposed module generates a continuous haptic sensation while transmitting the ultrasonic wave. In the distance measurement experiment, the input signal (Figure 9) for the creating ultrasonic wave was applied to the proposed module at every T ( = 0.1) second (the ultrasonic transmission envelope frequency was set to 10 Hz), and the signal was maintained for more than Ts ( = 2.5 msec) in order to create a stable ultrasonic oscillation. Therefore, the time that ultrasonic waves are not generated (Th) was 97.5 msec (= 100 msec 2.5 msec). To measure the distance while generating haptic information, we generated ultrasonic waves for Ts and generated a haptic sensation for Th. That is, each presented pattern consists of a

7 Measured distance(m) Measured distance error (m) Preprints ( NOT PEER-REVIEWED Posted: 29 June 2018 combination of an ultrasonic wave (20 khz) during Ts and a vibration signal (200 Hz) during Th. Because vibrations with a frequency above 1000 Hz are hardly felt by humans due to the vibration threshold [15], humans can hardly feel the haptic sensation during Ts. Therefore, as Ts increases, a person can sense that the prepared signal is discontinuous. In order to test whether a person could feel the haptic sensation, we prepared seven patterns by reducing Th by 5% as shown in Figure 9. Ten subjects participated in this experiment. Three subjects were females, and the remaining subjects were males. All of them were between 26 and 32 years old. The 10 subjects experienced randomly selected patterns for 2 seconds. Whenever we presented each pattern to a subject, we asked a subject whether the presented pattern was continuous or not. When Th became , all the subjects answered that the presented pattern is not continuous. This result is very reasonable considering the haptic temporal resolution of humans [16]. 20 khz 200 Hz T s =2.5 ms T h =97.5 ms T = 100ms T s = ms T s = ms T s = ms T s = 22 ms T s = ms Figure 9. Input signals for the oscillation to the proposed module The distance was also measured with the prepared seven patterns. Figure 10(a) shows the measured distance, and Figure 10(b) shows the measured distance errors. As we can see in Figure 10, regardless of the change in haptic generation time, we can see that the proposed module measures distance well when the distance to an object is less than 2 m. From these results, the proposed module is shown to be satisfactory for applications that generate a haptic sensation while measuring distance T h (100) T h (95) T h (90) T h (85) T h (80) T h (75) T h (70) T h (100) T h (95) T h (90) T h (85) T h (80) T h (75) T h (70) Distance between a transmitter and a target object (m) (a) Distance between a transmitter and a target object (m) (b)

8 Figure 10. Measured distance (a) and distance error (b) of the prepared seven patterns. We conducted another experiment to investigate the vibrotactile behavior of the proposed module as a function of the input frequency. In this experiment, we used a function generator (Protek 9305), an accelerometer (Bruel & Kjaer, Charge Accelerometer type 4393), and an oscilloscope (MSO/DPO 2000). A square wave voltage input was applied to the vibrotactile actuator. To measure the vibrational acceleration of the actuators, a mass of 100 grams was placed on the upper side of the proposed module, and the accelerometer was attached to the mass. The measured acceleration data were displayed on the oscilloscope. In this experiment, the input frequency was changed from 1 to 350 Hz. Figure 11 shows the haptic result of the proposed module as a function of the frequency. The vibration amplitude of the proposed module at the resonant frequency increased to about 2.86 g (g = 9.8 m/s 2 ). The result clearly shows that the vibration force of the proposed module is strong enough to stimulate the human skin. 4 3 Acceleration (g) Freqeuncy (Hz) Figure 11. Vibrational acceleration behavior of the proposed module according to the input frequency. 4. Conclusions This paper presented a new design for an all-in-one haptic/ultrasonic module which can generate not only an ultrasonic wave but also a haptic signal. In the proposed module, three energies (bending force, elastic returning force, and magnetic attractive force) were integrated into the design to maximize the haptic effect and the strength of the ultrasonic wave. After constructing the proposed module, we conducted experiments to investigate its performance. The results indicate that the proposed module can be used as an ultrasonic transmitter in the range of 0 ~ 1.5 m. In addition, the proposed module can generate an ultrasonic wave while delivering a haptic stimulus to the user. Author Contributions: Tae-Heon Yang and Dong-Soo Choi designed the research problems; Tae-Heon Yang, Dong-Soo Choi, Won-Cheol Bang conducted the experiments and analyzed the results; Sang-Youn Kim supervised the research. All authors discussed the results and wrote the paper. Funding: This work was supported by the Technology Innovation Program ( , Development of a filmtype transparent /stretchable 3D touch sensor /haptic actuator combined module and advanced UI/UX) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea). This work was supported by the Priority Research Centers Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (NRF-2018R1A6A1A ). Conflicts of Interest: The authors declare no conflict of interest.

9 References 1. Johnson, L. A.; Higgins, C. M. A navigation aid for the blind using tactile-visual sensory substitution. Proceedings of the 28th IEEE EMBS Annual International Conference, New York City, USA, August 30 September DOI: /IEMBS Available online: (accessed on 8 June 2018) 2. Zelek, J.; Audette, R.; Baltahzaar, J.; Dunk, C. A stereo-vision system for the visually impaired. Technical report, School of Engineering, University of Guelph Available online: (accessed on 8 June 2018) 3. Nie, M.; Ren, J.; Li, Z.; Niu, J.; Qiu, Y.; Zhu, Y.; Tong, S. SoundView: An auditory guidance system based on environment understanding for the visually impaired people. Proceedings of the 31st Annual International Conference of the IEEE EMBS, Minneapolis, USA, September DOI: /IEMBS Available online: (accessed on 8 June 2018) 4. Sainarayanan, G.; Nagarajan, R.; Yaacob, S. Fuzzy image processing scheme for autonomous navigation of human blind. Appl. Soft Comput. 2007, 7, , DOI: /j.asoc Available online: (accessed on 8 June 2018) 5. Benjamin, J. M.; Ali, N. A. An improved laser cane for the blind. Proceedings of the SPIE 0040 Quantitative Imagery in the Biomedical Sciences II, San Diego, USA, March DOI: / Available online: Cane-For-The-Blind/ / short?SSO=1 (accessed on 8 June 2018) 6. Yuan, D.; Manduchi, R. Dynamic environment exploration using a virtual white cane. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), San Diego, USA, June DOI: /CVPR Available online: (accessed on 8 June 2018) 7. Villanueva, J.; Farcy, R. Optical device indicating a safe free path to blind people. IEEE T. Instrum. Meas. 2012, 61, , DOI: /TIM Available online: (accessed on 8 June 2018) 8. Nada A. A., Fakhr M. A., Seddik, A. F. Assistive infrared sensor based smart stick for blind people. Proceedings of the Science and Information Conference, London, US, July DOI: /SAI Available online: (accessed on 8 June 2018) 9. Sekar, D.; Sivakumar, S.; Thiyagarajan, P.; Premkumar, R.; Kumar, M. V. Ultrasonic and voice based walking stick for blind people. International Journal Innovative Research in Electrical Electronics Instrumentation and Control Engineering. 2016, 4, , DOI: /IJIREEICE Available online: (accessed on 8 June 2018) 10. Gallo, S.; Chapuis, D.; Santos-Carreras, L.; Kim, Y.; Retornaz, P.; Bleuler, H.; Gassert, R. Augmented white cane with multimodal haptic feedback. Proceedings of the rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, Tokyo, Japan, September DOI: /BIOROB Available online: (accessed on 8 June 2018) 11. Dakopoulos, D.; Bourbakis, N. G. Wearable obstacle avoidance electronic travel aids for blind: A survey. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 2010, 40, 25 35, DOI: /TSMCC , Available online: (accessed on 8 June 2018) 12. Wang, Y.; Kuchenbecker, K. J. HALO: Haptic Alerts for Low-hanging Obstacles in white cane navigation. Proceedings of Haptics Symposium 2012, Vancouver, Canada, March DOI: /HAPTIC Available online: (accessed on 8 June 2018) 13. Nguyen, C. Haptic obstacle detector for the blind. Master, KTH Royal Institute of Technology, Stockholm, Sweden, August Available online: (accessed on 8 June 2018) 14. Yao, H.Y.; Hayward, V. Design and analysis of a recoil-type vibrotactile transducer. J. Acoust. Soc. Am. 2010, 128, , DOI: / Available online: (accessed on 8 June 2018)

10 15. Ryu, J.; Jung, J.; Park, G.; Choi, S. Psychophysical model for vibrotactile rendering in mobile. Presence-Teleop. Virt. 2010, 19, , DOI: /PRES_a_00011 Available online: (accessed on 8 June 2018) 16. Petrosino, L.; Fucci, D. Temporal resolution of the aging tactile sensory system. Percept. Motor. Skill. 1989, 68, , DOI: /pms Available online: (accessed on 8 June 2018)