A HAPTIC THERMAL INTERFACE: TOWARDS EFFECTIVE MULTIMODAL USER INTERFACE SYSTEMS

Transcription

1 A HAPTIC THERMAL INTERFACE: TOWARDS EFFECTIVE MULTIMODAL USER INTERFACE SYSTEMS Chang S. Nam Department of Industrial Engineering University of Arkansas, Fayetteville, AR 72701, U.S.A Jia Di, Liam W. Borsodi, and William Mackay Department of Computer Science & Computer Engineering University of Arkansas, Fayetteville, AR 72701, U.S.A {jdi, lborsod, ABSTRACT Despite powerful sensory inputs (e.g., tactile and force feedback), haptic interface systems are still in an early stage of development for accomplishment of a high degree of realism. One of the key elements missing is the ability to present thermal information, such as the thermal conductivity and temperature of an object being manipulated. This real time information is one of the obstacles to the wider application of haptic interface systems. However, thermal feedback should be incorporated into haptic interface systems to deliver more convincing and intuitive presence in virtual environments or teleoperation systems. This paper describes an advanced thermal interface system developed to provide the human operator with thermal information of an object being manipulated accurately and with no overt time delay. KEY WORDS Thermal user interface, human thermal perception, virtual environments, haptic force-feedback, thermal unit testing 1. Introduction As an increasingly powerful and dynamic interaction technology, haptic interfaces have found many application areas such as surgical simulation, medical training, scientific visualization, and assistive technology for the blind and visually impaired. The use of haptic devices as an interaction tool has provided users with a new and interesting sensorial experience, the sense of touch, through tactile and force feedback. That is, haptic interfaces allow users to synergistically integrate information from all their senses to make for richer sensorial experiences when interacting with computerbased applications. Many studies have also shown that tactile and force feedbacks can reduce the visual and auditory information overload that one can suffer from. Despite such powerful sensory inputs, haptic interface systems are still in a rather early stage of development to accomplish a high degree of realism [1]. One key element missing is the ability to present thermal information, such as the thermal conductivity and temperature of an object being manipulated. This lack of sensory feedback is one of the obstacles to the wider application of haptic interface systems. Thermal interfaces have a number of very desirable characteristics. For example, the temperature of an object plays an important role in tactile exploration and in the perception of the touched object [2]. The thermal conductivity information of an object can also be used to convey information about the material constitution of objects [3, 4]. Additionally, thermal feedback can act as a sensory substitute or adjunct for visual or tactile feedback, allowing the user to explore an object with its thermal information [5]. Taken all together, thermal feedback should be incorporated into haptic interface systems to deliver a more convincing and intuitive presence in virtual environments or teleoperation systems. Thermal devices to date, which have been developed to provide thermal feedback, can be categorized into three main types of systems. The first type of thermal devices used thermistors as a thermal sensor [e.g., 6, 7]. The use of the thermistor has two disadvantages for thermal feedback devices: slow response time (> 0.5 seconds) and larger than required size [8]. An array of thin film RTDs (resistance temperature detectors) have also been used to develop thermal devices [e.g., 5, 9]. However, more recent systems employed a thermocouple and a module with the Peltier effect to measure and present the temperature of an object to the human operator [e.g., 1, 10, 11]. Along side the development of thermal feedback systems, there has also been work on incorporating thermal feedback into haptic devices [3, 10, 11]. However, research in thermal interface is still in its infancy to be realized in real-world production settings systems. Motivated by this observation, we have developed an advanced thermal interface system that can provide thermal information of an object being manipulated to the human operator accurately and with no overt time delay. While the ultimate goal involves combining multiple sensory feedback modalities such as visual, auditory, haptic force, and thermal-feedback into an integrated interface system, in this paper we will focus on the development and evaluation of the thermal interface system as a first step. We will first give an overview of a human thermal perception system to understand how a

2 human being processes thermal information as well as to identify user requirements of thermal feedback systems. This is followed by a description of the major components and features of the thermal feedback device developed. Finally, we summarize the main contribution of the work. 2. Human Thermal Perception System In this section, properties of the human thermal perception system will be overviewed as an effort to identify user requirements for the development of thermal interface systems. 2.1 Thermoreceptors and Temperature Sense In human sensory physiology, the temperature sensation is closely associated with two types of thermoreceptors: the end-bulb of Krause that detects cold and Ruffini's end organ that detects heat. It is estimated that there are less warm spots than cold spots on the body [10]. Basic characteristics of this dual sensor system include: 1) Specific cold/warm spot: Thermal sensations are associated with stimulation of localized sensory spots in the skin. That is, there are specific cold and warm points in the skin, at which only sensations of cold or warmth can be elicited. 2) Different reaction time: Cold receptors, located at 0.15mm below the skin surface, react faster than the warmth ones, located at a depth of 0.3mm. That is, the reaction time for cold sensations for a temperature drop of greater than 0.1 C /sec is sec. On the other hand, the reaction time for hot sensations for a temperature rise of greater than 0.1 C /sec is sec. 3) Selective sensation: It is possible to prevent either the cold sensation alone or the warm sensation alone by selectively blocking free nerve endings [12]. According to the previous study [13], cold receptors are activated in a range from about 1 C to 20 C below normal skin temperature (approximately 34 C). Warm receptors are activated in the range of about 32 C to 45 C, peaking at about 37 C. Above about 45 C, cold receptors and pain receptors (nociceptors) fire at the same time in one field. Thermoreceptors are sensitive to changes in temperature, not to absolute temperature. Therefore, it is the rate of change in temperature that the human being senses. Many studies [e.g., 5, 10] have also confirmed that human subjects can sense rates of change in temperature as small as 0.01 C/sec (0.6 C/min). 2.2 Thermal Response Mechanism Temperatures that can be presented without any harm to the human being are between 15 C and 48 C [10]. However, it is also worth noting that the skin adapts to the temperature between 31 C and 36 C. Thus, the just noticeable difference (JND) measured in this range depends on the speed of the temperature change. Typically, the JND is found to be 2 C for a lower temperature variation of C/s, and 0.5 C JND for a temperature variation of +0.1 C/s [14]. Outside these intervals, it is estimated that the temperature sensation is either continuously cold or continuously warm. The ability of human subjects to discriminate changes in temperature is associated with many different factors. In their study on the psychophysics of temperature perception and thermal-interface design, for example, Jones & Berris [5] proposed the following four factors: the site of skin stimulated, the amplitude of the temperature change, the rate of the temperature change, and the baseline temperature of the skin (p. 138). 2.3 Requirements Specification for Thermal Interface System Based on the properties of the human thermal perception system and the results of the previous studies [e.g., 5, 8, 10], several user requirements for the development of thermal interface systems have been specified. Examples of the requirements include: To prevent injury, an upper temperature limit of 45 C and a lower limit of -5 C should be set. Temperature information should be presented no faster than 0.3 C/sec. Temporal transient resolution of cooling should be at least 20 C/sec with heating 10 C/sec. Absolute temperature as well as relative temperature changes should be presented for humans to feel. An effective thermal interface system should also implement various features conducive to human thermal perception, haptic interaction, and the use of thermal feedback devices. To identify such features and corresponding components, this study reviewed several previous thermal feedback devices [e.g., 1, 10, 11]. Examples include better performance, smaller size, simpler circuit, better accuracy, and higher reliability. 3. Thermal Interface System To develop more advanced thermal user interface system, this study used a variety of information obtained by analyzing the properties of the human thermal perception system, previous thermal feedback devices, and user/system requirement specifications. As mentioned in the previous section, several system architectures for developing thermal feedback devices have been proposed 14

3 (e.g., thermistor-, RTD-, or Peltier heat pump-based systems). Our system inherited the properties of Lawther s [8] general system architecture the use of Peltier heat pump and thermocouple. The system block diagram of the thermal feedback device developed in the study is shown in Figure 1. thermocouple as well as the temperature offset. The temperature regeneration subsystem is on the user end. The desired temperature is generated by a TEC. A TEC can be used to either heat up or cool down a subject. A heat sink is used to dissipate the extra heat generated by the TEC. The behavior of the TEC is controlled by an H- Bridge Driver, which is controlled by the microcontroller s (MCU) Pulse-Width-Modulation (PWM) module. Another thermocouple with converter is mounted on the TEC to detect the actual temperature generated. The detected temperature is then used as a feedback to the microcontroller for the control adjustment. The microcontroller gathers object temperature either from the temperature sensing subsystem or from the virtual environment on the PC and controls the TEC to regenerate the same temperature. 3.1 Temperature sensing subsystem Figure 1. System Overview of Thermal Feedback System However, there are two major differences between the two systems. First, to overcome disadvantages of using thermocouples, several approaches were made in Lawther s system. For example, a thermocouple cold junction compensator (LT1025) was used to compensate the nonlinear performance. A power transistor was also used as heater at the manipulator end to give one thermocouple junction a steady temperature. A feedback loop with an amplifier was needed to control the power transistor. Then, the output of LT1025 was directed into an amplifier to achieve a larger voltage swing that met the requirement of the A/D converter. However, all this circuitry caused a more complicated system with greater delay, less accuracy, higher power dissipation, larger size, and less reliability. Our system used a single IC, MAX6675, to accomplish the function of all circuitry mentioned in the example above. This integrated solution brings advantages such as better performance, smaller size, simpler circuit, better accuracy, and higher reliability. Another difference between the two systems is that unlike Lawther s system which used two different sensors to measure the temperature of thermoelectric cooler (TEC) and a user s finger (a thermocouple and a regular sensor, LM35DZ), our system used only one thermocouple to measure the temperature on the TEC. With a MAX6675, the same circuitry advantages could be achieved. The system can be divided into two subsystems (sensing and regeneration) and a microcontroller. The temperature sensing subsystem is on the manipulator end. To ensure fast response time and accuracy, a thermocouple was used as the temperature sensor. A thermocouple-to-digital converter, MAX6675, converts the analog signal from the thermocouple into digital values while compensating for the nonlinear behavior of a As suggested in Lawther s study [8], the touch temperature sensor must meet the following requirements: A fast response time in order to pick up sensations with no delay, Small and flat in order to be mounted on the finger of the manipulator and have full area contact with the object, and Linearity and absolute accuracy of the order of ±½ C and a range of 0 to 55 C. These requirements automatically reject sensors of several common types like Platinum films, semiconductor IC sensors, and thermistors. Thermocouples are based on the principle that when two dissimilar metals are joined a predictable voltage will be generated. This relates to the difference in temperature between the measuring junction and the reference junction (connected to the measuring device). Thermocouples have several disadvantages, including [8]: Α Thermocouple gives a voltage proportional to the difference in temperature between two junctions rather than an absolute temperature. Therefore, one of the junctions (the reference junction) has to be at a known temperature, to get the absolute temperature of the other junction. A Thermocouple s output is in the microvolt range: for a type-k thermocouple it approximates an average of 40.46mV/ C in the range 0 C to 50 C. This generates a requirement for a large gain capability in any circuitry it drives. This output value is not linear, as it also varies with temperature; at -100 C it is 35.54mV/ C and at +200 C 40.69mV/ C. Thus temperature compensation is needed [p. 40]. 15

4 However, these disadvantages can be solved by using a type-k Thermocouple-to-digital converter, MAX6675, from MAXIM. This IC includes an on-chip temperaturesensing diode to sense the ambient temperature, which is used as temperature on one junction of Thermocouple since that junction is directly connected to the converter. This resolves the first problem. MAX6675 also includes signal conditioning hardware to convert the Thermocouple s signal into a voltage that is compatible with the input channels of the on-chip 12-Bit A/D converter, solving the second problem. For the third disadvantage of Thermocouple, MAX6675 provides coldjunction compensation and a 0 C virtual reference to compensate the nonlinear performance. It also has a standard SPI interface to communicate with external devices such as a microcontroller. For our system, a type- K fast response self-adhesive thermocouple was used. This met most of the criteria, having a response time that is better 0.3s, a temperature range of -60 to +175 C, a thickness of 0.3mm, and is extremely robust. from the driver [8]. MI1023T has a size of inches, maximum cooler power of 9.2 watts, and 8V/1.8A maximum voltage/current requirements. 3.2 Temperature regeneration subsystem Since the manipulator and user are potentially in separate environments and it is impossible to tell which environment s ambient temperature will be higher, the regeneration subsystem must have the ability to generate a temperature that could be either higher or lower than the current ambient temperature. The rate of temperature change along with out-of-range protection must also be considered. Thermoelectric cooler (TEC), a semiconductor-based electronic component that functions as a small heat pump, can be used to regenerate thermal feedback. TEC, also known as the Peltier heat pump, is the only selfcontained, solid-state electronic method of thermal generation (cooling as well as heating). By applying a low voltage DC power source to a TEC, heat will be absorbed and transferred from one side and dissipated on the opposite side. Therefore, one face is cooled while the opposite face is simultaneously heated. Consequently, a thermoelectric cooler may be used for both heating and cooling by reversing the polarity (changing the direction of the applied current). The TEC s functionality comes from the Peltier effect. A typical single stage cooler consists of two ceramic plates with p- and n-type semiconductor material between the plates (Figure 2). These semiconductor materials are connected electrically in series and thermally in parallel. Heat is absorbed at the cold junction by electrons as they pass from the p-type (low energy level) to the n-type thermoelement (higher energy level). At the hot junction energy is expelled to a thermal sink as electrons move from the n-type to the p- type thermoelement. For our system, MI1023T was chosen as the most suitable cooler by considering the requirements such as size, cooler wattage, speed, and voltage/current needed Figure 2. Thermoelectric Cooler Due to the voltage/current requirements, a TEC cannot be directly driven by a microcontroller. Although linear amplifiers with a bridge configuration can be used to drive a TEC, a much more efficient way of driving the TEC is to use Pulse Width Modulation (PWM). This is a train of pulses of fixed frequency, with width proportional to the power required. An H-Bridge motor driver with ±2A/32V output capability was chosen to drive the TEC, as suggested in the previous work [8]. The magnitude of output current is controlled by the PWM signal from the microcontroller. The direction of current, which decides whether the TEC is heating up or cooling down, can be controlled by the microcontroller through one of its general purpose I/O ports. To apply appropriate control over the TEC a feedback loop is needed. The feedback loop consists of another thermal sensing system mounted to the user s contact point on the TEC. The sensor s input is converted into a digital signal and communicated back to the microcontroller. The measured temperature is then compared against both desired regeneration temperature and safety boundaries. The TEC control algorithm will dictate based on the comparison whether it should heat, cool, or deactivate. The feedback loop allows for both quick and simple control without the complexity of arcane algorithms along with a degree of safety for the user. 3.3 Microcontroller and communication with PC The system s regeneration speed is determined mainly by the thermocouple s conversion time. This relatively slow conversion time, when compared against most MCU s processing speed, allows for little restriction 16

5 on the choice of MCU. Since there is no special requirements other than standard MCU features a PIC18F452 from Microchip was chosen for our system. The PIC18F452 is an 8-Bit MCU capable of operating between MHz and contains an SPI, USART, 10-Bit A/D converter, and PWM. Once functionality is programmed it can convert and compare both thermocouples, and administrate the controlling PWM signal to regenerate the desired temperature. Additional functionality such as the USART will allow for communication with a PC, opening up other potential options such as tele-presence or virtual temperature generation. 4. Evaluation of thermal interface prototype To evaluate the effectiveness of the thermal interface prototype developed, a series of experiments was conducted. First, reaction times between input and feedback sensor settling were measured. Table 1 shows reaction times between the two sensors. It took less than 1.5 seconds in ranges of the temperature change tested. Table 1. Reaction time between input and feedback sensor settling Temperature Change ( C) Reaction Time (seconds) 23 to to to More detailed testing was also conducted to measure response and recovery times of both the temperature sensing and regeneration subsystems, including the times to 10%, 63%, 90%, and 100% change in temperature. As computer-controlled stimuli and timing were needed to properly test the thermal interface system, we first developed test software for the PC to manipulate the system prototype through a serial communication line. The test was conducted on the prototype by using the test software, written in Visual Basic and Visual C++. The software connected to the prototype through a serial port. The test software controlled the change of temperature through the serial communication port and logged the time of the temperature changes sent back through the serial communication port. The software kept track of time for a 10 percent, 63 percent, 90 percent, and 100 percent change in target temperature. The following tests were step changes of 10 degrees C in both the negative and the positive directions. The data shown is an average of 6 tests done by the test software. The test process started in a relaxed state, which is room temperature. Then a step change of 10 degrees C was applied to the prototype and the reaction times were recorded. After reaching the step change desired temperature the thermal interface prototype reset back to rest and recorded its recovery time. A test was done to show the time taken by the thermal interface prototype to drop 10 degrees C and to see the time taken by the prototype to rise back to room temperature. Table 2 shows test data collected during a negative 10 degrees C step change. Time measurements were taken at 10%, 63%, 90% and 100% of total temperature change. Table 2. Reaction and recovery times during -10 C step change Reaction time (sec) Recovery time (sec) 10% % % % Table 3 shows test data collected during a positive 10 degrees C step change. This test was done to measure the time taken by the thermal interface prototype to rise 10 degrees C and to see the time taken by the prototype to drop back to room temperature. Time measurements were taken at 10%, 63%, 90% and 100% of total temperature change. Table 3. Reaction and recovery times during +10 C step change Reaction time (sec) Recovery time (sec) 10% % % % Discussion and Conclusion The main goal of the study was to develop and evaluate an advanced thermal interface that can be incorporated into an integrated user interface system, which can provide multiple sensory feedback modalities. As shown in the previous section, preliminary tests on the system performance showed promising results. First, reaction times between the two sensors were very good, taking less than 1.5 seconds in ranges of the temperature change tested. Second, the longest time was the 10 degrees C drop in temperature because it took more time for the TEC to cool down or to heat up. With a 100 percent temperature change for all tests, however, the system was working fairly well. One of the biggest time consumptions in the thermal interface prototype was the settling time for the MAX 6675 chip, taking between.17 and.22 seconds to achieve an accurate thermocouple reading. To improve the time of the system it is recommended to use a stronger TEC or obtain better control of the power supply to the current TEC. There is still a temperature swing on the regeneration module that can be corrected using a PWM to control the power delivered to the TEC through the motor controller. The preliminary tests of the system also revealed several issues that should be addressed in the future 17

6 development. Examples include: Recovery times of the system could be high because of poor heat transfer between the sensor and the air. However, this issue could be avoided with continuous stimuli. Reaction times of the system could be bottlenecked by the initial sensor reading. The introduction of a PC controlled stimulus system will be able to resolve this issue. Future work will concern the development and user evaluation of a multimodal user interface system by integrating the thermal interface system developed in the study. References [1] P. Kammermeier, A. Kron, J. Hoogen, & G. Schmidt, Display of holistic haptic sensations by combined tactile and kinesthetic feedback, Presence, 13(1), 2004, th IEEE International Conference on Robotics and Automation, Minneapolis, Minnesota, 1996, [11] M.P. Ottensmeyer, & J.K. Salisbury, Hot and cold running VR: Adding thermal stimuli to haptic experience, Proceedings of the PHANToM Users Group, [12] L. Monteith, & R. Mount, Heat loss from animals and man (London Butterworths, 1974). [13] J.H. Martin, & T.M. Jessell, Modality coding in the somatic sensory system, In Principles of Neural Science (3rd ed.) (Kandel, E., Schwartz. J. H., & Jessell, T. M. eds, 1991) (pp ). New York: Elsevier. [14] C.E. Shenick, & R.W. Cholewiak, Cutaneous sensitivity, In Handbook of Perception and Human Performance (New York, 1986, ). [2] M. Benali-Khoudja, M. Hafez, J-M. Alexandre, & A. Kheddar, Thermal feedback interface requirements for virtual reality, Eurohaptics, Dublin, Ireland, 2003, [3] M. Bergamasco, A.A. Alessi, & M. Calcara, Thermal feedback in virtual environments, Presence, 6(6), 1997, [4] G.J. Monkman, & P.M. Taylor, Thermal tactile sensing, IEEE Transactions on Robotics and Automation, 9(3), 1993, [5] L.A. Jones, & M. Berris, The psychophysics of temperature perception and thermal-interface design, Proceedings of 10 th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Los Alamitos, CA, 2002, [6] R.A. Russell, Thermal sensor for object shape and material constitution, Robotica, 6, 1988, [7] D. Siegel, I. Garabieta, & J.M. Hollerbach, An integrated tactile and thermal sensor, Proceedings of IEEE International Conference on Robotics and Automation, San Francisco, 1986, [8] S. Lawther, Thermal and textural feedback for telepresence, MSc. Thesis Salford University, [9] M. Zerkus, B. Becker, J. Ward, & L. Halvorsen, Temperature sensing in virtual reality and telerobotics, Virtual Reality Systems, 1(2), 1993, [10] D.G. Caldwel1, S. Lawther, & A. Wardle, Tactile perception and its application to the design of multimodal cutaneous feedback systems, Proceedings of the 18