A haptic sensor-actor-system based on ultrasound elastography and electrorheological fluids for virtual reality applications in medicine W. KHALED 1, H. ERMERT 1, O. BRUHNS 1, H. BOESE 2, M. BAUMANN 2, G. J. MONKMAN 3, EGERSDOERFER 3, A. MEIER 3, D. KLEIN 4, H. FREIMUTH 4 1 Ruhr-University Bochum, Bochum, Germany 2 Fraunhofer-Institut fuer Silicatforschung, Wuerzburg, Germany 3 Fachhochschule Regensburg, Regensburg, Germany 4 Institut fuer Mikrotechnik Mainz, Germany Abstract. Mechanical properties of biological tissue represent important diagnostic information and are of histological relevance (hard lesions, nodes in organs: tumors; calcifications in vessels: arteriosclerosis). The problem is, that such information is usually obtained by digital palpation only, which is limited with respect to sensitivity. It requires intuitive assessment and does not allow quantitative documentation. A suitable sensor is required for quantitative detection of mechanical tissue properties. On the other hand, there is also some need for a realistic mechanical display of those tissue properties. Suitable actuator arrays with high spatial resolution and real-time capabilities are required operating in a haptic sensor actuator system with different applications. The sensor system uses real time ultrasonic elastography whereas the tactile actuator is based on electrorheological fluids. Due to their small size the actuator array elements have to be manufactured by micro-mechanical production methods. In order to supply the actuator elements with individual high voltages a sophisticated switching and control concept have been designed. This haptic system has the potential of inducing real time substantial forces, using a compact lightweight mechanism which can be applied to numerous areas including intraoperative navigation, telemedicine, teaching, space and telecommunication. Introduction Four German institutes are operating in a collaborative project, proposing to develop a haptic sensor-actor-system, to support a variety of applications in medicine, education, telecommunication and space. The mechanical consistency of an object is to be locally specified using a sensor system and represented perceptibly in an other place on a tactile display (actuator system) for the user. The sensor system uses ultrasonic elastography, whereas the actuator array is based on electrorheological (ER) fluids. Real time ultrasound elastography represents a recent development to determine strain and elasticity distributions [1-2]. Commonly used imaging techniques rely on the interpretation of two dimensional visual data displayed on a video screen. In addition to visual data displayed, a physician will employ tactile exploration making the simultaneous portrayal of both video
and haptic information most desirable. Over the past 30 years, many tactile data conversion methods have been investigated, mainly with the ultimate aim of assisting the blind [3]. More recently, interest has been directed toward the display of pictures on haptic surfaces, tactile imaging [4]. Such an imaging system would allow surgeons to document properties of hard lumps contained in soft tissue, and to assist in operations performed remotely. Some fundamental principles of the real time ultrasound (US) elastography (also called strain imaging) are described in the next section. The general design of the system is presented in the second section. The tactile actuator design and its components as well as the medical safety aspects using electrorheological fluids are discussed in the last section. 1. Real Time Elastography Palpation is a standard screening procedure for detection of breast, thyroid, prostate, and liver abnormalities. The pathological state of soft tissues is often correlated with changes in stiffness, which yields qualitative estimation of tissue elasticity characteristics. However, palpation is not very accurate because of its poor sensitivity with respect to small and to deeply located lesions as well as to its limited accuracy in terms of the morphological localization of lesions. Conventional diagnostic imaging modalities (X-ray, US, MRI) are not able to visualize the mechanical tissue properties directly. Therefore, elastography as a new method based on US or on magnetic resonance imaging techniques (MRI) is of growing interest because of the capability of elastic tissue property visualization. In addition to tumors in soft tissue elastography is also able to detect calcifications in blood vessels, for example in the coronary arteries. The advantages of US elastography over MRI elastography are easier applicability and real time capability. Real time US elastography has been recently developed based on high efficiency signal processing approaches [5]. Ultrasonic imaging is performed during compression of the medium by an external force, as shown in Fig. 1 (a), to determine the strain distribution. transducer force lateral transducer force rigid Free hand scan direction rigid soft axial B-mode image strain image soft lateral (a) (b) Fig. 1 Experimental inhomogeneous phantom with one inclusion under slight mechanical deformation using the ultrasound transducer. (a) Rigid body with circular shape in a soft phantom not visible in the traditional ultrasound image (B-mode image) on the left. It can be clearly seen in the strain image on the right. (b) Sequential acquisition of parallel tomographic slices using Elastography, combined with image segmentation, enables the reconstruction of a three-dimensional image A strain image is formed by comparing echo signal sets obtained prior to and immediately following less than 5% compression, where tissue properties are approximately linear and elastic. Using the exact measurement of temporal displacements between the two signal sets is the key to estimate strain. A phase root seeking algorithm has been developed [5] for a fast and accurate displacement shift estimation, improving the accuracy, reducing the time needed and establishing the first freehand, real time, two dimensional US elastography system. axial
The reprojection of 2-D-freehand-slices, as shown in Fig. 1 (b), into a volume data set requires a position sensing device, which may be electromagnetic, acoustic or optical. After acquiring a series of 2-D-elastograms, the volume is created by placing each image at the proper location in the volume. The position data acquired with each 2-D-elastogram determines the particular location of the image. Using the three dimensional volume data in building an equivalent virtual object displayed on the actively controllable 3-D object surface is the aim of the tactile displays. Any 2-D projections can be selected from this 3-D data set for the presentation on the tactile display system. (a) (b) (c) Fig. 2 In vivo results of a human prostate: (a) Histology, tumors have been stained, malign and benign tissue areas have been marked by pathologist. The tumor is in the lower left side (b) It is not easily seen in the B-mode image. (c) The tumor is clearly seen as a dark left side on the strain image. During a clinical study, radio-frequency ultrasonic echo data using B-mode and strain image mode of more than 100 patients is undergoing clinical examinations. Fig. 2 shows an example, where prostate slices with histological diagnosis following radical prostatectomies act as the gold standard. Cancerous areas have been stained and marked on the prostate slices. It has been shown in other works [7] that our system for real time ultrasound elastography is able to detect the prostate carcinoma with a high grade of accuracy, nearly 85%. Thereby the system can improve the early detection of prostate cancer and allow a more reliable diagnosis. 2. System Description A haptic system which consists of a sensor head and a separate actuator array is currently under development. The sensor system is based on real time US elastography. The elastography system is able to detect even small and far surface lesions which are not detectable by manual palpation or conventional US systems. On the other hand, the proposed actuator consists of a tactile display which has a hybrid configuration consisting of smart fluids with electrically controlled rheological properties, contained within electrically controlled micro-machined cells. ER fluids are suspensions of electrically polarizing particles in a non-conducting carrier liquid. While applying a strong electric field the ER fluid becomes highly viscous. This effect is fast and reversible and can be exploited in the design and construction of tactile elements. The ultrasonic device generates and transfers images into the workstation. The investigated object is slightly compressed by an applicator which leads to locally varying displacements in the tissue depending on its consistency. From these displacements the spatial distribution of strain can be derived. The data transfer to the haptic tactile array may be achieved by direct electrical connection or by wireless communications as appropriate. The touchable surface of the actuator system consists of numerous small micro-machined cells which are filled with the ER fluid. The elements of this actuator array are controlled by voltages which can be individually, and remotely, adjusted. By this means variable stiffness of the tactile elements is generated and a locally varying consistency of the surface is perceived by the user who presses his fingers
onto the virtual object. Due to their small size and their repetitive arrangement, the actuator elements, including the electrodes for the ER fluid, have to be manufactured using micromechanical production methods. The required field strength for influencing the consistency of the ER fluid is about 3 kv/mm and needs a sophisticated system of power supplies which individually controls the high voltages applied to the actuator. A new technological concept is being developed for this purpose. Fig. 3 shows the developed system design. Actuator Option: Mechanical Applicator or freehand with 3D Manipulator Ultrasonic transducer Sensor Ultrasonic workstation Ultrasonic device Haptic control system including field generation Mechanically inhomogeneous object Haptic actuator array Wireless or on transmission line transfer of haptic information Fig. 3 Scheme of the haptic sensor-actor system Furthermore, in order to achieve the required changes in stiffness, the evaluation of ER fluids and actuator designs together with various modes of operation are being investigated. The actuator array is envisioned to consist of 1024 (32x32) elements with a spatial haptic resolution of about 2 mm. The combined sensor-actuator system shall serve as a new technology from which the potential can be deduced for various applications. 3. Tactile Display Tactile interfaces for virtual environments have been able to exploit previous work in the areas of sensor substitution devices for the disabled. In the case of tactile interfaces, researchers are investigating how to provide contact force, slip, texture, vibration, and thermal sensation. Some products intended to simulate contact forces that occur when a user touches a virtual object and other products are providing thermal sensations. However the ability to support other types of tactile sensation is more problematic. Consequently, any form of tactile display must deliver a spatial resolution of between 2 and 6 mm, depending on the vibration frequency. This can be achieved using a matrix of elements, which provide a stimulus in the form of vibration or a physical movement in the vertical plane proportional to the elasticity of the object being portrayed. Table 1 shows actuation methods which have hitherto been used for the realization of tactile display arrays. Table 1 Actuators previously used in tactile displays as shown in [3] Physical actuation Array size Force Hor. Resol. Ver. strokebandwidth Power/unit Pneumatic air jet 7 x 7-6.5mm - 2 Hz 1 W Pneumatic cylinder 4 x 4 2 N 4 mm 10 mm 10 Hz 0.3 W Electromag. vibrator 20 x 20 0.5 N 12 mm 10 mm 100 Hz 1 W Piezoelectric vibrator 24 x 6 0.4 N 1.5 mm 5 mm 300 Hz 10-5 W Thermal shape 8 x 8 2.5 N 2 mm 2.5 mm <0.3 Hz 0.5 W Pulsed Elec. simulation 20 x 20-5 mm - 400 Hz 0.01 W ER fluid. 24 x 24 1 N 2.4 mm 2 mm >20 Hz 0.02 W
3.1 ER Fluids as a basis for the tactile display To reduce the scale of the actuator an active medium such as ER fluids is being used. ER fluids are fluids that experience changes in rheological properties, such as viscosity, in presence of an electric field. They behave as normal Newtonian liquids until subjected to a high electric field, then they undergo a phase change, as shown in Fig. 4, from liquid to a quasi-solid state. Some of the advantages of ER fluids are their high yield stress, low current density and fast response. ER fluids are also nonabrasive, nontoxic and nonpolluting, meeting health and safety regulations. Fig. 4 ER fluids in the presence of electric field changes viscosity and internal friction. ER fluid stuck between the charged electrodes (left), and drops down after switching the electrical field off (right). ER fluids can control the flow by simply passing the fluid between two electrodes. This makes the design of hydraulic valves extremely simple and their construction very small. Another possibility is to switch micro-machined elements with the ER fluid integrated into the tactile display directly. This has been done previously for binary (on/off) displays [7], and more recently with a degree of vertical resolution as shown in [8]. Test results from two versions, one with cylindrical elements and the other with planar elements, are shown in Table 2. Table 2 New electro-rheological actuators used in tactile displays as shown in [7] Physical actuation Array size Force Hor. Resol. Ver. stroke Bandwidth Power/unit Cylindrical 4 x 4 1.4 N 5 mm 30 mm >20 Hz 20 mw Planar 4 x 4 0.6 N 3 mm 40 mm >20 Hz 27 mw It can be shown that ER fluids enjoy a comparatively low power consumption. A total power consumption of less than 30 Watts for a 1024-element-array is required, which is much lower than for hydraulic or electromagnetic systems. 3.2 Technological realization of actuator elements The goal of the present development steps is to generate an experimental model for the haptic sensor actuator system. The actuator unit will contain an array of 1024 ER elements, which are electrically controlled. In the first step, individual single components of the sensor-actuator-system were produced, developed and tested. On the sensor side, we developed different real time strain imaging techniques, such as intravascular and high frequency US strain imaging. These techniques are used to perform surfaces of the material, and also deeply located 2D-sections or curved internal surfaces as for example the inner surfaces of arteries. As test objects we use gelatin materials to construct tissue-like phantoms for elasticity imaging. On the actuator side, single tactel elements of both large scale and micro-mechanical dimensions have been constructed and comparatively investigated with various ER fluids in order to evaluate the most convenient operation
modes. Fig. 5 shows the scheme of the actuator with the ER fluid in which a tubular piston is allowed to move. While pressing down the piston the ER fluid between the electrodes is subjected to shear forces while an upward flow is induced due to the displacement of the fluid below the piston. As a result, the motion of the piston generates a resistance force which depends on the consistency of the ER fluid which is in turn determined by the electrical field strength between the electrodes. Fig. 5 Scheme of a haptic actuator side view The tactile element of Fig. 5 has been coupled with a force measurement device and different newly developed ER fluids have been included in the investigations. An appropriate experiment was developed to measure the stress in the actuator, when using the piston, depending on the electrical Field strength as shown in Fig. 6. 6000 Shear stress\pa 5000 4000 3000 2000 1000 ERF 168-1 ERF 168-2 ERF 168-3 ERF 168-4 ERF 168-5 ERF 168-6 0 0 1 2 3 4 5 Field strength [kv/mm] Fig. 6 Shear stress of different ER fluids measured for different electrical fields (DC voltage, temperature 25 C) The ERF 168-3 was selected as particularly suitable in the haptic actuator, due to its high yield-stress at lower field strengths. To control the actuators with high voltages about 2 kv with a very small current flow we represented new promising switching concept on semiconductor basis. Going from design to production several methods, processes and materials can be used to produce micro-structured products. For realization by hot embossing or injection molding it is necessary to produce mould inserts possessing the negative pattern of the final microstructure. Mould inserts can be produced for example by high precision milling for structure sizes down to 100 µm. Smaller structures can be realized using processes like Advanced Silicon Etching (ASE) [9]. By using a special plasma etching technique it is possible to fabricate silicon structures independent of the crystal orientation with depths of several hundred micrometers and high aspect ratio. To achieve almost vertical side walls in the range of 80-90 the addition of monomer gases is necessary for side wall passivation. After successful fabrication of a mould insert different technologies for molding can be used [10], such as vacuum casting or hot embossing.
3.3 Medical safety aspects The need for high voltage to control ER fluid-based devices creates safety concerns for human operators. Parameters as accuracy, repeatability and resolution must be known within defined tolerances for all equipment used for instrumentation and measurement purposes - also over predetermined lifetimes. In addition to general mechanical, electrical, thermal, acoustic and ergonomic design criteria, other factors such as a nontoxic material and chemical compatibility are considered. Safety aspects are been initially addressed during the early part of the development stage. Difficulties identified at this stage can then be corrected in a timely manner thus saving much greater costs. 4. Conclusion The fundamental model of a haptic sensor-actuator array system has been developed. This system allows on-line real time display of mechanical properties as well as off line display modes like slow motion and quick motion. This is the first integrated haptic sensor actuator system based on ultrasound real time elastography and electrorheological fluids with interesting potentials of virtual reality applications in medicine. It can be used for medical teaching purposes, for applications in telemedicine, intraoperative applications like minimal invasive surgery, or in the field of electronic commerce, entertainment and education. Acknowledgements This work is funded by the German federal ministry of education and research (BMBF). References [1] Pesavento A., Lorenz A., Siebers S., Ermert H.: New real-time strain imaging concepts using diagnostic ultrasound. Phys. Med. Biol. 45, pp. 1423-1435, 2000. [2] Pesavento A., Lorenz A., Ermert H.: System for real-time elastography. Electronics Letters, Vol. 35, no. 11, pp. 941-942, 1999. [3] Monkman G.J.: An Electro-rheological Tactile Display Presence. Journal of Teleoperators and Virtual Environments, MIT Press, Vol. 1, issue 2, pp. 219-228, July 1992. [4] Monkman G.J., Böse H., Ermert H., Khaled W., Klein D., Freimuth H., Baumann M., Egersdörfer S., Bruhns O.T., Meier A., Raja K.: Smart Fluid Based Haptic System for Telemedicine. 7th International conf. on the medical Aspects of telemedicine, Regensburg, Germany, September 2002 [5] Pesavento A., Perrey C., Krueger M., Ermert H.: A time-efficient and accurate strain estimation concept for ultrasonic elastography using iterative phase zero estimation. IEEE Trans. Ultrasonics, Ferroelectrics and Frequency Control, vol. 46 (1999), 1057-1067. [ 6] Scheipers U., Lorenz A., Pesavento A., Ermert H., Sommerfeld H.-J., Garcia-Schürmann M., Kühne K., Senge T., Philippou S.: Ultrasonic multifeature tissue characterization for the early detection of prostate cancer. IEEE Ultrasonics Symposium, (2001), pp. 1265-1268 [7] Monkman G.J.: Addition of solid structures to electro-rheological fluids. Journal of Rheology, Vol. 35, pp. 1385-7, Oct. 1991. [8] Böse. H, G. J. Monkman, H. Freimuth, D. Klein, H. Ermert, M. Baumann, S. Egersdörfer, W. Khaled, O. T. Bruhns - ER Fluid Based Haptic System for Virtual Reality - 8th Interntional conf. on new Actuators, pp351-354 - Bremen, June 2002. [9] Niggemann M., Ehrfeld W., Weber L., Günther R., Sollböhmer O., Miniaturized plastic micro plates for application in HTS, Microsystem Technologies 6, pp. 48-53, 1999. [10] Franssila, S., Kiihamäki J., Kartunnen J., Etching through silicon wafer in inductively coupled plasma, Microsystems Technologies 6, pp. 141-144, 2000.