Synchronous stereo-video and biosignal recording a basic setup for Human-Computer-Interface applications

Transcription

1 Synchronous stereo-video and biosignal recording a basic setup for Human-Computer-Interface applications O.P. Burmeister, M.G. Litza, M. Nitschke and U.G. Hofmann Institute for Signal Processing, University of Luebeck, Germany Clinic for Neurology, University Hospital of Schleswig-Holstein, Campus Luebeck, Germany hofmann@isip.uni-luebeck.de Abstract The work presented reports on the development of a data acquisition system intended to be used in Human- Computer-interfacing applications to synchronously record upper limb movement and corresponding biosignals, be it EEG, ECoG, EMG or multi unit neuronal signals. Ultimately, this aims at the clinical recording of ECoG signals from awake patients undergoing tumor resection. For that purpose stereovideo camera frames are used to detect and triangulate hand and finger positions with high precision while at the same time a DSP-based 32-channel board acquires wideband biosignals. We validated the synchronous acquistion of both space coordinates and bioelectrical signals (EMG) by performing simple grasp experiments. Further research will be dedicated to the system s clinical application. I. INTRODUCTION One of the most depressing fears humans may face, is the outlook to become unable to voluntarily control body functions and movements. Unfortunately, this state is ultimately reached in patients suffering from traumatic tetraplegia, brain-stem strokes or long-term chronic diseases like amytrophic lateral sclerosis. Thankfully, the last couple of years have seen an immense increase in research efforts, knowledge and public awareness in order to alleviate at least some of the results of this sometimes locked-in called state [1]. Technological main stream of this developments are efforts to directly access the patients brain signals and utilize them for a Human- Computer-Interface (HCI) [2] to provide a new, non-classical path of communication. There are two general moieties of these HCI under development - the least traumatizing one is aiming at analysing macroscopical EEG signals from the brain (called Brain- Computer-Interface ) [3], whereas the other one is gaining microscopical signals from many single neurons from deeper cortical structures [4]. Clearly, while the former has the advantage to get absolutely non-invasive read-outs from the brain [5], [6], the latter features in principle higher information transfer rates, even suitable for real-time control tasks [7] [9]. Quite recently, another, less prominent subgroup of BCI s re-appeared on the scientific map: Macroscopic recordings of local field potentials from the surface of the brain, so called ECoG-recordings, proved to be helpful for predicting imaginary and real hand movements [10] [14], thus providing exiting new signals to be utilized for semi-invasive BCI s. All those recent developments however are based on a very small number of animal experiments or, due to good ethical reasons, restricted to some patients undergoing very specialized brain surgery. On the other side, recordings with strip electrodes from the cortex are quite common in neurosurgery (e.g. [15]) and used to the advantage of many, e.g. young tumor-patients [16]. It seems therefore rewarding for BCI research to widen the group of potential participants for subdural, BCI-oriented recordings by an experimental setup to be used beneficial in any neurosurgical operation room. In addition to the standard cortical mapping performed by strip electrodes, the awake patient could perform an easy movement task, while at the same time his local cortical field potentials are recorded. The paradigm to be investigated will be the correlation between hand-grasp movements and its generating brain activity. For that purpose it is neccessary to track the patients hand and finger positions in three dimensions with the least possible disturbance for the following surgery, while simultaneously recording its biosignal activity. The following describes the development of a recording system to perform synchronously and in one computer three-dimensional, marker-free hand tracking and general multichannel biosignal recordings to be used in a operation room setting. We will report on its realworld use in subsequent clinical publications. II. MATERIALS In order to achieve the minimal disturbance possible in the actual clinical setting, we decided to develop a stereo-camera based, marker-free system for handtracking. Both, stereo-video and biosignal recording system reside in the same industrial PC-chassis, equipped with an Intel Pentium IV, 3.06 GHz CPU and 1 GB SDRAM (DSM Computer, Munich, Germany). The utilized Intel Chip supports Hyperthreading, which permits both applications for handtracking and biosignal recording to run nearly in parallel. We used Windows 2000 as operating system and accessed a 250 GB S-ATA hard disk drive. A sketch of the complete system is depicted in Figure 1. A. Stereo-video system We are using two Sony XC-ST50CE monochrome CCD- Cameras with a Pentax H612A-TH lens, connected to a

2 Fig. 1. Sketch of the recording system PC2Vision framegrabber from Coreco Imaging (Saint-Laurent, Quebec). This framegrabber provides us with the ability to acquire the video data of two cameras synchronously without external genlocking. Due to movement artifacts when recording in interlaced mode we used their progressive scan mode instead. Each received image frame has a size of 760x280 pixels and is upscaled to 760x560 pixels, the frame rate has been manually reduced to 25. Faster frame grabbing doesn t seem necessary due to time restricitions given by the tracking algorithm. The cameras are fixed on a custom-built, portable optical bench. Both cameras are fixed on movable mounts such as to maintain permanent visibility of the tracked hand. For the laboratory setup presented here, we restricted our tracking to uniformly colored background, in order to reduce the complexity of the segmentation of the hand. B. Biosignal recording The data acquisition system is based on off-the-shelf DSP boards (M67, Innovative Integration, Thousand Oaks, CA). The board features a Texas Instruments C6701 highperformance DSP and is augmented with two analog-to-digital conversion modules (AD16 OMNIBUS modules). The DSPtarget is connected to the PC-host via the PCI-bus. This data acquisition system was developed for fast multichannel recording and allows to acquire data from 32 channels simultaneously with a sample rate up to 50kHz and 16 bit resolution [17]. Details can be found elsewhere [18]. Amplification of biosignals was achieved by a custom built, patient safe, wide band, 32 channel amplifier, but may be done by any type of analog multichannel amplifier, suitable for human use. III. METHODS A. 3D-Handtracking Although there are plenty of reports dealing with the tracking of the hand, most of them did not try to achieve a high precision in the hand-finger positions, but instead were interested in the gesture of the hand [19]. Gesture recognition may be done with or without markers and reduces the position detection in the image to a simple blob analysis. However, using markers seems prohibitive in the operation room. Therefore one of the biggest problems for markerfree handtracking is the segmentation of the hand out of any image [20]. In fact, the best suitable method for segmentation depends on some real world conditions impossible to foresee: 1) Is it possible to restrict the cameras background to a static or single-colored one? 2) Do monochrome or color cameras come to use? 3) Which lighting conditions will be experienced in the OR? Following earlier approaches [21] [23] we use a uniformly colored background with mono-chrome cameras, so the actual segmentation can be done via gray scale thresholding. In order to generalize this approach, static background subtraction [24] and a histogram based segmentation [25] can be used as well. Alternative methods by movement detection [24], skin detection [24], [26] and detection through stereo information [26] were dropped due to their computing time requirements or expensive equipment. In order to support the segmentation process Jennings [26] and Sato [27] have searched for circular structures, resembling finger projections, in images. Independent from the camera position fingers always look semicircular. Such structures were searched for by crosscorrelating images with finger templates. The video setup became independent from changes in light level by static, small source illumination and optical filtering of daylight. The handtracking application thus has two constraints: 1) the palm of the hand must be parallel to the image plane of the cameras, 2) the background must be static and single-colored. To compute the position of the hand in space, we first calculate the positions for each camera s frames. This enables us to triangulate the third dimension [28], based on a public domain stereo-calibration routine [29], included in the known OpenCV 1 computer vision library. Following segmentation, we perform a distance transformation [30] on the binary image of the hand. The distance 1

3 transformation gives the smallest distance of an unmarked (black) pixel to a marked (white) pixel in the binary image according to D(q) =min( p, q ),q O, where O is the set p O of all marked pixels. As can be seen in Figure 2 the maximum (a) Profile fingers through (b) Gray scale values of pixels on profile Fig. 4. Estimation of fingertips via gray scale profiles (a) Binary image of the hand Fig. 2. (b) Distance transform of (a) Computation of the handcenter by distance transformation of the distance transform locates the center of the palm. Due to the time-consuming computation of a full image distance transform we use a region of interest (ROI), in which the distance transform is calculated. That way we even reduce artifacts due to faultily marked, bright areas in the binary image by forcing the distance transform s center to lie at least on the palm. The ROI has to be chosen for the very first hand position only. The position of the distance transform s maximum defines the center and its scaled value, respectively, the size of a registered square estimating the orientation of the hand. This square is concentrically rotated such, that one side is oriented parallel to the hand s ulnar edge, providing a good estimate for the hands directionality (see Fig. 3). Its rotation value is deduced by minimizing the standard deviation of all distance transform values lying outside this side: α =argmin(std(s β )), β where S β is the set of all points lying on the right side of the palm s square. Fig. 3. Computation of fingertips and center of palm. Small blue dots: edges of the fingers, middle of the fingers. Big blue dots: estimation of fingertip The knowledge of the hand s orientation gives us an idea where fingers are supposed to be. That way we are able to confine our search for the fingertips via gray scale profiles. Considering a gray scale profile through a finger in the binary image we obtain a periodic sequence of background values, hand values, again background values. Anatomically the size of the hand value part has to be limited. So we get two conditions for a finger-like part in the gray scale profile. A profile through all four fingers and the gray scale values at the profile are depicted in Figure 4. The profiles are parallel to the closer side of the palm s square. Peaks in the profile s derivative give the location of the finger s edges, whereas a position in between peaks belongs to the finger itself. A line connecting central finger positions of two profiles directs to the fingertip position. (Fig. 3). An overview of the handtracking algorithm is depicted in Figure 5. B. Synchronizing stereo-video and biosignal recording The basic idea behind this project was to provide synchonous data from hand movement in space and corresponding biosignals, in the future from subdural electrodes. The problem when synchronizing stereo-video and biosignals, however, is the strongly differing sampling rate. Where normal video cameras are able to record images with a frame rate of 25 Hz, our biosignal recording system uses a minimum rate of 5000 Hz. There are two possibilities to cope with those different sampling rates. 1) Integration of biosignals over a given time period to approach the sampling rate of the cameras ( [31]) or 2) Matching of samples by timestamps. Even though the first method seems well suited for slow EEG-recordings, our intended recording modality, ECoG, provides wider frequency content (up to 200Hz), not to mention multiunit neuronal recordings (up to 15kHz). However, if we integrate the signal to approach the sampling rate of a camera, we will loose these important frequencies, since an integrator acts as lowpass filter. We therefore chose the latter way, assigning each hand-finger position as well as each biosignal sample point a high resolution system time stamp. Since both, signal acquisition and stereo-video boards reside in the same PC, we were able to avoid time-synchronizing problems to be considered for seperate computers. The delay produced by moving a camera image into memory, where it gets timestamped, was measured to be 30 msec. By subtracting that from the stamped time, we are able to perform stereo-video and signal acquisition synchronized to within milliseconds.

4 Fig. 6. signals Validation setup to synchronously record stereo-video and EMG Fig. 7. Exemplary space coordinates [mm] of handmovement trajectory Fig. 5. Application flow handtracking Clearly, that way we get many more biosignal sample points for only one stereo-video sample point, but it seems neccessary to correlate detailed biological signals with informations about the position of a hand. IV. RESULTS Tests of the system were performed in a laboratory setting not the operation room to validate the overall concept of this synchronous approach while recording biosignals. We chose to record surface EMG signals correlated to a handgrasp movement as proof of concept and due to their simple accessibility (see Figure 6). A test person sat in front of a black background and was asked to reach for a pole in grasping distance. This resulted in a movement distance of approximately 30cm. Stereo-cameras were positioned 60cm away from the target pole in a distance of 8cm from each other. Clearly, the precision to be expected depends on the optical and geometrical setup, but as well on the speed of the movement. All parameters can be easily adjusted to other geometries, but resulted in our case in a measurement precision of approximately 2mm 3 in a volume of about 50x40x20 cm 3 (see Figure 7). The exemplary trajectories in Figure 7 reveal the well known difficulty of precisely copying a given movement - a huge spread in precise coordinates can be found. We consequently detected the final grasp by the sudden disappearance of any finger related structures from the video frames (see Figure 8). (a) Open (b) Grasp Fig. 8. A cylindrical grasp to a target. The rectangle surrounding the hand shows the ROI for the distance transform. This time corresponded very well to an increase in EMG activity on the digital flexor muscle (see red vertical line in Figure 9). EMG signal from the volunteers main arm muscles were intercepted by selfadhesive electrodes (Bluesense, Ambu, Denmark) connected to the amplifier. The increase in grip force is visible by a clear rise in EMG signal variation as detected by the rms-value of the signal.

5 Fig. 9. EMG signals of digital flexor muscle showing an increase in grip force corresponding to a video detected handclosing (red line) V. CONCLUSION This project was intended to develop a single-computer system to synchronously record markerfree hand-arm-movements and its corresponding biosignals. We showed the validity of this concept by detecting hand-grasp movements and EMG signals at the same time with high temporal and local precision. We are now able to measure upper limb trajectories with a patient safe video system and will report on the clinical results elsewhere. ACKNOWLEDGMENT This work was partly funded by the European Union grant VSAMUEL IST and the German Research Ministry grant navegate 16SV1433. We want to express our gratitude to M. Zelazny for building the amplifier. REFERENCES [1] J. R. Patterson and M. Grabois, Locked-in syndrome: A review of 139 cases, Stroke, vol. 17, no. 4, pp , [2] M. A. L. Nicolelis, Actions from thoughts, Nature, vol. 409, pp , [3] J. R. Wolpaw, N. Birbaumer, D. J. McFarlanda, G. Pfurtscheller, and T. M. Vaughan, Brain-computer interfaces for communication and control, Clinical Neurophysiology, no. 113, pp , [4] J. K. Chapin, K. A. Moxon, R. S. Markowitz, and M. A. L. Nicolelis, Real-time control of a robot arm using simultaneously recored neurons in the motor cortex, Nature Neuroscience, vol. 2, pp , [5] N. Birbaumer and A. Kübler, The thought translation device (ttd) for completely paralyzed patients, in IEEE Trans. Rehab. Eng., vol. 8, 2000, pp [6] J. R. Wolpaw, D. MacFarland, G. Neat, and C. Forneris, An eeg-based brain-computer interface for cursor control, Electroencephalogr. Clin Neurophysiol., no. 78, pp , [7] D. M. Taylor, S. I. H. Tillery, and A. B. Schwartz, Information conveyed through brain-control: Cursor versus robot, in IEEE Trans. Neural Syst. and Rehab. Eng. 11, vol. 2, 2003, p. 195 ff. [8] M. Serruya, N. Hatsopoulos, L. Paninski, M. Fellows, and J. Donoghue, Instant neural control of a movement signal, Nature, vol. 416, pp , [9] M. Laubach, J. Wessberg, and M. A. L. Nicolelis, Cortical ensemble activity increasingly predicts behaviour outcomes during learning of a motor task, Nature, vol. 405, pp , [10] P. Kennedy, D. Andreasen, P. Ehirim, B. King, T. Kirby, H. Mao, and M. Moore, Using human extra-cortical local field potentials to control a switch, Journal of Neural Engineering, vol. 1, pp , [11] P. Kennedy, S. S. Mirra, and R. A. E. Bakay, The cone electrode - ultrastructural studies following long-term recordings in rat and monkey cortex, Neuroscience Letters, vol. 142, no. 1, pp , [12] E. C. Leuthardt, G. Schalk, J. R. Wolpaw, J. G. Ojemann, and D. W. Moran, A brain-computer interface using electrocorticographic signals in humans, Journal of Neural Engineering, vol. 1, pp , [13] T. Ball, M. P. Nawrot, T. Pistohl, A. Aertsen, A. Schulze-Bonhagen, and C. Mehring, Towards an implantable brain-machine interface based on epicortical field potentials, in Biomedizinische Technik, Biomedical Engineering, 38. Jahrestagung der Deutschen Gesellschaft für Biomedizinische Technik im VDE - BMT 2004, vol. 49, Ilmenau, 2004, pp [14] C. Mehring, J. Rickert, E. Vaadia, S. Cardosa de Oliveira, A. Aertsen, and S. Rotter, Inference of hand movements from local field potentials in monkey mortor cortex, Nature Neuroscience, vol. 6, no. 12, pp , [15] O. Suess, T. Kombos, O. Ciklatekerlio, R. Stendel, S. Suess, and M. Brock, Impact of brain shift on intraoperative neurophysiological monitoring with cortical strip electrodes, Acta Neurochir, vol. 144, pp , [16] P. M. Black, The present and future of cerebral tumor surgery in children, Child s Nerv System, vol. 16, pp , [17] U. G. Hofmann, A. Folkers, F. Mösch, D. Höhl, M. Kindlundh, and P. Norlin, A 64(128)-channel multisize neuronal recording system, Biomedizinische Technik, vol. 47, no. E1, pp , [18] U. G. Hofmann et al., A novel high channel-count system for acute multi-site neuronal recordings, submitted to IEEE TBME, [19] B. Stenger, A. Thayananthan, P. H. S. Torr, and R. Cipolla, Human hand pose estimation using hierarchical detection, in Intl. Workshop on Human-Computer Interaction, Prague, Czech Republic, [20] K. Dorfmüller-Ulhaas and D. Schmalstieg, Finger tracking for interaction in augmented environments, in Proceedings ISAR 01, New York, [21] K. Abe, H. Saito, and S. Ozawa, 3d drawing system via hand motion recognition from two cameras, in IEEE International Conference on Systems, Man and Cybernetics, [22] J. Segen and S. Kumar, Shadow gestures: 3d hand pose estimation using a single camera, in IEEE, Computer Vision and Pattern Recognition, vol. 1, [23] J. M. Rehg and T. Kanade, Digiteyes: Vision-based human hand tracking, Tech. Rep., [24] J. Martin, V. Devin, and J. L. Crowley, Active hand tracking, in IEEE Third International Conference on Automatic Face and Gesture Recognition, FG 98, Nara, Japan, [25] S. Ahmad, A usable real-time 3d hand tracker, in Proc. 28th Asilomar Conference on Signals, Systems and Computers. IEEE Computer Society Press, 1995, pp [26] C. Jennings, Robust finger tracking with multiple cameras, in Proc. of the International Workshop on Recognition, Analysis and Tracking of Faces and Gestures in Real-Time Systems, 1999, pp [27] Y. Sato, Y. Kobayashi, and H. Koike, Fast tracking of hands and fingertips in infrared images for augmented desk interface, in Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition. IEEE Computer Society, 2000, p [28] E. Trucco and A. Verri, Introductory Techniques for 3-D Computer Vision. New Jersey: Prentice Hall, [29] J.-Y. Bouguet, Camera calibration toolbox for matlab, doc/. [30] G. Borgefors, Distance transformations in digital images, Computer Vision, Graphics and Image Processing, vol. 34, pp , [31] J. D. Echard, M. Burrow, J. W. Miao, and D. Humphrey, Pattern recognition techniques for cortical control of a robotic arm.