Subpixel Resolution Binocular Visual Tracking Using Analog VLSI Vision Sensors

Transcription

1 1468 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: ANALOG AND DIGITAL SIGNAL PROCESSING, VOL. 47, NO. 12, DECEMBER 2000 Subpixel Resolution Binocular Visual Tracking Using Analog VLSI Vision Sensors Ziyi Lu and Bertram E. Shi, Fellow, IEEE Abstract This paper describes the application of an analog VLSI vision sensor to active binocular tracking. The sensor outputs are used to control the vergence angles of the two cameras and the tilt angle of the head so that the center pixels of the sensor arrays image the same point in the environment. One distinguishing feature of the sensor used here is the possibility to resolve target displacement with subpixel resolution via a phase-based algorithm, which integrates information over multiple pixels. Index Terms Analog integrated circuits, cellular neural networks, Gabor filters, image processing, image sensors, image orientation analysis, intelligent sensors, robot vision systems, stereo vision, visual tracking. I. INTRODUCTION WORK in analog VLSI vision sensors has been motivated by the increasing need for low-latency vision systems for applications such as visual serving and human computer interfaces. These sensors combine both sensing and image processing on the same silicon substrate. Both local and global image-processing operations are often implemented using analog VLSI circuits due to their high speed, low power, and low area [1] [3]. Visual tracking has been a popular application for computational sensors. Most of these systems have been based upon extracting the most salient point in the image using a winnertake-all network [4] [7]. Saliency has been measured by image intensity, temporal or spatial gradients, or combinations of these features. If the sensor is fixed, these sensors continuously report the location of the target in the image plane as it changes over time. These sensors can also be used in active vision systems, where their output is used to move the sensors so that the target is stabilized in the image. Note that since the winner-take-all network identifies the pixel corresponding to the most salient point, these systems can only resolve target motions with pixel-level accuracy. An alternative approach, based on zeroing the motion in a foveal region, is reported in [8] and [9]. This paper introduces the use of an analog VLSI computational sensor for visual tracking on an active binocular vision platform. The sensor outputs are used to control the vergence angles of the two cameras and the tilt angle of the head to keep Manuscript received September 2000; revised October This work was supported by the Hong Kong Research Grants Council under Grant HKUST675/95E and Grant HKUST 782/96E.This paper was recommended by Associate Editor A. Andreou. Z. Lu was with the Hong Kong University of Science and Technology, Kowloon, Hong Kong. He is now with Bell Laboratories, Lucent Technologies (China) Company, Ltd., Shanghai China. B. E. Shi is with the Department of Electrical and Electronic Engineering, Hong Kong University of Science and Technology, Kowloon, Hong Kong. Publisher Item Identifier S (00)11660-X. a target centered in the two sensor arrays. The trajectory of the target in environmental coordinates can be recovered using triangulation. One feature that distinguishes the sensor used in this paper is its ability to resolve target displacements with subpixel accuracy. This translates into a corresponding increase in the resolution with which the target trajectory in the environment can be recovered. Subpixel resolution may be especially advantageous for analog VLSI computational sensors, since the resolution and pixel count of these sensors is lower than pure charge-coupled device (CCD) or CMOS imagers due to the additional circuitry required at every pixel. For example, computational sensors reported for visual tracking, including the one here, have typically had on the order of 30 by 30 pixels or less. Subpixel resolution is achieved via a phase-based technique for disparity estimation using the outputs of filters similar to Gabor filters [10], [11]. Phase-based approaches have been shown to be robust in the presence of scale perturbations and contrast and luminosity imbalances [12]. In addition, they do not require a search for correspondences based on featureor region-based similarity measures. Although phase-based approaches usually use the output of Gabor filters, they are insensitive to the exact form of the filter transfer function [13]. In particular, the effect of replacing the Gaussian modulating function with the one-dimensional equivalent of the filter used in this work results in only slight degradation [14]. The remainder of this paper is organized as follows. Section II describes the general characteristics of the image sensor and the binocular vision platform used in this paper. Section III describes the phase-based technique for visual tracking used in this work. Section IV details our experimental results. Section V concludes with a summary of our findings and directions of future research. II. EXPERIMENTAL SETUP A. Vision Sensor The vision sensor contains a two-dimensional (2-D) array of phototransistors converting light into current and an array of continuous-time analog processing circuits, which spatially filter the input image. The analog processing array is based upon the cellular neural network architecture [15] [17]. We briefly review the important characteristics of the circuits and the image processing they perform below. For more detailed discussions of the theory and the circuit design, the interested reader is referred to [18] and [19]. The first image-processing operation performed by the circuits is subtraction of the dc offset from the input image [20] /00$ IEEE

2 LU AND SHI: SUBPIXEL RESOLUTION BINOCULAR VISUAL TRACKING 1469 This is done by mirroring the photocurrent twice. One copy of the current is used to compute the global average of the currents. This average is then subtracted from the other copy. The second operation is filtering the dc corrected image with a spatial filter whose impulse response can be approximated by where (1) otherwise. The parameters and control the shape of the filter. is the zeroth-order modified Bessel function of the second kind. The impulse response is similar to that of the Gabor filter, which replaces by a Gaussian function. The filter responds strongly to 2-D spatial frequencies near. For example, edges or bars oriented at angle result in large filter responses. The orientation angle is defined to be the angle between the normal to the edge and the -axis. The parameters and control the bandwidth of the filter. Fig. 1 shows cross sections of the real and imaginary parts of the impulse response as well as measured responses from one of the sensors. Fig. 2 shows the analog processing circuits connecting one pixel with its four nearest neighbors in the orientation selective image filtering array. The center frequencies of the tuning are set by Fig. 1. The top figures show cross sections of the real and imaginary parts of the impulse response h(m; n) for n =0. The envelope f(m; n) is shown in dotted lines. The filter was tuned to vertical orientations with parameters: =0:2,! =0:3,! =0:0, and =1. (c) and (d) show measured responses from the array tuned to the same filter parameters. The input was a small light spot focused onto the center of the array. Although qualitatively similar, the horizontal cross sections of (c) and (d) are not the same as and, primarily due to the additional high-pass filtering performed by the readout circuits but also due to component mismatch and the fact that the light spot is not confined to exactly one pixel. atan and atan The transconductance amplifiers with gains and are implemented with NMOS differential pairs with PMOS current mirror active loads. The bias currents through the differential pairs determine the gains and are supplied by NMOS transistors whose gates are connected external pins. Two separate pins control the tail currents for the transconductance amplifiers with gains and. In the following, we control the orientation tuning by changing the bias voltages at these pins. The sensor can be tuned to horizontal orientations by turning off and turning on so that and, and vice versa for vertical orientations. In both cases, we keep the values of and fixed and equal to each other. Three pins output voltages provide analog readout corresponding to the photosensor input and the real and imaginary parts of the filter output. Additional spatial high-pass filtering is performed by the readout circuitry, which enhances the orientation selectivity of the filter. Instead of sensing the voltage at each node in the array, we sense the sum of the currents entering each node from the two transconductors and. This sum is a discrete approximation to the directional derivative in the direction of the tuned orientation, which enhances the orientation tuning. Fig. 2. Analog processing circuits connecting pixel (m; n) with its neighbors. The filter input u(m; n) is provided as a current proportional to the input image intensity minus the average intensity. The voltages v (m; n) and v (m; n) at steady state represent the real and imaginary parts of the filter output. The filter parameters can be adjusted by controlling the conductances G ; G ; and G of the resistors and the gains G and G of the transconductance amplifiers via external bias voltages.

3 1470 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: ANALOG AND DIGITAL SIGNAL PROCESSING, VOL. 47, NO. 12, DECEMBER 2000 Scanner circuits enable an entire image to be read off serially by connecting each pixel in turn to the output. Alternatively, by stopping the scan, one pixel can be permanently connected to the output. Each pixel and its associated processing circuits occupy an area of m. The total project size including pads is mm. Transistor circuits operate in strong inversion. The measured static power consumption of the chip was 32 mw. B. Binocular Vision Head The binocular head used in these experiments is the Bisight system from HelpMate Robotics [21] mounted with two Fujinon H10X11E-MPX31 lenses. The pan and tilt of the head, the vergence angles of the two cameras, and the focus, aperture, and zoom of the lenses are controlled using a ten-axis PMAC controller VMEbus card from Delta Tau Corporation. In this work, only the tilt angle and the two vergence angles were actively controlled. The pan angle was held constant so that the head faces forward. The aperture was adjusted to its largest setting. The zoom was set to its smallest setting, corresponding to a focal length mm. Focus was adjusted so that images approximately 1 m away are correctly focused. A Motorola MC68040-based VME bus computer running VxWorks provides high-level control for the system. One sensor is mounted in the image plane of each lens, taking the place of the usual CCD camera. The generation of control signals and digitization of analog signals are provided by analog-to-digital (A/D) and digital-to-analog boards connected to the VME bus. Fixed pattern noise, measured with the sensor covered, is subtracted from the chip output after A/D conversion. Fig. 3. Coordinate frames associated with the binocular vision head. The view from the top. The Y axis points up out of the page. The view from the left side. The X axis points into the page. III. PHASE-BASED DISPARITY ESTIMATION AND TRACKING The filter outputs corresponding to the center pixel were used in a phase-based approach to estimate the disparity between the actual and desired target position. This approach exploits the fact translations in the input image result in phase shifts in the complex valued outputs of Gabor or similar filters. The phase shift is approximately linear in the amount of translation. Suppose an input image is convolved with the kernel in (1) Fig. 4. The disparity estimated from the sensor (solid line) as the camera vergence angle is swept over 10. less accurately the estimated translation reflects the local characteristics of the translation. The range of validity also depends upon the input image. Given a pair of images and, the two images are filtered, resulting in and. Define If the image is translated by, and the output becomes Then where Notice that the horizontal displacement can be approximately recovered if we set and [10] (2) The approximation holds for translations ( ), which are small compared with the width of and the periods of and. However, the wider the convolution kernel, the or, more accurately, by [11] (3)

4 LU AND SHI: SUBPIXEL RESOLUTION BINOCULAR VISUAL TRACKING 1471 Fig. 5. Recovered trajectories for the system tracking a vertical step edge translating in the X Z plane. The target is initially stationary, then translates at a constant linear speed of 25.4 cm/s and finally stops. Target translating from left to right for 1 s (25.4 cm) at a fixed Z displacement. Target translating from back to front for 1.5 s (38.1 cm) at fixed X displacements. The vertical lines show the exact lengths of the displacements for comparison. Fig. 6. Comparison of the measured variance of the estimated X coordinate (solid line) with that estimated assuming the variation is entirely due to sensor noise (dotted line) for a target at Z =120cm and various X coordinates. Comparison of the measured variance of the estimated Z coordinate with the estimate for a target located at X =0and varying distances Z. where the and denote the rate of change of the phase in the horizontal direction. In this work, we adopt the first measure since it is simpler and for vergence control we are primarily concerned with zeroing the disparity rather than estimating it. Similarly, can be estimated if we set and. Tracking of a target in the center of the sensor array is achieved by updating the tilt and vergence angles of the cameras to zero the disparity between the actual and desired target position. The sensor is configured so that the filter outputs of the center pixel in the array are permanently connected to the output. With the target in the center of the array, the sensor outputs are first recorded with the sensor tuned to horizontal and then to vertical orientations. During tracking, the sensor outputs are sampled every 2 ms. The phase differences between the current and reference outputs are used to estimate the dis-

5 1472 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: ANALOG AND DIGITAL SIGNAL PROCESSING, VOL. 47, NO. 12, DECEMBER 2000 Fig. 7. Recovered trajectory for target at constant depth Z and rotating in the X Y plane with speeds of 22 and 64 rpm. parity. The chip is tuned to horizontal and vertical orientations on alternate sampling intervals. Thus, the horizontal or vertical disparity information is updated every 4 ms. Assuming that the tilt and vergence axes are aligned with the camera axis, the angular movement required to zero the disparity is approximately proportional to the disparity. This movement is achieved using closed-loop position control with blended motion moves updated every sample period. By choosing equal reference outputs for the left and right sensors, the estimated horizontal disparity between the left and right images is zeroed during tracking, implying that both sensors track the same point. Its location can be estimated via triangulation. Fig. 3 shows the geometry of the binocular head looking from the top down and from the side. The and coordinates of the tracked point can be recovered from the vergence and tilt angles via where cm is the baseline length and is the projection of the distance from the origin to the tracked point onto the plane. IV. EXPERIMENTAL RESULTS In these experiments, the filter parameters for vertical orientations were,,, and. The parameters for horizontal orientations were the same, except the values of and are flipped. The tuned spatial (c) (d) Fig. 8. Recovered XY trajectories for three rotations of the target at speeds of 22, 37, (c) 51, and (d) 64 rpm. The dashed circle has radius 6.35 cm. The trajectories have been centered around the axis of rotation to facilitate comparison. frequency corresponds to a spatial period of 18.5 pixels. The period sets an upper bound on the range of disparities that can be measured, since phase wraparound makes large positive disparities appear the same as large negative disparities. In this case, the upper bound is a disparity of about 9 pixels around the center pixel, which covers nearly the entire array (25 pixels).

6 LU AND SHI: SUBPIXEL RESOLUTION BINOCULAR VISUAL TRACKING 1473 Fig. 9. Recovered trajectories for a target that rotates in the X Y plane and translates in depth (Z). The depth is initially constant, then increases at a constant rate of 20.3 cm/s for 1.25 s before coming to a stop. The speeds of rotation are constant at 22 and 64 rpm. The frequency response magnitude drops to half its peak value at radians away from the center frequency, corresponding to a space constant of 4.5 pixels. A smaller space constant reduces edge effects at the center pixel by decreasing magnitude of the convolution kernel for pixels falling outside of the array. On the other hand, it also decreases the range over which the phase difference is approximately linear in the disparity. A. Subpixel Disparity Estimation To examine the disparity estimated by the sensor, a vertically oriented step edge target was used. We initialized one camera so that the step edge transition was imaged by the center pixel and recorded the filter outputs of the center pixel. The vergence angle of the camera was swept over the range 5 to 5 relative to the starting position with increments of 0.2. The disparity for each location was estimated from the phase difference between the current and reference output using (3). The ideal and estimated disparities are plotted in Fig. 4. Assuming that the offset between the vergence axis and the lens center is small, the disparity should be proportional to, where is the vergence angle. From the figure, we observe that this is true for small. The degradation for larger is primarily because the approximation in (2) is less valid. Note however, that the sensor can distinguish the disparities much smaller than a single pixel. B. Tracking In these experiments, the same vertical step edge target was used. Only the left and right vergence angles were updated. The tilt angle of the binocular head was held constant at. The equations for recovering the and coordinates simplify to An table was positioned in the environment to provide controlled translations of the target both from left to right and directly toward the binocular vision head. Fig. 5 shows the recovered trajectories. The experimental results show much more variability in the estimated coordinate than in the estimated coordinate. From (4), small variations and in the vergence angles lead to variations in the estimated position of Since and are normally close to zero, the variation in the recovered position due to errors in the vergence angles is much smaller than that in the recovered position. Even with a stationary target, temporal variations in the vergence angles lead to variations in the estimated location. These variations are predominantly due to sensor noise introduced by sources such as the analog processing circuits, lighting variations detected by the photosensors, and electromagnetic interference picked up by signal lines and quantization noise during A/D conversion. The cumulative effect of these noise sources was measured by digitizing the chip output 1000 times with the camera centered on the step intensity edge. The measured vari- (4) (5)

7 1474 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: ANALOG AND DIGITAL SIGNAL PROCESSING, VOL. 47, NO. 12, DECEMBER 2000 ance of the estimated disparity was (0.10 pixel).the variations in the estimated disparity lead to fluctuations in the vergence angles with variance where mm is the focal length of the lens and mm/pixel is a conversion factor. Assuming that the variations in the left and right are uncorrelated, we can use (5) and (6) to estimate the variance of the fluctuations in the recovered and positions due to the sensor noise. Fig. 6 shows the estimated variance versus the measured variance of the estimates. The results match quite closely, indicating that sensor noise is indeed the major contribution to the fluctuations in the recovered position. The variation when tracking a moving target is larger due to additional tracking errors. C. Tracking In this experiment, tilt and vergence angles were allowed to vary to track a 2-in black square on a white background. The target rotated at varying speeds where the center of the square was displaced by 6.35 cm from the center of rotation. The distance to the target was fixed. The recovered trajectories for two different rotational speeds are shown in Fig. 7. As indicated by the sensitivity analysis, observed errors in the coordinate are much larger than those in the and coordinates. Fig. 8, which plots the recovered position over three rotations of the target at four different rotational speeds, illustrates the degradation in performance as the rotational speed is increased. D. Tracking In this final set of experiments, the same rotating target was used, but the target was also moved backward by 25.4 cm at a speed of 20.3 cm/s. The recovered trajectories are shown in Fig. 9. V. CONCLUSION This paper has described the use of analog VLSI vision sensors to provide visual feedback in an active binocular vision system. The sensor outputs are used with fairly simple postprocessing to control the tilt and vergence angles of the head so that a desired environmental point is tracked. The rate at which the visual feedback signals is extracted (250 Hz) is an order of magnitude higher than standard frame rates (25 30 Hz). Using triangulation, the three-dimensional coordinates of the tracked point can be estimated as the target moves. For stationary targets, the estimate is corrupted by noise in the sensor output. The resulting error in the estimated depth is on the order of 1 cm at a distance of 130 cm. For moving targets, the estimates degrade due to errors in tracking. The experiments reported here used ideal planar targets with high contrast and sharp edges. Applying the sensor to tracking in more natural environments was hampered by the primitive photosensor stage, which simply transduces light into current with linearly fixed gain. Reliable signals can be obtained only in environments where lighting and target contrast are controlled so that the currents from the photosensor stage are matched with (6) the input range of the orientation selective circuits. Integration of current-mode subthreshold versions of the sensor architecture [22] with edge-enhancing silicon retina architectures will hopefully enable more reliable operation in less controlled environments. Another issue we have not addressed is stereo mismatch. In this work, the camera positions were initialized to ensure that the initial target disparity is within the range that can be reliably estimated by the vision sensor. One possibility to address this problem may be to exploit the capability of electronically tuning the scale of the filters. Initially, the sensors could be tuned to large scales and low spatial frequencies, which would provide a coarse estimate of the disparity. This coarse estimate could then be refined by tuning the sensors to smaller scales and higher spatial frequencies. This coarse-to-fine strategy is common in computer vision and has been used by others in the context of phase-based disparity estimation and vergence control [11], [13], [23]. ACKNOWLEDGMENT The authors would like to thank S. F. Luk for his help in designing parts of the experimental platform used in these experiments. REFERENCES [1] T. Roska and L. O. Chua, The CNN universal machine: An analogic array computer, IEEE Trans. Circuits Syst. II, vol. 40, pp , Mar [2] T. Kanade and R. Bajcsy, Computational Sensors: A report from the DARPA workshop, in Proc. IUS, [3] C. Koch and H. Li, Eds., Vision Chips: Implementing Vision Algorithms with Analog VLSI Circuits. Los Alamitos, CA: IEEE Computer Society Press, [4] T. K. Horiuchi, T. G. Morris, C. Koch, and S. P. DeWeerth, Analog VLSI circuits for attention-based, visual tracking, in Advances in Neural Information Processing Systems 9, M. C. Mozer, M. I. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, pp [5] T. G. Morris and S. P. DeWeerth, Analog VLSI excitatory feedback circuits for attentional shifts and tracking, Analog Integr. Circuits Signal Process., vol. 13, pp , [6] V. Brajovic and T. Kanade, Computational sensor for visual tracking with attention, IEEE J. Solid-State Circuits, vol. 33, pp , Aug [7] G. Indiveri, Neuromorphic analog VLSI sensor for visual tracking: Circuits and application examples, IEEE Trans. Circuits Syst. II, vol. 46, pp , Nov [8] R. Etienne-Cummings, J. Van der Spiegel, P. Mueller, and M. Zhang, A foveated visual tracking chip, in ISSCC Dig. Tech. Papers, San Francisco, CA, Feb. 1997, pp [9], A foveated silicon retina for two-dimensional tracking, IEEE Trans. Circuits Syst. II, vol. 47, pp , Jun [10] T. D. Sanger, Stereo disparity computation using Gabor filter, Biol. Cybern., vol. 59, pp , [11] D. J. Fleet, A. D. Jepson, and M. R. M. Jenkin, Phase-based disparity measurement, CVGIP Image Understanding, vol. 53, pp , [12] A. Cozzi, B. Crespi, F. Valentinotti, and F. Worgotter, Performance of phase-based algorithms for disparity estimation, Machine Vision Applicat., vol. 9, pp , [13] C.-J. Westelius, H. Knutsson, J. Wiklund, and C.-F. Westin, Phasebased disparity estimation, in Vision as Process, J. L. Crowley and H. I. Christensen, Eds. Berlin, Germany: Springer-Verlag, 1995, ch. 11, pp [14] B. Crespi, A. G. Cozzi, L. Raffo, and S. Sabatini, Analog computation for phase-based disparity estimation: Continuous and discrete models, Machine Vision Applicat., vol. 11, pp , [15] L. O. Chua and L. Yang, Cellular neural networks: Theory, IEEE Trans. Circuits Syst., vol. 35, pp , Oct

8 LU AND SHI: SUBPIXEL RESOLUTION BINOCULAR VISUAL TRACKING 1475 [16], Cellular neural networks: Applications, IEEE Trans. Circuits Syst., vol. 35, pp , Oct [17] L. O. Chua and T. Roska, The CNN paradigm, IEEE Trans. Circuits Syst. I, vol. 40, pp , Mar [18] B. E. Shi, Gabor-type filtering in space and time with cellular neural networks, IEEE Trans. Circuits Syst. I, vol. 45, pp , Feb [19], Focal plane implementation of 2D steerable and scalable cortical filters, J. VLSI Signal Process., vol. 23, no. 2/3, pp , Nov./Dec [20] S. Espejo, R. Dominguez-Castro, R. Carmona, and A. Rodriguez-Vazquez, CMOS optical-sensor array with high output current levels and automatic signal-range centering, Electron. Lett., vol. 30, no. 22, pp , Oct. 27, [21] HelpMate Robotics Inc., Danbury, CT, [22] B. E. Shi, A low power orientation selective vision sensor, IEEE Trans. Circuits Syst. II, vol. 47, pp , May [23] W. M. Theimer and H. A. Mallot, Phase-based binocular vergence control and depth reconstruction using active vision, CVGIP Image Understanding, vol. 60, no. 3, pp , Nov Bertram E. Shi (S 93 M 95 SM 00 F 01) received the B.S. and M.S. degrees from Stanford University, Stanford, CA, in 1987 and 1988, respectively, and the Ph.D. degree from the University of California at Berkeley in 1994, all in electrical engineering. He then joined the faculty of the Department of Electrical and Electronic Engineering at the Hong Kong University of Science and Technology, where he is currently an Associate Professor. His research interests include analog VLSI and cellular neural networks, neuromorphic engineering, machine vision, image processing and speech recognition. Prof. Shi s IEEE activities have included work as Student Activities Chair of the IEEE Hong ong Section between 1996 and 2000, Associate Editor for the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I: FUNDAMENTAL THEORY AND APPLICATIONS between 1997 and 1999, and Secretary for the IEEE CAS Society Technical Committee on Cellular Neural Networks and Array Computing in Ziyi Lu was born in Wuxi, China, on November 18, He received the B.E., M.E., and Ph.D. degrees in electrical engineering from Southeast University, Nanjing, China, in 1993, 1995, and 1998, respectively. From 1998 to 1999, he was a Research Associate with the Department of Electrical and Electronic Engineering, Hong Kong University of Science and Technology, Hong Kong, where he engaged in computer vision research. Since 1999, he has been with the Bell Laboratories, Lucent Technologies, China, where he is currently working on signal- and speech-related topics. His current research interests include blind signal processing, speech recognition, and applications of artificial neural networks.