A Pixel Silicon Retina for Gradient Extraction With Steering Filter Capabilities and Temporal Output Coding

Transcription

1 160 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 37, NO. 2, FEBRUARY 2002 A Pixel Silicon Retina for Gradient Extraction With Steering Filter Capabilities and Temporal Output Coding Massimo Barbaro, Pierre-Yves Burgi, Alessandro Mortara, Pascal Nussbaum, and Friedrich Heitger Abstract A pixel analog very large scale integration retina is proposed to extract the magnitude and direction of spatial gradients contained in sensed images. The retina implements in a massively parallel fashion, at pixel level, an algorithm based on the concept of steerable filters to compute the gradients. An output rate of up to 1000 frames per second is achieved in a standard CMOS 0.5 m process. The retina provides address-event coded output on two asynchronous buses, one dedicated to the the gradient s direction and another to the gradient s magnitude. The gradient information is temporally ordered from largest to smallest gradient s magnitude. Rationales for such an order are borrowed from information theory. Precise timing of the address events is controlled by a decreasing threshold function, whose slope can be dynamically modified to regulate the data flow on the communication bus so as to reduce the number of collisions. Quantitative experimental results from a fully functional silicon demonstrator are presented. Index Terms Analog VLSI design, artificial retina, early vision, image processing, spatial gradient extraction, steerable filters, temporal coding. I. INTRODUCTION THE increasing demand for fast and efficient low-power and low-cost vision systems able to extract complex information from the visual scene is being led, nowadays, by the needs of the surveillance, automotive and mobile systems market, to cite but a few. Such a market asks for increasingly more specific vision tasks, which can encompass the operations of segmentation, recognition and classification applied to characters, faces, postures, etc. The traditional approach to deal with these kinds of task is to proceed accordingly to two main steps: 1) acquisition of the image on a low-cost CCD camera and 2) software processing of the raw image information on a digital platform (PC, Digital Signal Processors or Application-Specific Integrated Circuit). While the computational capabilities of digital platforms improve each year, they do not comply with low-cost and portability, as needed by the visual applications targeting the grand public. Indeed, the complexity of the vision tasks usually compels very high computational capabilities in the processing part, while real-time operations require the introduction Manuscript received January 23, 2001; revised September 21, This work was supported in part by Regione Sardegna and by CNR (Consiglio Nazionale delle Ricerche). M. Barbaro is with the Department of Electrical and Electronic Engineering, University of Cagliari-Piazza d Armi, Cagliari, Italy. P.-Y. Burgi, A. Mortara, P. Nussbaum, and F. Heitger are with the Swiss Center for Electronics and Microtechnology, 2007 Neuchatel, Switzerland ( pierre-yves.burgi@csem.ch). Publisher Item Identifier S (02) of high-performance DSP s, which usually are demanding in power consumption as well as costly (DSP s may however be a good trade-off between efficiency and flexibility). An attractive solution to the problem posed by these seemingly incompatible requirements (computational power, portability, low cost, and low power consumption) is to shift part of the computation into the sensor itself. A survey of the recent very large scale integration (VLSI) literature tends to indicate an emergence of such visual sensors (e.g., [1] [8]). This engineering effort is done in an attempt to substitute the CCD cameras with new CMOS imagers dotted with pre-processing capabilities at the level of each pixel. In this way, the camera is replaced with a more sophisticated device (i.e., a retina) able to preprocess the acquired images and to provide the next processing stages with band-limited information ready for being further processed using more limited resources. The advantage of this kind of architecture consists in performing a consequent number of low-level elaborations at pixel level in a parallel fashion, which optimizes computational resources. Power and time can generally be saved if the elaboration is accomplished with a simple analog circuitry surrounding the photodiodes, and if the processed information is dispatched off-chip only when it is needed, as proposed in this paper, rather than by systematically applying a raster access, as it is usually done with CCD and CMOS imagers. In the following, a pixel VLSI retina able to filter in a novel fashion the image is presented. The implemented algorithm is quite suitable for real-time elaborated visual tasks, since it provides within a very short time (1 ms) the magnitude and direction of the image s spatial gradients. The innovation of the proposed approach resides in the way this information is extracted and in its particular encoding. For one, filtering for extracting the gradients is based on the mathematical property of steerability specific to some functions, an approach that has not yet been exploited in analog VLSI, to the exception of filters based on resistive networks using discrete coefficients [9] (and excluding digital VLSI implementations, such as proposed in [10]). Second, the extracted gradients are encoded in a peculiar fashion by ordering in time their off-chip dispatching. Such an ordering is arbitrated according to the magnitude of the gradient vectors, starting from the largest magnitudes, followed by smaller and lesser relevant vectors. The dispatched information, ordered in time, also provides the gradient direction, which is a powerful clue for performing recognition operations. The computation is carried out concurrently by all the pixels and is preceded by some low-level preprocessing such as global normal /02$ IEEE

2 BARBARO et al.: A PIXEL SILICON RETINA FOR GRADIENT EXTRACTION 161 ization and low-pass filtering to ensure a proper extraction of the gradients. The proposed retina is planned to be the front-end part of a two-chip system able to implement several kinds of recognition operation performed in real time. The remainder of this paper is organized in five sections. In the Section II, the filtering operation based on steerable functions and the overall operational principle of the retina are briefly described. A system overview is then given in Section III, followed by a detailed description of the core circuit (the pixel) in Section IV. Measures obtained from the first physical implementation of the retina are presented and discussed in Section V. Finally, conclusions are drawn in Section VI. II. OPERATIONAL PRINCIPLE The purpose of the implemented pre-processing performed at the pixel level is to yield two images, one for each component of the local spatial gradient (direction and magnitude). The appearance of information on the output channels will be related to its importance: largest gradients first. Temporal coding of the processed information has, thus, been chosen as the means to dispatch the two images off-chip, since in this way it is possible to send in ordered fashion the data on the communication channel. That stronger gradients are considered more important than weaker ones can be justified on the basis of signal-to-noise ratios: strong gradients are less affected by internal noise (transistor mismatches, thermal noise, etc.). Moreover, from a human perception point of view, strong contrasts (gradients normalized by the local luminance) usually correspond to occluding contours [11], and by consequent are more salient than weak contrasts. Saliency is used in this paper to denote features that are key for interpreting visual scenes, which is clearly the case of occluding contours. However, the argument is given for contrasts, whereas the proposed retina extracts gradients. Generally, strong contrasts correspond to strong gradients, but it is not always necessarily so. Consequently, the notion of saliency defined in this paper is close to that applied to human perception, but not equivalent. The chosen strategy implemented on the chip is, thus, to provide the next processing stages with the most relevant data first, which for a natural image can be sparse [12], and to transmit gradually more and more information as weaker gradients are processed. It has previously been shown elsewhere [13] that such an asynchronous scheme tends to improve image processing capabilities, and leaves the possibility of stopping the acquisition as soon as the recognition process has been completed. The main idea for computing the spatial gradients in the proposed analog VLSI implementation is based on the definition of the first-order derivative of a 2-D function performed in the vector direction, which is expressed as where is the vector s angle. If we were to sweep this angle in time using steering functions, computation of the gradient (1) performed on a discrete square lattice at the pixel level would be given by where represents local luminance at pixel (as captured by the photodiodes), and is the frequency of the steering functions. In this way the local derivative in the direction of vector (i.e., the angle ) is continuously computed as a linear combination of two basis functions (the derivatives in the and directions). This is the principle of steerable filters [14]. A function is called steerable under some transformation (translation, rotation, scaling, etc.) if all transformed versions of this function can be expressed using linear combinations of a fixed, finite set of basis functions. Application of this principle through (2) makes the extraction possible of the first derivatives in any directions with a reduced neighborhood connection using interpolating functions [sine and cosine functions in (2)]. Inspection of (2) shows that in the proposed implementation, only the first nearest neighbor pixels in the and directions are involved in the computation of the gradient (because the derivatives in the and directions are numerically approximated using central differences). In spite of this limited neighborhood, it is possible to compute derivatives in directions that are smaller than the pixel grid (sub-pixel precision), contrary to the kernel approach where the number of direction is finite. This property is akin to the interpolating functions and, which are steering in time the direction of the derivative. By definition, the spatial gradient is the vector for which (2) reaches its maximum. Consequently, if the time scale of fluctuation of the image (i.e., the luminances) is substantially slower than, then (2) represents a sine wave in time, in the following called local steering function, which amplitude and phase are respectively related to the magnitude and direction of the gradient. With the proposed approach, coding of information is quite straightforward. Indeed, as the first revolution of the interpolating functions is completed, each local steering function went through a maximum, which can be determined using a simple peak-detection scheme and stored locally on small capacitances. Then, at the next period of the interpolating functions, a monotonic decreasing threshold function is triggered (by default, the slope of the decreasing function is constant, see below). As soon as the stored maximum values equate the current threshold value, address-event pulses, encoding the location of the corresponding pixels, are generated and transmitted on an asynchronous bus [15]. According to this scheme, the time of appearance of an event on the bus is directly related to the gradient s magnitude, and the dispatching of gradient information off-chip is ordered from high to low values. Similarly, the gradient s direction contained in the phase of the local steering functions, is also pulse-encoded in time by detecting the instant the local steering function is going through zero (zero-crossing detection). Because in a whole steering period two zero crossings occur (one on the positive flank, and a second on the negative flank), the convention for detecting the adequate zero crossing must be decided beforehand to lift up the 180 ambiguity in the phase. Finally, to avoid that the zero crossings of all pixels are (2)

3 162 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 37, NO. 2, FEBRUARY 2002 Fig. 1. Schematic representation of the signals activity during a frame acquisition. After the first period of the interpolating functions (a) and (b) is completed, the dynamic threshold (c) starts decreasing. Once the stored maximum (d) of the local steering funcion (e) equates threshold (c), a gradient s magnitude pulse (f) is generated. A zero-crossing pulse (g) is then emitted during the first descending flank of (e) following the magnitude pulse (f). detected within the first semi-period of the interpolating functions, which might possibly saturate the bus, zero-crossing detection is authorized only subsequently to the sending of the magnitude information. This strategy means that beside the fact that within a particular period the gradient s direction is naturally ordered in time, it is also dispatched accordingly to the gradient s magnitude. Fig. 1 schematically summarizes the different steps of the overall acquisition process described so far. It shows the interpolating functions labeled (a) and (b), the monotonic decreasing threshold (c), the result of the computation of a local derivative as a function of time (e) with the memorized peak value (d) and the generation of two impulses, the first encoding the amplitude (f) and the second the orientation (g). Since all communications are completely asynchronous (each pixel can access the bus independently from the others) collisions can arise [15]. Special care must, thus, be taken to reduce the probability of a collision on the bus, which would inevitably cause a loss of information. This is the reason for the choice of generating the minimum number of pulses: each pixel will be allowed to access only once the bus during a frame acquisition. Although this scheme slightly complicates the logical structure of the pixel, it is essential for maximizing the probability of collecting all available information. Another powerful means to avoid collisions is to dynamically control the monotonic decreasing threshold function: when information is sparse, typically in the first part of the acquisition (statistically, there are few strong gradients in an image [12]) the threshold can be decremented faster than later when weaker gradients, which are mainly due to the high frequency noise generated by the derivative operation, abound. Unless one knows in advance the statistical distribution of the magnitude of the gradients, a dynamic feedback based on the number of actual collisions would have to be used to regulate the dynamics of the threshold function. More restrictively, the final value of the threshold function could be Fig. 2. System overview of the chip. set to a fixed value to discard events corresponding to noise. To be robust, this last possibility calls for an accurate knowledge of the statistics of the processed images, which in practice is rarely available. III. SYSTEM OVERVIEW The architecture of the chip is divided into five main blocks (depicted in Fig. 2): a serial peripheral interface block (SPI), a digital control block, an analog bias block, a communication circuitry (one for each bus) and, finally, the array of pixels (photodiodes with their associated circuitry for performing the analog computation). The function of these different blocks is now described in more details. A. Array of Pixels The core circuit is, obviously, the array of pixels. This array is organized according to a Cartesian arrangement of pixels. Each pixel contains a photodiode for the light-to-current transduction along with all the analog circuitry necessary to implement the algorithm described in Section II. Two low-level computations are performed on the image before the extraction of the spatial gradients: global normalization and low-pass spatial filtering. Normalization is performed by regulating the total current, and is needed to ensure proper operation under different conditions of illumination, while low-pass filtering is necessary to partially reduce the high-frequency noise introduced by the derivative operation and for achieving subpixel accuracy of the derivatives. To properly implement the gradient-extraction scheme and minimize the number of events generated onto the two buses (one bus is dedicated to the magnitude information and the other to the direction), operations at the level of a pixel must be careful programmed. Fig. 3 shows the state machine (physically implemented in the pulse generator block of each pixel), which represents the overall operational march of the circuit. According

4 BARBARO et al.: A PIXEL SILICON RETINA FOR GRADIENT EXTRACTION 163 all the required biases). A second role of the SPI is to give access to registers of the digital control block, which can be programmed to change the operating mode of the retina. Fig. 3. State machine for the control of the pixels. to this state machine, the acquisition of a frame is initiated by the reception of a sync signal, provided by the user. Prior to an acquisition, the previously stored maximum values are cleared. Then, the local steering function starts to be computed. During the first period, the maximum values are detected and stored on local (parasite) capacitances. After this initial phase, the decreasing threshold is launched and the pixels wait for their corresponding locally stored maximum value to match the current level of the dynamic threshold. When matching eventually occurs, a communication pulse is generated on the adequate bus (the magnitude bus). Next, the pixel switches to the second state where it waits for a zero crossing, one occurring on the descending flank of the local steering function. At this instant, a second pulse is sent onto the direction bus and the pixel attains the third state. In this last state, the pixel is in stand-by and is prevented from generating any pulse, until a new sync signal is received and the whole process starts again. According to this scheme, a pixel emits at most one pulse per bus at each frame acquisition. Reception of a sync signal in any of these three states causes a reset of the state machine and the pixels start again from the initial state. It must be noted that to reduce the number of collisions on the bus transmitting the magnitude information a weak arbitrage has been implemented at pixel level. This mechanism is called collision avoidance and works as following: each time a pixel emits an impulse on the bus, it momentarily raises the global threshold value in order to prevent other pixels, with maximum values very similar to its own, from firing. B. SPI The SPI (serial peripheral interface) is used to program the different functions implemented on the chip. Each of these functions, performed in parallel by all the pixels, are based on analog computations. Consequently, several biases are needed to tune the processes. For different visual scenes or different high-level algorithms, different tunings of these analog computations might be required. Individual access to some of these parameters is, therefore, necessary. By those tuning, it is possible to change the normalization current, the width of the spatial low-pass filter, or act on some analog biases of the array of pixels to increase the gain of the photo-receptors or adjust the frame rate. To ensure flexibility and programmability of the chip, a total of 14 different analog biases are accessible through the SPI, and are visible from the user as digital registers (these registers are connected to digital-analog converters, contained in the analog bias block, with the output currents generating C. Digital Control Block The digital control block is used to generate the threshold function (a monotonic decreasing ramp) as well as the interpolating functions (sine and cosine). A digital code for each of these functions is computed at fixed time intervals and then converted into analog reference currents by the analog bias block. The slope of the monotonic decreasing ramp, as well as its starting and final values can all be set by addressing three specific registers. For increasing the flexibility of the operation of the retina, the user can also provide the interpolating functions (if the user does not want sine waves) and the threshold ramp (typically to optimize the flow of information on the buses). This option is possible through a multiplexed 8-bit bus, on which digital values of the sampled functions can be specified. A status register is set accordingly to indicate this particular operating mode. D. Analog Bias Block The analog bias block is the final interface between the digital world and the tuning of the analog computation. Its role is to generate fixed biases by converting the digital values of all 14 registers into currents or voltages, which are used as references by the pixels. The digital-analog conversion also concerns the interpolating functions and threshold ramp generated in the digital control block. The biases are provided globally to the whole retina, and there is no way to suppress, for instance, the fixed pattern noise specific to individual pixels. 1 E. Communication Circuitry The results of the computation are communicated to the user as pulses distributed over two 14-bit buses. One bus is dedicated to the communication of the gradient s magnitude and the other to the gradient s direction. For each type of information, the address of the pixel consists of 7 bits for each of the - and -axis. When a current pulse is generated in any pixel, the communication circuitry encodes the address in a digital word of the corresponding active pixel. A strobe signal is also provided to indicate whether the code is valid or not (collision-free code). If two or more pixels try to communicate at the same time, the strobe signal is kept low and the address is stamped as nonvalid. IV. CIRCUIT DESIGN The overall pixel structure is shown in Fig. 4. The starting point is the block containing the photodiode measuring the local luminance. The photodiode is not operating in isolation but is connected to its closest neighbors to perform low-pass filtering and compute the local spatial gradient [see (2)]. The multiplication between the basis functions and the interpolating functions (using a one-quadrant multiplier associated with two switches, see below) is carried out in current mode. As shown in the figure, the results of the multiplications are fed, with the proper sign, 1 In fact, the fixed pattern noise cannot even be probed in this retina as only the results of the processing, that is, spatial gradients, are available.

5 164 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 37, NO. 2, FEBRUARY 2002 Fig. 4. Block diagram of a pixel. to the closest neighbor pixels in each of the four cardinal directions. In a symmetrical way, the neighbor pixels provide their contributing currents, which are simply added at a summation node (Kirchhoff current law). The result of this summation represents the local steering function of the pixel (labeled with letter e, as in Fig. 1). This signal is then fed into a detection block that performs on it the following operations: 1) detection and memorization of its crest value; 2) comparison of this memorized value with a dynamic threshold; and 3) detection of its zero crossings (on the descending flank of the steering signal). The two residual blocks in the pixel structure are needed for encoding the gradient s information onto the bus and to ensure the correct sequencing of the state machine (sketched in Fig. 3). The layout of the pixel is shown in Fig. 5. The two main and largest blocks visible in this figure are the multiplier and the detection block, which together occupy 60% of the whole pixel area. Most of the remaining area is filled by the pulse generators and the photodiode, which take 20% and 15% of the total pixel area, respectively. The connections and the circuitry for signal conditioning occupy the residual space. The circuitry of these various blocks is described in more details in the next five sections. Fig. 5. Layout of a pixel. A. Photodiode and Signal Conditioning The circuit for the light-to-current conversion and signal conditioning (low-pass filtering and global normalization) is shown in Fig. 6. The transducer is made up of a photodiode realized with the junction between the n-well and the substrate. To increase sensitivity to light, the dimensions of the photodiode were chosen so as to fill with best efficacy the space not occupied by the circuitry dedicated to processing, while keeping a reasonable overall dimension for the whole pixel circuit. The resulting fill-factor is around 15%. With the kind of technology adopted (Alcatel MIETEC 0.5 m) and normal illumination conditions (see Section V), photodiode current ranges between few pa for dark current and around 10 na for full light exposure. Since the retina operates in current mode, the current normalization simply boils down to a weighted current mirror: weights self-adjust in order to keep the total output current fixed to a specific value (specified by one SPI register). Another advantage of the current mode is that the photo-currents are directly available, without prior integration (except for very small currents, in which case the parasite capacitances would impose a nonnegligible time constant). Fig. 6. Light-to-current conversion and signal conditioning (see text). Low-pass filtering is implemented as a diffusion network made of pseudo-conductances laterally connecting the south and east closest neighbor pixels. The network of pseudo-conductances is achieved by means of MOS transistors operating in weak inversion, as described elsewhere [16], [17]. The lateral and vertical pseudo-conductances implementing the diffusion network are both controllable by setting and (programmable through two SPI registers). All transistors sizes are m, m. B. Multiplier The multiplier, shown in Fig. 7, is key for generating the local steering functions, which involved the sum of four products, see (2). At any given instant of the computation of the local steering functions, both the positive and negative versions of and must be computed to be subsequently fed into one of the neighboring pixels. One-quadrant multiplier has been chosen as the basic unit in order to simplify the pixel circuitry. This is possible because the sine and

6 BARBARO et al.: A PIXEL SILICON RETINA FOR GRADIENT EXTRACTION 165 TABLE I TRANSISTOR DIMENSIONS FOR THE MULTIPLIER BLOCK (IN m) Fig. 7. Circuit diagram of the four-quadrant multiplier for the cosine product. The right part is replicated to obtain the sine product, while the left part is common to the two half circuits. cosine interpolating functions change sign at well-defined instants. Consequently, it is straightforward to generate a positive and a negative replica of the desired product. 2 These replicas are then switched back and forth at appropriate times so as to distribute the good version of the signal to the proper neighbors. In term of silicon area, this solution is more economical than a full four-quadrant multiplier. The core block of this multiplier is made up of matched transistors and. Both are biased in weak inversion [18], so the output current can be written as where is the low-pass filtered normalized photodiode current ( in Fig. 6). If voltages and are set so that then one gets the desired products. As the outputs are currents, the most precise way to generate the voltages is with reference currents. Two reference transistors in the bias block, identical to and, will yield voltages and on the basis of currents so that at the end where is a (rectified) sinusoidally varying reference current generated in the bias block. In this way the multiplications do not depend on any physical process parameter, but only on ratios of local parameters ( and must match one with the other, but they do not need to match to any other transistor in the array). Parameter can be exploited to amplify the currents coming from the photodiode. The current range of the photodiodes is indeed not sufficient to allow a proper working of the local steering function, and the combined action of the global normalization and multiplier amplification is necessary to reach a few picoamperes for dark current up to several hundreds of na for full light. The main drawback of this 2 Experimental results have shown that the relative error in the generation of these two replicas is under 1%. (3) (4) (5) approach is that the precision of the multiplications depends on the precise tuning of and. Such voltages are buffered before being applied to the array. An offset error in these buffers results in a multiplication gain uncertainty proportional to. This, however, has no great consequences as it can be compensated by varying the gain-setting current. Moreover, only the gradient s magnitude would be affected, with the effect of shifting in time by a fixed amount the pulses. According to the proposed asynchronous image processing scheme, such a shift is not detrimental as the order of the dispatching of the information is preserved. Full four-quadrant multiplication is implemented in the following way. Transistors,, provide the replicas of the cosine current, while transistors,, and are the current mirrors output which generate the currents fed to the neighbor pixels. The switch, composed of transistors and, selects the proper branch depending on the value of signals SIGN and NSIGN. When SIGN is high (and, thus, NSIGN is low), a positive current flows out of node IOUTCOS and a negative one from. The sign of these currents reverses when SIGN is low. Transistor is connected between supply voltage and transistor so that glitches that might occur during the transitions of the switch are nearly completely eliminated by the low impedance of the power supply. This elimination would not necessarily occur if the channel s charges were injected into the high-impedance node of the next stage (which would be the case if transistor was connected between transistors and ). Table I shows the physical dimensions of the transistors used in this block. The area for the multiplier block amounts to 1920 m. C. Detection Block This block is concerned with the extraction of the magnitude and direction of the local gradients, using the local steering functions generated by the previous stage. This is achieved by performing three basic operations, which consist in detecting in the input signal the zero crossings (on the descending flank) as well as the crest values (with memorization of their values), and determining the time when the threshold function matches the memorized maximum values. The circuit diagram of the detection block is shown in Fig. 8, while the physical dimensions of the transistors are summarized in Table II. The first task of the detection block is to determine the sign of the local steering function (represented as ) by means of comparator (described in [19]) implemented through transistors,,,,, and. This comparator generates the two-state voltage signal.if is low, then input current enters the circuit and sinks to the

7 166 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 37, NO. 2, FEBRUARY 2002 Fig. 9. Threshold pulse generator. Fig. 8. Circuit diagram of the detection block. TABLE II TRANSISTOR DIMENSIONS FOR THE DETECTION BLOCK (IN m) Fig. 10. Zero-crossing pulse generator. ground via transistor. When is high, it flows out from the input node and is sourced by. Thus, the logical level of indicates the sign of. When input current is negative, it is drawn from the maximum detection circuit (transistors,,,, and ). Maximum detection occurs only during the negative phase of, for which transistor is conducting. In this case, is also conducting and acts on the gate of to equate its drain current to the input current., however, can only act as long as is increasing while it is forced to cut off when the input current starts decreasing. When this latter condition arises, voltages on nodes 1 and 2 increase and turn off, thus, isolating the output transistor that now holds the maximum current reached by the input. An isolation device is necessary to avoid coupling to the hold node 4 via the gate drain capacitance of, which desaturates quickly as soon as the input current starts decreasing. Without this isolation mechanism, the fast variation of the drain potential would induce a charge on node 4, which in turn would alter the stored maximum values in a detrimental way. This problem becomes more acute whenever the maximum input current decreases. Indeed, the charge induced on node 4 does not depend on the value of the current but rather on values of the stray capacitances and supply voltage. Consequently, the relative error induced on the memorized crest value is larger for low currents and weaker for strong currents (that is, currents of the order of hundreds of na). For this reason, the current coming from the photodiode is amplified in the multiplier block (see Section IV-B) to obtain a local steering current which is not affected by this kind of coupling largely attenuated but not eliminated by. Once the maximum (crest) value is available on transistor, it can be compared to the dynamic threshold. The differential current is compared to zero by means of another comparator (transistors,,,,, ), identical to the one described above. The logical value of output voltage will now indicate whether the stored maximum is greater or lower than the threshold. A transition of this logical value constitutes an event. Reset transistors, not shown in Fig. 8, are necessary to clear the stored maximum values when starting a new frame acquisition. Both output signals and are buffered with an inverter before being applied to the next stage (pulse generators, see below). In this way, threshold-crossings are marked by high-to-low transitions while zero crossings are marked by low-to-high transitions. D. Pulse Generators Two slightly different circuits, shown in Figs. 9 and 10, have been designed to generate and communicate the thresholdcrossing and zero-crossing events. A current pulse must be generated for each of these events, but only once for each pixel during a frame acquisition. To ensure this scheme, the pixel communication circuitry must keep track of the latest history of the pixel in two 1-bit registers located in each pulse generator (MEMTH and MEMZ) using local stray capacitances. At each new frame acquisition, a sync signal is received so that all pixels are in state 1 (see Fig. 3). Resetting the zerocrossing pulse generator means that RESET and NRESET (Figs. 9 and 10) are set high and low, respectively. In state 1, any zero-crossing event should be discarded as long as a threshold-crossing event has not yet been communicated. To implement this scheme, the memory node of each pulse generator circuit (MEMTH in Fig. 9 and MEMZ in Fig. 10)

8 BARBARO et al.: A PIXEL SILICON RETINA FOR GRADIENT EXTRACTION 167 TABLE III TRANSISTOR DIMENSIONS FOR THE PULSE GENERATORS (IN m) is set to each time a sync signal is applied. In this initial state, threshold pulses are allowed (switches and of Fig. 9 are ON), while those of the zero-crossing pulse generator are disabled (switch of Fig. 10 is OFF). When the threshold matches the local crest value, a high-to-low transition on input signal occurs and transistors, and (Fig. 9) start conducting. At the same time switch is turned ON and the charge current of increases the voltage of the memory node, pushing MEMTH into the high logical state. This new state implies that switches and are turned OFF. In this way two current pulses flow out of nodes PULSEX and PULSEY. The duration of these pulses is set by a bias current (signal BIAS), which can be adjusted through a SPI register. When MEMTH goes high the zero-crossing pulse generator is enabled ( being turned ON), which corresponds to the machine state circuitry entering state 2 (characterized by the logical condition and ). In state 2 the threshold generator is disabled and only the zero-crossing generator can access its own bus. As soon as a zero crossing is detected on the descending flank, the local steering current produces a low-to-high transition of signal and node is pulled high through the coupling capacitor and switch. This change of state of node makes turn ON transistors MPX and MPY so that and current pulses are initiated. As previously described, the charging current is enabled and memory voltage MEMZ is pulled high. Under these conditions, the pulses are turned OFF. This new state represents state 3 ( 1 and ), during which both generators are disabled. Transistors and of Fig. 10 are needed to pull down nodes and subsequently to a zero-crossing event when the pulse generators are in the initial state (that is, prior to a threshold event). In this state, when the input signal goes high (detection of a zero crossing), node is also pulled high, but no pulse is generated because node is isolated from by means of, which is OFF. Without, would remain in the high state, and as soon as would be turned ON by the threshold pulse generator, would follow (high state) and generate a pulse stemming from a threshold detection and not from a zero-crossing detection. The physical dimensions of the transistors involved in the two pulse generators are summarized in Table III. Two pulses are needed for encoding the column ( -axis) and row ( -axis) address code. A third pulse (FREEZE) is used by the collision avoidance mechanism. When active, this pulse momentarily increases the threshold current by slightly charging the stray capacitance associated to node THRESHOLD Fig. 11. Microphotograph of the realized chip. of the detection block, shown in Fig. 8. In this way, other pixels with crest values closely matching the threshold current are prevented from firing during the time it takes for the currently emitting pixel to communicate on the bus its information. E. Communication Circuitry The role of the communication circuitry is to generate the adequate address code for each firing pixel. Any pulse (be it a - or -axis pulse) is collected on a common line and converted into a digital word with a redundant fixed weight code (fixed number of ones). The wired-or operator of the code of all emitting pixels is then computed and only if the resulting code (both for the - and -axes) is valid a STROBE signal is raised. In this way the STROBE signal can be used off-chip to latch the actual value of the - and -axes addresses. These addresses identify the location of the active pixel, while the time of their appearance on the bus encodes either the magnitude or direction of the spatial gradient (depending on which pulse generator has sent the information). For a more detailed description of the communication circuitry, see [20]. V. EXPERIMENTAL RESULTS A physical implementation of the chip has been realized in a 0.5- m analog CMOS process (Alcatel Mietec). In Fig. 11 the chip s microphotograph is visible. In this chip the total silicon area amounts to 86 mm, with the pixels taking 75.7 mm, and the digital part, included the analog bias circuitry, occupying a thin stripe representing 1.95 mm. Table IV summarizes the characteristics and performances of the device. A critical block for the overall circuit s performance is the multiplier. Experimental measurements have demonstrated good performance of this block in terms of power consumption and accuracy. Although the static power consumption depends on the input current level, it can be estimated to be 1.3 W for an input current of 5 na and a gain of 10 (the gain is specified by setting, see (5)). Output currents have been

9 168 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 37, NO. 2, FEBRUARY 2002 TABLE IV CHIP CHARACTERISTICS Fig. 12. Output current of the multiplier for gains set to 2, 1, 0.5, 00.5, 01, 01 (from top to bottom curves). Fig. 13. figure. Relative error of the multiplier for different gains, listed in previous measured for different gains (2, 1, 0.5, 0.5, 1, 2, negative values correspond to the negative version of the products) and for a range of input currents (0 to 100 na). The results are shown in Fig. 12, while Fig. 13 shows the relative error of the multiplier obtained for different gains. This relative error is always under 10% and decreases with either a higher gain or a lower input current level. The output of the multiplier is fed into the detection block. This latter shows a relative error in the stored crest value of around 2% for a maximum input current of 100 na. This relative error increases with decreasing amplitude of the local steering function, to reach a value of around 15% for an input current of 1 na. For even lower currents, the error dramatically increases and the information is not reliable anymore. Another critical characteristics of the detection block is the delay involved in the generation of the threshold and zero-crossing pulses. This delay ranges between 2 and 5 s, depending on the amplitude of the local steering function. This delay translates into an error of 1% in the crest value and 3% in the orientation angle if the frame acquisition time is set to 1 ms. Since this error is systematic, it could easily be corrected. The choice of 1 ms for the frame acquisition time is compatible with the pulse duration, ranging from 50 to 60 ns, depending on the bias parameters. This pulse duration cannot be further decreased since it is largely limited by the very big stray capacitances seen by each pixel on the output buses. If all events could be detected and transmitted (one for each pixel), 1 ms would be enough to fulfill the acquisition of a frame. Of course, in practice collisions occur, and, thus, the number of effective events is reduced. This number of collisions could be limited by using a slower frame rate. The circuit has been designed to operate at 1000 frames per second. Because the photodiodes are working in current mode, speed information flow does not depend on the illumination, at least not in the illumination range where edge detection is possible. To determine this range, we applied two methods. In the first method, we measured the irradiance, that is, the power per square meter received on the photocaptors. This radiometric quantity has been assessed using a calibrated photodiode (integrated with the same technology) positioned next to the retina to guarantee the same visual field. In the second method, a photometer was installed near the contrast pattern to measure the luminous flux, which can in turn be converted into an illuminance (in Lux). Starting with the radiometric measurements, we found that for a red light source (wavelength 660 nm as used for the character recognition application, [21], [22]), the sensitivity of the retina spans 0.3 mw/m to over 10 W/m (measured for a contrast of over 90%). While 0.3 mw/m corresponds to the lower limit where edges starts to come out from noise, 2 mw/m is required to ensure reliable edge detection of the kind exemplified in Figs. 15, 17, and 18. Also, this reliability is maintained at smaller contrasts only if the light source is stronger. Typically, we observed that for contrasts of 45% and 35% at least 7 mw/m and 20 mw/m, respectively, should be provided to detect with good confidence edges. Application of the photometric measurements indicate that for contrasts of 90%, 45%, and 35%, an illuminance of 5, 20, and 40 Lux, respectively, are needed. Contrasts smaller than 10% are difficult to detect in normal room light, which means that at least 100 Lux are necessary for their detection. Tests of the retina on synthetic and natural scenes have successfully shown the different blocks composing the integrated circuit are properly working. On highly textured surfaces, up to about 60% of all events could generally be collected, the remaining ones being lost in collisions occurring on the bus. This

10 BARBARO et al.: A PIXEL SILICON RETINA FOR GRADIENT EXTRACTION 169 Fig. 14. Output image, collected on the zero-crossing bus, corresponding to a black cross pattern on a white background. Arrows point into the estimated directions of the gradients. result is quite satisfying when one considers the relatively large number of pixels. About 6000 events out of can, thus, be collected and exploited to implement image analysis. Because the information is ordered in time with stronger gradients coming out first, most of these events represent salient information. Also, experimental results with this retina have shown that usually less than 10% out of all events are enough for performing high-level recognition tasks, such as for instance character recognition [21], [22]. Consequently, the potential of the retina to transmit the information is above what is generally needed by the targeted visual tasks. It must also be stressed that the number of collisions could be considerably reduced by optimizing the dynamic threshold, that is, by adapting the slope of the decreasing ramp accordingly to the number of detected collision (in all these experimental measurements, the threshold function was a linear ramp decreasing with a constant slope). Fig. 14 shows the gradient information collected by the retina when the focused image consists of a cross pattern cut in thick black paper. The extracted zero-crossing events are illustrated with small arrows pointing in the corresponding directions of the gradients. The measurements of the timings have been done using a logic analyzer tuned over a suitable time window for the frame acquisition rate set to 1 ms. The collected events correspond to high-contrasted regions, (dark-to-bright and bright-to-dark transitions) and are correctly identified at the proper spatial positions. The estimated directions of the gradients are approximately perpendicular to the iso-contrast lines of the cross pattern. In this experiment, 131 out of 400 salient events have been collected (33% of total). The high number of collisions is due to the fact that the pattern is almost a binary picture (black on white background), with basically four different directions of gradient. Consequently, the boundaries of the pattern yield threshold and zero-crossing events occurring within a very tight time window. In spite of this small number of events, the pattern is clearly visible. To quantitatively assess the error rate of the directions of the gradients, white circles on dark background were focused on the retina. Fig. 15 shows the measured directions (represented by oriented arrows). To acquire these measurements, a more Fig. 15. Output image, collected on the zero-crossing bus, corresponding to a white circle over black background (shown at right). Circle s diameter is 4.5 mm. Distance to lens 15 mm. Measured gradient s directions are encoded with oriented arrows. sophisticated interface than a logic analyzer has been used. It consisted of a dedicated architecture interfacing the retina programmed on a field-programmable gate array (FPGA). The amount of collected information is greater than that for the cross because the time window for the zero-crossing events is wider. This is visible in the figure by the higher density of recorded events. The average of the absolute error on the gradient s directions is 6.1. This average value did not take into account an abnormality in the directions, visible in the figure by arrows pointing inwards (instead of outwards). Those arrows, thus, suffer from a 180 shift. This dysfunction is accountable on a glitch occurring during the transitions of signal (see Section IV-C), whose role is to specify the sign of the local steering functions. Indeed, it might happen that this signal goes low for just a few nanoseconds, then raises high before going definitively low. This glitch necessarily gives wrong information on the sign of the local steering function, with the effect of authorizing the detection of a zero crossing on the wrong flank (in a future integration, this problem will be remedied). The frequency distribution of the errors on the direction of the gradients for the white circle on a black background (excluding the 180 shifts) is shown in Fig. 16. The standard deviation of this distribution is 7.7. The statistics of the errors depends on the contrast level as for weaker contrasts the signal-to-noise ratio increases. For instance, the same circle but on a gray background (same background as before but mixed with 30% of

11 170 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 37, NO. 2, FEBRUARY 2002 Fig. 16. Normalized frequency distribution of the errors on the direction of the gradients. Fig. 18. Output image of a human hand (collected on the zero-crossing bus). of steering periods can be reduced to four (corresponding to a frame rate of 1000 Hz),but in general such a reduced number leads to a larger incidence of collisions. Conversely, a too large number of steering periods could be detrimental as the maximum values stored on the parasite capacitances could be affected by the leaking current. Fig. 17. Output image corresponding to the printed letter R (collected on the zero-crossing bus). white pixels), is characterized by a standard deviation of 10.4 for the error distribution. The two last examples deal with more complex pictures. The first one (Fig. 17) corresponds to the image of a letter, taken from a set of characters used for testing an OCR system based on the new retina [21], [22]. The second example (Fig. 18) corresponds to a human hand. In both cases, the level of details is quite good in spite of the numerous collisions occurring on the buses. In all these experiments the number of steering periods for collecting the information was set to 10 (corresponding to a frame rate of 400 Hz). On this number, the first period is used for detecting the maximum and the last period is held in reserve for collecting the remaining orientation events. The total number VI. CONCLUSION A new pixel retina able to extract spatial gradients in a massively parallel fashion was presented. A detailed description of the analog circuitry as well as a brief description of the underlying algorithm were provided. Complex functionalities embedded in each pixel have been met by designing new circuits, such as the ones used to perform the computation of the local steering signals and the multiplier, and by reusing existing ideas (i.e., photodiode, detection block, and communication block). Computation time of less than 1 ms for the whole array is achievable. This timing allows an output rate of the order of 1000 frames per second, a rate that has only recently been proposed for a pixel vision chip using a MIMD architecture to compute spatial convolutions [23], whose results can be combined to obtain the orientation of edges. Although our approach is less flexible than this latter architecture, it has the advantage of directly providing the orientation of the edges on the output bus. The key to such fast processing is the application of the theory of steerable filters to compute the magnitude and direction of the gradients. Analog VLSI implementations of steerable filters, with interpolating functions steering in time, has not previously been described in the literature. The benefits of this new approach is speed and compactness (in term of silicon) of the filter. It must however be stressed that because the photodetectors are nonintegrating, the maximum achievable