Image Processing Vision System Implementing a Smart Sensor

Transcription

1 IEEE IMTC 2004 Instrumentation and Measurement Technology Conference Como, Italy, May 2004 Image Processing Vision System Implementing a Smart Sensor A. Elouardi, S. Bouaziz, A. Dupret, J.O. Klein, R. Reynaud Institut d'électronique Fondamentale Bât Université Paris Sud Orsay 91405, France Phone: , Fax: , elouardi@ief.u-psud.fr Abstract There are many vision algorithms of image processing with CMOS image sensors such as image enhancement, segmentation, feature extraction and pattern recognition. These algorithms are frequently used in software-based operations. Here, the main research interest focuses on how to integrate image processing (vision) algorithms with CMOS integrated systems or how to implement smart retinas in hardware, in terms of their system-level architectures and design methodologies. The approach is compared with state-of-the-art vision chips build using digital SIMD arrays and vision systems using CMOS sensors combined with FPGA technology. As a test bench, an advanced algorithm for an exposure time calibration is presented with experimental results of image processing. Keywords System on Chip, Image processing, Retina, CMOS/APS Sensors I. INTRODUCTION Conventional mechanisms for image processing consist of a standard camera and a high speed computers system (embedded PC104, FPGA or DSP module) that performs most (if not all) of the off-chip processing. The aim of this research is to integrate certain aspects of off-chip processing with the image sensor to achieve a vision System on Chip. The current implementation of most vision systems involves a number of different subsystems that rely mainly on digital signal processing. By shifting a few of these simple processes into the vision microchip, the complexity of subsequent off-chip processes can be reduced. The retina is the first element in the visual processing system of vertebrates. It serves 3 purposes: (1) optical sensing, (2) edge extraction and (3) motion detection. Analog systems, that perform this same spatial filtering in silicon, have been achieved silicon retinas [1]. Smart Retinas are integrated circuits in which the imagesensing element and processing circuits co-exist. Having analog and/or digital signal processing elements close to the pixel [3][4][7] or along with the image sensor array [5], enables to go beyond the transduction function and to achieve some low-level image processing tasks (early-vision). Most often, such circuits are for specific application. Their key features are their capability to enable massively parallel computations with a rather low power. Many approaches have been investigated: The approach presented by P. Dudeck [6] combines the architectural features of a general-purpose single instruction multiple data (SIMD) concept processing in the focal-plane. The implementation of an analog microprocessor in the pixel results in an Analog Processing Element (APE) and a programmable pixel-per-processor array. The suggested architecture is similar to that of retinas, but each processor has several analog memories, a communication register with neighbors pixels and a current multiplier. Consequently, the fill factor is low and the area of a pixel remains too large (98x98 µm²) to consider retinas of high resolution. These pixel dimension is 100 times larger than a standard CMOS/APS pixel. Another approach is presented by M. Arias-Estrada [8]. A CMOS imager is used to develop image processing architecture with the FPGA technology. The disadvantage here is the bottleneck related to the data flow between the sensor and the processing circuit and the deficiency of any kind of configuration or programmability of the array. This solution requires firm methods for a hardware implementation of the algorithm on FPGA processing circuits. In this paper, a new processing architecture approach is presented. It highlights a compromise between versatility, parallelism, processing speed and resolution. This enables to increase the system performances. The approach consists to set operators, usually integrated close in the pixels, at the array edge. Consequently, these functions are shared by a group of pixels, and the image processing is then carried out sequentially. This architecture results in a pixels array associated to a mixed analog-digital processors vector. Each processor is able to carry out, in situ, a wide range of low-level image processing algorithms [9][10]. The low-level information can be then processed by a processor. The aim when integrating such a processor, next to image sensor in a single circuit, is to increase the fill factor and to eliminate the input output bottleneck between the sensor and the processor. Our solution aims to take into account the algorithms response time with a significant resolution of the sensor, while reducing energy consumption for embedding reasons. The system becomes more compact and it can reach processing speeds suitable for real time applications. II. RETINA DESCRIPTION PARIS (Programmable Analog Retina-Like Image Sensor) is a retina architecture concept. It integrates in the

2 same circuit the acquisition photo-sensors and some hardware processing operators [11]. It assumes on chip parallel processing. So, it is not necessary to acquire all the image pixels out off the circuit. The final hardware processing objective is to reduce the data bandwidth exchange with external hardware environment. This architecture, shown in figure 1, is designed to support up to 256x256 pixels. We observe three kind blocks: an array of analog memories with photo-sensors, an analog processing vector and a microprocessor. PARIS1 is a 16x16 pixels VLSI prototype with 16 analog processors. This first circuit allows validating the integrated operators through some image processing algorithms like edge and movement detection. The pixel array is associated to a vector of processors operating in an analog/digital mixed mode. The column structure of PARIS architecture implements a mixed analog/digital unit: analog processor (AP), Boolean unit (BU) and analog registers. The figure 3 gives architecture details of the analog processor. The comparator allows analog results to be converted to logical predicate signal. The analog processing unit (AP) is based on switched capacitors integrators. It contains three capacitors, one OTA (Operational Transconductance Amplifier) and a set of switches controlled by the ARM processor. The capacitor Cout is used as an accumulator. Thanks to the OTA, the charge stored in capacitor Cin1 is transferred towards Cout so that it adds or subtracts. Fig. 3. Analog Processor Architecture Fig. 1. PARIS Architecture The analog processors execute instructions fed by a microprocessor achieving a program. Pixels can be randomly accessed. Each pixel consists of a photo-sensor and four analog capacitors acting as memories (figure 2). Furthermore when the capacitor Cin1 is first shorted, and then put in parallel with Cin2, the charges balance among Cin1 and Cin2. As a result, this sequence of operations achieves the division by two of the present charges on the two capacitors Cin1 and Cin2. Hence, the transferred charges can be divided by two. Pixels of the same line are simultaneously processed by the Analog Processors (AP) vector and the computing is iterated on image rows. The arithmetic operations (division, addition) are carried out in analog. The accumulation of the intermediate results is achieved in the analog processor by using the internal analog registers. We can distinguish two kinds of data processing: - low-level hardware operation: addition, subtraction - high-level software sequenced operation: MAC (Multiply-and-Accumulate), spatial filtering... The analog processor unit integrates a basic analog comparator which can be used by software sequenced operation as an analog to digital converter. Such a structure has the advantage of an analog ALU which avoid data conversion stage. However, all data processing are in analog but the control data flow is digital. Fig. 2. Pixel Architecture

3 III. GLOBAL ARCHITECTURE Most of intelligent vehicles applications use image sensors and image processing [12][13]. These computations require a significant computing power associated to data exchange mechanisms. These functions, originally achieved by FPGA or DSP circuits, can advantageously be carried out by a microcontroller based on RISC processor coupled to an electronic Retina. As microcontrollers have become more prevalent and their abilities have increased, it is possible to perform simple pixel processing on the fly as the pixel values are scanned out of the retina and so a full frame buffer is not necessary. On another side, a major advantage of Retinas versus CCD cameras technology is the ability to integrate additional circuitry on the same chip along the array of pixels. Since microcontrollers have asset of high integration, high computing power and low energy consumption, these characteristics make them necessary to the CMOS/APS imager sensors or Smart Retinas (known as intelligent sensors). Such microcontrollers support various Operating Systems and communication drivers. This suggests that it should be possible to associate a CMOS Retina chip with a low cost microcontroller to implement an on chip vision system. ARM7TDMI RISC processor. It is a low-power, general purpose microprocessor, operating at 50 MHz, that was developed for use in specific applications and custom integrated circuits. The retina, used as a standard peripheral of the microcontroller, is dedicated to image acquisition and lowlevel image processing. The processor waits for the extracted low-level information incoming data from the retina and processes them at video rate. The system sends an entire raw image sequences that we can view using a specific soft monitor. With all components listed above, we obtain a system vision that uses a fully programmable smart retina. Thanks to the analog processing units, this retina extracts the low-level information (e.g. edges detection). Hence, the system, supported by the processor, becomes more compact and can achieve processing suitable for real time applications. The advantage of this architecture type remains in the parallel execution of a consequent number of low level operations in the array by integrating operators shared by groups of pixels. This allows saving expensive resources of computation, and decreasing the energy consumption. In term of computing power, this structure is more advantageous than that based on a CCD sensor associated to a microprocessor. Figure 4 shows the global architecture of the system and figure 5 gives an overview of the implemented module. Fig. 5. Implemented module IV. CALIBRATION AND RESULTS A. Auto-Calibration Algorithm Fig. 4. Global Architecture To evaluate this architecture, we have implemented a prototype based on this idea. It is a three design parts. The first two chips are the smart retina and the microcontroller. The third part is a simple interface card implementing DAC/ADC converter and decoders circuits. The microcontroller is built around a CPU core: the 16/32-bit Luminosity variations cause unavoidable non-uniformities in focal plane arrays and others integrated sensors. Since these non-uniformities change with time, calibrating sensors once is not suitable and frequent calibrations are then required. Image calibration enable to reduce the sensor noise and increase contrast. Machine vision requires an image sensor able to capture natural scenes that may have a dynamic adaptation for intensity. Reported wide image sensors suffer from some or all of the following problems: large silicon area, high cost,

4 low spatial resolution, small dynamic range, poor pixel sensitivity, etc. The primary focus of this research is to develop a singlechip imager for machine vision applications which addresses these problems, but is still able to provide an on-chip automatic exposure time algorithm by implementing a novel self exposure time control operator. The secondary focus of the research is to make the imager programmable, so that its performance (light intensity, dynamic range, spatial resolution, frame rate, etc.) can be customized to suit a particular machine vision application. Exposure time is an important parameter to control image contrast. This is the motivation for our development of a continuous auto-calibration algorithm that can manage this state for our vision system. This avoids pixels saturation and gives an adaptive amplification of the image, which is necessary to the post-processing. The calibration concept is based on the fact that since the photo-sensors are used in an integration mode, a constant luminosity leads to a voltage drop that varies according to the exposure time. If the luminosity is high, the exposure time must decrease, on the other hand if the luminosity is low the exposure time should increase. Hence lower is the exposure time simpler is the image processing algorithms. This globally will decrease response time and simplify algorithms. We took several measurements with our vision system, so that we can build an automatic exposure time checking algorithm according to the scene luminosity. Figure 6 presents the variation of the maximum grey-level according to the exposure time. For each curve, we note a linear zone and a saturation zone. Thus we deduce the gradient variation (Ämax/Ät) according to the luminosity. The final curve can be scored out as a linear function (figure 7). The algorithm consists to keep the exposure time in the interval where all variations are linear and the exposure time is minimal. Control is then initialised by an exposure time belonging to this interval. When a maximum grey-level is measured, the corresponding luminosity is deduced and returns a gradient value which represents the corresponding slope of the linear function. Fig. 6. Maximum grey-level versus exposure time Fig. 7. Gradient variation according to the luminosity B. Image processing results The basis of the smart vision system on chip concept is that analog VLSI systems with low precision are sufficient for implementing many low-level vision (image processing) algorithms, often for application-specific tasks. Conventionally, smart sensors are not general-purpose devices, but everything in a smart sensor is specifically designed for the targeted application. The aim of this study is to investigate what image processing algorithms can be integrated on smart sensors as a part of early vision sequences and to examine their merits and the issues that designers should consider in advance. In this paper, we do not wish to limit implementations to application-specific tasks, but to allow for general-purpose applications such as DSP-like image processors with programmability. The idea is based on the fact that some of early level image processing in the general-purpose chips are commonly shared with many image processors, which do not require programmability on their operation. Indeed, the aim of our work is to explore the potential benefits of using a retina plus a DSP instead of the classical approach (imager associated to a DSP). We hence focus our works on the point where the gain of our approach can be significant in the low level image processing step of the image vision. Since the retina we use combines a fully random access to the pixel along with the ability to perform parallel computations, the low level image processing tasks can be performed at high speed while the ARM processor handle the high level processing tasks. General image processing consists of several image analysis processing steps: image acquisition, pre-processing, segmentation, representation or description, recognition and interpretation. The order of this image analysis can vary for different applications, and stages of the processes can be omitted. In image processing, the image acquisition is used to capture raw images from its input scene, through the use of video

5 camera, scanners and, in the case of smart retinas, the solidstate arrays. Local operation is also called mask operation where each pixel is modified according to the values of the pixel s neighbours (typically using convolution masks). This kind of operation is spatially dependent on other pixels around the processed pixel: the final value of the processed pixel is affected by its neighbouring pixels in the finite sized masks. The basic approach of the convolution is to sum products of the mask coefficients and the intensities of the pixels under the mask, at a specific location in the image. Denoting the gray levels of pixels under the mask K (3x3 Mask in this example) at any location by K ij, the response of a linear mask is: O = 3 3 K P xy ij x+ i 2, y+ j i= 1 j= 1 2 To evaluate the functioning of PARIS architecture (each column is assigned to an analog processor). We chose a traditional example consisting of a spatial filtering (a convolution with a matrix 3x3). The convolution kernel K used is the following: 0-1/4 0-1/4 1-1/4 0-1/4 0 Starting from an acquired image, the figure 8 shows the K filtering operation result of a NxN pixels image, obtained by PARIS1 with N=16. The first line is not taken into account for the computation of the final image. Such operation is achieved in 6887 µs. With K = k 11 k 12 k13 k21 k22 k23 k31 k32 k33 The gray level of the pixel located at (x, y) is replaced by Oxy if the center of the mask is at location (x, y) in the image. This computation is repeated as the mask is moved to the next pixel location in the image until all the pixels in the array are covered. Linear spatial filters are defined such that the final pixel value, Oxy, can be computed as a weighted sum of convolution mask (non-linear filters cannot be implemented in this way). In the above case, 3x3 local mask was taken as an example for the convolution mask. However, the size of the convolution mask is not restricted to 3x3, but can be expanded to 5x5, 7x7, 9x9, and larger, depending on the filter to be implemented. In aspects of on-chip integration with image sensors, local operations provide advantages of real time operation in image acquisition and processing, such as implementations of many practical linear spatial image filters and image enhancement algorithms. In addition, because the local operation is feasible for column structure implementations, low frequency processing is enabled and thus low power consumption is possible. However, since the local operations are based on a technique where local memory stores pixel values of the neighbours and processes them concurrently, implementation of the operation must contain some type of storage. Applications of local operations typically use an iterative technique for advanced image enhancement algorithms, which cannot practically be implemented on-chip. Nevertheless, in the case of column structure implementations, local operation still has a limitation on design area because of the restricted column width, even with flexible design area in the vertical direction. Therefore, in order to overcome these limitations, careful designs and system plans are required for the on-chip implementations. Fig. 8. Original image (left) and filtered image (right) Opposite to integration that is analogous to averaging or smoothing, differentiation can be predictable to sharpen an image leaving only boundary lines and edges of the objects. This is an extreme case of high pass filters. The most common methods of differentiation in image processing applications are first difference, gradient and laplacian operator. The difference filter is the simplest form of the differentiation with subtracting adjacent pixels from the centered pixel in different directions. The gradient filters represent the gradients of the neighboring pixels (image differentiation) in forms of matrices. Such gradient approaches and their mask implementations are represented with various methods: Roberts, Prewitt, Sobel, Kirsch and Robinson. With many different local operations in image processing algorithms, these operations can be categorized into three major groups: smoothing filters, sharpening filters and edge detection filters. We have successfully implemented and tested a number of algorithms, including convolution, linear filtering, edge detection, segmentation, motion detection and estimation. Some examples are presented below. Images are processed at different values of luminosity using the exposure time self calibration. Since the input signal is always smaller than the input range, no saturation occurs. When successive operations are performed, the coefficient applied to the input signals must be chosen so that their sum remains lower than maximum range to prevent saturations. The real limitation comes from the dynamic (the lower bound is due to noise, component mismatch, non linearities.) of the analog processor. Finally

6 local calibration may be locally achieved, thanks to the random access to the pixel. Figure 9 gives an example of images showing the adaptation of the exposure time to the luminosity and figure 10 gives examples of processed images. We have presented the PARIS architecture and the implementation of a prototype based vision system. The goal is the integration of a processor in the retina, to manage the system and optimize the hardware resources use. We propose evaluating such a system with a retina's high resolution starting from a complex application for autonomous collision avoidance and objects tracking at realtime. VI. REFERENCES 928 µs, 35 Lux 512 µs,80 Lux 2 µs, 1000 Lux Fig. 9. Exposure time adaptation to the luminosity Rough image Horizontal Sobel filtered image Binary Image Vertical Sobel filtered image Fig. 10. Examples of processed images V. CONCLUSION The require for real time image processing applications for portable and battery-operated devices has grown swiftly in recent years [14], and it has motivated the research on processing architectures that support focal plane data. It is concluded that on-chip image processing with retinas will offer benefits of low manufacturing cost, low power consumption, fast processing frequency and parallel processing. Since each vision algorithm has its own applications and design specifications, it is difficult to predetermine optimal design architecture for every vision algorithm. However, in general, the column structures appear to be the best choice for typical image processing algorithms. [1] A. Moini, "Vision chips or seeing silicon", Technical Report, Centre for High Performance Integrated Technologies and Systems, The University of Adelaide, March Kluwer Academic Publishers, ed. I [2] A. Elouardi, S. Bouaziz, R. Reynaud "Evaluation of an artificial CMOS retina sensor for tracking systems" Pro. of IEEE IV'2002, Versailles, France. [3] R. Burns, C. Thomas, P. Thomas, R. Hornsey "Pixel-parallel CMOS active pixel sensor for fast objects location" SPIE International Symposium on Optical Science and Technology, 3-8 Aug. 2003, San Diego, CA USA. [4] Abbas El Gamal, and al, "Pixel level processing Why, what and how?" SPIE Vol.3650, 1999, pp [5] Y. Ni, J.H Guan ''A 256x256-pixel Smart CMOS Image Sensor for Line based Stereo Vision Applications'', IEEE, J. of Solid State Circuits, Vol. 35 No. 7, Juillet 2000, pp [6] P. Dudek "A programmable focal-plane analogue processor array Ph.D. thesis, University of Manchester Institute of Science and Technology (UMIST), May [7] P. Dudek, J. Hicks "A CMOS General-Purpose Sampled-Data Analogue Microprocessor" Pro. of the 2000 IEEE International Symposium on Circuits and Systems. Geneva, Suisse. [8] Miguel Arias-Estrada. "A Real-time FPGA Architecture for Computer Vision" Journal of Electronic Imaging (SPIE - IS&T), Vol. 10, No. 1, January 2001, pp [9] A. Dupret, J.O Klein, A. Nshare "A programmable vision chip for CNN based algorithms" CNNA 2000, Catania, Italy: IEEE 00TH8509. [10] A. Dupret, J.O Klein, A. Nshare "A DSP-like Analog Processing Unit for Smart Image Sensors", International Journal of Circuit Theory and Applications : p [11] A. Nshare, J.O Klein, A. Dupret "An improved ARAM for PARIS, an original vision chip" CNNA2002, Frankfurt/Main, World Publishing 2002, pp [12] Shoji Muramatsu and al., Image Processing Device for Automotive Vision Systems, Pro. of IV'2002, Versailles, France. [13] D.M Gavrila, V.Philomin "Real-Time Object Detection for Smart Vehicles" Pro. of IEEE International Conference on Computer Vision, pp , Kerkyra [14] A. Gentile, J.L. Cruz-Rivera and al "Real-Time Image processing on a Focal Plane SIMD Array" IPPS/SPDP Workshops 1999: San Juan, Puerto Rico, USA.