Field programmable gate array based hardware implementation of a gradient filter for edge detection in colour images with subpixel precision

Transcription

1 Field programmable gate array based hardware implementation of a gradient filter for edge detection in colour images with subpixel precision M Schellhorn, M Rosenberger, M Correns, M Blau, A Göpfert, M Rückwardt and G Linss Ilmenau University of Technology, Faculty Mechanical Engineering, Department of Quality Assurance, Gustav-Kirchhoff-Platz 2, Ilmenau, Germany mathias.schellhorn@tu-ilmenau.de Abstract. Within the field of industrial image processing the use of colour cameras becomes ever more common. Increasingly the established black and white cameras are replaced by economical single-chip colour cameras with Bayer pattern. The use of the additional colour information is particularly important for recognition or inspection. Become interesting however also for the geometric metrology, if measuring tasks can be solved more robust or more exactly. However only few suitable algorithms are available, in order to detect edges with the necessary precision. All attempts require however additional computation expenditure. On the basis of a new filter for edge detection in colour images with subpixel precision, the implementation on a pre-processing hardware platform is presented. Hardware implemented filters offer the advantage that they can be used easily with existing measuring software, since after the filtering a single channel image is present, which unites the information of all colour channels. Advanced field programmable gate arrays represent an ideal platform for the parallel processing of multiple channels. The effective implementation presupposes however a high programming expenditure. On the example of the colour filter implementation, arising problems are analyzed and the chosen solution method is presented. 1. Introduction The use of colour images (multi-channel images) in geometric metrology becomes increasingly interesting. By the further spreading of digital cameras for surveillance, digital video and still photography, not least for mobile phones, the number of colour sensors continues to rise. That affects also the industrial image processing and metrology. Price differences between colour and black and white cameras became substantially smaller. Since human operators prefer colour images, colour cameras are increasingly used. But there are not only aesthetic reasons. For the measurement of geometric sizes with image processing the detection of edges is significant. Edges constitute typically only about 5% of the picture content, represent however the largest part of information content [1]. According to Novak and Shafer [2] 90% of the edges in a colour image correspond also to grey value edges. The remaining 10% correspond to colour edges, which can be detected only by the evaluation of the colour information. That means, colour images offer also from view of geometric metrology an increase of the information content.

2 2. Edge detection in colour images Why are geometric measurements more difficult in colour images than in grey value images (singlechannel images)? Typically methods [3] for subpixel-accurate edge detection presume that only one function value, i.e. the intensity respectively the grey value, varies within the image. But in colour images there is more than one value per pixel. In order to clarify this assumption, the technical structure of colour sensors must be understood Digital colour imaging techniques for machine vision cameras Light-sensitive photo-detectors form the basic elements of image sensors. The output voltage from each photo-detector is proportional to the incident light intensity integrated for an exposure time. Image sensors capture thus pure intensity or grey-value images. To acquire colour information the technology was inspired by the nature. The human eye contains three separate types of colour receptors. Each type responds to different wavelengths, which correspond to the colour ranges red, green and blue. Highest sensitivity is thereby in green range. All other colours result from different mixtures of these three basic colours (Young-Helmholtz theory). To make image sensors sensitive to colour, some way of imitating these Red-Green-Blue (RGB) response of human eye is needed. For the realization there are three fundamental approaches: colour-wheel cameras, three-chip cameras and single-chip cameras with colour filter array [4]. Single chip cameras represent thereby the most economical variant. Different arrangements of the colour filters are possible, the Bayer pattern however as far as possible became generally accepted. On a colour image sensor with Bayer pattern, half of the pixels, arranged in a chessboard pattern, are coated with a green filter. The other half of the pixels is alternating coated with red and blue filters [5],[6]. Thus each pixel is sensitive only for certain colour range. For a complete colour image with the same dimensions therefore the missing colour components for each pixel must be interpolated. For the colour interpolation different demosaicing algorithms are used. Simple algorithms interpolate the colour value from the pixels of same colour in the neighbourhood. In addition, for geometric measurements in [7] a appropriate demosaicing filter was already introduced. A digital colour picture consists thus of three colour layers, of which everyone contains the information of one colour channel (red, green or blue). Now the question arises, whether the advantages of colour cameras can be used and additionally the same accuracies, which can be obtained with grey value cameras of same resolution, are attainable? For this goal the use of all pixels for the measuring process and edge detection with subpixel precision is necessary. Accordingly the information from additional colour-channels must flow into the measuring process Methodology of edge detection in colour images For the detection of edges in colour images different approaches are possible [8]. According to the position of the image recombination in the algorithm, colour edge detection algorithm can be classified as vector methods, multi-dimensional gradient methods and output fusion methods (Figure 1.). Figure 1. Flowchart of colour edge detection algorithms after [8] classifying a.) Vector methods, b.) Multi-dimensional gradient methods, c.) Output fusion methods.

3 Output fusion methods work on basis of the grey value edge detection algorithms. As a state-of-the-art method a RGB to hue-saturation-intensity (HSI) conversion is performed. The actual measurement is accomplished only in the intensity channel, which represents a single-channel image comparably to a black and white image. Thus only intensity edges are measurable. Hue and saturation channels are not analysed, therefore hue and saturation edges are not detected. Multi-dimensional gradient methods determine the orientation and strength of an edge for each point. The gradients of the image components are computed by evaluation of the first or second derivative of the image function. There are several approaches [9], [10], [11] to combine them into one result. With vector methods the problem is omitted to combine the information of the individual channels, since the gradient finding take accomplished in vector space. Representation and use of the vectors vary strongly [8]. There are a lot of image processing algorithms for recognition/detection or colour measurement but only a view approaches for edge detection in colour images with subpixel precision. For recognition algorithms an exact edge localisation is less important than actual detection. For geometric measurements, a shift of the edge position through the detection algorithms is however problematic. Due to this fact, most image processing software use outputs fusion methods. This has the advantage that edge detection algorithms from the grey value image processing can be applied. Subpixeling methods in grey value images enable edges detection with a resolution larger than the pixel centre distance of the sensors. Depending on the quality of the image data, an increase of resolution of 1/10th to 1/100th pixels is attainable. As a disadvantage thereby effectively only one third of the sensor information are used. 3. Filter for edge detection in colour images with subpixel precision For this paper the goal of application is clearly defined. Special attention is given to the measurement of geometric features in images acquired with multi-channel systems. Therefore the usual procedures for the highlighting of edges are not applicable. A method is needed, which considers also differences between the channels, without by opposite signs a compensation of such differences can occur (averaging). Colour, saturation and intensity edges must be equally detected. For the research, presented in this paper, a gradient filter was used. This is justified by the fact that gradient filters can be implemented more simply in existing image processing software than edge detection algorithms for several channels. After the filtering the image must have the same resolution and size as the original image. In addition it must consist only of one channel, which contains all edge information. Beyond that the filter should soften as little as possible and do not affect the edge position. A further important aspect is the direction-dependency. An absolute direction-independency cannot be guaranteed by the firm pixel grid, but the pixel structure, which is regarded, should be possibly circular. In order to fulfil these demands a new filter presented in [12] was developed. The filter follows the Roberts gradient filter, which fulfils the demand of little softening. In order to evaluate horizontal/vertical edges exactly the same as diagonal edges, additional computations are introduced, which are missing with the Roberts gradient filter. Contrary to scalar differences, the presented filter is based on a vector approach. The colour channels of the picture are understood not as layer, but instead each pixel is a vector. Each component of the vector corresponds to the intensity value in one of the colour channels. It is based on the concept from an algorithm for edge detection in colour images, described in [13]. The used filter method thus is an intermediate form of the vector method and the multi-dimensional gradient method. Conventional edge filters are conceived for the filtering of single-channel images. In case of a multichannel image, these filters can only be applied separately for each channel. With the difference vector filter however not only intensity, but also colour hue and colour saturation edges are represented in an single-channel image.

4 Figure 2. Source image with 2 x 2 filter scope with horizontal, vertical and diagonal difference vectors. Each pixel (C1 to C4 in figure 2) is regarded as vector, whose components represent the values of the individual colour channels (RGB). By this attempt the filter is suitable also for higher dimensional colour spaces. First the difference vectors of the 2 x 2 filter scope are formed. The horizontal and vertical difference vectors are summarized separately. The diagonal difference vectors are, due to the larger pixel distance, weighted with the factor 1/ 2. Then the norm of the four difference vectors are added and multiplied by a scaling factor f (equation (1)). The scaling factor is necessary to adjust the edge value k to the range of values of the filtered image. For the computation of the scaling factor the largest sum of the difference vectors of a complete picture is determined. 1 2 (1) A disadvantage of this algorithm is however that the entire image must be completely filtered. Filters require a several arithmetic function for every pixel. This leads to an increased computation task. In order to compensate the additional computation time, filters could implemented to pre-processing hardware components such as Digital Signal Processors (DSP) or Field Programmable Gate Arrays (FPGA). These arithmetic and logic units can be inserted in the data stream either directly integrated in a camera or by adding a pre-processing card in the computer. Hardware implementation offers the possibility to operate filters with minimal time delay in real time. 4. FPGA implementation of the gradient filter algorithm Due to the increased computation time for such a filter, an implementation was considered on external logic. FPGAs are particularly suitable for the realization of filters. FPGAs are integrated circuits (ICs) that consist of a two dimensional array of general purpose logic blocks. These logic blocks can be configured to perform combinational functions or simple logic gates like AND and OR. In most FPGAs, the logic blocks also include memory elements, which may be simple flip-flops or more complex blocks of memory. This architecture allows the implementation of microprocessors, random access memory (RAM), read-only memory (ROM), digital signal processing (DSP) and logic functions in a single chip. In the presented implementation a low cost Spartan 3E FPGA (XC3S500E- 5) from Xilinx was used Difficulties for implementation of digital filters on FPGAs By the flexible programming possibilities of FPGAs very high system clocks can be realized. The fundamental programming conversion differs however from usual programming strategies. Arithmetic operations must be converted in hardware, which is not always possible without problems. Beyond that different programming can supply the same result, but exhibit substantial differences in resource consumption. Shifting operations, additions and subtractions can be implemented almost without restriction through basic logic elements. Multiplications are converted into multiplier networks or routed on one of the few existing hardware multipliers available only in a few FPGA. However divisions and modulo operations are usually not supported in hardware or only for terms of power-oftwo. Depending on necessary precision and obtaining speed, different algorithms can be implemented, in order to realize a division [14]. These implementations are highly resource-intensive. Especially squaring and extracting a root are costly in terms of resource consumption. These restrictions of hardware make a filter implementation difficult. It must be considered exactly, which hardware resources are used and how an existing algorithm can be simplified.

5 There are essentially three difficulties during the implementation of the filter on a FPGA. In order to determine the amounts of the individual difference vectors, the square root must be extracted from the sum of the squares of the components (equation (2)) (2) In addition a division is necessary for the computation of the scaling factor f. Finally, in order to determine the largest sum of the difference vectors of the entire image, a complete image must be buffered Square root calculation Four difference vectors must be computed for each pixel, this means that, with each pixel clock, four roots must be extracted. As mentioned in section 4.1, FPGAs do not support root operations in hardware. For the implementation of the filter three methods were analyzed, to accomplish the extraction of the root: look up table (LUT), Heron s method and the method of bisecting intervals. Equation (3) describes the rule of the bit widths valid for the calculation square roots: 2 (3) In the image processing bit widths between 8 and 12 bits are typical, larger bit widths are however possible. For the experimental setup a camera with 8 bit wide data bus was used. In order to obtain higher accuracies the computations were accomplished internally with 10 bit values. For 10 bit wide square roots, therefore the radicands must be 20 bit wide. The LUT access with binary search is for all strictly monotonous functions usable, so also for a square root. The memory, where the table is stored, is addressed with the 10 bit output value. The necessary memory size, which contains the square for each index, can be calculated with equation (4) (4) Block RAM 1 could be addressed accordingly. The programming effort to deposit each possible value is however enormously. The main problem is however, the LUT is limited to a firm bit width and thus is no longer scalable. The deposit of larger bit widths needs also increasingly more memory (table 1). Thus the use of a LUT for the implementation is unsuitable. Table 1. Memory consumption for variable bit widths. Radicand Square root Memory 8 bit 4 bit 16 byte 16 bit 8 bit 512 byte 20 bit 10 bit 2,5 KB 24 bit 12 bit 12 KB 32 bit 16 bit 256 KB The second approach was the realization of Heron s method. This special case of Newton s method provide an approximation of the square root x of the radicand a (equation (5)). The procedure converges very fast, if a good approximation is already present. Since the Heron s method can be derived from the Newton's approximation method, the convergence order is 2. The number of the correct digits is doubled with each step. 1 2 lim (5) 1 Embedded Block RAM memory is available in most FPGAs, which allows for on-chip memory implementation [15].

6 The computation leads thereby after a few iterations already to acceptable results. For the implementation on a FPGA the method is however problematic by the need of two divisions. The method of bisecting intervals represents a structured guess-and-check algorithm. The quantity of all possible results [L 0, H 0 ] is halved and divided thus into two intervals [L 0, x n ] and [x n +1, H 0 ]. After the decision in which of the intervals the square root is contained, this interval again halved (equation (6)) (6) This produces a convergent series of nested intervals witch converge to a unique point, which should therefore be a. Each decision halves the number of possible results. The method of bisection intervals is simple and robust. Since the interval with each step is halved, the method of bisecting intervals is very effective in binary system. For the computation of the 10 bit square root thereby 10 iterations are necessary. For the implementation of the filter the method of the bisecting intervals was selected. The reasons for the choice were the numerous advantages: scalability, parallelizability and smallest computation expenditure of the examined methods. For faster convergence and optimal hardware consumption the method was linked with a shorten LUT, which already limits the initial value. Table 2 shows the attainable clock delays and necessary resource consumption for different combinations. Table 2. Comparison of the square root calculation by combination of method of bisecting intervals with a LUT in different depth LUT condition Levels of bisecting Slices Delay in clock cycles intervals Scaling of the filtered image Since it is not possible, to buffer a complete image FPGA internally, and on the experimental hardware does not provide external memory, a compromise had to be closed. It was specified as boundary condition that D max changes only slightly from one to the next image. Thus the scaling factor of the current image can be estimated by the D max of the previous image added by a small security factor s. Since only one division per image must be accomplished, sequential algorithms can be used. The computation takes place according to each image during the rear vertical porch. 5. Results and discussion For the evaluation a camera with a complementary metal oxide semiconductor (CMOS) image sensor and universal serial bus (USB) 2.0 interface was build up. By the use of a 1/2-inch CMOS image sensor, with the maximum resolution of 1280 x 1024 pixels, customizable different display resolutions of VGA over XGA up to SXGA can be realized. Programming was accomplished in Very High Speed Integrated Circuit Hardware Description Language (VHDL). The Xilinx Integrated Software Environment (ISE) 11.4 was used for synthesis and implementation (mapping and place & route). The FPGA design including demosaicing and gradient filter requires 2366 slices and 5 x 1024 Byte of Block RAM on the XC3S500E-5. An excerpt of the FPGA utilization, reported by the Xilinx ISE, is shown in table 3. The post-routing static timing analysis suggest, that the maximum pixel clock frequency can be PCLK max = 50 MHz.

7 Table 3. Resource utilization statistics. Resource Number Utilization rate Occupied Slices % Slice Flip Flops % 4 input LUTs % RAMB16s 5 25 % BUFGMUXs 2 8 % MULT18X18SIOs % The block RAM is used to buffer five image rows, which are needed for the demosaicing process. Thereby sensor resolutions with a maximum line length of 1024 pixels can be filtered at the moment. However the filter can be adapted by an enlargement of the line buffer even for larger sensors. With the current realization a real time filtering of images with VGA resolution at 60 frames per second was tested. 6. Conclusion an outlook Based on a new filter algorithm for edge detection in colour images with subpixel precision, a scalable and parallelizable architecture was developed and implemented on a Xilinx XC3S500E-5 FPGA device. The filter architecture has been presented and the implemented hardware design was evaluated with a scalable CMOS colour camera. By the use of a Xilinx Spartan III FPGA a cost effective preprocessing was realized, which reduces the computation tasks on the host computer. As visible in Figure 3, the filter produces an image, in which edges are represented as black lines. Figure 3. Comparison of an original colour image and the filtered edge image. For further optimizations of the filter architecture different attempts are applicable. For the next realization stage a filter computation with a higher processing clock is intended. Thus the clock delay can be minimized further. In order to adapt the filter better to the requirements of industrial metrology, an additional parameterization by the user is planned. The parameter for the scaling factor f is to be computed within a user defined area of interest (AOI). Thus edges, with special interest for a measurement task, can be strengthened. For realization, on the one hand, a delivery of the AOI coordinates from the measuring application to the hardware and, on the other hand, appropriate parameter fields for the filter must be implemented. In addition a parameterization of the safety factor s would be conceivable, in order to minimize disturbing environmental influences (lighting). Beyond that a new hardware platform, based on a Spartan 6 FPGA device (XC6SLX45), is planned. With the emigration on the new hardware the access to DSP48 slices is possible, which offers a better hardware support for filter operations. For further investigations to the attainable measuring accuracy at present studies with a new edge local criterion are accomplished. Acknowledgements The presented work is the result of the research within the project Qualimess which is situated at the Ilmenau University of Technology, Germany as part of the research support program InnoProfile, funded by the Federal Ministry of Education and Research (BMBF) Germany.

8 References [1] Marshall A and Sicuranza G L 2006 Advances in Nonlinear Signal and Image Processing; EURASIP Book series on Signal Processing And Communications (New York: Hindawi Publishing Corporation), p. 329 [2] Novak C L and Shafer S A 1987 Color edge detection; Proc. of DARPA Image Understanding Workshop, vol. 1 (Los Angeles, USA, February 1987) p [3] Kühn O 1997 Ein Beitrag zur hochauflösenden zweidimensionalen Geometriemessung mit CCD- Zeilensensoren dissertation, Ilmenau University of Technology, Faculty of Mechanical Engineering [4] Russ J C 1998 The image processing handbook 3rd Ed. (Berlin: Springer) pp [5] Bayer B E 1975 Color Imaging Array U.S. Patent No [6] Parulski K A 1985 Color filters and processing alternatives for one-chip cameras in IEEE T - Elec. Dev. ED-32 No. 8 pp [7] Correns M, Schumann M, Weissensee H, Rosenberger M, Schellhorn M and Linss G 2009 New demosaicing algorithm especially for measurement of geometries by image processing: Proc. IMEKO XIX World Congress - Fundamental and Applied Metrology (Lisbon, PT, 6-11 September 2009), pp [8] Ruzon M A and Tomasi C 2001 Edge, Junction, and Corner Detection Using Color Distributions; IEEE TPAMI, vol. 23, no. 11 (IEEE Computer Society) pp [9] Cumani A 1991 Edge Detection in Multispectral Images, GVGIP, vol. 53, no. 1 (Elsevier Science) pp [10] Drewniok C 1994 Multispectral Edge-Detection Some Experiments on Data from Landsat-TM, Int. J. Remote Sensing, vol. 15, no. 18 (Taylor & Francis) pp [11] Saber E, Telkap A M and Bozdagi G 1997 Fusion of Color and Edge Information for Improved Segmentation and Edge Linking, Image and Vision Computing, vol. 15, no. 10 (Elsevier) pp [12] Correns M and Brückner P 2008 Transformation of colour images for precision edge probing: Prosp. in mech. Engrg. / 53. IWK Ilmenau University of Technology (Ilmenau, DE, 8-12 September 2008) [13] Usbeck C and Brückner P 2000 Method for determining the edge position in color images, particularly for color and intensity transitions U.S. Patent No [14] Kilts S 2007 Advanced FPGA design: architecture, implementation, and optimization (Hoboken: John Wiley & Sons) pp [15] Xilinx 2008 Getting Started with FPGAs, (San Jose: Xilinx) [16] Xilinx 2005 Using Look-Up Tables as Distributed RAM in Spartan-3 Generation FPGAs, Application Note XAPP464 (San Jose: Xilinx)