SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 1. Localization and Recognition of Trac Signs. M.M. Zadeh, T. Kasvand, C.Y.

Transcription

1 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 1 Localization and Recognition of Trac Signs for Automated Vehicle Control Systems M.M. Zadeh, T. Kasvand, C.Y. Suen Concordia University, Computer Science Department, CENPARMI, GM De Maisonneuve Blvd west, Montreal, PQ, H3G 1M8, Canada phone: (514) zadeh@cenparmi.concordia.ca, mahmoud.zadeh@space.gc.ca Abstract We present a computer vision system for detection and recognition of trac signs. Such systems are required to assist drivers and for guidance and control of autonomous vehicles on roads and city streets. For experiments we use sequences of digitized photographs (to be replaced by video tapes taken from a moving car) and o-line analysis. The system contains four stages. First, region segmentation based on colour pixel classication called SRSM (Supervised Region Segmentation Method). SRSM limits the search to regions of interest in the scene (image). Second, we use edge tracing to nd parts of outer edges of signs which are circular or straight, corresponding to the geometrical shapes of trac signs (circle, triangle, rectangle, octagon). The third step is geometrical analysis of the outer edge and preliminary recognition of each candidate region, which may be a potential trac sign. The nal step in recognition uses colour combinations within each region and model matching. This system may be used for recognition of other types of objects, provided that the geometrical shape and colour content remain reasonably constant. The method is reliable, easy to implement, and fast. This diers from the road signs recognition method in the PROMETEUS (EU45), [1]. The overall structure of the approach is sketched in Fig. 1. keywords: Segmentation Keywords: Computer Vision System, Road sign Recognition, Coloured Geometrical Region

2 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 2 I. Introduction Due to the ever increasing number of cars and trucks on the roads, with resultant trac jams, delays, and accidents, in Europe a large and well-funded research project (called PROMETEUS) was planned in 1986 and the research started in This was a "pre-competitive" project involving most car manufacturers in Europe and several universities. Automatic recognition of highway signs was but one part of the PROMETEUS project. Meanwhile, similar projects have been undertaken in USA and Japan. Due to the large number of participants and a certain condentiality, it is dicult to evaluate what exactly has been achieved [1] to [7]. We were asked by CRIM (Centre de recherche informatique de Montreal) to evaluate the possibilities of recognizing trac signs on city streets rather than on highways. The design of trac signs is standardized for size, shapes, and colour compositions. Some examples are shown in Fig. 2. Furthermore, the signs are placed in visible and fairly predictable locations and orientations relative to the view-point of the driver (but it requires "scene understanding" to make realistic use of this information). However, trac signs form a very unique and easily visible set of objects in a road or street scene. The only signicant variables are the sizes of the signs as they appear in images (due to distance), and illumination of the scene (bright sunlight, overcast, fog, at night). Our study has been limited to street scenes during daylight hours, for signs both in direct sunlight and in shadows. The approach consists of preliminary segmentation of the image to limit the analysis only to regions which are potential road signs, followed by renement of segmentation and increasingly detailed recognition as more and more information becomes available from image analysis. A. Region Segmentation Based on SRSM The rst and most dicult step in automatic recognition of trac signs is the detection of likely regions which might be trac signs in the images of street scenes. Recognition is greatly simplied after a "fair" segmentation of the image. Fortunately, in the present case the objects of interest (trac signs in an image) consist of compact regions with a very limited set of shapes, colours, and colour combinations. However, illumination can vary and the purity of the colours is degraded by weathering ("wear and tear") of the signs. In cities the signs are often in clusters and a sign may be partly in direct light and partly in shadow (due to buildings, trees, posts, wires, etc.). Due to distance, the size of a sign may vary from a few pixels to a rather large region in the image. Of the three major alternatives to segmentation, the rst is an improved version of the classical region splitting and merging method described in [8]. The second is pixel classication based on colours. We use this method for preliminary classication to detect potential regions which could contain or be

3 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 3 trac signs. The third is edge based segmentation, which produces a profusion of edge pieces, but which we only use for rening the potential regions for trac signs. Each colour in an image is given by three independent values (red, green, and blue, or RGB) at every pixel, which are the additive primary colours. The colour of each pixel may be plotted as a point in the (Euclidean) RGB space, see Fig. 3a. TV cameras vary in their abilities to capture the RGB values (of a pixel), and these values are highly dependent on the illumination levels in the scene (and to a much lesser degree on the orientation of the light and the view angle with respect to the sign). Hence, the points (pixel values) in the RGB space do not stay xed even though the colours of trac signs are xed. One alternative is to attempt to transform the RGB space (for example, to HSI space of hue, saturation, and intensity), to try to have better control over the variations in pixel values for the same colour. Another is to determine the nature of the variations and to create sub-spaces in RGB which enclose most of the variations for each colour in the image of each trac sign. We chose the latter approach and found that the variations tend to cluster around and along fairly straight lines from R,G,B =0,0,0 (black) to the combination of primary colours dening each colour used on trac signs. The sub-spaces could be approximated reasonably well by conical regions in RGB space, see Fig. 3b. Black and white "colours" require dierently shaped sub-spaces. B. Teaching colours to the computer The colours on trac signs are standardized, except for variations due to illumination and weathering. The classical approach to colour classication would require numerous samples of (pixel values for) each colour used in trac signs, followed by clustering procedures or statistical methods. However, since each type of TV camera used to record images from street scenes can modify the colours unpredictably, we chose to experiment with an interactive approach. We select a few typical images (from the image data base) which contain trac signs under dierent illumination conditions. On the display we select ("point to") typical sample pixels for the colour, and we create an analytical description of the subspace (cylinder, cone, spherical section) in RGB space from the obtained samples. Minimally, even two colour samples may suce, namely, one for the colour in shadow and the other for the same colour in sunlight. Optimal speed in pixel classication would be obtained if we could treat the RGB space as a look-up table (of 256x256x256=16,777,216 bytes). Correct pixel classication is the most sensitive step in the process and a certain minimal quality in colour response from a TV camera is required.

4 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 4 C. Processing sequence After on-line interactive teaching has been completed, we expect to be able to process sequences of images taken by the same TV camera. (At present we do not have a TV camera of sucient quality, and the image recording equipment, to verify this claim). C.1 Pixel classication Each pixel in the image is classied. If the pixel falls into one of the sub-spaces dened as "trac sign colours", a corresponding colour code (class) is recorded in the output image. Pixels not belonging to any colour class become zeroes (0's). The output image is either an image of colour class codes or we create one binary (0,1) image for each colour class. Each pixel in the output image is still an isolated entity, whether the pixel is 0 or a colour class code. C.2 Pre-processing and preliminary segmentation The image of colour codes from pixel classication contains several types of noise. Isolated noise pixels and very small regions can be removed by rather "mild" erosion and subsequent dilatation and some logic. These operations have to be "mild" to preserve small (distant) trac signs and narrow coloured regions within the signs. This is followed by region labelling to assign unique identity codes (labels) to each isolated but connected group of pixels of the same colour. Each labelled region represents and isolated region of a desired colour. Individual labelled regions, and adjacent (touching) regions of known colours (which are now known), are potential regions for trac signs, depending on the colour combinations on a given sign. However, even though each labelled region, or a combination of regions, represents a probable trac sign, a region may be partly "camouaged" by similar colour in the background or foreground of the sign. Thus, "something else" becomes attached to the region containing a likely road sign. The region may also be "broken" due to low spatial resolution (and noise cleaning). For example, a red "don't enter" sign can become two red "half-moons" face to face. The next step is to remove the parts of regions which are unlikely to belong to a road sign, using geometrical analysis of edges (contours). The analysis is now concentrated onto the labelled regions. C.3 Edge tracking and circle and straight edge detection Trac signs have pre-dened and known shapes. The outer edges of trac signs are either circular or consist of sections of straight lines (triangle, rectangle, octagon, but ve, six, and seven sided signs are not used). Thus, the outer edge of a trac sign is smoothly curved (circular) or it consists of a piece-wise sequence of linear sections. The angles between the linear sections are known and, except for the rectangle, the length ratios between sides (and their slopes) are also known.

5 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 5 The additions to a region of a potential trac sign (a colour class) was caused by similar colour either in the foreground or background of a part of the sign. The outer edges of the camouaging regions are often "ragged" and do not obey the regularity expected of trac signs. By tracking the outer edge of a region and subsequent analysis of the edge, we can detect and separate circular and straight sections, and remove the ragged parts of the edges. Similar logic applies to the outer edge of regions that may be missing from the sign due to something in front of the sign. The insides of potential regions for signs that are fully camouaged by similar background colour have not been investigated. C.4 Geometrical Analysis After tracking the outer edge of a sign and removal of unlikely portions of the edge, the edge has also been split into circular and straight sections. New information is now available. For example, if a signicant part of the outer edge is circular, we have probably detected a circular trac sign. Likewise, if straight edges obey the rules for trac signs (angles between and length ratios of straight sections), the region is likely to be a sign of a given type. In most cases we can now recreate the shape of the expected trac sign, which becomes an improved potential region. This results in the preliminary recognition of groups of trac signs of similar shape. Since the colour combinations inside the region are known from classication, a fairly good estimate can be made which trac signs the region may represent. D. Model Matching The outer edge of a region is in contact with the background of pixels which do not belong to any of the trac sign colours. If portions of the outer edges have been camouaged, we have reasonable estimates of which sections of the edge belong to a potential trac sign, and we can recreate to a reasonable certainty the missing portions of the outer edge. The analysis of the outer edge is rotation and size invariant. The interior of the potential region (trac sign) is now known, we know which colours occur within this region, and we know which colour is adjacent to which other colour. The size and orientation of the region is also known. To further reduce the choice of which trac sign model to match to the found region, we compute the area (pixel counts) ratios for each colour within the region. This test is also invariant to rotation and size of the region. However, since there are dierent trac signs with the same area colour ratios (for example, a "turn left" or "turn right" arrow inside a circle), we split the region horizontally and vertically for individual area colour ratios. These tests depend on the rotation of the trac sign. To further verify the recognition, we can scale and match a model of the expected trac sign to the potential region in the image, or use neural nets.

6 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 6 II. Conclusions These investigations have shown that the described sequence of processing steps is robust and gives very good results, provided that the trac sign in the image is not too small. The total number of images of street scenes is about 60, and the number of dierent trac signs is rather small (stop, yield, turn, no parking, one way, and a few more). Hence, statistical estimates of recognition accuracies are premature. A. Example A cluster of trac signs, obtained from Germany via the Internet, is shown in Fig. 4 and Fig. 5. Note that there is a shadow across the sign cluster, and the background can act as "camouage" for the red colour. The red coloured edge on the triangular (yield) sign was used for teaching. Only two samples were taken, (R,G,B) =(190,32,32) and (R,G,B) =(222,60, 56), and used to construct a cylindrical sub-space in RGB. Using this subspace, the red pixels in the same image were classied, see Fig. 5b. Note that a "camouage" eect occurred where parts of a red roof in the background became attached to the red coloured region in the yield sign, since the subspace was rather large. Since the outer edges of the roof region are rather ragged, the camouage eect was easily removed. Fig. 5c shows the outer edges of red regions after circular and linear segments are retained. This subspace for red was also used to detect the red colours in a street scene form Montreal (Fig. 6a), which was rst photographed and the paper print was digitized with a page scanner. The results of pixel classication before noise removal is shown in Fig. 6b. Fig. 6c shows the results after noise removal and detection of circular and straight line segments on the outer edges of each region. The geometrical analysis of each region "looked for" triangle, rectangle, octagon, and circle. Only a circular region was found which represented the NO-PARKING sign. The region was split into four quadrants (1,2,3,4) and the pixel count (area) and percentage of each colour was determined. The results on this sign are: First quadrant; area =108 pixels, percent red =62%, percent white =33%, percent black =6%. For the second quadrant; 120, 72%, 27%, 1% (area, red, white, black). For the third and forth quadrants; 120, 52%, 48%, 1%, and 108, 68%, 32%, 0%. Fig. 7 shows another sample by recognition of an (octagon) STOP sign. References [1] L. Priese, J. Klieber, R. Lakmann, V. Rehmann, R. Schian, "Echtzeit-Verkehrzeichenerkennung mit dem Color Structure Code", Labor Bilderkennen, Universitaet Koblenz-Landau,Reinau 1, D Koblenz, Germany, [2] L. Priese, et al, " New Results on Trac Sign Recognition", University of Koblenz-Landau Germany [3] S. Estable, et al, "A Real-Time Trac Sign Recognition System", Daimler-Benz Research Centre, Stuttgart and Ulm, Germany, and CNRS, F-63177, Aubiere, France

7 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 7 [4] C. Baghdassarian, et al,"recognition of arrows in the environment of road markings", Ecole Nationale Superieure des Mines de Paris, France [5] G. Piccioli, et al,"robust road sign detection and recognition from image sequences", Univesita di Genoa, Italy [6] F. Stein and G. Medioni, "Pami, Structural Indexing: Ecient 2D Object Recognition", vol. 14/ 2, Dec. 1992, USA [7] H. Akatsuka and S. Imai, "Road Sign-posts Recognition System"' Nippondeso Co. Ltd, Japan [8] Y. Li, C. Saldanha, M. Lalonde, "Geomodelling: Georeferencing Real World Objects", CRIM,Vision Interface '96, May 1996, Toronto, Canada.

8 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 8 Teaching of Computer sample Images Sequence of Images Installation Processing Region Segmentation Based on SRSM: Color Classification Based on SRSM Noise Removal Labelling Process Edge Extractor: Polygonal Approximation Straight-Curve- Line Detector Edge Tracking Geometrical analysis: Shape Definition Model Matching Shape Identification Sign Classification: Road Sign Recognition Looking in data base Color analysis of signs Figure 1: Road Sign Recognition Based on SRSM

9 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 9 Figure 2: Some examples of traffic signs

10 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 10 B P_bg b P β r ϕ g G R P_rg Figure 3a: Pixel value as point in RGB space B defined cluster W (255,255,255) WHITE * * * * : Some samples by teaching of computer B (0,0,0) BLACK G R Figure 3b: A conical subspace representing one colour

11 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 11 Inputs: _ Original images _ RGB-Value of two points Red Color B RGB-Space Color Classifier G R Noise Removal Errosion & Dilatation Labelling of Connected Region Labelling Process Edge Detector Edge Tracking Recognition of Road Sign Fig. 4: Illustrated sketch of operations

12 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 12 Fig. 5a: Original image (in colour) Fig. 5b: After colour classification Fig. 5c: After linear/circular segmentation and model matching

13 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 13 Fig. 6a: Original image (in colour) Fig. 6b: After colour classification Fig. 6c: After linear/circular model matching

14 SUBMITTED TO SPIE'S INTELLIGENT SYSTEM AND AUTOMATED MANUFACTURING 14 Fig. 7a: Original image (in colour) Fig. 7b: After colour classification Fig. 7c: After linear/circular model matching