3M Transportation Safety Division. Improved Daytime Detection Of Pavement Markings With Machine Vision Cameras

Transcription

1 3M Transportation Safety Division Improved Daytime Detection Of Pavement Markings With Machine Vision Cameras

2 Abstract Automotive machine vision camera systems commonly rely on edge detection schemes to locate pavement markings. These enable robust operation of advanced driver assistance system (ADAS) functions such as lane departure warning and lane keeping systems, well as autonomous driving functions. Edge detection algorithms commonly detect first-order gradients in pixel intensity, and binarize images so only pixels with gradients above a threshold value are retained. In this paper, we demonstrate that by increasing the luminance of a pavement marking over a range of illumination conditions, and by placing a contrast stripe adjacent to that pavement marking that has low luminance values under different illumination conditions, it is possible to increase both the Weber contrast of a marking and a representative contrast gradient based on a Sobel operator. A contrast stripe adjacent to a white marking reduces the dependence of the Sobel contrast gradient on illumination levels, and also enables higher Sobel contrast gradient values even for darker(e.g. soiled, aged) markings. If illumination and luminance of the markings result in a Weber contrast greater than ~4, the Sobel contrast gradient is relatively constant. We further demonstrate that widths of both pavement marking and contrast stripe may be selected for a given camera field-ofview, focal length, sensor size, and pixel density to optimize the pixel luminance values and gradients. We anticipate that appropriately sized pavement markings for a desired viewing distance with appropriate absolute luminance values and gradients will enable more robust lane detection. Keywords: Edge detection, Pavement marking, Contrast gradient, Machine vision Background Detection of pavement markings is critical to some current advanced driver assistance systems (ADAS) (e.g. lane departure warning and lane keeping systems) and autonomous driving systems like autopilot. If the detection is not accurate or robust, this can deleteriously affect the operation of these lane keeping systems (1,2). It is desirable to quantify the factors that influence robust detection of pavement markings with machine vision camera systems. Many common lane detection algorithms include calculation of gradients in pixel intensity for each point in the image, followed by application of a threshold so that only pixels where the gradient is greater than that threshold level are retained via binarization prior to feature extraction (3). This substantially decreases the complexity of data in an image, thereby reducing the computing load and processing time. Smaller gradients decrease the signal-to-noise ratio, making false negatives more likely. Similarly, if gradients are small enough not to be differentiable from other gradients that might exist in the image, then the artifacts from other features in the image might result in false positives. The magnitude of the contrast gradient is also a function of the ambient illumination. When illumination levels are low, less light is available to be reflected from light-colored surfaces. This decreases the gradient between light and dark regions of the image. Furthermore, this decreases the differentiation between gradients arising from other lighter regions of the image. When illumination levels are high and directed toward the camera, such as in glare conditions, surfaces that in other illumination conditions may have produced substantial gradients can have little or no gradient at all. In cases where the luminance of the substrate results in low contrast gradient values between markings and the substrate, longitudinal contrast striping can provide high contrast gradient values at the edges. Previous studies have investigated the effect of marking luminance on detection based on human factors (4) and machine vision systems (5,6), however, these studies did not evaluate results in the context of an edge detection scheme that related back to quantitative luminance measurements under different illumination conditions. This study correlates markings, contrast stripes, pavement substrates, and image and contrast gradient data obtained with a machine vision camera (with specifications similar to current automotive forward-facing camera) with calibrated luminance for the same surfaces obtained with a calibrated photometric camera. This correlation is investigated for a variety of pavement marking and substrates having different values of luminance (Y) over ranges of distance, illumination levels and the presence or absence of contrast stripe. This study further characterizes the effects of tape width and contrast tape width on the magnitude of the contrast gradient as a function of distance. The learnings of this study contribute to enabling design of pavement markings and contrast stripe of specific construction for optimized detection by an ADAS camera system with a given field-of-view, focal length, focal plane, sensor size and pixel density. Additionally, they will enable appropriate sizing of pavement marking tape and contrast stripe to achieve a desired viewing distance, thereby enabling more robust detection via edge detection at longer distances.

3 Experimental Pavement Marking Tape Preparation and Installation Pavement marking tape substrates were embossed with the same pattern as 3M 3M Stamark High Performance All Weather Tape Series 380AW and were slit to 4 wide and 10 yd long. Three samples were painted with Sherwin- Williams Bonding Primer and Sherwin-Williams exterior acrylic latex paint tinted to appropriate shades of gray to approximate target values of Y. When dry, the tapes were characterized for Y. Y, as defined in the CIE xyy color space, of embossed samples was measured according to ASTM D using a Hunterlab MiniScan EZ colorimeter (available from Hunter Associates Laboratory, Reston, Va.) with a 45 :0 illuminating and viewing geometry. After trimming edges, the tapes were installed on asphalt pavement so that the right edges of each tape were placed at 24 centers (Figure 1). One longitudinal edge of the pavement marking tape was immediately adjacent to the pavement substrate. The other longitudinal edge of the pavement marking was placed immediately adjacent to a 4 wide and 10 yd long black contrast tape (3M Stamark Contrast Tape 380AW-5, St. Paul MN). For comparison, experimental tape substrates with the same embossed pattern with no paint and with no paint and retroreflective optics also were installed in the same fashion, with black contrast on one side of the tape. FIGURE 1 Pavement marking tapes and contrast tapes on an asphalt substrate that were imaged in this study. Skips are 10 yd long and placed with centers at 20, 30, 40, 60, 80, and 100 m from the cameras. Photographs were taken facing northwest from intersection of the lines denoting sunset, solar noon and dawn on the map of the installation (Figure 2). The positions of sunrise and sunset are marked on the map for July 5, 2017, when many of the data reported in this paper were collected. FIGURE 2 Map of the installation with the orientation of the roadway on which the samples were installed, as well as the locations of sunrise, sunset and solar noon for July 5, Samples were aligned parallel with the major axis of the roadway.

4 Characterization of Pavement Substrates Y of the pavement substrate immediately adjacent to the tapes was measured with the same Hunterlab MiniScan EZ colorimeter and settings used for measurement of the tapes. Machine Vision Image Collection Machine vision images were collected with a Point Grey Grasshopper3 12 MP Color USB3 camera with lenses (Edmund Optics) that were 16 and 75 mm with 44 and 10 degree fields of view, respectively. Calibrated luminance measurements of pavement markings and of roadway substrates were performed with a Radiant Vision Systems Radiant ProMetric I-16 with a 200mm e-lens. The Point Grey camera was mounted 50 inches above the ground with a 1.8 downward tilt and a focus distance of 50 m, and the ProMetric was mounted 53 inches above the ground with a 2.8 downward tilt and a focus distance of 60 m. The cameras were housed in a shed with a transparent acrylic window, and positioned on the centerline of the array of pavement marking samples at a distance of 20 m from the centerpoint of first row of tape samples. A 1.0 neutral density filter was used. Images were taken with both cameras at intervals of five to fifteen minutes, depending on the day. Images taken with the Point Grey camera were set to be collected at full auto exposure settings. The algorithm for auto exposure automatically controls either or both the exposure and the gain to achieve a specific average image intensity (7). The ranges of allowed values of auto exposure, auto shutter, and auto gain were not limited. A 10 x 10 Spectralon panel from Spectra Vista Corporation (Poughkeepsie, NY) was placed flat on the ground at a distance of 15 m to provide white standards in the ProMetric image for calibration. Another set of images was collected for two separate concrete materials (Y = 34.8 and 17.5) that were placed in the sample array. The luminance of each concrete substrate was measured with the ProMetric camera and compared to the asphalt. Image Analysis Point Grey Machine Vision Images were uploaded into ImageJ (8) Images were converted from color into gray scale. For images collected with the 16 mm lens with a 44 degree horizontal field of view, the horizontal resolution and the vertical resolution were each cut in half by averaging every two pixels in the vertical and horizontal directions to down-convert 12 MP images to 3 MP images to more closely approximate the pixel density of conventional automotive machine vision cameras. Images were also collected with 75 mm lens with a 10 degree field of view; it was necessary to reduce the vertical and horizontal resolutions by a factor of eight. To remove image noise, a low pass filter was applied to the image with a Box Blur kernel of 1/9[ 1,1,1; 1,1,1; 1,1,1 ]. Next, a Sobel operator, an algorithm to detect edges, was applied to the image. The operator calculates the horizontal and vertical derivatives of pixel intensity, and then calculates the magnitude of the gradient in pixel intensity. The horizontal gradient, I/ x, or Gx, and vertical gradient, I/ y, or Gy, are calculated as follows, where A is the matrix representing the source image Gₓ = * A (1) Gᵧ = * A (2) And the magnitude of the gradient is given by G = (G²ₓ + G²ᵧ ) ½ (3)

5 The magnitude of the contrast gradient was calculated and recorded in a new matrix for each pixel in each image. For each tape interface, the maximum contrast gradient in each horizontal trace that intersected a marking was recorded, and these results were averaged along the length of the tape to calculate a mean value for the magnitude of the Sobel contrast gradient. Radiant Vision ProMetric Images were collected in TrueTest software (Radiant Vision Systems, Redmond, WA). Regions of interest corresponding to the tape surface, the contrast striping, and the pavement substrate were identified and the luminance was averaged over a range of 100 to 10,000 pixels, depending on the viewable area size of the marking. The focal plane was selected to be in the center of the samples at 60 m, and edges of the regions of interest were selected so as to have a sufficient margin between them and the edge of the features to mitigate focus distortion effects. A Weber contrast was calculated from the luminance results as follows: Weber Contrast = (I I b ) I b (4) where I is the intensity of the feature of interest and Ib is the intensity of the background. For the background, the luminance of either the black contrast tape or the pavement substrate were used in the calculation. Determination of Appropriate Pixel Density to Mitigate Focus Distortion Effects Images collected with the 75 mm, 10 degree field of view (FOV) lens illustrated that focus distortion effects created spurious trends in contrast versus distance if the pixel density of the image was too high, or by corollary, the pixel size too small. Focus distortion increasingly blurs the edges of the line as the measurement plane gets farther from the plane of focus. With small pixel size, the step change in intensity is small for each pixel in these blurred edges, and goes through a maximum at the focal plane. With a lower density image that has larger pixels that are still sufficiently small such that it is possible to fit at least one pixel entirely within the region of the image corresponding to the line, the magnitude of the Sobel contrast gradient is relatively constant for distances shorter than the focal plane distance and decreases at longer distances (Figure 3). FIGURE 3 Sobel contrast gradients calculated from images collected with the Point Grey camera with lenses with 10 and 44 degree FOV at a 50 m plane of focus. The 10 degree FOV image needed to be reduced by a factor of 64 before the pixels were large enough for the contrast gradient not to go through a maximum; the 44 degree FOV image only needed to be reduced by a factor of four to mitigate focus distortion.

6 Results Characterization of Tapes, Pavements Substrates and Calibration Panels Under Standard Conditions Y was measured for all surfaces investigated in this study (Table 1) after installation of the tapes. TABLE 1 Y of Samples Sample White Embossed 72.7 White Embossed with Optics 58.0 Light Gray Embossed 43.0 Medium Gray Embossed 28.3 Dark Gray Embossed 16.0 Contrast Tape Embossed x 10 Spectralon Asphalt Pavement 9.0 Light Concrete 34.8 Dark Concrete 17.5 Y Characterization of Tape Samples and Pavements Under Outdoor Conditions In ADAS and autonomous driving situations, distances other than the 30 m viewing distance of ASTM E a and the IS EN 1436 European Standard for Road Markings are relevant to lane detection and the reaction time assumed for autonomous driving systems. For highway traveling speeds, it is desirable to be able to obtain information from machine vision systems at distances greater than 50 m to enable reasonable reaction times at highway speeds. For this reason, this investigation evaluated gradient detection at a distance of m.luminance measurements were obtained with the ProMetric at observation distances of 20, 40, 60, 80 and 100 m and at various intervals throughout each day. Brighter substrates exhibit a large range in luminance throughout the day, whereas dark substrates exhibit little variability. At the extreme, the luminance of the black contrast tape is relatively constant over the bulk of the day, and shows less variability in luminance than even the dark asphalt surface. For example, the luminance of the asphalt decreased by 50% between 11:30 and 6:30 on July 5, 2017, from a maximum of 5000 cd/m2, and luminance of the black contrast tape decreases only by 15% from a maximum of 1800 over the same period. From the luminance data, the Weber constant for each tape and substrate was calculated assuming a background of either a black contrast tape or the substrate. The Weber constant for each case is linear as a function of Y, but the slope of the line is positively correlated with the level of illumination. The Weber contrast was calculated from the luminance data for each tape against both the black contrast tape and against the pavement substrate for each distance and type of tape for measurements collected from 1:30 and 6:30 pm July 5, 2017 as a function of the Y of the tape and distance (Figure 4).

7 FIGURE 4 Weber contrasts for tape relative to black contrast tape and relative to pavement collected from 1:30-6:30 pm July 5, 2017 as a function of distance and Y. Effect of the Variability of Substrate Luminance and Illumination on the Magnitude of the Weber Contrast The luminance from two concrete samples, asphalt pavement and tape was measured with the ProMetric camera under sunny and cloudy illumination to demonstrate the effect of Y of the substrate on the Weber contrast (Figure 5) for a white tape (Y = 58.0). The values of Y of the light concrete, dark concrete and asphalt were 34.8, 17.5, and 9.0, respectively. FIGURE 5 Weber contrast for Y=58.0 tape against concrete and asphalt substrates.

8 Characterization of the Magnitude of the Sobel Contrast Gradient of Tape Samples Against Contrast Tape and Asphalt Pavement Under Outdoor Conditions From images collected with the machine vision camera, the magnitude of the Sobel contrast gradient was calculated for each of the five tapes at the longitudinal edges adjacent to the pavement and to the black contrast stripe for each tape and for each distance throughout the day (Figure 6). These measurements were collected at the same distances and illumination levels as the luminance measurements collected with the ProMetric camera that were used to calculate the Weber contrasts. FIGURE 6 Mean magnitude of the Sobel contrast gradient values for tape relative to black contrast tape and relative to pavement collected from 1:30-6:30 pm July 5, 2017 as a function of distance and Y. Characterization of the Effect of Pavement Marking Width on Contrast Gradient of Tape Samples Against Asphalt Pavement Under Outdoor Conditions 10 yd segments of 4 wide 3M Stamark High Performance Tape 380I ES were applied to pavement in widths of 2, 4 and 6. Images were collected with the Point Grey camera from distances of 20 to 80 m with the 10 degree field of view lens and converted to MP. From these images, the magnitude of the Sobel contrast gradient was calculated for each of the three tapes at the longitudinal edges adjacent to the pavement for each distance (Figure 7).

9 FIGURE 7 Magnitude of Sobel contrast gradient as a function of distance and width of tape (10 degree horizontal field of view, MP). Characterization of the Effect of Contrast Tape Width on Contrast Gradient of Tape Samples Against Contrast Tape and Asphalt Pavement Under Outdoor Conditions 10 yd segments of 4 wide 3M 3M Stamark High Performance Tape 380I ES were applied to pavement. Black contrast pavement marking tapes (4, 3, 2, 1, and none) were applied along one longitudinal edge of each segment. Images were collected with the machine vision camera from a distance of 20 to 80 m with the 10 degree field of view lens and converted to MP. From these images, the contrast gradient was calculated for each of the five tapes at the longitudinal edges adjacent to the pavement and to the black contrast stripe for each distance Figure 8). FIGURE 8 Magnitudes of the Sobel contrast gradient as a function of distance and contrast tape width.

10 Discussion Variability in Luminance and Weber Contrast as a Function of Illumination, Y, and Distance Brighter substrates exhibit a large range in luminance throughout the day, whereas dark substrates exhibit little variability. Thus, dark substrates such as black contrast tape provide a largely invariant baseline for contrast calculations over the bulk of daytime hours. Also, the brighter the substrate, the larger the difference in luminance relative to a darker substrate, enabling higher contrast in low-illumination conditions. The range of values achieved for Weber contrast increases with Y, as illustrated in Figure 4. At any set distance, illumination and substrate, the Weber contrast increases with Y in a linear fashion. For a given Y, the distribution of Weber contrast drops to lower values with increasing distance. The same trends are observed for both the tape-asphalt interface and the tape-contrast tape interface, but the magnitude of the ranges of Weber contrast and maximum value of Weber contrast achieved are much larger for the tape-contrast tape interface. Variability in Contrast Gradient as a Function of Illumination, Y, and Distance The magnitude of the Sobel contrast gradient is relatively independent of distance provided enough pixels are available and focus distortion not an issue, as shown in Figure 6. Like the Weber contrast, the Sobel gradient increases with illumination levels. The magnitude of the gradient increases as a function of Y for the tape-asphalt and tape-contrast tape interfaces. The magnitude is greater for the tape-contrast tape interface, so much so that it reaches a plateau value of for the average value between 12:30 and 6:30 pm for tapes with Y By comparison, the largest average magnitude for the same interval for the tape-pavement interface is about 250. The use of contrast also allows for markings of substantially lower Y to achieve the same Sobel gradient, all else being equal. As shown in Figure 6, the mean values of the Sobel gradient for Y=72.7 marking on Y=9.0 asphalt are similar to those for Y=43.0 marking next to Y=3.4 contrast. If the contrast marking next to the Y=43.0 marking was abraded away, however, the Sobel contrast gradient would decrease by 20% if the Y of the marking stayed constant and more likely, would decrease by an even larger percentage as the marking Y decreased. For distances up to the focal plane where focus distortion is not an issue and where there are enough pixels to be fully within the edges of a pavement marking, the magnitude of the Sobel gradient increases linearly with the Weber contrast, and by extension, linearly with the Y, of the tape when the interface is a tape-black contrast tape interface. This linear increase in magnitude of the Sobel contrast gradient is observed up to a Weber contrast of ~4 and a distance of 60 m for the experimental camera and marking parameters detailed in this study, as illustrated for a 40 m distance (Figure 9). The line fits for the Sobel contrast gradient for Weber contrasts up to ~4 have R2 values that range from and standard errors that range from for distances of m. For distances beyond the focal plane, focus distortion and small pixel numbers both contribute to noise. Configurations with Weber contrasts greater than four (e.g. tapes with Y=58.0 and Y=72.7 bounded by contrast tape) exhibit a plateau in the magnitude of the Sobel contrast gradient at the tape-contrast tape interface. The existence of a plateau is potentially advantageous. Soiling, weathering, or abrasion of a pavement marking can reduce its value of Y, and if the initial value of Y is in this plateau region, it provides a buffer for darkening before the magnitude of the Sobel gradient starts to drop rapidly. For the tape-pavement interface, a somewhat different result is observed for distances shorter than the focal distance (Figure 10). For Weber contrasts less than four, the magnitude of the Sobel contrast gradient does generally increase for the population of all samples, but for each tape with a different value of Y, the magnitude goes through a maximum with respect to Weber contrast. Thus, this increases the standard error of this linear regime, and substantially reduces the monotonicity and linearity of the response. For example, at 40 m, the R2 value is 0.91, and the standard error is 52.1 for the line fit for Weber contrasts less than four. This comes about because the luminance values of both the tape and pavement are changing throughout the test interval, depending on the illumination, so different combinations of tape luminance and substrate luminance can result in the same Weber contrast, but different magnitudes of the Sobel gradient. This could contribute to errors in the line identification process if the magnitude is not large enough to survive thresholding, for example. For the tape-pavement interface, the plateau region was never observed, so any change in luminance of the marking should affect the magnitude of the Sobel contrast gradient. The Weber contrast value never exceeded four, even though the asphalt used in this study was quite dark (Y=9.0).

11 A marking with Y=58.0 exhibited Weber contrasts of (Figure 5). against two concrete samples (Y=34.8 and Y = 17.5) These values of Weber contrast translate to relatively low Sobel contrast gradients of ~ on Figs 9-10, suggesting that these edges may be more likely to be lost in binarization than an edge for that same marking against a contrast stripe (Figure 9). FIGURE 9 Magnitude of Sobel contrast gradient versus Weber Contrast for Tape-Black Contrast Tape at 40 m. For Weber Contrast < 4, R2 = 0.99 Standard Error =22.8. FIGURE 10 Magnitude of Sobel contrast gradient versus Weber Contrast for Tape-Asphalt at 40 m. For Weber Contrast < 4, R2 = 0.91, Standard Error = 52.1.

12 Effects of Tape Width and Contrast Tape Width on the Magnitude of the Sobel Contrast Gradient as a Function of Distance Increasing the width of the pavement marking tape increases the number of pixels in the region of interest for any given distance. Thus, it is possible to detect the marking at longer distances. The magnitude of the Sobel contrast gradient varies only 1-3% over a distance of 80 m for 6 pavement marking tape. By comparison, the magnitude of the Sobel contrast gradient for 2 pavement marking tape drops by 30% over a distance of 80 m (Figure 7). A similar argument holds true for the width of the contrast tape. For the camera specifications in this study, a contrast tape width of 2 is the threshold width above which the contrast gradient plateaus for distances of meters. Narrower contrast tapes result in much lower Sobel contrast gradient magnitudes except at the shortest distance of 20 m (Figure 8). Conclusions Edge detection is a common component of many lane detection algorithms in ADAS and autonomous driving systems. Robust edge detection relies upon contrast gradients that are determined with first-order derivative expressions (e.g. Sobel operator) and that have larger magnitudes than the binarization threshold. This study demonstrates that the magnitude of the Sobel contrast gradient increases with increasing Y of pavement markings, and is substantially higher for the interface between pavement marking and a black contrast stripe than it is for the interface between a pavement marking with the same Y value and a dark asphalt pavement substrate. For lighter colored pavement substrates, the magnitude of the Sobel contrast gradient is smaller yet, increasing the value of using contrast marking. Read case study: Michigan DOT test of new 3M TM Connected Roads Prototype Solutions Pavement marking tapes with high values of Y and black contrast exhibit plateau behavior over a wide range of illumination conditions that result in Weber contrasts above a threshold level (~4 for the camera specifications used in this study), and a linear dependence on Weber contrast for Weber contrast below that threshold. The plateau in contrast gradient provides a buffer for real-world soiling or degradation of the pavement marking before substantial reduction in the magnitude of the Sobel contrast gradient occurs. Without a contrast stripe, no plateau is observed. For these systems, all changes in Y of the pavement marking will immediately affect the magnitude of the Sobel contrast gradient. Wider pavement markings exhibit less decrease of the Sobel contrast gradient with increasing distance. This result may be critical for pavement markings designed for detection at longer distances that enable longer reaction times. There is a threshold level of the width of the black contrast stripe, and for widths larger than this threshold, the magnitude of the Sobel contrast gradient reaches a plateau over a wide range of distances. For the camera specifications and markings used in this study, the threshold width of contrast was 2. The Weber contrast and the Sobel contrast gradient depends on both the Y of the marking and contrast stripe, as well as the illumination level. A marking with a high Y adjacent to a contrast stripe with a low, minimally varying Y results in the best Sobel contrast gradient in both high light and low light conditions. This improves the likelihood that that edge will survive binarization and be accurately detected. Learn more about 3M Pavement Markings Ask a Question

13 References 1. Young, A. Self-driving cars vs. American roads: Will infrastructure speed bumps slow down the future of transportation? Salon, Apr. 20, Accessed July 31, United States House of Representatives Committee on Transportation and Infrastructure, 113th Congress. How autonomous vehicles will shape the future of surface transportation. Nov. 19, CHRG-113hhrg85609/html/CHRG-113hhrg85609.htm. Accessed July 31, Narote, S.,Bhujbal, P., Narote, A., and Dhaned, D. A review of recent advances in lane detection and departure warning system. Pattern Recognition, Vol. 73, 2018, pp Wang, J.-H. and Cao, Y. Effects of Road Marking Luminance Contrast on Driving Safety. Publication FHWA-RIDOT- RTD Rhode Island Department of Transportation, May Marking_Luminance.pdf Accessed July 31, Carlson, P. Road Markings for Machine Vision. NCHRP Project (6). TAMMU Technology Conference, May Davies, C. Pavement Markings Guiding Autonomous Vehicles A Real World Study. Proceedings of the Automated Vehicles Symposium 2016, Breakout 20: Physical Infrastructure, Work Zones, and Digital Infrastructure, July Accessed July 31, Technical Reference: FLIR Grasshopper 3 USB3 Vision. Accessed July 29, Accessed July 26, 2017.

14 Transportation Safety Division 3M Center, Building 225-4N-14 St. Paul, MN Please recycle. Printed in USA. 3M All rights reserved.