Challenges and Solutions for Bundling Multiple DAS Applications on a Single Hardware platform.

Transcription

1 Challenges and Solutions for Bundling Multiple DAS Applications on a Single Hardware platform. Gideon P. Stein Itay Gat Gaby Hayon MobileEye Vision Technologies Ltd. Jerusalem, Israel. gideon.stein@mobileye.com Abstract: This paper addresses the key challenges in bundling multiple camera based DAS applications onto the same hardware platform. In particular, we discuss combinations of lane departure warning (LDW), Automatic High-beam Control (AHC), traffic sign recognition (TSR) and forward collision warning (FCW). The advantages of bundling are in cost reduction and that it allows more functions to be added to the car without increasing the footprint on the car windshield. The challenge in bundling is that the different applications traditionally have different requirements from the image sensor and optics. We show how algorithms can be modified so that they can all work together by relying less the particular physics of the camera and making more use of advanced pattern recognition techniques. This shift in algorithm paradigm means an increase in computational requirement. The introduction of new automotive qualified, high performance vision processors makes these new algorithms both viable and affordable paving the way to bundles of application running on the same platform. Keywords: Driver assistance systems, camera, vision 1 Introduction In the last few years we have started to see camera based driver assistance systems (DAS) entering the market: lane departure warning (LDW), Automatic High-beam Control (AHC), traffic sign recognition (TSR) and forward collision warning (FCW) to name a few. The key challenge now, apart from expanding the performance envelop of each function, is the bundling of multiple applications on the same hardware platform. The advantage of bundling is in cost reduction but more importantly, it allows more functions to be added to the car without increasing the footprint on the car windshield. The difficulty in bundling is that the different applications traditionally have different requirements from the image sensor and optics. For example, traditional AHC makes significant use of color information and thus requires a color sensor, while lane detection and traffic sign recognition require the extra sensitivity of the monochrome sensor for low light conditions. In order to overcome these conflicts new algorithms need to be developed which put more focus on the computation and less on the specific physics of the sensor. In particular, we introduce a new AHC algorithm that uses a higher resolution image sensor and requires only very weak color information. This then allows the same sensor to be naturally integrated with LDW, TSR and vehicle detection applications where mostly shape and pattern recognition come into play with some color information giving a more measured boost in performance. This shift in algorithm paradigm, however, means an increase in the computational requirement. The introduction of new automotive qualified, high performance vision processors makes these new algorithms both viable and affordable paving the way to bundles of application running on the same platform. To more accurately follow the design process this paper should first describe the four basic applications (LDW, FCW, TSR and AHC) and the lens and imager requirements of each individual application. One would then show how the requirement spaces intersect leading to possible solutions that fulfill all the requirements. However, it is often easier to use a concrete example, so this paper reverses the process and the next section (section 2) describes a particular camera (imager and lens) to be used in the discussion. Section 3 describes each application in detail and why this camera configuration is acceptable. The major change was in AHC where a totally new algorithm was developed to work with this camera (section 3.4). Section 4 describes the camera exposure control concept which is based on multiple exposures. We conclude with section 5 where we review the Mobileye EyeQ chip family which was developed to support multiple DAS applications. Due to space and time constraints this paper focuses on the nighttime performance of the application bun-

2 dle. It is the more challenging case since it must include AHC. 2.1 The Sensor 2 The Camera Solution The example sensor we use for discussion is a wide VGA sensor with resolution This is the highest resolution sensor available today in an automotive qualified part whose specifications are publicly known. The sensor has a 6µm 6µm square pixel. It has a global shutter and can switch exposure and gain settings every frame. This is key for the ability to use the multi-exposure approach described in section 2.3 and in more detail in section 4. One unusual aspect of the proposed sensor is that it has only red and clear pixels. The red pixels are arranged as they would be on a standard Bayer pattern but the green and blue filters are missing. This is done in order to combine most of the advantages of the monochrome sensor in terms of low light sensitivity and imager resolution with some color information; in particular, the ability to distinguish between red and white lights. 2.2 The Lens The lens example discussed is a 5.7mm lens with a low F number (for example the Marshall 5.7mm F1.6). Some sort of IR cutoff filter is required in order to give a consistent response with a variety of windshields whose transmission characteristics in the IR spectrum can vary widely. The filter selected is an IR cutoff filter with the 50% cutoff set at 700nm rather than the typical 650nm. This filter lets in more light and increases the low light performance in detection of lanes and traffic signs under halogen headlights in particular and also increases the detection range for red taillights. The sensor/lens configuration gives a FOV of 0.06 o per pixel or 45 o x29 o over the whole sensor. However due to assembly tolerances we will only assume that pixels are available giving an effective FOV of 39 o 24 o. Once the sensor and lens have been fixed it is often convenient to specify the lens focal length (F) in pixels. In this case: F = 950pixels (1) Camera Mounting in the Vehicle In this example the camera orientation is centered around the horizon. This was a compromise since the TSR might prefer a small upward tilt for better detection of overhead signs. This would be unacceptable for LDW. The camera height is 1.2m. 2.3 Multiple Exposures No single gain/exposure settings works well for all applications and in fact some applications (such as AHC and TSR) require more than one gain/exposure setting on their own. The solution is to sequence through a set of gain/exposure settings. There are three frames captured in sequence with different gain/exposures in each. A complete set of three frames is captured in 66msec. See section 2.3 for details. 3 Review of the Individual DAS Applications This section reviews the basic performance requirements of the four DAS applications and the imager requirements that are derived from desired performance. It then shows how the suggested camera manages to match these requirements. 3.1 Lane Departure Warning The lane departure warning system is designed to give a warning in the case of unintentional lane departure [1]. The warning is given when the vehicle crosses or is about to cross the lane marker. Driver intention is determined based on use of turn signals, change in steering wheel angle, vehicle speed and brake activation. There are various LDW systems available. The Mobileye algorithm is predictive in that it computes time to lane crossing (TLC) based on change in lane to wheel distance and warns when the time to lane crossing is below a certain threshold. Other algorithms give a warning if the wheel is inside a certain zone around the lane marker. In either case, the core of the algorithm is the lane detection algorithm. The lanes markers are detected in the camera image and then, given the known camera geometry and it s location relative to the vehicle, the position of the vehicle relative to the lane is computed. This information is then collected over time, often using the Kalman filter. Wheel to lane marker distance must be given with an accuracy of better than ±5cm. With forward looking camera this distance is not observed directly but is extrapolated from the forward view. The closer we observe the road markings the less we have to extrapolate and the more accurate our estimates will be, especially on curves. Due to the car hood and the location of the camera, the road is seldom visible closer than 6m in front of the wheels. In some cars, with longer hoods, this distance is even greater. The camera orientation and FOV also have a contributing factor. In order to see the road 6m in front of the vehicle using a camera mounted at 1.2m height

3 the camera vertical FOV must be: α = arctan( ) = 11.3o (2) below the horizon. In our case the camera vertical FOV below the horizon is 12 o so the closest visible point on the road is at 5.7m. This means the limiting factor is most likely to be the car hood not the camera specifications.. The lane departure warning system must work on sharp curves (with radius down to 125m). With a horizontal FOV of 39 o the inner lane markers will still be visible on such curves. The output of the lane detection module can also be used to support other applications. For example it is used for detection the current in-path vehicle (CIPV) for headway monitoring and FCW. In order to correctly perform this lane assignment on curves we need to detect lane markings at 50m and beyond. With the proposed camera, a lane mark of width 0.1m will be just under two pixels wide at 50m and can be detected accurately. The expectation from the system is greater than 99% availability when lane markings are visible. This is particularly challenging in low light conditions when the lane markings are not freshly painted (have low contrast with the road) and the only light source is the car halogen headlights. In these conditions the lane markings are only visible using the higher sensitivity of the clear pixels (i.e. using a monochrome sensor or the proposed red/clear sensor). Note: with the more powerful Xenon HID headlights it is possible to use a standard RGB sensor in most low light conditions. 3.2 Forward Collision Warning The core technology behind Forward Collision Warning (FCW) and headway distance monitoring is vehicle detection. To reliably detect vehicles in a single image the system currently requires that a vehicle be 13 pixels width. For a car of width 1.6m, the proposed camera gives initial detection at 115m and multi-frame approval at 100m. A narrower FOV will give a greater detection range however it will reduce the ability to detect passing and cutting-in vehicles. The FOV of around 40 o was found to be almost optimal given the sensor resolution and dimensions. A key component of the FCW algorithm [2] is the estimation of distance from a single camera as in [3] and the estimation of scale change from which one can determine the time to contact. Computing either value from a Bayer pattern mosaic would be difficult since no row or column is of uniform color. However since the red/clear sensor has every second row and column of uniform color (in this case clear) it is possible to find and track vertical and horizontal edges accurately. 3.3 Traffic Sign Recognition The TSR module is designed to detect all speed limit signs and end-of-speed limit signs on highways, country roads and urban settings. In addition iconic supplemental signs (e.g. rain, on exit (arrow), trailer) must be detected and identified. The signs can be of the regular (fixed) type at the sid of the road or the electronic overhead signs. Partially occluded, slightly twisted and rotated signs must also be detected. It is important that the system ignore the following signs: signs on truck/buses, exit road numbers, minimum speed signs, embedded signs. The TSR module which focuses on speed limit signs does not have a specific detection range requirement. These signs only needed to be detected before they leave the image. The most difficult case is to detect a 0.8m diameter sign on the side of the road when the vehicle is driving in the center lane of a three lane highway. In this case lateral distance between the sign and the vehicle centerline can reach 10m. In the above case, the sign will leave the edge of the image when it is: w = = 26pixels (3) 10 This translates to a detection at distance of: Z = FW w = = 29m. (4) Within the time of one frame the car can travel 7m. Therefore, in the worst case, the sign is last completely in the image, at a distance of 36m and when it is 21 pixels in diameter. An example is shown in figure 1c. This presents a challenge in terms of imager resolution. The numbers in the speed sign are confined to a central region which is half the diameter of the sign or about 11 pixels in this worst case. For speed signs which can include up to three digits this is clearly pushing the limits of OCR technology [4]. Superresolution techniques using prior models have been suggested [5] to improve the image resolution. We follow that approach and implicitly apply constraints in the training data due to the limited class of objects that must be identified. Tests show that reliable detection is possible with a red/clear sensor for signs down to 18 pixels in outer diameter (figure 1b). It is clear however that monochrome imagers will perform significantly better than Bayer pattern color sensors with the same number of pixels since Bayer mosaics have only 50% of the pixels of uniform response. The red/clear sensor is a compromise, where 75% of the pixels are of uniform response. However, the impact of using a red/clear sensor was found to be minimal when compare to the full monochrome imager. Recognition rates by the classifier dropped by only a fraction of one percent when, in tests, the value

4 (a) (a) (c) (d) Figure 1: (a) A 0.8m diameter sign first recognized at 50m when it is 15 pixels in the image. Robust recognition at 40m when it is 19 pixels in diameter. (c) Sign leaves the image at 36m when it is 21 pixel in diameter. (d) example of an electronic overhead sign for one out four pixels of a monochrome sensor was replaced by interpolation of its eight-way connected neighbors. Another significant reason why monochrome and red/clear imagers are better than RGB imagers is improved low light performance. At night there is a limit on the allowable camera exposure so as to avoid motion blur. However the sign must also be bright enough to be readable. This topic is described in more detail in the context of multiple exposures (section 4) Vehicle Detection Support for TSR Having multiple applications running on the same ECU does not just create problems. It sometimes provides solutions. A particularly difficult problem in TSR is posed by the speed limit signs on the backs of trucks and buses. They look like regular speed signs at a distance and since the relative speed to such a Figure 2: (a) The speed limit sign on the truck is recognized but ignored (marked in green). The circular shape of a tree behind a truck is also an initial candidate. All the speed limit signs on the truck are ignored. The 90 KMH speed limit sign is recognized (marked in blue). sign is small their size also increases slowly like a distant sign. So TSR by itself could not solve the problem. The solution was found by using the vehicle detection module. A traffic sign that was detected in the back of a vehicle was considered invalid. An example can be seen in figure Automatic High-beam Control A typical automatic high-beam control (AHC) system detects the following conditions and switches to low beams: Headlights of oncoming vehicles Taillights of preceding vehicles Street lights or ambient light indicating that high beams are not required Vehicle speed The host vehicle lights are switched back to high beams when non of these conditions exist (often after a specific grace period). One of the key challenges is to avoid false positive detection on reflections from street signs and reflectors and on stationary lights on distant buildings. One approach [6] is to compare images from two sensors: one with a red filter and the second with a cyan filter. The latter will respond only to non-red light sources and will give zero response to red light. By comparing corresponding pixels from the two imagers one

5 can detect the color of the lights source. The number of pixels of each color above a certain intensity are counted and if the count is above a threshold the systems switches to low beams. The use of low resolution imagers precludes the use of this system for any other application. A second approach [7] uses an RGB sensor to give better color differentiation and typical light sources can be located in the full CIE color space. This approach would be able to tell the difference between green, yellow and red lights. A powerful green traffic light will not be confused as an oncoming vehicle. Since a single sensor with a Bayer pattern mosaic is used the lens is defocused so as to spread a light source over multiple pixels. The use of the color mosaic reduces both the effective imager resolution (by 50%) and the low light response (to less than one third). This precludes the use of the same sensor for TSR or LDW. Given that the three other applications already require a high resolution monochrome sensor, a new AHC algorithm was developed that was primarily monochrome but makes use of the high resolution imager that is available. In other words, sophisticated pattern recognition techniques are used on the higher resolution monochrome imagers to identify light sources instead of relying on color information. Some color information is available from the red clear sensor but only when the light source is at a medium to short distance and covers a large enough area in the image (typically more than 16 pixels). A detailed description of the AHC algorithm is beyond the scope of this paper however the algorithm highlights are as follows: Detect bright spots in the sub-sampled long exposure image and then perform clustering and classification in the full resolution image. Classify spots based on brightness, edge shape, internal texture. Get further brightness information from the short exposure frames and classify obvious oncoming headlights based on size and brightness. Track spots over time and compute change in size and brightness. Pair up matching spots based on similarity of shape, brightness and motion. Classify pairs as oncoming or taillights based on distance, brightness and color, and estimate distance. Unmatched spots might be motorcycles taillights. For spots above a threshold size and brightness, search for other supporting evidence such as red color and the patch of brightness where the road in illuminated by the motorcycle headlights. Figure 3a shows a typical highway scene with oncoming vehicles, preceeding vehicles, traffic signs and reflectors. In the long exposure frame, even the reflections off the signs on the right can appear as saturated pixels. Figures 3b, 3c and 3d show some characteristic differences between taillights and reflectors which can be used for classification: Taillights have a smooth roll-off in brightness at the edges (Figure 3c) as opposed to the sharp edges on the rectangular sign (Figure 3d). The brightness of the circular sign drops off gradually like a light source but in this case it is due to the dark ring of a speed sign. However any light source which is that large should be much brighter. (See the oncoming lights figure 3b for example). The red pixels on the circular signs are darker than the saturated clear pixels further indicating that it is not a red light. The two reflectors in figure 3c have sharp edges. The two reflectors in figure 3d are not at the same height and their size/distance ratio does not match a taillight pair so they cannot be a pair of taillights. Figure 4 gives an example how the red pixels can be used to determine the color of a light source. 4 Multiple Exposures No single gain/exposure setting works well for all applications and in fact, some applications (such as AHC and TSR) require more than one gain/exposure setting on their own. The solution is to sequence through a set of gain/exposure settings. There are three frames captured in sequence with different gain/exposures in each. A complete set of three frames is captured in 66msec. Figure 5 shows a example of such a frame sequence. Figure 5a is a long exposure frame optimized for LDW and FCW. It has long exposure and low gain to give strong lane markings even in low light conditions. The outline of the vehicle ahead and it s point of contact with the ground can also be determined which is then used to compute distance [3] and time-to-contact [2]. The long exposure can also be used to detect distant taillights for AHC and candidate circles for TSR. However the long exposure causes motion blur on the traffic sign so it is not readable. Figure 5b has a medium exposure and high gain. It is has little motion blur and is used for TSR. The gain

6 (a) (a) (c) (d) Figure 3: (a) A typical highway scene with oncoming vehicles, preceeding vehicles, traffic signs and reflectors. Figures, (c) and (d) highlight some of the features that are used for classification. (see text) (a) (c) Figure 4: Use of the red pixels to determine color of the light source. (a) An in path vehicle. The halo of the taillights is smooth since the response of the red and clear pixels to red light is identical. The halo around the oncoming headlights shows a reduced response on the red pixels indicating that it is a white light source. Figure 5: A sequence of three different gain/exposure combinations: (a) A long exposure primarily used for lane and vehicle detection gives clear lane marks but produces motion blur on traffic signs and distant oncoming lights tend to blur. Medium exposure and high gain image has no motion blur and is used for TSR. (c) short exposure is used together with the long exposure frame for accurate interpretation of light sources.

7 (a) (a) Figure 6: (a) A detail of the lane marking as it appears in the long exposure image. corresponding point in the TSR image. The lane marking brightness is similar but note the increased background noise. and exposure settings used for the TSR frame are determined dynamically. Once the outline circle of a traffic sign has been detected and tracked in two images (any combination of long exposure frames and TSR frames) it is possible to predict the location and size of the sign in the next image. It is also possible to determine the maximum allowable exposure that will keep the motion blur below a certain limit. It is typically kept to under one pixel. In some cases the maximum allowable exposure combined with the maximum gain, give an image which is too bright and the traffic sign is saturated. In order to avoid this situation, the brightness of the sign is predicted based on the distance (derived from the image size of the circle), the angle based on the predicted location of sign in the image and the high/low beam status. The gain and exposure are then set accordingly. Further adjustment can be made using closed loop control. Since the TSR frame uses a high gain it is has significant image noise and not optimal for LDW (see Figure 6). However the TSR frame does give useful information on well marked highways and can effectively double the frame rate on such roads (which are more typical for fast moving vehicles). This is an opening for future development which could give a performance boost beyond the current LDW technology. Figure 5c is a short exposure frame that helps in accurate classification of the light sources detected in the long exposure frame. Figures 7a and 7b show in detail the center of the image in both long and short exposures respectively. The bright cluster detected on the left in figure 7a clearly appears as two distinct pairs of oncoming headlights in the short exposure image and not one close pair of oncoming lights. The cluster on the right taillight of the in-path vehicle is clearly split into a taillight on the close vehicle which matches the left light and also a taillight on a distant vehicle. The long and short exposure frames also help with the detection and recognition of overhead electronic Figure 7: Use of the short exposure frame to help classify the light source clusters detected in the long exposure frame (a). signs. Figure 8 shows how the long exposure is used to detect candidate overhead signs and the short exposure is used to read the content of the signs. 5 The Mobileye EyeQ Family The Mobileye EyeQ family was developed to support the computational requirements of multiple DAS applications running concurrently. The first generation, Mobileye EyeQ, combines two general purpose CPU s and four vision computation elements (VCE) that provide hardware acceleration for pattern recognition, object tracking and other basic building blocks of computer vision algorithms. The two CPUs and the VCE units can all operate in parallel under the control of one of the CPU units which is also in charge of camera gain/exposure control and CAN communication with the host vehicle. The Mobileye EyeQ can be found in a number of serial production programs including the 2008 GM Cadillac CTS and Buick Lucerne, the 2008 Volvo S80, V70, and XC7 models where it performs vehicle and lane detection and the newly announced BMW 7 Series where it performs Speed Limit Information (SLI), High Beam Assist (HBA) and Lane Departure Warning (LDW). The second generation, Mobileye EyeQ2, follows the same architecture concept but with significant enhancements that give it almost a six fold increase in computational power. The main goal of the Mobileye EyeQ2 was to be able to add pedestrian detection and more, to the bundle of applications running concurrently on a single, low cost, ECU. Figure 9 shows the block level diagram of the new chip. The main features

8 (a) Figure 8: (a) The long exposure is used to detect candidate overhead signs The short exposure is used to read the signs. are: Two Floating Point MIPS34K hyper threads RISC CPU running at 333MHz (twice chip speed). 1 MByte internal SRAM Five enhanced Vision Computation Elements (VCEs) with increased computational power and flexibility. Three new Vector Microcode Machines (VMP) The Vector Microcode Processor (VMP) is an innovative design for a coprocessor module for ASICs targeting computer vision applications: it operates very efficiently on 2D patches of data and has some very powerful and unique instructions for vision processing. The VMP is a fully programmable VLIW vector processing unit with local memory but without cache giving deterministic memory access and single clock instructions. The pipeline is handled by the programmer and compiler at compile time. Mobileye EyeQ2 will be launched in serial production vehicles starting 2009 with an application involving a consolidated feature package of lane detection, vehicle detection, pedestrian detection and fusion. Figure 9: Block level diagram of the Mobileye EyeQ2. References [1] ISO 17361:2007 Intelligent transport systems Lane departure warning systems Performance requirements and test procedures [2] O. Mano, G. Stein, E. Dagan and A. Shashua. Forward Collision Warning with a Single Camera In IEEE Intelligent Vehicles Symposium (IV2004), June. 2004, Parma Italy. [3] G. P. Stein, O. Mano and A. Shashua. Vision-based ACC with a Single Camera: Bounds on Range and Range Rate Accuracy In IEEE Intelligent Vehicles Symposium (IV2003),June 2003, Columbus, OH. [4] S. V. Rice, G. Nagy and T. A. Nartker Optical Character Recognition: An Illustrated Guide to the Frontier Published by Springer, 1999 [5] M.F. Tappen, B. C. Russell, and W. T. Freeman. Efficient Graphical Models for Processing Images In The Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)., June 2004 [6] J. S. Stam et al., Continuously Variable Headlamp Control US Patent number: US 6,593,698 [7] K. Schofield et al., Vehicle Headlight Control Using Imaging Sensor US Patent number: US 5,796,094