Panoramic Vision System for an Intelligent Vehicle using. a Laser Sensor and Cameras

Transcription

1 Panoramic Vision System for an Intelligent Vehicle using a Laser Sensor and Cameras Min Woo Park PH.D Student, Graduate School of Electrical Engineering and Computer Science, Kyungpook National University, 1370 Sankyuk-dong, Buk-gu, Daegu, South Korea , , mwpark@vr.knu.ac.kr Kyung Ho Jang Research Professor, Graduate School of Electrical Engineering and Computer Science, Kyungpook National University, 1370 Sankyuk-dong, Buk-gu, Daegu, South Korea , , khjang@knu.ac.kr Soon Ki Jung Professor, School of Computer Science and Engineering, Kyungpook National University, 1370 Sankyuk-dong, Buk-gu, Daegu, South Korea , , skjung@knu.ac.kr ABSTRACT We propose a multipurpose panoramic vision system for eliminating the blind spot and informing the driver of approaching vehicles using three cameras and a laser sensor. A wide-angle camera is attached to the trunk and two cameras are attached under each side-view mirror to eliminate the blind spot of a vehicle. A laser sensor is attached on the rear-left of the vehicle to gather information from around the vehicle. First of all, the proposed system displays laser sensor data in three dimensions and performs the calibration between laser sensor data and a rear image for finding road boundaries and gathering information on other vehicles. The proposed system computes the homography between the rear image and the laser sensor data interactively and overlaps the laser sensor data on the rear image for improving driver s recognition. Then, the system generates a panoramic mosaic view to eliminate the blind spot. First, the system computes the focus-of-contraction (FOC) in images taken from the rear-view and the homography between each side-view image and a rear-view on pre-processing. Next, the system performs the image registration process after road and background region segmentation using laser sensor data. Finally, it generates various views such as a cylindrical panorama view, a top view, a side view and an information panoramic mosaic view for displaying varied safety information. 1

2 INTRODUCTION BACKGROUND Today, vehicles are widely used for transport. The human-active area is increased by vehicle usage thus improving the quality of life. On the other side, however, many people are killed or injured in car accidents. Therefore, the present vehicle includes various safety equipments which help to prevent car accidents. For example, the lane departures warning system (LDWS) informs the driver when the vehicle is moving out from the lane. The driver state monitoring (DSM) wakes the driver up with an alarm when the driver falls into a doze. Vehicles including these kinds of safety devices are called intelligent vehicles [1]. Recently, the main issue of study in the field of intelligent vehicle design is the safety features. Especially, many of the assisted devices focus on reducing the driver s mistakes or eliminating the blind spot of a driver. Chief of all, eliminating the blind spot can reduce many car accidents. The blind spot of a driver means that a driver cannot see the field of the road [2, 3, 4]. Figure 1 shows the driver s blind spot and dangerous objects. From figure 1, a driver does not identify a motorbike in the blind spot when the motorbike is encroaching on the blind spot. In this case, car accidents frequently happen. Therefore, to reduce the blind spot, many drivers generally use the assisted mirror (e.g. convex mirror) and the wide-angle camera. However, this approach does not eliminate the blind spot of the vehicle but only reduces it. Therefore, we suggest a multi-purpose panoramic vision system. Figure 1. Driver's Blind Spot and Approaching Vehicles RELATED WORK Recently, expensive vehicles include various safety systems. Especially, many drivers are interested in safety devices for eliminating the blind spot [3, 4]. In most of the systems, only a wide-angle camera is used. The driver can see a wider area than the rear-view mirror. However, it is difficult for the driver to recognize the situation as shown in the rear scene because the image of the wide angle camera has a dense distortion. Another approach uses many cameras that mount standard lenses. In this case, image registration techniques are required for supporting the wide-view area. The Ford Interim Summer 98 of the Washington 2

3 University generates a panorama image to eliminate the blind spot [5]. Three CCD cameras are used to eliminate the rear blind spot. This panorama image is generated by using the mosaic algorithm. Ford s CamCar uses four cameras for generating panorama images [6]. However, these approaches only eliminate the blind spot of the rear-view. The concept car Z22 of BMW uses three cameras for eliminating the rear blind spot [7]. Three cameras are attached on the left/right and rear of the vehicle that instead of two side-view mirrors and a rear-view mirror. This system attempts to generate a panorama image, but it does not offer a perfect panorama image. General Motors presented the Panoramic Vision TM System of Magna Donnelly. They introduce the next generation rear-view mirror system using two cameras attached on the rear of the vehicle and four side cameras attached on each door [8]. However, it is at present just a concept design system. Nissan also attaches an around view on Infiniti EX35 [9]. The around view is a parking assist system for eliminating the blind spot around the vehicle. It has four wide-angle cameras that are attached on front/rear/left/right of the vehicle toward the road. It just performs the lens calibration for generating undistorted images. Undistorted images are displayed in a top-view when a driver parks the car. Another safety system that interested drivers are the blind spot information system (BLIS). This system is a safety system of another type to protect a driver from the blind spot. The blind spot information system intends to inform a driver of approaching vehicles that are in the blind spot. The BLIS has been included in high-end vehicles of Volvo and Ford [10]. In this paper, we propose a multi-purpose panoramic vision system for eliminating the blind spot and informing the approaching vehicles by using two standard cameras, one wide-angle camera and a laser sensor. In the proposed system, two standard cameras are attached under each side mirror and one wide-angle camera is attached to the trunk. For supporting wider area, two standard cameras are out-panning. A laser sensor is attached to the rear of the vehicle for gathering the varied information around the vehicle. To enhance visualization, input images of three cameras are registered. And then, the driver recognizes the object in the blind spot from the displayed image without confusion. Therefore, the system can eliminate the blind spot and the approaching vehicles are displayed in the image. The registration algorithm is not the general solution. However, the approach used in this study is suitable for eliminating the driver s blind spot. The remainder of this paper is as follows: Section 2 describes the proposed system. The laser sensor calibration technique is presented in Section 3. The panoramic vision generation method is presented in Section 4, along with extensive simulation results in Section 5. Finally, concluding remarks are presented in Section 6. 3

4 SYSTEM OVERVIEW In the proposed system, two standard cameras (30 degrees) are attached under each side mirror and one wide-angle camera (120 degrees) is attached on the trunk of the vehicle. For visualizing a wider area, two standard side-view cameras are out-panned about 15~30 degrees. Furthermore, LD-OEM laser sensor made by SICK is attached on the left rear of the vehicle. A laser sensor measures the distance from the vehicle to other objects. Figure 2 shows the configuration of the proposed system. Figure 2. System Configuration and Range of Devices The proposed system provides a multi-purpose panoramic vision system for eliminating the blind spot of a driver. The main functions of the proposed system are panorama image generation around the car and a warning system for approaching vehicles in the blind spot. However, the general registration method is not applied because acquired images using cameras have a different center of projection (COP). Also if we only use laser sensor data, the driver cannot clearly recognize any safety hazards. Therefore, we present a novel method for the image registration and a method to effectively display the laser sensor data. Figure 3 show the flow chart of the proposed system. Figure 3. Work Flow of the Proposed System 4

5 The proposed system consists of the laser sensor data calibration part and the panoramic mosaic view generation part. The calibration system for laser sensor data consists of pre-processing step and a visualization step. In the pre-processing step, an undistorted map is computed to calibrate lens distortion and a homography for overlapping laser sensor data on the rear image is computed. The pre-processing step is executed only once. The visualization step is performed while driving. A homography between an image data and distance data obtained from laser sensor data is calculated for several frames. Then, the proposed system overlays some information in the image such as approaching vehicles. The panoramic mosaic view generation consists of a pre-processing step, segmentation step, alignment step and visualization step. In the pre-processing step, the focus-of-contraction (FOC) and the homography between cameras are computed. This step is also executed only once for system initialization. After the pre-processing step is finished, the other steps are executed while driving. In the segmentation step, the road area and the background area in the input image are separated using results of laser sensor data and the FOC. In the alignment step, image registration of the side-view images and the rear-view image is performed for each of the road parts and background parts in the images. In the visualization step, various views such as a cylindrical panorama view, a top view, a side view and an information panoramic mosaic view for visualizing varied safety information are generated along with a panoramic mosaic view including the position of approaching vehicles. LASER SENSOR DATA CALIBRATION The laser sensor measures the distance from the driver s vehicle to other objects. However, the laser sensor data is not easy for a driver to interpret. Therefore, laser sensor data is overlaid using a homography on a rear image because the laser sensor data is not synchronized with the image data on the hardware level. Some other information such as the position of approaching vehicles or the boundary line of the road on the images are augmented to improve driver s recognition. PRE-PROCESSING STEP In this step, an unwarping map is generated for the calibration of lens distortion. And then, a homography is computed between the rear image and the laser sensor data for its registration on the rear image. Finally, the homography is adjusted using user interaction for achieving a more correct homography matrix. Calibration of lens distortion For eliminating a rear blind spot, a wide-angle lens camera of 120 degree field of view is used. The wide-angle lens camera has dense radial and tangential distortion. Therefore, distortions of the rear image are calibrated using the distortion coefficients and intrinsic parameters [11, 5

6 12]. However, the distortion removal process uses a long execution time. Therefore, unwarping map is computed only once in the pre-processing step. Then, the unwarping map is applied to the rear images while driving. Homography The laser sensor data only gives the distance from the vehicle to other objects. This data is used internally for other functions of an intelligent vehicle. Also, laser sensor data is useful in the removal of the blind spots. However, simply displayed laser sensor data is not easily understood by the driver. Therefore, the proposed system overlays the laser sensor data on the rear image for faster recognition of hazards in the blind spots. For the alignment between the rear image and the laser sensor data, a homography matrix is used. The homography means the transform matrix between two dimensional planes [13]. Because the laser sensor data exists in the x-z plane and the rear image data exists in the x-y plane, two types of data can be aligned by using the homography. Therefore, the homography is computed in advance. For computing the homography, we need 4 or more matched features between the laser sensor data and the rear image data. Therefore, we select some features in the laser sensor data. And then, we obtain their corresponding features in the rear image. We extract inlier feature set using RANSAC Algorithm [14] with the feature correspondences and compute the homography using the inlier feature set. The homography is computed using the following equation, wwuu h 11 h 12 h 13 uu pp = wwvv HHHH = h 21 h 22 h 23 vv, (1) ww h 31 h 31 h 33 1 Where pp and pp are the corresponding feature points and HH is the homography matrix. Interactive adjustment of the homography After computing the homography, we refine the homography more correctly using user interaction. First, we overlap the laser sensor data using the current homography on the rear image. After that we adjust the feature points of the laser sensor data until getting the correct homography interactively. Whenever we adjust the feature positions of the laser sensor data, the proposed system updates the overlaid image using the refined homography. In this way, we can get a more correct homography for the overlay between the rear image and the laser sensor data. Figure 4 shows that we get a more correct homography by adjustments to red points interactively. The red points are matched feature sets to compute the homography. 6

7 Figure 4. Interactive Procedure to Compute the Homography VISUALIZATION STEP In this step, the visualization is performed using the overlaying of the laser sensor data while driving. First, laser sensor data is interpreted into the 2D points data. And then the 2D points are overlaid on the rear image. Display of laser sensor data Laser sensor data captures distances and angles by using an LD-OEM sensor made by SICK. However, the drivers cannot recognize this raw data because these captured data consists of hexadecimal code. Therefore, the laser sensor data is displayed on the x-y plane using OpenGL in three dimension space. Visualization Simple plotting of 2D laser sensor data cannot offer the driver with its fast interpretation. Therefore, the proposed system overlaps the laser sensor data using the refined homography on the rear image while driving. Overlapped laser sensor data has many function subsidiaries. The main function is the warning function about hazardous objects in the blind spots. The proposed system warns the driver about the fast approaching vehicles. This function is called blind spot information system (BLIS) [10]. Most BLISs have a warning function with a sound. However, the proposed system displays fast approaching hazardous vehicles with the distance data for other vehicles in the image. The proposed system improves driver s recognition ability for hazards in the blind spots. The road region and the background region are segmented using overlapped laser sensor data in the rear image. Segmented road and background data can be used for registration between the rear image and the two side images. PANORAMIC MOSAIC VIEW GENERATION Two side mirrors and one rear mirror attached a vehicle do not eliminate the blind spot perfectly. Therefore, the proposed system attaches three cameras to the rear and sides of the vehicle, respectively. Attached cameras eliminate the blind spots of the driver. But, the 7

8 separated images captured by three cameras require an effective visualization for easy understanding of the driver. Therefore, the proposed system performs a registration of three images. In this case, the general panorama mosaic method cannot be applied because the centers of projection for the three cameras are not same [15]. Therefore, we present a novel image registration method. A stitched panoramic mosaic image enhances driver s recognition ability for the blind spots. PRE-PROCESSING STEP In this step, the focus-of-contraction in the rear image is first computed. Then, homographies between the rear image and two side images are computed. Because the pre-processing step is executed only once, it can reduce the execution time of the proposed system. Focus of contraction Generally, if a camera moves forward or backward, optical flows in the captured image expand from one point or gather to one point. This point is called focus of expansion (FOE) or focus of contraction (FOC) [16]. Figure 5. Optical Flow and FOC on the Rear Image In the proposed system, the vehicle moves forward and the rear camera is moving backward. Therefore, the focus of contraction (FOC) is computed using the following equation, xx TT ffff = 0, ffff = 0. (2) For computing the FOC, feature extraction and matching in rear images is performed using KLT algorithm [17]. A fundamental matrix is computed using EIGHT-POINT algorithm [13] and RANSAC [14] in the corresponding points. Then, an epipole is computed using the equation (2). In the proposed system, this epipole means the FOC. In this equation, ff is the fundamental matrix and ee is an epipole and xx and xx are a corresponding point set. Homography between images In this part, homographies between the rear image and two side images are computed using 8

9 the equation (1). For computing the homography between two images, a set of 2D corresponding points on a 3D plane is required. Therefore, the proposed system assumes that the road is the basic plane for registration. Figure 6 shows the geometry between time t and time t t. In this figure, t is the time interval required for the car to move from the position of the rear camera to the position of two side cameras. Finally the panorama image is generated using images at time t. However, homographies (HH rr, HH ll ) cannot be computed between the rear image at time t and two side images at time t because of the scale difference. Therefore, homographies (HH rr,bb, HH ll,bb ) are computed between the rear image at time t and two side image at time t t and the homography (HH tt,tt tt ) between the rear image at time t and the rear image at time t t for reducing the scale difference effect. Finally homographies are computed using the following equations, 1 HH rr = HH tt,tt tt HH rr,bb, 1 HH ll = HH tt,tt tt HH ll,bb. (3) Figure 6. Geometry between time tt and time tt tt SEGMENTATION STEP The road and the background areas in the image are segmented using the laser sensor data. First, a line model is defined in RANSAC algorithm [14] and inlier points near the line in laser sensor data are extracted. Next, the inlier points are transformed using the homography from the laser sensor coordinates to the rear image coordinates. In the transformed inlier set, the line model is extracted through the FOC. These line models are the road boundaries in the rear image. Then, the road boundary of the two side images is transformed using homographies (HH rr, HH ll ) between the rear image and two side images. Therefore, the road region and the background region are segmented using the road boundaries in the images. ALIGNMENT STEP The proposed system cannot apply the general mosaic method because the images do not have a common center of projection. Therefore, the proposed system performs the registration for 9

10 each of the road and the background region. In the registration of the road regions, homographies (HH rr, HH ll ) are performed for the road region of the images at time t. In the registration of background regions, the simple homographies cannot be applied because it does not exist on common plane. Therefore, it is assumed that the background regions of the side images are modeled as shown in figure 7. Figure 7. Alignment Model VISUALIZATION STEP In this step, the blind spots of the driver are displayed in various situations. The proposed system displays them with various views such as the cylindrical panorama view, the top view, the side view and the information panoramic mosaic view for visualizing varied safety information. Information panoramic mosaic view This view is the basic display view. When the vehicle is running at over 10km/h, the information view displays panoramic mosaic view with the states of the vehicle such as speed of vehicle or the distance until the nearest approaching vehicles. Cylindrical panoramic view Another view is the cylindrical panoramic view that is made by using a cylindrical projection [18]. This view is also used when the vehicle is running at over 10km/h. This view presents a complete panorama image to the driver while driving. Side view The side view means the extended image from the images in each side camera. When you use the turn signal for changing the lane, the view is displayed to prevent the car accident with an approaching vehicle. It is the registration image based on the images in each side camera. Top view The last view type is top view that is made by using a simple image warping [19]. The top view is actually a bird s eye view that displays the blind spot as if the driver is seeing it from the sky. In this case, the driver can easily distinguish obstacles around the vehicle. This view 10

11 is operated when the vehicle is running at below 10 km/h or is reversing for parking. It looks like the Around View Monitor of Nissan but it is fundamentally different [9]. The Around View Monitor uses four wide-angle cameras pointed toward the road placed around the vehicle to eliminate the blind spots during parking. However, the proposed system does not change the position of attached cameras for generating the top view. The proposed system generates the top view using only a three-dimensional image warping technique. EXPERIMENTAL RESULTS In all the experiments, three cameras and a laser sensor were used on a real vehicle and data from each devices was captured at 70~80km/h on a real road. First, the proposed system captures the distance of objects around the vehicle using a LD-OEM laser sensor made by SICK. Then, it calibrates the distortion of the rear image and computes the homography between the laser sensor data and the rear image. Figure 8. Visualization of overlapped laser sensor data Figure 8 illustrates the result of laser sensor calibration. The left image in the figure show the result of visualization of the real laser sensor data and the right image in the figure displays overlapped laser sensor data with the nearest approaching vehicle (red rectangle) on an undistorted rear image. Next, the proposed system generates a panoramic mosaic view using an overlapped rear image and two side images for eliminating the blind spots. First, the proposed system computes FOC and the homographies between the rear camera and each side camera on the pre-processing step. Then, it segments the road and background regions using the laser sensor data and RANSAC algorithm. After the segmentation is completed, the registration is performed with the 3D alignment model. Variou s panoramic mosaic views are used on the driver s intention. Figure 9 shows the information panoramic mosaic view such as informing the information of the vehicle. Figure 10 shows that the proposed system warns an approaching vehicle. 11

12 Figure 9. Information Panoramic Mosaic View Figure 10. Information Panoramic Mosaic View ( with an approaching vehicle ) Figure 11 shows another view made by the cylindrical projection. This view is the most natural view to the driver. Figure 12 shows the side view with warning for the approaching vehicle. Figure 13 shows the top view such as seeing from the sky. It is effective when the vehicle is running at below 10 km/h or is reversing for parking. Figure 11. Cylindrical Panorama View 12

13 Figure 12. Side View Figure 13. Top View CONCLUSION In this paper, we propose a multi-purpose panoramic vision system for eliminating the blind spots that are mainly caused of car accidents. The proposed system establishes two cameras in each side of the vehicle and a wide-angle camera in the rear of the vehicle and a laser sensor in rear-left of the vehicle to eliminate the blind spots perpectly. Therefore, the proposed system provides various type views depending on the driver s intention for eliminating the blind spots. The proposed system consists of two subparts such as the laser sensor data calibration part and panoramic mosaic view generation part. In the laser sensor data calibration part, the unwarping map for calibration of the lens distortion and the homography for overlapping laser sensor data on the rear image are computed in the pre-processing step and other information such as approaching vehicles are displayed in the image while driving. In the panoramic mosaic view generation part, the FOC and the homography between cameras are computed in pre-processing. The images are segmented using the road boundary into the road region and the background region while driving. And then, the registration is performed on the images of three cameras and the visualization on various type views is provided with the laser sensor data. 13

14 The proposed system has an advantage of visualization for the blind spot but also improves on other problems. The registration is performed for each of the road regions and background regions but this method is not correct in the background case. Because of the registration for the background region, the registration image has a little distortion for approaching vehicles. Therefore, future studies will focus on the novel registration method based on multi-perspective panoramic image generation method. Also, the experiment could not be conducted in bad weather such as a snowy or rainy day. But, the proposed system will be operated well if quality of images is guaranteed. Future research is needed to compensate for the current disadvantages. REFERENCES (1) Intelligent Vehicle Technologies. nologies (2) Blind Spot. d-spots.cfm (3) Blind Spot (vehicle). (4) R. Andrew Hicks and Ronald K. Perline, Blind-spot Problem for Motor Vehicles Applied Optics, Vol. 44, No. 19, pp , 2005 (5) Ford Interim Summer (6) Ford's CamCar Technology Eliminates blind spots. news/fulls-tory.asp?id=409 (7) The Mechatronic Car - Operating the Z22. /z22.htm (8) PanoramicVision TM /panoramicvisionsystem.asp (9) NISSAN - Around View Monitor. INTRODUCTION/DETAILS/AVM/ System. (10) Volvo invents BLIS. spot-info-system-actual-happiness/ (11) S. Nayar, Catadioptric Omnidirectional Cameras, in Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition(USA, 1997), IEEE Computer Society (12) Camera Calibration and 3D Reconstruction. mentation/camera_calibration_and_3d_reconstruction.html (13) R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge University Press, USA, 2004 (14) M. Fischler and R. Bolles, Random Sample Consensus: a Paradigm for Model Fitting with Application to Image Analysis and Automated Cartography, Communications of the ACM, ACM New York, USA, June 1981, Vol. 24, Issue 6, pp

15 (15) R. Szeliski, Image Alignment and Stitching: a Tutorial, Technical Report MSR-TR , Microsoft Research, USA, December 2004 (16) Didi Sazbon, Hector Rotstein and Ehud Rivlin, Finding the Focus of Expansion and Estimating Range using Optical Flow Images and a Matched Filter, Machine Vision and Applications, Springer Berlin, Germany, October 2004, Volume 15, Issue 4, pp (17) Carlo Tomasi and Takeo Kanade, Shape and Motion from Image Streams: a Factorization Method, Technical Report CMU-CS , Carnegie Mellon University, USA, Jan 1991, PA (18) Kyung Ho Jang, Soon Ki Jung, Minho Lee, Constructing Cylindrical Panoramic Image using Equidistant Matching, Electronics Letters, IEEE Computer Society, USA, September 1999, Volume 34, Issue 20, pp (19) William R. Mark, Leonard McMillan and Gary Bishop, Post-rendering 3D Warping, in Proceedings of the 1997 symposium on Interactive 3D graphics(usa, 1997), ACM New York 15