Independent Work Report Spring, TrafficAssist. A monocular camera based approach to identifying. traffic situations

Transcription

1 Independent Work Report Spring, 2013 TrafficAssist A monocular camera based approach to identifying traffic situations Horia Stefan Radoi, prof. Alain Kornhauser (advisor,orfe) Abstract This paper accompanies a software designed to implement a traffic situation identificator. It identifies lane markings and other traffic elements (other vehicles, obstacles in the road etc.), and was designed to be used as part of the visual input of an autonomous vehicle, in addition to the stereo vision. It manages to identify road situations in real time, and to signal the driver of possible scenarios. It creates a platform for a robust, platform and vehicle independent software that requires little to no calibration to identify relevant traffic situations. 1

2 1. Introduction Imagine you are out, on a Casual Wednesday, after celebrating your friends anniversary. Both you and your friend had a couple too many drinks, and are in no way capable of driving. You decide to call a taxi, and you insist that your friend should do the same. He insists that he is fine, and convinces you that he can drive, and that he can get home safely. He has, after all, driven for the past 5 years, and knows the road like the back of his hand. The bar you met up is close to a highway, and from the highway, your friend can get home in 5 minutes. He drives off, and as you are about to enter the taxi, you hear a loud crash, which you ignore. The following day you find that your friend was involved in an accident, and he is in serious trouble with the law. You are happy that your friend was not involved in a fatality, and try to make sense of what happened the previous night. Unfortunately, the scenario we presented is not unique, and can happen multiple times a day. It is estimated that, in 2010, in the United States, there were 32,885 deaths and around 2,239,000 injuries 1 2 as a result of the 5,419,000 auto vehicle crashes. Out of these fatalities, 31% were alcohol related. Since it is beyond the scope of this paper to criticize drunk driving, and given the inevitability of such behaviour, given the current legislation and human mentality, it is up to car manufacturers and engineers 1 nrd.nhtsa.dot.gov/pubs/ pdf 2 drv_factsheet.html 2

3 to provide safety nets, should this behaviour happen, and limit the damages to both the driver and to the people involved in crashes. Recent years have seen a multitude of safety devices implemented in auto vehicles, starting from the mandatory seat belts and airbags for protection, continuing with ABS, Automatic Shifting, Cruise control and GPS receivers integrated into the car, and ending with assistance for stop and go traffic and parking assistance. These additions have the role of protecting the driver and aiding him in the driving activity, by removing mechanical distractions(changing gears, pressing the gas pedal while on the highway) and providing an audible or visual assistance in the case of slow movement (parking or pulling out of a parking spot, stop and go traffic etc.). Perhaps the most important function that the future reserves is that of fully autonomous vehicles: vehicles that can drive themselves based on information they acquire from their surroundings and from GPS data, vehicles that can interact with each other, that can cluster and drive together, making better informed decisions about traffic flow, and, slowly, developing a fully autonomous traffic system, in which the driver is completely removed from the control loop. This future has the potential of drastically reducing the number of drunk related accidents, by the sheer virtue of removing the concept of a drunk driver. TrafficAssist, the software that accompanies this paper aims to help shape such an environment. It provides a monocular camera assistance system that can be attached to any vehicle s dashboard, and it will help the user identify traffic lanes and oncoming traffic. The software is aimed at being integrated 3

4 into an autonomous vehicle, and provides valuable information about the traffic conditions, information which can be used together with a stereo vision system to determine exact positions of vehicles and road obstacles based on the velocity and orientation of the car they are attached to. It is designed to work using a general, off the shelf webcam for image acquisition, and it is developed as a stand alone application on a non dedicated laptop. In its final implementation, it will use a similar input configuration, but run on a dedicated vision processing server integrated into the vehicle. 2. Overview 2.a. Background The necessity of autonomous vehicles rose as the demand for more safety in auto vehicles increased. There is currently legislation for more than 70 safety devices that can be and should be incorporated into an auto vehicle, and all are designed to aid the driver either in driving or in case of a collision. One characteristic of driving is its repetitiveness, which can become strenuous on a driver, whose full attention needs to be focused on the road and on his surroundings, most of the time exercised in an environment which rarely changes (while driving on a highway or Interstate, or in stop and go traffic). At the same time, the road infrastructure is carefully legislated, and the driving environment has a strict set of rules for every possible situation. It is a shame, then, that such a repetitive action cannot be automated, and the human driver ( Errare humanum est to err is human) be removed from the driving loop, and allowed the freedom to enjoy his driving experience without him having to worry about other 4

5 traffic participants. Previous work has been done in autonomous vehicle engineering. Starting with the DARPA challenges in 2004, 2005 and 2007, and continuing with the Google Driverless Car, it has been proven that autonomous vehicles can be built and that they can interact with human driven traffic and obey traffic laws. Combined with the emergence of hybrid and electric cars, the field of autonomous 7 vehicles has a huge potential of growth and is already being implemented, if at a lower level now, at a 8 higher level as soon as next year, with Mercedes 2014 S Class. Teams that have participated in the DARPA challenges have continued their research, and, including Princeton s Autonomous Vehicle Engineering 2.b Implementation The software was designed to be used as a standalone application, or as part of an integrated system. In its current form, it identifies lanes and has a clear representation of object edges. It will be able to implement an object classification and object tracking, by grouping the identified elements into three categories, using the vehicle it is located in as a point of reference: moving in the same direction as the vehicle (small difference in size, object shrinks as time goes by, or size difference is small when it

6 brakes), stationary objects in the scenery, or objects not moving relative to the vehicle (they appear as growing at a constant rate that is consistent to the measured speed of the vehicle) and objects moving towards the vehicle (size growing very fast, and rate of change consistent with a speed greater than that measured by the test vehicle). The software has the capability to be integrated into a larger system, as a supplementary visual input method for an autonomous vehicle, in addition to its stereo vision sensors, and can better interpret data if it has access to an accelerometer (to measure the real speed of the test vehicle). While GPS positioning would be a nice addition to the project, it is not necessary, since the software can be programmed to identify an environment based on visual cues (ie. many wide lanes for a highway environment, houses or buildings for an urban environment etc.) 3. TrafficAssist 3.a Hardware This project was developed and tested on a Sony VPCF127FX laptop computer(intel Core i7 CPU 1.73 GHz x 8), running Ubuntu 12.04, 64 bit. We have decided on a top of the shelf webcam ( Logitech C920, 1080p widescreen). This was consistent with our wish to make the hardware requirement as accessible as possible, and, at the same time, took advantage of this webcam s high pixel count (possibility to capture highly detailed images for the puppet Matte and the background) and auto focus. While, at times, the constant autofocus proved 6

7 to be a nuisance in image acquisition, the algorithms used were able to account for this flaw. For the purposes of testing this software, we have also used a 2004 BMW Z4, and acquired images in the area surrounding Princeton University, on Route 1 and Washington Road. The camera was placed on the passenger s side, close to the center of the vehicle, at the dashboard level. It was physically placed at 1.5 meters from the front of the car, at a height of 1 meter from the road level. 3.b Software OpenCV and Python OpenCV is an open source vision platform, released under a BSD licence, free for both academic and commercial use. It was written in C and C++, and provides interfaces for C, C++, Python and Java, under Windows, Linux, Mac OS, ios and Android. OpenCV uses the full computational power of a multi core computer, and provides a good framework for real time image processing. OpenCV contains over 500 optimized functions, for immediate usage. It features a growing online community, and very well documented community support, through both their own wiki and the StackOverflow website. This availability of algorithm implementation made it an essential tool for this project. The learning curve is steep, but once one understands the basic functionality, OpenCV proves to be a powerful image manipulation software. The OpenCV libraries and functions are built in a way that enables interaction between different programmers, by providing cross platform and language independent support. OpenCV features tutorials for multiple platforms, and the documentation files 7

8 feature all available programming implementations (C++, C, Python etc.). The OpenCV version used in this project is OpenCV This project was coded in Python. It is a very lightweight programming platform, providing the power of C++ and enabling the same object oriented programming functionality, while reducing the amount of code needed to be written. It provided a very good educational environment for the initial implementation of the OpenCV functions. Unfortunately, since the OpenCV developers are still implementing functions in Python, the lack of documentation for those functions proved to be an obstacle in implementing the program. The Python build used in the early phases of project was c Implementation The software is designed to use both real time input and pre recorded input. For consistency, during testing, a pre recorded video sequence was used as input. Once a stream is determined, the software processes each frame individually, by first determining the edges of the objects present (Using Sobel Operators and Canny edge detectors). This information is going to be used by the object classifier. The Canny edge detector is used with Hough transformations to determine the straight lines in the image. These lines, resulting from the prominent edges, are the basis of the lane identification of the software. 8

9 Future development includes identifying objects into the three classes described earlier (moving with, away from and towards the camera) and extrapolating the POV image to a top down (bird s eye) view of the world. By providing this extrapolation, a future autonomous vehicle can identify the best route to avoid road congestion or to tackle traffic problems as they arise. 4. Tests and results The system was tested in a highway setting around Princeton, NJ. The videos captured identify common traffic situations, such as stop and go traffic, following a vehicle, following a vehicle that brakes and accelerates. A test video was captured in an urban environment as well (Washington Road), and another test video captured a curving road (exiting a turnpike). So far, the results are satisfying, with the sobel operator identifying contours and creating a good basis for lane detection. Future developments include restricting the region of interest of the image to focus only on the road ahead (thus ignoring what happens to the sky and on the reflection of the car), and a better identification of discontinuous lines. Some test results can be seen in the Appendix. 9

10 5. Related Work 5.a PAVE In 2005 and 2007, Princeton s Autonomous Vehicle Engineering team participated in the two DARPA challenges. Both teams led by professor Alain Kornhauser, composed of undergraduate students, built two autonomous vehicles to participate in each respective competition. Even though the cars were functional, some of the algorithms used in their development became obsolete, as did some of the hardware on the vehicles. In 2012, a group of PAVE students including myself started porting the software of PAVE from the Microsoft Robotics Studio to the Robot Operating System (ROS), which has a better support for OpenCV implementations. The hardware used on these vehicles included both color stereo vision cameras and monocular, black and white cameras. The approach that PAVE takes is that since humans can drive using only vision, it is not unreasonable to consider a self driving car that utilizes only vision porcessing. The end goal is that the TrafficAssist software is going to be used on the PAVE vehicle. 5.b Google Self Driving car The Google Self Driving car is perhaps one of the best known examples of autonomous vehicles. The Google project, started under the coordination of Sebastian Thrun (the coordinator of the Stanford teams at the 2004, 2005 and 2007 DARPA challenges), has been leading both technological 10

11 advances and legislation to be pushed towards a future of autonomous driving vehicles. In a presentation given earlier this semester, during ORF 401, Joshua Herbach, a former Princeton Undergraduate and member of the PAVE project, currently working on the Google Self Driving car, showed some of the mechanics of the Google vehicle. While different from the PAVE approach in that most of the information regarding the environment comes from the huge LIDAR unit located above the vehicle (to obtain a representation of its surroundings), the Google car uses vision as a means of verification of its input data. This approach takes into account the huge database that Google has at hand (with the Google Street view), and uses a group of people to map out the specifics of each environment (traffic signs, speed limits etc.). In addition, the Google approach is sensitive to change, in that, if a car identifies that an environment is not consistent with its databases, it prompts its driver to take control of the car, until the area is updated. While Google s approach is working, it is severely ignoring the potential of vision processing: modifications in the environment happen all the time (branches falling, potholes forming, water on the road etc.), and a vision based approach can easily identify these obstacles and act accordingly. 6. Limitations and further work The current limitations of the project come from the restricted time frame given by the University in completing the independent work. While the project is not currently in a finished state, it will still be improved and more features can be added. 11

12 The current lane detection system assumes continuous lane markings, and does not take into account perspective. So far, it has only been tested from one point of view, but we plan to extend it to be independent of location (perhaps needing only some calibration once it is mounted into a new place). Some alternative algorithms or approaches to implemented algorithms are available, and they are presented as more efficient and more exact approximations of lane detections or object 9 identification. A quick means of efficiency is to restrict the ROI of the image to only an area located underneath the horizon line (since that is mostly where the road will be). Alternatively, ffthe image can be classified into color regions, and those corresponding to sky or side of the road can be ignored (for the purposes of lane detection and traffic identification). 7. Conclusions While this project is still in an early state, we are working on developing it into a fully functional, standalone project. Throughout this project we have identified the potential of monocular cameras in image processing, and have formed a basis of knowledge as to how a series of 2D images can be used to simulate a 3D environment. Moreover, starting at the 2D Point of View perspective, the software has shown a potential of extrapolating to a 2D, bird s eye perspective, thus aiding in programming an Artificial intelligence module to determine the best path it can take to reach its destination. 9 Kim, ZuWhan. "Robust lane detection and tracking in challenging scenarios." Intelligent Transportation Systems, IEEE Transactions on9.1 (2008):

13 8. Appendix Some of the images resulted from testing the system: Fig 1.a Test frame of a highway traffic situation Fig 1.b Canny edge detection and Hough line transformation used on the previous image to detect lanes. Some false positives are identified in the reflection of the car and on the sky, but these issues can be resolved by restricting the ROI of the image to only the road surface (or anything below the horizon line). 13

14 Fig 1.c Two successive Sobel operators applied to the same image, once in a vertical direction and once in a horizontal direction, are here superimposed to create a good representation of the edges of the objects. Individually, the Sobel operators can provide useful information about the surroundings (the horizontal one providing a good approximation of where the shadows of the vehicles are, and the vertical one providing another means to identify traffic lanes) Honor Pledge This paper represents my own work, in accordance with University Policies. [Horia Stefan Radoi] 14

15 Acknowledgements I would like to thank my independent work advisor, professor Alain Kornhauser, for coordinating my work and steering me in the right track, even though I was a pain to work with. I would also like to thank the Computer Science department at Princeton University, along with the IW coordinators, for always reminding me of deadlines and for all the passion and dedication they have in teaching me, us to become better scholars. I would also want to thank all the people who have supported me in conducting my reasearch, Kevin Yang for allowing me to use his car for weird purposes and all my friends at Colonial Club who have supported me in their own ways. Lastly, I would want to reiterate my gratitude to professor Adam Finkelstein for allowing me to learn OpenCV and develop as a vision processing programmer. References and bibliography Academic papers Kim, ZuWhan. "Robust lane detection and tracking in challenging scenarios." Intelligent Transportation Systems, IEEE Transactions on9.1 (2008): Wang, Yue, Eam Khwang Teoh, and Dinggang Shen. "Lane detection and tracking using B Snake." Image and Vision computing 22.4 (2004):

16 Wang, Yue, Dinggang Shen, and Eam Khwang Teoh. "Lane detection using spline model." Pattern Recognition Letters 21.8 (2000): Wang, Yue, Dinggang Shen, and Eam Khwang Teoh. "Lane detection using catmull rom spline." IEEE International Conference on Intelligent Vehicles OpenCV, Python Tutorials Tutorials: Rahman, Abid, OpenCV Python Tutorials, <opencvpython.blogspot.ro> Rahmbihia, Jay, Paranoid Android, < < < < (2007). Pisarevsky, Vadim. "Introduction to OpenCV." Intel Corporation, Software and Solution Szeliski, Richard. Computer vision: Algorithms and applications. Springer,