Driver Assistance for "Keeping Hands on the Wheel and Eyes on the Road"

Transcription

1 ICVES 2009 Driver Assistance for "Keeping Hands on the Wheel and Eyes on the Road" Cuong Tran and Mohan Manubhai Trivedi Laboratory for Intelligent and Safe Automobiles (LISA) University of California at San Diego, La Jolla, CA, USA Abstract Motivated from a basic tip for safe driving, Keeping Hands on the Wheel and Eyes on the Road, this paper introduces a vision based system for driver activity analysis by observing 3D movement of driver s head and hands from multiview video. From the results of upper body and head pose tracking, semantic descriptions of driver activities are extracted in two steps: First, we determine basic activities of each upper body part (e.g. Two, one, or no hand is on the steering wheel; Head is looking left, straight, or right). Then these basic activities are combined in a fusion step to extract higher level of semantic description of driver activities (e.g. whether the driver is following the above safety tip or not). Our experimental evaluation with real-world street driving shows the promise of applying the proposed system for both post analysis of captured driving data as well as for real-time driver assistance. Index Terms Driver activity analysis, Upper body pose analysis, Intelligent vehicles, Active safety. I. INTRODUCTION OMPUTER vision and machine learning technologies Cplay an important role and are increasingly used in today s vehicles to improve the safety as well as comfortability. However, to be effective, such technologies need to be human centric and need to work in a holistic manner which takes into account different components of the system involving driver (e.g. looking at driver to recognize driver activity and attention state), vehicle (e.g. looking at vehicle speed, steering angle, braking), and vehicle surround (e.g. looking at road and other cars to understand surrounding situation). [14, 15]. Among those components, we consider in this paper the part of looking at the driver which is a very important component in driver assistance systems (it is shown that a large portion of accidents is caused by human errors like driver inattention or cognitive overload [1]). We propose a vision based system for driver activity analysis by observing movement of upper body extremities including head and hands. The motivation is that from computer vision viewpoint, extremal parts have less occlusion and could be tracked more robustly compared to other upper body parts. In the scene of driving where the types of upper body motion are quite restricted compared to general case, we can also use our knowledge of upper body model to predict the whole upper body motion from extremities movement as an inverse kinematics problem. We have developed an approach for doing so which we called XMOB (extremity Movement OBservation) upper body pose tracker [12]. From driver assistance systems viewpoint, only coarse level of head and hand blobs movement have already exposed several important cues about driver state and activity. For example, hand position can be used to determine how many hands are currently on the steering wheel, which is an indicator of the readiness to control the vehicle. Hand blobs movement may indicate whether the driver is in rest state or is doing some actions (which may or may not relate to the driving task). Since the driver typically sits in a fixed position, head blob tracking roughly reveal the sitting posture which is important, e.g. for smart airbag deployment [13]. In this paper, we also incorporate head pose estimation which is kind of a more detailed level of information. However it is worthwhile because head pose is a strong cue for determining driver focus of attention. Our experiments with real-world street driving scene show the potential of the proposed system in vision based analysis of driver activity. Although the system currently works in offline mode which does post analysis of the saved driving scene data, it has strong promise to be applied in real-time driver assistance system such as active safety system (which predicts critical situations and alert driver if needed) since several major parts of the system currently run in real-time. On the other hand, it should be mentioned that post processing of saved data is also an important task. More concretely, in our lab we have built several vehicle testbeds that capture rich contextual realistic data of the environment, vehicle, and driver which can be used later for studying driver behavior as well as developing better analyzing algorithms. Since there is a huge amount of such data that cannot be processed manually, there is a need for post processing tools to automatically extract semantic descriptions from these data. The remaining part of the paper is organized as follows: Section 2 is the related research studies. Section 3 describes the details of the proposed system for driver activity analysis. Section 4 provides experimental evaluation and finally we have some concluding remarks and discussions in Section 5. II. RELATED RESEARCH STUDIES There were several research studies for driver activity /09/$ IEEE 97

2 analysis based on tracking different upper body parts. Park and Trivedi [9] proposed a hierarchical framework for driver activity analysis at multiple levels: from low level of individual body-part pose to middle level of single body part action, and then high level of driver interaction with vehicle. However this paper just proposed the ideas of such a framework, no real implementation was done. Cheng et al. [3] proposed using multi-perspective, multi-spectral (thermal infrared and color) cameras combined with data about vehicle dynamics (e.g. steering angles, speed) for turn analysis and hand grasp analysis. Although the idea of using different camera types to improve the system s robustness is theoretically true, using thermal camera seems not be a good choice since in many real driving situation, the heat from car engine, sunshine, could make thermal images very noisy. In this paper we propose a system for driver activity by observing movement of upper body extremities (head and hands) from multiview color cameras. At high level, this system follows quite similar ideas of the hierarchical framework proposed in [9]. However, this is the first proposed system with real implementation and evaluation. The XMOB upper body tracker is motivated by previous researches in inverse kinematics of arms, e.g. Soechting and Flanders, researchers in neurophysiology, found that the desired position of the hand roughly determines the arm posture [10]. Compared to other upper body pose tracking methods (e.g. [4, 6]), XMOB only needs image evidences to track extremal parts (head and hands) which are the easiest parts to track with less occlusion. From the 3D movements of head and hand blobs, XMOB then predicts whole upper body motion as an inverse kinematics problem. By breaking the very large search problem of upper body pose estimation into two sub problems (tracking extremal parts first, and then predict the most likely corresponding sequence of remaining inner joints), the complexity is also reduced to achieve realtime performance. In our proposed system for driver activity tracking, we also incorporate head pose estimation because it reveals important information about driver focus of attention. Moreover realtime and robust head pose tracking systems like HyHOPE (Hybrid Head Orientation and Pose Estimation) [8] have been developed. We chose head pose rather than for example a more detailed level like eye gaze because systems for eye gaze tracking are typically not as robust as for head pose tracking and it was also shown that for some tasks like lane change prediction, head pose is a better cue than eye gaze [5]. In the next section, we will go into more details of the proposed vision based system for driver activity analysis. III. THE PROPOSED SYSTEM FOR DRIVER ACTIVITY ANALYSIS BY OBSERVING HEAD AND HAND MOVEMENTS The flowchart of the proposed system is shown in Fig.1. From synchronized multiview video input, the upper body Fig. 1. Flowchart of the proposed system for driver activity analysis pose and head pose are tracked by XMOB and HyHOPE respectively in two different streams. However the pose estimation results are then synchronized and used for extracting semantic descriptions of driver activities in two steps: First, we determine basic activities of individual upper body part such as determine if the head is looking left, straight, or right; if hand is in rest or moving state (currently we use 4 set of basic activities as shown in Fig.1.C). Then there is a fusion step using these basic activities as input for higher level of semantic description of driver activity. Fig. 1.D shows the 6 types of semantic driver activity descriptions (events) that we currently detect. A. XMOB Upper Body Tracker The skeletal upper body model used in XMOB upper body tracker is shown in Fig. 2. The length of body parts are considered fixed, which means there is only kinematics movement at the joints. There are 4 joints in the model: two shoulder joints, each has 3 Degree of Freedom (DOF) and two 1 DOF elbow joints. An upper body pose at time t can be represented by 7 positions in 3D of inner joints and extremities X t = {P hea t, P lha t, P rha t, P lsh t, P rsh t, P leb t, P reb t } As mentioned above, the idea is to break the estimation of X t into two sub problems: First estimating the subset of extremities {P hea t, P lha t, P rha t } which are the easiest parts to track with less occlusion. Then by observing sequence of these extremity movements over time, we predict the 98

3 P reb P rsh P rha P hea P lha P lsh P leb Fig. 2. Upper body model used in XMOB upper body tracker corresponding sequence of the remaining subset {P t lsh, P t rsh, P t leb, P t reb } as an inverse kinematics problem. The main ideas of XMOB upper body tracker [12] are: Regarding head and hands tracking, XMOB has a semisupervised procedure, in which user starts by moving only their extremities, to robustly segment skin color of a particular user from background colors of a particular scene. By learning a specific color clustering model for each case, we can achieve better performance compared to a general model for arbitrary human skin color, e.g. [7] and background colors. Regarding the prediction of corresponding sequence of inner joint positions {P t lsh, P t rsh, P t leb, P t reb } from head and hand movement {P t hea, P t lha, P t rha }: Since upper body kinematics is redundant, there could be many solutions for upper body pose with the same head and hand positions. Therefore, there is no precise solution in every case but we want to find the most likely solution which could be true in many cases. XMOB took a numerical approach. Using geometric constraints between upper body joints and extremities, at each frame it determines a set of hypotheses for possible inner joint locations. Then by observing extremity movements in a period of time, XMOB uses heuristic rules of minimizing joint displacement and preserving left-right symmetry to predict the corresponding sequence of inner joint positions. B. HyHOPE Head Pose Tracking We use the implementation HyHOPE in [8] for head pose tracking. The main idea of HyHOPE is to improve the performance by combining a static head pose estimation with a real time 3D model-based tracking system. From an initial estimate of head position and orientation, the system generates a texture mapped 3D model of the head from the most recent head image and using particle filter approach to find the best match of this 3D model from each subsequent frame. HyHOPE also exploit GPU (Graphics Processing Unit) programming to run in real-time. C. Extracting basic activities of each upper body part There are 4 set of basic activities as shown in Fig. 1.C. These basic activities are simply extracted from upper body pose and head pose tracking results by a thresholding process. For set 1, we roughly mark a 3D region for the steering wheel and determine if hand position is inside that region (hand on wheel) or not. For set 2, a distance threshold (which is 10cm in our experiment) between consecutive hand positions is used to determine if hand is in rest or motion state. For set 3, since we assume that the driver sitting in a fixed position, sitting pose is determined by a distance threshold (which is 20cm) between the referenced 3D head position and current 3D head position projected on the direction of car length. For set 4, an angle threshold of head pan angle (which is 200 in our experiment) is used to determine looking left/straight/right. D. Combining basic activities of each upper body part for higher level of semantic description We represent the relation between basic activities as separate, sequential, or concurrent based on their time gap as well as the percentage of overlapping between them (as illustrated in Fig. 5) Fig. 5. Three types of relation between basic activities As shown in Fig. 1.D, the fusion of basic activities into higher level of semantic description works in a rule-based manner in which AND operator represents concurrent relation and THEN operator represents sequential relation. These rules can be implemented by 2 types of state machine: 2-state state machine for rules with no THEN operator and 3-state state machine for rules with 1 THEN operator. Fig. 6 and Fig. 7 show examples of these 2 types of state machine. Automatically return to the start state s 0 s s 1 Hand on wheel AND Hand on wheel AND 2 Head look left Hand move Turn left Start state detected Fig. 6. Example of a 3-state state machine for rule 2 in Fig.1.D Automatically return to the start state s 0 Head look straight AND Hand on s 1 wheel AND Hand rest Start state Normal going forward detected Fig. 7. Example of a 2-state state machine for rule 1 in Fig.1.D 99

4 this driver activity analysis should finally be incorporated with other component of looking at vehicle and surround environment to have a holistic sensing system for intelligent driver support. Fig. 8. Camera setup for the experiment in LISAP testbed IV. EXPERIMENTAL EVALUATION All of the data used here was collected from LISA-P testbed ( The testbed is centered on the computing, power and mounting resources in the LISA-P to allow frame-rate collection of data simultaneously from multiple sources. For this experiment, we used 2 color cameras for upper body pose tracking and 1 color camera for head pose tracking (Fig. 8). Real driving data of different drivers was captured and then analyzed by the proposed system. Fig. 9 shows some results for visual evaluation of the HyHOPE head pose tracking with large range of head rotation, different lighting conditions, and some occlusions. Fig. 10 shows results of XMOB upper body tracking in several different driving activities. Regarding the runtime performance, we did the analysis on a Pentium(R) D CPU 2.8 GHz and HyHOPE head pose tracking ran at around 15 fps (frame per second) and the runtime of XMOB upper body tracking is around 6 fps. Fig. 11 shows the results of extracting basic activities for each upper body part (described in Section 3.C) based on the above pose tracking results. These extraction results are also compared with the manually annotated ground truth and we see that the proposed system can capture these basic activities quite well. Fig. 12 is a sample result of combining basic activities of each upper body part for higher level of semantic description (described in Section 3.D). V. CONCLUDING REMARKS We have proposed a vision based system for driver activity analysis by observing 3D movement of upper body extremities including head and hands. The system was evaluated with real-world street driving scene and showed promise to be applied for both real-time active safety systems as well as analyzing systems which post process driving data captured from rich contextual realistic situation. For future direction, ACKNOWLEDGMENT We are thankful to our colleagues at CVRR lab for useful discussions and assistances. We especially acknowledge Dr. Erik Murphy Chutorian for his contribution in developing the HyHOPE system. The first author also thanks Vietnam Education Foundation (VEF) for its sponsorship. REFERENCES [1] World report on road traffic injury prevention: Summary. Technical report, World Health Organization, [2] S. Y. Cheng, M. M. Trivedi, "Turn-Intent Analysis Using Body Pose for Intelligent Driver Assistance", IEEE Pervasive Computing, 2006 [3] S. Y. Cheng, S. Park, M. M. Trivedi, "Multi-spectral and Multiperspective Video Arrays for Driver Body Tracking and Activity Analysis", CVIU, [4] A. Datta, Y. Sheikh, and T. Kanade, Linear Motion Estimation for Systems of Articulated Planes, IEEE Conference on Computer Vision and Pattern Recognition, [5] A. Doshi and M. M. Trivedi, "On the Roles of Eye Gaze and Head Pose in Predicting Driver's Intent to Change Lanes", IEEE Trans on Intelligent Transportation System, [6] V. Ferrari, M. Jiménez, A. Zisserman, Progressive Search Space Reduction for Human Pose Estimation, IEEE CVPR, [7] G. Gomez and E. Morales, Automatic Feature Construction and a Simple Rule Induction Algorithm for Skin Detection, ICML Workshop on Machine Learning in Computer Vision, [8] E. Murphy-Chutorian and M. M. Trivedi, "HyHOPE: Hybrid Head Orientation and Position Estimation for Vision-based Driver Head Tracking", IEEE Intelligent Vehicles Symposium, June [9] S. Park, M. M. Trivedi, Driver Activity Analysis for Intelligent Vehicles: Issues and Development Framework, IEEE Intelligent Vehicles, [10] J. Soechting and M. Flanders, Errors in pointing are due to approximations in sensorimotor transformations, Journal of Neurophysiology, [11] C. Tran and M. M. Trivedi, "Human Body Modeling and Tracking Using Volumetric Representation: Selected Recent Studies and Possibilities for Extensions", ACM/IEEE ICDSC, September 2008 [12] C. Tran and M. M. Trivedi, Introducing XMOB: Extremity Movement Observation Framework for Upper Body Pose Tracking in 3D, IEEE International Symposium on Multimedia, [13] M. M. Trivedi, S. Y. Cheng, E. Childers, S. Krotosky, Occupant Posture Analysis with Stereo and Thermal Infrared Video: Algorithms and Experimental Evaluation, IEEE Trans on Vehicular Tech, Nov [14] M. M. Trivedi and S. Y. Cheng, "Holistic Sensing and Active Displays for Intelligent Driver Support Systems", IEEE Computer Magazine, [15] M. M. Trivedi, T Gandhi, J. McCall, "Looking-In and Looking-Out of a Vehicle: Computer-Vision-Based Enhanced Vehicle Safety", IEEE Trans on Intelligent Transportation System,

5 Fig. 9. Visual evaluation of HyHOPE head pose tracking with large range of head rotation, change in lighting condition, and some occlusions Fig. 10. Visual evaluation of XMOB upper body tracking in several different driving activities. Top - Upper body pose tracking results in 3D. Bottom Superimposed 3D pose tracking results on image Fig. 11. Basic activities extraction results. Top Head (right, straight, or left), Middle Number of hands on wheel, Bottom Hand motion (rest or move) Fig. 12. Combining basic activities of each upper body part for higher level of semantic description. Result of alert type 1 detection (rule 4 in Fig. 1.D: Head look straight AND No hand on wheel AND Hand rest) 101