Balancing Privacy and Safety: Protecting Driver Identity in Naturalistic Driving Video Data

Transcription

1 Balancing Privacy and Safety: Protecting Driver Identity in Naturalistic Driving Video Data Sujitha Martin Laboratory of Intelligent and Safe Automobiles UCSD - La Jolla, CA, USA scmartin@ucsd.edu Ashish Tawari Laboratory of Intelligent and Safe Automobiles UCSD - La Jolla, CA, USA atawari@ucsd.edu Mohan M. Trivedi Laboratory of Intelligent and Safe Automobiles UCSD - La Jolla, CA, USA mtrivedi@ucsd.edu ABSTRACT Naturalistic driving dataset is at the heart of automotive user interface research, detecting/measuring driver distraction, and many other driver safety related studies. Recent advances in the collection of large scale naturalistic driving data include the second Strategic Highway Research Program (SHRP2) consisting of more than 3000 subjects and the 100- Car study. Public access to such data, however, is made difficult due to personal identifiable information and protection of privacy. We propose de-identification filters for protecting the privacy of drivers while preserving sufficient details to infer driver behavior, such as the gaze direction, in naturalistic driving videos. Driver s gaze estimation is of particular interest because it is a good indicator of driver s visual attention and a good predictor of driver s intent. We implement and compare de-identification filters, which are made up of a combination of preserving eye regions, superimposing head pose encoded face mask and replacing background with black pixels, and show promising results. Author Keywords De-Identification; Driver Safety; Privacy; Human Factors; ACM Classification Keywords K.4.1 Computers and Society: Privacy; I.4.0 Image Processing and Computer Vision: Image Processing Software INTRODUCTION The 100-Car Naturalistic Driving Study (NDS) collected data for more than one year, yielding in nearly 2,000,000 miles and 42,300 hours driven, 82 crashes, 761 near crashes and 8,295 critical incidents [10]. Preliminary analysis on the data has shown nearly 80% of crashes and 65% of near-crashes involved some form of driver inattention within three seconds prior to the event [10], and a strong correlation between accumulated off-road eye glances and the risk of crashes and near-crashes [18]. Similarly, the SHRP-2 naturalistic driving study (NDS) has been collecting data for the past two years which will result in approximately 4 petabytes of data, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. AutomotiveUI 14, September , Seattle, WA, USA Copyright c 2014 ACM /14/09...$ million hours of video, 3000 subjects, 5 million trips, 33 million miles driven and 4 billion GPS points [2]. Public access to raw data, however, for neither the 100-Car study nor the SHRP2-NDS data is available due to personal identifiable information and protection of privacy. There is, therefore, a need to protect the privacy of individuals while preserving sufficient details to infer driver behavior in naturalistic driving videos. Camera sensors looking at the driver, an integral part of intelligent vehicles [12], are of particular concern for invasion of privacy, as they can be used to recognize the driver s identity. Typical protection of privacy of individuals in a video sequence include blacking out and blurring of faces or people, commonly referred to as De-Identification. While this will help to protect the identities of individual drivers, it impedes the purpose of sensorizing vehicles to look at the driver and his behavior. In an ideal situation, a de-identification algorithm applied to the video of looking at the driver would protect the privacy of drivers while preserving sufficient details to infer driver behavior (e.g. eye gaze, head pose, hand activity). Any form of de-identification on video sequences of looking at the driver can only degrade the performance of detecting or measuring driver behavior. However, the degradation in performance could be minimized by using appropriate methods of de-identification. Therefore, we propose, de-identification filters which preserve disjoint regions of the face depending on the type of driver behavior study. For example, a deidentification filter which preserves only the mouth region can be used for monitoring a driver s yawning [21] or talking and a de-identification filter which preserves the eye regions can be used for detecting driver s fatigue [13] or gaze direction. Since an ensemble of facial features is often used to identify a person, de-identification filters which preserve only a subset of facial features and that too in a disjoint manner are promising for privacy protection. In addition to preserving a subset of disjoint facial regions, we explore the advantages of superimposing a head pose encoded face mask to provide spatial context. In this paper, we choose the driver s gaze as the driver behavior to preserve. We implement and compare de-identification filters, which are made up of a combination of preserving eye regions for fine gaze estimation, superimposing head pose encoded face masks for providing spatial context and replacing background with black pixels for ensuring privacy protection, as shown in Figure 1c. The eye region preserving 1

2 (a) Preserving scene [6] (b) Preserving action [1] (c) Preserving gaze direction (this paper) Figure 1. Comparison of selected works in de-identification from different applications: (a) Google street view: removing pedestrians and preserving scene using multiple views, (b) Surveillance: Obscuring identity of actor and preserving action and (c) Intelligent vehicles: Protecting driver s identity and preserving driver s gaze (this paper). filters will also be useful for other works, such as understanding the relationship between eyes-off road and lane keeping ability [14]. Furthermore, with similarly designed filters that match the designer s criteria of what to preserve, institutions may be more inclined to publicly share de-identified naturalistic driving data. The research community can then benefit tremendously from large amounts of naturalistic driving data and focus on the design and evaluation of intelligent vehicles. As consumer vehicles are also instrumented with an increasing number of sensors, looking inside at the driver is raising concern over individual privacy. The first work on deidentification of drivers in naturalistic driving videos can be found in [9], but it lacks the spatial context we present in this paper with the face mask, which shows a significant improvement in driver s gaze-zone estimation. To the best of our knowledge, no other work has explicitly proposed or evaluated de-identification algorithms for inside the vehicle. We say explicitly because some researchers have used a derivative of raw camera sensory data to do further analysis. In [20], Cheng and Trivedi represent data from multiple camera sensors as voxel data and perform occupant posture analysis. Similarly, [19] uses EXtremity Movement OBservation (XMOB) for 3D upper body pose tracking to determine whether the driver s hand is on the wheel. In this study, we intentionally apply de-identification filters on data from a camera looking at the driver, and quantify the level of deidentification and the effects of de-identification on estimating driver s gaze direction. RELATED WORKS The term De-Identification has been popularly used in literature to address the concern of privacy invasion from sensorized environments. A sensorized environment can be anything from when a photographer captures a moment in time where the camera is the sensor and the scene is the environment. The need for de-identifying people in sensorized environments are typically for two reasons, either the person is not intended to be in the image or the presence and action of the person is intended but not their identity. The former is a prevalent reason for de-identification in applications like Google-street view [7, 6], while the latter has applications in surveillance [1], as illustrated in Figure 1. DE-IDENTIFICATION FILTER A de-identification filter is that which takes an image or a sequence of images where a person could be recognizable and makes it unrecognizable, meanwhile preserving the necessary information for which the image was captured. In the driving context, a de-identification filter as applied to a raw image of looking at the driver will ideally output an image where, semantically speaking, the driver s identity is protected and the driver s behavior is preserved. In the 360 panoramic view, the Google street view car captures not only the appearance of location specific objects such as buildings, billboards and street signs but also privacy invasion material such as people and license plates. Google protects individual privacy by introducing a system that automatically blurs faces and license plates [7]. However, recent studies have shown that face is one of many identifiable features associated with people, such as silhouette [3], gait and articles of clothing. To this end, many researchers have proposed to remove persons from the Google street view and replace them with background pixels using multiple views of the same scene [6] or with similar looking pedestrians from a controlled dataset [11]. Challenges of De-Identification in Naturalistic Driving De-Identification of drivers inside the vehicle cockpit presents a few questions, issues and challenges. First, given an image containing the driver s face, whether deidentification should be applied locally inside a region of interest (e.g. the driver s face) or over the entire image. The former is not recommended because if the face is detected or 2

3 Figure 2. Illustrating three different de-identification filters, which semantically share the same goal of obscuring driver s identity and preserving driver s behavior, but in different degrees: (a) the first filter output is on the lower end of privacy because it masks only parts of the face and leaves spatially contextual information (i.e. hair color/length, body shape/posture), (b) the second filter provides more privacy while still preserving gaze, and (c) the third filter preserves only deduced information and therefore provides the highest privacy among the three. tracked incorrectly, then de-identification on the wrong portion of the image leaves the driver s face vulnerable for identification. Robustness of face tracking algorithms is made difficult by the changing illumination conditions and large spatial head movements present in typical naturalistic driving data. Therefore, the latter proposition of applying de-identification uniformly to the entire image is more ideal for its limited or lack of dependence on face detection or tracking modules. It would, however, mean sacrificing resolution or key details in the background, which do not reveal the driver s identity. For example, distorting the whole image can deteriorate the performance not only of recognizing driver s identity, but also of inferring whether the driver is wearing a seat belt or in inferring driver s hand movements. This leads to the second issue of what should be preserved in the process of de-identification. Driver fatigue monitoring systems would benefit largely from preserving mouth behavior such as the number of times the driver yawned [21], and eye behavior such as the proportion of time the eyes are closed (PERCLOS) [13]. In addition to fatigue monitoring, preserving eye gaze behavior can be used as a proxy to determine what information the driver is processing [17]. For example, coarse gaze direction estimation is a good indicator of driver s intent to change lanes [4, 5]. Figure 2 illustrates three different de-identification filters which share the same goal of obscuring driver identity and preserving driver behavior, but in varying degrees. Therefore, depending on the study of driver behavior, specific de-identification algorithms can be designed. Evaluating the degree of de-identification is a key part of designing the filter because failure to provide adequate protection of privacy is unacceptable. Evaluations can occur in one of two ways: human user study and algorithmic face recognition. While the latter is more objective, it does not compare to a human s ability to recognize faces. In a user study for face recognition, participants are typically asked to match the driver in a de-identified image against a pool of possible candidate photographs. One of the main issue in designing this user study is in choosing the candidates. It would be ideal that not all individuals represented in the deidentified images be in the list of candidates and vice versa. This represents a realistic situation of person identification where the subject in real life encounters many unknown faces to match with known (candidate) faces. De-Identification by Parts, with and without Spatial Context In this study, we are interested in protecting the driver s identity and preserving the driver s gaze direction. There is a trade-off, however, between accurately and robustly estimating driver s gaze and protecting the driver s identity, because the same facial features that are useful for gaze estimation play a key role in recognizing a person s identity. We explore methods to de-identify the driver yet allow for gaze estimation by preserving key facial regions in the foreground and obscuring other regions in the background. Gaze estimation can be accomplished using any one of the combinations of facial regions in the first row of Figure 3, where some are more robust than others to large head movements, facial deformations, lighting conditions etc. While presenting these facial regions in a disjoint manner helps to hamper face recognition, it also hampers gaze estimation. However, superimposing these disjoint facial regions onto a generic face model, as shown in the second row of Figure 3, may provide the necessary spatial context for gaze estimation. In this study, we explore the benefits of preserving the region around the eyes while either replacing the rest of the face region with black pixels or an appropriate face mask. To extract the location of the eyes, facial features are reliably detected using supervised descent method [22]. These facial landmarks are also used to estimate head pose. Head 3

4 Figure 3. Illustration of different combinations of patches around facial landmarks to estimate or predict driver behavior (i.e. driver s gaze, driver is talking) while protecting driver s identity. The second row is the same as the first row but with an underlying face mask to provide spatial context. However, some combinations are more susceptible than others to face recognition. Figure 4. Illustrates the five gaze zone regions of interest: Left, Front, Right, Rear Mirror, and Inside. drivers in two user studies, face recognition and gaze zone estimation. Experiment Design pose is computed using seven facial landmarks (eye corners, nose corners and nose tip) and their corresponding points on a 3D-generic face model [8]. Using head pose, appropriate face masks are generate for each image. To generate a face mask, a 3D mean face model is rotated using head pose, is scaled using the distance between the detected eye corners in the 2D image plane, is aligned using the nose tip and is finally projected onto the image by discarding the axis value that is perpendicular to the image plane. Furthermore, using head pose, a de-identification scheme which preserves only oneeye can determine which eye is more visible from the camera perspective. The LISA-A tested [16] is used to collect data of four drivers driving on urban roadways and on freeways around the University of California, San Diego (UCSD) campus. Among the collected sensory inputs, one is a video feed from a camera mounted to the left of the driver on the front wind shield near the A-pillar. While each drive lasted approximately 20 minutes, we choose sample images from the video sequence where the driver is looking at particular gaze zones to conduct the face recognition and gaze zone estimation user study. Figure 4 illustrates the five gaze zones considered: Left, Front, Right, Rear Mirror, Inside. A total of 20 human subjects were asked to participate in this two part user study: 10 participants for the face recognition study and the other 10 participants for the gaze-zone estimation study. In the user study for recognition of faces in deidentified images, we used five images of each driver for the five gaze zones times two for two types of de-identification, One-Eye and Two-Eyes. In addition, approximately 5 random images from four other drivers were used to increase variability in the dataset. So there was a total of 5 x 4 x = 78 de-identified images presented to each of the ten participants. Figure 5 shows the layout of the user study as seen by participants. When facial feature tracking becomes unreliable and deidentification is only applied to the estimated face region, parts of the true face region becomes exposed and vulnerable to recognition. Some possibilities include diffusing [15] or scrambling the background (i.e. non-face region). Visual querying, however, raises concern that any form of deidentification on the background, except for pixel replacement, could provide some information towards driver s identity. For this reason, we replace everything around the detected face region with black pixels. However, removing background information can remove contextual information often helpful in e.g. determining driver gaze (look) zone. Given a test sample, participants choose one of the candidates who best identifies with the person in the de-identified image. The candidate images are random pictures and not taken from the collected dataset. Among the 12 candidates, the image with a question mark is available for use when the participant could not conjecture who is in the de-identified image. The participants were instructed that people in the de-identified images may not necessarily be available as one of the candidates and not all candidates are necessarily represented in the de-identified images. This represents a realistic situation of person identification where the subject in real life encounters many unknown faces to match with known (available) faces. Therefore, a thorough experimental analysis is conducted using three de-identification methods, where the background (i.e. non-face region) is replaced with black pixels: the first preserves the region around one-eye with black pixel replacement for the rest of the face region, the second preserves the region around both eyes with black pixel replacement for the rest of the face region and the third preserves the region around both eyes with superimposed face mask on the rest of the face region. These de-identification methods, henceforth, will be referred to as One-Eye, Two-Eyes and Mask with TwoEyes, respectively. The second user study is comprised of nine expert participants classifying the gaze of the driver in the de-identified image into one of six categories: Left, Front, Right, Rear Mirror, Inside, and Unknown. These experts represent re- EXPERIMENTAL EVALUATION In the following sections we describe the collected dataset and present participants response to de-identified images of 4

5 Figure 5. Layout of face recognition toolbox for user study. Given a deidentified image, participants choose one of the 12 candidates that best matches the driver in the de-identified image. Table 1. Face Recognition User Study: Statistics on Participants Response De- Drivers Samples RecognitionUnknown Identification Method Rate Rate % 34% % 38% One-Eye % 38% % 46% All 200 5% 39% % 46% % 40% Two-Eyes % 30% % 54% All 200 8% 43% searchers who are familiar with the camera perspective. By watching unperturbed videos of drivers not in this study but from the same camera perspective, the expert participants were familiarized with estimating gaze zones. This is especially challenging because the camera position is biased to the left. Testing samples contain five images of two drivers for each of the five gaze zones times three for all three types of de-identification. A total of 5 x 2 x 5 x 3 = 150 de-identified images is presented to each participant. Face Recognition Performance First step of evaluation, one participant took part in the user study for recognition of faces on images before they were deidentified. With a 100% recognition rate, we claim the pictures of candidates satisfactorily represented the raw images of looking at the driver. Second step of evaluation tested the level of face recognition with two-types of de-identification methods: One-Eye and Two-Eyes. De-identified images by way of face mask superimposed with the actual eyes of the driver on a black background is not Table 2. Gaze Zone Estimation User Study: Statistics on Participants response De- Identification Method Gaze-zone Samples Accuracy Unknown Rate Left 90 67% 1% Front 90 82% 0% One-Eye Right 90 76% 9% Rear Mirror 90 67% 0% Inside 90 32% 7% Two-Eyes All % 3% Left 90 92% 2% Front 90 87% 2% Right 90 77% 1% Rear Mirror 90 61% 1% Inside 90 39% 3% All % 2% Left 40 98% 0% Front 40 85% 0% Mask with Right 40 88% 0% Two-Eyes Rear Mirror 40 88% 0% Inside 40 65% 0% All % 0% part of the face recognition study, because it doesn t reveal any more information about the driver s identity than the deidentified images with two-eyes only. On the contrary, fusing facial features from two different sources, one source being the driver and the other source being the mean face model, can only introduce confusion. Table 1 details the number of drivers, the number of samples accumulated over all participants per driver, the recognition rate and the percentage of times participants responded with Unknown for de-identification with One-Eye and Two-eyes. Given there are 12 possible candidates to choose from, random chance of recognition is 1/12 = 8.3%. Table 1 shows the mean recognition rate is less than or equal to chance for most of the drivers considered. On average, the recognition rate is higher when using the Two-Eyes de-identification method than when using the One-Eye de-identification method, as expected, however, both are below chance level. It s important to mention that the nature of the experiment where the choices are given to pick one from is very conservative and subjects could use elimination tactics without actually identifying the driver. For example, one participant noted that eye color (e.g. dark or light) was one of his criteria for choosing a candidate. Despite this, recognition rate of any particular driver as well as overall is at most at chance level. Gaze Zone Estimation Performance The gaze zones are as illustrated in Figure 4: Left, Front, Right, Rear Mirror and Inside. A total of nine expert participants classified the driver s gaze direction as perceived in the de-identified images of looking at the driver. Table 2 lists the number of samples over all participants, the classification accuracy, and the percentage of times participants responded with Unknown for each method of de-identification for each 5

6 Left Front Right Rear view Down Left Front Right Rear view Down Left Front Right Rear view Down Left Front Right Rear view Down Left Front Right Rear view Down Left Front Right Rear view Down (a) One-eye (b) Two-eyes (c) Mask with two-eyes Figure 6. Confusion matrix for the five gaze zone classification by participants on de-identified images with (a) one-eye (b) two-eyes and (c) two-eyes superimposed on face mask. Gaze zones are as depicted in Figure 4. Each row represents true gaze and each column represents the participants estimate of the gaze zone. On average, gaze zone estimation accuracy is 65%, 71% and 85% for de-identification with one-eye, with two-eyes and with two-eyes superimposed on face mask, respectively. gaze zone. Given there are five gaze zones, random chance of gaze zone accuracy is 1/5 = 20%. As shown in Table 2, gaze zone accuracy is well above chance for all three deidentification methods for all of the gaze zones considered. On average, gaze zone estimation was 65%, 71% and 85% with One-Eye, Two-Eyes and Mask with Two-Eyes, respectively. Confusion matrix, as given in Figure 6, gives insight into the misclassification of gaze zones. For instance, Rear Mirror gaze-zone is significantly confused with Right gaze-zone, because it is not incorrect to assume a rightward gaze when a driver is gazing at the rear-view mirror. Similarly, Inside gaze is significantly confused with Front gaze and Right gaze. It is expected since gazing at the gauge and instrument panel invoke gazes similar to Front and Right, respectively. Some of these misclassifications, however, decreased when presented with more spatial context. For instance, Left gaze-zone misclassification is high when using One-Eye but decreased significantly when using Two-Eyes with and without the mask. On the other hand, Rear-view gaze zone misclassification is consistently large when using One-Eye and Two-Eyes but decreased when using Mask with Two-Eyes. CONCLUDING REMARKS In the design of driver assistance system, when looking at the driver, driver s identity is irrelevant to understanding and predicting driver behavior. We explored three de-identification schemes, which are made up of a combination of preserving eye regions, superimposing head pose encoded face mask and replacing background with black pixels. Eyes especially, because it can provide finer detail on gaze zone estimation. A two part user study using human participants showed face recognition to be well below chance and gaze zone estimation accuracy to be 65%, 71% and 85% for One-Eye, Two-Eyes and Mask with Two-Eyes, respectively. Acknowledgment We acknowledge support of the UC Discovery Program and associated industry partners. We also thank our UCSD LISA colleagues who helped in a variety of important ways in our research studies. Finally, we thank the reviewers for their constructive comments. REFERENCES 1. Agrawal, P., and Narayanan, P. Person de-identification in videos. Circuits and Systems for Video Technology, IEEE Transactions on 21, 3 (2011), Campbell, K. Analyzing driver behavior using data from the shrp 2 naturalistic driving study, May Collins, R. T., Gross, R., and Shi, J. Silhouette-based human identification from body shape and gait. In Automatic Face and Gesture Recognition, Fifth IEEE International Conference on (2002). 4. Doshi, A., and Trivedi, M. Investigating the relationships between gaze patterns, dynamic vehicle surround analysis, and driver intentions. In Intelligent Vehicles Symposium, 2009 IEEE, IEEE (2009), Doshi, A., and Trivedi, M. M. Head and eye gaze dynamics during visual attention shifts in complex environments. Journal of vision 12, 2 (2012), Flores, A., and Belongie, S. Removing pedestrians from google street view images. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE (2010). 7. Frome, A., Cheung, G., Abdulkader, A., Zennaro, M., Wu, B., Bissacco, A., Adam, H., Neven, H., and Vincent, L. Large-scale privacy protection in google street view. In Computer Vision, 2009 IEEE 12th International Conference on, IEEE (2009), Martin, S., Tawari, A., Murphy-Chutorian, E., Cheng, S. Y., and Trivedi, M. On the design and evaluation of robust head pose for visual user interfaces: algorithms, databases, and comparisons. In Automotive User Interfaces and Interactive Vehicular Applications (2012). 9. Martin, S., Tawari, A., and Trivedi, M. M. Toward privacy-protecting safety systems for naturalistic driving videos. Intelligent Transportation Systems, IEEE Transactions on (2014). 10. Neale, V. L., Dingus, T. A., Klauer, S. G., Sudweeks, J., and Goodman, M. An overview of the 100-car naturalistic study and findings. National Highway Traffic Safety Administration, Paper, (2005). 11. Nodari, A., Vanetti, M., and Gallo, I. Digital privacy: Replacing pedestrians from google street view images. In Pattern Recognition (ICPR), st International Conference on, IEEE (2012), Ohn-Bar, E., Tawari, A., Martin, S., and Trivedi, M. Vision on wheels: Looking at driver, vehicle, and 6

7 surround for on-road maneuver analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2014), Pei, Z., Zhenghe, S., and Yiming, Z. Perclos-based recognition algorithms of motor driver fatigue. Journal-China Agricultural University 7, 2 (2002), Peng, Y., Boyle, L. N., and Hallmark, S. L. Driver s lane keeping ability with eyes off road: Insights from a naturalistic study. Accident Analysis & Prevention 50 (2013), Perona, P., and Malik, J. Scale-space and edge detection using anisotropic diffusion. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 7 (1990), Tawari, A., and Trivedi, M. M. Head dynamic analysis: A multi-view framework. In New Trends in Image Analysis and Processing ICIAP Springer, 2013, Taylor, T., Pradhan, A., Divekar, G., Romoser, M., Muttart, J., Gomez, R., Pollatsek, A., and Fisher, D. The view from the road: The contribution of on-road glance-monitoring technologies to understanding driver behavior. Accident Analysis & Prevention (2013). 18. Tian, R., Li, L., Chen, M., Chen, Y., and Witt, G. Studying the effects of driver distraction and traffic density on the probability of crash and near-crash events in naturalistic driving environment. Intelligent Transportation Systems, IEEE Transactions on (2013). 19. Tran, C., and Trivedi, M. M. 3-d posture and gesture recognition for interactivity in smart spaces. Industrial Informatics, IEEE Transactions on 8, 1 (2012), Trivedi, M. M., Cheng, S. Y., Childers, E. M. C., and Krotosky, S. J. Occupant posture analysis with stereo and thermal infrared video: Algorithms and experimental evaluation. Vehicular Technology, IEEE Transactions on 53, 6 (2004), Wang, Q., Yang, J., Ren, M., and Zheng, Y. Driver fatigue detection: a survey. In Intelligent Control and Automation, WCICA The Sixth World Congress on, vol. 2, IEEE (2006), Xiong, X., and De la Torre, F. Supervised descent method and its applications to face alignment. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (2013). 7