A novel biometric signature: multi-site, remote (> 100 m) photo-plethysmography using ambient light.

Transcription

1 Technical Note PR-TN 2010/00097 Issued: 03/2010 A novel biometric signature: multi-site, remote (> 100 m) photo-plethysmography using ambient light. W. Verkruijsse; M.P. Bodlaender Philips Research Europe

2 PR-TN 2010/00097 Authors address W. Verkruijsse HTC34-71 M.P. Bodlaender HTC36-01 KONINKLIJKE PHILIPS ELECTRONICS NV 2010 All rights reserved. Reproduction or dissemination in whole or in part is prohibited without the prior written consent of the copyright holder. ii

3 PR-TN 2010/00097 Title: Author(s): Reviewer(s): Technical Note: A novel biometric signature: multi-site, remote (> 100 m) photoplethysmography using ambient light. W. Verkruijsse; M.P. Bodlaender Boeve, H.M.B., IPS Facilities PR-TN 2010/00097 Project: Unobtrusive Monitoring ( ) Customer: Keywords: Abstract: remote sensing, video cameras, intelligent cameras, biometric authentication, diabetes, remote patient monitoring, unobtrusive monitoring, Biometric, unobtrusive sensing We propose a novel biometric signature based on the principle of multi-site remote photo-plethysmography (PPG). This signature can be measured using video cameras. Traditional PPG uses a contact sensor to measure slight skin color changes that occur with every heartbeat. Recently we have demonstrated that these changes can also be measured remotely with a video camera and advanced video processing software. PPG signals can be collected from multiple body sites by tracking multiple regions of interest in the video. Literature indicates that individual cardiac activity has identification potential. The PPG waveform reflects the heart beat and the unique vascular system at the measurement site. The specific PPG waveforms at multiple body sites (e.g., forehead, cheeks, hands, ears) can together form a uniquely identifying signature that is hard to forge. Advantages of multi-site remote PPG measured in ambient light are:: can be done at large sensor-subject distances (e.g., > 100 m), the signature can be measured with minimal or no cooperation of the subject, as measurement during motion is feasible using advanced video processing software make-up or masks intended to mislead visual identification techniques can be detected, multiple individuals can be analyzed simultaneously, stress-level indicators such as heart rate variability, respiration rate and other vital signs such as arterial oxygen saturation can be extracted from the same signal, multi-site remote PPG integrates with surveillance cameras, making it practical to use. Conclusions: Remotely measured multi-site PPG signals may provide biometric signatures and warrant further investigation. iii

4 PR-TN 2010/00097 iv

5 PR-TN 2010/00097 Briefing Chart

6 PR-TN 2010/00097 Scientific Discussion We propose a new biometric signature that can be acquired at long distances with a video camera. This signature is based on photo-plethysmography (PPG). At the core of PPG technology is the optical detection of the dynamic cardiovascular pulsewave, generated by the heart, as it travels throughout the body 1. PPG is traditionally performed using contact probes and dedicated illumination sources (e.g., pulse-oximetry). Recently, we demonstrated 2 non-contact PPG using a conventional video camera and normal ambient light. As a consequence, PPG sensing can be done at long distances (> 100 m has been demonstrated) and inconspicuously, as no energy projection is needed. ECG and PPG signals are closely related and both carry the cardiac signature. The PPG waveform also carries the local vascular network signature, thereby improving biometric identification. PPG is closely related to ECG (electro-cardiogram), which has considerable identification potential as a biometric signature 3,4. ECG and PPG signals are similar in that they are both difficult to forge. An important difference is that in PPG signals, the cardiac signature is uniquely modified by the impedance network of the vascular system (arterial capillary venous). Cardio-vascular waves thus arrive at different times with different amplitudes and waveforms at different body sites (e.g., cheek, forehead, hand, neck). While a single-site PPG signal may contain a weaker identifying signature than an ECG signal, multi-site PPG records the stronger unique signature of the vascular network with the unique cardiac signature. Extensive literature 5-10 indicates that multi-site PPG reveals unique conditions of the vascular network, expressed by the spatio-temporal distribution of PPG signals. PPG waveforms vary at different body sites. Both the waveform shape (left), amplitude and arrival time (right) are unique individual identifiers. Key advantages of camera based PPG are: The signature can be measured with minimal or no cooperation of the subject as it is a non-contact measurement using only ambient light, make-up or masks intended to mislead visual identification techniques can be detected, the camera allows for straightforward multi-site PPG signal acquisition from the same video sequence, even for multiple individuals within the same field of view, can be integrated with existing surveillance cameras, thus reducing infrastructure investment costs and providing the security operator with visual context information, 2

7 PR-TN 2010/00097 PPG signatures may be acquired from existing recorded video footage, measurements can be placed in a context using video information, for example the influence of posture of the subject can be corrected for (sitting/standing), stress-level indicators such as heart rate variability and other physiological parameters such as respiration, SpO2 may be retrieved simultaneously. PPG can often be detected inconspicuously without cooperation of the subject. Pre-recorded footage can be analyzed for PPG. Illustration of multi person PPG: simultaneous heart rate detection. Predicted accuracy and stability Multi-site PPG theoretically provides a potentially strong signature since it combines the cardiac signature with the unique signature of the vascular impedance network. We anticipate that the uniqueness of multi-site PPG signatures is higher than that of for example an ECG which only records the individual cardiac signature. On the basis of single site PPG we were able to distinguish 4 individuals in a limited scope experiment. Multi-site PPG will drastically increase the information content and accuracy of the signature. As a first order example of multi-site PPG, the degree of left-right symmetry of PPG signals (e.g., left hand, right hand) has been used to identify vascular peripheral diseases 6-9, autonomic dysfunction 10,11 and diabetes 5,8,12 in individuals. A regular photograph (A) reveals no particular features. Multi-site PPG of the region of interest (square) shows distinct spatial variation in the low frequencies (B). Accuracy of multi-site remote PPG measurements (e.g., forehead, cheeks, hands) is robust under realistic lighting conditions. While camera movements (e.g., a hand-held camera, or unstable camera positioning) are straightforwardly compensated for, subject movement is a greater challenge. We have developed advanced video processing algorithms that allow for reliable multi-site PPG signature recording for subjects that move considerably (e.g., walking

8 PR-TN 2010/00097 or talking subjects). The figure below demonstrates that even during heavily moving and challenging illumination conditions the heart rate can be reliably retrieved with our algorithms. In fact, the control (3 lead ECG) suffers more from motion artefacts than the video based method. A video-still of a subject on a bicycle fitness machine. The subject s head rotated and moved from left to right heavily during a sprint at more than 2 rotations per second. The video-based, single-site PPG signal derived heart rate is more robust to motion than a 3-lead ECG (Nexus, inc.). Regarding stability of the signature, it is important to note that instability (irreproducibility) of traditional contact PPG is often due to variations in probe-attachment, probe pressure, or shifting. With the camera-based PPG method such instabilities are eliminated. Subject movement artifacts can be almost entirely eliminated with our video processing algorithms. Instability of the signature caused by physiological variation is of concern. The arteriovenous shunts for example have thermo-regulatory functions and can alter the impedance of the local vascular network and thus the PPG waveform. Similarly, respiration is known to have an impact on low frequency PPG signals 12. However, our preliminary modeling suggests that certain characteristics of multi-site PPG are largely invariant for such physiological changes. A simple but possibly useful enrichment of the PPG signature is to analyze the relative PPG waveforms of the three color channels in a color camera (RGB). Since different colors of light have different skin penetration depth, information regarding mean vessel depth can be retrieved. Multi-site variation of such vessel depths adds to the uniqueness of the PPG signature. Methods of acquisition Acquisition is done with a conventional, preferably color, video camera. In all our preliminary measurements we have used off-the shelf equipment. Even low-end cameras such as web-cams are capable of reliable PPG measurements. We believe that many existing security camera systems are also capable of PPG measurements when using our video processing algorithms. For multi-site PPG at long distances the magnification power of the tele-lens becomes critical. We have reliably measured PPG signals at a distance of 115 meters with a consumer grade digital camera combined with an amateur telescope (100x). Existing binoculars or sniper scopes could easily be coupled to our video processing software to acquire multi-site PPG signals. Predicted acquisition time In principle, a PPG waveform can be determined with footage of just 2 or 3 heartbeats (e.g., 3 seconds). This could provide a robust signature for a nonmoving subject. However, robustness is enhanced if longer acquisition times (e.g., seconds) are used that would measure the unique lower frequency variations. 4

9 PR-TN 2010/00097 Acquisition time for these images was 10 seconds. Sensorsubject distance was 4 m for A and B, and 12 m for C. Note the PPG amplitude differences between volar and palmar hand sides and amplitude variation within the face and neck. For PPG amplitude analysis (as shown above) it is possible to use several relatively short acquisition periods (e.g., 2 seconds) and thus enhance the accuracy of the signature (noise reduction) by combining the acquisition periods. In the case of inconspicuous detection these acquisition periods may be done at different sites with different cameras. Also, images from multiple cameras aimed at the subject from different angles can be combined into one PPG signature. In airports, for example, where camera coverage overlaps, a quasi-continuous measurement may take place even while the subject moves around the premises. Predicted minimum/maximum range of subject in proximity to sensor While there is no minimum range (< 1m is fine), the maximum range of detection is at least 100 m and limited by atmospheric conditions. Heart rate detection at 115 meter has been demonstrated with sub-optimal tele-magnifications. The image shown below is cropped from the original image, wherein only 3% of the total available pixels is used to detect the PPG signal. With more powerful tele-lenses we estimate that multi-site PPG will be possible for at least 100 meter sensor-subject distance. Sensor subject distance: 75 meters. The measurement was made through a glass window (camera indoors, subject outdoors). A clear PPG signal is measured, demonstrating long distance PPG. The image on the left is cropped; only 3% of the total image was used to acquire the PPG signal. Mechanisms to potentially increase range between subject and sensor The accuracy and robustness of the PPG signature depend primarily on image quality. With off-the-shelf telelenses and cameras we have demonstrated PPG on a subject at 115 m. With more powerful tele-lenses, superior camera sensitivities and higher pixel resolution we anticipate that atmospheric conditions (fog, smoke) are ultimately the limiting factors in terms of maximum

10 PR-TN 2010/00097 detection distance. In the case of poor atmospheric conditions the application of infra-red based PPG is likely to improve the signal to noise ratio due to its significantly smaller scattering cross-section than visible light. In good atmospheric conditions the application of visible light is preferred due to its inherently stronger PPG amplitudes. Other factors that influence the maximum subject-sensor range are subject movement, illumination level, and illumination stability. We are confident that our video processing algorithms can minimize the impact of subject movement. We have measured clear PPG signals at very low illumination levels (about 1 Lux, approximately the illumination level provided by a 25 LCD screen at 3 meters). As with subject movement, our algorithms can also minimize the impact of illumination instability. Prior relevant experience The current team working on remote PPG at Philips Research possesses extensive experience in video and digital signal processing, camera and skin optics. Prof. Gerard de Haan is a world renowned specialist in the field of movement estimation of subjects in video, which is of critical importance to the successful acquisition of PPG signals. Arno van Leest, PhD has deep knowledge in the field of watermarking techniques, retrieving minute signals hidden in video. Vincent Jeanne, MSc has independently discovered the remote PPG phenomena and worked in computer vision for unobtrusive emotion detection. Ihor Kirenko, PhD is an expert in video compression and image enhancement. Wim Verkruijsse, PhD has extensive experience in skin optics, and was the first to publish the principles of ambient light remote PPG. The team is supported by a broad complement of professional hardware developers at Philips Research, and has more than 300 first filings. 6

11 PR-TN 2010/00097 References 1 Photoelectric plethysmography of the fingers and toes in man. Proc.Soc. Exp. Biol. Med., 37: , 1937, Hertzman AB. 2 Remote plethysmographic imaging using ambient light. Opt. Express 16: , 2008, Verkruysse, W, Svaasand, LO, Nelson, JS. 3 ECG to identify individuals, Pattern Recognit., 38: , 2005, Israel SA, Irvine JM, Cheng A, Wiederhold,MD, Wiederhold,BK. 4 EigenPulse: Robust human identification from cardiovascular function, Pattern Recognit. 41: , 2008, Irvine JM, Israel SA, Scruggs WT, Worek WJ. 5 Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 28:R1-39, 2007, Allen J. 6 Non-invasive in vivo assessment of changes in peripheral arterial properties with estimation of arterial volume compliance. Physiol. Meas. 28: , 2007, Zheng D, Allen J, Murray A. 7 Similarity in bilateral photoplethysmographic peripheral pulse wave characteristics at the ears, thumbs and toes. Physiol. Meas. 21: , 2000, Allen J and Murray A. 8 Right-left correlation of the sympathetically induced fluctuations of photoplethysmographic signal in diabetic and non-diabetic subjects. Med. Biol. Eng. Comput. 43:252-7, 2005, Buchs A, Slovik Y, Rapoport M, Rosenfeld C, Khanokh B, Nitzan M. 9 Age-related changes in the characteristics of the photo-plethysmographic pulse shape at various body sites. Physiol. Meas , 2003, Allen J, Murray A. 10 Autonomic control of the cerebral circulation during normal and impaired peripheral circulatory control. Heart 82:365 72, 1999, Cencetti S, Lagi A, Cipriani M, Fattorini L, Bandinelli G, Bernardi L. 11 Microvascular optical assessment confirms the presence of peripheral autonomic dysfunction in primary biliary cirrhosis. Liver Int. 29: , 2009, Stevens S, Allen J, Murray A, Jones D, Newton J. 12 Respiratory variations in the reflection mode photo-plethysmographic signal. Relationships to peripheral venous pressure. Med Biol Eng Comput. 41:249-54, 2003, Nilsson L, Johansson A, Kalman S.