2009 Second International Workshop on Computer Science and Engineering A Real-Time Driving Fatigue Monitoring DSP Device Based On Computing Complexity of Binarized Image CHEN Xiang Collage of Information Science and Engineering Northeastern University Shenyang, China yizhiye@gmail.com ZHANG Zhifei, SONG Yang, CHEN Renyi Collage of Information Science and Engineering Northeastern University Shenyang, China chenrenyi12345@163.com Abstract This paper presents a driving fatigue monitoring method with machine vision algorithm, which is based on computing complexity. And a DSP system based on this method is also introduced. In this method, we firstly binarizated image which contains driver's facial information, to distinguish face area from background.. Then the binarizated luminance components are horizontally and vertically accumulated for collecting face pixel to draw out a face and eyes's location block. Thus, with accumulated index, analyz the relative location ratio between eyes and index of other facial parts, face width, and eyes horizontal position. Compare these ratios to PERCLOS based thresholds, we can ensure whether the driver's eyes are open or not, and infer drivers fatigue state. Finally, this method has been fulfilled with an intact device based on DSP system. Series of experiments indicate this contactless device has advantages of easy operation, sensitive response and splendid accuracy etc. Keywords-Driving fatigue monitoring, DSP, Computional complexity, Binarization, PERCLOS I. INTRODUCTION Driving fatigue has been a significant factor to traffic accidents. According to American Census and Statistics Department's survey, accidents caused by drowsy driving has accounted for about 20% of the total, and 40% especially in the serious ones, making up 83% of traffic death rate. Another American survey in 2007 also indicated that over 53% of the blamed drivers were too drowsy to control cars. It can be seen that driving fatigue monitoring and alarm is highly important for traffic safety. Currently, there are three kinds of index in driving fatigue monitoring methods: 1) Through driver's physiological signasl to define fatigue state. Physiological signal includsg EEG (Electro Encephalon Gram), electrocardiosignal, myoelectric signal, is the most representable of said three kinds. In these signals, though EEG has always been valued as "golden index", this method needs adhere electrodes to driver's head, and signals can be varing with surrounding factors as well as individuals' difference. When it comes to electrocardiosignal and myoelectric signal, the devices based on this method is simple, easy to apply, and nonintrusive as well as portable, but it also has limitation of sensitivity and diagnosis. 2) Through monitoring car related index to assess driver's fatigue state. A relatively sized method is through a sensor collecting the movement of steering wheel and pedals to define safety index. For example, if the steering wheel stopped moving for a long time, it could be inferred that the driver is sleeping. These methods are mostly based on Sensors & Testing technology, though it can reflect driver's status, the contact sensors are too inconvenient and intrusive to use. 3) Through monitoring driver's particular figures to define the fatigue statues. Figures includes pupils diameter, eyes character, head motion and driving posture. And eyes character has been used by most institutions studying driver's drowsy driving. This standard called PERCLOS (the time ratio when driver's eyes are closed) is regarded as a significant index of driver's fatigue. When people are sleepy, eyelids will blink faster and the time closing eyes will be longer. Research indicates that, in the ordinary course of events, the time eyes are closing is about 0.2 to 0.3 seconds, but when this time will be prolonged to 0.5 to 1 seconds when it comes to a drowsy driver. Now, PERCLOS has been acknowledge as the most efficient, portable and realtime assessment of driver fatigue. In this paper, PERCLOS amount to the determination of driver's fatigue state. [10] The index character of PERCLOS shows in Figure 1. Fig 1. PERCLOS index character In sum, though there are many assessment monitoring driver's fatigue status, but most of them are limited to the level of theory study. And even existed monitoring methods and devices have lots of limitation in application and usage. Individual differences and inconvenient device block commercialization and development. This paper identifies the core element of this problem and 978-0-7695-3881-5/09 $26.00 2009 IEEE DOI 10.1109/WCSE.2009.26 10.1109/WCSE.2009.627 84
situation, presents a practical, real-time and contactless new method to monitoring driver's fatigue status. And fulfill this method with DSP device, preventing traffic accident caused by drowsy driving. II. MONITORING METHOD A. Basic Theory This monitoring method is entirely based on the process and analyse facial optical character to get face index. And through computing complexity to compare with PERCLOS standard to infer drivers state. First, take skin colors in consideration. Though everyone's skin is unique, most share similar color that can help distinguish face from background. When this mechanism comes to a camera image process, according to taken face pixels information varing within certain color range from an image, we can locate the face area. Secondly, with facial character in consideration. Because the color character of eyes balls, pupils and eyebrows, it is easy to find out that eyes area bears sharpest monochrome contrast, while other area tend to unitary color. With said character, we use particular binarization algorithm to locate the eyes area to analyse its state. Finally, with modified binarization processing, face and eyes area can be efficiently converted into mathematic module. Through comparing a serious ratio of face index and setting PERCLOS threshold to infer whether the driver's eyes are closing, if the monitoring result continuously closing, an alarm will be activated. As state above, this method's precondition is to quickly locate face and eyes area to analyse the eyes state. And with implementation embedded DSP (DM642), a monitoring and warning mechanism can be achieved without complex device and contact unit. B. Selection for Color Space The color space in common use is RGB and YCbCr. RGB, composed by three components of R (red), G (green), B (blue), generates different colors by superposing these three components, is in mostly extensive application to display and storage color image. YCbCr color space is a other way of encoding RGB information. The actual color displayed depends on the actual RGB colorants used to display the signal. Y is the luma component and Cb and Cr are the blue-difference and red-difference chroma components. An chromatogram of RGB and YCbCr shows in Figure 2. Fig 2. RGB and YCbCr chromatogram C. Modified Binarization Process Binarization processing is an important basic operation in image process community. Based on the thresholded value, the image can be segmented into a binary image, usually consisting of background and foreground. The aim of binarization process in this monitoring method is first to locate face area and process them into white color and the background into black color. Binarization processing is an important basic operation in image process community. Based on the thresholded value, the image can be segmented into a binary image, usually consisting of background and foreground. The aim of binarization process in this monitoring method is first to locate face area and process them into white color and the background into black color. To modify binarization processing, the fist job is to reduce calculation amount to satisfy real-time demand. A more classic of binarization process method is Gaussian Model, which includes vast double counting of e-index. Actually, large said computing statistics is seriously waste of time and effect the demand of real-time. The binarization process method we take in this paper is through computing statistics of facial skin color to build a similarity matrix based on Gaussian Model. We find that the values of numbers in said matrix decrease from center to around, appearing an elliptic isoplethic curve. Through empirical analysis, we directly take the geometrical existence of elliptic equation to determine the binarized threshold values. Finally with said procession, we reduced 720 530=38.16 of calculation to e-index, using only 0.04s to process an image. While every calculation to e- index take 15um, meaning processing an image of 720 530 take 5.7s. Thus the method we take can be 142.5 times faster than the old, and then satisfied the real-time demand [7]. A binarized face image is shown in Figure3. Fig 3. Binarized face image Then we use this binarization to build a facial skin similarity matrix from an image with a face in it. Because the value range of Cb and Cr in the image is 0 to 255, we need to calculate every pair of (Cb, Cr) value to build a skin similarity matrix of 256 256 pixel value. A serious of analysis of this matrix indicates that, the term values in the center tends to be much bigger and decrease radiantly to the distance, then the values near the edge will almost reach 0. Connecting the similar values in the matrix, an ellipse curve will emerge. In these terms, the bigger the 85
values, the much it will represent skin color [6]. The values distribution is shown in Figue4 (the values in the matrix is just illustrative not actual values). then, every line and row's pixel in the binarized image will be taken into horizontal and vertical accumulation operation. From the first line's accumulated value, find the first line A i with none 0 value and the last Line A j with none 0 value. Area betwenn Line Ai and Line Aj is the face part that required. In a similar way, using vertical accumulation operation, we can find Row Bi and Row Bj. Thus the edge of face could be located, and the face pixel could be intercepted to draw a face location block [1]. An binarized face image with face block is shown in Figure 5. Fig 4. Values distribution of similarity matrix Thus, it was known from consumption analysis of matrix to build a mathematical model to determine face area. Experiments indicates that: The range from 100 to 130 of Cb and 135 to 155 of Cr will be optimum to represent facial character. Namely the binarized threshold value is defined as 100 to 130 of Cb and 135 to 155 of Cr, meaning the prolate axis of the elliptic equation equals130-100=30, minor axis of the elliptic equation equals155-135=20 and the centre of ellipse iscb,cr=115,145. Thus the elliptic equation will be (x-115)/10 2 +(y-145)/15 2 =1 (1) Where: x is Cb value of pixel, y is Cr value of pixel; Substitute Cb and Cr value of pixel to the left part of the equation, if the result is less than 1, the pixel will be evaluated to whiter color (adjust the luminance component of the pixel to 255 ). Otherwise this pixel is determined belong to background and evaluate this pixel to black color (adjust the luminance component of the pixel to 0). So the outcome will be a binarized luminance component matrix. Finally, we can get a binarized image, in which white color representing face are and black color representing background. D. Location of Face In the previous step, according to the facial skin color similarity matrix we can fix the binarized threshold value into a lesser range, namely reduce the ellipse area as small as possible. This process can ensure the white area in the binarized image represents facial skin area. Thus in the face location process, we can omit some complex operation in general usage like Gray-level integration projection. Instead of said complex operation, simple accumulation will be enough, avoiding complex recalculation, the real-time demand is also increased. Through binarization processing, the original image is converted to monochromatic image. In this image, the white section is face area and the black area is background. And Fig 5. Face location block E. Location of Eyes Location of eyes mainly adopt the method of computational complexity. The calculation of complexity can be summarized as: from any number sequence's first number, let the first number minus the second, the second minus the third... till the last number, finally the accumulation of said differences will be the complexity. It can be seen that complexity can reflect the pixel's variation in any row (line). Therefore, we can extract luminance components of color space from the origin image of face location block, and compute them into complexity as (2): a 1- a 2 + a 2- a 3 + a 3- a 4 + (2) Meanwhile, find out the rows whose complexity bears the max. From the view of humans' facial character, the horizontal line cross eyes has the biggest monochrome aberration. So we can use this property to locate eyes area and draw out eyes location block [3]. An binarized eyes image with face block is shown in Figure 6. Fig 6. Eyes location block F. Eyes State Determination We use three rules in the eyes state determination to increase accuracy [2,4,9]. 1) By analyzing the relative location relationship between eyes and other facial parts. According to the horizontal complexity of human face, eyes are 1st peak values, noses and mouth may be 2nd peak values. Normally, the values change in some range. So we can calculate the ratio of 1st peak value to 2nd peak value to determine whether the 86
ration is in certain range of the threshold values. With experiment experience and PERCLOS standard, we can assume that the threshold value is 1.5, and if the ratio is no more than 1.5, the target person closes the eyes. 2) By comparing eyes' complexity to its actual width. As the width of eyes are different from different person, the comlexity changes in a large range, which means the more width of eyes, the larger complexity is. We can calculate the ratio of 1st value to the width of eyes location block (ratio of 1st value to Bi-Bj ) and compare whether the ratio is in threshold range. We assume the threshold is 4 and if the ratio is less than 4, the person closes eyes. 3) By analyzing the eyes' horizontal positon in face. In our experiment, the horizontal position of 1st peak values (eyes complexity) is 1/4 up on the face when eyes are open; the horizontal position of 1st peak values is below middle on the face. So we can just determine whether the horizontal position of 1st peak values ( the number of line of 1st peak values) is in the range of threshold. If not, the target person closes the eyes. If any of the above 3 judgments can prove the eyes are closed, the person can be recorded as fatigue state for once. If the person gets fatigue state for 4 times, we can determine the driver is in drowsy driving. III. SOFTWARE AND HARDWARE IMPLEMENTATIONS A. Segmentation of Module Function To implement the entire monitoring method requires program and debug every different step and function. We regard all the monitoring algorithm as a monitoring processing module that including components as follows: 1. Binarization processing module, used for binarize the image collected by the camera. 2. Horizontal and vertical accumulation module, used for ensure the face edge and intercept face pixels to draw out face location block. 3. Horizontal complexity calculation module, used for draw out eyes location block and find out the first and second complexity peak value. 4. Driving state judgement module, used for calculate ratio of the first to the second complexity peak value, 5. Alarm module, used for warning drivers when they get drowsy. With function segmentation, we can definite software mechanism steps to prepare for encode software process. B. Software Proscess The software process has been shown in previous parts at some extent. And taken as a whole, the concrete procession is shown in the software view as follows: a) Initialize settings, including the ports of observation monitor and camera to ensure signal standards accord with each other. This initialization only operates once after starting up. b) Extra the pixel information to build a similarity matrix. c) Using binarization process to build mathematic module for distinguishing face area from back ground. d) Ensure face edge to intercept face pixels to draw face block. e) Calculate said complexity to draw eyes location block as well as get 1st peak and 2nd peak values. f) Use 3 rules above to determine whether the eyes of driver is open or closed. g) The 3 determination operates in order, if any of the determination shows that the eyes are closed, add to 1 to accumulator. When the value in accumulator is greater than 3, we can determine the driver is in drowsy fatigue; and at this moment, light LED, turn on light activated switch alarm the speaker. If all 3 determinations do not find that the eyes are closed, make the value of accumulator to 0. Only if 4 continuous pictures show that eyes are closed, we can determine that the driver is in drowsy fatigue, by this,we can avoid errors caused by winks. h) The LCD will display the original image with location block for observation. Then we need to operate Step H to Step B repeatly, so this part is infinite loop program. Any loop will take 50ms and process 4 images will take about 0.2s. As people usually wink for 0.05s, this program can output image at rate of 20 frame/s to display real-time effect. C. Hardware Design and Experiment In view of the demands of video capture capability and image process velocity. We choose TI company's TMS320DM642 within DM642 development module produced by ICETEK as the core unit of entire system. It is a low-cost high performance video & imaging development platform designed to jump-start application development and evaluation of multi-channel, multi-format digital and other future proof applications [5,8]. The figure of DM642 board is shown in Figure 7. Fig 7. DM642 development modules In this development modules, DSP chip, video process unit, audio process unit and many peripheral interface ports are integrated together. Based on this DSP system, we build an entire system as shown in Figure 8. 87
Fig 8. System sketch map Where: (1) DSP chip, (2) Video decode chip, (3) Video encode chip,(4) Flash memory, (5) RAM, (6) LED light, (7) Supply unit, (8) Light activated switch, (9) Filter, (10) Digital camera, (11) LCD, (12) Speaker. When the function start, camera collects an image with face information and transport it to decode chip. DM642 will receive the image use program flashed in flash memory to process it. Then the processed image will be encode and displayed on LCD for experiment observation. In said procession, if any time the fatigue index exceed threshold values, LED light will be activated, and light activated switch will control speaker to give an harsh alarm. As the monitoring method presented in this paper uses a serious of threshold comparison, some limitation of adaptive capacity can not be avoided. Therefor, there are some precondition, like sufficient light, naked eyes or colourless glasses, to ensure proper functioning. Amount of experiments indicates that only after 0.5s of long-time eyes closing, the alarm be activated and the false rate is below 8.7%. It means that this device can efficiently monitoring driver's fatigue state to prevent traffic accident. IV. CONCLUSIONS First, a fatigue monitoring method based on Computing Complexity of Binarized Image is presented. Furthermore, with eyes information collected from previous step, we build 3 rules based on PERCLOS to determine eyes state and infer whether the driver is drowsy driving. Then we fulfil the software implementation hardware design and experiment to verify the vitality and efficiency. In the future work, a serious of improvement is proposed to adapt said method and device to multitudinal environment for further generalization. The figure of encapsulated device is Fig 9. Fig 9. Encapsulated device ACKNOWLEDGMENT This work is supported by national innovation experiment program for university students (Grant No. 080136), and National High-tech R&D Program of China (863 Program). REFERENCES [1] Canny J. "A computational approach to edge detection", IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986, 8(6), pp.679~698. [2] Cao ZhanHui, Zhang Ke, LiI YanJun. "Transition Region Extraction Algorithm based on Local Fuzzy Complexity", Fire Control and Command Control, 2008, (1), pp.25-26. [3] Gao Wen, Yao HongXun. "The Method of Principal Component Analysis on Human Face Recognition System[C]", Proceeding of the Second International Conference on Multimodal Interface, 1999, IV, pp.9195. [4] Kittler,J.,Illingworth,J. "Minimum Error Thresholding". Pattern Recognition, 1986, 19 (1), pp.41~47. [5] LI ZhengHui. "Optimization Technology of Image Coding Based on DM642 Evaluation Board", Microcomputer Information, 2009,(19), pp.145-147. [6] Nobuyuki Otsu. "A Threshold Selection Method From Gray-Level Histograms". IEEE Trans On System,Man,and Cy-bernetics. 1979, 9(1), pp.62-66. [7] Sauvola,J,Pietikainen,M. "Adaptive Document Image Binarization". Pattern Recognit, 2000, 33 (2), pp.225~236. [8] Texas Instruments, TMS320DM642 Video/Imaging Fixed-Point Digital Signal Processor Data Manual, SPRS200B, May2003. [9] Q Ji, X.J.Yang. "Real-time Eye,Gaze,and Facepose Tracking for Monitoring Driver Vigilance", Real-Time Imaging, 2002 (8), pp.357-377 (2002). [10] Xin Qin, Song YiWei, Zhu XueFeng. "The Research Development on Driving Fatigue Based on PERCLOS", Techniques of Automation and Applications, 2008, (6), pp.45-47,(2008). 88