Night-time pedestrian detection via Neuromorphic approach

Transcription

1 Night-time pedestrian detection via Neuromorphic approach WOO JOON HAN, IL SONG HAN Graduate School for Green Transportation Korea Advanced Institute of Science and Technology 335 Gwahak-ro, Yuseong-gu, Daejeon REPUBLIC OF KOREA Abstract: - There have been many researches carried out in pedestrian detection field. However, many of the computer vision require high computation making it difficult to implement it electronically while keeping the cost low. Also most of these algorithms are very specific to its target of object and the condition at which it performs. Therefore, it is still difficult to find algorithm that performs well in both day and night time. In this paper, we have approached the pedestrian detection problem at varying luminance from a neuromorphic view. Successful implementation of neuromorphic neuron based on the programmable linear MOSFET conductance has allowed us to keep the computational load low. Also the robustness of the proposed pedestrian detection was confirmed when both pedestrians and cyclists were successfully detected by the system in day and night time conditions. The system showed accuracy rate of over 90% in both day and night time conditions Key-Words: - Neuromorphic circuit, MOSFET, pedestrian detection, night-time, neural network 1 Introduction There have been many researches being done in both academia and industry about pedestrian detection. Pedestrians are exposed to lot of danger on the street as reflected by statistics collected over the years such as, in United States, in 2001, pedestrian fatalities accounted for 12 % of all fatalities[1]. Recently there has been much works done to improve the safety of personnel involved with vehicle such as passengers and pedestrians. Many of vehicle manufacturers offer such system, for example; pedestrian detection to assist in emergency braking should the driver fail to acknowledge the pedestrian in time and head and eye detection to give off alarm should the driver be detected as sleeping. However, these systems have limited impact on improving the safety since the system s reliability comes into question. Pedestrian detection problem, for example, is tackled by using different forms of sensors such as radar, cameras or combination of both [2]. The usage of radar allows the user to detect obstacles that are further away than a visual sensor (camera) can however, in a heavy traffic environment or on busy street of major cities of the world obstruction of other objects will reduce the accuracy and the reliability of the result. Use of visual sensor, however, can detect the pedestrian or any other object using right algorithm. There have been many works done in using a camera with computer vision algorithm or a combination of camera and radar to improve the accuracy and reliability. Although, conventional computer vision algorithms are effective in their condition of usage [3], they, at most times, lack the robustness of the human vision. Most of the computer vision s accuracy suffers heavily to the change in luminance and as such, the detection rate at night time is very low. In this paper, the early vision of bio-inspired implementation of primary visual cortex, based on the neuron of Hodgkin-Huxley formalism and the visual cortex experimentation by Hubel and Wiesel [4] are described. Also, the elements of neuromorphic processing of visual cortex are proposed with the orientation tuned map of synaptic weights and the spiking neuron, based on the programmable linear MOSFET conductance. 2 Neuromorphic Processing The environment of pedestrian at which the image is captured may differ greatly between each capture since the vehicle is in motion. There will be a big changes in the luminance, background and object, so a robust detection algorithm is needed to perform accurately under this widely changing environment. Much of computer vision algorithms are effective in their specific usage, however they lack the robustness of human vision and for most times will underperform in varied conditions especially in ISBN:

2 conditions where the luminance level is low such as night-time. 2.1 Primary Visual Cortex Although there is not a definite model of visual cortex, Hubel and Wiesel s research on cat s striate cortex confirmed the idea on the functioning of simple cell. It is from this discovery which motivated various theories of object recognition from characters to complex natural images [5]. One of the findings of Hubel and Wiesel s result was that the neuron in striate cortex of a cat when given input with light orientation will give different intensity of neural spikes as seen in Fig. 1. The finding suggested that the neuron in the visual cortex is fixed on certain orientation but will give certain amount of a reaction to other orientations than its orientation. complexities in its implementation; however the asynchronous spikes are considered as principle element of high level or large scale neural computing system. The circuit of Fig. 3 was proposed as a voltage-controlled linear conductance circuit by a PMOS transistor M3 and a pair of identical MOS transistors M1 and M2, while the conductance of MOS transistors is one of essential components in the analogue circuit design. The circuit of Fig. 3 has been investigated for various neural networks applications, from implementing synapses to neurons [6]. Fig.2. (a) An electrical equivalent circuit of a neuron, Hodgkin-Huxley formalism (b) dynamics [7] Fig.3. CMOS voltage-controlled linear conductance circuit for neuromorphic VLSI and the chip photograph of test circuit Fig.1. Response of the cat s cortex when a rectangular slit of light of different orientations is shown [4] 2.2 Neuromorphic Implementation These researches on neurophysiology introduced the principles of biologically plausible electronic implementation. The Hodgkin and Huxley s model of neuron, one of the electronic implementation of neuron, which is utilized in this paper to show feasibility of implementation of proposed bioinspired visual processing electronically. The Hodgkin-Huxley (H-H) is a widely adopted idea of neuron s biophysical characterization as shown in Fig. 2. H-H formalism is not used as much in neural networks VLSI as it does not give any major advantages compromising the demanding 2.3 Simulation The neuromorphic system of neuron and synaptic network was designed for evaluating the feasibility of mimicking the primitive behavior of brain neural system in electronic hardware using the CMOS electronic circuit of Fig. 3 [8, 9]. The various input stimuli are six 50 x 50 pixel sized rectangles at different angles as to give same effect as the inputs give to the cat in Hubel and Wiesel s experiment. The simulation was carried out using combination of HSPICE and MATLAB. The result of stimulation shown in Fig. 4 shows consistent outcome as the outcome of Hubel and Wiesel s experiment shown in Fig. 1 where the tuned feature orientations are evaluated. ISBN:

3 the outline of the pedestrian occluded by textile can be clearly seen. Fig.4. Simulated neuron behaviour of visual cortex in Fig. 1 by the CMOS neuromorphic neuron based on the model of Fig. 2 and circuit of Fig. 3 3 Pedestrian Detection The pedestrian detection system has similar steps as to any computer vision algorithm such that features are extracted then used to make detection against its set of database of the object as seen in Fig. 5. However, the feature extraction and how the detection is made are fundamentally different to the systems implemented in computer vision fields. Fig. 6. From left to right: close-up of pedestrians - heads and torsos are shielded; the image after enhanced neuromorphic orientation processing The proposed pedestrian detection was run on Daimler s data set [10]. The resolution of the image is at standard definition of 640 by 480. The Fig. 7 below illustrates the each process of the pedestrian detection system. The input image is first converted into grey scale image followed by orientation feature extraction. The activity level of the image shown on bottom left of Fig 7, shows that the highest activity level is in the middle, which is where the pedestrian s head is positioned as shown by the final, bottom right, image in Fig. 8. Fig. 5. From left to right: neuromorphic vision for human detection, inspired by the visual cortex; the neural network detector of head-torso shape The initial step in the pedestrian detection is to extract orientation features from the input image using the neuromorphic approach mentioned in previous section. Once the features are extracted, neural network with a head-torso template is applied to obtain saliency image where winner-takes-all approach is applied to obtain an activity level map of the image. The performance of orientation feature extractor can be seen on an image of occluded pedestrian in Fig. 6. In the resulting image after neuromorphic orientation feature extraction, as shown in Fig. 6, Fig. 7. From top left to clockwise: original image; image after neuromorphic processing; detected pedestrian s head located in the original image; probability of detected head area of pedestrian. Other participants on the road other than the pedestrians are cyclists which can be under danger. So in order to test the robustness of the system and to increase the safety of other road participants than the pedestrians, the system was tested on detecting cyclist. For this detection, the image was captured from an ordinary vehicle-purpose DVR with the high definition of 1280 by 720 and the frame rate of 30 frames per second. Same template as the pedestrian ISBN:

4 was used in detecting the cyclist since the torso and head of the cyclist is visible. The actual size of the template used was 31 by 64. The detection result shown in Fig. 8 was from an image captured in London, UK and even though the image was captured in sunny day time, ground level has darker contrast compared to the sky. This difference in contrast can, at times, be problematic for some algorithms relying heavily on luminance information. Before the orientation feature extraction process, the input image was cropped such that the frame of the vehicle and the sky was not included in the process. This was done on the assumption that pedestrians must be on the ground plane or close by it. As expected in the orientation extraction image in Fig. 9, there is lot of high frequencies which suggest that they are pedestrian when, in fact, it is due to lightings in low luminance that makes such result. Fig. 10. Saliency image of Fig. 9 and its activity level map Fig. 8. From top left to clockwise: original image, image after enhanced neuromorphic processing, saliency map related to head-torso, probability of detected head area of human object In Fig. 8, the head area of human object is located as a white patch in bottom left image, where the patch is evaluated as the summation of local saliency map and operated by the threshold value. The corresponding location in original image shows the correct detection of cyclist. The distance range of detection is determined by the scale of neuromorphic processed image, to match the NN detector. Some extent of the system s robust could be confirmed by successful detection of different objects (cyclist and pedestrian) using the same system. However, biggest concern so far has been detection of pedestrian in night-time or at the time frame where the luminance level is low such as dusk and twilight times. The night-time image was captured from London, UK with resolution of 1280 by 720 and 30 frames per second as seen in Fig. 9 below. Fig. 9. The image captured (left) and the orientation features extracted from it. The saliency image seen in Fig. 10 shows lot of possible candidates but they are mostly high frequencies from vehicle and shop lights. Thus in this case, for the winner-take-all process we applied a head template of a pedestrian to reduce the noise. An assumption made in this process is that the pedestrians heads are not next to each other like one would do in, for example, a group photo. Fig. 11. The detection result of pedestrians at night time. Finally, in Fig. 11, successful detection of two close by pedestrians are detected. It is worth noting that IR sensor was not involved in the capture of the image yet successful detection of pedestrian was made which could be quite challenging even for a naked eye. Table 1 and 2 shows the detection rate of the neuromorphic vision system in day and night time respectively. The table shows that the proposed system managed accuracy rate of over 90% from images captured at HD resolution (1280 x 720) and 30 fps. The location at which the images were captured for day and night time were from the same route to have ISBN:

5 similar conditions between the different time except the luminance condition. Table 1. Detection rate for daytime Acknowledgement: This research work was supported in part by the Ministry of Knowledge and Economy of Korea under the Grant for Next Generation Passenger Electric Vehicle Development, and in part by the National Research Foundation of Korea under the Grant F. Table 2. Detection rate for night time 4 Conclusion The neuromorphic visual information processing is proposed and evaluated for the pedestrian detection, which shows the performance of over 90% of detection in simulated test as well as the detection in both day time and night time. The biggest challenge for pedestrian detection or any object detection using visual sensors is the change in luminance over time, where the most of pedestrian detection methods using the ordinary sensor have poor detection rate at night or dusk environment. In this paper, using neuromorphic approach, the improved performance of detection is demonstrated by the successful operation in the night time. It is worth noting that in the night time driving there are still other light sources for human driver such as headlights or etc. The basis for this algorithm came from biological eye and brain to give robustness to the algorithm so that it can be used in different applications without many modifications to the original algorithm. The algorithm is proven to possess such robustness as the human head-torso detection applicable to both pedestrian detection and cyclist detection at the same time. The neuromorphic visual processing demonstrated the advantage of its applicability to various uncertain environment like half-transparent obstacle, in addition to the low cost and reliable implementation in CMOS VLSI based on the earlier test device and simulated evaluation. References: [1] US Department of Transport, Technical Report: Pedestrian Roadway Fatalities, April 2003, HS [2] D. Gerónimo, A.M. López, A.D. Sappa, T. Graf, Survey of Pedestrian Detection for Advanced Driver Assistance Systems, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(7): [3] A. Yilmaz, O. Javed, M. Shah, Object Tracking: A Survey, ACM Computing Surveys, 2006, 38(13):1-45. [4] D. H. Hubel, T. N. Wiesel, Receptive fields of single neurons in the cat s striate cortex, J. Physiol., 1959, 148: [5] M. Risenhuber and T. Poggio, Hierachical models of object recognition in cortex, Nature Neuroscience, 1999, pp [6] I. S. Han, Mixed-signal neuron-synapse implementation for large scale neural networks, Int. Journal of Neurocomputing, 2006, pp [7] M. Hausser, The Hodgkin-Huxley theory of action potential, Nature neuroscience supplement, vol. 3, 2000, pp [8] W. J. Han and I. S. Han, Bio-Inspired Visual Signal Processing based on Neuromorphic Circuit, Proc. WSEAS, 2010, pp [9] W. J. Han and I. S. Han, Tunable linear conductance by two MOSFETs and its application to analogue-mixed VLSI for mobile communication and biologically plausible neuromorphic hardware, Proc. WSEAS, 2009,pp [10] M. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp , ISBN: