Real Time Neuromorphic Camera Architecture Implemented with Quadratic Emphasis in an FPGA

Transcription

1 International Journal of Electronics and Electrical Engineering Vol. 5, No. 3, June 2017 Real Time Neuromorphic Camera Architecture Implemented with Quadratic Emphasis in an FPGA Elizabeth Fonseca Chavez1, Mario A. Ibarra-Carrillo2, Julio C. Sosa1, and Ruben Ortega-Gonzalez1 1 Polytechnic/Escom, City of México, México UNAM/Telecomm City of Mexico, Mexico buzonliz@yahoo.com, maixx@yahoo.com.mx, {jcsosa, rortegag}@ipn.mx 2 wide range of lighting conditions. The output signals of the sensor is a stream of events that encode the positions and the precise timestamps of the intensity changes. Then, an Address-event Representation Protocol (AER) allows communication between the DVS and a microprocessor-based system [2]. Vision systems under biological inspiration, usually are constructed with analog electronic circuits that mimic such neuro-biological architectures present in the nervous system [3]. For example, the fly has extraordinary abilities to detect motion. Reichardt proposed the basic model of the fly s vision nervous system (1956) such as the correlation-based Elementary Motion Detector (EMD) [4]. An EMD is driven by the combination of two neighboring photoreceptors. This combination transforms the motion of an object into two separate signals by a delay. There are others articles with modified model of Reichardt for to improve detection of horizontal movement and reduce system noise. Placing an additional process to each adjacent photoreceptors: log-normal filter [5], high pass filter [6]-[8], low pass filter [6], [8] and [7], logarithmic filter [9], delay exponential [5], and automatic gain adaptable [8]. Most validates these modified with Matlab model. There are another way to validate their modified models and some authors use the FPGA to store massive data and have more processing power, as shown by the articles [8], [9]. This document is aimed at implementing a neuromorphic camera using an inexpensive camera and developing a specific application processor. Thus, the present work is divided into three parts. The first part is the proposal of a new architecture for Reichardt EMD. The second part consists of simulation and comparative testing between classical architecture and the proposed architecture. Simulations are to quantify and validate the capabilities of the new model. The last part describes the implementation of the new architecture in FPGA and testing in real time. The camera used has a 640x480 frame size, delivery, according to the data sheet, up to 30 frames per second. The FPGA will perform its calculations with integer data of 5 bits. The scope of the developments presented below involve an architecture in which the sensors act along a Abstract A neuromorphic camera has one neuromorphic vision sensor which was designed to detect an approaching object in real time. This sensor is the Dynamic Vision Sensor (DVS) which works like a retina, but on a silicon chip. The DVS responds to changes in intensity under a wide range of lighting conditions and offers the possibility of fast, computationally efficient visual processing for navigation in mobile robotics. Reichardt proposed a model of fly s vision such as the correlation-based Elementary Motion Detector (EMD). As a proposal, we present a new architecture for the EMD such that increases sensitivity. We call this new design the "EMD with emphasis". Because the "EMD with emphasis" requires more arithmetic operations than the classic EMD, it is implemented in a pipeline for its operation at a frequency of 25 MHz on a FPGA. The experimental results show that the performance of the proposed model is better than that of the classic model. The tests were successful for motion detection for size, contrast, as well as for movement: horizontal, expansion and rotation. Our system works with 30 fps and, a very low latency, dynamic range of 56dB, a resolution of 640x480. There are two special features: the first implies that the system operates with inexpensive cameras. The second characteristic has to do with the way the VHDL is used, ie, the system is easily transportable between FPGA branches. Index Terms neuromorphic, Reichardt, elementary motion detector, EMD, DVS, FPGA I. INTRODUCTION In recent years, the neuromorphic cameras have become more and more common in machine vision oriented tasks. One neuromorphic camera has one vision sensor was designed to detect an approaching object in real time; this sensor is the Dynamic Vision Sensor (DVS), it solves these problems by using patented technology that works like the human retina. Dynamic Vision Sensors are biologically inspired vision systems that asynchronously generate events upon relative light intensity changes [1]. The DVS is a silicon Chip, it offers the possibility of fast, computationally efficient visual processing for navigation in mobile robotics. A silicon retina camera responds to changes in intensity under a Manuscript received July 29, 2016; revised April 2, Int. J. Electron. Electr. Eng. doi: /ijeee

2 horizontal line. A model that makes detecting both, horizontally and vertically, is not implemented, this is a simplification that allows us to clarify the developments in this Article. A. Classical Model II. EMD MODEL The fly's vision system comprises several thousands of photoreceptors. These sensors are connected to an array of motion sensing neurons. Reichardt [3] proposed the model which approximates the behavior of such neurons. In this model, he called Elemental Motion Detector or EMD. This model is based on the correlation. Suppose this, A1 and A2 are two neighboring photoreceptors and representing the input signals, left and right. B1 and B2 representing the corresponding delayed signals [9]. The EMD consists of two correlators as shown in Fig. 1. The left correlator calculates the product A2B1, which means motion detection right. In turn, the right correlator calculates the A1B2 product, which means the detection of movement to the left. Both correlators are combined by a difference between their effects (A2*B1 - A1*B2). Thus, a positive result indicates movement to the right while a negative result implies movement to the left. The classic EMD model when tested in the MATLAB, showed little sensitivity to certain experimental pattern. Therefore, it was decided to add a quadratic emphasis on motion detection model thus achieved an improvement in sensitivity. The Fig. 2 shows the architecture with III. THE SIMULATIONS To start, Fig. 3a shows the image obtained from classical EMD, when exposed to a pattern that moves leftward camera. Meanwhile, Fig. 3b shows the image obtained by the EMD with emphasis being exposed to the same pattern: we clarify that the pattern was moved manually and trying to maintain the same speed in both tests. The reader can see the difference in the texture of the images obtained. Figure 3. (a) Resulting images from the classical EMD model. (b) Resulting images from the emphasis EMD model Figure 1. Classical EMD model B. Modified Emphasis EMD Figure 2. Emphasis EMD model Because the images shown in Fig. 3 depend on an empirical assessment, it is necessary to define a method to quantify the quality of the two EMD. Then, an experiment is performed to quantify the reliability of both architectures. The experiment consists of two stages. The first stage consists of expose the camera to a pattern that shifts leftward. The second step is to evaluate the percentage of pixels that indicate a movement to the left and must also evaluate the amount of pixels that erroneously indicate a move to the right. The experiment was performed 100 times, this is, 100 frames with a pattern moving to the left of the camera were taken. Fig. 4 shows graphs obtained from the experiment performed on the classical model of EMD. The Fig. 4a indicates the relationship between the percentage of pixels detected as indicators of movement to the left, and the index picture analyzed. The Fig. 4b shows the percentage of pixels erroneously detected as indicating movement to the right, and the index picture analyzed Int. J. Electron. Electr. Eng. 231

3 Figure 4. About of classical EMD model: (a) Relationship between the percentage of pixels that indicate movement to the left and the index picture analyzed. (b) Percentage of pixels erroneously detected as indicating movement to the right, and the index picture analyzed Taking averages of the graphs of Fig. 4, it is obtained that the real left detected is 88%. So then, the percentage of falsely detected left is 12%. So we can conclude that the movement was correctly detected. Fig. 5 shows graphs obtained from the experiment performed on the model of EMD with The Fig. 5a indicates the relationship between the percentage of pixels detected as indicators of movement to the left, and the index picture analyzed. The Fig. 5b shows the percentage of pixels erroneously detected as indicating movement to the right, and the index picture analyzed. Taking averages of the graphs of Fig. 5, it is obtained that the real right detected is 98 %. So then, the percentage of falsely detected is 2%. So we can conclude that not only was correctly detected, a better detection was also performed. Figure 5. About the EMD model with emphasis: (a) Relationship between the percentage of percentage of pixels that indicate movement to the left and the index picture analyzed. (b) Percentage of pixels erroneously detected as indicating movement to the right, and the index picture analyzed IV. IMPLEMENTATION ON FPGA The architecture model involves the construction of two blocks of RAM. So while a memory block stores the image newly captured by the camera, the other block stores a previous image. This second image is the one that represents the delay defined by Reichardt EMD. The memory blocks also have the characteristic of having dedicated writing ports and having reading dedicated ports. Both memory blocks are connected to a block that is responsible for implementing the correlator EMD with Figure 6. Internal structure of the EMD with emphasis The block that implements the EMD with emphasis operates in pipeline mode, as shown in Fig. 6. It is possible for the reader to see that the four products, required by the EMD with emphasis, are performed in a single pipeline stage, that is, four multiplications in parallel, which is possible thanks to the FPGA. In the next stages of the pipeline other EMD operations are performed. The FPGA Cyclone IV used is mounted in the development system DE The Cyclone IV has the ID, EP4CE115 and contains 114,480 logic elements (LEs), the largest offered in the Cyclone IV E series, up to 3.9-Mbits of RAM, and 266 multipliers. The camera used is the OV7670, widespread in the local market. This camera can works up 30 FPS, dynamic range of 56dB, format YUV of 8 bits. Fig. 7 shows the elements to implement and probe the new architecture of neuromorphic camera. The Fig. 7 does not show the monitor which is used to observe what sees the neuromorphic camera Int. J. Electron. Electr. Eng. 232

4 Fig. 8 shows the probes and results: (a) Test Pattern to test sensitivity to rotation. (b) Result of the classic EMD to rotation. (c) Result of EMD with emphasis to rotation. (d) Test Pattern to test sensitivity to lateral displacement. (e) Result of the classic EMD to lateral displacement. (f) Result of EMD with emphasis to temporal displacement. (g) Test Pattern to test sensitivity to the approach and departure. (h) Result of the classic EMD to the approach and departure. (i) Result of EMD with emphasis to the approach and departure. Figure 7. Elements to construct a neuromorphic camera and elements to test it V. RESULTS A. Real Time Experimental Results In this section one experiments are proposed to test the system implemented in the FPGA. The experiment consists of performing three tests using high-contrast patterns: patterns in black and white. For these tests the results delivered by the FPGA implementation of the classic EMD against the results of the implementation in FPGA with an emphasis EMD are compared. B. Discussion The result of simulation with MATLAB shows improved motion detection, with emphasis model. The test was done with webcam and with reproduction of video. The simulation results show that using MATLAB neuromorphic camera based on EMD with emphasis delivers a better quality picture. This can be seen by comparing the graph of Fig. 4 with the graph of Fig. 5. In both figures, the reader can see that the camera may be based on an EMD with emphasis delivery fewer false detections. The results of tests on FPGA show that the EMD with emphasis delivery sharper images and more defined edges. The reader may notice some differences in position of the elements shown in the figure. This is because the images were photographed directly from the monitor. In the case of used to zoom in and zoom out pattern, both EMD showed a slight insensitivity. VI. CONCLUSION The neuromorphic camera needs a dynamic vision sensor to detect a movement, the Reichardt correlator provides low computational cost and efficient solution for motion detection, but for better performance we modified the Reichardt model (the quadratic model emphasis ) for to replace with success the DVS. The experimental result shows that the performance of the proposed model is better than the classic model. Our system has a latency of 0.2 microseconds because the EMD with emphasis was implemented in a pipeline. Lower costs, low storage, no redundant data, and resolution of 640x480. The detection is good for the Reichardt classic model, but with the modified Reichardt the detection is better. The tests were successful for motion detection for size, contrast, as well as for movement: horizontal, expansion and rotation. We have a neuromorphic camera ready for use, with open source. The future work will be to reduction of bits of capture, reduction of noise using a filter, and to create a test module in the FPGA. REFERENCES Figure 8. Test patterns and results of EMD classic camera and with [1] T. Delbruck, V. Villanueva, and L. Longinotti, Integration of dynamic vision sensor with inertial measurement unit for electronically stabilized event-based vision, in Proc. IEEE International Symposium on Circuits and Systems, 2014, pp [2] A. Rios-Navarro, E. Cerezuela-Escudero, M. Dominguez-Morales, A. Jimenez-Fernandez, G. Jimenez-Moreno, and A. Linares- Barranco, Real-time motor rotation frequency detection with event-based visual and spike-based auditory AER sensory integration for FPGA, in Proc. International Conference on 2017 Int. J. Electron. Electr. Eng. 233

5 Event-based Control, Communication, and Signal Processing, 2015, pp [3] C. Mead, Neuromorphic engineering: Overview and potential, in Proc. IEEE International Joint Conference on Neural Networks, [4] W. Reichardt, M. Egelhaaf, and R. W. Schlögl, Movement detectors provide sufficient information for local computation of 2-D velocity field, Naturwissenschaften, vol. 75, no. 6, pp , [5] D. C. O'Carroll, P. D. Barnett, and K. Nordström, Computational models reveal non-linearity in integration of local motion signals by insect motion detectors viewing natural scenes, in Proc. Seventh International Conference on Intelligent Sensors, Sensor Networks and Information Processing, 2011, pp [6] A. Fabrice and F. Nicolas, Optic flow sensors for robots: Elementary motion detectors based on FPGA, in Proc. IEEE Workshop on Signal Processing Systems Design and Implementation, 2005, pp [7] F. Aubépart and N. Franceschini, Bio-inspired optic flow sensors based on FPGA: Application to micro-air-vehicles, Microprocessors and Microsystems, vol. 31, no. 6, pp , [8] J. Plett, A. Bahl, M. Buss, K. Kühnlenz, and A. Borst, Bioinspired visual ego-rotation sensor for MAVs, Biological Cybernetics, vol. 106, no. 1, pp , [9] T. Zhang, H. Wu, A. Borst, K. Kuhnlenz, and M. Buss, An FPGA implementation of insect-inspired motion detector for highspeed vision systems, in Proc. IEEE International Conference on Robotics and Automation, 2008, pp Elizabeth Fonseca-Chávez, Mechanical Electrical Engineer graduated from FES-C UNAM, Master in engineering systems of the Faculty of Engineering of the UNAM. She is Assistant Professor in Telecommunications and Computation Engineering Department of the Faculty of Engineering, UNAM. Her areas of interest: Digital Systems Design, FPGA y CPLDs. She is an IEEE member. buzonliz@yahoo.com Mario A. Ibarra-Carrillo, Mechanical Electrical Engineer graduated from the Faculty of Engineering, UNAM, Master in Engineering with a concentration in Digital Signal Processing, graduated from the Division of Graduate Studies of the Faculty of Engineering of the UNAM. He is Professor Associated in Telecommunications Engineering Department of the Faculty of Engineering, UNAM. His areas of interest: Digital Signal Processing, DSP, Digital Systems Design, FPGA and CPLDs. He is an IEEE member. maixx@yahoo.com Julio C. Sosa received the Engineering degree in Electronic Engineering in 1997 from ITLAC, Mich. Mexico. The M. Sc. in Electrical Engineering, in 2000 from CINVESTAV-IPN, Mexico and the Ph.D. in Technology of Information, Communication and Computation in 2007 from the University of Valencia, Spain. He is Titular Professor at Instituto Politécnico Nacional - ESCOM, México D. F. since His current research interests are computer architecture, signal and image processing, sensor networks and embedded systems. jcsosa@ipn.mx Rubén Ortega-González received the B.Sc. degree in electrical engineering from the Instituto Politécnico Nacional, Mexico City, Mexico, 1999 and the Ph.D. degree in Electronics Engineering from the Universidad Politécnica de Valencia, Valencia, Spain, in He has been professor in Escuela Superior de Computo, Instituto Politécnico Nacional since His main research fields are in modeling and control of power converters applied to the distributed generation in microgrids, and digital signal processing Int. J. Electron. Electr. Eng. 234