Autonomous Optical Guidance System

Transcription

1 Autonomous Optical Guidance System John Ciambriello, Ryan Shoaf, Brandon Staton, and John Fridenmaker Dept. of Electrical Engineering and Computer Science, University of Central Florida, Orlando, Florida, Abstract Real time autonomous target tracking systems require a delicate balance between hardware and software. This paper explains the processes used to build a real time target tracking system using two 168 MHz microcontrollers. The system is split into two components which the optical microcontroller uses a camera to capture a image frame which is stored into SRAM. The image processing software scans the frame for the target, computes the center location, and sends to offset to the flight microcontroller. The flight microcontroller uses the offset to determine the correction necessary to stay on collision course. Index Terms Autonomous, Dual Microcontroller Embedded, Flight, STM32F417IGT6, Tracking. I. INTRODUCTION Hardware and software needs depend on the system implementation and the requirements. Real time systems require either enough hardware to offset the software requirements or simplistic software which meets the system hardware. Although simplistic software to meet system hardware is unusual, this project s requirements met these needs. Since the microcontrollers used in the project have a maximum frequency rate of 168 MHz, the software needs to be fast enough to process the image while maintaining a high FPS. Since the system could be split into two components which could use parallel processing, the system is controlled by two microcontrollers to increase speed, optical and flight microcontrollers. The optical microcontroller controls the image processing, camera, LCD, SRAM, and flight microcontroller. The flight microcontroller controls the data flow between the ultrasonic transducer, gyroscope, and all servos. The optical portion is split into stages, POST, target acquisition, and flight mode. The POST stage verifies all optical components are functioning correctly. The target acquisition phase loops while displaying the image on the LCD until target is selected. The flight stage switches between running the image processing code and capturing frames. The flight portion has a POST phase and does not activate until flight mode. During flight mode, it monitors flight altitude while adjusting the servos controlling the plane when the target offset is received. All processes end once collision or landing mode is activated. Collision mode guides the airplane into the target while landing mode will allow the plane to glide till it lands. A. System Design II. SYSTEM HARDWARE DESCRIPTION The system consists of four circuit boards, mounted in a layered arrangement. Aluminum brackets provide mounting provisions for the system boards, display, camera, and switches to the test airplane. The system uses two STM32F4 microcontrollers. This family of microcontrollers is available with up to 1MB of internal memory, several serial and parallel bus technologies, dedicated timers, and up to 140 general purpose input output (GPIO) lines. The optical microcontroller receives, moves, and analyzes camera data. This data is used to calculate steering advice data that is provided to the flight microcontroller. The camera is connected to the optical microcontroller by means of a standard Digital Camera Interface (DCMI). Simplification of optical target detection is facilitated with a single servo under microcontroller control to maintain level attitude in the roll axis. The optical microcontroller also sends image frames to the display before flight. A one megabyte static ram module is provided to store multiple image frames plus any other semi-persistent data. The image data is then shuttled to the display and SRAM by means of a dedicated 16 bit data bus. The selected microcontroller contains multiple bus interconnects and DMA provisions that will allow transfer of the image data from camera to memory and to the display with very low core processing utilization. The selected display features a 16 bit 8080 architecture interface. This type of interface eliminates the need for several microcontroller frame synchronizing signals, reducing circuit complexity and microcontroller workload. This display will display the center 320x240 pixels of each frame before flight to provide the user with a what you see is what you get (WYSIWYG) aiming interface. The display also provides detailed power on self test (POST) indications for system components and subsystems. The selected camera is the Omnivision This camera is capable of encoding image data in RGB565 format. This data format encodes the color information of a pixel in a manner that does not require color matrix

2 calculations of any kind to determine the encoded value of a single pixel. This is in stark contrast to YUV or Bayer encoding schemes, in which the value of a pixel transmitted from the camera, must be mathematically evaluated with the values of the pixels around it or a series of constants. Direct encoding of the pixel data reduces microcontroller computational workload. Camera settings are configured via the serial Inter Integrated Circuit (I2C) bus communications, which write values to the camera s registers. Camera resolution is a balance between performance and memory storage requirements. It was determined that selecting 640X480 camera resolution strikes the best balance between both considerations. This simple type of camera is much more appropriate than expensive cameras that provide television or compressed still image data that would require much more interface circuitry and/or higher microcontroller workload to convert video data. The flight microcontroller is connected to an ultrasonic transducer, a solid state gyroscopic sensor, the servos installed in the test plane, and the aircraft battery. The flight microcontroller is responsible for directing the aircraft toward the target within allowable aircraft attitude and maneuvering constraints. A solid state gyroscopic sensor and an ultrasonic transducer will provide the flight microcontroller with attitude and altitude measurements, which are transmitted via the I2C bus. Positioning of the aircraft servos is performed by generating pulse width modulated (PWM) signals for each servo which change the position of each aircraft control surface. Typical model airplane servos respond to 50Hz PWM signals with high pulse duration of 1 to 2.1 milliseconds. The flight microcontroller will use the aircraft roll attitude measurement and use a typical aircraft servo to keep the camera level to simplify target detection. The system monitors the voltage level of the propulsion battery and is entirely powered by a separate battery. The power supply board contains six monolithic low dropout (LDO) DC to DC regulators and a single boost regulator. These regulators supply 3.3 volts for system logic, 5 volts for servo power and miscellaneous circuitry, and 28 volts for display backlighting. The Linear Technology LT1963 LDO regulator family was selected to meet all expected voltage and current requirements for 5v and 3.3v and the Texas instruments TPS61040 for 28V, segregate the power connections of servos and logic systems, and provide power enable lines for power sequencing. For safety reasons, a standard radio receiver will remain in the aircraft and be connected to a device that can interrupt the pulse width modulated signal to the motor current controller. The purpose of this configuration is to allow our group to stop the propulsion motor in the event that the aircraft appears to be proceeding in a dangerous manner. This device utilizes the controller board from a standard servo with a fixed resistive divider that will provide a voltage to a small relay whenever the radio transmitter stick is in any other position than dead center. Whenever there is no PWM signal to the motor controller, it removes all power to the propulsion motor. The overall system has been designed to minimize layers of hardware abstraction to reduce complexity and maximize performance. Strategic hardware choices minimize data encoding and processing steps and allow the development team to accomplish does more with less. B. Hardware Testing Testing was performed in a tiered progression. Individual subsystems (camera, LCD, servos, ultrasonic transducer, etc.) were implemented and tested first for applicability and proper operation. Related subsystems were then tested for proper interaction. The ability of the selected microcontroller to configure the camera via I2C and receive data was verified. Testing and verification of the gyroscopic sensor and ultrasonic transducer was performed to verify their usefulness through their data outputs. Microcontroller control of servos was tested while developing code to properly control them. The power delivery capability of the power supplies was tested under load. Current consumption of the system boards was verified before connecting the major system assemblies and applying power. In addition, the team performed data handling and integrity tests with applicable programs, and microscopic inspection to identify physical defects in the assembly of the system boards and that no defective components were received. III. OPTICAL MICROCONTROLLER DESCRIPTION A. Optical System Design and POST Procedures The optical microcontroller is responsible for handling and interpreting real time image data from an image sensor and sending steering advice to a separate microcontroller which in turn steers the plane accordingly. In order for the system to run in real time the ability to transfer large amounts of image data is essential. To handle this process the microcontroller uses a combination of a DCMI (Digital Camera Interface), DMA (Direct Memory Access) and a FSMC (Flexible Static Memory Controller) in order to minimize the time of transfer as well as CPU time necessary. This will

3 allow the processor to continue handling other request and procedures. The optical microcontroller is also responsible for handling the state of its internal systems and the state of the system as a whole. In order to maintain complete control of the system at all times a Finite State Machine was implemented. Figure 1 shows a visual description of how the optical microcontroller works. Fig. 1. This figure shows the operation of the optical microcontroller. The Optical Microcontroller starts up in its POST state after it goes through all of its initialization. In this state, it tests various peripheral components of its sub system as well as communication between the two micro controllers. The first peripheral to test is the LCD connection. This allows for visual feedback for the user to see. Once the LCD has been verified the I2C communication and camera configuration are tested. In order to do this the camera configuration registers are written to then read back and check for matches. Following the Camera test are the memory test and MCU to MCU test. To verify the memory connections are correct and the memory is working, combinations of test are performed here. One test verifies every data line while another test verifies every address line. The values are written to the memory then read back for verification. Since the MCU to MCU communication is a simple UART protocol, we need to verify clock rates are correct and the TX and RX lines are connected in the proper configuration. To test the UART a simple message is sent to one microcontroller and an acknowledgement is sent back. If the acknowledgement is received then the MCU to MCU test was verified. Once the Power On Self Test has been completed, a check for any fault flags is done. If a fault has occurred, the system state is changed to fault and the error is posted to the LCD for the user. If no errors have occurred then the systems state becomes idle. In the idle state the optical microcontroller waits for the user to advance to the next processes in the system. A timeout routine has been implemented to ensure the system does not remain in idle state for an extended amount of time. Once a confirmation has been accepted by the user, the system will change to target acquisition mode. B. Target Acquisition Mode Target acquisition mode is essentially the start of the image processing routines for the optical microcontroller. In this state, the DCMI, FSMC, and DMA are used to stream the image data to the LCD for the user. It will stay in this state until the user presses a button to accept the target. In order to ensure the target is properly captured, a cross hair will be displayed on the LCD. Since the target is selected based on the center pixel, the target acceptance button should only be pressed when the target is centered on the screen. To capture the target, the FSMC and DMA are reconfigured to write to the SRAM instead of transferring the image to the LCD. The DCMI is configured in snapshot mode which allows for the image processing routine to collect the image data and find the target faster. Once the system is configured for snapshot mode, the backlight and communication to the LCD is temporarily turned off in order to save energy. The optical microcontroller will take the center pixel color as a representation of the target and store the value into memory and switch the state of the system to target tracking. Signals are sent to the flight microcontroller to power to the motors and prepares for launch. If the flight microcontroller confirms it is ready, the target tracking begins and the plane is launched. During the target tracking state, an infinite loop consisting of image

4 snapshots being store in memory and image processing routines occur. After image processing, a steering direction signal is sent to the flight controller to adjust its heading and a new image is captured. This process is repeated until the image processing routine determines a collision is about to occur or the target is lost. In either case, power to the motor is shut off. If a collision is about to occur the plane holds its heading for a few seconds and begins its landing routine. Once the signals have been sent to the flight microcontroller to land for either case the optical microcontroller is essentially done and remains idle for the rest of the process. No more images will be captured and processed in this state. C. Optical System Testing The individual component tests for the optical microcontroller correspond to the POST. While each section was developed, the test used to verify correct functionally were used for POST. Other than the POST tests, overall system tests were conducted to ensure proper utility. The LCD and camera display test ensured proper connections and register settings. Reading memory locations where the frame is stored verified correct frame storage as well as target color. The final tests included changing camera registers values to verify the affect on the image frame which were used for image processing. 5) Target must be rigid or not be allowed to move independently. If the target is non-rigid, the calculation for target s center could be fail and cause erratic behavior. Also, if the target is non-rigid, the target might be affected by outside variables such as wind blowing a balloon which will violate the abrupt movements. B. Image Processing Description The image processing is responsible for taking the stored frame from optical processor and finding the center of the target. In order to find the center of the target, the code will calculate the threshold for the target color, and then scan through the image frame until the program encounters a target color. From the first target value, the center is calculated then sent to the flight microcontroller. IV. IMAGE PROCESSING DESCRIPTION A. Image Processing Assumptions The image processing operates under a set of assumptions in order to process the image while minimizing the time. These are the assumptions: 1) The target size or distance may vary to make up for camera resolution since software is pixel based. 2) The target will be symmetrical over 3-dimensional Cartesian coordinate system to ensure rotational invariance. Spherical preferred but software will still compute center. 3) The target will need high contrast. Since optical processing algorithm is based on using color thresholds, target cannot have similar color to surround area since the plane could change target. 4) The target will not produce dramatic abrupt direction changes because target could be lost since slower processing time can allow for target to move out of camera. Although the target target could still be in range of camera after an abrupt movement which will not affect tracking, restricting abrupt movements removes possible failures. Fig. 2. This figure shows the flow diagram of the image processing software. The program is executed by the optical microprocessor. The first step for image processing is calculating the color threshold from the target color which is done for each frame. The color threshold calculates a tight threshold and a loose threshold for red, green, and blue color values. The tight threshold will adjust the target color within 10% while the loose threshold will adjust the target color within 65%. The loose threshold will allow for large gradient values over the target from

5 lighting problems. In order to not select extraneous pixels, a pixel with a loose threshold will not be accepted as a target value unless near a tight threshold. offset position is sent to the flight microcontroller, the height and total width values are compared to 50% of either the height or width of the frame depending which value is smaller. If the height and width are greater than or equal to the set value, then the collision mode sequence will be sent to the flight microcontroller. If a target color is not found during the scan of the frame, then a landing sequence will be sent to the flight microcontroller. Fig. 3. This figure shows the amount of coverage the calculated target threshold covers using double thresholds. If the first pixel of memory is 0x0000, then the code will go into optical fail safe detection. Inside optical failsafe detection, the code will look at five blocks of data, in diagonal sections, and check if the surrounding five pixels are black. If all sections return black, optical failsafe mode will be triggered which sends landing mode to the flight microcontroller. Checking diagonal blocks over the image, the amount of processing time is reduced. If the first pixel is not black, it will go into scan mode which looks for a target color. Once in scan mode, the code will look for the first pixel within color threshold. From the first target color which is set as the start position, the program will loop checking the pixels below until a pixel does not match the color threshold while incrementing the height counter. After the height is calculated, the width starting position is determined by mowing down half the height and looping target color check to the left and right. To determine a closer center the left and right width values are accumulated then halved to determine an offset from the start position. The offset, which is measured in pixels from the center of the frame, sent to the flight microcontroller is calculated as follows. X = (Start Position + offset) x-origin (1) Y = (Start Position + height/2) y-origin (2) Collision Mode will be activated depending on the width and height ratio of the target center. Before the Fig. 4. This figure shows the results of the find target center algorithm. The lines shows what the program sees as the height and width for the target center. After the image processing software completes the scan mode, the next process is to examine the target color. In order for the software to be as accurate as possible, the amount of interference from outside sources should be minimized. Since the software has to execute as quickly as possible, the amount of external interference could not be accounted for internally. Since the lighting and shadowing of the target has a big impact on the camera image because of the gradient and hot spots produced, it needs to be taken into consideration. This problem is corrected by evaluating the target color, after the offset coordinates have been sent, and calculating adjustments to meet a specific value which will allow the software to cover the surface of the target as shown in Figure 3. The adjustments calculated will produce new values for specific registers in the camera which will adjust the contrast, color gain, or brightness to set the image into optimal conditions for image processing. This will allow for flight during varying lighting conditions such as overcast or late afternoon.

6 C. Image Processing Testing Since the image processing software needed test images to ensure proper function, the image format used was RGB888 using portable pixel map (PPM) files. These files would be easy to save to a file and view results of software. The first test was ensuring proper conversion from RGB888 to RGB565 formats since the actual format for the system was RGB565. The file was converted by bit shifting the color channels then anding together in two bytes. Once an image was converted between RGB888 and RGB565, multiple tests for the color threshold were conducted. After trial and error to find the best solution for large gradients, a double threshold was implemented as shown in Fig. 3. The height and width were then calculated based on the target pixels found below the starting point and to the left and right of the center of the height as shown in Fig. 4. V. FLIGHT MICROCONTROLLER DESCRIPTION A. Flight Hardware Control The flight microcontroller is responsible for monitoring and handling adjustments to the autonomous flight of the aircraft. Data is read from an electronic 3- axis gyroscope and is used to determine the current angular velocity of the aircraft around the yaw, pitch, and roll axis. This angular velocity is integrated over time, to calculate and track the real time angular tilt of the aircraft as shown in the following function: Angle += ((ADC RAW / GYRO SCALER) (3) / TIME SCALER) Monitoring of the angular tilt of the aircraft allows for the proper correction of the attitude of the aircraft to keep stable flight. The steering of the aircraft is handled through servo control over the camera mount, ailerons, elevators, and rudder of the aircraft. PWM signals with a period of 20ms and adjustable duty cycles are output on timers to the camera mount, aileron, elevator, and rudder servos which allow full control by the flight microcontroller. Also handled are steering control codes sent over via UART communication with the optical microcontroller which are used to update the flight path of the aircraft. Adjustments to servos are limited by thresholds set to prevent a control code from compromising the stability of the aircraft during flight. In addition to control code flight path adjustments, the flight microcontroller also monitors the attitude of the aircraft during these adjustments, and corrects the attitude of the aircraft if steering adjustments or outside forces have taken the aircraft outside the set safety thresholds. The flight microcontroller also incorporates an ultrasonic transducer which is used to gather real time altitude measurements. Monitoring the altitude of the Fig. 5. This figure shows the flow diagram of the flight microcontroller.

7 aircraft during flight allows corrections to be made in the event the aircraft exceeds set low and high altitude thresholds. Altitude data is also used for purposes of determining during landing when the aircraft is at or very near to the ground. The flight microcontroller is also responsible for monitoring and triggering of the killswitch of the aircraft. If at any time during operation the signal indicating that the killswitch should be triggered is detected, the flight microcontroller adjusts the PWM signal output to the killswitch servo, which in turn cuts power to the propulsion motor. This allows for immediate landing in the event an emergency shutdown is required. B. State and Procedure Control During aircraft operation, various static scalars and function priorities need to be adjusted depending on the current state of the aircraft. These scalars are used to control the throw thresholds of the servos, flight path adjustment thresholds, among other state specific variables. Efficient control over these values is made possible by use of a flight state machine. This allows that, anytime the aircraft changes states, that the specific thresholds and variables set for that state are also active. This helps, for example, during takeoff when control code flight path adjustments need to be made secondary to ensuring the aircraft achieves stable flight. Fig. 6. This shows the initialization and communication check of flight components. The flight initialization state is used during system startup to perform power on self tests to check the various devices of the flight system. Initialization of the gyroscope, ultrasonic transducer, and servos are all performed and confirmed to be working before entering the takeoff state. The init_confirm() function will handle changing to takeoff state after user input that the system is ready is confirmed. If any components fail their tests, the aircraft enters a fault state, where it remains for system debugging. The takeoff and flight states handle control of the aircraft during its flight to the target destination. These states perform similar functions, however priorities and thresholds for each state differ, and their corresponding state variables allow for control over this. Scalars over the degree at which control instructions are obeyed differ between these states, as during takeoff the priority needs to be throughout both the takeoff and flight states, the aircraft s altitude and attitude are constantly monitored. If at any time during these states an altitude or attitude hazard condition arises by exceeding specified thresholds, the aircraft enters an altitude or attitude adjust state respectively. The altitude adjust and attitude adjust states are responsible for quickly and efficiently attempting to correct the altitude/attitude of the aircraft and regain safe and stable flight. Once stable flight is regained, the aircraft returns to its previous state. The collision and landing states handle control of the aircraft during target collision and landing. At the time that the optical microcontroller sends a collision control code, the aircraft enters the collision state which will function to prepare the aircraft for collision with the target. Cutting power to the motor is the main priority of the collision state. Attitude stability functions continue running during both collision and landing states to attempt to bring the aircraft to the ground as smoothly as possible. C. Flight Microcontroller Testing The flight microcontroller individual system testing consisted of checking each component. The first test was to ensure proper control over the servo and verify accurate travel distance. Since these will be controlling the movements of the airplane, it will be necessary to have precise control. The 3-axis gyroscope was tested to validate data during certain positions and the accuracy of the readings. The gyroscope will be used to determine the airplane s angles in each axis during takeoff, flight, and landing or collision, so tests in each position and

8 function were conducted. The ultrasonic transducer was also tested to determine the accuracy displayed. Since the plane will be in close proximity to the ground, it needs to have at least 1-2 feet accuracy. The ultrasonic was also tested for lateral speeds to ensure the planes speed would not interfere with the readings. The microcontroller was also tested while connected and manipulating all components in order to guarantee operating capacity was not exceeded. VI. OVERALL SYSTEM TESTING All software was tested on functioning development boards in order to ensure correct software before testing on system boards to help eliminate any hardware bugs or incorrect wiring. Since only individual testing was conducted by each member for each component, the integrated testing between components was the next step. The testing for the overall system will be done in multiple phases starting with similar components such as image processing and optical system. The image processing software will be installed on the optical microcontroller and tested by capturing an image and evaluating the result. While the image processing and optical microcontroller are being integrated, the flight system and the airplane will be integrated and tested by moving all components. After each component operates properly on the airplane, the airplane will then be tilted and twisted to observe the results and compare to the expected reaction. The next step was to integrate flight and optical components which is primarily testing communication between microcontrollers through UART. Using a stationary target for testing purposes, the system will be test acquisition mode and ensure proper target is acquired. Next, flight mode is activated using the button which can then test both flight and image processing components by moving the airplane. If the target is being tracked properly, the plane will adjust to correct course for collision. Once all subsystems are verified and indoor testing is completed, the complete system will be tested with actual flight tests. systems that form the core of such a guidance system. The price of a single instance of our system, built with commercial off the shelf components, costs less than four hundred dollars. A more compact system using custom integrated circuits and volume pricing would likely reduce the price by fifteen percent. The most attractive aspect of this system is it s tremendous scalability. In its initial configuration, this system will control a model airplane three times the size of our test airplane. Even larger planes could be controlled with simple driver circuits capable of switching significantly higher current. A system this small placed in much larger airplanes minimizes the impact on payload capacity. Intricate and powerful sensor arrays could be mounted in these larger aircraft. BIOGRAPHY John Ciambriello ia a Computer Engineering student graduating in May of His experience in the embedded systems industry has open his eyes to the possibilities and varieties of the embedded systems world in which he intends to continue working in. Ryan Shoaf is a Computer Engineer student with interest in cyber security and embedded systems. He plans to continue with his master in computer networks and computer security. Brandon Staton is a 25 year-old Computer Engineering student. Brandon s career goals are to work in the embedded systems industry for a company such as Lockheed Martin, Google, Microsoft, or Intel. VII. CONCLUSION The technology now exists to create an extremely small, inexpensive autonomous drone guidance system that will guide a typical model airplane to chase a moving target. Readily available microcontrollers all but eliminate the need for custom application specific integrated circuits. Extremely inexpensive digital camera sensors facilitate simple but capable optical processing John Fridemaker is an Electrical Engineering student graduating in May His interests include circuit packaging, physical implementation, and testing.