Introduction to Remote Sensing Lab 6 Dr. Hurtado Wed., Nov. 28, 2018

Transcription

1 Lab 6: UAS Remote Sensing Due Wed., Dec. 5, 2018 Goals 1. To learn about the operation of a small UAS (unmanned aerial system), including flight characteristics, mission planning, and FAA regulations. 2. To fly a remote sensing mission with a UAS. 3. To obtain and process visible-wavelength imagery from a UAS for use in SfM/MVS (structure-from-motion/multi-view stereo) photogrammetry. 4. To obtain and process near-infrared and visible multispectral imagery from a UAS for use in NDVI vegetation mapping. Introduction In this lab we will acquire remote sensing data using a UAS. We will do this at a field site of geologic interest on UTEP campus. Our flights will be used to obtain highresolution aerial imagery in visible and near-infrared wavelengths. These data will be processed to produce 3D models (in the form of point-clouds) and derived data products to include orthorectified image mosaics and gridded digital elevation models (DEMs). These data products will be further processed to produce a topographic contour map of the field site as well as an NDVI vegetation map. Background 3DR Solo Quadcopter UAS This link has the specifications for the 3DR Solo (Figure 1): It is a consumer-grade quadcopter UAS that is designed to carry a GoPro camera (see below) and similarly-sized payloads. It is capable of being flown manually (using a remote control and a mobile device app) and also autonomously. In autonomous mode, it can fly itself over a preprogrammed, GPS-guided flight path (see below). 1

2 Figure 1. 3DR Solo UAS. GoPro Hero 4 Black High-Definition Video Camera This link has the specifications for the GoPro Hero 4 Black high-definition camera (Figure 2): It is a consumer-grade action camera that works seamlessly with the 3DR Solo. It is capable of taking both still images and video at resolutions up to 4K (3840 x 2160 pixels at up to 30 frames per second [fps]). We will use an unmodified GoPro camera that takes truecolor visible-wavelength images i.e. it has the same red-green-blue wavelength sensitivity as the human visual system. While the GoPro is capable of taking still images, for simplicity we will use the camera to shoot 4K video at 30-fps and extract still images from the video for use in our data processing (see below). Figure 2. GoPro Hero 4 Black camera. Back-Bone Ribcage-Modified YI 4K High-Definition (Infared) Video Camera The following links have information about the modified YI 4K high-definition, infrared-capable camera (Figure 3): This device is a heavily-modified version of the YI 4K consumer-grade action camera. Like the GoPro, the unmodified version of the YI 4K is capable of taking both still 2

3 images and video at resolutions up to 4K (3840 x 2160 pixels at up to 30 fps). The modifications to the YI 4K camera we will use were done by a Canadian company called Back-Bone, whose Ribcage modified cameras are typically used in applications where non-standard, detachable lenses, filters, and other optical elements are desired. particular, the modification allows the camera to take images in the near-infrared. In Figure 3. Modified YI 4K camera. Note that the CCD sensors used in most consumer-grade cameras everything from GoPros, to point-and-shoots, to DSLRs, to your phone camera are already inherently sensitive to near-infrared light. That capability is not useful for taking truecolor visible-wavelength images, so, to make a camera that takes true-color images, near-infrared light needs to be blocked while allowing the visible-wavelength light to pass through. A filter, colloquially referred to as a hot mirror, accomplishes this. Manufacturers install the hot mirror as part of the optical path in front of the sensor, either as a discrete optical element or integrated into one of the lenses. If the hot mirror filter is removed, as in the Ribcage-modified cameras sold by Back-Bone, the sensor will detect the full spectrum of light it is inherently sensitive to, including both the visible and near-infrared wavelengths. Figure 4 shows the native (i.e. no hot mirror) spectral response of the sensor used in the YI 4K camera. Note that the G and R channels are both sensitive to near-infrared radiation in a way that the G and R channels would not be if the hot mirror filter was installed (Figure 5). Comparing Figures 4 and 5 shows that the hot mirror filter also sharpens the sensitivity of the R, G, and B channels so that they are centered at the wavelengths corresponding to red (~650 nm), green (~550 nm), and blue (~450 nm), respectively, with minimal overlap between channels. 3

4 Figure 4. Native spectral response of the sensor in the YI 4K camera. Note the broader spectral sensitivity, particularly in the G and R channels whose sensitivities extend into the near-infrared, than for a sensor with a hot mirror filter installed (c.f. Figure 5). Figure 5. Generic spectral response in RGB for a typical true-color camera. This example shows color film sensitivity, but the spectral responses of the GoPro Hero 4 Black and an unmodified YI 4K (i.e. a digital sensor with a hot mirror filter installed) would be similar. For the NDVI, recall that we ideally want to compare the reflectance in the visible (typically red, but blue works too) to the reflectance in the near-infrared. Note that there is substantial overlap between the spectral response curves for the R, G, and B channels in Figure 4, so, to make the modified YI 4K camera more useful to us for vegetation mapping using the NDVI, it is advantageous to add our own filter to further modify the camera s spectral response. Specifically, we would like to minimize the overlap (or cross-talk ) between the RGB channels. One way of doing this is by using a bandpass filter that is designed to only let in certain wavelength bands while blocking others. (The hot mirror filter is actually an example of a bandpass filter.) The bandpass filter behavior we would like is to not block the light that the R or B channels would be sensitive to, but 4

5 very effectively block the light the G channel would see. It turns out that, rather than an expensive, optical-quality lens filter with those properties, we can get away with using an inexpensive gel filter (the type used for theater lights) instead (Figure 6). Figure 7 shows the resulting spectral sensitivity for the modified YI 4K camera with this gel bandpass filter installed. Figure 6. Spectral response for gel filter. Note that it allows all of the light that the B channel of a camera sensor is sensitive to (c.f. Figures 4 and 5). It also allows all of the light that the R channel is sensitive to, including the near-infrared (c.f. Figures 4 and 5). However, it blocks almost all of the light that the G channel is sensitive to (c.f. Figures 4 and 5). Figure 7. Cumulative spectral response curve for modified YI 4K camera when used with the gel filter (c.f. Figures 4 and 6). Note the sensor is optimally sensitive to visible light centered around 450 nm in the B channel and near-infrared light nm in the R channel, with comparatively less sensitivity in the wavelength region between (~ nm). 5

6 While the modified YI 4K is capable of taking still images, for simplicity we will use the camera to shoot (near-infrared!) 4K video at 30-fps and extract still images from the video for use in our data processing (see below). Mission Planner Software For the purposes of planning our UAS flight, i.e. pre-programming a GPS-guided autonomous flight path, we will use Mission Planner ( The software allows the user to define an area on a map (using Google Earth or other images as a base) by drawing a polygon. Then a tool can be used to automatically generate a grid-pattern flight path to completely image the polygon area, assuming a certain amount of user-defined overlap between images, the field-of-view of the camera, the camera look-angle, and the planned flying altitude and velocity of the UAS. The resulting flight path will be defined by a series of GPS waypoints and associated actions the UAS will take at each waypoint (i.e. take-off/land, change/maintain velocity, change/maintain direction, change/maintain attitude, trigger camera, etc.). Mission Planner can connect to the UAS flight control system via WiFi, and these waypoints can be uploaded as a file to the UAS prior to flight. A pre-loaded mission plan can be triggered with the Mission Planner software during flight. (In fact, the 3DR Solo can be completely controlled, either manually or autonomously, just using Mission Planner on a laptop rather than with the remote control and mobile device app). Agisoft Photoscan Professional SfM/MVS Software Much of the data processing of the UAS data will be done using Agisoft Photoscan Professional ( This software implements techniques known as structure-from-motion (SfM) and multiview stereo (MVS) that together can be used to extract 3D information from sets of overlapping images. Amazingly, this 3D information includes the 3D shape of the target (e.g. the surface of the Earth and any objects in the field of view) as well as the 3D camera locations (e.g. the flight path of the UAS)! (We will read a couple of papers on this subject, so the details are not discussed here.). The resulting 3D model, in the form of a point-cloud (a set of hundreds to millions of points in [x,y,z] space), can be used to make derived data products that can be 6

7 further processed in ENVI to make the final results we want in this lab (see below). It is important to note that the SfM/MVS algorithms take substantial computing power, and processing in Photoscan can take a long time for a large image dataset. Photoscan is also not free software, but we have licensed copies running on various capable computers in the Department (see below). Note that the raw imagery we obtain from our UAS flights will be in the form of video files. Note also that the final products we want are: (a) a topographic contour map of the area of interest; and (b) an NDVI map of the area of interest. A number of processing steps will therefore be required to: (1) convert the video files to sets of still images; (2) produce a geographically-correct (i.e. orthorectified) mosaic of the still images; and (3) produce a 3D model from the still images from which topographic information can extracted. Item (1) can be done using any number of free tools (I use resulting still images can be ingested by Photoscan for SfM/MVS processing. The 3D model produced by Photoscan can be used to accomplish items (2) and (3). Specifically, an RGB orthomosaic can be made as a byproduct of the SfM/MVS process. orthomosaic can be exported from Photoscan for use in ENVI to make the NDVI map. Similarly, the point-cloud can be used to produce a gridded elevation map (a digital elevation model, or DEM). The DEM can be exported from Photoscan for use in ENVI to make a contour map. Further details on this processing are given below and will be discussed in class. The The Instructions Part I: Mission Planning As a class, we will use Mission Planner to design a series of flight profiles, some using the GoPro and some using the modified YI 4K camera, that will achieve complete coverage of the study area with both cameras. (There will probably be something like two GoPro flights and two YI 4K flights.) Collectively, we will decide on the appropriate and safe flight parameters (speed, altitude, etc.) and go over applicable rules and regulations that affect our flight planning. 7

8 Questions: What are the most important considerations in planning a UAS mission? Do you anticipate having to alter the flight plans while in the field (why or why not)? Part II: Data Acquisition As a class, we will fly our pre-planned missions. We will autonomously fly the area of interest multiple times (probably at least four) so that we obtain complete data coverage with both the GoPro and the YI 4K. We will acquire most (if not all) of the imagery from (< ft) overhead with a vertical camera look-angle (i.e. nadirlooking) during autonomous flights. However, we may also have to fly some portions of the area (e.g. where there is steep topography) at low altitudes (<<100 ft), under manual control, and with the cameras at a non-vertical look angle. We will need everyone to play a role in carrying out these missions safely and successfully. Collectively, we will exercise appropriate and safe flight procedures and go over applicable rules and regulations that affect our flight operations while in the field. Following FAA regulations (14 CFR Part 107; for safe and legal flight, we must have a licensed remote pilot-in-command as well as spotters/operators to manually fly the UAS, issue commands to the UAS with Mission Planner, and visually monitor the mission in progress. (All of you will get experience manually flying the UAS, even without a license, as an operator flying under the supervision of the pilot-in-command.) To achieve our mission objectives, we will also need payload operators/monitors to ensure the camera systems are correctly configured, operating correctly, and properly targeted, as well as to download the imagery from the cameras between missions. In addition, prior to our flights, we will need to deploy a small number (~8) ground control point (GCP) targets and also measure their geographic positions (UTM northing, UTM easting, and elevation in meters above sea level) using a handheld GPS receiver. This GCP GPS data will be used to georeference the UAS imagery and the resulting data products (see below). The targets will be retrieved after our flights are complete. 8

9 Questions: What are the most important considerations in executing a UAS mission? What aspects of the flight operations struck you as the most interesting/exciting/boring/surprising/etc.? Part IV: Data Pre-Processing Note: For Parts IV-VI, we will divide the class up into groups of 3 and partition the data into those groups for data processing. How the groups will be partitioned is TBD. Each group will need to do the following: 1. Download video files from the cameras. 2. Convert the video files to still images. 3. Remove non-useful images from the datasets. Questions: What considerations go into deciding how many frames to extract from the video? How do you decide what still images to omit from the dataset? Part V: SfM/MVS Data Processing Agisoft Photoscan Professional is not available in the 4 th floor computer lab, but it is available on 1 computer in the 3 rd floor computer lab, on 1 computer in room 301, and on 2-3 computers in Dr. Giles lab on the 2 nd floor. While there are numerous processing options and functions, the basic workflow is accessed from the Workflow menu in Photoscan. As a class we will go over where licensed copies of Photoscan are installed and the basics of how to use it. Each group will need to do the following: 1. Import the still image dataset. 2. Set camera calibration parameters (use the fisheye option). 3. Align the images (process at low to medium quality). This step finds matching points among multiple images using SfM, resulting in a preliminary, sparse pointcloud as well as a model for the relative location and attitude of the camera for every image in the dataset. This can take Photoscan some time (tens of minutes) to process. 9

10 4. Use the GPS-surveyed GCPs visible in the images (and the preliminary point cloud as a guide), to manually georeference the entire image dataset. Note that you only need to find a given GCP (and enter its GPS location) in one image, and the software will automatically find it in all the other images based on the SfM model generated in the previous step. 5. Re-do the image alignment (process at medium quality). This will georeference the sparse point cloud and determine absolute camera locations. This can take Photoscan a long time (hours) to process. 6. Build the dense cloud (process at medium quality). This step builds a georeferenced, dense point-cloud using MVS. This can take Photoscan a long time (hours) to process. 7. Build a mesh (optional). This step makes a tessellated irregular network, or TIN, from the georeferenced, dense point-cloud. This can take Photoscan a long time (hours) to process. 8. Build a texture map (optional). This step generates a high-resolution texture map from the images that will be draped onto the TIN produced in the previous step for visualization as a photorealistic, volumetric model in 3D. This can take Photoscan a long time (hours) to process. 9. Build a DEM (required for GoPro, optional for YI 4K). Use the default settings, but make note of the spatial resolution (pixel size) of the product. This can take Photoscan some time (tens of minutes) to process. 10. Build an orthomosaic (required for YI 4K, optional for GoPro). Use the default settings, but make note of the spatial resolution (pixel size) of the product. This can take Photoscan some time (tens of minutes) to process. Questions: What were the most time-consuming or otherwise most problematic steps? Qualitatively, how good was your model, DEM, and orthomosaic (i.e. coverage, resolution, presence/absence of artifacts, etc.)? Part VI: Generate Final Data Products For the GoPro data, you will need to do the following: 10

11 1. Export the DEM from Photoscan as a file in geotiff format. Note that the pixel values in this image will be elevations in meters above sea level. 2. Load the DEM geotiff into ENVI. You should edit the ENVI header to ensure the correct pixel size is listed. 3. Use ENVI to generate contours from the DEM using a 0.5-meter contour interval. (Hint: the Overlay menu in the Image window has the function you need you will need to pick an appropriate range of elevations and number of contours). 4. Save your contours as an ENVI vector file. View the vector file in a vector window and save your resulting plot. For the YI 4K data, you will need to do the following: 1. Export the color orthomosaic from Photoscan as a file in geotiff format. Note that the pixel values in this color image will be RGB values, but use Figure 7 to determine the actual wavelengths that go with the R, G, and B channels based on the spectral response of the sensor. 2. Load the color orthoimage geotiff into ENVI. You may find it useful to edit the ENVI header to reflect the wavelength information mentioned above. You should also edit the ENVI header to ensure the correct pixel size is listed. 3. Using your knowledge of ENVI, compute an NDVI image from the color orthoimage. (Hint: remember what you did with Band Math in Lab 5). 4. Apply a color map to your NDVI image to highlight the areas with vegetation and the areas without vegetation. Save your resulting plot. Questions: Comment on the utility of UAS remote sensing for making data products such as these. Is it an efficient process (compared to the alternatives, whatever those may be)? Is it an effective processes (compared to the alternatives, whatever those may be)? How would you describe the quality and accuracy of your resulting data products? How might they be improved? 11

12 What to Turn in and How Prepare a short report. Include in your report responses to the questions/thought prompts posed throughout the instructions. Also include representative images (inline with the text, not as separate files) showing the results of the UAS remote sensing missions and your data processing (to include the final contour map and the final colorcoded NDVI map and any intermediate steps you deem necessary). Only include those images that are necessary to illustrate what you did and your results/interpretations/conclusions. If you do not talk about it in your report, I do not need to see it. Limit your report to no more than 4 typed pages each (normal margins, 12-point font, reasonable spacing), not including the figures and references. Follow the general format of a journal article for your reports, including the formatting and presentation of figures and the referencing style. Note that this means I do not just want a numbered list of answered questions. Instead, write in a narrative style suitable for a scientific paper. Submit your report as a PDF file named lab6 <yourname>.pdf that you should place in the D R O P B O X folder on tejas. 12