Rectifying the Planet USING SPACE TO HELP LIFE ON EARTH

Rectifying the Planet USING SPACE TO HELP LIFE ON EARTH

About Me Computer Science (BS) Ecology (PhD, almost ) I write programs that process satellite data Scientific Computing! Land Cover Classification & Deforestation Detection GPU computing & Artificial Intelligence Working with Plant Labs for 2.5 years

Requirements Mission 1: Image Everywhere, everyday, for everybody. 100s of satellites, each with an 11 Mpix camera ~10x15 km view area, 3-5 meter resolution Move Fast & Break Things - IN SPACE! (As fun as it sounds) Orthorectify ~1 million images per day

Using RGB, what is our initial strength? Time series data analysis! What will I need for time series analysis? Imagery correctly rectified through time! (i.e. trees that don t move) So, we need the best spatial precision (& accuracy) Using space hardware that takes shortcuts In the fastest time

Experimentation Published prior work?? Started with inspiration from GDAL-Correlator Google Summer of Code 2012 project Danger: SURF is patented! Use OpenCV instead Make a GRASS GIS module Fast(er) experimentation in C++ Easy built-in geo vector & raster handling Further analysis can be done in the same environment

Spec also says: 100-200 meters initial spatial error JUST KIDDING (hardware issues) 100 km initial spatial error

How is rectify formed? Minimal interpolation/resampling steps Therefore RAW to orthorectified in one sample Unknown initial location + topography We must use some sort iterative refinement And, to future proof, it must be parallelizable

Rectification Stack For each pixel, we must adjust for: Bayer masking Telescope geometry Orbital geometry Satellite pointing accuracy Topography Listed in sensor-to-ground order We can calculate the first three with a rigorous model Last two must be iterated over Samples calculated from ground location to satellite sensor

Demosaicing RAW (RGGB) More difficult than simple bilinear interpolation High frequencies must match across colors Malvar, Henrique S., Li-wei He, and Ross Cutler. "High-quality linear interpolation for demosaicing of Bayer-patterned color images." Acoustics, Speech, and Signal Processing, 2004. Proceedings.(ICASSP'04). IEEE International Conference on. Vol. 3. IEEE, 2004. Highly parallelizable!

Optical correction Found to be inconsequential with current telescope NO OP (for now)

Orbital Correction Warp the scene to the curvature of the Earth Use PROJ.4 s +proj=tpers Tilted Perspective Simulates a perspective view from a height Input coords in whatever the working projection is Generally UTM

Georectification Originally very human-intensive A three step process pl.area search - 100 km -> 1 km pl.rectify - 1 km -> 20 m OSM snapping - 20 m -> ~5 m (?) Reference imagery: Landsat 8 Automatic tie point extraction Explanation To Be Continued

Orthorectification We *think* we know where the image belongs now So use PROJ.4 again! Calculate intersection angles using XYZ space +proj=geocent Adjust to SRTM, as processed by CGIAR-CSI Relative to average control point elevation

Auto-geo-rectification

Rectification In Depth Automatic ground control point collection Keypoint detection and description - Computer Vision We use OpenCV - STAR & FREAK Agrawal, M. and Konolige, K. and Blas, M.R. CenSurE: Center Surround Extremas for Realtime Feature Detection and Matching, ECCV08, 2008 A. Alahi, R. Ortiz, and P. Vandergheynst. FREAK: Fast Retina Keypoint. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. CVPR 2012 Open Source Award Winner. Both are relatively fast and simple ~50,000 points collected per 10 x 10 km patch of land 10,000 matching points can t be wrong!

Point Descriptors 512-bit vector Information dense Sensitive to rotation & scale Can be compared in hardware with the popcnt() instruction In SSE 4.2 or OpenCL 1.2 (AKA Hamming Distance)

Index Tiles Precompute reference descriptors into an index Reproject Landsat 8 into working UTM zone Uses FLANN (Fast Library for Approximate Nearest Neighbors) Any number of Landsat 8 panchromatic scenes Year round imagery in each index file 15 m (& 3 m) resolution tiles Clouds masked out with the QA band High spatial accuracy (~18 m) Storey, James, Michael Choate, and Kenton Lee. "Landsat 8 Operational Land Imager On-Orbit Geometric Calibration and Performance." Remote Sensing 6.11 (2014): 11127-11152.

Currently +10 million descriptor points on a tile Query 50,000 points in approximately one second with FLANN LZ4 compressed index & database files for file size and I/O speed

Keypoint Match Noise ~150 of 512 bits of the vector may not match Leads to many incorrect tie point pairs RANSAC RANdom SAmple Consensus Create random transforms until one explains most of the data Works with ~50% error PairPare Assume a set of point pairs shouldn t rotate or scale relative to each other If they do, one (or both) of the pairs in the set is a bad match Randomize all matches to create a null hypothesis Similar to RANSAC, but works with ~95% error

Area Search First pass generates a corse match Rectification works at ~3 km max Look for potential matching sets of points in each tile Search is linear in the number of index tiles Also search multiple rotations FREAK keypoints are accurate to +/- 10 degrees of rotation

OSM Snapping Extract lines from the satellite raster SELECT all lines that can be seen from space Target lines the size of two lane roads Create more tiles For each vertex in each OSM line Measure Mutual Information score Compare rasterized OSM and extracted lines Add each matching point pair to the larger key point collection Continue with Landsat 8 rectification

Transformation Equation Develop a set of correction equations x = A + B*x + C*y y = E + F*x + G*y (+ D*z) (+ H*z) Solve the coefficients with OpenCV function cv::solve() 1st order equations work better than 2nd or 3rd order Corrects relative to the average CP elevation for proper orthorectification later Coefficents are saved for fast re-rectification from RAW

Thank you PLANET.COM Seth Price seth@planet.com @planetlabs