INTRODUCTION TO GEOPROCESSING CONFLATION TOOLS AND WORKFLOWS. Dan Lee and Silvia Casas

Transcription

1 INTRODUCTION TO GEOPROCESSING CONFLATION TOOLS AND WORKFLOWS Dan Lee and Silvia Casas

2 Agenda What is Conflation? Geoprocessing Conflation Tools Conflation Workflows Conceptual workflow strategy Demo: real world scenario Conclusions and Future Work

3 What is Conflation? Translated by Esri localization Zusammenführung Fusione 合并 Объединение 補正 Combinação Combinación Birleştirme Assemblage

4 When using multi-source spatial data together Common obstacles in analysis and mapping: Spatial and attribute inconsistency caused by differences in data collection and modeling High cost to fix the problems Adjacent datasets Overlapping datasets

5 Conflation reconciles multi-source datasets and optimizes data quality and usability Conflation involves the process of: Identifying corresponding features, also known as feature matching (FM) Making spatial adjustment and attribute transfer Combining matched and unmatched features into one unified dataset with the optimal accuracy, completeness, consistency, and integrity Resolving conflicts and connection issues among adjacent datasets, also known as edge matching, to ensure seamless data coverage Long-term benefits: No longer living with various imperfect datasets More confidence in reliable analysis and high quality mapping What s the way to get there?

6 Geoprocessing Conflation Tools

7 Our initial focuses Have you used these tools in ArcMap? Develop highly automated tools in Geoprocessing framework Starting with linear features (roads, parcel lines, etc.) Aiming at high feature matching accuracy (not promising 100%) Providing information to facilitate post-processing In ArcGIS and Pro 1.3 Build workflows Conflation: Edgematching tools and workflows 12:30 1:15pm, today Demo Theater 2 Spatial Analysis, Hall B

8 Feature matching (FM) for overlapping datasets Based on proximity, topology, pattern, and similarity analysis, as well as attributes information 1:1 and 1:m matches m:1 and m:n matches

9 FM-based tool #1 - Detect Feature Changes (DFC) Finding feature differences Output CHANGE_TYPE Spatial change (S) Attribute change (A) Spatial & attribute change (SA) Spatial and line direction change (S_LD) Spatial, attribute, and line direction change (SA_LD) No change (NC) New update feature (N) To-be-deleted base feature (D) DFC

10 FM-based tool #2 Transfer Attributes (TA) From source features to target features Transfer fields (e.g. ROAD_NAME) Target features are modified TA

11 FM-based tool #3 Generate Rubbersheet Links (GRL) Rubbersheeting moves source locations towards target locations based on established links Generate Rubbersheet Links (GRL) From source features to target features Followed by Rubbersheet Features (RF) Adjusting input features GRL RF

12 Conflation Workflows

13 Conceptual workflow strategy Unification of overlapping datasets

14 Unification of overlapping datasets Conceptual workflow strategy seta Contains updates setb Spatially accurate A popular scenario and requirements: To unify the two datasets into one with combined spatial and attribute information Unmatched from seta (2) Generate Rubbersheet links (1) Identify matched & unmatched Matched (4) Transfer attributes Make a copy (3) Rubbersheeting (5) Append setc Best of both

15 This model reflects the conceptual workflow strategy. With simple and highly similar inputs, the process can produce 100% accurate result.

16 Simple and highly similar input streets Together Update features with new streets and attributes Base features with spatial accuracy and attributes

17 Results Changes detected Attributes transferred New features adjusted and added to base Rubbersheeting links generated

18 Three components in conflation workflows Preprocessing Conflation and evaluation Postprocessing In same projection Data validation Selection of relevant features Conflation tools Workflow tools Queued review Interactive editing

19 Supplemental tools and guidelines for download You can also search by conflation at arcgis.com to find the download. Improved toolbar and new workflow tools are coming soon.

20 Demo: Real world scenario Unification of complex overlapping datasets

21 Data overview Two road datasets (northwest of San Diego, CA): SAN 1128 features OSM 1206 features Both datasets: Have common and uncommon features and attributes Are well preprocessed

22 Breakdown of Conceptual workflow into sub-workflows Same goal and same strategy as the conceptual workflow Step4 QA #4 Step1a QA #1 Step5 Step1b Step3 QA #3 QA #2 Step2 Let s get started

23

24 DFC result and potential match issues QA #1 Matched flags (CFM_GRP) Unmatched flags (N and D in DFC output) Demo of QA #1

25 Review potential match issues Total 13 CFM_GRP were flagged 6 were true match issues due to data complexity and dissimilarity 7 were false alarm (ignorable) Match issue due to data complexity Match issue ignorable

26 Review DFC result: CHANGE_TYPE N and D ((CHANGE_TYPE = 'N') OR ( CHANGE_TYPE = 'D' )) AND (NEAR_DIST > 0) Inspect records with high potential for errors: 107 reviewed 3 wrong Ns or Ds flagged Wrong N Wrong D

27 Feature matching accuracy estimates Matched groups: Total: 980 groups Correct: 974 groups Incorrect: 6 groups Accuracy = 974/ 980 = 99.34% Unmatched: Total: 317 (121 Ns Ds) Correct: 314 (120 Ns Ds) Incorrect: 3 (1 N + 2 Ds) Accuracy = 314 / 317= 99.05% (biased by the total count) Overall feature matching accuracy (average of matched and unmatched) 99.19% Ready to join with inputs to tag Ns and Ds

28 Extract matched features for GRL and TA processes SAM: 1008 non-n out of 1128 OSM: 1012 non-d out of 1206 Ready for GRL process

29

30 GRL result Generated total regular links and 4 identity links QA #2 Intersecting links Long links Links of different node types Flagged match issue areas No link locations Demo of QA #2

31 Intersecting links 10 locations of intersecting links, total 16 links: 8 correct, 4 modified, 4 wrong

32 Long links Shape_Length >15, total 123 links: 96 correct, 7 modified, 20 wrong

33 Links at different vertex types qanotes = 'src_tgt_vxtype_diff, total = 192 links, 155 correct, 23 recheck, 10 modified, 4 wrong

34 Locations of missing links 33 areas generated for 605 no link locations on matched source features (TGT_FID > -1 ) AND ((FREQUENCY >1) OR ((FREQUENCY =1) AND (srcvxtype = 0) AND (NEAR_DIST>2))) 59 links were added; other locations are not critical

35 QA regular links - summary Review selection criteria matters Total 326 (1.69%) of links were reviewed: 19 were modified 27 were to be removed 280 were correct, including 23 to be rechecked 33 no link areas were reviewed: 59 links were added Links at other locations were not critical Ready for rubbersheeting

36 19300 of regular links were selected by (REV_FLAG <> 'Wrong' OR REV_FLAG IS NULL) to use

37 Rubbersheeting result QA #3 Links after RF DFC result after RF Flagged Links Flagged match issue areas Flagged no link areas

38 GRL and DFC result after rubbersheeting Many regular links became identify links; RF result is less different from target

39 How good is the rubbersheeting result? Three indicators showing spatial improvement Less spatial differences Improved location alignment Before RF After RF Regular links Identity links Link-length distributions before/after RF (Not on the same scale due to the big difference in values)

40 QA #3 Check rubbersheeting result Matched features and N features Ramps may need editing Ready to do TA

41 Transfer attributes Attaching all flagged and reviewed information to target

42 Attribute transfer result 1:2 match QA #4 Flagged unmatched features (Ns and Ds) Flagged match issue areas Multi-source m:n transfers (srcm_inmn > 1) 3:2 match Demo of QA #4

43 Check attribute transfer result CHECK_TA = 'Recheck' OR REV_FLAG = 'wrongd' OR srcnearfid > -1 OR srcm_inmn >1 69 records were reviewed: 15 Unique_ID values were corrected Retransfer via corrected Unique_ID Almost there

44 Select adjusted N features; append them to target ( CHANGE_TYPE = 'N' AND REV_FLAG IS NULL ) OR( REV_FLAG = 'isn' )

45 N features in final result

46 Unification of overlapping datasets completed! Processing Time Automated processing Step 1 (a, b) Step 2 1 min 45 sec 1 min 47 sec Step 3 1 min Steps 4, 5 36 sec Total 5 min 8 sec Interactive processing (not counting final review) Review Count (locations or groups) Edit Count (shape or value) QA #1 (CFM_GRP and DN) QA #2 (links) QA #3 QA #4 (attribute transfer) Total x x Time (3-4 review counts per minute) ~ 2-3 hrs.

47 Thanks to: Department of Public Works (DPW), Los Angeles County, USA. Conclusions and Future Work Institut Cartogràfic i Geològic de Catalunya (ICGC), Barcelona, Spain. Kevin Hunt, New York State Department of Transportation, USA. Richard Fairhurst, Riverside County Transportation and Land Management) CA, USA RCTLMA, National Institute for Water and Atmospheric Research (NIWA) and Land Information New Zealand (LINZ) - Crown Copyright Reserved. Resource Management Service, LLC, Birmingham, AL, USA. All others who supported us along the way.

48 Conflation can be done more efficiently now It takes a workflow: Use the best practice in preprocessing. Run automated tools to obtain highly accurate results and evaluation information. Interactively review and edit the results. The time is worth-spending. Three user stories

49 User story 1: Enhancing county roads by spatially more accurate city roads County centerline attributes and direction must be retained. Original_county_centerlines Temecula_city_centerlines Updated_county_centerlines Original_county_centerlines Updated_county_centerlines Temecula_city_centerlines Use DFC to find matching features and line direction differences For 1:1 matches, flip city centerlines of opposite direction (Flip Line) ~ 98%+ accuracy For m:n matches, merge/split city or county centerlines to get 1:1 matching segments, recalculate address ranges for county roads as needed, and flip city centerlines of opposite direction (tools + scripts) Transfer city centerline geometry to county centerlines (script) Data/information source: RCTLMA (Riverside County Transportation and Land Management) CA, USA Acknowledgement: Thanks to Richard Fairhurst, for providing the information and screenshots.

50 User story 2: Combining electoral roads and topographic roads There is no most accurate dataset. ~ 90% accuracy ~ 99% accuracy Electoral roads Topographic roads Information source: Land Information New Zealand (LINZ) Acknowledgement: Thanks to Douglas Kwan, LINZ, for providing the information.

51 User story 3: Transferring attributes from State routes to Street segments Segmentation for the datasets was different State routes Street Segments State Routes and Street segments were split by end points to provide a more similar segmentation between the two datasets. 99.5% match rates Data/information source: NYSDOT, USA Acknowledgement: Thanks to Kevin Hunt, for giving us the opportunity to work with him and share his data.

52 Consider conflation a higher priority Study the tools and workflows; understand the results Start with small test areas Customize the workflows for your organizations Improve data quality and usability Bring new live and value to your data Work with broader communities Data sharing and collaboration Seamless analysis and mapping Please send us your feedbacks

53 Our future work Align Features New tools and enhancements Align Features tool Better feature matching Better rubbersheet links Integrated review and editing Plug-in to the Universal Error Inspector (ArcGIS Pro future release) Interactive tools, including reference imagery Formalization of workflows Common scenarios oriented Other feature types Contextual conflation (spatially related features) Please send us your use cases and requirements

54 Recent papers Baella B, Lee D, Lleopart A, Pla M (2014) ICGC MRDB for topographic data: first steps in the implementation, The 17 th ICA Generalization Workshop, 2014, Vienna, Austria. ssion_8.pdf Lee D (2015) Using Conflation for Keeping Data Harmonized and Up-to-date, to be presented at the ICA-ISPRS Workshop on Generalisation and Multiple Representation, 2015, Rio de Janeiro, Brazil Lee D, Yang W, Ahmed N (2014) Conflation in Geoprocessing Framework - Case Studies, GEOProcessing, 2014, Barcelona, Spain. Lee D, Yang W, Ahmed N (2015) Improving Cross-border Data Reliability Through Edgematching, to be presented at The 27th International Cartographic Conference, 2015, Rio de Janeiro, Brazil Yang W, Lee D, and Ahmed N, Pattern Based Feature Matching for Geospatial Data Conflation, GEOProcessing, 2014, Barcelona, Spain.

55 Please take our Survey Your feedback allows us to help maintain high standards and to help presenters Thank you for attending! Questions, comments? Find ESRI UC in the Esri Events App Find the specific session Scroll down to the bottom of the session Answer survey questions and submit

56