Best Practices for Automated Linking Using Historical Data: A Progress Report

Transcription

1 Best Practices for Automated Linking Using Historical Data: A Progress Report Preliminary; Comments are welcome Ran Abramitzky 1 Leah Boustan 2 Katherine Eriksson 3 James Feigenbaum 4 Santiago Perez 5 World Economic History Congress August Stanford 2 Princeton 3 UC Davis 4 Boston University 5 UC Davis 1

2 The Linking Challenge A recent explosion of research using linked historical data Census to census, other sources to the census In historical records, we usually don t have unique IDs, so we need to link people from one census to another on fields that will not change across censuses Names, Year of Birth, Place of Birth 2

3 James Alexander in

4 James Alexander in 1910 Nice and Simple? 4

5 James Alexander in 1910 Nice and Simple? What if... there are 20 more James Alexander s from Wales with the same age? there is another 29 years old James Alexander? How about another Jmaes Alexander or Jim Alexander? 4

6 Linking is inevitably imperfect Lots of things can go wrong: Enumeration error, transcription error, mortality, return migration, under-enumeration between Census years, and people with same attributes make it impossible to know the correct match with certainty,... 5

7 Linking is inevitably imperfect Lots of things can go wrong: Enumeration error, transcription error, mortality, return migration, under-enumeration between Census years, and people with same attributes make it impossible to know the correct match with certainty,... We face a trade-off Erroneously deeming two unrelated records as a match (Type I error) Erroneously neglecting true matches (Type II error) 5

8 Two General Ways to Link 1. Linking by hand Advantage: We humans trust other humans Disadvantages: Expensive; non replicable; impossible to search for a single record in a census without some use of automated algorithms; names of immigrants may be very hard for US hand linkers 6

9 Two General Ways to Link 2. Linking using automated algorithms Advantages: Rule based, cheap, replicable, can compare any two records Disadvantage: May not match the holistic similarity of different names that humans perform based on experience 7

10 The Goal of this Paper Evaluate various automated methods Discuss their pros and cons discrepancy rates sample size representativeness relative to population of interest Compare to human linkers What do humans and automated linkers see for linking data? Suggest best practices and provide transparent automated linking code 8

11 Paper s Take-Away Automated methods almost entirely reproduce hand links when humans and machine use the same information for linking Hand and automated methods both have some discrepancies with hand links that are based on rich set of genealogical variables But discrepancies are reasonably low In cases of discord, not at all obvious who is right: hand vs. machine 9

12 Paper s Take-Away Automated methods almost entirely reproduce hand links when humans and machine use the same information for linking Hand and automated methods both have some discrepancies with hand links that are based on rich set of genealogical variables But discrepancies are reasonably low In cases of discord, not at all obvious who is right: hand vs. machine We evaluate three commonly used automated methods and find all perform very well All three trade off Type I and Type II errors at different rates 9

13 We Evaluate Three Automated Linking Methods 1. ABE/Ferrie-style: Using exact names/nysiis adjusted/jaro-winkler distance With or without requiring uniqueness within 5 year band 2. Expectation Maximization (EM): combining age and name distance into a single score reflecting the probability that each two records are a true match 3. Machine Learning (ML): train an algorithm with data linked by hand to make matches like a human RA, using various record features 10

14 Need Ground Truth Data to Study Type I vs Type II Errors 100% for-sure ground truth linked census data does not exist 11

15 Need Ground Truth Data to Study Type I vs Type II Errors 100% for-sure ground truth linked census data does not exist 1. Data linked from the Union Army Records to 1900 Census by Dora Costa s team Data was created using trained RAs who had access to extra information not typically available (e.g. pension records) Carefully (and expensively) hand-collected 2. Two independent hand linkers observing only the variables we show the automated methods (names, year of birth, state of birth) Linked by trained Assistant Professor and Undergrad (made very similar links) 11

16 Need Ground Truth Data to Study Type I vs Type II Errors 100% for-sure ground truth linked census data does not exist 1. Data linked from the Union Army Records to 1900 Census by Dora Costa s team Data was created using trained RAs who had access to extra information not typically available (e.g. pension records) Carefully (and expensively) hand-collected 2. Two independent hand linkers observing only the variables we show the automated methods (names, year of birth, state of birth) Linked by trained Assistant Professor and Undergrad (made very similar links) 3. Hand linked Iowa 1915 to to 1940 from two different transcriptions 11

17 How Do Automatic Methods Do? Two ways to assess links and methods PPV is a measure of accuracy or precision Positive Predictive Value: CorrectMatches Matches Of the links made by the algorithm, what share agree with the reference data? TPR is a measure of recall or recall True Positive Rate: CorrectMatches Observations Of the links to be made, what share did the algorithm find? These measures trade off against one another 12

18 How Do Automatic Methods Do? Two ways to assess links and methods PPV is a measure of accuracy or precision Positive Predictive Value: CorrectMatches Matches Of the links made by the algorithm, what share agree with the reference data? TPR is a measure of recall or recall True Positive Rate: CorrectMatches Observations Of the links to be made, what share did the algorithm find? These measures trade off against one another We calculate these measures for our algorithms We include several variations of each algorithm in the following graphs 12

19 13

20 14

21 Automated methods almost entirely reproduce hand links when using the same set of matching variables We have two datasets in which human linkers and automated algorithms made links, all using same information for linking names, year of birth, state of birth 1. Union army data linked by hand to 1900 census 2. Iowa 1915 sons linked by hand to 1940 census 15

22 Linking Union Army to 1900 with Same Information: All Automated Methods Have ~90% Accuracy 1.9 PPV=(#correct/#matches) ABE ABE(DoraB) ABE-JW ABE-JW(DoraB) EM EM(DoraB) ML ML(DoraB) Solid shapes treat human links with limited information as the truth TPR=(#correct/#observations) % agreement between hand and automated methods 16

23 Iowa records linked by hand to 1940 census Over 90% agreement between hand and automated methods 17

24 When the human and machine disagree... who is correct? 18

25 When the human and machine disagree... who is correct? 18

26 When the human and machine disagree... who is correct? 19

27 When the human and machine disagree... who is correct? 20

28 Why is linking so hard? Common names, name changes, limited variables, age misreporting and age heaping, enumeration errors, death, return migration 21

29 Why is linking so hard? Common names, name changes, limited variables, age misreporting and age heaping, enumeration errors, death, return migration But we think transcription errors are a BIG deal Cursive is really hard to read 21

30 Why is linking so hard? Common names, name changes, limited variables, age misreporting and age heaping, enumeration errors, death, return migration But we think transcription errors are a BIG deal Cursive is really hard to read We have two copies of the 1940 Federal Census FamilySearch and Ancestry.com We know for sure which links are true because the same underlying census manuscript pages were the same Allows us to isolate transcription errors 21

31 We find that transcription error is generally high (~10 to 15%), and higher for foreign born 22

32 An Upper Bound for Linking? When we link 1940 to itself Of the links made, automated methods get almost 100% correct But we can only link 43 to 67% of the census Common names Brutally bad transcription 23

33 How do linked samples compare? Ultimately, we want to do economics with the linked data 24

34 How do linked samples compare? Ultimately, we want to do economics with the linked data Will the results of our analysis depend on which method we choose? 24

35 Parameter Estimates Are Very Stable Across Linking Methods We regress occupation score in 1900 on Occupation score in the Union Army data and height in the Union Army data We also regress occupation score on height within the Union Army data Hand and automated linked samples recover similar estimated effects Occupation scores at enlistment correlates positively with occupation score in 1900 All methods estimate similar positive correlations, though only ABE and ML have sufficient statistical power Height on occupation score is a null in Costa data All methods recover that 25

36 Parameter Estimates Are Very Stable Across Linking Methods Occupation Score Intragenerational Elasticity 26

37 Parameter Estimates Are Very Stable Across Linking Methods Height Effects on Occupation Score in Union Army Records 27

38 Why is our takeaway so different than Bailey et al (2017)? Our takeaway on automated methods is more positive Our paper: 3-25% false positives Bailey et al: 25-82% false positives 1. Available matching variables matter Bailey et al LIFE-M data have middle name, exact date of birth, etc Bailey et al assumes hand links are ground truth We find similar discrepancy rates for hand linked and automated samples using same matching variables 28

39 Why is our takeaway so different than Bailey et al (2017)? 2. Bailey et al exclude better-performing automated methods designed to reduce false positive rates and include rarely used methods Widely-used ABE conservative method requiring name uniqueness within 5 year band is only discussed in the online appendix (= 19% false positives ) Better-performing second-generation automated methods are not evaluated (JW, EM) Never and rarely used methods known to have high false positive rates are included (e.g., ABE with Soundex [= 43% false positive ] is never used, random matches [= 54-69% false positive ] are rarely used) 29

40 Thanks! This project is preliminary and comments are welcome! 30