Preserving privacy in record linkage of anonymised administrative and survey data

Transcription

1 Preserving privacy in record linkage of anonymised administrative and survey data Pete Jones Census Transformation Programme Office for National Statistics

2 Presentation overview Introduce the ONS Administrative Data Census Outline the Data Linkage Strategy for 2021 Census Challenges of linking administrative, survey and census data in the UK Research methodology for linking pseudonymised data Results of our matching research Linkage quality and alternative approaches

3 Definitions LA = local authority, between 2,200 and 1 million people, average size = 160,000 Postcode = An alpha-numeric code assigned to a postal address to assist the sorting of mail PR = Patient Register list of all patients registered with an NHS doctor in England and Wales CIS = Customer Information System list of people who have a National Insurance Number tax register HESA = Higher Education Statistics Agency list of students registered on a Higher Education course in England and Wales SC = School Census list of pupils registered at state schools in England and Wales CCS = Census Coverage Survey a sample enumeration of approx 1% of households in England and Wales

4 ONS Administrative Data Census Project Office for National Statistics (ONS) researching alternative methods for producing census and population statistics in England and Wales National Statistician made a recommendation to Government in March 2014 that there should be a predominantly online census in 2021 Major focus on increasing the use of administrative data to enhance Census outputs in 2021 Longer term aspiration is to develop a census that is based primarily on administrative data and surveys The ONS Admin Data Census Project have started to produce annual research outputs in the run-up to 2021 Record linkage between administrative datasets and surveys will underpin the methodology for producing estimates

5 2011 Census Matching

6 2021 Census Matching

7 Challenges of linking administrative, survey and census data No population register in the UK No Person Identification Number (PIN) to facilitate automating linkage of records across sources: - Matching records on the basis of name, date of birth, sex and address Data quality Admin data source collections are not designed to be linked to Census or survey data: - Lags / capture errors / incompleteness Efficiency intention is to link admin data sources on 100% basis: - Increased reliance on auto-matching (scale and frequency) - Trade off between quality and efficiency Public acceptability Establishing a privacy preserving approach that still enables us to meet our statistical objectives: - Pseudonymisation / De-identification - Separated functions (e.g. Trusted Third Party)

8 Pseudonymisation in record linkage In recent years ONS have adopted a pseudonymisation approach for preserving privacy when linking administrative data Pseudonymisation is an irreversible type of encryption (Secure Hashing Algorithm, SHA-256) Substitutes identifiable information (i.e. name, sex, date of birth, address) with meaningless hash values: John = XY Jonny = SDAG The pseudonymisation process is applied to administrative records prior to record linkage SHA-256 is one of the most stringent types of encryption But also very limited for comparing information that is similar but not exact

9 Methodological Research Hashing data makes many of the traditional methods for resolving inconsistencies redundant - Cannot run direct string comparison algorithms - Cannot use clerical resolution Developed alternative ways of tackling data capture inconsistencies (1) The development of match-keys that can be derived in preprocessing and hashed before linking two datasets

10 Deterministic Matching: Match-Keys Key Type Unique records on EPR (%) 1 Forename, Surname, DoB, Sex, Postcode % 2 Forename initial, Surname initial, DoB, Sex, Postcode District 99.55% 3 Forename bi-gram, Surname bi-gram, DoB, Sex, Postcode Area 99.44% 4 Forename initial, DoB, Sex, Postcode 99.84% 5 Surname initial, DoB, Sex, Postcode 99.44% 6 Forename, Surname, Age, Sex, Postcode Area 99.46% 7 Forename, Surname, Sex, Postcode 99.19% 8 Forename, Surname, DoB, Sex 98.87% 9 Forename, Surname, DoB, Postcode 99.52% 10 Surname, Forename, DoB, Sex, Postcode (matched on key 1) % 11 Middle name, Surname, DoB, Sex, Postcode (matched on key 1) 99.90%

11 Use of Match-Keys to produce Administrative Data Population Estimates Admin Data based estimates compared with 2011 Census population estimates Admin data is: More than 13 % lower 8.5% to 13% lower 3.8% to 8.5% lower Within 3.8% 3.8% to 8.5% higher 8.5% to 13% higher More than 13 % higher

12 Probabilistic Matching Probabilistic matching methods are required to resolve more difficult linkage decisions Need to be able to compare and measure the similarities between names and date of birth Have developed a method based on similarity tables that are constructed prior to the pseudonymisation process Reception Server (Data Import Area) Data Storage Area Original Dataset (Source 2) Forename Surname DoB PostCode John Davis 02/04/1993 B1 2TG John Thomas 23/07/1986 M2 1JH John Smith 16/06/2003 BH12 1LT List of Jon Reed 19/09/1993 DT8 4PB unique Jon Ellis 16/06/2008 KT1 1LL John Extract list of Jonny Johnson 06/01/2002 N7 4ER Jon unique Jonny Daniels 21/10/1949 LN22 1AR Jonny forenames Jonny Barker 14/10/1974 PO11 7TG Jonathan Jonny King 26/02/1998 SO1 4KW Jonathan Khan 03/06/1999 E1 2BB Jonathan Wright 11/10/2004 CR21 2JJ Jonathan Walker 10/07/2002 W5 6AD

13 Similarity Tables Follow the same process for the 2 nd dataset import Reception Server (Data Import Area) Data Storage Area Source 2 Dataset Forename Surname DoB PostCode John Davis 02/04/1993 B1 2TG John Thomas 23/07/1986 M2 1JH John Smith 16/06/2003 BH12 1LT List of unique Jon Reed 19/09/1993 DT8 4PB forenames Jon Ellis 16/06/2008 KT1 1LL Identify any John Jonny Johnson 06/01/2002 N7 4ER additional Jon Jonny Daniels 21/10/1949 LN22 1AR names not on Jonny Jonnie Barker 14/10/1974 PO11 7TG list Jonathan Jonny King 26/02/1998 SO1 4KW Jonnie Jonathan Khan 03/06/1999 E1 2BB Jonathan Wright 11/10/2004 CR21 2JJ Jonathan Walker 10/07/2002 W5 6AD

14 Similarity Tables Run string comparison algorithm between all names on the list List of String comparison List of unique Forename Matches Score unique algorithm forenames John John 1 John John John Jonny 0.88 Jon Jon John Jon 0.91 Jonny Jonny John Jonathan 0.82 Jonathan Jonathan Jonny Jonny 1 Jonnie Jonnie Jonny John 0.88 Jonny Jon 0.89 Jonny Jonathan 0.79 Jon Jon 1 Jon John 0.91 Jon Jonny 0.89 Jon Jonathan 0.81 Jonathan Jonathan 1 Jonathan John 0.82 Jonathan Jon 0.81 Jonathan Jonny 0.79

15 Similarity Tables (example) PR_Forename PR_Surname PR_DoB PR_Sex PR_Pcode SC_Forename SC_Surname SC_DoB SC_Sex SC_Pcode Jon Smyth 13/02/1965 M PO15 5RR John Smith 08/02/1965 M PO15 5RR PR_Forename SC_Forename Similarity Score PR_Surname SC_Surname Similarity Score PR_DoB SC_DoB Similarity Score John John 1 Smith Smyth /02/ /02/ John Jonny 0.88 Smith Smithers /02/ /02/ John Jon 0.91 Smith Smithson /02/ /02/ John Jonathan 0.82 Smith Smith 1 13/02/ /02/ Jonny Jonny 1 Smyth Smith /02/ /02/ Jonny John 0.88 Smyth Smithers /02/ /02/ Jonny Jon 0.89 Smyth Smithson /02/ /02/ Jonny Jonathan 0.79 Smyth Smyth 1 13/02/ /02/ Jon Jon 1 Smithers Smith /02/ /02/ Jon John 0.91 Smithers Smyth /02/ /02/ Jon Jonny 0.89 Smithers Smithson /02/ /02/ Jon Jonathan 0.81 Smithers Smithers 1 13/02/ /01/ Jonathan Jonathan 1 Smithson Smith /02/ /03/ Jonathan John 0.82 Smithson Smyth /02/ /04/ Jonathan Jon 0.81 Smithson Smithers /02/ /05/ Jonathan Jonny 0.79 Smithson Smithson 1 13/02/ /06/

16 Similarity Tables (example) PR_Forename PR_Surname PR_DoB PR_Sex PR_Pcode SC_Forename SC_Surname SC_DoB SC_Sex SC_Pcode Jon Smyth 13/02/1965 M PO15 5RR John Smith 09/02/1965 M PO15 5RR PR_Forename SC_Forename Similarity Score PR_Surname SC_Surname Similarity Score PR_DoB SC_DoB Similarity Score John John 1 Smith Smyth /02/ /02/ John Jonny 0.88 Smith Smithers /02/ /02/ John Jon 0.91 Smith Smithson /02/ /02/ John Jonathan 0.82 Smith Smith 1 13/02/ /02/ Jonny Jonny 1 Smyth Smith /02/ /02/ Jonny John 0.88 Smyth Smithers /02/ /02/ Jonny Jon 0.89 Smyth Smithson /02/ /02/ Jonny Jonathan 0.79 Smyth Smyth 1 13/02/ /02/ Jon Jon 1 Smithers Smith /02/ /02/ Jon John 0.91 Smithers Smyth /02/ /02/ Jon Jonny 0.89 Smithers Smithson /02/ /02/ Jon Jonathan 0.81 Smithers Smithers 1 13/02/ /01/ Jonathan Jonathan 1 Smithson Smith /02/ /03/ Jonathan John 0.82 Smithson Smyth /02/ /04/ Jonathan Jon 0.81 Smithson Smithers /02/ /05/ Jonathan Jonny 0.79 Smithson Smithson 1 13/02/ /06/

17 Similarity Tables (example) # PR_Forename # PR_Surname # PR_DoB PR_Sex # PR_Pcode # SC_Forename # SC_Surname # SC_DoB SC_Sex # SC_Pcode EFIJ2465 CTYG0289 GXCX6714 M XXY1234 VRXM2613 XHDK5456 LRQP3671 M XXY1234 # PR_Forename# SC_Forename Similarity Score # PR_Surname # SC_Surname Similarity Score # PR_DoB # SC_DoB Similarity Score VRXM2613 VRXM XHDK5456 CTYG GXCX6714 JVNJ VRXM2613 XFVZ XHDK5456 RDDM GXCX6714 LRQP VRXM2613 EFIJ XHDK5456 LLZY GXCX6714 NFBN VRXM2613 UAXM XHDK5456 XHDK GXCX6714 XPKA XFVZ6018 XFVZ CTYG0289 XHDK GXCX6714 LIOO XFVZ6018 VRXM CTYG0289 RDDM GXCX6714 GXCX XFVZ6018 EFIJ CTYG0289 LLZY GXCX6714 MTVL XFVZ6018 UAXM CTYG0289 CTYG GXCX6714 URHR EFIJ2465 EFIJ RDDM5656 XHDK GXCX6714 ATNY EFIJ2465 VRXM RDDM5656 CTYG GXCX6714 QZIF EFIJ2465 XFVZ RDDM5656 LLZY GXCX6714 HIFN EFIJ2465 UAXM RDDM5656 RDDM GXCX6714 HJRM UAXM3111 UAXM LLZY2510 XHDK GXCX6714 FFKD UAXM3111 VRXM LLZY2510 CTYG GXCX6714 UUGF UAXM3111 EFIJ LLZY2510 RDDM GXCX6714 YZWA UAXM3111 XFVZ LLZY2510 LLZY GXCX6714 UASD

18 Candidate Matches The similarity tables identify all the candidate pairs that achieve a specified similarity threshold on forename, surname and DoB Source 1 Forename Source 2 Forename Forename Score Source 1 Surname Source 2 Surname Surname Score Source 1 DoB Source 2 DoB Source 1 DoB Overall Score EFIJ2465 ZASG CTYG0289 XHDK GXCX6714 AFIQ EFIJ2465 VRXM CTYG0289 XHDK GXCX6714 LRQP EFIJ2465 HDNR CTYG0289 CTYG GXCX6714 EYGI The researcher will only ever see the hashed fields Hashed variables are now redundant (can delete them) The only usable information is the scores themselves But what do you do with the scores?

19 Clerical Matching In the 2011 and 2021 Census, clerical matching was relied on to produce high quality links between the Census and the Census Coverage Survey: - 70% were automatched - 30% were clerically matched Impractical to rely on clerical review when linking very large datasets Clerical matching is also traditionally used for setting thresholds in probabilistic matching Using lots of administrative data requires ONS to move away from doing lots of clerical matching Need to develop methods that automate the classification of match statuses between candidate pairs Tested supervised and unsupervised methods

20 Supervised Method ONS currently unable to undertake large-scale clerical work but can have access to a small sample set of de-encrypted candidate pairs Modelling approach that moves away from setting two thresholds logistic regression Clerically match a small sample of unencrypted records: - Fit a logistic regression model where y-variable is the decision to match or not - Predictor variables are the similarity scores, name frequencies, geographic distances The idea is to substitute the clerical decision with an automated procedure Regression equation can be applied to remaining candidates between the two datasets Generates a single cut-off point (match where p >= 0.5)

21 Model Design Piloted in matching School Census records to the NHS Patient Register (all 12 year olds) Following auto-match, used similarity tables to identify 7303 records A clerical decision was made for 5% of records (365 candidate pairs) Fitted a logistic regression model with the dependent variable as the clerical match decision (binary outcome Yes or No ) and the following variables as predictors: - Agreement between forenames (SPEDIS Score) - Agreement between surnames (SPEDIS Score) - Forename weight (highest on both sources) - Surname weight (highest on both sources) - Sex agreement (agree = 2, disagree = 1) - Postcode agreement (full=5, sector=4, district=3, area=2, none=1) - DoB agreement (full=3, M/Y=2, D/Y=1) - Distance between OA centroids

22 Classifying Matches Observed Classification Tablea Predicted Match No Yes Percentage Correct Match No Yes Overall Percentage 98.6 a. The cut value is.500

23 Unsupervised Method Also tested an unsupervised approach to setting thresholds Unsupervised methods do not rely on any clerical matching at all Fellegi-Sunter framework with duplicate link method (Blakely & Salmond, 2002)

24 Comparison between supervised and unsupervised method

25 Testing the Algorithms Major requirement to understand quality of links we can expect between linked coverage survey and administrative datasets Plan to use a capture-recapture methodology called Dual System Estimation (DSE) DSE is sensitive to matching errors (false positives and false negatives) Tested our pseudonymised linkage algorithms by comparing them to a gold standard set of links that had been produced 2011 Census teams Linked 1% of NHS Patient Register records to the 2011 Census and Coverage Survey data for selected local authorities This had been done to a very high standard previously (< 0.1% false positives, and less than 0.25% false negatives)

26 2011 Census Comparison Exercise Made comparisons in 8 Local Authorities: Powys Westminster Birmingham Mid-Devon Lambeth Southwark Aylesbury Vale Newham When comparing against the gold standard we produced statistics on precision and recall - Precision = number of true matches / number of matches made - Recall = number of true matches / number matches available

27 Results of comparison exercise 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Comparison of PR to Census/CCS match rates: Census QA and B2011 Census QA match rate B2011 match rate B2011 Precision B2011 Recall

28 Summary Tables Local Authority PR Count Census QA Beyond 2011 B2011 True Positives Birmingham 21,313 17,482 17,255 17,185 Westminster 9,626 6,268 6,178 6,152 Lambeth 10,532 6,740 6,684 6,633 Newham 13,461 9,193 9,032 8,990 Southwark 9,993 6,627 6,496 6,472 Powys 1,648 1,554 1,539 1,536 Aylesbury Vale 2,732 2,455 2,448 2,441 Mid Devon Local Authority Census QA match rate B2011 match rate B2011 false positives B2011 false negatives Birmingham 82.0% 81.0% 0.4% 1.7% Westminster 65.1% 64.2% 0.4% 1.9% Lambeth 64.0% 63.5% 0.8% 1.6% Newham 68.3% 67.1% 0.5% 2.2% Southwark 66.3% 65.0% 0.4% 2.3% Powys 94.3% 93.4% 0.2% 1.2% Aylesbury Vale 89.9% 89.6% 0.3% 0.6% Mid Devon 88.6% 88.6% 0.2% 0.2%

29 Quality of Privacy Preserving Record Linkage Quality of linkage using pseudonymised data is likely to be sufficient for many applications - For example, building a person spine from administrative data But ONS needs to be able to link data to a very high standard for some aspects of our population estimation methodology Dual System Estimation requires very high quality matching between a coverage survey and the administrative data - Approx 1% of population - Targeting < 0.1% false positives, <0.25 false negatives Need the capacity to use clerical matching when it s needed Exploring other alternatives in privacy preserving record linkage For example, Trusted Third Party Approach

30 Trusted Third Party Models

31 Summary ONS needs to link large amounts of data using a privacy preserving approach Methods have been developed that automate the process of linking pseudonymised administrative data to reasonably high quality The quality of linkage does however need to be better in some circumstances ONS are reviewing alternative approaches to preserve privacy in record linkage We continue to develop methods for automating the linkage of administrative and survey records