Session 10: Quality of Register-based Statistics

Transcription

1 Course on Register-based Statistics INEGI Aguascalientes April 2011 Anders & Britt Wallgren Statistics Sweden and Örebro University Session 10: Quality of Register-based Statistics 1. Specific quality issues for register-based statistics? 2. Errors in sample surveys and register-based surveys 3. The users and the producers view of quality 4. Detailed knowledge of a register s characteristics 5. Overall appraisal of quality 6. Main quality issues in different kinds of surveys 1

2 Many say that Administrative Data has low quality? Explanations? Dagens Nyheter, 2 March 2003 Statistics lie about Page 173 Murder and manslaughter murder Year Deaths by Police cause of reported During the period , death: crime: on average only 57 per cent of the reported cases were actual cases of murder and manslaughter, says Mikael Rying. The other cases related Average : to everything from suicide and alcohol poisoning to simple coding errors in the police 100/181 = 55% reports. Source: Statistical Yearbook of Sweden

3 Chart 1.1 Four principles on how to use administrative data 2. These administrative registers should be transformed into statistical registers. Many sources should be used and compared during this transformation. 3. All statistical registers should be included in a coordinated register system. This system will ensure that all data can be integrated and used effectively. Cancer Register Cause of fdeath thregister Multi-generation Register Fertility Register Population Register Police reported crime: Administrative data that has not been transformed into statistical data Not included in the system, no harmonisation 3

4 Many say that Administrative Data has low quality because people report to help their purposes! Page % of Swedish people s tax deductions are pure tax evasion Taxpayers submit errors worth billions in their tax declarations. Deductions for the sales of shares 1/3 of all share sales contain errors Tax errors are difficult to judge and amount to SEK billion Many inadvertent errors occur because of the complicated rules Deductions for management fees taxpayers claimed deductions of a total of SEK 515 million 66% of these deductions contain incorrect information A deduction d for fees for fund managers is the most common error, the fee is deducted automatically Deductions for other expenditure taxpayers claimed deductions of a total of SEK 2.8 billion 82% of these deductions contain incorrect information

5 Many say that Administrative Data has low quality because people report to help their purposes! Page % of Swedish people s tax deductions are pure tax evasion Taxpayers submit errors worth billions in their tax declarations. Taxpayers submit errors worth billions in their tax declarations. Deductions for the sales of shares 1/3 of all share sales contain errors Complicated rules and unclear Tax errors are difficult to judge and amount to SEK billion Many inadvertent legislation... errors occur because of the complicated rules Errors can be found primarily in the deductions. Another perspective on these errors is when they are compared to the total income for all filing tax declarations the error is of the size 0.3%. Deductions for management fees taxpayers claimed deductions of a total of SEK 515 million 66% of these deductions contain incorrect information A deduction d for fees for fund managers is the most common error, the fee is deducted automatically Deductions for other expenditure taxpayers claimed deductions of a total of SEK 2.8 billion 82% of these deductions contain incorrect information

6 Many say that Administrative Data has low quality because people report to help their purposes! Income from employment Deductions for other expenditure Taxable income The income is correct The deduction is too high but is accepted Taxable income is incorrect according to the tax rules but is not used for the statistics ti ti Tax The tax is incorrect and too low according to the tax rules, but statistically it is correct, as this is the tax that the Decision by Tax Board: person actually paid Disposable income Statistically correct

7 Platek and Särndal (2001) state that a (new) theory is needed. Holt (2001) says that the most important aspect of quality of register-based statistics is not accuracy, but relevance. It is necessary to distinguish between surveys with their own data collection and ones that are registerbased. Otherwise, there is a risk of uncritically ii using the traditional error models developed for sample surveys and censuses. There should also be a distinction between the quality of a register-based survey and that of a statistical register, as a register has many possible uses. Page 175

8 Quality of a Register and a Register-based Survey: Administrative systems Primary statistical register The system of statistical registers Many purposes! How is it? One purpose! p Is it good/bad? A particular register-based survey Quality depends on: 1. Administrative sources 2. The Register system Page The methods used to create the Statistical Register

9 Errors in Sample Surveys and Register-based Surveys: Surveys based on data collection: Same quality issues in all surveys Quality of one survey does not affect other surveys Register-based surveys: Different quality issues in different surveys Quality of one survey affects many other surveys Sampling Errors and Integration Errors Cochran (1963): 12 of 13 chapters on sampling errors We give 40 quality indicators, 22 of integration errors Page 177 9

10 Model errors: Classification errors in the Employment Register 1993 Number of persons in test data Estimate in Employment Register Estimated classification errors: Employed Not employed Total Employed Not employed Total Employed LFS % 5% 100% Not employed LFS % 87% 100% Total Quality of estimates for two domains in the Employment Register Quality of estimates for two domains in the Employment Register True number of: Nr of persons Expected number of Standard error of Domain Employed Not employed in domain employed in the register the register estimate = = => Both systematic and random errors Page

11 Random variation in register data: Page 179 Road accidents, boys 0 12 years, actual values and estimated t trends A. County with roughly boys B. County with roughly boys Number of injured per 1000 boys/year Number of injured per 1000 boys/year Sample survey data for small areas or domains are not published because of the sampling error says NO! In Register statistics we have a similar problem! 11

12 Measurement errors: Page 180 Groves et al, Sample Survey Methodology: The cognitive process in answering questions: - Comprehension of question - Retrieval of information - Judgment and estimation - Reporting an answer Register statistics: Administrative rules and legislation, accounting principles and practice are more important than psychology 12

13 Chart 10.5 Measurement errors comparison of data collection methods Collecting data in sample surveys Underlying structure of question: Will you please try to understand our questions and try to remember? It is not necessary that you answer, and it does not matter what you answer, we will not do you any harm Questionnaire to persons: Does the right person in the household answer? Collecting data in admin. systems Underlying structure of question: 1. Report last month s turnover before e the 12 th this month! 2. Pay 25% of reported turnover before the 12 th this month! 3. If you don t report and pay, you will have to pay extra! Reports to authorities from persons: In our yearly tax form, we only add our signature and perhaps try some deductions d Page

14 Chart 10.5 Measurement errors comparison of data collection methods Collecting data in sample surveys Questionnaire to enterprises: Does the right person within the enterprise answer? Collecting data in admin. systems Reports to authorities from enterprises: Regular duty of professionals, the enterprise s s accounting system as a rule generates the report Errors are errors in the accounting routines or typing or scanning errors Page

15 Chart 10.5 Measurement errors comparison of data collection methods Collecting data in sample surveys Interviewer effects can be disturbing Leading questions in market research is often a problem Variables collected via questionnaire or interview in sample surveys (or censuses): Measurement errors are important t Collecting data in admin. systems No interviewers, no interviewer effects Legally complicated questions Statistical variables in registerbased surveys are often derived variables based on administrative variables: Relevance errors and model errors are important Page

16 The Producer s view of quality detailed knowledge Risk of random and systematic errors by major error source Page Error source Risk of random errors Risk of systematic errors Specification error Low High Frame error Low High Nonresponse error Low High Measurement error High High Data processing error High High Sampling error High Low Source: Biemer and Lyberg (2003, p. 59) The steps to to create create a register a register corresponding quality quality characteristics characteristics Steps to create a register Determining the research objectives The inventory phase The planning phase Quality characteristics of a register Group 1: Subject matter competence and capacity for development 4. Supplier contacts, receipt of administrative Group 2: Supplier contacts and editing data 5a. The integration phase the object set Group 3: Characteristics of the object set, both in a base register and in general 5b. The integration ti phase the object type Group 4: Characteristics ti of the definition iti of objects 5c. The integration phase the variables Group 5: Characteristics of the variables 6. Documentation Group 6: Characteristics of the documentation

17 The Producer s view of quality detailed knowledge Page Group 1. Subject matter competence and capacity for development Quality indicator Implies How is this measured/appraised? 1a Good contacts with users Subject matter 1b Perform own analyses competence 1c Sources are integrated Diversified use intended Qualitative appraisal 1d Search for new sources 1e Changes are planned Capacity for development Group 2. Indicators for how supplier contacts t and editing function Quality indicator Implies How is this measured/appraised? 2a Group from the statistical office Great or restricted Qualitative appraisal regularly in touch with supplier or subject matter one person occasionally in contact expertise respectively 2g Both usual editing i of each source Diversified ifi use Qualitative i appraisal and consistency editing comparing intended many sources

18 The Producer s view of quality detailed knowledge Page Group 3.1 Indicators for the characteristics of the object set in a base register Quality indicator Implies How is this measured/appraised? 3.1a Delay in reporting Overcoverage and undercoverage in current version of register give errors in cross-sectional data and errors changing over time Compare present version of register with version based on all available sources when all information is available. Indicate the size of revisions and length of delay. Indicate frequency of changes. Group 4 Indicators for the characteristics of object definitions Quality indicator Implies How is this measured/appraised? 4c Are register maintenance questionnaires conducted? Quality awareness How often are such questionnaires conducted, and regarding which categories of objects? 18

19 The Producer s view of quality detailed knowledge Page Group 5. Indicators for the characteristics of the variables Quality indicator Implies How is this measured/appraised? 5d Missing values and imputation errors 5f Wrong preliminary values Quality defects of administrative source Revision error Extent and causes of missing values are given for every variable Investigate number/proportion of wrong records which have been rectified or annulled Group 6. Indicators for the characteristics of the documentations Quality indicator Implies What should be documented? 6c Population Time Time series level shifts and their effects. What comparable over series processing has been done to provide comparability over time quality time? 6d Variables comparable over time Time series quality Time series level shifts and their effects. What processing has been done to provide comparability over time? 19

20 Overall appraisal of quality (for the users): page 190 Register-based statistics can sometimes be used through the calculation of one summarizing value: Employment in the manufacturing industry in our municipality has increased by 1.6%. But often a number of large tables with hundreds of estimates are studied in an effort to discover patterns of relationship and of time series patterns. The patterns found give rise to conclusions: The labor structure in our municipality has become worse with respect to age and level of education, hampering the high-tech sector. The overall quality of registers and register-based surveys Quality Register Particular register-based survey Relevance Only definitions are given Are the definitions adequate and functional? This is discussed in detail Cross-sectional quality What comparisons can be made? The quality is described only for the Time series quality What comparisons can be made? particular use. Is the quality good or bad? The Longitudinal quality What comparisons can be made? quality of estimates are described

21 A register for research based on LISA, a longitudinal register on persons used by many researchers in Sweden: Variables with different quality levels: l Level of Employed Income education (L) Cross-sectional quality Time series quality Longitudinal quality 21

22 L2000 Changed codes: cases Missing 9 All L Missing All Category 1, 2, 3 and 4 has been corrected: L Missing 9 All L Missing All Real changes : cases

23 Employed during November each year a variable intended to have good time-series quality of macro data Classification errors in the Employment Register 1993 Number of persons Eti Estimate t in new Emp. Register in test data Employed Not employed Total Employed LFS Not employed LFS Total Per cent of total number of persons Estimate in new Emp.Register Employed Not employed Total Employed LFS Not employed LFS Total Classification error Net error: = 0.3% Gross error: = 7.1% Changes in employment status for one individual bad longitudinal quality Year 1 Year 2 Year 3 Year 4 Year 5 Income for individual every year Limit in model every year Status of employment, estimated value Empl Not empl Empl Empl Not empl

24 Main quality issues of: Page Sample surveys 2. Censuses 3. Register-based surveys Good quality? The sampling Good coverage Good coverage miracle! of population: of population and time: Small areas Small areas Longitudinal surveys 24

25 Main quality issues of: Page Sample surveys 2. Censuses 3. Register-based surveys Errors, bad quality? Frame errors Frame errors Relevance errors Nonresponse errors Nonresponse errors Administrative data Measurement errors Sampling errors Measurement errors Integration errors Data integration Data collection Data collection The data collection process is not perfect 25