Handbook on precision requirements and variance estimation for ESS households surveys

Transcription

1 ISSN Methodologes ad Workg papers Hadbook o precso requremets ad varace estmato for ESS households surveys 03 edto

2

3 Methodologes ad Workg papers Hadbook o precso requremets ad varace estmato for ESS household surveys 03 edto

4 Europe Drect s a servce to help you fd aswers to your questos about the Europea Uo. Freephoe umber (*): H(*) The formato gve s free, as are most calls (though some operators, phoe boxes or hotels may charge you). More formato o the Europea Uo s avalable o the Iteret ( Catalogug data ca be foud at the ed of ths publcato. Luxembourg: Publcatos Offce of the Europea Uo, 03 ISBN ISSN do:0.785/3579 Cat. No: KS-RA-3-09-EN-N Theme: Geeral ad regoal statstcs Collecto: Methodologes & Workg papers Europea Uo, 03 Reproducto s authorsed provded the source s ackowledged.

5 Ackowledgmets Ackowledgmets The Europea Commsso expresses ts grattude ad apprecato to the followg members of the ESS Task Force, for ther work o Precso Requremets ad Varace Estmato for Household Surveys: Experts from the Europea Statstcal System (ESS): Mart Axelso Loredaa D Cosglo Kar Djerf Stefao Falors Alexader Kowark Mārtņš Lberts Ioas Nkolads Swede Statstcs Swede Italy ISTAT Flad Statstcs Flad Italy ISTAT Austra Statstcs Austra Latva CSB Greece EL.STAT Experts from Europea uverstes: Yves Berger Ralf Müch Uted Kgdom Uversty of Southampto Germay Uversty of Trer Coordators of the Task Force from Eurostat ut Qualty, methodology ad research (formerly Methodology ad research ): Jea-Marc Museux Desa Camela Florescu Charma Coordator Methodologsts from elsewhere: Ncola Massarell Eurostat ut Labour market Albrecht Wrthma Eurostat ut Iovato ad formato socety (formerly Iformato socety; toursm ) Oo Hoffmester FAO The hadbook receved sgfcat cotrbutos from Gullaume Oser (Luxembourg STATEC), workg for SOGETI S.A., ad specfc cotrbutos from Mchele D Alò (Italy ISTAT) ad from AGILIS S.A. experts. The hadbook receved useful commets from the revewers Jula Aru (Estoa Statstcs Estoa), Harm Ja Boostra (Netherlads CBS), László Mhályffy (Hugary Hugara Cetral Statstcal Offce), Karm Moussallam (Frace INSEE) ad Paul Smth (UK Offce for Natoal Statstcs), ad from the DIME 3 members. Europea Statstcal System. Itally attached to Eurostat ut Govermet ad sector accouts; Facal dcators, member of the Eurostat etwork of methodologsts. 3 Drectors of Methodology of the Natoal Statstcal Isttutes (NSIs) of the Europea Statstcal System (ESS). Hadbook o precso requremets ad varace estmato for ESS household surveys

6 Foreword Foreword The qualty of Europea statstcs s essetal for users. Europea legslato requres that a certa level of qualty of statstcs s guarateed, ad ths qualty has to be assessed. Oe mportat dmeso of qualty s accuracy. The accuracy s the geeral statstcal sese the degree of closeess of estmates to the true values ad ts compoets are varace ad bas. The scope of ths tatve s the varace requremets ad estmato. Dfferet statstcal domas have bee cofroted wth smlar eeds related to the varace of estmates. These eeds rage from settg up precso (varace) requremets ad a process to assess complace to the requremets ( EU Labour Force Survey LFS), to developg procedures to streamle the producto of stadard errors of Europea statstcs o the bass of the formato provded by Natoal Statstcal Isttutes (NSIs) of the Europea Statstcal System - ESS ( the Commuty Survey o ICT 4 Usage Households ad by Idvduals). The tatves lauched by dfferet statstcal domas o smlar ssues called for a harmosato of the methods the ESS. I agreemet wth the ESS Drectors of Methodology (DIME), the Eurostat Drectorate Methodology, corporate statstcal ad IT servces ( charge of the methodologcal coordato ad support both at Eurostat ad ESS level) set up a Task Force (TF) wth a geerc madate to ssue geeral recommedatos o varace requremets ad estmato for ESS household surveys. The mplemetato of the geeral recommedatos ad the specfc agreemets at stake for LFS ad ICT are decded by the doma specalsts. Actually, a LFS doma specalst Eurostat set up a doma specfc TF whch ru parallel wth the DIME TF, dscussed the geeral recommedatos ad provded valuable feedback to the DIME TF. The coordato betwee the two TFs was esured by the Eurostat methodologsts. Wth respect to the ICT, doma specalsts Eurostat are curretly assessg the use of methods to estmate stadard errors cetrally Eurostat. For effcecy reasos, the DIME TF was a thk tak composed of a lmted umber of hgh profle experts Member States, two hgh profle academc experts ad the Eurostat methodologsts volved ther respectve projects. The hadbook prepared uder the auspces of the DIME TF was addtoally submtted for revew by experts desgated from other Member States tha those who partcpated to the DIME TF ad was subsequetly developed o specfc ssues. We expect that the geeral recommedatos ssued wth ths coordated approach wll provde a bass for a more harmosed/stadardsed approach of smlar problems other surveys. Atoo Bagorr Head of Ut Dael Defays Drector 4 Iformato ad Commucato Techology. Hadbook o precso requremets ad varace estmato for ESS household surveys 3

7 Table of cotets Table of cotets. Itroducto Precso requremets The two approaches for specfyg precso requremets Precso measures relato to type of dcators....3 Precso requremets ad reportg domas Examples of precso thresholds/szes Recommedatos for a stadard formulato of precso requremets Best practces o varace estmato Overvew of samplg desgs household surveys Sources of varablty of a estmator Varace estmato methods: descrpto, evaluato crtera ad recommedatos Some recommeded varace estmato methods to accout for dfferet sources of varablty Software tools for varace estmato: presetato Some examples of methods ad tools used for varace estmato Samplg over tme ad sample coordato Varace estmato for aual averages Varace estmato for estmators of et chage Estmato of gross chage Computg stadard errors for atoal ad Europea statstcs Recommedatos for mprovg computato of stadard errors for atoal ad Europea statstcs Possble methods for mplemetg the tegrated approach of varace estmato Possble ways of assessg complace wth precso requremets Refereces Appedx Glossary of statstcal terms Desg effect Metadata template for varace estmato Sutablty of varace estmato methods for samplg desgs ad types of statstcs Sutablty of software tools for samplg desgs ad related ssues o varace estmato Mmum effectve sample sze for logtudal estmates Idex Hadbook o precso requremets ad varace estmato for ESS household surveys 4

8 Itroducto. Itroducto The objectve of ths hadbook s to preset the results of the work o Precso Requremets ad Varace Estmato for ESS Household Surveys, ad more specfcally the geeral recommedatos ssued by the Task Force, set up uder the auspces of the DIME. The hadbook covers oly the varace compoet of accuracy (ad ot the bas). The recommedatos are le wth the ESS hadbook for qualty reports (Eurostat, 009a) ad wth the madate of the Task Force. They comprse: a recommedato for a stadard formulato of precso requremets EU regulatos, by takg to accout survey specfctes such as dcators ad regoal dsaggregato. The pot of formulatg these requremets s to acheve uform ad uambguous uderstadg betwee the Natoal Statstcal Isttutes (NSIs) ad Eurostat; a revew of varace estmato methods, wth a vew to establshg a more harmosed approach whe computg stadard errors ad cofdece tervals for statstcs at atoal ad EU levels. The hadbook recommeds good practces ad detfes bad oes whch should be avoded; a recommedato for a tegrated approach to the creased avalablty of stadard errors the ESS, wth a vew to achevg a fully harmosed approach. The recommedato assesses a rage of possble approaches from fully cetralsed to decetralsed; a recommedato o how to assess NSIs complace wth the precso requremets. Hadbook o precso requremets ad varace estmato for ESS household surveys 5

9 Precso requremets. Precso requremets Oe objectve of ths chapter s to detfy ad assess the exstg approaches for settg up precso requremets ad to propose oe of them as best practce. Other objectves are to provde the approprate precso measures for each type of dcator ad to troduce the cocept of domas ad how to hadle them whe defg precso requremets. Ths chapter sets out kow examples of precso thresholds/szes used dfferet cotexts ad by dfferet sttutos. These are ot meat to be prescrptve but to gve some feasble bechmark whe defg precso thresholds. Fally, the chapter proposes a set of stadard formulatos of precso requremets for regulatos, for level (aual, quarterly, mothly, etc.) estmates, ad for estmates of et chage (for overall atoal populatos ad for atoal breakdows).. The two approaches for specfyg precso requremets Mchele D Alò ad Stefao Falors (ISTAT) There are two ma strateges for settg up precso requremets: specfyg mmum effectve sample szes wth whch the NSIs have to comply, or precso thresholds that have to be met by the ma target dcators of the survey. Both strateges ca be defed at ether the plag or estmato stage, after the survey has bee carred out. The approach to settg up precso thresholds for survey estmates has already bee used the EU Labour Force Survey (EU-LFS) ad the Commuty Survey o Iformato ad Commucato Techology (ICT). The EU-LFS Framework Regulato (Coucl Regulato No 577/98 of 9 March 998) troduced precso requremets the form of precso thresholds whch have to be met over certa sub-populatos by estmates of aual averages ad estmates of chages over two cosecutve quarters. Ths esures that the EU-LFS atoal samples ca acheve a sgfcat degree of represetatveess. Hadbook o precso requremets ad varace estmato for ESS household surveys 6

10 Fgure..: Precso thresholds EU-LFS Precso requremets It s mportat to ote that the curret LFS precso requremets refer to a theoretcal stuato whch uemployed people accout for 5 % of the workg-age populato. These requremets are thus a referece for desgg the survey, but caot tell us aythg about the qualty of the actual survey results. Ths approach, wth the cotext of the curret arragemets for formulatg precso requremets, prevets the relatve stadard errors used as precso thresholds from becomg meagless (whe the proporto approaches zero, the relatve stadard error approaches fty). However, ths s a crtcal pot the curret formulato of precso requremets. I partcular, t s dffcult to say whether or ot such requremets for theoretcal stuatos are met whe the relatve stadard errors ca oly be relably computed for the actual estmates (Eurostat, 00c). A more straghtforward formulato of precso requremets the form of precso thresholds (referrg to the qualty of the actual estmates) s provded the methodologcal maual for the ICT survey (Eurostat, 00b). For the household survey: The estmated stadard error ( ) shall ot exceed percetage pots of the overall proportos ad shall ot exceed 5 percetage pots for the proportos related to the dfferet subgroups of the populato, where these subgroups costtute at least 0 % of the total populato the scope of the survey. Aother approach s to formulate precso requremets terms of mmum effectve sample szes to be acheved by the coutres. Hadbook o precso requremets ad varace estmato for ESS household surveys 7

11 Precso requremets Ths s the case wth the EU-SILC 5 Framework Regulato (Europea Parlamet ad Coucl Regulato No 77/003 of 6 Jue 003). Fgure..: Mmum effectve sample szes EU-SILC The cocept of effectve sample sze bascally refers to the mmum sample sze that would be requred, uder smple radom samplg wthout replacemet, to obta the same level of precso as wth the actual samplg desg. I practce, however, may samples are selected wth complex desgs (mult-stage selectos, weght adjustmet for o-respose, calbrato, etc.). It follows that the mmum effectve sample szes uder smple radom samplg have to be adjusted for desg effects Deff, vz. the varato desg effcecy caused by samplg desg compoets such as stratfcato or clusterg. Ths leads us to the cocept of acheved sample sze. Desg effect s foud to be subject to terpretato ad s ot easy to forecast because t also depeds o dcators, domas ad the estmato methods used. See Appedx 7. for more formato. If deotes the acheved sample sze, the the effectve sample sze eff s gve by (Kalto et al, 005): eff / Deff. (..) 5 EU Statstcs o Icome ad Lvg Codtos. Hadbook o precso requremets ad varace estmato for ESS household surveys 8

12 Precso requremets The acheved sample sze refers to the umber of (ultmate) respodets. Therefore, the real sample sze at the plag stage should be adjusted to the atcpated o-respose. I practce the value of Deff s ukow ad has to be estmated. The desg effect of a estmator ˆ of the parameter s defed as the rato betwee the varace V (ˆ ) of the estmator uder the actual samplg desg ad that whch would be obtaed from a hypothetcal smple radom sample wthout replacemet of the same sze: θ V(θ) ˆ Deff Deff ˆ. (..) * V (θˆ ) ˆ* s a equvalet estmator of uder smple radom samplg wthout replacemet. See Appedx 7. for more formato. Whe desgg a survey, defg the mmum level of precso s a very mportat step: very hgh precso atteds to waste resources, whle very low precso makes the results less usable. From a EU perspectve, t s desrable to have accurate statstcs at atoal level so that we ca compare ot oly the performace of coutres agast specfed targets but also ther performace betwee each other. Specfyg a mmum sample sze makes t possble to calculate a cofdece terval that cludes the true value of the parameter wth probablty close to. A commo practce whe determg cofdece tervals cossts of assumg that the estmator follows a ormal dstrbuto. A cofdece terval wth a gve cofdece level s the derved usg percetle values of the ormal dstrbuto of mea 0 ad varace. The halflegth of the cofdece terval represets the (absolute) marg of error of the estmator, whle the relatve marg of error s obtaed by dvdg the absolute marg of error by the estmated value of the parameter (see Appedx 7.). As a geeral formula, let us cosder a smple radom samplg wthout replacemet, ad assume that we are seekg a relatve marg of error of 00 k % for the total Y of a study varable y. Thus, the mmum sample sze s gve by: SRS m z / N S y, (..3) k Y z N S / y where z / S y s the varace of y over the whole target populato (see Appedx 7.) ad of the ormal dstrbuto of mea 0 ad s the percetle value at 00( / )% varace. I may practcal applcatos The populato quattes Y ad S y are actually ukow ad have to be estmated usg data from auxlary sources (prevous surveys, admstratve sources, expert judgmet, etc.). I the above formula k s the relatve marg of error expressed as a proporto, whle 00 k % s the relatve marg error expressed as a percetage. Equato (..3) accouts for the fte populato correcto. Calculato of mmum sample szes had so far reled o a sgle estmator for whch a specfed level of precso was desred. Ths made t dffcult for Eurostat to assess complace sce t requred kowledge ad motorg of the actual samplg desg. Noetheless, there are also practcal stuatos where mmum sample szes are ot determed o the bass of a precso crtero. For stace, may ESS surveys, budgetary costrats may be such that they put a strct lmt o the total umber of tervews whch ca Hadbook o precso requremets ad varace estmato for ESS household surveys 9

13 Precso requremets be coducted at EU level. Mmum sample szes at coutry level are the determed by allocatg the total umber of EU-level tervews amog the coutres, basg the allocato method o a geeral compromse betwee EU ad coutry accuraces. Ths s doe by allocatg a mmum umber of uts to the small coutres, thereby esurg a mmum level of precso each of them. There are also termedate stuatos where budgetary costrats at EU level are weaker, so a more ambtous EU precso target ca be set. I such cases, sample szes at coutry level should be adjusted for desg effects ad atcpated o-respose. The choce of the actual samplg desg s dctated by a trade-off betwee reducg cost ad reducg varablty. Ths adjustmet of atoal sample szes trggers addtoal costs whch should be cosdered the total budget at EU level (gve that budgetary costrats are weaker), o codto that the o-respose ad desg effects are kept uder cotrol. Determg mmum sample sze works oly f precso thresholds have bee set frst. Coversely, the oly practcal way for coutres to take o board precso thresholds s to esure that a mmum umber of uts have bee sampled. The two ma strateges are therefore equvalet theory, but may dffer practce, especally for mult-purpose surveys. Large-scale surveys are usually desged to estmate a great umber of parameters wth referece to may dfferet domas of terest. I ths cotext, precso requremets expressed as precso thresholds seem to be more flexble, eve though they also refer to a reduced set of target dcators. A gve effectve sample may acheve satsfactory precso for oe dcator but may be less satsfactory for others. Besdes, samplg desgs that meet desg requremets may ed up producg low-qualty output (e.g. a mmum sample sze does ot cotuously acheve satsfactory precso case of dyamc pheomea, a mmum sample sze does ot aturally cover for all sources of varablty lke calbrato). What really matters to data users s output qualty. Therefore, for EU regulatos, precso requremets expressed as precso thresholds are recommeded. They are a mportat strumet terms of qualty assurace. Preferece s gve to output qualty uder the assumpto that t cludes all of the effects (samplg desg, o-respose, calbrato, mputato, etc.). Wth regard to allocato large-scale surveys, bear md that surveys have, most cases, multple objectves. Ths meas that t s urealstc to hope for sample dmesos that guaratee predetermed levels of precso for all estmates of terest. A further problem arses from the eed to produce parameter estmatos for qute a large umber of domas. It s recommeded to lmt precso requremets to the ma target dcators for key reportg domas order to avod cumulatve coflctg costrats o a sgle data collecto strumet. Though we may be seekg the optmum autoomous soluto terms of precso of estmates for each doma, the result, practce, s a compromse of dfferet ams, each of them demadg a specfc type of respose ad where the soluto may be at odds wth the other solutos avalable. So we eed to establsh a optmum sample sze ad allocato a multvarate framework, assumg that a optmum for each varable cosdered dvdually may ot be optmum for the overall set of varables of terest. The extesve rage of objectves ad tes calls for multvarate allocato methodologes, order to get a overall pcture of how to acheve optmum determato of sample sze. More precsely, determg the sample sze ad the samplg allocato has to be able to guaratee precso thresholds for each varable of terest ad for a varety of domas. The method proposed by Bethel (989) amed at determg optmum sze from a multvarate vewpot the case of a Hadbook o precso requremets ad varace estmato for ESS household surveys 0

14 Precso requremets desg wth oe stage of stratfcato ad wth referece to a sgle doma. The method has bee geeralsed for mult-purpose surveys the cotext of mult-stage samplg desgs whe multple domas are uder study. The method s based o flatg the varace of the estmator Yˆ h of the total Y h stratum h uder smple radom samplg by meas of a estmator of the desg effect Deff for each doma of terest (Falors ad Russo, 00). Summary The two ma strateges for settg up precso requremets refer to: precso thresholds, to be met by a few ma target dcators of the survey, ad mmum effectve sample szes, to be esured by the NSIs. For regulatos, t s recommeded to express requremets by defg mmum precso thresholds to be met by a few ma target dcators. Precso ad accuracy are cocepts that are well defed ad documeted the ESS qualty framework ad are easly uderstood by users of statstcs. It s recommeded to lmt precso requremets to the ma target dcators for key reportg domas order to avod cumulatve coflctg costrats o a sgle data collecto strumet. Mmum sample sze s a meagful cocept for data producers who eed to desg strumets to collect data ad estmate costs. However, precso requremets expressed as precso thresholds seem to be more flexble, eve though they also refer to a reduced set of target dcators. A gve effectve sample may acheve satsfactory precso for oe dcator but may be less satsfactory for others. Besdes, samplg desgs that meet desg requremets may ed up producg low-qualty output (e.g. a mmum sample sze does ot cotuously acheve satsfactory precso case of dyamc pheomea, a mmum sample sze does ot aturally cover for all sources of varablty lke calbrato). Precso requremets expressed as precso thresholds whch are assumed to cover these sources are a mportat strumet for qualty assurace. What really matters to data users s output qualty. Therefore, for EU regulatos, precso requremets expressed as precso thresholds are recommeded. The two frameworks are evertheless equvalet theory: mmum sample szes are a traslato of precso thresholds a deal survey samplg cotext. The techcal dffculty assocated wth the effectve sample sze framework has to do wth determg the desg effect that measures the dstace betwee the actual desg ad the deal stuato. Desg effect s foud to be subject to terpretato ad s ot easy to forecast because t depeds also o dcators, domas ad the estmato methods used. Hadbook o precso requremets ad varace estmato for ESS household surveys

15 . Precso measures relato to type of dcators Precso requremets Desa Camela Florescu ad Jea-Marc Museux (Eurostat) Whe settg up precso requremets for a survey, the precso measures should be geared to the type of dcators. Most of the commoly used dcators ESS surveys belog to oe of the followg categores: the total or the mea of a cotuous varable (e.g. the total or the mea household come); the case of a qualtatve varable, the terest geerally les the total or the proporto of populato elemets a gve category (e.g. total umber or proporto of uemployed people the populato); a o-lear fucto of several totals, meas or proportos (ratos, regresso coeffcets, etc.). Frst of all, we eed to clarfy the dfferece betwee percetages ad proportos, ad betwee proportos ad ratos. Percetages ad proportos are coceptually equvalet but are expressed dfferet ways. A dcator may refer to the percetage of dvduals havg access to Iteret whch ca, for stace, take the value of 50 % or to the proporto of dvduals havg access to Iteret whch s, the same case, 0.5. Both ratos ad proportos are parameters of a populato. A rato s a rato of two totals or meas. A proporto s a specal case of a rato. The umerator ad the deomator are couts of elemets, the case of a proporto. The umerator s the cout of elemets a doma A, ad the deomator s the cout of elemets a doma B. Doma A has to be a subset of doma B. Ratos ad proportos are usually estmated by dvdually estmatg the umerator ad the deomator. It s ot mportat whether the populato parameter s a rato or a proporto whe t comes to varace estmato. The mportat pot varace calculatos s whether or ot the estmator of the deomator has samplg varablty. I may practcal cases, the varace of the estmator the deomator s zero. Ths happes, for stace, wth proportos whe the populato total the deomator s kow from exteral sources. I ths case, we have to estmate the varace of the estmator the umerator ad dvde the result by the squared value of the deomator. A example s the proporto of dvduals havg broadbad coecto, provded the whole umber of dvduals s kow from a exteral source. O the other had, the varace of the estmator the deomator may be strctly postve; ths meas that the varablty of the deomator has to be take to accout calculatos. Ths occurs, for example, wth doma estmators, vz. estmators for sub-populatos. For stace, a statstc may be the uemploymet rate, where the total labour force (the populato of employed ad uemployed persos) s estmated from a sample of observatos. I order to estmate the varace of such o-lear statstcs, we ofte resort to the Taylor learsato method (Teppg, 968; Woodruff, 97; Wolter, 007; Oser, 009). Accordg to the above deftos, both proportos ad ratos ca have a costat deomator (varace of the deomator s zero) or a varable deomator (varace of the deomator s ot zero). Hadbook o precso requremets ad varace estmato for ESS household surveys

16 Precso requremets However, for smplfcato purposes ad the varace estmato cotext, the cocept of rato s used to desgate a rato of two estmators where the deomator has a ozero varace (a o-lear statstc), whle the cocept of proporto s used to desgate a lear statstc (wth costat deomator). Coeffcets of varato (relatve stadard errors) are geerally ot recommeded for estmatg the precso of percetages/proportos. Ths s because the value of the percetage/proporto has a strog mpact o the value of the coeffcet of varato, especally whe the percetage/proporto s low, ad because the coeffcets of varato for the percetages/proportos of ay characterstc are ot symmetrcal. Cosder a smple radom sample wthout replacemet of sze. Let us assume that we wat to estmate a proporto P 0 P over the etre populato, ad that the exact sze of the populato (deomator of the proporto) s kow from exteral sources. Thus, the coeffcet of varato CV of the estmated proporto s gve by: P CV. (..) P Therefore, the lower the value of proporto P, the hgher wll be the coeffcet of varato CV. Furthermore, let us exame the mpact of the proporto o the mmum sample sze eeded to acheve a coeffcet of varato of 5 %. Fgure..: The mpact of the proporto o the mmum sample sze eeded to acheve a coeffcet of varato of 5 %, uder smple radom samplg estmated proporto Sample sze Whe a precso threshold s expressed as a coeffcet of varato, the proporto has a strog mpact o the mmum sample sze eeded to atta ths threshold: the sample sze teds towards fty, as the proporto approaches zero. Therefore, the use of coeffcets of varato precso requremets would lead to more strget codtos for coutres/regos wth small values of proporto, ad would thus requre a huge crease sample sze. Ths could result values of sample szes whch ca ever be attaed uder stadard budgets. Hadbook o precso requremets ad varace estmato for ESS household surveys 3

17 Precso requremets I partcular, t ca pose serous accuracy problems whe estmatg a proporto of dvduals rare sub-populatos, for stace, dvduals havg a partcular professo or actvty status: Table..: Mmum sample sze to esure a coeffcet of varato of 5 % Proporto the total populato (P)* Mmum sample sze eeded () Clergy Admstrators, publc sector Scetsts Retred farmers Retred factory workers *Source: Frech Cesus, 990 O the other had, wheever the precso threshold s expressed usg a absolute measure of accuracy lke stadard error, the the mmum sample sze eeded creases as the proporto approaches 0.5 (from both drectos), albet ot to fty. The use of stadard errors precso requremets would therefore mpose less strget codtos. As a result, survey targets expressed terms of stadard errors are more tractable. Fgure..: The mpact of the proporto o the mmum sample sze eeded to obta a stadard error of 0.5 percetage pots, uder smple radom samplg estmated proporto Sample sze The above therefore dscourages the use of coeffcets of varato for percetages/proportos, but ecourages ad recommeds the use of stadard errors. However, for specfc surveys, experts should decde to use that precso measure whch s the most demadg the case of that proporto value whch makes the study varable the most relevat. Ths meas: Hadbook o precso requremets ad varace estmato for ESS household surveys 4

18 Precso requremets the use of the stadard error may be preferred f the study varable becomes more relevat as the estmated proportos get closer to 0.5, sce t mposes more demadg requremets (bgger sample sze) for proportos earer to 0.5; the use of the coeffcet of varato may be preferred f the study varable becomes more relevat as the estmated proportos ted to 0, sce t mposes more demadg requremets (bgger sample sze) for proportos earer to 0. However, we should ote the huge crease the sample sze wheever the proporto approaches 0 ad should set a low threshold of the proporto uder whch the requremet does ot apply; the use of ether stadard error or coeffcet of varato s equally preferable f the study varable becomes more relevat as the estmated proportos approach, sce they both relax the burde (lower the sample sze eeded). Summary It s recommeded to use precso measures whch are geared to the type of dcators they refer to. The geeral deftos of rato ad proporto are: a rato s a rato of two totals or meas, whle a proporto s a specal case of a rato where the umerator ad the deomator are couts of elemets doma A ad doma B respectvely, where doma A s a subset of doma B. However, for smplfcato purposes ad the varace estmato cotext, the cocept of rato s used to desgate a rato of two estmators where the deomator has a o-zero varace (a o-lear statstc), whle the cocept of proporto s used to desgate a lear statstc (wth costat deomator). Recommeded precso measures are: coeffcets of varato ad other precso measures expressed relatve terms for totals ad meas of cotuous varables; stadard errors ad other precso measures expressed absolute terms for proportos, but also for ratos ad chages whch are close to 0. The secod recommedato ams to avod stuatos where precso requremets lead to a huge crease the sample sze whe the dcator approaches 0. Moreover, absolute precso measures for the percetages/proportos of ay characterstc are symmetrcal. However, for specfc surveys, experts should decde to use that precso measure whch s the most demadg the case of that proporto value whch makes the study varable the most relevat. Ths meas: the use of the stadard error may be preferred f the study varable becomes more relevat as the estmated proportos get closer to 0.5; the use of the coeffcet of varato may be preferred f the study varable becomes more relevat as the estmated proportos ted to 0. However, we should set a low threshold of the proporto uder whch the requremet does ot apply; the use of ether stadard error or coeffcet of varato s equally preferable f the study varable becomes more relevat as the estmated proportos approach. Hadbook o precso requremets ad varace estmato for ESS household surveys 5

19 .3 Precso requremets ad reportg domas Precso requremets Desa Camela Florescu ad Jea-Marc Museux (Eurostat) Precso thresholds ad/or mmum effectve sample szes ca be defed at EU level, at coutry level or at doma level. There are usually o precso requremets for EU estmates as ther relablty s a drect cosequece of the relablty of atoal estmates. Thus, practce, precso requremets are mostly lad dow at atoal ad doma levels. It s recommeded that precso requremets be formulated for a certa doma level for a specfc survey, wheever a regulato stpulates that relable estmates are requred at that doma level. A doma s a subgroup of the whole target populato of the survey for whch specfc estmates are eeded. A doma may cosst of a geographcal area, such as a NUTS rego, or a major populato cetre, e.g. captal ctes. It may also comprse a specfed populato category, such as a major atoal or ethc group (OECD). For stace, the focus may be o ot oly the uemploymet rate of the etre populato, but also of breakdows by age, geder ad educato level. Uts a doma may sometmes be detfed pror to samplg. I such cases, the doma ca be treated as a separate stratum from whch a specfc sample may be take. Stratfcato esures a satsfactory level of represetatveess of the doma the fal sample: these are called plaed domas. Precso thresholds ad/or mmum effectve sample szes are mostly set up for plaed domas. O the bass of the result (..3) Secto., the mmum sample sze requred to acheve a relatve marg of error of 00 k %, for the total Y d of a study varable y, over a doma U of sze N, s gve by: d d d _ m z / N d S yd, (.3.) k Y z N S d / d yd where S yd s the varace of y over the doma ad z / s the percetle value at 00( / )% of the ormal dstrbuto of mea 0 ad varace. I the above formula k s the relatve marg of error expressed as a proporto, whle 00 k % s the relatve marg of error expressed as a percetage. The populato values Y d ad S yd are ukow ad have to be estmated usg data from auxlary sources (prevous surveys, admstratve sources, expert judgmet, etc.). Equato (.3.) accouts for the fte populato correcto. The formula (.3.) s applcable whe the whole sample s selected by smple radom samplg, whch case t ca be assumed that the sample s d of sze d (whch s supposed to be costat) from the doma U d has bee selected by smple radom samplg wthout replacemet. 6 Whe several precso thresholds are defed (at both atoal ad doma level), the mmum effectve sample sze should atta each precso target as specfed (..3) ad (.3.). Wheever costrats are tght, such that they lead to a sample sze whch caot be 6 The formula (.3.) caot be appled f the selecto probabltes are dfferet for the uts cluded the same doma. Hadbook o precso requremets ad varace estmato for ESS household surveys 6

20 Precso requremets attaed gve the avalable resources, there wll eed to be a compromse soluto, whch wll volve removg ad/or relaxg certa objectves. O the other had, there are may uplaed domas for whch uts caot be detfed pror to samplg. The eed for estmates of certa domas s ofte evdet oly after the samplg desg has bee decded or after the samplg ad the feldwork have bee completed. However, t s recommeded that survey maagers avod settg requremets for uplaed (reportg) domas, especally for domas whch represet a small share of the total populato. Sample szes for sub-populatos are radom varables, sce formato of these sub-populatos s urelated to samplg desg. The survey maager caot cotrol the sze of a uplaed doma sample eeded to esure complace wth establshed requremets. Besdes, the radom sze of the sample bulds a addtoal compoet of varablty to the doma estmates (Eurostat, 00). These two ssues occur partcular relato to rare sub-populatos 7 (say, where the doma accouts for less tha 5 % of the total populato). To llustrate the varablty of the sample sze from uplaed domas, cosder a smple radom sample s wthout replacemet of sze selected from a target populatou, of sze N. Let s d be the part, of sze d, of the whole sample s whch falls to a domau d U d U. d s a radom varable whch satsfes the followg propertes: Ed Pd P P, (.3.) V d d d N where d Pd deotes the relatve sze of the doma U d the populato U. For N example, wth = 8000 ad P d =0.05, we get a coeffcet of varato for the sample d of early 5 % ad a relatve marg of error of early 0 %, whch s ot eglgble. For a fxed sample sze = 8000, the lower the relatve sze of the doma ( P d ), the hgher wll be the coeffcet of varato for the sample sze d from the doma. Fgure.3.: The coeffcet of varato (CV) (%) for the sample sze d from the doma, plotted agast the relatve sze P of the doma (fxed whole sample sze = 8000) d CV(%) P d 7 Small doma estmato s excluded whe we are talkg about precso requremets for rare populatos. Hadbook o precso requremets ad varace estmato for ESS household surveys 7

21 Precso requremets Whe uplaed domas form a part of the whole populato, e.g. age groups, geder, educato levels, the survey maager ca estmate what the doma sample szes would be f the atoal sample structure mrrored the populato structure by doma. Precso measures ca be calculated for uplaed doma estmates to terpret the expected doma sample szes terms of statstcal accuracy. The precso of estmates for uplaed domas ca be mproved by post-stratfcato. However, bas ca be troduced at the same tme. The effect o the mea square error should be cosdered before post-stratfyg. Summary It s recommeded that precso requremets be formulated for a certa doma level for a specfc survey, wheever a regulato stpulates that relable estmates are requred at that doma level. Whe several precso thresholds are defed (at both atoal ad doma level), the mmum sample sze should atta each precso target. Wheever costrats are tght, such that they lead to a sample sze whch caot be attaed gve the avalable resources, a compromse soluto should be sought by removg ad/or relaxg certa objectves. I sample surveys, some of the domas are uplaed,.e. the doma uts caot be detfed pror to samplg. It s recommeded that survey maagers avod settg requremets for uplaed (reportg) domas, especally for domas whch represet a small share of the total populato. The survey maager caot cotrol the sze of a uplaed doma sample to esure complace wth requremets. I addto, the precso of estmators over uplaed domas s kow to have a varace compoet related to the ucertaty of the sample sze from such domas. These occur partcular wth rare subpopulatos (say, where the doma accouts for less tha 5 % of the total populato)..4 Examples of precso thresholds/szes Desa Camela Florescu ad Jea-Marc Museux (Eurostat) Specfyg what degree of precso s desred s a mportat step whe plag a sample survey. A very hgh level mght mea wastg of resources, whle a very low oe mght make the results less usable. I practce, questos arse as to whch precso thresholds are lked to acceptable qualty, but there s o commo stadard. Precso thresholds are geerally survey-specfc ad deped o users eeds ad o the requred relablty. Furthermore, ad over ad above statstcal cocers, determg precso thresholds s also a poltcal ad resource-related decso. Some examples of precso szes/thresholds used dfferet cotexts by dfferet sttutos are gve below. They are ot meat to be prescrptve but rather to gve some feasble bechmarks to be used whe defg precso thresholds: A coeffcet of varato of 5 % or less meas a satsfactory level of relablty for estmates, whle a coeffcet of varato of more tha 5 % meas lower relablty (Ardlly, 006). Hadbook o precso requremets ad varace estmato for ESS household surveys 8

22 Precso requremets I the ICT household survey, the estmated stadard error may ot exceed percetage pots for the overall proportos ad 5 percetage pots for the proportos relatg to the dfferet subgroups of the populato, where these subgroups comprse at least 0 % of the total populato wth the scope of the survey (Eurostat, 00b). I the EU-SILC, a methodologcal documet (Eurostat, 00) sets out how to use the compromse power allocato method to allocate the EU sample sze (whch should ot exceed to sample households) to coutres. The ma household come measure s the poverty rate, ad vares roughly the 5-5 % rage. At atoal level, takg a proporto (percetage) of 5 % as the bass for computatos, a smple radom sample of households s requred (except perhaps for the smallest coutres) to estmate ths wth percetage pot error (the absolute marg of error) (95 % cofdece terval). Ths correspods to a absolute stadard error of aroud 0.5 percetage pots. The Europea Health Itervew Survey (EHIS) methodologcal samplg gudeles (Eurostat, 009b) show how to allocate the EU sample sze ( dvduals) to coutres by usg the compromse power allocato method. Ths sample sze derves from the cosderato that a average of or dvduals per coutry would make for good precso (for a sample sze of ad a percetage of 0 %, the absolute stadard error s 0.3 percetage pots). Natoal effectve sample szes have bee computed by takg the percetage of people wth severely hampered actvty as the most crtcal dcator. Ths dcator was selected because of low prevalece some Member States, whch could lead to precso problems for some sub-groups. The correspodg errors absolute percetage pots (stadard error absolute terms) were the computed for the atoal effectve samples. At atoal level, the absolute stadard error vares from 0. to 0.4 percetage pots. Ths correspods to a absolute marg of error of betwee 0. ad 0.8 percetage pots (95 % cofdece terval). At ISTAT, coeffcets of varato should ot exceed 5 % for domas ad 8 % for small domas; whe they do, ths serves as a dcato to use small area estmators. Note that ths s just a rule of thumb ad that ot all domas are equvalet because they are assocated wth the percetage of the populato they represet, ad ths populato ca vary. Statstcs Caada apples the followg gudeles o LFS data relablty (Statstcs Caada, 00): o f the coeffcet of varato (CV) 6.5 %, the there are o release restrctos; o f 6.5 % < CV 33.3 %, the the data should be accompaed by a warg (release wth caveats); o f CV > 33.3 %, the the data are ot recommeded for release. Summary There are o geeral precso thresholds/szes that would hold good for all ESS surveys. They ted to be survey-specfc ad purpose-specfc, deped o users eeds terms of relablty, ad are related to avalable resources. The hadbook evertheless presets some (o-prescrptve) examples of precso thresholds/szes used by dfferet sttutos for specfc cases. Hadbook o precso requremets ad varace estmato for ESS household surveys 9

23 Precso requremets.5 Recommedatos for a stadard formulato of precso requremets Desa Camela Florescu ad Jea-Marc Museux (Eurostat) The DIME Task Force ssued a proposal for a stadard formulato of atoal precso requremets for percetages/proportos 8 (as ths s the type of dcator most ofte ecoutered household surveys) EU Regulatos. Ths s lked to the strategy of settg requremets terms of precso thresholds (see Secto.). Precso thresholds refer to the actual value of the estmated dcator. Ulke precso thresholds, a complace crtero fxed at the desg stage would be dffcult for Eurostat to motor ad may be frutless sce the ma am s the qualty of the output. The proposed stadard formulato of precso requremets s ssued for dcators of the proporto type, for atoal estmates of level, ad for et chages the atoal estmates of level, as follows: Precso requremets for estmates of level (e.g. aual, quarterly, etc. estmated percetages): o For overall atoal estmates: The survey should be desged such that the estmate of the stadard error does ot exceed... percetage pots for the estmated percetage... 9 for the total referece populato. o For estmates of atoal breakdows (domas): The survey should be desged such that the estmate of the stadard error does ot exceed... percetage pots for the estmated percetage... 0 for the populato breakdows, where such populato breakdows comprse at least...% of the total referece populato. Precso requremets for et chages the estmates of level (absolute chages the estmated percetage betwee successve years, quarters, etc.) o For overall atoal estmates: The survey should be desged such that the estmate of the stadard error does ot exceed... percetage pots for the chage betwee... 3 of the estmated percetage... 4 for the total referece populato. o For estmates of atoal breakdows (domas): 8 Percetages ad proportos are coceptually equvalet. See Secto. for more formato. 9 These ellpses wll be replaced by the ame of the ma target dcator. 0 These ellpses wll be replaced by the ame of the ma target dcator. Accordg to Secto.3, survey maagers should avod settg requremets for uplaed domas, especally for domas whch represet a small share of the total populato. Precso requremets ca be set for plaed domas, e.g. NUTS, where the samplg desg provdes for stratfcato by NUTS. Breakdows ca also be defed accordg to ther cotrbutos to the target dcators. 3 The perod of tme cocered by the chage wll be metoed here. For example, the ellpses ca be replaced by two successve quarters. 4 These ellpses wll be replaced by the ame of the ma target dcator. Hadbook o precso requremets ad varace estmato for ESS household surveys 0

24 Precso requremets The survey should be desged such that the estmate of the stadard error does ot exceed... percetage pots for the chage betwee... 5 of the estmated percetage... 6 for the 7 populato breakdows, where such populato breakdows make up at least...% 8 of the total referece populato. If the cofdece terval of the et chage cludes the value 0, the the chage the estmates s ot sgfcatly dfferet from 0 at the correspodg sgfcace level. The requremets for the estmates of level ad of et chage should be accompaed by addtoal provsos for the relaxato ad/or exempto of requremets for small ad very small geographcal domas (breakdows) (e.g. coutres, NUTS or NUTS3 regos). These provsos are partcularly relevat to estmates of atoal breakdows, where there are oly few populato uts small coutres breakdows, thus requrg a hgher samplg fracto tha for bgger coutres. The provsos address a poltcal cocer cocerg the burde o small coutres/regos. The followg provsos ca be used: The same requremet s relaxed to a threshold of... percetage pots for geographcal domas wth a populato of betwee... ad... habtats. Geographcal domas whose populato s below... habtats are exempted from these precso requremets cocerg chages. Ths proposal o formulatg a commo stadard of atoal precso requremets s accompaed by the followg explaatos ad clarfcatos: The type of estmate to whch the stadard formulato refers to s the estmated percetage (coceptually equvalet to the estmated proporto). The cocept of stadard error s closely related to survey desg sce t reflects the expected varablty of the parameter estmator (the parameter ths case s the populato percetage). Typcally, the stadard error remas a ukow value whch tself has to be estmated, by usg a approprate estmator (called the estmator of the stadard error ). Cosequetly, stadard error should be used cojucto wth estmator. For determg a partcular sample ad a partcular estmated percetage, we ca calculate a estmate of the stadard error by usg a approprate estmator. Hece, as the requremets cocer the survey output, the formulato refers to the estmate of stadard error rather tha just to stadard error. Estmate of the stadard error should be used cojucto wth estmated percetage. The measuremet ut of a percetage s percetage pots. Both stadard error ad absolute marg of error coserve the measuremet ut of the target dcator. A estmate of stadard error s therefore expressed percetage pots the formulato of requremets. 5 The perod of tme cocered by the chage wll be metoed here. For example, the ellpses ca be replaced by two successve quarters. 6 These ellpses wll be replaced by the ame of the ma target dcator. 7 Accordg to Secto.3, survey maagers should avod settg requremets for uplaed domas, especally for domas whch represet a small share of the total populato. Precso requremets ca be set for plaed domas, e.g. NUTS, where the samplg desg provdes for stratfcato by NUTS. 8 Breakdows ca be also defed accordg to ther cotrbutos to the target dcators. Hadbook o precso requremets ad varace estmato for ESS household surveys

25 Precso requremets Let us cosder a et sample sze of uts (dvduals). Assumg smple radom samplg, f the estmated percetage of dvduals wth access to the Iteret s 50 % (50 percetage pots), the the estmate of the stadard error s aroud 0.56 percetage pots. The half-legth of the cofdece terval (the estmate of the absolute marg of error) s aroud. percetage pots, for a cofdece level of 95 %. The upper ad lower lmts of the cofdece terval are determed by addg ad subtractg. percetage pots to ad from 50 percetage pots. Thus, the lower lmt of the cofdece terval s 48.9 % (48.9 percetage pots) ad the upper lmt s 5. % (5. percetage pots). Cofuso may arse f the percetage sg % s used stead of percetage pots to express the estmate of stadard error or of the absolute marg of error. The rsk s that the upper ad lower lmts of the cofdece terval are determed after calculatg the percetage of. % out of 50 percetage pots ad the addg ad subtractg the result to ad from 50 percetage pots. For ths reaso, the threshold for the estmate of stadard errors s expressed percetage pots (ad ot % ) the formulato of requremets. The precso requremets cocer the survey output (the actual estmates), whle measures have to be take at the desg stage to esure complace wth the requremets. Ths s the ratoale for the expresso the survey should be desged such that... the formulato. At the desg stage, survey desgers should take to cosderato the expected o-respose, the varablty of the varable of terest the populato, the desg effect, etc. order to estmate the sample sze eeded. Meetg survey output requremets by adoptg measures at the survey desg stage s ot a easy task. Ths s because of the varablty of the varace estmates across all possble sample realsatos ad the fact that the varace estmate of the pot estmate obtaed wth oe sample s subject to ths varablty. It s oe of the reasos why complace wth requremets s accompaed by provsos o tolerace (see chapter 5 for more detals). Ulke the marg of error, the use of the stadard error precso requremets does ot assume a ormal dstrbuto of the sample meas across all possble sample realsatos. 9 The requremets cover oly precso ad ot the bas, so the whole cocept of accuracy s ot fully covered. The precso should corporate the effects, e.g. of o-respose, calbrato, etc. However, the elmato of bas caot be guarateed. It s commo practce to set up a cotrol mechasm for the level of o-respose: o I EU-SILC, uder Commsso Regulato No 98/003, the precso requremets for publcato of the data must be expressed terms of the umber of sample observatos o whch the statstc s based ad o the level of tem orespose (besdes the total o-respose at ut level). The Commsso shall ot publsh a estmate f t s based o fewer tha 0 sample observatos, or f o-respose for the tem cocered exceeds 50 %. The Commsso shall publsh the data wth a flag f the estmate s based o 0 to 49 sample observatos, or f o-respose for the tem cocered exceeds 0 % ad s 50 % or below. 9 However, bootstrap cofdece tervals, for stace, are ot based o the ormalty assumpto. Hadbook o precso requremets ad varace estmato for ESS household surveys

26 Precso requremets The Commsso shall publsh the data the ormal way whe they are based o 50 or more sample observatos ad the tem o-respose does ot exceed 0 %. All data publcatos must clude techcal formato for each Member State o the effectve sample sze ad a geeral dcato of stadard errors for at least the ma estmates. o The OECD Programme for Iteratoal Assessmet of Adult Competeces (PIAAC) outles a mmum overall respose rate of 70 % as the goal, ad goes o to state that (OECD, 00): data from all coutres wth a mmum respose rate of 70 % wll geerally be cluded teratoal dcators ad reports uless sample motorg actvtes ad/or o-respose bas aalyses dcate serous levels of bas the coutry data; results from coutres wth respose rates of betwee 50 % ad 70 % wll typcally be cluded teratoal dcators ad reports uless problems resultg from such respose rates are compouded by other factors, such as uder-coverage bas; results from coutres wth respose rates below 50 % wll ot be publshed uless the coutry ca provde evdece that the potetal bas troduced by the low respose rates s ulkely to be greater tha the bas assocated wth respose rates of betwee 50 % ad 70 %. The proposed stadard formulatos have some lmts caused by the depedece of the estmated stadard error o the actual value of the percetage (estmated percetage). For dyamc pheomea partcular, the chage the value of the dcator may trgger a eed to readjust the sample sze to esure cotued complace wth requremets. However, cotuously adaptg the sample sze s ether feasble or desrable. The followg possbltes should therefore be evsaged: o The survey desgers may cosder the most demadg value possble of the estmated percetage whe they estmate the sample sze eeded. If the requremets set thresholds for the estmate of the stadard error, the theoretcally ths value s 50 %; practce, however, t ca be the earest percetage value to 50 % out of the actual rage of relevat values for the specfc survey. 0 The feasblty of such reflectos should be assessed by the doma specalsts for each survey. o Both requremets for the estmates of level ad et chage may use multple thresholds for the estmates of stadard error, whch should be set up fucto of the values of the estmated percetages. The ratoale behd ths s to allevate the dfferet treatmet (burde) mposed o coutres wth dfferet values of estmated percetages. The thresholds may be determed as: the stadard errors that correspod to the upper boudares of each bad defed by the values of estmated percetages; the stadard errors that correspod to the md-pots of each bad defed by the values of estmated percetages. 0 If the requremets set thresholds for the estmate of the relatve stadard error (coeffcet of varato), the the most demadg percetage terms of sample sze s 0 %, or percetages that ted to 0 %. Hadbook o precso requremets ad varace estmato for ESS household surveys 3

27 Precso requremets o The threshold for the estmate of the stadard error may be expressed as a model fucto of the estmated percetage for the requremets of the estmates of both level ad et chage. Ths s fact the approach for revsg the curret precso requremets (Eurostat, 00d) preferred by the Group of Experts o Precso Requremets for the Labour Force Survey (LFS). The ma prcple that guded the choce of ths approach s that the ew precso requremets for the LFS should be ether more restrctve or more relaxed tha the curret oes. Ad practce, t mposes a eutralty costrat the revsed text. The advatages of ths approach are: the requred precso of desg would be fxed, whle allowg the threshold to vary wth the actual value of the estmate; complace wth requremets wll ot be flueced by the actual value of the estmate. O the oe had, ths approach avods havg to tghte up the requremets just because of a chage the value of the estmate; t also avods ay pressure to crease the sample sze, eve wth a effcet samplg desg. O the other had, t avods relaxg the requremets ad havg to deal wth a possble budgetary request to cut the sample sze, thereby reducg the qualty of the survey estmates as a whole. The requremet for the precso of the et chage of estmates may be adapted by establshg the requred dfferece at whch a estmate of chage has to be sgfcat. o For overall atoal estmates: The survey should be desged such that a dfferece of... or more percetage pots the estmated percetage... 3 betwee... 4 s sgfcat at the 0.05 level, for the total referece populato. o For estmates of atoal breakdows (domas), the formulato ca be adapted aalogously. Determato of the (maxmum) estmated varace (of the estmator of et chage) whch allows us to coclude that the actual chage s sgfcat ca be doe by usg the followg statstcal test: We reject the ull hypothess f: H 0 : P -P =0 H : P -P 0. Pˆ Vˆ( Pˆ Pˆ Pˆ ) z, (.5.) where ˆP = estmated percetage of tme t, Ths s, for stace, the approach of the U.S. Curret Populato Survey. See U.S. Cesus Bureau (006), p.3-. The specfc value for the absolute chage requred to be sgfcat wll replace the ellpses. 3 These ellpses wll be replaced by the ame of the ma target dcator. 4 The perod of tme cocered by the chage wll be metoed here. For example, the ellpses ca be replaced by two successve quarters. Hadbook o precso requremets ad varace estmato for ESS household surveys 4

28 ˆP = estmated percetage of tme t, Vˆ( Pˆ ˆ P ) = estmate of the varace of the estmator of the et chage, z = the quatle of the stadard ormal dstrbuto. Precso requremets From the above formula, the ull hypothess wll be rejected whe the et chage of estmates s hgher tha ts estmated absolute marg of error. I other words, rejecto of the ull hypothess occurs whe the cofdece terval of the chage does ot clude the value 0. Let us take a umercal example. Suppose the et chage of estmates betwee t ad t s equal to 0 percetage pots. If the estmated absolute marg of error s 5 percetage pots (for α = 5 %), the the cofdece terval rages from 5 to 5 percetage pots. As the cofdece terval does ot clude the value 0, the chage s statstcally sgfcat. For the same example, the absolute et chage s hgher tha ts estmated absolute marg of error, whch case the ull hypothess s rejected. Ths approach explctly requres sgfcace of et chage. However, sgfcace ca also be a cosequece of applyg the requremets ad ot just the bass of the requremet tself. Precso requremets for gross chages 5 ca also be establshed. However, all requremets for a survey (for estmates of level, of et chages ad of gross chages) should be parsmoous ad should be assessed from the pot of vew of redudacy ad cosstecy. The specfc choce for formulatg requremets should be aalysed ad decded by the specalsts for each survey. Precso thresholds should be agreed by the specalsts of the statstcal doma, based o techcal feasblty studes. Summary It s recommeded to follow a stadard formulato of precso requremets for EU regulatos whch ams at uform ad uambguous uderstadg wth the ESS. Ths formulato (whch s preseted ths secto) s ssued for dcators of the proporto type, for both: estmates of level (e.g. aual, quarterly, etc. estmated proportos), for the overall atoal estmates ad estmates of atoal breakdows (domas); et chages of estmates of level (absolute chages of the estmated proportos betwee successve years, quarters, etc.), for the overall atoal estmates ad estmates of atoal breakdows. Both requremets should be accompaed by addtoal provsos for relaxg ad/or exemptg requremets for small ad very small geographcal breakdows. The proposed stadard formulatos have some lmts caused by the depedece of the estmated stadard error o the actual value of the estmated proporto. The followg possbltes should therefore be evsaged: 5 See Secto for the defto of gross chages. Hadbook o precso requremets ad varace estmato for ESS household surveys 5

29 Precso requremets the survey desgers may cosder the most demadg value possble of the estmated percetage whe they estmate the sample sze eeded; the requremets may use multple thresholds for the estmate of stadard error, to be set as a fucto of the values of the estmated percetages; the threshold for the estmate of the stadard error may be expressed as a model fucto of the estmated percetage. Precso requremets for gross chages ca also be establshed. All requremets for a survey (for estmates of level, of et chages ad of gross chages) should be parsmoous ad should be assessed from the pot of vew of redudacy ad cosstecy. The specfc choce for the formulato of requremets should be aalysed ad decded by the specalsts for each survey. Precso thresholds should be agreed by the specalsts of the statstcal doma, based o techcal feasblty studes. Hadbook o precso requremets ad varace estmato for ESS household surveys 6

30 Best practces o varace estmato 3 3. Best practces o varace estmato Ths chapter starts wth formato o the ma samplg desgs used the ESS household surveys. Ths s followed by revews of all varablty sources of estmates whch should be take to accout, as far as possble, whe estmatg varace. The chapter the revews the varace estmato methods, ther characterstcs ad applcablty, ad evaluates the methods agast defed crtera. Later o, t troduces a matrx (Appedx 7.4) whch offers gudace o the choce of sutable varace estmato methods relato to samplg desg ad type of dcators, also lstg usutable methods. There s also gudace o avalable methods for corporatg the effect o varace of drect samplg, mplct stratfcato, uequal probablty samplg, calbrato, ut orespose, mputato, coverage errors ad measuremet/ processg/ substtuto errors. The the chapter presets tools for varace estmato. Some examples of curret methods ad tools used NSIs ad Eurostat are brefly descrbed. The last part of the chapter dscusses surveys usg samplg over tme ad gudes the reader towards estmato of varace for a aual average of estmates ad for estmators of et chage that preset a covarace structure duced by a rotato patter (e.g. as the LFS). The fal part troduces the cocept of gross chages ad sets out basc deas for varace estmato of gross chages. 3. Overvew of samplg desgs household surveys Desa Camela Florescu (Eurostat) ad Oo Hoffmester (FAO) A draft vetory of samplg desgs used the EU Labour Force Survey (EU-LFS), the Iformato ad Commucato Techology (ICT) household survey, ad EU Statstcs o Icome ad Lvg Codtos (EU-SILC) hghlghts ther dversty ad complexty. The samplg desgs used are: smple radom samplg of dvduals; stratfed radom samplg of dvduals; (stratfed) cluster samplg of households, addresses, etc. Clusters are selected the frst stage ad all elgble dvduals the clusters are tervewed; (stratfed) drect cluster samplg of households, addresses, etc. Some dvduals are selected the frst stage ad the all elgble dvduals lvg the household or at the address of the selected dvduals are tervewed; (stratfed) mult-stage samplg of dvduals. Clusters are selected the frst stage(s) ad some dvduals are selected from clusters the last stage; (stratfed) mult-phase samplg of dvduals. A master sample, a mcrocesus or a sample draw from aother survey sample s used as samplg frame. Some dvduals are selected from that frame or from clusters formed later phases; Hadbook o precso requremets ad varace estmato for ESS household surveys 7

31 Best practces o varace estmato 3 (stratfed) mult-stage cluster samplg of households, addresses, etc. Clusters are selected a hgher stage tha the frst ad all elgble dvduals the clusters formed the last stage are tervewed; (stratfed) drect mult-stage cluster samplg of households, addresses, etc. Some dvduals are selected a stage hgher tha the frst oe ad all elgble dvduals lvg the household or at the address of these selected dvduals are cluded; (stratfed) mult-phase cluster samplg of households, addresses, etc. A master sample, a mcrocesus or a sample draw from aother survey sample s used as a samplg frame. Clusters are selected from that frame or from clusters formed later phases ad all elgble dvduals the clusters selected the last samplg phase are tervewed; (stratfed) drect mult-phase cluster samplg of households, addresses, etc. A master sample, a mcrocesus or a sample draw from aother survey sample s used as samplg frame. Some dvduals are selected from that frame or from clusters formed later phases ad all elgble dvduals lvg the household or at the address of these selected dvduals are cluded. Systematc samplg (wth equal or uequal selecto probabltes) s ofte used as a samplg scheme the dfferet samplg stages. The above classfcato makes a clear dstcto betwee drect ad drect samplg. For example, drect cluster samplg, the ultmate samplg uts are clusters (e.g. households), whle drect cluster samplg a sample of clusters s obtaed from a sample of other uts (e.g. dvduals). A sample of dvduals may be selected from a populato regster ad the a sample of households s obtaed by takg all households that have at least oe of ther curret members the orgal sample of dvduals. Fgure 3..: Idrect samplg of households through dvduals Selecto of dvduals Selecto of households I practce, alteratve ames are sometmes used for drect samplg, such as etwork samplg. The dstcto betwee drect ad drect cluster samplg s deemed relevat, sce dfferet weghts should be appled to these desgs. Whe a smple radom sample of Hadbook o precso requremets ad varace estmato for ESS household surveys 8

32 Best practces o varace estmato 3 households s selected, every household has a equal probablty of selecto. O the other had, a drect selecto of households through dvduals leads to the selecto of households wth probabltes proportoal to ther sze (accordg to the umber of household members). Weghtg for the selected households s by the geeralsed weght share method. See Secto 3.4 for more formato o adjustmet of weghts ad for varace estmato for drect samplg. A addtoal dstcto should be made betwee mult-stage samplg ad mult-phase samplg. Both samplg procedures volve samplg at dfferet stages or phases. However, mult-stage samplg refers to samplg desgs whch the populato uts are arraged herarchcally ad the sample s selected stages correspodg to the levels of the herarchy. The samplg uts are dfferet for the dfferet stages. O the other had, mult-phase samplg the same type of samplg ut (e.g. dvduals) s sampled multple tmes. I the frst phase, a sample of uts s selected ad every ut s measured o some varable. The, a subsequet phase, a subsample of uts of the same type s selected oly from those uts selected the frst phase ad ot from the etre populato. I mult-stage samplg, samplg uts are selected varous stages but oly the last sample of uts s studed. I mult-phase samplg, the sample of uts selected each phase s studed properly before aother sample s draw from t. Ulke mult-stage samplg, mult-phase samplg formato may be collected at the subsequet phase at a later tme; ths evet, formato obtaed o all sampled uts of the prevous phase may be used f ths appears advatageous. Mult-phase samplg ca be used whe we do ot have a samplg frame wth suffcet auxlary formato to allow for stratfcato, or whe we caot detfy the samplg frame the populato subgroup of terest. The frst phase s used to measure the stratfcato varable o a tal sample of uts or to scree out the tal sample of uts o the bass of some varable. The, usg oly the strata or the part of the sample for whch we wat addtoal formato, a probablty sample of those elemets s selected for addtoal data collecto o a secod varable. For example, a frst phase ca scree out a sample of dvduals to detfy oly those who have bee a vctm of a robbery, whle the secod phase ca ask more detaled formato (e.g. whether the dvduals reported the robbery to the polce) to a sub-sample of the detfed vctms of a robbery. Mult-phase samplg reduces costs, tme ad the respose burde. Moreover, the formato from both phases ca the be used to compute a regresso or a rato estmate. For stace, a rato ca be the share of dvduals who reported a robbery to the polce the total umber of dvduals who have bee a vctm of a robbery. Mult-stage samplg s a partcular case of mult-phase samplg arsg by mposg the requremets for varace ad depedece of the secod phase desgs. Ivarace meas that every tme the th PSU (prmary samplg ut) s cluded the frst stage samplg, the same subsamplg desg must be used. Idepedece meas that subsamplg a gve PSU s depedet of subsamplg ay other PSU. See Särdal et al (99), Secto Two-phase samplg s sometmes called double samplg. Mult-phase samplg ca be detfed whe a survey sample s draw from a master sample, a mcrocesus or from aother survey sample. Whe calculatg weghts, t s recommeded that selecto probabltes for those frst-phase uts be take to accout ad that the Hadbook o precso requremets ad varace estmato for ESS household surveys 9

33 Best practces o varace estmato 3 addtoal varablty resultg from mult-phase samplg be accurately corporated the calculato of varace estmates. A addtoal dstcto should be made betwee tervewg all, some or oe of the elgble members of the selected households. Varace estmato should take ths to accout. Most surveys employ mult-stage desgs, whereby a sample of households s draw usg ay covetoal samplg desg (smple radom samplg wthout replacemet, systematc samplg, stratfed samplg, mult-stage samplg ad so o) ad the dvduals are selected for tervew from every sampled household. There are two ma optos at ths stage. The frst opto cossts of selectg ad tervewg oe perso per sampled household. The respodet s geerally selected by the ext/last brthday method or the Ksh grd method. A alteratve s to survey some or all of the household members above a certa age lmt. Oser (0) dscusses how may people should be tervewed per household the EU Safety Survey (EU-SASU), partcularly whether oe or all members should be tervewed every sampled household. The paper starts wth a revew of some techcal aspects relato to selectg ad tervewg all members of a household, rather tha oe. The choce has mplcatos for data qualty, maly samplg varace, o-respose rate ad measuremet errors, ad for the overall cost of the survey. The advatages of tervewg all household members are: The survey costs are reduced: order to acheve a target sample sze, ths opto meas cotactg far fewer households tha f oe perso was tervewed per household. For face-to-face surveys, the umber of trps to a segmet area ca be mmsed, whch helps save moey by reducg travel costs. Nevertheless, wth telephoe or web surveys the cost of cotactg a household s geerally small. Household respodets may help tervewers by provdg cotact formato for the other household members ad the tmes whe they are lkely to be avalable. Further, f ther experece was postve, household respodets help to locate ad motvate other household members to respod, a burde whch would otherwse fall o tervewers. Thus, because the feldwork ca be supervsed more easly, orespose s lkely to be reduced. Havg all the members of a household tervewed may also produce more accurate results, especally to household-level questos. The dsadvatages of tervewg all household members are: It ofte leads to less accurate results terms of samplg varace, maly because the members of a household ted to be more homogeeous tha the geeral populato wth regard to the varable of terest. Data for mult-respodet households may be subject to certa bases o sestve topcs (e.g. domestc volece, persoal atttudes). Ths measuremet bas could be reduced f oly oe perso per household were tervewed. To compare the effect of selectg oe, some or all persos from each sampled household, Oser (0) uses varace estmato formulae (whch assume a smple radom samplg of households, a costat umber of household members ad a costat overall sample of dvduals) uder dfferet scearos related to dfferet values of vctmsato rates ad Hadbook o precso requremets ad varace estmato for ESS household surveys 30

34 Best practces o varace estmato 3 tra-cluster correlato coeffcets. The desg effect s also cosdered. See Oser (0) for more detals, cludg cosderato of cost. Fally, some NSIs draw household samples usg balaced desgs. A samplg desg s sad to be balaced f t esures that the Horvtz-Thompso estmators of some balacg varables are equal to the kow totals. The cube method proposed by Devlle ad Tllé (004) eables balaced samples to be selected. As there s ofte o such thg as a exact balaced samplg desg, the cube method geerally proceeds two steps: a flght phase whch exact balace s mataed, ad a ladg phase whch the fal sample s selected whle complyg as closely as possble wth the balace codtos. Devlle ad Tllé (005) derve a varace approxmato for balaced samplg. It stems from cosderg balaced samplg as a calbrato exercse at the desg stage ad, lke calbrato (see Secto 3.4), t reles o the resduals of regresso of the study varable o the balacg varables. Summary Samplg desgs used household surveys are hghly dverse ad complex. Varace estmato should take to accout samplg desg. It should dstgush betwee drect ad drect samplg, betwee mult-stage ad mult-phase samplg ad, the case of household surveys, betwee eumeratg oly oe or more members of the same household. 3. Sources of varablty of a estmator Mārtņš Lberts (CSB) Ths secto revews the ma sources of varablty of a estmator ad gves geeral gudeles o how to corporate such varablty compoets to varace estmato. We have a parameter of terest deoted by whch we would lke to estmate from a sample s S ( 0 S 0 deotes the set of all possble samples that ca be draw from the target populato). We have a estmator deoted by ˆ whch we are usg to estmate. By defto, ˆ s a stochastc varable. It s a fucto of the set-valued radom varable S ~ whose realsatos are the possble samples (see Appedx 7.). ˆ ~ ˆ S : ss ˆ s ˆ. (3..) 0 s ˆ I practce, the expected value of ˆ deoted by E s ofte ot equal to (because of o-respose errors, measuremet errors, coverage errors ad other errors). The dfferece betwee E ˆ ad s the bas of ˆ : ˆ Eˆ ps ˆ B. (3..) s S 0 Bas s a compoet of the mea square error. It s oe of the accuracy measures of a populato parameter estmator. The bas of a estmator s the average error of the estmator s Hadbook o precso requremets ad varace estmato for ESS household surveys 3

35 Best practces o varace estmato 3 over all possble samples. A estmator s based f, o average, ts value dffers from the true value. I ths hadbook, however, we are ot cocered wth bas. The sample-to-sample varablty of ˆ aroud square error of a estmator. We wll express ths varablty by varace V ˆ E E ˆ s the other compoet of the mea ˆ E ˆ ˆ ˆ ˆ ˆ p s E E E s ss 0 V ˆ :. (3..3) The samplg varablty s the varablty of the statstc computed for all possble samples take from a populato. Precso refers to how close estmates from dfferet samples are to each other. The cocepts of bas ad varablty are llustrated the fgure below. Fgure 3..: Bas ad varablty (precso) of a estmator 6 We eed to kow the varace estmator. It s ot possble to measure the true value of populato. But t s possble to buld a varace estmator V ˆ order to be able to measure the varablty of a V ˆ from a sample of the V ˆ ˆ for V ˆ. Before buldg V ˆ ˆ t s recommeded to explore the dfferet sources of varablty for a estmator. There are several sources of varablty for a estmator ˆ, amely: Samplg desg ad estmator The frst source of varablty of a estmator comes from the procedure used selectg the sample (commoly kow as the samplg desg). Cosder a fte populato U of sze N. A radom sample s of sze s selected from the populato accordg to a samplg 6 Charles As, P. E. Statstcal Egeerg. Hadbook o precso requremets ad varace estmato for ESS household surveys 3