Secion on Govenmen Saisics JSM 2008 Assessing he Effec of Calibaion on Nonesponse Bias in he 2005 ARMS Phase III Sample Using 2002 Census of Agiculue Daa Mogan S. Eap 1, Jaki S. McCahy 1, Nick D. Schaue 2, Phillip S. Ko 1 1 Unied Saes Depamen of Agiculue, Naional Agiculual Saisics Sevice, Reseach and Developmen Division 3251 Old Lee Highway Room 305, Faifax, VA 22030 2 USDA, NASS, Saisics Division, 1400 Independence Ave. Mail Sop 2053, SW Washingon, DC 20250 Absac The Agiculual Resouce Managemen Suvey (ARMS) is conduced by he USDA and collecs deailed economic daa fom US poduces. As wih mos suveys, ARMS suffes fom uni nonesponse (70.5% in he 2005) wih he poenial o inoduce nonesponse bias. Nonesponse bias was assessed by maching ecods sampled fo he ARMS wih hose fom he 2002 Census. Mean elaive bias was assessed fo 17 vaiables by compaing esimaes based on census daa fo all ARMS cases (espondens and nonespondens) vesus ARMS espondens, using boh uncalibaed and calibaed base sample weighs. Nine of he 17 had significan bias using he ARMS base weighs. The ARMS calibaion weighs educed he bias so ha i was no longe significanly diffeen fom zeo in 90% of he sudy vaiables. This suggess ha calibaion is an effecive ool in educing nonesponse bias o accepable levels. Key Wods: Nonesponse Bias, Response Rae, Calibaion Weighs, Mean Relaive Bias 1. Inoducion Suvey nonesponse happens; he quesion is, how do we addess i? In 2003, he Fedeal Govenmen s Fedeal Commiee on Saisical Mehodology (FCSM) fomed a subcommiee of Ineagency Council on Saisical Policy (ICSP) epesenaive nominees o updae Fedeal sandads fo saisical suveys. The Subcommiee on Sandads fo Saisical Suveys concluded ha in ode o ensue he qualiy, objeciviy, uiliy, and inegiy of Fedeal Govenmen daa, nonesponse bias should be assessed when suveys exhibi insufficien esponse aes. Unde he guidance of he FCSM and ICSP, ICSP epesenaives ecommended Fedeal suvey sandads and guidelines o he Execuive Office of he Pesiden s Office of Managemen and Budge in 2004. Afe public eview, he Execuive Office of he Pesiden ulimaely eleased he Office of Managemen and Budge Sandads and Guidelines fo Saisical Suveys on Sepembe, 22, 2006. The Unied Saes Depamen of Agiculue s (USDA) Naional Agiculual Saisics Sevice (NASS) helped develop he OMB s new sandads and guidelines fo saisical suveys. This pape focuses specifically on Sandad 3.2. Sandad 3.2 addesses esponse aes and analysis of nonesponse bias, equiing ha Agencies mus appopiaely measue, adjus fo, epo, and analyze uni and iem nonesponse o assess hei effecs on daa qualiy and o infom uses when suvey esponse aes fall below 80 pecen. (Office of Managemen and Budge, 2006, p. 14) Sandad 3.2 sipulaes ha esponse ae ae compued using sandad fomulas o measue he popoion of he eligible sample ha is epesened by he esponding unis in each sudy, as an indicao of poenial nonesponse bias. (p. 14) In 2005, he Agiculual Resouce Managemen Suvey (ARMS) Phase III esponse ae was 70.5 pecen (n = 34,937), necessiaing an assessmen of nonesponse bias accoding o Sandad 3.2, Guideline 3.2.9. Guideline 3.2.9 saes, Given a suvey wih an oveall uni esponse ae of less han 80 pecen, conduc an analysis of nonesponse bias using uni esponse aes as defined above, wih an assessmen of whehe he daa ae missing compleely a andom. As noed above, he degee of nonesponse bias is a funcion of no only he esponse ae bu also how much he espondens and nonespondens diffe on he suvey 506
Secion on Govenmen Saisics JSM 2008 vaiables of inees. Fo a sample mean, an esimae of he bias of he sample esponden mean is given by: B n y y y y y n n Whee: y The mean based on all sample cases; y The mean based only on esponden cases; n n The numbe of cases in he sample; and n The numbe of nonesponden cases. y The mean based only on nonesponden cases; n n Fo a mulisage (o wave) suvey, focus he nonesponse bias analysis on each sage, wih paicula aenion o he poblem sages. A vaiey of mehods can be used o examine nonesponse bias, fo example, make compaisons beween espondens and nonespondens acoss subgoups using available sample fame vaiables. In he analysis of uni nonesponse, conside a mulivaiae modeling of esponse using esponden and nonesponden fame vaiables o deemine if nonesponden bias exiss. (p. 16) Cuenly NASS calculaes he unweighed uni esponse aes (RRU) fo he ARMS based on he fomula povided unde Guideline 3.2.2: RRU C C R NC O e(u) Whee: C = The numbe of compleed cases o sufficien paials; R = The numbe of efused cases; NC = The numbe of nonconaced sample unis known o be eligible; O = The numbe of eligible sample unis no esponding fo eason ohe han efusal; U = The numbe of sample unis of unknown eligibiliy, no compleed; and e = The esimaed popoion of sample unis of unknown eligibiliy ha ae eligible. (p. 14) NASS sums he numbe of posiive usables, ou of business, and non-fams and calculaes he pecenage his sum epesens of he oal numbe of epos o calculae he esponse ae fo ARMS Phase III. The ARMS is conduced in hee phases. Phase I sceens fo poenial samples fo Phases II and III. Phase II collecs daa on copping pacices and agiculual chemical usage and Phase III collecs deailed economic infomaion abou he agiculual opeaion, as well as he opeao s household. Phase III is he only phase of he ARMS wih esponse aes lowe han 80 pecen. Due o lowe esponse aes wih ARMS Phase III, he poenial fo nonesponse bias is geae hee. NASS weighs he ARMS Phase III esponden sample in such a way ha esimaed vaiable oals fo a lage se of iems mach ages deemined fom ohe souces. This is done hough a weighing pocess called "calibaion." Calibaion is he pocess of adjusing suvey weighs so ha ceain ages ae me. NASS uses official esimaes of fam numbes, con, soybean, whea, coon, fui and vegeable aces as well as cale, milk poducion, hogs, boiles, eggs and ukeys as calibaion ages. Fo example, afe calibaion he sum of he weighed suvey daa will equal he NASS esimae fo con aces. In addiion o educing confusion in he use communiy ha migh esul fom NASS eleasing alenaive esimaes fo he same oals, calibaion weighing poduces 2005 ARMS Phase III esimaes wih geneally lowe vaiances and, hopefully educed nonesponse biases. This epo descibes an ongoing eseach effo aimed a measuing he poenial fo nonesponse bias in he ARMS Phase III and he success o failue of calibaion in emoving i. 2 507
Secion on Govenmen Saisics JSM 2008 Nonesponse bias is vey difficul o measue diecly. Founaely, an indiec measue of nonesponse bias is available fo he 2005 ARMS Phase III, heeafe called simply he ARMS. The Census of Agiculue, conduced evey five yeas, is a mandaoy collecion of daa fom all known agiculual opeaions. NASS has daa fom he Census on iems of inees fo many of he ARMS nonespondens; howeve, he Census iself is incomplee. An esimaed 17.90 pecen of all fams wee missing fom he 2002 Census Mailing Lis, and 12.26 pecen of fams on he Lis failed o espond o he Census. Moeove, no all ARMS sampled fams could be mached o 2002 Census ecods. Neveheless, by compaing he 2002 Census values of ARMS espondens o he full sample of ARMS espondens, we can measue he diffeence beween he aveage ARMS esponden and he aveage of he full sample wihou any nonesponse adjusmen. Addiionally, we hope o measue he educion of ha diffeence fom using a calibaion-weighing pocess simila o he one used fo he 2005 ARMS. While he 2002 Census daa do no pefecly mach he 2005 ARMS daa, hey ae coelaed (Eap, McCahy, Schaue, & Ko, 2008, Appendix A), so he pesen evaluaion will compae espondens on he 2005 ARMS suvey o nonespondens using hei 2002 Census daa fo each. 2. Mehod Ou analyical daa se consiss of census values fo fams sampled fo he ARMS ha also povided 2002 expendiue daa on he Census. In he 2002 Census, only a sample of fams eceived he long vesion ha asks fo expendiue daa. Howeve, 2002 Census daa wee available fo 81.4% of all 2005 ARMS III sampled opeaions of which only 48% compleed he Census long foms wih expendiue daa and wee included in his analysis. 1 The base sample weighs 2 (each fam s ARMS sample weigh befoe calibaion muliplied by is Census sample weigh) fo he subse of fams esponding o he ARMS wee calibaed so ha he final weighed oals compued fom hem equaled he aw weighed oal compued fom he enie mached se fo he following vaiables: cale, con, coon, pigs, soybeans, whea, fui, vegeables, boiles, and ukeys. Each of hese age vaiables plus egg and milk poducion was used opeaionally o calibae he ARMS daa. As in he opeaional pogam, he ARMS esponden subse was calibaed independenly in 20 egions. These included he 15 leading cash eceips saes (Akansas, Califonia, Floida, Geogia, Illinois, Indiana, Iowa, Kansas, Minnesoa, Missoui, Nebaska, Noh Caolina, Texas, Washingon, and Wisconsin). The emaining 33 saes (Alaska and Hawaii ae no sampled fo he ARMS) wee gouped using he five poducion egions: 1) Alanic, 2) Souh, 3) Midwes, 4) Plains, and 5) Wes. Ou analysis focuses on 17 specific (non-calibaion) vaiables colleced on boh he ARMS and he Census: 1. Toal Aces 2. Toal Sales 3. Aces Rened 4. Copland Aces 5. Toal Poducion Expenses 6. Cop Expenses 7. Seed Expenses 8. Feilize Expenses 9. Chemical Expenses 10. Livesock Puchases 11. Feed Puchases 12. Hied Labo Expenses 1 The mach ae fo 2005 ARMS Phase III was significanly highe fo nonespondens (86.5%) han fo espondens (79.2%) (z = 16.04, p <.05). 2 The sample weigh was used o deemine which opeaions wee o epo expendiue daa in he Census (USDA, 2002). 508 3
Secion on Govenmen Saisics JSM 2008 13. Machiney and Equipmen Value 14. Govenmen Paymens 15. Opeao s Age 16. Opeao s Race 17. Fam Type. Leing y denoe he 2002 Census peliminay-sample o calibaed-sample mean among he ARMS esponden subse fo a sudy vaiable, and y denoe he coesponding peliminay-sample mean among he enie mached sample, i is a simple mae o compue he elaive bias of he fome wih espec o he lae, elbias = significance of his value is much hade o assess since he samples on which ovelapping. y and y y y. The saisical y ae based ae complex and Founaely, we can easily es he pesisence o absence of a sysemaic bias acoss he 20 egions. To his end, we compue he following measue of bias of an ARMS-esponden mean (befoe o afe calibaion) wih espec o he mean of he enie mached sample in evey egion: M = log( y ) log( y ) = log y y y y = log 1 y y y. y This measue is convenienly symmeic, log( y ) log( y ) [log( y ) log( y )], while eaining he scale-invaiance popey of he elaive bias (i.e., muliplying he epoed iem value on each fam by a fixed faco does no affec he oveall elaive bias). The bias measue M fo a sudy vaiable in a egion can be eaed as an independen andom vaiable. The null hypohesis of no bias (again, eihe befoe o afe calibaion) can be esed agains an alenaive hypohesis of a pesisen bias (p%) acoss all he egions. The convenional es based on he 20 obsevaions (one pe egion) is asympoically nomal unde boh he null and alenaive hypoheses. We follow he sandad pacice of appoximaing he disibuion of his es saisic wih a Suden s having 19 degees of feedom. This may lead o libeal infeences (he inappopiae ejecion of he null hypohesis when i is ue) because he M-values fo he sudy vaiable may no be nomally disibued wih a common vaiance acoss egions. Neveheless, by aking logs we ceae a es saisic ha is moe nealy nomal and homoscedasic han elaive biases would be. A sign and a anked-sign es of he 20 paied obsevaions fo a sudy vaiable befoe and afe calibaion was conduced. The sign es is no as poweful as he ohe wo ess (i.e., i moe ofen fails o find ha M is significanly diffeen fom 0 when, in fac, hee is a pesisen bias acoss he egions), bu i assumes neihe ha M is nomal no homoscedasic. The signed-ank es assumes he lae bu no he fome. We include i in ou esuls fo compleeness. 3. Resuls Ou esuls ae summaized in Table 1. Chemical expenses, machiney and equipmen value, govenmen paymens, aces ened, fam ype, fuel and oil expenses, opeao s age, and copland aces (Vaiables 1-8) do no exhibi significan biases using eihe calibaed o uncalibaed weighs. Alhough chemical expenses (Vaiable 1) did no exhibi significan bias, significanly less bias was exhibied using he calibaed weighs vesus he uncalibaed weighs. 4 509
Secion on Govenmen Saisics JSM 2008 In almos 90 pecen (8/9) of he sudy vaiables (9-17) exhibiing pesisen biases using he base sample weighs, calibaion weighing is able o educe he bias so ha i was no longe significanly diffeen fom zeo (9-16) (p <.05) accoding o he -es, and in 50% of hese vaiables we saw a significan educion in bias levels (9-12) (p <.05) accoding o he paied -es. Afe calibaion, only one sudy vaiable, feilize expenses has a significan bias (p <.05) accoding o he -es (bu no accoding o he sign es). The bias of livesock puchases is indicaed o be he lages of he sudy vaiables. Using only he sampling weighs, i was highly significan in ems of each of he es saisics. Afe calibaion, while sill lage in magniude, he indicaed bias was educed o he poin ha i was saisically insignifican accoding o all he es saisics. Fo his vaiable, calibaion does educe he bias significanly if no compleely. 4. Discussion ARMS daa ae used by fam oganizaions, commodiy goups, agibusiness, Congess, Sae Depamens of Agiculue, and he USDA. The USDA uses ARMS daa o evaluae he financial pefomance of fams and anches, which influence agiculual policy decisions. The Depamen also uses he ARMS Phase III daa fo objecive evaluaion of ciical issues elaed o agiculue and he ual economy; heefoe, i is essenial ha measues be aken o minimize he effec of nonesponse bias in ARMS, especially fo Phase III. In he eseach on adjusmen fo nonesponse bias in he 2005 ARMS Phase III, he mean esimaes of feed puchases, oal poducion expenses, oal sales, seed expenses, livesock puchases, copland expenses, oal aces opeaed, hied labo expenses, and feilize expenses, using 2002 Census daa, demonsaed significan bias using jus he base sample weighs. Alhough he magniude of he elaive bias of he mean esimae emained high fo livesock puchases using he calibaed weighs, calibaion educed he magniude of his bias so ha i was no longe significan (see Table 1). Fo his analysis, he calibaion pocess vaied slighly fom ha of he 2005 ARMS Phase III, in ha egg and milk poducion wee no included as calibaion ages, since hese daa iems wee no colleced fo he 2002 Census. This may help o explain why he magniude of he elaive bias of he mean fo livesock esimaes in Table 1 emained high even afe he daa wee calibaed. While i was no possible o use hese as calibaion ages in his analysis, hei use in he ARMS III suvey may educe he bias fo livesock puchases in published ARMS daa. Accoding o Guideline 3.2.13 of he Office of Managemen and Budge Sandads and Guidelines fo Saisical Suveys, NASS should Base decisions egading whehe o no o adjus o impue daa fo iem nonesponse on how he daa will be used, he assessmen of nonesponse bias ha is likely o be encouneed in he eview of collecions, pio expeience wih his collecion, and he nonesponse analysis discussed in his secion. When used, impuaion and adjusmen pocedues should be inenally consisen, sampled on heoeical and empiical consideaions, appopiae fo he analysis, and make use of he mos elevan daa available. If mulivaiae analysis is anicipaed, cae should be aken o use impuaions ha minimize he aenuaion of undelying elaionships. Due o he boadness of he ARMS Phase III daa use communiy and he suvey s impac on agiculual policy, i is cucial ha he calibaion pocess effecively adjuss fo nonesponse bias. Assuming ha he adjusmen pocess is even moe effecive han demonsaed hee (paiculaly fo livesock puchases and feilize expenses) when all calibaion ages (including egg and milk poducion) ae available and used, i appeas ha NASS is appopiaely addessing he issue of nonesponse bias in ARMS Phase III hough he calibaion pocess. Limiaions of his analysis include: 1) Inabiliy o eplicae he 2005 ARMS Phase III calibaion pocess exacly; 2) Inabiliy o assess fams no coveed o esponding o he Census of Agiculue o fo which expendiue daa wee no available; 3) Inabiliy o ecognize localized biases in he ARMS daa (ess wee limied o pesisen biases acoss egions); and 4) Assessmen of nonesponse bias was conduced using 2002 daa as opposed o he 2005 daa, since Census daa is only available evey five yeas. 510 5
Table 1: Mean Compaisons and Indicaed Biases fo Maching Recods Using Base Sampling Weighs vesus Calibaed Weighs Secion on Govenmen Saisics JSM 2008 6 511
Table 1 (Coninued): Mean Compaisons and Indicaed Biases fo Maching Recods Using Base Sampling Weighs vesus Calibaed Weighs Secion on Govenmen Saisics JSM 2008 7 512
Secion on Govenmen Saisics JSM 2008 Knowing ha he analyzed daa come fom he 2002 Census and no fom he 2005 ARMS Phase III Suvey does no limi, bu senghens he analysis. I allows us o focus eniely on he impac of he nonesponse pe se. Fuue eseach includes analyzing nonesponse bias of all sudy vaiables, especially livesock puchases and feilize expenses, using he 2007 Census daa. We expec ha analysis of 2007 Census daa will povide a moe poweful sudy, since he 2007 daa conains equivalen calibaion age vaiables fo egg and milk poducion, expendiue daa fo all Census espondens, and moe cuen efeence daa. Refeences Eap, M.S., McCahy, J.S., Shaue, N.D., & Ko, P.S. (2008), Assessing he Effec of Calibaion on Nonesponse Bias in he 2005 ARMS Phase III Sample Using Census 2002 Daa, Appendix A. Reseach and Developmen Division Saff Repo RDD-08-01, Unied Saes Depamen of Agiculue, Naional Agiculual Saisics Sevice. Hoppe, R. (2006). 2005 Agiculual Resouce Managemen Suvey (ARMS) Phase III Suvey Adminisaion Analysis. Census and Suvey Division Saff Repo SAB-06-14, Unied Saes Depamen of Agiculue, Naional Agiculual Saisics Sevice. Ko, P.S. (2005), Using Calibaion Weighing o Adjus fo Nonesponse and Coveage Eos Suvey Mehodology 32: 133-142. Unied Saes Depamen of Agiculue. (2002). 2002 Census of Agiculue, Vol. 1, Appendix C. Washingon, DC: U.S. Depamen of Agiculue. Unied Saes Depamen of Educaion. (2003). Naional Cene fo Educaion Saisics Saisical Sandads. Washingon, DC: U.S. Depamen of Educaion. Unied Saes Execuive Office of he Pesiden. (2006). Office of Managemen and Budge Sandads and Guidelines fo Saisical Suveys. Washingon, DC: U.S. Execuive Office of he Pesiden. 8 513