Generalizability Theory: An Analysis of Variance Approach to Measurement Problems in Educational Assessment

Transcription

1 Journal of Stude n Educaton ISSN , Vol., No. 1 Generalzablty Theory: An Analy of Varance Approach to Meaurement Problem n Educatonal Aement Huan Alkharu College of Educaton, Sultan Qaboo Unverty P.O.Box: 3 Al-Khod, P.C.: 13, Sultanate of Oman Tel: E-mal: huen5@qu.edu.om Receved: January 3 Accepted: February 0 Publhed: February, 01 do:10.596/je.v1.17 URL: Abtract Of ncreang nteret to the educatonal aement reearcher the relablty of core obtaned by varou meaurement procedure lke tet, ratng cale, urvey, and obervaton form. Tradtonal method of relablty baed on Clacal Tet Theory (CTT) conder only one ource of meaurement error. Generalzablty Theory (GT) extend CTT by provdng a flexble and practcal framework for etmatng the effect of multple ource of meaurement error through an applcaton of analy of varance procedure. It help educatonal aement reearcher to determne approprate condton n term of tem, occaon, rater conducve to obtanng an optmal level of core relablty. Th paper hghlght the utlty and applcablty of the GT analy n the educatonal aement reearch. The paper begn wth a bref htory of the GT tracng t growth and development and then dcue t defnng feature and advantage and dadvantage. The paper llutrate the applcaton of the GT n the educatonal aement reearch. The paper cloe wth a recommendaton for educatonal aement reearcher to take advantage of the GT n the development of meaurement procedure and further reearch agenda n th area. Keyword: Generalzablty theory, Educatonal meaurement, Relablty 184

2 Journal of Stude n Educaton ISSN , Vol., No Introducton The generalzablty theory (GT) can be vewed a an extenon of the clacal tet theory CTT through an applcaton of certan analy of varance (ANOVA) procedure to meaurement ue. Jut a the reearcher, ung ANOVA, attempt to dentfy and etmate the effect of potentally mportant factor, GT reearcher attempt to dentfy and etmate the magntude of the potentally mportant ource of meaurement error. The connecton between GT and CTT paralleled by the connecton between factoral and mple ANOVA. A reearcher, ung mple ANOVA, partton varance nto two component, uually named between-group varance and wthn-group varance. The between-group varance thought of a a ytematc varance aocated wth the factor that dtnguhe group from one another. The wthn-group varance thought of a random and treated a error. In the ame way, CTT partton varance nto true-core varance and error varance. The former the thought of a a ytematc varance aocated wth dfference between object of meaurement. The latter treated a random varance unrelated to the true-core varance (Shavelon & Webb, 1991). By applyng factoral ANOVA ntead of mple ANOVA, the reearcher acknowledge multple factor contrbutng to the total varance n the obervaton, and hence partton t nto part correpondng to each factor, to nteracton among the factor, and to a random error. Smlarly, GT acknowledge multple nfluence on meaurement varance. Wherea CTT, lke mple ANOVA, partton varance nto only two ource, GT, lke factoral ANOVA, partton varance nto many ource correpondng to a ytematc varance among the object of meaurement, to multple error ource, and to ther nteracton (Shavelon & Webb, 1991). Th paper attempt to provde an overvew of the fundamental of the GT. It hghlght t feature, aumpton, advantage, and dadvantage. The paper begn wth a bref htory of the GT to trace t growth and development. Then, an example provded to llutrate the applcaton of the GT n the educatonal aement reearch. The example focue on the mple one-facet degn, whch the mot common meaurement degn ued n the GT analy. However, t hould be noted that the generalzablty analy preented n th paper can be extended to almot any type of unvarate and multvarate degn contng of two or more factor, whch may be random or fxed, and whch may be croed or neted. It hould be acknowledged that the paper an expotory ummary of the GT whch orgnated from Brennan (001), Shavelon and Webb (1991), and other meaurement expert. The reader are encouraged to utlze the reference cted n the paper for more detal about the topc.. A bref Htorcal Overvew of GT Table 1 ummarze mportant and practcal contrbuton n the htory of GT (Brennan, 1997; Crocker & Algna, 1986; Shavelon & Webb, 1981). Although everal reearcher can be credted wth pavng the way for GT (e.g., Burt, 1936; Hoyt, 1941), t wa formally ntroduced by Cronbach and h aocate a an extenon to the CTT (Cronbach, Gleer, Nanda, & Rajaratnam, 197; Cronbach, Rajaratnam, & Gleer, 1963; Gleer, Cronbach, & Rajaratnam, 1965)

3 Journal of Stude n Educaton ISSN , Vol., No. 1 Table 1. A ummary of mportant and practcal contrbuton n the htory of GT Year Contrbuton 1936 Burt appled ANOVA approach to meaurement problem n the analy of examnaton mark Hoyt howed that ANOVA can be employed to compute relablty coeffcent by treatng tudent and tem a ource of varaton Fnlayon' tudy of grade agned to eay wa probably the frt treatment of relablty n term of varance component. - Ebel publhed an artcle on the relablty of ratng n whch he condered two type of error varance: one that ncluded and another that excluded rater man effect. Ebel alo condered ngle-facet croed and neted degn to 1959 The rater man effect n Ebel' (1951) artcle played the role of the tem man effect n Lord' (1955, 1957, 1959) artcle about condtonal tandard error of meaurement and relablty under the aumpton of the bnomal error model. Ebel and Lord' work were eventually captured the dtncton between relatve and abolute error n GT to 1965 Cronbach and h colleague Gleer and Rajaratnam had pretty much completed ther developmentnof unvarate GT Cronbach and Nada began the work on multvarate GT, n whch each of the level of one or more fxed fact aocated wth a dtnct unvere core. 197 Cronbach, Gleer, Nada, Rajaratnam publhed a book enttled, The Dependablty of Behavoral Meaurement: Theory of Gneralzablty for Score and Profle. Th monograph tll the mot defntve treatment of the GT to 1981 Cardnet, Tourneur, and Allal emphazed the role that facet other than tudent mght play a object of meaurement, whch wa known a prncple of ymmetry of the GT Crck and Brennan degned a computer program called GENOVA for conductng GT analy Feldt and Brennan devoted about one thrd of ther chapter on relablty to GT Shavelon and Webb publhed a relatvely hort monograph enttled, Generalzablty Theory: A prmer

4 Journal of Stude n Educaton ISSN , Vol., No Brennan provded a very bref ntroducton about GT n the Educatonal Meaurement: Iue and Practce ntended prmarly for claroom ue. 3. Defnng Feature of GT GT a meaurement theory for etmatng the dependablty of meaurement obtaned by any knd of procedure lke tet, ratng cale, urvey, and obervaton form. Dependablty refer to the accuracy of generalzng from a tudent' oberved core on a tet or other meaure to the average core that tudent would have receved under the poble condton uch a all poble form, all poble tetng occaon, or all poble tem. Th average core called a "unvere core", whch analogou to the CTT' concept of "true-core". Thu, GT conder core a dependable f they permt accurate nference about the unvere of admble obervaton that they are meant to repreent (Allal & Cardnet, 1997; Shavelon & Webb, 1991). In GT, any oberved core condered to be a ample from a unvere of admble obervaton. Th unvere cont of all poble obervaton that would be acceptable a ubttute for the obervaton n queton. For example, a core obtaned by a partcular tudent on a partcular tetng day not the only acceptable ndcator of h or her performance. A core obtaned on a dfferent day would alo be acceptable, a would a core from a dfferent form of the ame tet, or pobly a dfferent et of tem from the ame form. Each of thee charactertc of the tetng tuaton called a facet, and the level of the facet are called condton. The term "facet" and "condton" are analogou to "factor" and "level" n the lterature on expermental degn (Shavelon & Webb, 1991; Webb, Rowley, & Shavelon, 1988). The unvere defned n term of the facet of the obervaton that determne the condton under whch an acceptable core can be obtaned. A one-facet unvere defned by one ource of meaurement error. For example, f the decon maker want to generalze from the core on one et of tet tem to a much larger et of tet tem, tem are a facet of the meaurement and the tem unvere would be defned by all admble tem. If the decon maker want to generalze from performance on one occaon to performance on a much larger et of occaon, occaon are a facet and the occaon' unvere would be defned by all admble occaon (Shavelon & Webb, 1991). Tradtonal method of relablty that are baed on CTT conder only one ource of error n a meaurement at tme. For example, tet-retet relablty conder only the occaon of tetng a the ource of error. Parallel-form relablty conder the form of the tet a the only ource of error. Internal contency relablty conder only the tem a the ource of error. A uch, CTT provde very lmted nformaton. More pecfcally, nformaton about one knd of relablty (e.g., tet-retet) cannot be ued to make nference about other knd of relablty ( e.g., nternal contency). Furthermore, even f dfferent knd of relablty are preented, t dffcult to ue the combned nformaton to determne how many tet form, tem, and occaon need to be ued to obtan dependable meaure (Webb, Rowley, & Shavelon, 1988)

5 Journal of Stude n Educaton ISSN , Vol., No. 1 However, GT extend CTT by provdng a flexble and practcal framework for etmatng the effect of multple ource of error. In partcular, CTT tate that an oberved core for any tudent obtaned through ome meaurement procedure can be decompoed nto the true core and a ngle error. In contrat, GT recognze that multple ource for error uch a error attrbuted to tet tem, tetng occaon, and tet form may occur multaneouly n the meaurement proce. A uch, the bac approach underlyng GT to decompoe an oberved core nto a component for the unvere core and one or more error component (Crocker & Algna, 1986; Shavelon & Webb, 1991). 4. Aumpton of GT The aumpton underlyng GT are bacally the ame aumpton of CTT. Frt, the data examned n a generalzablty analy hould be nterval or ordnal n nature. Second, GT aume that a tudent' oberved core compred of h or her unvere core and/or more ource of error. Thrd, the error are aumed to be ndependent of the unvere core and uncorrelated. In other word, all of the effect n the meaurement model are ndependent. Fourth, GT aume that the ample ued to etmate the error varance and elected of tudent, tem, or occaon and compre random ample from ther repectve populaton. However, thee fact can ometme be treated a fxed. In partcular, the concept of "randomne" tate that even though condton of a facet have not been ampled randomly, the facet may be condered to be random f condton not oberved n the tudy can be exchanged wth the oberved condton. For example, f the reearcher wllng to exchange the 30 tem on a tet for another ample of 30 tem, the facet mght be treated reaonably a random. The ffth aumpton that the tandard error are the ame at all core level. In other word, the ame tandard error of meaurement often appled to all object of meaurement regardle of the underlyng unvere core (Shavelon & Webb, 1991; Strube, 00). 5. An llutraton of Generalzablty Analy To llutrate the applcaton of GT, conder the followng example: A tet contng of a random ample of n = 5 tem, from a unvere of tem, wa admntered to a random ample of n = 5 tudent, from a populaton of tudent. Table preent hypothetcal data for th example. Th degn called a one-facet degn becaue the tem facet the only facet of potental meaurement error beng nvetgated. The generalzablty queton that how dependable are core made under dfferent condton of tet tem to draw nference about the unvere contng of all condton? 188

6 Journal of Stude n Educaton ISSN , Vol., No. 1 Table. Data from a hypothetcal one-facet degn Student Item The oberved core ( X ) for one tudent () on one tem () can be expreed n term of the followng lnear model (Brennan, 001; Shavelon & Webb, 1991): X e, model (1) Where = the overall mean n the populaton of tudent and unvere of tem. = the core effect attrbutable to tudent. = the core effect attrbutable to tem. = the core effect attrbutable to the nteracton of tudent and tem. e = the random error for tudent' core. Becaue there only one core for each tudent-tem combnaton n Table, the term and e are confounded. In other word, after accountng for tudent effect and tem effect, we do not know f dfference between tem core reflect the tudent-tem nteracton, the random varablty, or both. Conequently, th confoundng repreented wth the notaton (,e) and often referred to a the redual effect. Thu, model (1) mplemented a a two-way non-factoral analy n whch the nteracton not etmated. A uch, model (1) can be wrtten a follow (Brennan, 001; Shavelon & Webb, 1991): X, e, model () The unvere core for tudent ( ) defned a the expected value of a tudent' oberved 189

7 Journal of Stude n Educaton ISSN , Vol., No. 1 core acro the unvere of tem. Smlarly, the populaton mean for tem ( ) defned a the expected value over tudent. The bac aumpton underlyng model () are that all effect are ampled ndependently, and the expected value of each effect over the populaton of tudent and the unvere of tem equal to zero. Gven thee aumpton, the model condered a random-effect tudent-croed-wth-tem ( ) one-facet degn. In fact, the choce of the approprate ANOVA model determned by the model of amplng, ether fxed or random, of the level of each facet (Brennan, 001; Shavelon & Webb, 1991). Once the data have been collected, the tandard procedure of ANOVA are appled to determne the mean quare and to etmate the varance component correpondng to all ource of varaton n the degn. For th mple degn n whch tudent are croed wth tem, varance component can be etmated by the random effect model for the three ource of varaton: tudent ( ), tem ( ), and redual (,e ). Table 3 how the tandard ANOVA table for the tudent-by-tem degn along wth correpondng computatonal formula (Brennan, 001). Table 3. ANOVA formula for the tudent-by-tem degn Source of varaton Sum of quare Degree of freedom Mean quare Varance component Student () SS n ( X X) df n 1 SS MS df S MS n MS, e Item () SS n ( X X) df n 1 SS MS df MS MSe, n Redual (,e) SS ( XX X X df ( n 1)( n 1), e ), e MS SS e,, e MS e e,, dfe, Table 4 preent the ANOVA reult from the hypothetcal one-facet tudy. The varance component reveal nformaton about how dfferent ource of varablty affect the repone to a ngle tem. To nterpret the magntude of the etmated varance component, we can take the um of the varance component, called the total varance, and create percentage of th um that each etmated varance component account for (Shavelon & Webb, 1991). In th cae, the varance component for tudent account for 18% of the total varance, uggetng that averagng over all the tem, the tudent n the ample dffer n the contruct beng meaured. In th example, tudent conttute the object of meaurement, not error, and a uch th varablty derable. It reflect ytematc ndvdual dfference n the contruct beng meaured. Th varance component known a the unvere core varance. It preent the varance of core averaged over all condton of tet tem, defned by the unvere of admble obervaton (Strube, 00)

8 Journal of Stude n Educaton ISSN , Vol., No. 1 The varance component for tem account for 37% of the total varance, uggetng that ome tem were more dffcult than other, averagng over all the tudent. The redual term account for 45% of the total varance. Th ugget that there are mportant ource of varance not accounted for by dfference between tudent, dfference n tem dffcultly, or both. It hould be emphazed that n GT, the magntude of the etmated varance component are of central mportance rather than ther tattcal gnfcance (Brennan, 001). Once the varance component have been etmated, the prncple of GT are ued to determne the allocaton of the component for the etmaton of two type of error varance: relatve and abolute error varance. Table 4. ANOVA etmate of varance component for the one-facet croed degn example Source of varaton Student () Sum of quare Degree of freedom Mean quare Varance component Item () Redual (,e) Relatve Error Varance % of total varance GT dtnguhe between two type of error varance that correpond to relatve decon and abolute decon. Relatve decon are decon about ndvdual dfference between tudent. Abolute decon are decon about the abolute level of performance (Shavelon & Webb, 1991; Strube, 00). The relatve error varance ( ) of prmary concern when reearcher are ntereted n decon that nvolve the rank orderng of ndvdual. In th cae, the error ource are lmted to the nteracton of the ndvdual wth the facet() formed by random amplng of the condton of meaurement. Th becaue nteracton nvolvng the object of meaurement reflect change n relatve tandng acro facet level (Brennan, 001; Shavelon & Webb, 1991; Strube, 00). For the one-facet degn preented n Table, the etmate of the relatve error varance can be found by averagng the redual varance over the number of tem ued n the meaurement (Brennan, 001). Ung the etmate obtaned n Table 4, the relatve error varance etmate.37. The quare root of th ndex, whch.61, condered the relatve tandard error of meaurement (Shavelon & Webb, 1991; Strube, 00). 7. Abolute Error Varance For ome tuaton, partcularly n the area of crteron-referenced aement, one may make decon about whether a tudent can perform at a prepecfed level. In thee ntance, 191

9 Journal of Stude n Educaton ISSN , Vol., No. 1 the abolute error varance ( ) of concern (Brennan, 001; Shavelon & Webb, 1991). It reflect both nformaton about the rank orderng of tudent and any dfference n the average core (Shavelon & Webb, 1991). All ource other than the object of meaurement are a ource of error for abolute decon (Strube, 00). A uch, n the one facet ( ) degn example, the abolute error varance nclude the varance component due to both the tem effect and the redual effect averaged over the number of tem ued n the meaurement. Ung the etmate obtaned n Table 4, the etmate of the abolute error varance.51. The quare root of th ndex, whch.71, repreent the abolute tandard error of meaurement (Shavelon & Webb, 1991; Strube, 00). 8. Generalzablty Coeffcent The dependablty of a meaurement procedure aeed by a generalzablty coeffcent, an ndex that analogou to the CTT' relablty coeffcent. It range from 0 to 1 wth hgher value reflectng more dependable meaurement procedure. Value approachng 1 ndcate that the core of nteret can be dfferentated wth a hgh degree of accuracy depte the random fluctuaton of the meaurement condton (Alla & Cardnet, 1997; Shavelon & Webb, 1991; Strube, 00). The generalzablty coeffcent are avalable for both relatve error and abolute error. For the cae of relatve comparon of oberved core, the correpondng etmate of the generalzablty coeffcent for the ( ) degn defned by the followng formula (Barennan, 001; Shavelon & Webb, 1991):, If decon were baed on the abolute value of the oberved core, the correpondng etmate of the generalzablty coeffcent would be a follow (Brennan, 001; Shavelon & Webb, 1991):, Ung the value of the one-facet degn preented n Table 4, the etmate of the generalzablty coeffcent are =.48 and =.40. A ndcated earler, GT provde a framework for examnng the dependablty of meaurement procedure. Performng a generalzablty analy to pnpont the ource of meaurement error allow the reearcher to determne how many condton of each facet are needed (e.g., number of tem, number of occaon) to obtan an optmal level of generalzablty (Marcoulde & Goldten, 1990). The tem, n the degn preented n Table, repreent a ource of meaurement error. Thu, ncreang the number of tem n the 19

10 Journal of Stude n Educaton ISSN , Vol., No. 1 meaurement procedure ncreae the generalzablty coeffcent becaue an ncreae n the number of tem would lead to a decreae n the etmate of both relatve and abolute error. 9. Generalzablty Stude and Decon Stude There are two tage n the applcaton of GT. The frt the generalzablty (G) tudy carred out by the developer of the meaurement procedure. It degned to provde nformaton about the ource of varablty (.e., facet of the unvere) that nfluence the generalzablty of obervaton. On the ba of th nformaton, varou modfcaton of the ntal G tudy degn can be analyzed n the econd tage called a D tudy (Shavelon & Webb, 1981, 1991). The purpoe of D tudy to determne the mot approprate meaurement procedure for a partcular tuaton n whch the nformaton obtaned n the G tudy wll be ued. For example, f the reult of a G tudy how that ome ource of error are mall uch a the error attrbutable to tem, then the reearcher may elect a meaurement procedure that reduce the number of level of the facet (.e., number of tem) or elect to change the actual data collecton degn, ay, for example, from a croed ( ) degn n whch every tudent wa admntered the ame ample of tet tem to degn n whch the tem are neted wthn tudent (.e., each tudent admntered a dfferent ample of tem). Alternatvely, f the reult of a G tudy how that ome ource of error n the degn are large, the reearcher may need to ncreae the level of that facet n order to obtan an acceptable level of generalzablty. In general, the D tudy addreed the queton: What hould be done dfferently f you are gong to rely on th meaurement procedure for makng future decon? In the cae where no change hould be made, the G tudy act a the D tudy (e.g., employ the ame ample of tem ued n the ntal G tudy) (Brennan, 001; Shavelon & Webb, 1991). 10. Advantage and Dadvantage of GT The foregong dcuon ndcate that GT ha four man advantage (Shavelon, & Webb, 1991; Thompon, 003): 1. It can be ued to ae multple ource of error n a pecfc meaurement tuaton. Compared to CTT, GT provde more reaonable etmate of dependablty n crcumtance where multple ource of error are preent. A reearcher wth a meaure contng of everal tem done by dfferent rater to meaure one contruct n repondent erouly nflatng relablty when only lookng at the eparate relablty etmate.. It nform the reearcher about the magntude of the type of error, o that decon concernng whether error magntude are wthn acceptable range can be appled to future tude. Hence, a dered level of generalzablty can be obtaned n thoe tude. If error are condered larger than dered, reearcher can ue GT to more precely evaluate method for reducng the error n future tude. Once a reearcher know whch ource of error, he or he can dentfy thoe error ource whch are mutable, plan way to reduce them, and degn the mot optmal future meaurement tuaton

11 Journal of Stude n Educaton ISSN , Vol., No GT dtnguhe between relatve and abolute decon. Relatve decon are thoe ued to compare ndvdual to each other. Abolute decon are thoe baed on an ndvdual' abolute level of performance. 4. The poblty of treatng ource of error a fxed or random allow the reearcher the flexblty to conder meaurement error that wll generalze to ether a unvere of facet or to only a fxed number of facet. Depte the power of GT, t ha a number of dadvantage or lmtaton (Shavelon, & Webb, 1991; Strube, 00; Webb, Rowley, & Shavelon, 1988). Thee nclude: 1. It ha not been readly acceble to reearcher becaue of t techncal development and preentaton.. It may reult n coeffcent that are tethered to the partcular ample ued n conductng the tudy. The ablty to generalze the fndng to another populaton lmted, partcularly when level of facet of the larger populaton are not ncorporated n the tudy' ample. 3. It applcaton requre ubtantal effort n the degn, data collecton, and analy. Etmaton of error ource requre that the degn of the generalzablty tudy nclude all relevant facet and that enough data are collected on each facet to accurately etmate t error varance. 4. By takng multple ource of error nto account, the generalzablty coeffcent tend to be lower than relablty coeffcent from CTT. 11. Concluon GT, a a meaurement theory, provde a framework for examnng the dependablty of almot any type of meaurement procedure n almot any type of degn. It extend CTT n everal mportant way. Frt, t recognze multple ource of meaurement error, etmate each ource eparately, and provde a mechanm for optmzng the relablty. Second, although GT provde a relablty coeffcent, called a "generalzablty coeffcent", the theory focue on varance component that ndex the magntude of each ource of error. Thrd, GT dtnguhe between relatve decon, where nteret focue on the dependablty of the dfference among ndvdual, and abolute decon, where core are themelve nterpretable wthout reference to other' performance. Fourth, GT dtnguhe between generalzablty (G) tude and decon (D) tude. G tude etmate the magntude of a many potental ource of meaurement error a poble. D tude ue nformaton from a G tudy to degn a meaurement that mnmze error for partcular purpoe. Although GT provde the mot flexble and practcal approach for examnng the dependablty of the meaurement procedure, t ha not been readly acceble to reearcher becaue of t techncal development. It hould be emphazed that GT a contnuou work n progre, where there are ome mportant theoretcal and tattcal topc that clearly need to be addreed more fully, and that there are potental area of applcaton where the theory ha been largely not ued

12 Journal of Stude n Educaton ISSN , Vol., No. 1 Reference Allal, L., & Cardnet, J. (1997). Generalzablty theory. In J.P. Keere (Ed.), Educatonal reearch, methodology, and meaurement: An nternatonal handbook ( nd, pp ). Cambrdge, Unted Kndom: Cambrdge Unverty. Brennan, R.L. (199). Generalzablty theory. Educatonal Meaurement: Iue and Practce, 11, Brennan, R.L. (1997). A perpectve on the htory of generalzablty theory. Educatonal Meaurement: Iue and Practce, 16, Brennan, R.L. (001). Generalzablty theory. New York: Sprnger. Burt, C. (1936). The analy of examnaton mark. In P. Hartog & E.C. Rhode (Ed.), the mark of examner (pp ). London: Macmllan. Cardnet, J., Tourneur, Y., & Allal, L. (1976). The ymmetry of generalzablty theory: Applcaton to educatonal meaurement. Journal of Educatonal Meaurement, 13, Cardnet, J., Tourneur, Y., & Allal, L. (1981). Extenon of generalzablty theory and t applcaton n educatonal meaurement. Journal of Educatonal Meaurement, 18, Crck, J.E., & Brennan, R.L. (1983). Manual for GENOVA: a generalzed analy of varance ytem (Amercan College Tetng Techncal Bukketn No. 43). Iowa Cty, IA: Amercan College Tetng. Crocker, L., & Algna, J. (1986). Introducton to clacal and modern tet theory. Belmont, CA: Wadworth Group/ Thomon Learnng. Cronbach, L. J., Gleer, G. C., Nanda, H., & Rajaratnam, N. (197). The dependablty of behavoral meaurement: Theory of generalzablty core and profle. New York: Wley. Cronbach, L. J., Rajaratnam, N., & Gleer, G. C. (1963). Theory of generalzablty: A lberzaton of relablty theory. Brth Journal of tattcal Pychology, 16, Ebel, R. L. (1951). Etmaton of the relablty of ratng. Pychometrka, 16, Feldt, L. S., & Brennan, R.L. (1989). Relablty. In R. L. Lnn (Ed.), Educatonal meaurement (3 rd ed., pp ). New York: Macmllan. Fnlayon, D. S. (1951). The relablty of markng eay. Brth Journal of educatonal Pychology, 35, Gleer, G. C., Cronbach, L. J., & Rajaratnam, N. (1965). Generalzablty of core nfluenced by multple ource of varance. Pychometrka, 30,

13 Journal of Stude n Educaton ISSN , Vol., No. 1 Hoyt, C. J. (1941). Tet relablty etmated by analy of varance. Pychometrka, 6, Lord, F. M. (1955). Etmatng tet relablty. Educatonal and Pychologcal Meaurement, 15, Lord, F. M. (1957). Do tet of the ame length have the ame tandard error of meaurement? Educatonal and Pychologcal Meaurement, 17, Lord, F. M. (1959). Tet of the ame length do have the ame tandard error of meaurement? Educatonal and Pychologcal Meaurement, 19, Marcoulde, G.A., & Goldten, Z., (1990). The optmzaton of generalzablty tude wth reource contrant. Educatonal and Pychologcal Meaurement, 50, Shavelon, R.J., & Webb, N.M. (1981). Generalzablty theory: Brth Journal of Mathematcal and Stattcal Pychology, 34, Shavelon, R.J., & Webb, N.M. (1991). Generalzablty theory: A prmer. Newbury Park, CA: Sage. Strube, M. J. (00). Relablty and generalzablty theory. In L.G. grmm & P.R. Yarnold (Ed.), Readng and undertandng more multvarate tattc (pp. 3-66). Wahngton, DC: Amercan Pychologcal Aocaton. Thompon, B. (003). A bref ntroducton to generalzablty theory. In B. Thompon (Ed.), Score relablty: Contemporary thnkng on relablty ue (pp ). Thouand Oak, CA: Sage. Webb, N.M., Rowley, G. L., & Shavelon, R. J. (1988). Ung generalzablty theory n counelng and development. Meaurement and Evaluaton n Counelng and Development, 1,