Recommendations on Assessment Tools for Monitoring Digital Literacy within UNESCO DLGF

Transcription

1 GAML5/REF/ Recmmendatins n Assessment Tls fr Mnitring Digital Literacy within UNESCO DLGF Maart Laanpere, Tallinn University, Centre fr Educatinal Technlgy GAML Fifth Meeting Octber 2018 Hamburg, Germany

2 2 Intrductin This reprt summarizes a desk research prject that aims t advise UIS in designing an instrument fr assessment f digital literacy skills in the cntext f cllecting data n SDG indicatr The SDG Target 4.4 declares that By 2030, substantially increase the number f yuth and adults wh have relevant skills, including technical and vcatinal skills, fr emplyment, decent jbs and entrepreneurship and it cntains three indicatrs (UIS, 2017): Prprtin f yuth and adults with infrmatin and cmmunicatins technlgy (ICT) skills, by type f skill Percentage f yuth/adults wh have achieved at least a minimum level f prficiency in digital literacy skills Yuth/adult educatinal attainment rates by age grup, ecnmic activity status, levels f educatin and prgramme rientatin UNESCO Institute f Statistics (UIS) is respnsible fr develpment and validatin f new methdlgies fr indicatrs under SDG Target 4.4. While indicatrs and have been already implemented in reprting fr 2017, the status f indicatr is still under develpment (UIS, 2017). Althugh many cuntries have been cllecting statistical data n digital skills r ICT literacy f their citizens fr varius purpses, there is n cmmn agreements n what cnstitutes a "minimum r basic level" f prficiency in digital literacy that wuld allw aggregatin f natinal data n the glbal level. As a result, there exists a serius knwledge gap abut the glbal state f digital literacy skills f yuth and adults, while these skills play increasingly imprtant rle in achieving SDG. As a majr milestne in the prcess f develping UIS's framewrk fr digital literacy, UIS cmmissined a reprt "A Glbal Framewrk f Reference n Digital Literacy Skills fr Indicatr 4.4.2" (Law et al, 2018). This recent reprt reviews digital literacy assessment framewrks used in 47 cuntries and summarizes cnsultatins with a number f experts, resulting with suggestin t use Eurpean DigCmp framewrk as the fundatin fr UIS Digital Literacy Glbal Framewrk (DLGF), while expanding it by five additinal cmpetences and adding tw cmpetence areas. The reprt raises three challenges. First, the need fr mapping existing instruments fr digital skills assessment t DLGF, pinting ut that "... there is nt a ne-size-fits-all assessment f digital cmpetence that can serve all purpses and cntexts". Secnd, it als calls fr cst-effective crss-natinal R&D prgrammes t develp and validate "cntext-sensitive and fit-fr-purpse digital literacy indicatrs and assessment instruments". Third, the reprt pints ut the discrepancy between the prficiency levels and related measurement scales f SDG indicatr versus DigCmp. While SDG Indicatr fcuses n "minimum level f prficiency", DigCmp discriminates eight prficiency levels. There exist sme supra-natinal initiatives r services in the field f digital cmpetence assessment, but thse have fcused n internatinal research prjects invlving several cuntries (e.g. Internatinal Cmputer and Infrmatin Literacy Study, ICILS) r prfessinal certificatin in specific ccupatinal fields (e.g. Internatinal Cmputer Driving Licence, ICDL). Such supra-natinal initiatives culd definitely infrm UIS in designing a glbal instrument fr cllecting reliable and valid data n SDG digital literacy target, but nne f these practices was specifically designed t infrm SDG Indicatr

3 3 Anther relevant categry f digital literacy assessments are supra-natinal framewrks and plicy indicatrs fr measuring digital skills. The clsest t DLGF is a new standard n digital cmpetence framewrk fr Eurpean citizens (DigCmp), which has already been used fr varius purpses in several Eurpean cuntries (Carreter et al., 2017). Eurpean agencies DG Cnnect and EurStat have already used DigCmp t redesign their Digital Skills Indicatr (DSI) in The DSI survey is carried ut n a sample that is representative t general ppulatin f a cuntry, asking respndents abut digital activities carried ut within the previus three mnths and assuming that "persns having realised certain activities have the crrespnding skills" (Eurpean Cmmissin, 2016). The DSI instrument defines three levels f prficiency: belw basic, basic, and abve basic level. Hwever, there is n cmmn Eurpean instrument fr knwledge- r perfrmance-based assessment f digital cmpetence f citizens based n DigCmp. These three challenges raised by Law et al. are addressed by current desk research exercise that has three bjectives: mapping existing digital literacy assessment t DLGF evaluating advantages and disadvantages f selected assessments that cver a large part f the DLGF, with emphasis n their cst-effectiveness fr rllut n ppulatin scale recmmending the next steps n develping an assessment tl suitable fr SDG indicatr Mapping existing Natinal and Crss-Natinal Assessments f ICT and Digital Literacy Skills against the Cmpetencies f DLGF Definitins f Cnstruct Digital literacy is a relatively new cncept that entered t the semantic space that was already partly ccupied by cmpeting cncepts such as ICT, media, infrmatin and cmputer literacy (r cmpetence). Ferrari (2013) was amng the first authrs wh tried t settle the relatinship between these existing labels and newcmer (digital literacy/cmpetence) in a similar manner with the definitin suggested by Law et al (2018): "Digital literacy is the ability t access, manage, understand, integrate, cmmunicate, evaluate and create infrmatin safely and apprpriately thrugh digital technlgies fr emplyment, decent jbs and entrepreneurship. It includes cmpetences that are variusly referred t as cmputer literacy, ICT literacy, infrmatin literacy and media literacy". This definitin builds n previus practices by incrprating vcabulary frm predecessrs (e.g. frm infrmatin, media, ICT literacy framewrks), resulting with a list f 26 cmpetences gruped int seven cmpetence areas. As experience with Eurpean DigCmp has demnstrated, such cmpetence framewrk can be used fr varius pragmatic purpses: re-designing the utdated curricula and prfessinal develpment prgrams, develping plicy indicatrs, prfessinal accreditatin, recruitment and (t a lesser extent) research. Such "pragmatic" cmpetence mdels are usually created by a panel f experts using tp-dwn design apprach, resulting with relatively large sets f cmpetences and 5-8 prficiency levels. These mdels suit well fr the purpse f curriculum design r develping diagnstic self-assessment tl. Hwever, it is quite challenging t cme up with

4 4 a scientifically reliable and valid perfrmance-based test that measures 22 cmpetences f DigCmp n 8 prficiency levels (where each cmpetence is a separate cnstruct t be measured independently). As an alternative t pragmatic apprach, recent psychmetric appraches t measuring digital literacy have been guided by Multidimensinal Item Respnse Thery (MIRT) that understands Cmputer & Infrmatin Literacy (ICILS 2013) r Digital Infrmatin Literacy (Sparks et al, 2016) as a single latent trait that cannt be directly bserved in test situatin and, thus, shuld be inferred indirectly thrugh the statistical analysis f the test results. Like any mathematical mdel, MIRT has sme assumptins that need t be fulfilled in rder t make valid inferences n the basis f test results. Fr instance, mntnicity assumptin requires that the instrument des nt make knwledgeable persns mre likely t participate in the test (Chenery et al., 1988). Assumptin f lcal independence means that perfrmance in ne item in a test des nt influence perfrmance in ther items. While such assumptins are relatively easier t guarantee in case f knwledge-based multiplechice tests, the same might be quite difficult in case f authentic perfrmance-based assessment. Psychmetric tests are ften created in a bttm-up manner, by experimenting with a large set f test items and gruping these n the basis f factr analysis perfrmed n pilt test results. Such assessment instruments (fr instance, ICILS and iskills) are usually fcusing n a single cnstruct, which might be multidimensinal (having several aspects). Tw appraches t digital literacy assessment that were described abve illustrate a struggle between internal and external validity in the cntext f educatinal assessment. Validity in general is understd as degree t which test results can be interpreted and used accrding t the stated purpses f the assessment (AERA et al., 2014). Internal validity refers t methdlgical crrectness/cherence f a research instrument, while external validity can be interpreted as its reusability thrugh the relevance r usefulness fr a wider audience. Pragmatic apprach t defining and measuring digital literacy tends t result with prer internal validity, but higher external validity f the assessment instrument, as it is better understd and accepted by varius stakehlders (mst f whm may nt have backgrund in mathematical statistics r psychmetry). On the ther hand, psychmetric apprach guarantees higher internal validity quite ften in expense f reduced external validity. Law et al (2018) recmmend t use Pathway Mapping methdlgy fr peratinalising DLGF, fcusing rather n users perceptin f digital literacy in varius cntexts and cncerning with external validity f assessment. Eventually, the digital literacy assessment based n DLGF will have t address the challenge f balancing internal and external validity, bth thrugh the methdlgical cnsideratins and design f the digital literacy assessment instrument. Framewrk fr Analysing existing Digital Literacy Assessments In rder t decide abut the methdlgy fr selecting existing DL assessments and mapping these t DLGF, I reviewed sme previus studies that had similar gals. Carreter, Vurikari and Punie (2017) have reviewed 22 existing instruments that are used fr assessment f digital cmpetence in line with DigCmp framewrk in varius Eurpean cuntries. They gruped these instruments int three majr categries based n the data cllectin apprach:

5 5 perfrmance assessment, where individuals are mnitred by human bserver r sftware while being engaged in slving authentic, real-life prblems by using cmmn sftware tls (e.g. brwser, wrd prcessr, spreadsheet) knwledge-based assessment, where individuals are respnding t carefully designed test items that measure bth declarative and prcedural knwledge self-assessment, where individuals are asked t evaluate their knwledge and skills with questinnaires that might range frm structured scales t free-frm reflectin. These appraches can be strengthened by cmbining them and als by cnducting secndary datagathering (e.g. by prviding an e-prtfli that cntains creative wrks, certificates and ther dcumentary evidences). Hwever, it wuld definitely increase the cst and duratin f assessment, while decreasing its scalability. Carreter et al did nt exactly map the selected instruments t DigCmp mdel n the detailed granularity level (t cmpetences and prficiency levels), but their wrk guided me t sme relevant assessments that have nt yet been published. An alternative set f digital literacy assessments and their analysis framewrk is suggested by Sparks et al (2016), wh cnsider bth the fcus f assessment and item design: 1. Assessments having infrmatin literacy fcus with multiple-chice and cnstructed respnse items (such as ILT, SAILS, RSA, ISS, ILAAP, ICILS) 2. Assessments having technlgy literacy fcus with multiple-chice and cnstructed respnse items (such as IC3, ICDL/ECDL) 3. Assessments having digital infrmatin literacy fcus with perfrmance-based tasks (such as CLA+, PIAAC PS-TRE, iskills) Sparks et al (2016) are discussing als variatins in the design f digital literacy assessment instruments caused by different purpse f the test. Sme assessments are designed fr research purpses, while the thers supprt accreditatin r institutinal quality assurance and curriculum design. Yet anther purpse is self-assessment t supprt prfessinal develpment, where the instrument has been designed t meet the needs f test-taker as the primary user. Yet anther alternative typlgy f digital literacy assessments is presented by van Deursen et al (2017), wh are mainly cncerned with differences in item design: Type 1: Self-reprting surveys with questins that ask fr the frequency f technlgy use, which are assumed t deliver indirect evidence fr the cmmand f skills. When an individual uses an applicatin that is cnsidered t be difficult t use (r uses a large variety f applicatins), this is held t be an indicatin f a high level f skills. An example f this kind f assessment is EurStat's Digital Skills Index. Type 2: Surveys with questins that request self-assessments f skills n a predefined scale. This is the mst cmmnly used methd, sme examples are: Digital Skills Wheel by DigitalDannelse, MDS questinnaire by van Deursen.

6 6 Type 3: Perfrmance tests in a labratry r ther cntrlled envirnments that prvide subjects with particular assignments t bserve their cmmand f Internet skills, fr instance: PIAAC PS-TRE test. The mst thrugh and relevant apprach fr analysing digital literacy assessment tls has been applied by Siddiq et al (2016), they have addressed the differences between ICT literacy assessments regarding reliability, validity, match t DigCmp (n the level f single cmpetence) and interactin types used in the item design. The analysis criteria used by Siddiq were: Cuntry f implementatin Year f implementatin Sample size in empirical study Target grup: primary, secndary r high schl (n interest in adults) Item interactin type: multiple chice, interactive, authentic Framewrk used fr cmpetence mdelling Availability f test items r their descriptins Match with adapted DigCmp mdel Data: qualitative, quantitative r mixed Duratin f the test (in minutes) The current study addresses the three categries f instruments fr digital literacy assessment, as described by Carreter et al (2016) t identify the existing practices and evaluate their applicability in the cntext f data cllectin fr SDG indicatr The applicability analysis is mainly inspired by Siddiq et al (2016) and is fcusing n relevance (purpse, match with DLGF, cst-effectiveness) f the given instrument, but als cnsiders its reliability and validity, fllwing the discussin abve. The existing digital literacy assessment practices and instruments were searched frm three types f surces: 1. scientific research publicatins, 2. plicy dcuments in educatin and emplyment dmains, 3. prfessinal certificatin framewrks and related technical dcuments. As a result f this search, 44 assessments were selected fr clser review: 30 frm Siddiq's sample (leaving ut the assessments that were lder than 10 years) and 14 additinal nes, see the Annex 1. In additin t indicatrs used by Siddiq (see the list abve), tw additinal criteria were applied in analysis (inspired by Sparks et al): the purpse f assessment: research, mstly internatinal cmparative studies credentialing the digital skills, mstly fr emplyment and qualificatin

7 7 educatinal statistics, mstly fr plicy planning/evaluatin r quality assurance f educatinal system diagnstic self-assessment, mainly fr persnal use. the fcus f assessment: technical skills: manipulatin f cmmn sftware applicatins, e.g. wrd prcessr, spreadsheets, presentatins, internet brwser, client infrmatin literacy: a single cnstruct that has several facets r dimensins related t skills t find, evaluate, make use f infrmatin digital cmpetence: a set f multiple (20+) cmpetencies that represent varius capabilities f a persn t slve prblems in authentic cntext using digital technlgies. Overview f existing assessments and their mapping t DGLF The current analysis invlved 44 assessments f DL, 7 f thse being designed fr and implemented n a glbal r multinatinal level. The ldest instruments date back t 2006, while the mst recent nes have been develped in The main bdy f cases (30) was taken frm Siddiq's paper that fcused n digital literacy testing nly in primary and secndary schls. Additinal 14 cases represent mstly the DL tests that target wider ppulatin: tw instruments target specifically university students, five are designed fr adult emplyees, tw have been implemented bth in schls and with adult respndents. While a few f the assessments in ur selectin were implemented n relatively small samples (50-60 respndents), majrity f the instruments have been tested with thusands f users. My selectin f DL assessments is cmpsed s that it includes varius alternative designs due t different purpse (R - research, S - statistics, C - credentialing, D - diagnstic self-assessment) r fcus (T - technical skills, I - infrmatin literacy, D - digital cmpetence). There are als variatins in item types (MC - multiple chice items, INT - interactive/dynamic items, AUHT - authentic perfrmance) and duratin f the test, ranging frm 20 t 180 minutes. Mst f the instruments are designed in accrdance with a specific digital/infrmatin literacy framewrk r natinal curriculum (in case f schl-riented tests). In 18 cases ut f 44, the set f test items have been explicitly available fr analysis and mapping with DLGF framewrk, while in the remaining cases I had t rely n infrmatin abut the specific cmpetence within a framewrk/curriculum and prvided examples f tasks. The methd f mapping the specific assessment t DLGF invlved cmparing the test items (r underlying assessment framewrk) with DLGF cmpetencies and judging abut ptential match. Prficiency levels were nt included in analysis, nly cntent f the descriptin f a cmpetency. As a result (see Annex 2), I can cnclude that the best match t DLGF was fund in these existing assessments: DigCmp level test fr 9th and 12th grade in Estnian schls: it cvers the whle DigCmp framewrk, has bth self-reprting and knwledge based test cmpnent, but reliability f items related t cmpetence areas 3-5 are weak, des nt cver DLGF dmains 0 and 6.

8 8 PIX test in France: extremely prmising slutin (bth instrument and pen-surce sftware), advanced platfrm and item design (incl. adaptive testing), des nt cver cmpetence areas 5-6 in DLGF Digital Cmpetence Wheel in Denmark: a gd example f effective and user-friendly instrument, cvers mst f the DLGF cmpetence areas, attractive visual feedback; disadvantage: n knwledge-based items, nly self-reprting. MDS by van Deursen et al (NL and UK): reliable, valid and effective instrument that cvers mst f the DLGF cmpetencies; disadvantage: n knwledge-based items, nly selfreprting.

9 9 Cuntry Year Sample Target Item type Duratin Fcus Purpse Belgium schl INT 100 D R Australia schl MC/INT 120 D C Australia schl MC/INT 120 D C Krea schl MC/INT 50 D R Krea 143 schl MC/INT 50 T R UK schl MC/INT 100 D C Italy schl MC 60 D R Lithuania schl MC/INT 60 D C Krea schl 40 I C Hng Kng 199 schl MC D R Chile schl MC/INT D R Israel schl AUTH I R Germany schl MC/INT 120 I R Nrway schl MC D R Nrway schl MC I R Nrway 1793 schl MC I R Nrway schl MC/INT 70 I R USA schl MC/INT 30 D R USA schl INT 75 I C Krea schl MC/INT 45 T C Krea schl T C Hungary 1470 schl MC 60 T R Hng Kng 1622 schl INT 90 I R Germany schl MC I R USA schl MC/INT I R Hungary schl INT 45 I R Netherl schl AUTH I R Russia schl INT 120 I R glbal schl INT 60 D R USA schl INT 45 D R NL & UK any MC 25 D R Finland schl MC 60 T C/D Estnia schl MC/INT 45 D C EU - DSI 2015 adults MC 20 D S ECDL - glbal adults MC T C IC3 - glbal adults MC/INT 45 T C USA - ILT 2014 adults MC 60 I C USA - SAILS adults MC I C USA - RRSA university MC I R Canada 2016 university MC I C USA - CLA+ any AUTH D C PIAAC 2013 adults INT D R France - PIX 2017 any MC/INT D D Denmark 2017 any MC D D

10 10 Analysis f Advantage and Disadvantage f Different Appraches t Assessment It is likely that perfrmance assessment and additinal analysis f secndary (qualitative) data are nt cst-effective, neither scalable appraches t digital skills assessment in the cntext f glbal assessment f digital literacy in the cntext f SDG. Self-assessment wuld be the easiest and mst cst-effective t implement, but will likely suffer frm lw reliability and validity. Hwever, it shuld be pssible t cmbine self-assessment with knwledge-based r perfrmance assessment. Fr instance, Põldja et al. (2014) have designed and validated an instrument called DigiMina that cmbined self-assessment f teachers' digital cmpetence with peer-assessment, knwledge-based tests and e-prtfli cntaining teacher's reflectins and creative wrks. Within the DigCmpEdu prject, JRC tried t balance internal and external validity f assessing schl's digital capability with the design f the SELFIE tl, s that schls were allwed t expand the scientifically validated cre instrument with additinal items frm pre-designed, publicly available pl r even design their wn additinal items that seemed relevant t them (JRC, 2018). The future instrument that will be designed by UIS fr digital literacy assessment might als benefit frm similar balancing f needs fr glbal standardisatin (cntributing t internal validity) and lcal cntext (cntributing t external validity). The purpse f DL assessment has a majr impact n design f the instrument, its items and testing prcedures. Fr instance, the instruments that are designed fr research purpses apply mstly psychmetric apprach described in 1.1. It results with large effrts invested in item design and validatin, sampling and data cllectin prcedures, and analysis f results. The test items are mstly knwledge-based, cmbining multiple chice and interactive tasks, in a few ccasins als simulatins (PIAAC PS-TRE, OECD PISA). The data is cllected in a shrt time frame, in cntrlled envirnment (grup wise in cmputer labs). This type f DL assessments is cstly, ften funded directly by gvernments and repeated infrequently (nce every 3-5 years). The DL assessments designed fr statistical purpses (e.g. EurStat DSI) are als applied n carefully cmpsed sample, but the design f the instrument and items (mstly multiple-chice items) is significantly cheaper and allw frequent use, expecting lw effrt frm respndents and analysts. Data cllectin fr statistical purpse takes usually place in relatively wide and flexible time frame in lsely cntrlled settings. Cmpared with research-fcused DL assessments, statistical data gathering is clearly less reliable and weaker regarding the validity (bth cnstruct validity and internal validity). DL assessments designed fr credentialing purpses (e.g. ICDL/ECDL) address mainly the needs f emplyers r academic institutins wh are increasingly mre cncerned abut cmparability f digital skills f their emplyees r students. This is why credentialing-fcused assessments are designed fr frequent and scalable use in highly cntrlled envirnment. High level f standardizatin and fcus n universal, practical, technical skills (e.g. wrd prcessing, spreadsheets) increases reusability f items and instruments ver time and acrss natins, thus reducing the cst that is left t be cvered mainly by test-takers. Althugh the credentialing-type f DL assessment are usually highly reliable, their cnstruct validity and internal validity is likely weaker than thse f research-fcused instruments. Diagnstic assessments f DL d nt target high reliability and internal validity, as they priritize external validity and perceived usefulness f assessment t the end users (test-takers). Diagnstic self-assessments (e.g. DigiMina, DigitalCmpetenceWheel, MDS) are likely t include mainly selfreprting scales and smetimes als authentic tasks. Authentic assessment reduces the pssibilities

11 11 fr autmatic scring and requires invlvement f human reviewers. This is why such assessments are cstly and d nt scale up, are mstly used n the level f a single rganizatin. Hwever, diagnstic self-assessment that uses nly multiple-chice items (e.g. DigitalCmpetenceWheel, MDS) is scalable and lw-cst, suffering mainly fr lw reliability and biased sample, as persns with lw self-esteem and pr digital skills are unlikely t take the test. Lking frm the perspective f UNESCO DLGF, it is pssible t cmbine tw r three types f DL assessment. As the aim f evaluating SDG Target is t establish "the minimum r basic level f prficiency in digital literacy" amng the ppulatin f Earth, the statistical purpse seems t be the main pririty. Yet, the uptake, reliability and external validity f the future DLGF-based assessment can be increased by adding sme elements f credentialing-fcused apprach r diagnstic selfassessment apprach. Recmmendatin n Assessment Tls fr Mnitring Indicatr Based n the analysis abve, five general recmmendatins are prvided regarding the design and delivery f DLGF based Digital Literacy assessment: the cre test f DLGF shuld be based n self-reprting, similar t Danish Digital Cmpetence Wheel and Dutch MDS, respnses n 3-5 pint scale - nne, basic, advanced; test duratin shuld nt exceed min, autmatic assessment fr all items; test shuld be pilted with at least 1000 respndents in 3 different languages, different cuntries, t validate the items (CFA); an internatinal steering grup f researchers shuld be frmed t evaluate and imprve quality f items and prcedures, als reliability and validity f assessment. t increase the validity and reliability, enhance the self-reprting instrument with additinal knwledge-based test (similar t PIX, MC/INT items, adaptive); there's n need t cver the whle DLGF mdel with this additinal test - nly the cmpetencies that were ranked higher by respndent; knwledge-based test items shuld be cmpsed in line with Evidence Centered Assessment Design (ECD) apprach, ready fr MIRT analysis. sftware architecture fr testing tl shuld be similar t PIX, with built-in data uplad t UIS database (in prcessed, annymized frm); sftware and test items shuld fllw annual versining in Github; user interface has t be respnsive, test runs als n smartphnes and tablets; items and tests shuld be available fr exprt in IMS QTI 2.2 frmat; special attentin t meeting high natinal (and EU) standards fr privacy and data prtectin: data is annymized in early stage, n private infrmatin f respndents stred. sftware architecture (API, interfaces, prtcls) shuld enable ptential extensins fr e- prtflis (fr diagnstic self-assessment) and micrcredentials (e.g. Open Badges).