Wrkplace Digital Essential Skills in Rural Small Businesses Nrm Leckie Julie Rdier David Gyarmati May 2016
SRDC Bard f Directrs Richard A. Wagner Partner, Nrtn Rse Fulbright LLP Grdn Berlin President, MDRC Maria David-Evans IPAC Immediate Past President and Frmer Deputy Minister, Gvernment f Alberta Rbert Flynn, Ph.D. Emeritus prfessr, Schl f Psychlgy, University f Ottawa Pierre-Gerlier Frest, Ph.D., FCAHS Directr and Palmer Chair Schl f Public Plicy University f Calgary Suzanne Herbert Frmer Deputy Minister, Gvernment f Ontari The Scial Research and Demnstratin Crpratin (SRDC) is a nn-prfit research rganizatin, created specifically t develp, field test, and rigrusly evaluate new prgrams. SRDC's tw-part missin is t help plicy-makers and practitiners identify plicies and prgrams that imprve the well-being f all Canadians, with a special cncern fr the effects n the disadvantaged, and t raise the standards f evidence that are used in assessing these plicies. Since its establishment in December 1991, SRDC has cmpleted ver 250 prjects and studies fr varius federal and prvincial departments, municipalities, as well as ther public and nn-prfit rganizatins. SRDC has ffices lcated in Ottawa, Trnt, and Vancuver, and satellite ffices in Calgary and Winnipeg. Guy Lacrix, Ph.D. Prfessr f Ecnmics, Université Laval Renée F. Lyns, Ph.D. Funding Chair and Scientific Directr Emeritus, Bridgepint Cllabratry fr Research and Innvatin, University f Trnt Sharn Mansn Singer, Ph.D. Frmer President, Canadian Plicy Research Netwrks Frmer Deputy Minister, Gvernment f British Clumbia Jim Mitchell Funding partner f Sussex Circle SRDC President and CEO Jean-Pierre Vyer Fr infrmatin n SRDC publicatins, cntact Scial Research and Demnstratin Crpratin 55 Murray Street, Suite 400 Ottawa, Ontari K1N 5M3 613-237-4311 1-866-896-7732 inf@srdc.rg www.srdc.rg Vancuver Office 789 West Pender Street, Suite 440 Vancuver, British Clumbia V6C 1H2 604-601-4070 604-601-4080 Trnt Office 481 University Avenue, Suite 705 Trnt, Ontari M5G 2E9 416-593-0445 647-725-6293 Published in 2016 by the Scial Research and Demnstratin Crpratin
Digital Essential Skills in Rural Small Businesses Digital Essential Skills in Rural Small Businesses is a natinal pilt prject that was implemented by Restiguche Cmmunity Business Develpment Crpratin (CBDC) frm January 2012 t March 2016. The aim f this prject is develp a flexible training platfrm that increases access t basic wrkplace digital skills training in small rural businesses, specifically, amng lw literacy wrkers, wh ften lack bth digital skills and suitable access t digital skills training. In respnse t this gap, Restiguche CBDC, in cncert with a private learning sftware develper, created an nline digital skills training platfrm. The design and implementatin f this training mdel was assessed in this pilt prject with a view t supprting the wider distributin f the platfrm in the future. The purpse f this reprt is t present the results f the evaluatin by the Scial Research and Demnstratin Crpratin, which assesses the success f the implementatin n a number f pre-determined indicatrs and describes the utcmes f participants and their emplyers. Prject Objectives and Research Questins The prject had tw primary bjectives related t the implementatin and effectiveness f the digital skills training mdel. These gals can be expressed as a series f related research questins fr the evaluatin: Design and Implementatin: the gal was t develp a digital skills training prduct that is flexible and suitable fr lw literacy wrkers in rural small businesses. The evaluatin seeks t address whether the design and delivery f this prduct achieved these aims, specifically, by assessing several key indicatrs f flexibility, suitability, and prgram satisfactin: Did the digital skills training mdel prvide sufficient flexibility fr participants in terms f ptins fr access and custmizatin? Was the digital skills training mdel suitable fr lw-literacy wrkers in a diverse range f ccupatins in rural rganizatins in terms f its usability, cntent-accessibility, and autnmy? Were participants and emplyers satisfied with ther aspects f the training mdel, such as the extent f supprt ffered, r technical aspects f the platfrm? Indicatrs f Effectiveness: the gal was t develp a training mdel that wuld address basic wrkplace digital skills gaps f wrkers in rural small businesses in a way that supprts bth emplyee and rganizatinal perfrmance. The evaluatin seeks t understand whether these gals were achieved by addressing questins related t a number f key training utcmes: Did participants experience imprvements in their digital skills r related utcmes such as cnfidence and use in infrmatin and cmmunicatin technlgies (ICT)? Did the training mdel assist rganizatins in assessing and enhancing digital skills f their emplyees and in turn did they experience any perfrmance gains? Were there differences in utcmes based n the cntext r characteristics f wrkers, their jbs, r their emplyers and wrkplaces? Scial Research and Demnstratin Crpratin 1
Wrkplace Pilting Prcess The pilting f the digital skills training prgram was cmpleted with a ttal f 67 emplyees in nine rural small and medium-sized enterprises. The pilting was an intrinsic part f the prject s research prgram as well as the prduct develpment prcess, as data was cllected thrughut t infrm nt nly the evaluatin but als future revisins t the platfrm. It was cnducted in three primary phases: Baseline Pre-Training Activities: included emplyee and emplyer briefing sessins, administratin f the Canadian Adult Achievement test t measure literacy levels, and the pretraining emplyee and emplyer research surveys. Training Wrkshps: included activities related t managers registratin f emplyees and their assignment t the varius wrkplace digital skills training wrkshps; pre and psttraining digital skills assessment tests fr emplyees; and participatin in the nline training wrkshps themselves (three f five wrkshps were pilted) in ttal, invlving up t a maximum f five hurs f access t the platfrm. Pst-Training Research: included the administratin f pst-training emplyee and emplyer surveys t measure supplementary utcmes beynd thse assessed thrugh the platfrm. Key Findings Indicatrs f Implementatin Success Flexibility: Did the digital skills training mdel prvide sufficient flexibility fr participants in terms f ptins fr access and custmizatin? Indeed, participants valued the flexibility in access t the platfrm at bth different times and in different lcatins. When asked at the end f each wrkshp what they liked abut the training, ver a third (35%) f participants in Wrkshp 2 and all participants (100%) in Wrkshp 3 said they liked the ability t take the training when they wanted. Furthermre, 39% f thse in Wrkshp 2 and 80% in Wrkshp 3 said they liked flexibility in lcatin. In terms f accessibility, a relatively large prprtin f participants far in excess f traditinal wrkplace LES training mdels made use f the training platfrm utside wrk. Abut a quarter f participants (23%) said that they used the platfrm utside their nrmal wrk lcatin and time which is mre than duble the rate expected (10%) frm traditinal wrkplace LES training mdels (Gyarmati et al., 2014). With respect t custmizability, a sizable majrity f participants valued being able t custmize parts f their training experience, such as the ability t retake parts f a wrkshp. 89% f respndents in wrkshp tw and 75% in Wrkshp 3 said that they liked either smewhat r a lt the ability retake part f a wrkshp. The respnses were similar fr retaking a test: 89% and 88% f Wrkshps 2 and 3, respectively, said they liked this feature. Scial Research and Demnstratin Crpratin 2
Suitability: Was the mdel suitable fr lw-literacy wrkers in a diverse range f ccupatins in rural rganizatins, in terms f its usability, cntent-accessibility, and autnmy? Indeed, participants reprted the platfrm as highly usable in terms f its navigability and the level f cmfrt they experienced thus meeting a majr bjective f the prgram. Fur in five participants said that it was easy t navigate the platfrm while a similar prprtin said that they felt cmfrtable during the training. Mst participants als fund the training cntent t be highly accessible in terms f clarity and ease f understanding At least 85% f participants in each wrkshp said that they fund the language used n the platfrm easy t understand. Furthermre, when asked in the fllw-up survey, after all wrkshps were cmplete, 71% f participants felt they understd the cntent f the training well. In terms f autnmy, results indicate that the training platfrm is suitable fr independent learning fr a majrity f participants. Almst tw thirds f participants (64%) smewhat r strngly agreed with the statement, I was cmfrtable with having t d the training n my wn. While nly abut 15% f participants tk the training n their wn mst f the time, almst half (42%) tk it n their wn at least sme f the time. 1 Taken tgether with the fact that participants reprted the platfrm as highly usable, the training cntent as highly accessible and given that a large prprtin made use f the platfrm utside f wrk these results suggest that the training platfrm is, indeed, suitable fr independent learning fr a majrity f participants. At the same time, given there are n well-established benchmarks fr apprpriate levels f autnmy fr nline learning, participants preferences prvide a reasnable guide and suggest that while flexibility in access is certainly imprtant, independence f learning may be less imprtant t participants. Satisfactin: Were participants and emplyers satisfied with ther aspects f the training mdel, such as the extent f supprt r technical aspects f the platfrm? Large prprtins f participants reprted being happy with all aspects f the training, particularly, with the assistance prvided by the Guide and the Instructr. Large prprtins were happy with the help prvided by the Instructr and the Guide, the cmputer hardware, the assessments and survey n the platfrm, and the training verall. In terms f speed f the platfrm, cnclusins are difficult t make. Only abut half f participants said they were satisfied with the speed f the platfrm; hwever, it is unclear whether this relates t cnstraints with the sftware itself, r with the available hardware participants were accessing it frm, and/r issues with having access t high-speed internet in the wrkplaces. One 1 The fact that participants were invlved in a pilt prject may have led t higher levels f interactin with Instructrs than wuld therwise be the case in a wider nn-pilt implementatin f the platfrm. The pilting f the training platfrm was dne in grup sessins facilitated by Instructrs with the aim f nt nly supprting learners, but als t gather infrmatin abut bugs and pssible imprvements. It is therefre understandable that many participants interacted with the Instructr and their clleagues and that nly a minrity cmpleted wrkshps n their wn. Scial Research and Demnstratin Crpratin 3
factr that must be recgnized is that the platfrm being used in the pilting was a Beta versin, and s the final versin wuld be prgrammed t ensure maximum perfrmance in terms f speed. Cncerns were expressed by participants in regards t Internet cnnectins in sme f the testing sites. This wuld suggest the future need t wrk with emplyers and develpers t ensure participants have access t high-speed internet and sufficient hardware. Satisfactin amng emplyers with key cmpnents f the training platfrm was fairly high, with sme prviss. Mst emplyers were satisfied with such elements as managing the rganizatin s training prgram thrugh the platfrm, assigning wrkshps t emplyees based n need, and mnitring emplyee prgress. Hwever, mre than half f the participating rganizatins recmmended imprvements in technical supprt and navigatin instructins. Indeed, in the full-prductin f the training mdel, a manager instructin manual will be prvided, which shuld address these cncerns. Indicatrs f Training Effectiveness Participant Outcmes: Did participants experience imprvements in their digital skills and related utcmes such as cnfidence and use f ICTs r their attitudes twards educatin and training? One f the primary bjectives f the training prgram was, indeed, met t effectively enhance digital skills f participants. Participants in the training prgram realized large DES gains, as measured by the platfrm assessments. A cmparisn f the results f the pre- and pst-training assessments revealed average gains f between 34 and 37 pints (n the 100-pint platfrm scale) acrss the three wrkshps. A substantial prprtin experienced gains f ver 20 percentage pints in each learning step f each wrkshp (as much as 70 pints n ne learning step). Very few participants experienced n gains at all. The prject was successful in enhancing the skills f thse at the lwer end f the literacy distributin, thus meeting an imprtant bjective f the prject t meet the needs f lwerliteracy wrkers. Average skills gains in Wrkshp ne were 47.2 percentage pints amng thse in the lwer literacy grup (Level 2 r less) cmpared t 32.5 percentage pints amng thse in the higher literacy grup (Level mre than 2), a statistically significant difference. The research als demnstrated that the training helped thse with lwer educatinal credentials, with slightly higher average gains than thse with higher educatinal credentials. There were psitive ICT utcmes fr a significant prprtin f training participants, thus fulfilling anther majr bjective f the prject. The prprtin f participants wh reprted being cnfident with ICTs increased significantly ver the curse f the training, frm 59% t 70%. ICTs als appeared t be a large part f mre participants jbs. When asked whether ICTs were a large part f their wrk, the prprtin f participants wh agreed rse frm 39% at baseline t 50% at fllw-up. Imprtantly, cnfidence in using ICTs als increased in cntexts utside f the wrkplace. The prprtin f participants expressing cnfidence in their use f ICTs increased quite dramatically fr several nn-wrk related tasks including searching fr infrmatin at hme and in Scial Research and Demnstratin Crpratin 4
helping with their children s hmewrk: frm 62% and 20% expressing cnfidence at baseline, respectively, t 76% and 33%, respectively, after training was cmplete. Attitudes twards educatin and training were very high befre and after the training. A large prprtin f participants demnstrated psitive attitudes t educatin befre the training began. Fr the mst part, these were maintained at high levels after the training, ften in excess f 90% f participants exhibiting psitive attitudes. A majrity f emplyers als reprted that the digital skills training psitively affected the level f interest and engagement in wrk-related training in general and digital skills training in particular, bth fr themselves and amng their emplyees. Organizatinal Outcmes: Did the training mdel assist rganizatins in assessing and enhancing digital skills f their emplyees and in turn did they experience any perfrmance gains? Emplyers prvided crrbratin f the bserved gains in digital skills and ICTs f participants. The majrity reprted bserving imprvements in their emplyees digital skills and ICT prficiency. Mrever, mst f thse wh reprted these gains als attributed them t the DES training platfrm rather than ther factrs r activities within the rganizatin. A majrity f emplyers als reprted that the digital skills training psitively affected attitudes within their rganizatin twards wrk-related and digital skills training. Owners/managers were asked t judge the effect f the DES training n their wn attitudes and thse f their emplyees tward training. The results indicate that the majrity f rganizatins felt that the prgram had a psitive effect n their wn attitudes tward digital and general training, as well their emplyees interest participatin in general and digital skills training. While sme emplyers reprted psitive imprvements in varius perfrmance measures particularly in regard t human resurce measures, very few wuld attribute these changes t the digital skills training. This is nt entirely unexpected, given that the pst-training surveys were administered t emplyers immediately fllwing the end f the training and therefre wuld have allwed very little time t bserve effects n rganizatinal utcmes. Future prjects shuld allw fr a lnger fllw-up perid in an effrt t assess lnger-term utcmes fr rganizatins. Cntext, Subgrups: Were there differences in utcmes based n the cntext r characteristics f wrkers, their jbs, r their emplyers and wrkplaces? An analysis f subgrup differences cnfirms that thse with lwer literacy r ICT skills, can indeed, benefit frm digital skills training and ften t a greater extent than thse with higher levels. Results indicate that gains in digital skills were abut 15 percentage pints larger fr thse with lw levels f pre-training literacy (levels 1 r 2 cmpared t 3 r abve). Similarly, participants in the lwer end f the ICT skill distributin befre training experienced average gains f 47.3 percentage pints in digital skills cmpared t 32.5 percentage pints amng thse with higher starting ICT levels. Scial Research and Demnstratin Crpratin 5
Outcmes f a digital skills training interventin can als vary in imprtant ways based n the demgraphic characteristics f participants. Fr instance, with respect t the language f the participant, French speakers experienced an average gain f 46 percentage pints in digital skills cmpared t 22 percentage pints fr English speakers in the first wrkshp. Imprtantly, utcmes can als vary based n participants mtivatin fr training, their understanding f its gals, and perceived levels f supprt they have frm their emplyer. On a cmpsite measure f participant mtivatin and emplyer supprt, results indicate that thse with higher mtivatin and perceived supprt befre training, experienced average digital skills gains f 41.9 percentage pints cmpared t thse in a lwer mtivatin and supprt grup f 29.1 percentage pints. This underlines the imprtance f ensuring that emplyees and emplyers have a shared understanding f the gals f training and are slidly mtivated and engaged in the training prcess frm the nset. Scial Research and Demnstratin Crpratin 6