Using Data to Strengthen Technical Assistance

Transcription

1 Using Data to Strengthen Technical Assistance This document was written in response to a request for information from an RTT-ELC State about using data to strengthen technical assistance (TA) provided to programs in their Tiered Quality Rating and Improvement System (TQRIS). It summarizes the value of TA data, explains how data from multiple systems can be linked to answer essential questions, identifies important data elements to consider, highlights promising practices and lessons learned, and provides key considerations for strengthening data practices. The information in this document is based on interviews with staff in Arizona, Delaware, Georgia, Indiana, Pennsylvania, and South Carolina. These States were chosen based on collaborative discussions with national TA partners. Figure 1 shows the multiple data systems States can link to evaluate and improve technical assistance to early childhood programs. Figure 1. Leveraging data to strengthen technical assistance AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 1

2 Once data are linked, States can use this new information to answer key questions, such as Who provided the technical assistance? Who received it? What happened during the session? Was the TA effective? How long did it last? This information can be used to guide continuous quality improvement efforts. Guidelines for Decision Making About Using Data to Improve TA Practices Technical assistance is one of several quality improvement strategies. States often contract with early learning professionals who have subject matter expertise and skills related to adult learning to provide targeted technical assistance to improve classroom practices and the overall quality of early learning programs. Policy makers who are deciding what data to collect in order to measure TA effectiveness can find guidance from research on professional development initiatives, and coaching and mentoring. Studies on coaching and consultation have typically examined outcomes related to quality improvement, use of a new curriculum, new practices, interactions with children, and children s developmental outcomes (see Figure 2). Figure 2. Effective professional development initiatives and coaching and consultation. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 2

3 Linking Data Across Systems To Answer Management And Policy Questions About Technical Assistance Investments in data systems that collect, integrate, maintain, store, and report information across multiple early childhood programs are worth making and sustaining over time. A technical assistance data system that relies on rigorous data collection can serve multiple purposes. Data collection can be used to: 1. Track service delivery 2. Track the fidelity of implementation of specific TA initiatives 3. Provide support to TA providers 4. Provide data to evaluate the progress of those participating in the intervention or initiative Figure 3. Important types of information to collect about technical assistance States need access to the types of information shown in Figure 3 to make informed decisions. In addition to the data that may be found in TA data systems, relevant data may also exist in other data systems such as those that are used to manage QRIS, 1 workforce registries, training registries, licensing systems, and subsidy systems (see Figure 4). Linking data from different early childhood education (ECE) systems can generate new information to answer essential TA management and policy questions. Instead of building entirely new systems to capture the data needed to answer questions about TA, States can link data found in other ECE systems and focus new data collection efforts only on the essential elements that are not in existing systems. Linking existing data avoids the inefficiencies and risks associated with data entry. It reduces the potential for data entry errors and creating duplicate files. This allows TA specialists to focus more on 1 In the 20 States' RTT-ELC grants, a QRIS is referred to as a Tiered Quality Rating and Improvement System (TQRIS) to highlight the importance of common tiers, or levels, across all types of early learning and development systems statewide. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 3

4 services and less on administrative tasks. Depending on the data being linked, data linking can be an effective way to coordinate program status and information across both quality and compliance functions of the State s early childhood system. Figure 4. Data systems containing data relevant to TA In recent years, significant advancements in information technology have made it possible to link data from multiple data systems to support the types of analysis required to answer key management and policy questions. Linking data across the systems that support TA can help States meet management objectives for example, streamlining business processes and strengthening data quality and expand their ability to answer essential policy questions about TA. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 4

5 While each State will have its own specific questions, Figure 5 provides examples of the types of questions that States are trying to answer. Figure 5. Examples of questions States are trying to answer As States face competing priorities such as meeting multiple reporting requirements, developing research plans, implementing effective TA services, and supporting TA specialists in their work State personnel struggle with the inevitable tension between the value of data collection and the burden of collecting data. They must make intentional decisions about what is essential versus what is nice to know. Collecting unnecessary information is a burden on programs, TA consultants, and information technology staff. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 5

6 Important Data Elements to Consider Based on the research and discussions with States, the following four categories of data elements should be considered for collection: Demographic data about the early learning practitioners and programs receiving TA services Demographic and qualification data about the TA providers who deliver services to programs and practitioners Data about each program s quality improvement plan and the TA activities associated with it Data on the methods and strategies used by TA providers to deliver services to programs and practitioners Figure 6 offers examples of data elements to consider within each of these four categories. Figure 6. Examples of data elements to consider AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 6

7 Leveraging TA data in practicefigure 7 provides examples of how the raw data elements might be combined to provide answers to key management and policy questions. 3 Figure 7. Leveraging TA data in practice AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 7

8 Promising State Practices for TA Data Collection and Management Based on current research and shared experiences from other States, States are adopting and adapting TA data collection and management practices. In addition, they are engaging in TA data collection and management practices that can be used to inform future TA management and policy decisions. Following are examples of some promising State practices and lessons learned. The information presented here is based on interviews with staff in Arizona, Delaware, Georgia, Indiana, Pennsylvania, and South Carolina. These States were chosen to be highlighted in this brief based on collaborative discussions with national TA partners. States Are Linking Their TA Data to Data in Other ECE Systems. Delaware links TQRIS TA data with data from its Professional Development Registry and State licensing data system. This avoids duplication of data entry for some program and practitioner data. Georgia has developed a data system for Quality Rated, the State s TQRIS,. The State has defined business and data element requirements and also uses web services that link several of Georgia s early learning programs and initiatives, including licensing, professional development, and Quality Rated components. The Indiana Child Care Information System (CCIS) is an early childhood data system that shares data between licensing, subsidy, and Paths to Quality, the State s QRIS. A TA data module allows TA specialists, licensing inspectors, and QRIS raters to communicate and coordinate more effectively regarding program status across both compliance and quality functions. Pennsylvania links early learning data across multiple systems using the Pennsylvania Enterprise to Link Information Across Networks (PELICAN) data system. The PELICAN system includes data related to child care certification, child care subsidy, the State s preschool program, Early Intervention, and the State s TQRIS. The PELICAN Keys to Quality data system that is used to support the TQRIS includes a module for TA that allows TA specialists to collect and report TA data on requests for assistance, contacts, quality improvement plans, TA goals, and other information related to TA. States Are Using TA Data to Answer Essential Management and Policy Questions. Arizona plans to use data to analyze the complex interactions that result when multiple TA specialists are working with a program. The State will look at the impact that dynamic TA may have on effectively and efficiently moving a program up in QRIS levels in a reasonable amount of time. Delaware uses a combination of data to organize and manage TA specialists workflow. TA specialists document the type and frequency of TA provided to programs in the TQRIS database. TA activities are also linked to specific standards in the TQRIS. In addition, Delaware uses its data to engage TA consultants in an ongoing feedback loop to provide opportunities to refine the system as lessons are learned and needs change. Georgia plans to use data to examine how the content of TA plans may be linked to changes in Environment Rating Scale (ERS) scores and quality ratings. The State is also engaged in a study to look AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 8

9 at TA data to gain a better understanding of how infant/toddler teachers and administrators experience TA interventions. Administrative data is also used throughout Georgia s robust TQRIS validation project. Indiana developed a TA case management module that supports TA specialist workflow and caseload management. The State is in the process of conducting a research study using both quantitative and qualitative data to determine whether there are specific program and TA factors associated with advancement or no advancement in QRIS levels. Pennsylvania used TA data to conduct research into the effectiveness of the TA being given to programs participating in its TQRIS. It found that the odds of advancing a TQRIS level for providers receiving TA are 2.2 times greater than the odds for programs not receiving TA. The research suggests that programs at higher TQRIS levels retained knowledge and skills longer than programs at lower TQRIS levels. 4 Pennsylvania used this research to create a TA Accountability Plan, part of which sets a contractual performance expectation. South Carolina is planning to expand on the work done in Pennsylvania by examining the correlation between how well a program retains the knowledge and skills learned during the TA process for a program that is voluntarily participating in TA, versus a program that is required to participate in TA due to regulatory or compliance issues. States Are Using Mobile Technology to Produce Efficiencies and Strengthen Data Quality. TA consultants in Arizona, Delaware, Georgia, Indiana, and Pennsylvania all use tablets or ipads to record TA data. This allows real-time data uploads. Offline data entry options allow data to be captured and stored if Internet connection is not available. States also found that access to printer hardware allows an easy way to generate and leave information on the spot. States found that policy makers and managers must recognize that some TA specialists resist the use of new technology, and they must develop a proactive plan to address this. Initial training should direct TA specialists about how to enter data and also instruct them about the use and capabilities of the hardware that is involved. Finally, States should maintain user-friendly tech support services such as a live help desk and an online chat feature to assist staff who use mobile technology to perform their work. State Lessons Learned About Data Collection and Management The following are summaries of some of the most important lessons States learned as they tackled data collection issues. If You Can, Start Small, and Scale Up Only When the System Is Stable Initially, some States grappled with the impulse to collect every data element they could imagine they might ever need. States unanimously agreed that it is important to focus on collecting only the data elements that are most relevant to the management and policy questions they are trying to answer. By starting with the end in mind, States can be more intentional in and prepared for meeting their data collection and analysis goals. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 9

10 Arizona took the intentional approach to start small, which allowed flexibility as the system evolved and increased data entry accuracy and the State's ability to keep a high level of data quality over time. The State initially focused on collecting and reporting data related to program QRIS levels. Now, Arizona is turning its attention to adding data elements that will help State policy makers gain a deeper understanding about how program quality progresses during participation in the system. Delaware staff decided to delay designing management reports until a later phase when they felt they had accurate data and a better sense of what they wanted to know. Georgia staff focused on defining and collecting only the data that are key levers to change. They also emphasized the importance of defining each data element. In South Carolina, following a six-month period of data entry, the State team reviewed the data entry trends and found that many of the fields requested by TA specialists were never used. The State team reevaluated the system and made decisions about which fields to retain, add, and delete by asking, What data do we need to better understand the impact that TA has on program quality? Engage Researchers and TA Specialists Early and Often States found that engaging researchers, TA specialists, and other key stakeholders early in the process of defining and identifying important data elements for collection was an invaluable and effective strategy. Doing so ensured that the State was better prepared for validation and evaluation work. In addition, engaging stakeholders builds buy-in to data collection practices and ensures that the system development and linkages plan are designed to meet the needs of the stakeholders, not the interests of the developers. States also emphasized the importance of continuing to engage stakeholders on an ongoing basis. Engaging stakeholders on a routine basis provides information from the field that can be used to refine and enhance the data collection and management process as lessons are learned and needs change over time. The ongoing engagement of stakeholders includes the sharing of management reports and the review of any data quality concerns. Continuous data quality improvement protocols should be developed, updated, and communicated on a routine basis. Delaware engaged TA specialists during its initial data system development and on a routine basis afterward. Delaware found that sharing data back with their TA consultants helped them better understand the value of the data they enter, as well as the pitfalls of haphazard data entry practices. Indiana engaged its research team to identify and define the data elements that would be needed to support future QRIS and TA evaluation efforts during the development of its TA case management module. The research team participated in data system planning sessions to define the data elements they would need to answer the research questions outlined in their QRIS validation and evaluation scope of work. South Carolina gathered TA specialists and researchers to help define the data elements that each group would need to support their work. The State formally engaged the TA specialists again at the six-month milestone to reflect on adjustments that needed to be made to the data fields being collected. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 10

11 Using Qualitative Data Has Pros and Cons All States mentioned the collection and use of qualitative data. They pointed out that qualitative data can be useful in case-specific issues, but can be a challenge for broader quantitative analysis. Arizona experienced difficulties in analyzing the impact of TA strategies because open text fields were used by TA consultants to record their activities. The State discovered inconsistencies in how TA consultants were describing and defining TA activities. In addition, it found that sifting through narrative information to try to identify key variables of TA was very time consuming. As a result, Arizona plans to define and add drop-down lists of TA strategies, from which each TA consultant may choose. Similarly, both Delaware and Pennsylvania staff mentioned that while qualitative data collection may be necessary in some cases, States should make sure that users understand and are trained in how to enter this kind of information. For example, Pennsylvania trains TA specialists on how to develop and enter quality improvement plan goals for each program. This training focuses on how to write goals that emphasize child or family outcomes. Indiana staff mentioned that while qualitative data may have drawbacks, it can be particularly helpful in situations where a program is challenging a rating or making a complaint about TA services. Having ongoing notes can help support a State s response to these kinds of issues. The High-Stakes TQRIS Environment Affects State Strategies Along with the proliferation of QRIS across the country comes increased accountability and scrutiny. A few States mentioned their struggles with the QRIS success criteria and the reporting requirements of multiple funding sources. Georgia indicated that it strives to balance the collection of information required to meet funding and reporting requirements for Federal agencies with the collection of information that will help it understand what truly improves program quality. Pennsylvania mentioned another unintended consequence of the high-stakes TQRIS environment: although continuous quality program improvement has been a focus for many years in Pennsylvania, there is a new push for programs to improve their quality even faster. Staff are wondering about the potential impact that applying pressure to improve faster will have on programs especially programs already struggling to maintain or increase their quality. Summary and Key Considerations As States continue to adopt and refine QRIS as a framework to build and improve early care and education systems and as a strategy to improve ECE program quality, they have an increased, highstakes interest in investing in and implementing only the most effective TA supports. AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 11

12 Work With Information Technology Staff and Information Technology Vendors to Link Data Elements That May Exist in Current Data Systems The following national resources can assist States in the effort to link data systems. QRIS Resource Guide: Data Collection and Evaluation 5 from the National Center on Child Care Quality Improvement provides States with key considerations for QRIS data collection and evaluation. It includes information relevant to the Initial Design Process and the Approaches to Implementation of a QRIS. The INQUIRE Data Toolkit 6 was developed in 2013 by the Quality Initiatives Research and Evaluation Consortium (INQUIRE) to provide tools to support effective data collection and the use of data to answer important policy and reporting questions through the use of common data elements. The Toolkit contains two components: a Linkages Guide and a Dictionary of Common Data Elements. The Toolkit provides step-by-step guidance about identifying key policy questions, identifying the data elements required, and performing analysis on the data. Answering Key Questions with an Early Childhood Data System 7 from the Institute of Education Sciences (IES) was designed to support States working to create or evaluate their current early childhood policy questions. The brief examines examples of policy questions from States and outlines why those questions are needed, how to create them, and who should be involved in the process. The brief was a product of a series of conversations sponsored by the Statewide Longitudinal Data Systems (SLDS) program. Common Education Data Standards (CEDS) 8 is the result of a national collaborative effort to develop voluntary, common data standards for a key set of education data elements to streamline the exchange, comparison, and understanding of data within and across P-20W (preschool through workforce) institutions and sectors. It contains an exhaustive dictionary of early childhood data elements and tools to assist the user in mapping the elements needed to answer specific policy questions. Which ECIDS System Model is Best for Our State ECIDS? 9 from IES provides information on three different approaches that States can take to develop an Early Childhood Integrated Data System (ECIDS) that collects, integrates, maintains, stores, and reports information from early childhood programs across multiple agencies. Engage Researchers and TA Specialists Early in the TA Data Collection Planning Process Stakeholders can help State policy makers determine the data most needed to support the work they have been tasked to perform. It is helpful to put guiding principles in place when having these discussions. For instance, when making data collection decisions, the group can agree to focus on collecting only those data elements that are most closely related with the ability to report and evaluate items that are relevant to key goals and outcomes. State staff can inform stakeholders whether and how the data may be used to measure contract performance and scope of work expectations. Ensure That Ongoing Two-Way Communication Practices and Data Feedback Loops Are in Place State policy makers and stakeholders can use these practices to further refine and enhance TA data collection, report, and analysis. For example, sharing TA management reports with TA specialists will AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 12

13 help them better understand the impact of their practices and give them the opportunity to share insights with policy makers and managers about any unexpected reporting results or data trends. Share TA Research Results and Management Practices with Others States are doing a tremendous amount of great work. Consider partnering with your State-level peers to share strategies, practices, and lessons learned. ELC TA and other national TA providers can support States in connecting and learning from one another. Finally, sharing your insights with Federal partners can help inform the national agenda, so give thought to the best way to share information that could support the national ECE landscape. ENDNOTES 1 Zaslow, Tout, Halle, Whittaker, Lavelle, and Child Trends (2010). 2 Aikens and Akers (2011). 3 Friese, King, and Tout (2013). The examples in Figure 7 were partially adapted from the INQUIRE Data Toolkit that was developed in 2013 by the Quality Initiatives Research and Evaluation Consortium (INQUIRE) to provide tools to support effective data collection and the use of data to answer important policy and reporting questions through the use of common data elements. 4 Pennsylvania Office of Child Development and Early Learning (2011). 5 National Center on Child Care Quality Improvement (2014). 6 Friese, King, and Tout (2013). 7 Institute of Education Sciences (2013). 8 Common Education Data Standards (2014). 9 Duarte, Sellers, and Cochenour (2014). REFERENCES Aikens, Nikki, and Lauren Akers. Background Review of Existing Literature on Coaching: Final Report. First 5 LA, Boller, Kimberly, Kate Tarrant, and Diana D. Schaack. Early Care and Education Quality Improvement: A Typology of Intervention Approaches: OPRE Research Report # Office of Planning, Research and Evaluation; Administration for Children and Families; U.S. Department of Health and Human Services, Common Education Data Standards. Accessed November 10, Duarte, Steve, Jeff Sellers, and Missy Cochenour. Which ECIDS System Model is Best for Our State ECIDS? Institute of Education Sciences, National Center for Education Statistics, AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 13

14 Friese, Sarah, Carlise King, and Kathryn Tout. INQUIRE Data Toolkit: OPRE Report # Office of Planning, Research and Evaluation; Administration for Children and Families; U.S. Department of Health and Human Services, Institute of Education Sciences. SLDS Issue Brief: Answering Key Questions with an Early Childhood Data System. National Center for Education Statistics, data_system.pdf National Association for the Education of Young Children and National Association of Child Care Resource & Referral Agencies. Early Childhood Education Professional Development: Training and Technical Assistance Glossary National Center on Child Care Quality Improvement. QRIS Resource Guide. Office of Child Care, Administration for Children and Families, U.S. Department of Health and Human Services, Pennsylvania Office of Child Development and Early Learning. Research Brief, Issue 4: Keystone Stars Technical Assistance Zaslow, Martha, Kathryn Tout, Tamara Halle, Jessica Vick Whittaker, Bridget Lavelle, and Child Trends. Toward the Identification of Features of Effective Professional Development for Early Childhood Educators: Literature Review. Office of Planning, Evaluation and Policy Development Policy and Program Studies Service; U.S. Department of Education, This brief was compiled in collaboration with State and national partners. ELC TA sincerely thanks the following State staff who generously contributed their time to help us understand promising State practices for data collection and management for decision making about policies and management related to technical assistance: Arizona Ginger Sandweg and Brooke Travis; Delaware Rena Hallam and Evelyn Keating; Georgia Laura Johns; Indiana Melanie Brizzi, Janet Deahl, and Meghan Smith; Pennsylvania Katrina Coburn; and South Carolina Herman Knopf. In addition, ELC TA thanks Child Trends staff members Kelly Maxwell and Kathryn Tout for their significant contribution to the content of this brief. Finally, ELC TA thanks the following national partners for their content input and review: National Center on Child Care Quality Improvement, National Center on Child Care Professional Development Systems and Workforce Initiatives, and INQUIRE (supported by the Office of Planning, Research and Evaluation in the Administration for Children and Families, U.S. Department of Health and Human Services). This resource was developed as part of the Early Learning Challenge Technical Assistance (ELC TA) Program through a contract from the U.S. Department of Education, run in partnership with the U.S. Department of Health and Human Services Administration for Children and Families. The findings, conclusions and opinions expressed in this document are those of the authors and do not necessarily reflect the official position or policies of the U.S. Departments of Education and Health and Human Services. ELC TA is administered by AEM Corporation, in partnership with ICF International. For more information, visit AUGUST 2015 EARLY LEARNING CHALLENGE TECHNICAL ASSISTANCE 14