The study, Continuous Quality Improvement in Afterschool Settings: Impact Findings from the Youth Program Quality Intervention, found that a cycle of assessing staff practices, planning based on the assessment and targeted training improves the quality of services delivered to young people.
It says the Youth Program Quality Intervention (YPQI) model increases program quality among a wide range of afterschool systems, is sustainable and cost-effective, and might boost staff retention.
The findings come at a time when the afterschool field has made quality improvement a top priority. While growing evidence shows that afterschool programs can boost academic achievement, improve social and civic skills, and reduce risky behavior, many programs don’t realize that potential – and “a primary reason for this may be the quality of experiences available to youth in these settings,” the report says.
The study validates a specific type of professional development: a continuous process in which managers support staff in their efforts to improve.
“This is a tangible example of a positive connection between research and practice and represents an important step forward for the field,” said Bob Granger, president of the William T. Grant Foundation, which funded the study conducted by the David P. Weikart Center for Youth Program Quality. “After-school staff come and go quickly, are part-time, and programs have limited infrastructure to support improvement. Despite those challenges, the YPQI produced substantial improvements in line-staff practice.”
The YPQI process, said a staffer at one of the OST sites in the study, has “taken the program to a whole new level.”
A Strategy to Improve Quality
The YPQI seeks to improve quality at the point of service – the place where youth and staff come together. The intervention leads with performance assessment, then engages staff in a multi-month cycle of planning and improvement based on the assessment.
Under the YPQI, a team of instructors led by a site manager engage in a cyclical process of assessing, planning and improving. “The cycle is critical,” said Charles Smith, executive director of the Weikart Center, which created the model. “You’ve got to enact the whole cycle to get the largest effects.”
The process uses observation to measure best practices such as safe environment, supportive environment, peer and youth-adult interactions, and youth engagement. The YPQI is carried out by program staff, often with support from local trainers and assessors who have been trained by the Weikart Center, a division of the nonprofit Forum for Youth Investment.
The YPQI is being implemented by more than 70 afterschool networks in 27 states.
Measuring the Model’s Impact
The study examined 87 after-school sites in five provider networks, over four states, between 2006 and 2008. Sites included a mix of rural and urban settings and a diverse set of program types and funding streams. It used random assignment to create an experimental group that went through the YPQI and a control group that did not.
Researchers collected data from managers, staff and youth in all sites prior to the randomization, at the end of the implementation year of the study (spring 2007) and again at the end of the follow-up year (spring 2008).
It is “the first experimental investigation of a data-driven, continuous improvement intervention in the afterschool field,” says the report.
Strong Impacts Last
The study draws the following conclusions:
1. The YPQI improves program quality, and high implementation of continuous improvement practices leads to higher quality.
At afterschool sites assigned to the YPQI, the quality of instruction improved both overall and in several areas that site teams targeted for improvement. For example: A low baseline score for one practice measured by the process – “youth have opportunities to set goals and make plans” – signals an area that needs improvement. Staffers go through a targeted training in how to engage youth in planning practices then implement those strategies with support from their manager. Over time, staff members see youth completing activities with more intentionality, and an external assessment reveals an increase in scores on this scale.
In a field characterized by high staff turnover, Smith noted, “just having these standards for practice in place in a program will advance the learning curve for new staff quickly, increasing returns on investments in new staff.”
2. The YPQI might increase staff tenure and works across challenging staffing conditions.
The length of staff employment at participating sites increased, “suggesting that the YPQI continuous improvement practices may reduce staff turnover over time.” Also, quality improvements were not significantly affected by manager turnover, staff education or youth-adult ratios.
3. The YPQI works across different types of afterschool systems and policies.
Sites implementing the model included a fee-based school-age afterschool system in a large urban district, a state network of 21st Century Community Learning Centers (both middle school and high school), a coalition of nonprofit youth programs and an afterschool system funded by a state human service agency. The study found “almost no significant differences in effects between these networks.”
4. The YPQI appears to be a sustainable, cost-effective, lower stakes model for continuous quality improvement.
Researchers returned to intervention sites one year after the post-intervention data were collected, and “found that improvements were maintained in the areas we were able to measure: the manager’s improvement focus, staff participation in the continuous improvement cycle, and staff employment tenure. In other words, the improvements set in motion by the YPQI were still present a year after the intervention ended.”
The staff time required to implement the YPQI was deemed reasonable in comparison to similar interventions in other fields. Costs to implement the YPQI are described in the report.
Low Stakes Drive Results
The performance data for individual programs were not made public, and funding levels were not tied to scores. Quality improved as managers and frontline staff embraced the continuous improvement process and used assessment and targeted supports to change practices.
“The ‘lower stakes’ aspect of the YPQI is critical,” said Nicole Yohalem, special projects director at the Forum for Youth Investment. “Keeping the focus on improvement rather than punishment helps participants feel supported and committed to the change process.”
The report, an executive summary and a policy brief are available at www.cypq.org/ypqi.