Lessons for Youth Program Quality Improvement Initiatives

The YMCA of Greater Seattle—one of the largest youth-serving agencies in Western Washington—is measuring its work a little differently.

No stranger to measuring program impact, this is the first time the Y launched a comprehensive, organizationwide evaluation of quality at all points where staff and youth interact.

The distinction is an important one: A strong body of research supports the link between quality youth programs like those offered through the Y and long-term positive outcomes for youth, including reducing risk behaviors and improving academic performance and interpersonal skills.

Lessons for Youth Program Quality Improvment Initiatives, released by the Raikes Foundation, is a case study of YMCA of Greater Seattle’s experience with the Youth Program Quality Intervention (YPQI)—an evidence-based continuous improvement model to strengthen the skills of staff and the quality of services in after-school and youth development programs.  The model was developed by the David P. Weikart Center for Youth Program Quality, a division of the Forum for Youth Investment.

The report lays out the goals, successes, and challenges of the process. It is easy to read and engaging, and follows the format of the YPQI’s five-step process. Through each step, we learn how the leaders, program directors, and program deliverers defined and maintained their goals while following the YPQI process. 

A Candid Look at Challenges and Solutions

“This tool is exactly what we wanted to happen. We have external measures that parents complete, and we track program participation and frequency. This is the third prong we needed to provide an intermediate indicator of where our outcomes might be.” – YMCA Executive

The study offers real-life examples of obstacles to implementing YPQI, and pairs these with concrete recommendations from YPQI professionals about how the Y and other organizations looking to use YPQI can address these and similar challenges.

Here’s a quick look at the process and some of these lessons.

Step One: Prepare

The first step in the YPQI is to identify and train staff ("initiative leads") who will make decisions about how the process will work within the specific organization and community context. In this case, two mid-level executives shared the leadership role for this process. They defined the timeline and the budget (including funds for training, coordinating and staff involvement) and planned communication strategies to share the expectations and timeline in a way that fostered participant buy-in, cooperation, and fidelity to the YPQI.

They also set the scope for this work: all teen and young adult programs located within schools, Y branches, and camps, and one program that serves foster youth ages 17 to 24 (because of its similarity with other teen programs).

In the words of one of the initiative leads: “There’s a difference between managing to those who get it and those who don’t, it’s two different processes.”

Among the lessons learned:

• Timeline: The scope and pace of the YPQI rollout sets the stage for every step that follows. The larger the number and greater the diversity of programs involved in a YPQI rollout, the harder it is to establish schedules and expectations that work for everyone. In contrast, if too small a portion of the organization is involved, many cycles of rollout are required to shift organizational culture and see positive changes.

• Communications: The Y had “internal champions [of the YPQI] in the middle of the organization,” who were key to getting staff excited about the initiative early on, and to clarifying the improvements it could offer. This helped keep momentum moving when other priorities and challenges threatened to get in the way.

That said, developing an effective communications plan isn’t as easy as it may seem. In this case, the plan wasn’t fully developed and didn’t adequately take into account the frequency and level of communication necessary to maintain staff participation and buy-in. 

Step Two: Assess

This stage is two-prong: self-assessments and external assessments. Staff received training on how to score the YPQA self-assessment and then schedule observations to assess their own programs. At the Y, 13 peer assessors conducted the external observations.

Among the lessons learned in this phase:

• Implementers at the YMCA learned that assessments from "external" assessors negatively affected the morale of many program directors and deliverers. This was one of the challenges that could not be predicted or laid out in the initial steps of the process.

To address this, the study suggests introducing the concept of peer assessment carefully. In this case, it may have been more effective if these assessments were "formally reframed as ‘peer assessments’ with time consistently allocated for debriefing to minimize miscommunication and unaddressed resentment."

Step Three: Plan

Staff review the self-assessment and external YPQA scores and create an improvement plan based on the data.

Among the lessons learned in this phase:

• Maintaining momentum. In many instances, staff reported losing interest at the point of reviewing their data. Only 14 of 20 programs completed action plans after reviewing their scores, despite deadlines and repeated requests.

The fix? Staff said it would have been helpful to have a small, trusted group of peers who understand their program help them review the data and complete an action plan. Also, it would have been good to have more attention from staff leads to keep this phase moving forward.

Step Four: Improve

YPQA scores showed that the Y saw program quality improvements as a result of this initiative.

Program staff carry out action plans to improve point-of-service quality and ensure the right professional developments. The good news: The YPQI scores showed that the Y did indeed see program quality improvements as a result of this initiative.

Among the lessons learned in this phase:

• When new tasks emerge in the YPQI process, it’s useful to know what can be adapted, adjusted or even delayed if necessary and what is critical in order to keep the process moving forward smoothly.

“There’s a balancing act for each site around what’s efficient and doable. That’s one of the biggest tricks—when do you allow for flexibility with each site and when does a mandate work better?” one Program Director told the authors of the study.

Step Five: Repeat

The steps are repeated to initiate a cycle of continuous program improvement.

Among the lessons learned in this phase:

• In the second year, the Y planned to expand YPQI to most of its youth development programs. While the study doesn't cover whether “Repeat” will continue into the future, there was no shortage of optimism (and high-level buy-in) that this effort makes a difference and was worth continuing.
  

  The YPQI has paid off for the long term, says Jessica Werner, who was associate executive at the Metrocenter YMCA and how heads Youth Development Executives of King County. “The entire continuous improvement process that the PQA provides has changed the culture, empowered staff to improve programs and provided common language across previously siloed departments and programs at the YMCA of Greater Seattle,” she said.

These are just a few of the take-aways. Click the link below to read the full report. It includes a very cool chart comparing the planned implementation steps with what actually happened at each stage.