Quality + Time = Quantity?

Karen Pittman
October 1, 2001

Gone are the days when anyone believed that all it takes to get a pilot youth program to scale is a favorable evaluation. Going, it seems, are the days when anyone believes that all it takes to keep a program afloat is luck, a good accounting system and some compelling anecdotes. Outcomes-based accountability has brought discipline to some programs but fear to many. Good evaluations do not ensure automatic growth. Bad or even mediocre evaluations, however, may lead to funding cuts.

But things are changing. The San Francisco-based Community Networks for Youth Development (CNYD) has been promoting quality and peddling assessment for some time. It pulled out the stops a few years ago, enlisting the support of Michelle Gambone and Jim Connell, two results-oriented researchers who decided it was time to help community youth programs “show” what they can and cannot deliver. CNYD staff recruited Bay Area programs and funders into a continuous improvement project. Preliminary results are in. The news is powerful, and I am more excited than I have been in years. Four reasons:

First, to get the quantity of programming needed, we will have to hold youth programs more accountable. The question is, for what? Gambone and Connell, through their Community Action for Youth Project, join a growing group of researchers, evaluators and practitioner advocates who suggest that the most important thing programs can do is meet program standards that are linked via research to desired youth outcomes.

Echoing much of the program literature, they suggest and provide easy survey questions for five key supports and opportunities: safety, relationships, youth participation, skill-building and community involvement. Putting these to the test in San Francisco documented what we all know – programs’ games aren’t always as good as their talk. When young people were asked whether they consistently got what they needed in these areas, the reports were disappointing, even on relationships – the raison d’etre of youth work.

Second, youth programs and intermediaries have to demonstrate their willingness and capacity to improve. The CNYD programs absorbed the results, set benchmarks and made improvements. When youth were resurveyed, their responses reflected
the change. The before-and-after data are impressive. In some cases, programs met or exceeded their benchmarks in every area.

Third, youth programs have to show that the supports and opportunities they offer matter. Youth workers by and large – especially those working outside of the youth care/youth counseling fields – have not been quick to embrace standards. They are almost as leery of impact assessments. The outcomes that youth workers are asked to have an impact on (e.g., pregnancy, school achievement) have complex determinants that are often beyond youth workers’ abilities to influence consistently. The outcomes that youth workers feel they can best influence (e.g., confidence, decision-making) are often neither easily measured nor highly valued. Ditto for the program inputs (such as opportunities for involvement).

Connell and Gambone are completing a powerful meta-analysis of longitudinal data sets that demonstrates that the relationships, safe places, challenging experiences and opportunities to participate do matter. Their argument (and mine): Hold programs accountable for providing high quality inputs, hold researchers accountable for proving that these inputs make a difference.

Fourth, we will have to show that these supports and opportunities are not just applicable to youth programs and nonacademic youth outcomes. We will have to show that they are the black box ingredients behind improved school success.

They are. Connell, wearing his other hat as director of the Institute for Research and Reform in Education, has helped schools move the dial on academic outcomes by increasing their capacity to provide two of these supports – relationships (multiple supportive relationships with adults and peers), and challenging and engaging activities and learning experiences.

When they give their pitch, Connell and Gambone talk about the three M’s. We have to show that the supports and opportunities that youth need, and that community programs and schools offer, (1) matter, (2) are measurable and (3) are movable. We’re on our way.

Karen Pittman is executive director of the Forum for Youth Investment. Contact: karen@iyfus.org.

Pittman, Karen. "Quality + Time = Quantity?" Youth Today, October 2001, p. 55.

©2000 Youth Today. Reprinted with permission from Youth Today. All rights reserved.