Improving the Quality of After-School Programs

Robert C. Granger
January 16, 2004

The process that led to this commentary was important to the paper?s accuracy and tone.
We began by approaching the authors of the reports it is based upon, and all agreed to assist the
review. These authors?Mark Dynarski, Jean Grossman, Elizabeth Reisner, Karen Walker, and
Richard White?provided invaluable help as they responded to questions, clarified their work,
and reacted to drafts.
We also shared the penultimate draft with a number of influential scholars, advocates,
policymakers, funders, and practitioners in the after-school field. Many of these people provided
thoughtful comments and reactions and told us about related work.
By its nature, the commentary goes beyond the data in the underlying reports, as we draw
implications for policymakers, practitio ners, and evaluators. Thus, the opinions are ours. But
the work of others improved the paper greatly.
This commentary has been submitted for possible publication to Education Week. While under
consideration, this document is available on the William T. Grant Foundation?s website at
www.wtgrantfoundation.org. Any comments or questions should be directed to Robert Granger,
bgranger@wtgrantfdn.org.
Improving the Quality of After-School Programs
1
Over the last half-decade, after-school programs have moved from the periphery to the
center of the nationa l education policy debate. It happened very quickly. Between 1998 and
2002, federal funding for the 21st Century Community Learning Centers program grew from $40
million to $1 billion. State support has also grown. In a 2002 ballot initiative, Californians
voted for a six- fold increase in funding for after-school programs. Now, the heat is on to
demonstrate that after-school programs are effective.
But what do we expect after-school programs to achieve? Unfortunately, there?s no
single answer. Over the last century, they have served several distinct purposes. Some programs
have been designed to supervise youth and prevent them from being the victims or perpetrators
of crime. For others, the main goal has been general youth development, helping youth explore
new talents and interpersonal skills. Many programs serve as a child-care bridge between the
end of school and the end of parents? workdays. More recently, the focus on test-based
accountability in elementary and secondary education has led to an increased emphasis on
academic performance in after-school programs.
After reviewing the results from several recent evaluations, we have four
recommendations for policymakers, evaluators, and those operating after-school programs:
1. Programs must increase attendance or they will not achieve their goals. The most
consistent finding among these studies is that many young people attend sporadically and for a
short period of time. In the typical program, the average participant in elementary and middle
school programs attended between one and two days per week. No program can make a
difference if it does not change the daily experiences of youth, and it cannot do that if attendance
is poor.
From polling data, we know that the general public feels that much of the responsibility is
the parents?. In effect, the argument is that they should ?make the kid go.? We are sympathetic
to the argument, but as parents we are also realistic. These are voluntary programs, and if they
are not engaging, it is ha rd to enforce participation, particularly as young people age and have
more alternatives.
After-school programs can help young people explore and deepen interests, make
decisions, try out different roles, and get involved with people, ideas, and activities. Within each
study, the researchers saw that when such experiences occurred, young people were more likely
to participate. Unfortunately, in too many programs, the activities did not provide the rich, ageappropriate
experiences that lead to sustained attendance and engagement.
In a few of the studies, attendance rates were higher when staff tracked down nonattendees
or when programs required five-day per week participation. The former is the more
promising approach, even though both may produce the appearance of higher participation rates.
Participation mandates (three absences and you?re out) may simply weed out young people who
have fewer resources when even limited participation would have been worthwhile. But when
staff hit the streets or the phones to pull kids into a program, they can boost participation among
the very young people who will benefit most. Participation is also helped when programs are
located in neighborhoods where the young people have fewer alternatives.
Improving the Quality of After-School Programs
2
2. We need to be more realistic about what it takes to create discernible effects on
achievement test scores. In the national samples used to norm the Stanford 9 achievement test,
fifth grade students scored only one-third of a standard deviation higher than fourth-graders on
reading and one- half of a standard deviation higher on math. This reading score difference is
about as large as the difference in moving from 1000 to 1070 points on the combined SAT or
100 to 105 on an IQ test. In other words, everything that happens to a student between the end of
fourth grade and the end of fifth grade?a whole school year of full-day classroom instruction,
interactions with family, conversations with friends, and homework?is associated with an
important but not huge gain on an achievement test. With this as a backdrop, consider the
typical after-school program with youth attending one to two days per week for two to three
hours per day. While it is reasonable to expect that after-school activities can affect performance
as measured by achievement tests, it is likely that such effects will be small. This is particularly
true for reading scores, since they are traditionally less responsive than mathematics scores to
instruction.
Therefore, even if the programs are helping, effects on achievement tests are likely to be
hard to detect statistically. We should balance a focus on test scores with an examination of
intermediate effects?such as more parental involvement in school-related activities, more
diligent homework completion, more school attendance, and better grades?which may pay off
in improved test performance over time. Although the studied programs often did not have
statistically significant effects on achievement test scores, some programs did have promising
effects in these other areas.
3. Programs need to reach vulnerable kids who would otherwise be on their own
after school. It seems reasonable to expect that so-called ?latch-key? children would benefit the
most from after-school programming. Unfortunately, only one of the four recent evaluations
kept track of students? after-school care arrangements. Perhaps surprisingly, the researchers
found that many of the program participants would not have been on their own, but with a
parent or sibling if the programs were not available. Reducing sibling care is perhaps a good
thing, but it is less obvious that time spent in an after-school program would be more worthwhile
than time spent with a parent after school. Future evaluations should focus on whether the
programs have had an impact on the number of children under adult supervision after school.
As noted above, it will be difficult to discern the impact of after-school programs on
academic achievement. It may be even more difficult to identify impacts on hard-to-measure
outcomes such as the risk of committing or being the victim of a crime. But if programs truly are
reaching a large number of youth who would otherwise be on their own after school,
policymakers and the public will be more willing to give the programs the benefit of the doubt.
4. Build on examples that are demonstrable winners. In all the studies, there are
individual programs where kids are safe, engaged, and attending consistently. Some programs
may only appear to be succeeding because they are starting with highly motivated children. But
it is hard to believe that this is the full story.
Improving the Quality of After-School Programs
Within the after-school field, there is reasonable agreement on the key ingredients
required for success: interesting activities, supportive relationships, and the capacity to deliver
such things. As with public schools, almost everyone would agree on how an effective program
looks and feels. There is much less agreement on how to get there. Progress will come when
evaluators learn to ask the right questions by listening to policymakers, advocates, and
practitioners and when people in those roles learn from evaluation results.
Indeed, the most recent wave of evaluations offers a number of valuable lessons for all
the interested parties. Future programmatic reforms should focus on raising participation rates,
particularly among children who would otherwise be on their own after school. In doing so,
practitioners should be assertive and creative in how they recruit and retain students. Similarly,
evaluators and policymakers need to be clear about the nature and magnitude of expected effects
and be sure studies are prepared to measure them.
When federal and local tax dollars are combined with the value of time and resources
donated by volunteers, the country is making an investment in after-school programs that
warrants asking whether they are delivering. The unequivocal summary from these recent
reports is that some are, but the average program needs to get better. Americans need effective
programs to ensure that children are spending their time after school safely and productively.
Our advice is to turn the political and scholarly agenda toward making that happen.
(Note: A more complete review of four recent after-school program evaluations is available on
the William T. Grant Foundation?s website at www.wtgrantfoundation.org.)


#

tags