Modeling the Magic: Deconstructing Social-Emotional Practice

November 13, 2015

On November 12, Karen sat down with Charles Smith, Executive Director, and Gina McGovern, Design and Innovation Specialist of the Weikart Center for Youth Program Quality.  The Weikart Center is housed at the Forum for Youth Investment.

Karen: The Social and Emotional learning project that the Susan Crowne Exchange is leading “is one of the most exciting efforts that’s underway current to make the invisible visible, to understand the how the abilities and practices really play out in the day to day lives of young people and how we can be intentional about improving them.”

The evaluation, which will be released in February 2016, examines how eight out-of-school programs in seven states support young people in developing key social and emotional skills. It looks at what the intentional staff practices are, and what the impact is on young people.

Background on the Project

Charles: We call it the Social and Emotional Learning Challenge, or the “SEL Challenge.” The Susan Crowne Foundation wanted to do something innovative related to SEL and its application in learning communities and in educational settings, with a particular focus on programs served vulnerable youth.”

They were interested in understanding the kinds of services and experiences that help build resiliency in groups of vulnerable youth. We were very interested in getting into that conversation because the issue of innovation, what the staff do with kids, how to help children build

Many kinds of settings where adolescents spend time when outside of school. 

A caveat: We don’t think that we have discovered anything remarkably new here. It’s all about how we do the talking, the words we choose and the voices that we use to describe what good youth practice is.

We reached out to Reed Larson at University of Illinois … and then we found eight really great partners, expert orgs who know their work and have been refining their curriculum model over many years. We got that group together and focused our energies on what we think are the best, most promising practices.

Karen: Give us a flavor of the organizations you selected for this evaluation.

Gina: We chose eight programs across seven states for the SEL Challenge.

Philadelphia, where teens serve as apprentice boat builders. They start from a draft drawing, learning carpentry skills and sailing techniques, and building a boat from scratch and sailing it.

The Possibility Project in New York City has a group of 45 young people who share life stories and then work together to craft together write and preform for the community and their families based on their lives and issues that are important to them.

Attitude, Harmony and Achievement (AHA) in Santa Barba, a group of young women working on relationship wisdom and learning to talk about the types of relationships they want and are meaningful and important to them and work through the relationships they have in their lives to build really strong and meaningful ones.

Across the programs, as different as they are, we have been able to describe the things that they have in common, the specific practices that they use to build skills with kids.

KP: We hear a lot about research-practice partnership. But you are on to something different with this partnership. Is that true? a unique approach, this partnership with this group of direct service organizations, isn’t it?

CS: Yes, that’s a central part of the recipe. For those who know Weikart’s work, we consider the first phase of Weikart’s work on quality. Phase 2 is moving from the context to the person side. In this way, Social and Emotional Learning becomes a way to work on what’s happening with the person, in addition to what’s happening with the setting.

it’s very difficult to find a rigorous evaluation base focused on specific adult practices and the relationship to the development of specific skills in.

Our work on quality came out of a research-practice partnership a lot like the one we’re mounting with this Challenge. The Youth Program Quality Assessment, our key quality metric, was developed through conversations with expert practitioners. That’s because if we want to turn to the literature, it’s very difficult to find a rigorous evaluation base focused on specific adult practices and the relationship to the development of specific skills in children; They tend to be quantitative.

So when we decided we wanted to build stanards for social and emotional learning—to talk about what adults do to build skills in kids—we needed to turn to expert practitioners.

We contact Reed Larson at the University of Illinois who has published more than 28 peer-reviewed articles aligned with our sox domains of social-emotional learning. He has been working on this and interviewing youth for years. He had a powerful qualitative base for us to use as a starting framework.

Then, we used a selection process where we looked for programs that could talk about this work, both their curriculum and the skills kids build in relationship to the curriculum. They could talk about real-time assessments how they help kids test their own ability to regulate, how handled the tough spots—in their own applied language.

We found people who could really talk about what they are trying to do and why. We applied a qualitative method—iterative and thematic, with feedback loops in the community—and we developed standards to describe what good work is.

We brought rigorous methods to bear on the very rich content that the experts provided.

KP: What are the skills that really seem to be relatively universal across this diverse set of programs and also observable. What are those skills you found?

Gina: Our method was to dig into six domains of social and emotional learning:    

  • Emotion management
  • Empathy
  • Teamwork
  • Responsibility
  • Initiative
  • Problem-solving

Across each we examined, what are the things that these expert practitioners are doing in their programs that lead youth to build skills in these areas?

While we were asking about that, a lot of things that have also been in the literature for decades that are still reinforced to be very important, which we call “curriculum teachers.”Curriculuum is set of sequences and supports that are part of the program that surround what happens in the day to day. There are four categories of these features:

  1. Content sequence. The programs each had a project that was based on a real work experience—building a boat, crafting a performance—that created opportunities along the way as tasks were getting harder and skills were getting repeated, for young people to have instances and opportunities for social and emotional learning. The project content makes room deliberately for the SEL content to emerge
  2. Safe space. This includes all the things we know are good for strong relationships between staff and youth, that creates a culture that reinforces that people are valued, where learning from mistakes is important, and there is a routine and consistency that young people can begin to expect
  3. Responsive practices. This where we start to describe and categorize the types of things staff were doing—such as coaching, modeling, scaffolding and facilitating. These are the things staff do in real time. They build in regular check ins that are often on top of the work doing with the youth that facilitate who the youth are, where strengths are where need support
  4. Strong support for the staff – in order for them to implement SEL with youth, the staff need to be supported. It starts with recruiting right youth to program, and having enough staff in program to address issues and move forward. It includes time to plan and debrief after and before each sessions, support for professional development through learning communities, sometimes counseling and interaction with peers. Another deep support for staff is to have a continuous improvement process where they’re reflecting on work, continuing to build and improve the experience that the youth are having and continuing to innovate each program year based, on the previous year’s experience.

How do We Know When Youth Master SEL?

Karen: A question from Twitter is, How do any of us know when young people have actually internalized these skills enough to be useful for them?  How do we measure that—and what’s enough?

Charles: Karen, you like the tough questions. In the SEL Challenge, we took the advice of these expert practitioners. We built a learning community and created these descriptions of standards, what we call practice indicators—more concrete descriptions of the parts within each standard.

One of the validation points was to go to the programs and measure the staff practices and skill change for the young people. Presumably the extent that the program staff practices are very high quality compared to other programs and to youth skills change over time to improve, that leads us to believe we are getting advice from the right people.

An important part of the story is that we did that kind of measurement in all of these programs we found consistently, statistically significant and substantial youth gained skills over the period of the program. We reconfigured our Assessment to test the SEL and used those measures as performance measures and saw very strong results.

These programs were extremely high quality on YPQA compared to the reference sample we used.

There is a ton of literature and information about what changes inside of individuals, including [NEW] and CASEL references. We were not focused on what happens inside of individuals in this study, we were focused on what the adults did and the behaviors—what the kids were doing—when adults thought they were demonstrating the skills, so the behaviors. We did this because it is observable and we can use this to give people performance feedback.

KP: To help people understand what this growth looks like, give us a concrete example of what it looks like for young people who experienced growth in one of those domains?

Charles:  You can use many terms but I’m happy we’re using “SEL” has “e” in it and emotion management is such an important skill. Emotional dynamics happen really fast and can create states and moods that can last for hours or days. Emotion often doesn’t get enough attention in our thinking about youth development and learning.

There is a scale of youth skills. We developed the behavioral measures by asking staff, “What do you see when the kids are demonstrating the skills you’re trying to build? What are they doing?”

The young people came in less able to check for misunderstandings when negative emotions occur, to manage positive emotions like elation and pride with confidence that doesn’t belittle or exclude others, maintain composure to work through conflicts or disagreements, actively reach out to others when they  have emotional difficulties, they can accept feedback nondefensively, they can allow others to express and take responsibility for their own emotions

That’s just a sample of the kinds of youth behaviors that the expert practitioners who work with the kids in the SEL challenge, that the kids could do at the end of the program that they could not do as well at the beginning.

KP: that Helps with their agency and identity

To what extent were you surprised? You were working with very good youth development programs that in addition to their content had an explicit commitment to build these skills. How surprised were you about their ability to talk about these practices and how they fit in with the official curriculum?

Charles: Once we got to know these folks, we were not very surprised. They have very clear ideas about what they did and why. They have very clear ideas about how the context can cue both behavior very immediately—the stuff that happens in the moment in the program setting can cause behavior to happen, and emotional states to happen, very quickly and powerfully. So they were very careful to set up that safe space where people were calm and where emotions and reactivity were managed. They were explicit about the responsive practices: they studied and had skills and practices they could apply in the moment—for problem-solving, how teams interact. 

If you think back fifteen years ago, when we were really focused on conflict resolution as a method and there were many step by step approaches to conflict resolutions. This is like that, where the staff have been trained specifically on how to respond to difficult circumstances when they happen. Any time you respond, you are both scaffolding the student to figure out how to do that themselves, but you are also modeling—the staff is doing the thing they want the kids to do.

We thought that there was great intentionality, and that’s really promising.  

The world is trying to move in that direction, we’re to deal with micro-aggressions and how to deal with kids who experience stress and trauma.

Gina: I’d like to add that there were 250 programs that applied for the SEL challenge and they were all strong. We selected the eight that we chose in large part because they were able to describe what their program is like and what they do and why. The staff in these programs are so humble despite the fact that they are experts.

An expert is someone who has multiple experiences with a situation and has been able to learn from those and apply what they’ve learned and adapt to new experiences. All of the staff in these programs were explicit that no matter how much experience they have, regardless of whether they have 10 years of this work behind them, they can’t let themselves feel like that. They are always learning about the kids whom they’re working with. Each new young person is a different young person that they’re trying to create an experience with. That’s a large part about what makes them so talented and strong.

KP: Being truly youth centered leads to humility. When you did the measurement did you just get ratings from adults, or did kids rate themselves?

We took a simple approach, we said that an “efficacy belief” about your skill in one of the six domains is probably a good indicator.

We wanted to know if the kids felt like they were good at the skills and practices. We asked them through surveys that came from ChildTrends and Lipmann et al.

Then we developed indicators of youth behavior, working with the staff.  

Again, we think that SEL is powerfully local: how you express empathy, how you demonstrate compassion, how you manage and self-regulate emotion has a lot to do with who you are and where you are.

It was important to have staff tell us what the important behaviors are and then rate their importance,

The really interesting thing is that the beliefs and behaviors has zero correlation– as we know, knowledge isn’t really correlated with action—just because people know something doesn’t mean that they do it. So: We have indicators of youth beliefs and staff ratings of child behaviors and they are not correlated within the domains.

In short:

We asked them about their beliefs using Child Trends measures and developed staff ratings about their behaviors.  

Karen: Whenever I hear the world curriculum I think of something that is sequential and linear. Are you at all concerned that as you roll out this exciting SEL “Curriculum” make people think it’s overly linear?

Charles: Yes, we are. But we want to reclaim that word a little. Weikart comes out of the early childhood field, where there was a broader sense of curriculum—the word includes not just the content or ideas you want kids to learn but the experiences they had, the parts of the daily routine that produced those experiences, and the staff supports necessary to implement the curriculum. We are coming from a broad concept of curriculum.

If we focus just on content day by day, that misses an important part of the power of these offerings focused on social and emotional learning.

KP: You don’t just have a class on emotion management, the full curriculum is about not just the content of what’s being delivered but

Charles: these programs were designed for kids they recruited very specifically to develop experiences.

These programs offer both a content and experience that these youth needed to have—they were doing SEL within a context of a concrete content or experience-building for youth.

Why we did this? Type of measurement?

The beliefs were general (I am this I can do this) the behavior were specific, did the youth do this. The context of reference was different. and that was part of or point because The field of SEL measurement is so confusing and there is so much that we wanted to take a different types of measurements and use them in this study – incidentally they all went up—but we wanted to use them so we can say what are the implications if we do different measurements?

Broader implications of this work:

KP: 250 programs applied, all of which were strong. Rationale for choosing the ones selected was that they were intentional in their focus in growing SEL skills and their ability to talk about the staff practices needed to do it. They had their own theories of change. Now that this is done, is there a value to bring these learning groups together in communities not to redo the research but to have these experiences to not only work with diverse practitioners to not only know there is common language but to use it.

CS: Yes, the design of the Challenge had that outcome in mind from the start. We could have done a book on SEL practices but we called it standards. We could have just presented the results but what we wanted to do was develop performance measures. It’s just like with the kids: we can tell humans lots of things but that doesn’t really have any impact on their behavior or change their thinking. We want people to us the standards, the descriptors of good practice, we want them to decide which ones are important for them and their youth and then try to do it and put perf measures on it and get some feedback. Get a cycle going where they can start to work with these ideas.

This based on Larson et al.

The point is for people to interact with the data—to measure their performance to think about the data.

“There are local experts in every community. If these standards serve any purpose it’s help local communities bring their experts together. There are people who know how to do this work everywhere and we need to highlight their work and give value to it. We hope that that’s one of the primary impacts of the SEL Challenge.”

K: Can we move this beyond after school? When we think about other places where youth spend their time –school, job training, treatment programs—that have official practices that are given to staff that may or may not be explicit about these developmental practices, can we use this same approach?  Can we bring these to them to help make the often “invisible” skills and practices and more visible?

Charles: we hope so. I think this is happening in all settings: we are starting to realize that the way we get along can be managed and changed and we’re getting along well and feeling positive and our bodies are healthy and feeling vibrant, lots of good things happen. There are few new ideas under the sun, basically everyone would agree that settings and groups function better when they are capable of empathy and have compassion, work together and stick to it to solve problems, divvy up tasks into roles and people can stick to that role till they have success or renegotiate it. These is lots of empirical literature linking these kinds of skills to lots of workplace-related outcomes. We think there are lots of applications of this work to other settings.

KP: As someone who has tried to get youth development principles into toehr settings for three decades, I think we’re onto something. When you are coming into a setting that already has demands on it and say, “Now we want you to do this other thing.” They say, “We don’t have time.” But when you come in and bring an authentic way to assess where they are and to allow them to look at whether their official curriculum or practices are actually do all they can to support the developmental, we’re finding a huge amount of excitement about that.

Gina: I think the most exciting thing is that I hope the standards and curriculum features will help folks already working toward increasing SEL impact and provide guidance for both those who are interested in intensive and intentional curriculum that does build skills, also examples of how staff build these  every day. Ways to ease into it and take it on deeply full force. I hope the guide is useful in both ways.

Charles: Self-Regulation and Toxic Stress, by a federal agency OPRE report jan 2015, Table 1 is really interesting, looks at similar issues from birth to early adulthood. Excellent.

I hope the folks on the call get to look at the Guide, flip to part 2 and look at the vignettes. We structured this guide to move this content out in the voices of the folks who were our expert practitioners for the challenge. When you hear how these folks talk about doing Social and emotional learning, you will believe that the hidden curriculum can become the official.  They are producing results. It’s all about the vignettes. Their voices are super powerful.

KP: we’re talking about adolescence, do these practices also apply to elementary school as well?

CS: I was recommending the PRE because there are some changes that would need to be made to apply our standards to middle childhood—kids in first to fifth grade. We hope to do a challenge for elementary age kids next. Local experts can draw on the parts that work for adolescents and those that work for younger kids.

GET MORE: The guide and technical report will be released in February, along with the online resources including opportunities to interact with partners as a learning community and an online SEL strengths builder that helps people do continuous improvement work, applying the standards into programs.

Learn more by checking out the guide and listen to the recording

This is part of the Readiness Roundtable Series, a lively and candid conversations with some of the most influential, boundary-pushing leaders working to improve the lives of children and young people in the U.S.