Monthly Archives: May 2015

Are outcomes evaluations right for youth charities?

At long last nonprofits are lurching into the brave new world of outcomes evaluations, but could this be a misstep for youth charities?

Believe it or not, in 2015 we still haven’t come to an agreement on the fairest way to assess whether an organization is worth supporting. In general there are three types of measurements of a charity’s success – input (resources used), output (people served), and outcomes (effect on the people served). Of these, input is the only one readily discernible on financial statements. Input metrics such as salaries and the overhead percentage indicating how much a nonprofit spends on things other than direct program costs have been the philanthropists’ St. Peter for a long time now, dividing charities between the good and the bad, the worthy and the unworthy, though recent debates are threatening to slam the pearly gates of input shut for good.

 

LOS FELIZ, CA - NOVEMBER 20:  Fortune teller Deseret (R) gives a tarot card reading at "A Night of Celebration" in honor of director Rob Minkoff and the completion of "The Haunted Mansion" at Minkoff's home on November 20, 2003 in Los Feliz, California.  (Photo by Amanda Edwards/Getty Images)

“That one? But I heard it spends 20% of its budget on management.”

 

Ever since “The Overhead Myth” was published in 2013, funders and fundees have been scrambling to get on board with “rigorous” outcomes evaluations. The manifesto, co-authored by representatives of GuideStar, the BBB Wise Giving Alliance, and Charity Navigator, rightly pointed out that the practice of vilifying nonprofits for spending too small a percentage of their income on programs is a terrible way of determining if an organization is actually doing a good job. It doesn’t take into account differences in service fields and constituents that might necessitate more spending on management and fundraising (“overhead”) and is plunging plenty of good orgs into the “nonprofit starvation cycle,” in which infrastructure and services are gradually cut in a desperate effort to meet short-term overhead percentage goals – as little as 10% depending on how finicky the funder, but typically 20-40% of the total. Outcomes, on the other hand, seem like a fair and honest way to decipher a nonprofit’s contributions.

And they can be. If you’re interested in UNICEF’s present efforts in Nepal, the amount of dollars they spent there (input) reveals less than the number of people they served (output) and seems almost like a meaningless statistic compared to 50% of Kathamundu families who lost their homes in the earthquake are living in UNICEF-built dwellings, 85% of schools have reopened, 100% of children within the charity’s jurisdiction have access to clean drinking water, or other outcomes (those are hypothetical achievements, by the way – UNICEF’s outcomes-based evaluation of their work in Nepal hasn’t been published yet).

But youth charities present a special case. Despite embracing rigor and promising to prove long-term efficacy, their work by its nature makes taking accurate outcome measurements very tricky:

  • Many of them address issues that cannot be resolved during one program session or evaluated meaningfully in the short term. “Recovering from childhood trauma is a very long process,” said developmental psychologist Dr. Sascha Kahrs, referring to the many nonprofits serving young victims of abuse. “Each time a child learns a particular coping skill or is able to express what they’ve been keeping inside is a step closer, but you won’t really know if he or she recovers until you can observe them as young adults.” In the same vein, how can a youth charity honestly say if their programs are immediately inspiring a generation of scientists or empowering bullied children? And even if nonprofits such as Free Arts LA were able to track down and evaluate participants ten years post-program, the most current outcomes would only reflect how effectively the organization worked a decade earlier.
  • Variables can’t easily be quantified or standardized. “How can you tell if a five-year old gained a more positive sense of self-esteem?” Dr. Kahrs mused. “There isn’t a meter you can simply read off. He or she might be happy for the hour that they’re at the program but still feel pessimistic and unsure at home, especially if their home life isn’t all that great.” He points out that children are generally not very good at describing how they feel. Unlike houses built or gallons of potable water delivered, the goals scattered throughout youth charity mission statements (empowerment, enrichment, inspiration, comfort) can be reported on so many different scales that comparing results becomes confusing. Which program outcome for a charity serving gravely ill children is the best: a) immediately reduced feelings of isolation from 9 to 4, on a 1 to 10 scale; b) 95% of parents polled observed a positive change in the child’s outlook after participation in the program; or c) last year 473 children who recovered from their illness ranked the program among the top three experiences they had during their treatment?
  • Positive results almost always require cooperation but outcomes assessments penalize cooperation. One of the purposes of outcomes evaluations is to weigh one nonprofit against another, in order to decide which of them to fund. The impetus therefore is not only for charities to take sole credit, if possible, for what was achieved through group effort, but also to eschew group efforts in the first place. Yet in the case of child abuse victims, the scientific consensus favors a “the more, the merrier” approach. Dr. Kahrs explains: “There’s no single intervention that works for everyone. It’s not even likely that a child can recover from one type of intervention, even if they’re responding positively to it. What you need is lots of different opportunities – cognitive behavioral therapy, creative outlets, support from adults, and so on.” Alongside the nonprofit starvation cycle, then, the pressure on youth charities to demonstrate unique, impressive outcomes creates a perverse incentive that’s detrimental to their interest in helping children.
  • Outcomes evaluations are expensive to acquire. It’s easy to count the number of polio vaccines administered. Designing, implementing, and following through on a system that parses subjective feedback into useful statistical data requires expert help and can eat up a good chunk of an organization’s budget.

 

youth charities los angeles

Focusing too hard on impressive-sounding outcomes might force a nonprofit to take its eyes off the prize – doing the best it can to help children live safe, happy, and fulfilling lives.

 

All of this isn’t to say that outcomes evaluations are a bad thing. On the contrary, they are essential. Nonprofits serving children must track their outcomes because they need feedback to improve their services, and as a method of fairly assessing charities outcomes still trumps input almost every time. Perhaps it can even benefit from input critiques such as “The Overhead Myth.” That document’s authors called for charities to educate funders on “the real cost of results.” Correspondingly, youth charities should be proactive and transparent about the realistic “context of results.” Rather than seizing the opportunity to trumpet an impressive-sounding program statistic, they should couch it as an opportunity to explain their long-term efforts to the public, and to locate themselves as an essential part of a child’s support network that might include parents, schools, other nonprofits, and more. But in order to ensure that the tracking of outcomes is beneficial rather than detrimental to the work of youth charities the funders also need to do their part. They need to understand that the special circumstances surrounding youth charities mean they need to do more work to understand outcomes, such as making site visits and reading detailed reports, and shouldn’t keep rewarding those that summarize results in a snazzy, oversimplified infographic. They need to stop putting nonprofits with common stakes in naked competition with one another. And finally, they need to stop demanding organizations throw money chasing evaluative rigor that can’t realistically be achieved, lest financial rigor mortis sets in.

Do you have more questions about outcomes evaluations and how they impact your nonprofit’s attractiveness to donors? Please contact Wild Fundraising at 619-436-7161 or contact@sarahwildfundraising.com for a personalized assessment.