Some Hard Truths About Evaluation

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

Sean Stannard-Stockton, our blogger colleague at Tactical Philanthropy, poses these very difficult questions for his latest Giving Carnival:  How should social impact (or nonprofit effectiveness) be evaluated? How can we best understand the output of nonprofit or for-profit social enterprises? It is not enough to simply say that an organization is “doing good.”  How much good are they doing and how effective are they at turning “inputs” (donations and/or investment dollars) into social “outputs”?

I offer this modified version of an earlier post in reply ...

We seek a kind of scientific or moral certainty from a formal evaluation.  But it can provide neither.  The questions that funders most often bring to an evaluator—Was this program worth our $25,000 investment?  Should we continue funding it?—are questions only they can answer.  There’s simply no absolute scale against which an evaluator can measure the value of a philanthropic investment.

Here are some of the typical moves in the evaluation game:

WE:	Was this youth development program worth our $25,000 investment?
EVALUATOR:	Dunno. What are the kinds of outcomes you’d like from a youth development program?
WE:	Better grades in school.
EVALUATOR:	Hmm. Don’t see any of that. Would you settle for a lower dropout rate? I see some of that.
WE:	That sounds good. How much lower?
EVALUATOR:	I see a 15 percent lower dropout rate. Is that worth $25,000 to you?
WE:	Don’t know. Let me think about it.

The evaluator’s first job is to determine who is asking the evaluation question, and his second job is to discover what that person values.  Then and only then can he design an evaluation that attempts to detect those values.  An evaluator isn’t measuring the effects of a given program willy-nilly.  He can suggest various outcomes that we might look for from a given program, but, for the most part, he’s trying to detect the outcomes we think are worth seeing.

And so we’re at the stage where we’ve detected a 15 percent lower dropout rate for youth participating in our program, as in the example above.  Assuming it’s not an effect of how the young people were selected for the program, is the outcome—together with other secondary outcomes—worth what it cost to run the program?  We can try to compare this program with other programs we know, but they appear to be very different from one another.  Some have a mentoring component, others don’t; some include leadership training, others focus on academics.  The annual cost per student is $5,000 in one program, but $10,000 in another, but there are significant differences in the quality of program delivery.  At a certain point, the annual cost per student starts looking ridiculous—but what is that point? And is it the same for you as for me?

And so it goes.  What then is the value to society of a given nonprofit program or organization?  The simplest answer is that it’s whatever the market is willing to pay for it.

For those of us in the nonprofit sector, one of the hardest truths about evaluation is that the value to society, in dollar terms, of nonprofit work is as much determined by market forces as is the price of cheddar in Manhattan.  And we, the people who ask the evaluation questions—the funders, the donors, the board members—are that market.

Good evaluators know this.  They would gladly tell us, but often they intuit that we’d rather not know.  Frequently their role is to describe all the potentially valuable outcomes of a given program or organization and then retire from the scene, leaving us to ponder what kind of investment it might be worth.

The counsel most useful to us at that point might come from a priest or a philosopher, or, if we’re on a very tight budget, a Ouija board.

Comments

[Phil]

I am much more interested in metrics of the bad being done by all entities, whether business, nonprofits, or government. How can we measure the good done without knowing the bad? Without an objective and measurable good/bad universal scale, we are not being scientific.

[Holden]

The purpose of metrics isn't to establish the dollar value of a project. As you say, that's determined the same way as the dollar value of cheddar in Manhattan.

Metrics serve two vital purposes: (1) helping us to decide *between* different charitable options, as we try to do as much good as possible (not just "some good") with our donations; (2) helping the charities themselves to learn what works and what doesn't, and improve their own ability to help people.

There is nothing easy about metrics, but they're the only way I know of to these benefits, benefits we simply can't leave on the table.

[Albert Ruesga]

Phil: Why measure the bad when we can all simply agree that mistakes were made?

Holden: I think some metrics accomplish what you say for some projects. When a youth development program starts costing $25,000 per young person per year, might it not be better to sock that money away into college accounts for the participants? I don't know. So many axes of value to compare -- quality of instruction, intensity of intervention, paucity of other opportunities: how do we assign weights to these? We can never compare two youth development programs the way we compare two cans of soda: they differ in too many respects.

As for internal evaluations, those we use to improve the programs we ourselves design and run, I suppose we need to ask whether the cost in dollars and opportunity is worth the improvements we imagine we'll eke out of the process.

In any event, aren't we always informally evaluating our own work, constantly trying to find new ways to improve it? And is the explicit articulation of "metrics" really part of this process?

[Holden]

Yes, the explicit articulation of metrics is essential. Informal evaluation and formal evaluation both have advantages and disadvantages - and that means if you're using only one, you're missing out.

Of course there is such thing as unnecessary and overly costly evaluation. But in this particular sector at this particular time, which way do we need to be pushing? I'd argue that we're way too far toward having NO information about what charities accomplish. Never mind a formulas - we literally have no reasonable way, either analytical or intuitive or emotional, to decide between different donations.

From my experience in the for-profit sector, I have been constantly, and I mean constantly, knocked off my chair by the extent to which measured reality diverges from what I expected based on my own internal logic and intuitions. It isn't because I have bad intuitions, it's because reality is a tough cookie. The history of scientific progress is the same story.

[Gerry]

The only metrics I've seen in this discussion are micro level. Without an understanding of the whole situation, measurement is almost useless. The blind men are arguing about metrics to count hair on the elephant without having a single idea about the elephant itself.

The most arbitrary thing here is dollar value. It is different for everyone, and philanthropy is not different. What is more important, how the donor values the result or how the beneficiary values what was given?

Phil, not only must we measure the bad, the negative outcomes of any activity, but we must remember what is missing. Who isn't being served, who is being left outside of every metric, not counted in the total.

[Sean Stannard-Stockton]

I think Gerry's comment regarding who's value are we counting is spot on. I tried to make this point in a recent post.

I think we should be measuring total value generated. My point in the post was that with a charitable donation, only the donor is putting a number on the value. Using the logic of the market system, the donor is only valuing the benefit that accrues to them personally. But the true value of the work done by the nonprofit is the benefit that accrues to society as a whole.

[Holden]

Letting the donor name the value is just being honest. Value to society *is* valued by the donor. To the extent it isn't, the donor won't pay, no matter what a formula says.

Saying "Fixing this problem would be worth $X to society" is, in the end, just not useful to anyone. If that's wrong, please explain who is supposed to make use of it. By contrast, saying "Spending $X would result in Y lives saved" is meaningful, and helps donors make the decisions that are theirs whether you like it or not.

[Sally Wilde]

Looking at this conversation from the outside, it appears to me that you agree on so many things: that explicit metrics are valuable, although you might not all agree on the range of their application; that only the donor (here interpreted broadly as “funder”), not the evaluator, can decide if the outcomes of a given intervention are worth $N to the donor, who, after all, is likely the one who raised the evaluation question in the first place; and, finally, that any responsible donor would consider the value to society as a whole of a given intervention, not just the value to him or her.

The way I interpret it, the discussion introduced a series of correctives. Sean wanted to stress the fact that it’s not enough for an organization to do good—although one might ask, not enough for whom? It is enough for me that a particular youth development organization provide a good experience for the young people it serves, and that its staff continuously try to improve the program, with or without the assistance of formal evaluations.

Holden reminds us that there are various uses of formal evaluations, each with its own kind of logic. One kind of evaluation helps us choose between two charities or “different charitable options.” I took it as one of the points of Albert’s post to illustrate how fiendishly difficult this can sometimes be, and how our desire to impose metrics threatens to take us deep into the realm of pseudoscientific hooey.

Gerry rightly reminds us of the multiplicity of points of view from which value can be measured, what some critics have called the embarrassment of axiologies.

I forgot where I was going with this …

[Mark Petersen]

I guess I see metrics as not the end game, but the starting point. Yes, they are necessary, and the rigor involved in quantifying output/impact is important. And we require these from our grantees.

But once we have the stats - what then? I hope they open up the way for a conversation between donor and NPO on whether the 'market' considered it a worthwhile purchase, which determines then whether to re-invest. There are all sorts of nuances and intangible benefits to society and to individuals that result from a financial intervention. The numbers themselves don't tell the whole story.

[Holden]

Do we all agree on everything? Let me repeat what I find most important:

Formal evaluation driven by good, concrete metrics is essential. Nobody has said at any point that metrics are the end goal or the only thing we need. And we all know they're fiendishly difficult to conceive and measure. But they are ESSENTIAL.

If we all agree on that, perhaps we can go to the next step and talk about what sorts of metrics should be used? GiveWell's take is right here. I'm interested in your thoughts.