White Courtesy Telephone

Defining Nonprofit Success in the New Millennium

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

Some people say a nonprofit organization is “successful” if* …

1. it manages to do its work entirely with volunteers.
2. it has an inspiring brochure.
3. it has paid staff and an annual operating budget of over $1 million.
4. it’s “scalable.”
5. it maximizes its social benefit-to-cost ratio.
6. it minimizes its cost per client per unit of social utility produced.
7. it provides a high quality service to a few of our most vulnerable neighbors.
8. it provides a fairly low quality service to many of our most vulnerable neighbors.
9. it’s replicable.
10. it operates in more than one city.
11. it operates in Nambia.
12. it operates in more than one country.
13. it has branch offices on other planets.
14. it’s “sustainable.”
15. it requires no subsidy to pay for its operations.
16. its operating budget is over $100 million.
17. it has a president/CEO who’s a household name.
18. the organization is a household name.
19. it’s not currently embroiled in scandal.
20. it’s funded by the Ford Foundation.
21. it’s funded by the Ford Foundation and the Bradley Foundation.
22. Betsy DeVos is a board member.
23. Neil deGrasse Tyson is a board member.
24. Bill Clinton is board member.
25. Colin Kaepernick is a board member.
26. God is a board member.
27. Beyoncé is a board member.
28. its work “disrupts” traditional models.
29. it fosters a strategic, integrated competence in a pro-active, extended paradigm to create digital, re-engineered synergies that affect its double bottom line.
30. it accomplishes its mission.
31. at least 51 percent of its funders agree that it has accomplished its mission.
32. at least 51 percent of clients served agree that it has accomplished its mission.
33. it accomplishes its mission then goes out of business.
34. building on previous success, it expands its mission.
35. the organization is important to a marginalized segment of the population.
36. year after year, in its own quiet way, it manages to scrape enough money and volunteers together to serve a community largely ignored by other brand name nonprofits.
37. the organization changes the basic equations of power in favor of the oppressed.
38. it works toward a vision of the Good shared by an enlightened elite.
39. it works toward a commonly shared vision of the Good.
40. it helps focus our minds on eternal things.
41. it has been roundly criticized in one of the President’s tweets.

* This is an updated version of a post published a little more than a decade ago.

Related articles

 

Debunking Theories of Change, Part 2 of 2

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

This post is part 2 of 2 in a series devoted to questioning the utility of theories of change in nonprofit and philanthropic work.  To review part 1 of the series, click here.  You can download a full version of the argument by clicking here.

A Convention for Representing Theories of Change

Consider the program theory for a mentorship program:

(1)   A child’s close relationship with an adult mentor --> An increase in the child’s sense of self worth --> A reduction in the child’s susceptibility to violence

Here we read the arrow symbol (-->) as “causes” or “leads to.”  A theory of change, as outlined in (1), is essentially two linked explanations.  The first explains the increase in the child’s sense of self-worth on the basis of that child’s close relationship with an adult mentor; the second explains the reduction in the child’s susceptibility to violence on the basis of his increased sense of self-worth.  In this essay, I adopt, for various reasons I describe below, the conventions used by the great American philosopher of science Carl Hempel for representing explanations in the natural sciences.  Those conventions have us represent (1) as two linked syllogisms:

(2a)	1.	If I provide child A with a close relationship with an adult mentor, then child A's sense of self-worth will increase.
	2.	My program provides child A with a close relationship with an adult mentor.
	_______________________________
	3.	Therefore, child A's sense of self-worth will increase.

 

(2b)	1.	Child A's sense of self-worth will increase.
	2.	If child A's sense of self-worth increases, his susceptibility to violence will decrease.
	_______________________________
	3.	Therefore, child A's susceptibility to violence will decrease.

These two syllogisms  are linked in the sense that the conclusion of the first syllogism (“Child A’s sense of self-worth will increase”) becomes the initial premise of the second syllogism.

The syllogism outlined in (2a) is an example of what Hempel calls a “deductive-nomological explanation.”  Its first premise (“If I provide child A with …”) has the look and feel of a general law of human psychology: it tells us something about how normally constituted individuals tend to behave under certain ideal conditions.  Compare it, for example, with the following “lawlike” statement from physics: “If at time t = 0 we drop a ball from a height h above the earth, and we drop it in a perfect vacuum, and there is nothing to obstruct its path as it falls, then it will hit the ground at time t = the square root of (2h/g).”  The second premise in (2a) has a different character.  It’s an “observation” sentence that makes assertions about certain observable facts—viz., my program provides child A with a close relationship with an adult mentor; the mentor is well trained; etc.

Identifying each link in a theory of change with a deductive-nomological explanation as defined by Hempel has several advantages.  First, it provides a common language and framework for arguing about the utility of theories of change.  No longer need we suspect—as I often have—that people who disagree about theories of change are arguing past one another, that they understand “theories of change” in very different ways.  Second, Hempel’s formalism requires that we make explicit the laws and conditions that need to obtain in order for certain effects to follow from our social interventions.  The very schematic theory of change outlined in (1), for example, does not do this. By contrast, the theory of change described in (2a) is richer: it offers the evaluator a number of avenues for exploring the possible failure of a given social intervention.  If, for example, the mentored child’s sense of self-worth doesn’t increase, we can examine the reasons for this failure in two places: either adult mentorship (even under the best conditions) doesn’t tend to increase a child’s sense of self-worth—a question that can be explored empirically; or the program we’ve designed simply doesn’t instantiate the conditions necessary for the success of an adult-child mentorship relation.*

But enough theory.  Let’s look at some real cases …

 

Putting Theories of Change to the Test

Imagine that you are the executive director of an organization whose mission is to put a park within walking distance of every home in America.  You’ve done this work successfully in your city and now you’re approaching the XYZ Foundation for funding that will help expand your work to an adjacent town.  You send in your proposal confident that the XYZ Foundation will note your track record of success and fund your expansion.  You’re surprised when a program officer from the Foundation gives you a call and asks you to produce a theory of change for your program.  “You mean like our inputs, outputs, and expected outcomes—that kind of thing?” you ask.  “No, that’s a logic model,” he says.  “I want to see an articulation of the underlying beliefs and assumptions that guide your service delivery strategy and are believed to be critical for producing change and improvement.”   “Well, uh, our beliefs and assumptions are that our program will succeed in creating a park in Neighborhood X of Town Y and that this will induce the residents of Neighborhood X to use and enjoy it.  But we told you that in our proposal,” you answer him, somewhat testily.  “Do your best,” your program officer insists.

Although you suspect your program officer is deeply in the grip of an ideology, your need for funding drives you to comply.  You begin hesitantly:

(3)	1.	Every other time we've created a nice park within walking distance of residents' homes, they have used and enjoyed it.
	2.	Our effort in Neighborhood X will succeed in creating a nice park within walking distance of residents' homes.
	_______________________________
	3.	Therefore, residents of Neighborhood X will use and enjoy it.

You send this to your program officer and after several e-mail exchanges, you agree on this emendation of your original theory of change:

(4)	1.	If we create a park within walking distance of residents’ homes, and residents participate in the design of the park, and the city commits to its upkeep, then residents will use and enjoy it.
	2.	We will create a park within walking distance of the residents of Neighborhood X, and residents will participate in the design of the park, and we will get the city to commit to its upkeep.
	_______________________________
	3.	Therefore, the residents of Neighborhood X will use the park and enjoy it.

You receive funding from the XYZ Foundation and build the park in Neighborhood X, just as you described in your proposal, and just as outlined in your theory of change (which to you still appears completely superfluous—but never mind).  A year later, when you conduct an evaluation of your program you discover, to your great surprise, that the residents of Neighborhood X are not using the park.  After polling the residents, they tell you they feel unsafe in the park because drug dealers have started using it to conduct their business.  Soon after, you hear from your program officer.  “Your theory of change was wrong,” he e-mails you.  “Please refer to the attached document for the correct theory of change.”  You open it and find this:

(5)	1.	If we create a park within walking distance of residents’ homes, and residents participate in the design of the park, and the city commits to its upkeep, and residents feel safe in the park, then residents will use and enjoy it.
	2.	We will create a park within walking distance of the residents of Neighborhood X, and residents will participate in the design of the park, and the city will commit to its upkeep, and we’ll take steps to ensure that residents will feel safe in the park.
	_______________________________
	3.	Therefore, the residents of Neighborhood X will use the park and enjoy it.

So, OK, you now convince Town Y’s police department to station a police cruiser next to the park.  One year later, same problem: residents aren’t using the park.  It turns out that because of the rising levels of violence in Neighborhood X, residents don’t even feel safe walking out of their homes, much less to a nearby park.  You wonder if you’ll get another e-mail from your program officer with yet another version of your theory of change:

(6)	1.	If we create a park within walking distance of residents’ homes, and residents participate in the design of the park, and the city commits to its upkeep, and residents feel safe in the park, and residents feel safe walking from their homes to the park, then residents will use and enjoy it.
	2.	We will create a park within walking distance of the residents of Neighborhood X, etc.
	_______________________________
	3.	Therefore, the residents of Neighborhood X will use the park and enjoy it.

There are several things to be noted about this simple example:

1. The most relevant parts of your theory of change were already implicit in your original grant proposal.
2. Connected with this last point, if your program officer can’t “get” what your theory of change is simply by reading your proposal, he or she might simply be in the wrong business.
3. Even in a very simple example like this one, articulating the “underlying beliefs and assumptions that guide your service delivery strategy and are believed to be critical for producing change and improvement” can get ridiculously complicated.  In general, you cannot know ahead of time what the “relevant” beliefs and assumptions will be.
4. Your theory of change is not working for you: you are working for it.  It’s like a dead albatross around your neck that you occasionally remove and dip in formaldehyde to keep your funders happy.

Our problems are compounded when we look at more complicated examples.  In her article, “How Can Theory-Based Evaluation Make Greater Headway?”** Carol Weiss describes a program that gives tenants representation on the management board of their public housing project:

This program espouses multiple goals, such as improving the maintenance of the property, decreasing crime in and around the housing project, and encouraging residents to take greater responsibility for their lives in such areas as work and education.  These are ambitious goals and, to some, inflated hopes, but let us take the most realistic objective, improved maintenance of the project … (p. 503)

What’s interesting about the description that Ms. Weiss gives of this project is that many of us immediately share her intuition about which of  its goals is the most realistic.  Even without seeing a detailed proposal, or a theory of change, we can imagine ways that tenant representation on the management board of a public housing project might lead to improved maintenance of the property.  Ms. Weiss points out that there are multiple causal paths that can lead to this goal, and she sketches a program theory that I reproduce below as Figure 1:

 
Figure 1

What’s striking about this theory of change is its overwhelming complexity—and this in spite of the fact that it doesn’t begin to articulate all the underlying beliefs and assumptions that might be critical to the success of the project—e.g., How much power do residents wield on the management board compared with non-resident managers?  How constrained is the board by a lack of funding?  How much time can residents reasonably dedicate to the management of a complex property? etc.  Weiss argues that:

The small everyday steps in Figure 1 can be couched in social science language, such as sense of communal responsibility, access to authority, local knowledge, bureaucratic responsiveness, and so on.  In the conduct of the evaluation, the evaluator will need to operationalize these concepts and develop specific indicators that capture their essential meaning. (p. 506)

This sounds like an enormous undertaking for what starts out as a fairly simple project.  If, after a year, the condition of the property doesn’t improve, is it really practical to ask our evaluator to do as Weiss suggests?  How much would such an evaluation cost?  Would he or she test each of the branches in Figure 1?  And just as with our previous example, we can imagine a thousand reasons, not pictured in Figure 1, why the program might have failed.  How, in that case, would the theory of change we’ve so carefully crafted guide our evaluation efforts?

By the way, we can also imagine scenarios in which the property improves but for none of the reasons outlined in Figure 1!  One of the resident managers, for example, might have a brother-in-law whose contracting company does excellent work for half the price; or one of the resident-managers might be very successful at organizing resident work crews who dedicate one Saturday each month to repairs and upkeep; or the very presence of residents on the management board might inspire a local donor to endow and thereby double the public housing project’s maintenance budget; etc.  Real life is extremely complicated.  When human actors are involved, its twists and turns are almost always unpredictable.

It takes enormous effort to dream up a program theory like that pictured in Figure 1.  And as we’ve seen, it can easily turn out to be a completely fiction.  At this point we need to ask, whom does it serve and to what end?

Adding It All Up

To summarize:

1. An airtight, explicit theory of change, even for the simplest intervention, might be beyond the powers of any mortal known.  If you’re a funder, think twice before you require it.
2. What typically passes for a theory of change is a narrative of how certain causes can plausibly lead to certain effects.  If this is all you’re really after, then read the proposal.  Everything you want to know is likely already there.
3. Because a theory of change is more like a narrative than an explanation in the social sciences, it will be more compelling if it’s written by a good story teller on your staff.  This moves you from the domain of sound scientific practice to the realm of compelling literature. This raises for me serious questions about the weight we should assign to these narratives in grantmaking work.
4. Funders who do not require highly articulated, explicit theories of change from their grantees are not treating them as idiots savants.  They’re treating them as fully rational human beings who have had enough experience of the world to know which effects generally follow from which causes.

I am not here denying that theories of change, and the process of constructing them, have helped many people in the nonprofit world.  But while I myself have found it enormously useful to discuss my plans with my Jack Russell Terrier, I would never dream of requiring this of my grantees.  There are myriad ways that organizations can fruitfully plan their social interventions.  All will likely involve some anticipation of hoped-for effects, and most, I assume, will fall far short of fully developed theories of change.  Some interventions will be designed entirely on the basis of intuitions or gut feelings.  While this is not a style of program planning I practice or endorse, I’ve also not yet seen any evidence that its failure rate is higher than that of allegedly more “scientific” approaches to planning social interventions.

The acid test for me is this: Take a close look at your theory of change:  Does it tell you anything beyond what you’ve already included in your grant proposal?  Does it suggest to you ways that you might improve your work?  Given all the factors we’ve considered in this article, are you confident of its accuracy?  Is it absolutely required by your funders?  No?  Then, borrowing a suggestion from David Hume, commit it to the flames!  For it can do nothing but confound you and waste untold hours of your time.

-----

* Hempel’s formalism certainly has its limits, especially when one uses it to describe change processes that are not strictly linear.  And as Hempel himself acknowledged, special consideration must be given to so-called “probabilistic explanations.”  But these complexities only serve to underscore my point about the questionable utility of theories of change in nonprofit work.

** Weiss, Carol 1997. “How Can Theory-Based Evaluation Make Greater Headway?” Evaluation Review, Vol. 21 No. 4.



Debunking Theories of Change, Part 1 of 2

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

Editor’s note: A full version of the argument presented in this post is available by clicking here.

In the Spring 2010 issue of the Stanford Social Innovation Review, Paul Brest, President of the William and Flora Hewlett Foundation, took several of my colleagues* to the conceptual woodshed for their skepticism about “theories of change.”  As Mr. Brest and others understand them, theories of change include our beliefs about the causal paths that lead from our social interventions to our hoped-for effects.  These beliefs naturally shape funders’ approaches to social change. Grantmakers will not, for example, fund programs that attempt to reduce violence by requiring gang members to wear Styrofoam shoes, for this intervention would not be supported by any plausible theory of change.

If we agree with Mr. Brest that a theory of change includes one’s beliefs about the way the world will behave given this or that intervention, we might wonder what there is to be skeptical about.  Do the skeptics deny that grantees have beliefs about the effects of their social interventions?  At the very least, Mr. Brest is concerned that these skeptics feel it’s unimportant to require grantees to articulate their theories of change:

Schambra and Somerville argue that funders should focus their grantmaking on community-based organizations, and that funders should trust an organization’s leaders without requiring them to articulate theories of change ….  These skeptics are implicitly analogizing grantees to idiots savants—individuals who are able to do complex calculations … in their heads without knowing, let alone being able to explain, how they do it.  [p. 48, emphasis mine]

It should be noted that one of the interesting characteristics of idiots savants is that the complex calculations they perform in their heads are often correct.  If the nonprofit world harbored the analogues of idiots savants—executive directors, for example, who could correctly intuit the interventions required for lifting whole communities out of poverty—I would strongly encourage them rather than force them to give an account of their methods.  Despite their inability to explain how they arrived at their program designs, I would prefer that we went about changing the world the way they did, rather than the way most of us didn’t.

Mr. Brest’s claim that the skeptics alluded to in his article are analogizing grantees to idiots savants also needs to be challenged.  An idiot savant cannot explain to you how he is able, in under a second, to calculate the square root of a 55-digit number.  A talented nonprofit executive director, by contrast, can draw on a lifetime of experience and whatever research is available to him to give an account of the design of his social intervention.

In many cases, of course, nonprofit executive directors are working from implicit theories of change.  Since these implicit theories of change are ubiquitous, there’s a clear—but vacuous—sense in which theories of change are indeed as “powerful” as Mr. Brest claims.**  But this is not what’s at issue between him and the skeptics he alludes to in his article.  In writing about the evaluation of social interventions, for example, Mr. Brest claims that “Identifying the proper variables and metrics requires that one articulate the program’s goals and its underlying theory of change” (p. 48).  Even more forcefully than this, “Improving the lives of disadvantaged populations ... requires proven theories of change” (p. 47, emphases mine).

Should we really be requiring grantees to articulate their theories of change?  The simple answer, as I argue below, is that there’s little need for this requirement since most grantees already do a sufficiently good job of describing “the empirical basis underlying their intervention,” as Mr. Brest puts it.  They do this when they write their grant proposals.  The more surprising answer, perhaps, is that requiring grantees to produce explicit theories of change—beyond what they usually include in their grant proposals—does little to improve the art or science of grantmaking.

Highly elaborated theories of change are generally urged upon grantees by well-meaning people who have a limited understanding of how they function in the social sciences.  Because theories of change are often shrouded in the impenetrable verbiage of philanthropy, it’s also not surprising that most of us have little inkling of their theoretical and practical limits.

Speaking of practical limits, do you, dear reader, know with unerring certainty what a theory of change is?  If you do not, you are entirely forgiven and in very good company.  Take a little time to review the literature on theories of change and you’ll discover, as I did, that theories of change are everything and they are nothing.  Unfortunately, as I argue below, the “power” that Mr. Brest ascribes to them is largely the power to waste enormous amounts of your time.

The Radical Polysemy of “Theories of Change”

The term “theory of change” is used widely in philanthropy and other fields to describe a family of concepts whose members, upon close inspection, vary significantly in meaning.  “Theory of change” is a term of obscure origin, as far I can tell, associated in the minds of many with the culture and language of formal evaluation.  A grantmaker or grantee scanning the available literature on theories of change can hardly help but be confused by the exercise.  The GrantCraft publication titled Mapping Change, for example, opens by telling the reader not what a theory of change is, but rather what it does:

A theory of change describes a process of planned social change, from the assumptions that guide its design to the long-term goals it seeks to achieve. 

From this the reader might conclude that a theory of change is the narrative of some social change process with goals appended.  But would any such narrative qualify?  After all, “describing a process of planned social change” can mean writing a rambling manifesto, penning the succession plan for a drug cartel, or developing marketing guidelines for a smokeless ashtray.   The authors acknowledge this ambiguity when, several paragraphs into their article, they add:

What does “theory of change” really mean, in practice?  Grant makers who use the term may be describing anything from a detailed map to a general storyline.  What they agree on is that a theory of change is valuable if it helps them and their grantees understand the relationship between the problems they’re addressing and the strategies they’re using to get the work done.

At this stage, the careful reader might have some inkling of what a theory of change does and why it might be useful.  He might still have little idea what it is.  If the essential mark of a theory of change is that it helps us better understand how our efforts do—or do not—lead to our goals, then it’s possible that a glass of Merlot can qualify as a theory of change, for as Heraclitus reminds us, “It is better to hide ignorance, but it is hard to do this when we relax over wine.”

On one popular website the authors tell us that a theory of change “defines all the building blocks required to bring about a given long-term goal,” while on another we’re informed that it’s “the product of a series of critical-thinking exercises that provides a comprehensive picture of the early—and intermediate—term changes in a given community that are needed to reach a long-term goal articulated by the community.” In some publications they’re equated with so-called “logic models,” in others they’re not. 

One of the claims frequently made for theories of change is that they’re essential for helping us evaluate the effects of our social interventions.  As Mr. Brest himself informs us, “most evaluations of social interventions require understanding the theories that mediate between inputs, activities, and outcomes.” (p. 48)  Interestingly, many of the formal evaluation textbooks I researched didn’t use the term “theory of change” at all.  The closest cognates I found were the terms “program impact theory,” used, for example, in the 7th edition of Evaluation: A Systematic Approach by Peter H. Rossi et al., a popular textbook on evaluation; and the more common “program theory,” used in a significant corpus of books and papers by such authors as Carol Weiss and others.  Generally, these scholars understand a program theory to be a sequence of causes and effects, in which our interventions function as the instigating causes and certain hoped-for social benefits function as their ultimate effects.

To take a very simple-minded example, suppose we diagram the program theory for a mentorship program as follows:

(1)   A child’s close relationship with an adult mentor --> An increase in the child’s sense of self worth --> A reduction in the child’s susceptibility to violence

Here we might read the arrow symbol (-->) as “causes” or “leads to.”  But scan the theories of change produced by professional and amateur evaluators and you’ll find a significant range of choices regarding which types of causal connections to include (those drawn from individual belief-desire psychology, from the psychology of social movements, from rational actor theory, etc.); the necessary conditions one should make explicit (an inclination on the part of subjects to cooperate, a robust economy, favorable weather; etc.); and what unit of analysis to focus on (the individual, the family, the neighborhood)—among many other variations.

Given the general lack of consensus about what a theory of change is or what it should include, it’s not surprising that the skeptics alluded to in Mr. Brest’s article might fail to agree over its utility.  If we are ever to make any progress in our debates about theories of change, we will need a common language for describing and evaluating them. 

Coming up next: A definition, some case studies, and some damning conclusions

-----

* Bill Schambra, Bill Somerville, Sean Stannard-Stockton, and Jeff Berndt.

** Mr. Brest's article is titled "The Power of Theories of Chnage."

Beige Philanthropy in the Crosshairs

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

Think of them as two armies battling against overwhelming indifference—and frequently against one another—for the soul of philanthropy.

One camp is arrayed under such banners as “metrics” and “evaluation,” and has a distinctly business school cast.  The other champions a style of philanthropy concerned with social justice, and seeks to expose the root causes of our social ills.

These are caricatures, but they’re not entirely without foundation, so indulge me for a moment longer.

We fail to appreciate how closely united these two camps are in their rejection of philanthropy as usual—“beige philanthropy,” as I like to call it.  Most striking, perhaps, is their primary point of contact: a shared concern for effectiveness.

Here’s how it works: Those in the Metrics camp argue that few foundations make any real attempt to hold themselves accountable.  Without measuring the effectiveness of their grantmaking, they’re unable to track progress against stated goals (if they exist).  Even if these grantmakers are able to see with their own eyes the good they help produce, it’s not clear that this good significantly outweighs the cost of producing it.  Accountable to nobody but themselves, they end up shortchanging the communities they aim to serve.

Those in the Social Justice camp come to exactly the same conclusion but via a different path.  They argue that most of the giving done by mainstream foundations is based on an incomplete or flawed analysis of what it takes to achieve goals like ending poverty or ensuring that all children thrive.  These foundations content themselves with triaging society’s victims, never wondering about the causes of their victimhood, and suspecting, perhaps, that they might themselves be implicated in the crime.

It’s an easy matter to strawman mainstream philanthropy, to claim that it’s grown soft on flattery and self-congratulation.  It’s a far better strategy, in my view, for those in the Metrics and Social Justice camps to make their arguments incontrovertible—to give their speculations the heft of widely shared conviction.  There’s also a lot they can learn from one another.

March Metrics Madness

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

I was recently invited by Bill Schambra, director of the Bradley Center for Philanthropy & Civic Renewal, to speak on a panel titled Metrics Mania.  Panelists were asked to comment on an essay commissioned from Gary Walker, the founding director of Public/Private Ventures, on the subject of evaluating social programs.  Here is the text of my remarks ...

Bill Schambra saw my role on this panel as the guy who when asked about metrics replies “metrics schmetrics,” but I’m going to have to disappoint him—to some extent.  Measurement and evaluation, when done properly, are not just a bit of value-added for philanthropic or nonprofit work, they’re absolutely essential.  Only a fool would disagree with that proposition. 

But here I mean not just the kinds of formal evaluations described by Gary Walker in his essay, but informal evaluation as well: the kinds of course corrections we naturally make when we embark on a project, take a false step, and adjust what we do accordingly.  Evaluation is not and should not be the sole province of the highly compensated consultant.  We evaluate all the time; our own eyes and ears notice things the most astute consultant will never notice; and we’ll often be our own worst critics.

Now here’s where the metrics schmetrics comes in, perhaps: We’ve written more nonsense about evaluation than just about any other subject in philanthropy.  Worries about evaluation, engendered in part by logic models the length of whale intestines, have become the math anxiety of the philanthropic world.

My general thesis—if I could call it that—is that from the perspective of somebody like Mr. Walker whose organization has been commissioned to conduct lucrative, large-scale evaluations of social programs (lucrative by nonprofit standards), the Impact Revolution might seem like a good thing.  But from the ground, from the perspective of many people working in community-based organizations, this so-called revolution has brought with it new sources of irritation, new ways of adding meaningless make-work to already overburdened nonprofit staff members.

It has not been a people’s revolution, in other words, but rather one championed by elites—like myself, I’m afraid— sometimes unable to see far enough beyond our own measuring sticks to understand the limitations of formal evaluation techniques, and the trade-offs in staff time and other resources that these formal techniques require.

The Axiology of Nonprofit Impact

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

In a celebrated 2001 essay, wordsmith Tony Proscio asked, “Are grantmakers a species of investor, building benevolent enterprises that produce a measurable return for society, or are they more passive enablers of good, seeking mainly to support those who pursue charitable ends by whatever path?”

He was commenting on the tendency of the foundation sector to import the language of business while remaining largely oblivious to its insights.  But his question still manages to provoke.  The debate over the “measurable return” of charitable enterprises rages on between the wielders of business metrics and the Luddites who oppose them.

The Nonprofit Roundtable of Greater Washington attempts to stake a middle ground with a new report titled, Beyond Charity: Recognizing Return on Investment.  The report includes the kinds of cost-saving arguments familiar to many of us: each dollar spent on Johnny’s education saves us tens of thousands of dollars in prison costs and helps turn Johnny into a productive citizen who pays taxes to support local schools and fire fighters.  But the report also attempts to highlight nonprofit contributions that are a little harder to quantify—strengthening community, improving the quality of life, stimulating reform.

I understand the need for return-on-investment arguments, but I also worry about their impact on audiences that might have a limited understanding of the civilizing effects of the nonprofit sector.

What’s the ROI, for example, on a provocative question that interrupts, if only for a moment, our relentless consumerism and reminds us of what we once aspired to become?  And where do we learn to distinguish between those cases in which metrics apply and those in which social goods are less susceptible to measurement? 

It used to be that notions like “character” and “good citizenship” figured prominently in arguments for maintaining the quality of a good public education.  Now it’s mostly about the dreary but important business of building a better workforce.

What do we lose when we attempt to convert every currency to a single coin?

Evaluation as Symbolic Capital

Evaluations too often function as ritual objects in our sector, tokens of a rite of passage: we did this thing, and it matters little what the evaluator heard or what he recorded in his report.  It matters only what we saw on our Vision Quest: our totem animal, a jaguar, running through the forest, bolting past knots of trembling stakeholders, drawing up to its prey: a program officer clucking his approval.

Some Hard Truths About Evaluation

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

Sean Stannard-Stockton, our blogger colleague at Tactical Philanthropy, poses these very difficult questions for his latest Giving Carnival:  How should social impact (or nonprofit effectiveness) be evaluated? How can we best understand the output of nonprofit or for-profit social enterprises? It is not enough to simply say that an organization is “doing good.”  How much good are they doing and how effective are they at turning “inputs” (donations and/or investment dollars) into social “outputs”?

I offer this modified version of an earlier post in reply ...

We seek a kind of scientific or moral certainty from a formal evaluation.  But it can provide neither.  The questions that funders most often bring to an evaluator—Was this program worth our $25,000 investment?  Should we continue funding it?—are questions only they can answer.  There’s simply no absolute scale against which an evaluator can measure the value of a philanthropic investment.

Here are some of the typical moves in the evaluation game:

WE:	Was this youth development program worth our $25,000 investment?
EVALUATOR:	Dunno. What are the kinds of outcomes you’d like from a youth development program?
WE:	Better grades in school.
EVALUATOR:	Hmm. Don’t see any of that. Would you settle for a lower dropout rate? I see some of that.
WE:	That sounds good. How much lower?
EVALUATOR:	I see a 15 percent lower dropout rate. Is that worth $25,000 to you?
WE:	Don’t know. Let me think about it.

The evaluator’s first job is to determine who is asking the evaluation question, and his second job is to discover what that person values.  Then and only then can he design an evaluation that attempts to detect those values.  An evaluator isn’t measuring the effects of a given program willy-nilly.  He can suggest various outcomes that we might look for from a given program, but, for the most part, he’s trying to detect the outcomes we think are worth seeing.

And so we’re at the stage where we’ve detected a 15 percent lower dropout rate for youth participating in our program, as in the example above.  Assuming it’s not an effect of how the young people were selected for the program, is the outcome—together with other secondary outcomes—worth what it cost to run the program?  We can try to compare this program with other programs we know, but they appear to be very different from one another.  Some have a mentoring component, others don’t; some include leadership training, others focus on academics.  The annual cost per student is $5,000 in one program, but $10,000 in another, but there are significant differences in the quality of program delivery.  At a certain point, the annual cost per student starts looking ridiculous—but what is that point? And is it the same for you as for me?

And so it goes.  What then is the value to society of a given nonprofit program or organization?  The simplest answer is that it’s whatever the market is willing to pay for it.

For those of us in the nonprofit sector, one of the hardest truths about evaluation is that the value to society, in dollar terms, of nonprofit work is as much determined by market forces as is the price of cheddar in Manhattan.  And we, the people who ask the evaluation questions—the funders, the donors, the board members—are that market.

Good evaluators know this.  They would gladly tell us, but often they intuit that we’d rather not know.  Frequently their role is to describe all the potentially valuable outcomes of a given program or organization and then retire from the scene, leaving us to ponder what kind of investment it might be worth.

The counsel most useful to us at that point might come from a priest or a philosopher, or, if we’re on a very tight budget, a Ouija board.