Debunking Theories of Change, Part 1 of 2

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

Editor’s note: A full version of the argument presented in this post is available by clicking here.

In the Spring 2010 issue of the Stanford Social Innovation Review, Paul Brest, President of the William and Flora Hewlett Foundation, took several of my colleagues* to the conceptual woodshed for their skepticism about “theories of change.”  As Mr. Brest and others understand them, theories of change include our beliefs about the causal paths that lead from our social interventions to our hoped-for effects.  These beliefs naturally shape funders’ approaches to social change. Grantmakers will not, for example, fund programs that attempt to reduce violence by requiring gang members to wear Styrofoam shoes, for this intervention would not be supported by any plausible theory of change.

If we agree with Mr. Brest that a theory of change includes one’s beliefs about the way the world will behave given this or that intervention, we might wonder what there is to be skeptical about.  Do the skeptics deny that grantees have beliefs about the effects of their social interventions?  At the very least, Mr. Brest is concerned that these skeptics feel it’s unimportant to require grantees to articulate their theories of change:

Schambra and Somerville argue that funders should focus their grantmaking on community-based organizations, and that funders should trust an organization’s leaders without requiring them to articulate theories of change ….  These skeptics are implicitly analogizing grantees to idiots savants—individuals who are able to do complex calculations … in their heads without knowing, let alone being able to explain, how they do it.  [p. 48, emphasis mine]

It should be noted that one of the interesting characteristics of idiots savants is that the complex calculations they perform in their heads are often correct.  If the nonprofit world harbored the analogues of idiots savants—executive directors, for example, who could correctly intuit the interventions required for lifting whole communities out of poverty—I would strongly encourage them rather than force them to give an account of their methods.  Despite their inability to explain how they arrived at their program designs, I would prefer that we went about changing the world the way they did, rather than the way most of us didn’t.

Mr. Brest’s claim that the skeptics alluded to in his article are analogizing grantees to idiots savants also needs to be challenged.  An idiot savant cannot explain to you how he is able, in under a second, to calculate the square root of a 55-digit number.  A talented nonprofit executive director, by contrast, can draw on a lifetime of experience and whatever research is available to him to give an account of the design of his social intervention.

In many cases, of course, nonprofit executive directors are working from implicit theories of change.  Since these implicit theories of change are ubiquitous, there’s a clear—but vacuous—sense in which theories of change are indeed as “powerful” as Mr. Brest claims.**  But this is not what’s at issue between him and the skeptics he alludes to in his article.  In writing about the evaluation of social interventions, for example, Mr. Brest claims that “Identifying the proper variables and metrics requires that one articulate the program’s goals and its underlying theory of change” (p. 48).  Even more forcefully than this, “Improving the lives of disadvantaged populations ... requires proven theories of change” (p. 47, emphases mine).

Should we really be requiring grantees to articulate their theories of change?  The simple answer, as I argue below, is that there’s little need for this requirement since most grantees already do a sufficiently good job of describing “the empirical basis underlying their intervention,” as Mr. Brest puts it.  They do this when they write their grant proposals.  The more surprising answer, perhaps, is that requiring grantees to produce explicit theories of change—beyond what they usually include in their grant proposals—does little to improve the art or science of grantmaking.

Highly elaborated theories of change are generally urged upon grantees by well-meaning people who have a limited understanding of how they function in the social sciences.  Because theories of change are often shrouded in the impenetrable verbiage of philanthropy, it’s also not surprising that most of us have little inkling of their theoretical and practical limits.

Speaking of practical limits, do you, dear reader, know with unerring certainty what a theory of change is?  If you do not, you are entirely forgiven and in very good company.  Take a little time to review the literature on theories of change and you’ll discover, as I did, that theories of change are everything and they are nothing.  Unfortunately, as I argue below, the “power” that Mr. Brest ascribes to them is largely the power to waste enormous amounts of your time.

The Radical Polysemy of “Theories of Change”

The term “theory of change” is used widely in philanthropy and other fields to describe a family of concepts whose members, upon close inspection, vary significantly in meaning.  “Theory of change” is a term of obscure origin, as far I can tell, associated in the minds of many with the culture and language of formal evaluation.  A grantmaker or grantee scanning the available literature on theories of change can hardly help but be confused by the exercise.  The GrantCraft publication titled Mapping Change, for example, opens by telling the reader not what a theory of change is, but rather what it does:

A theory of change describes a process of planned social change, from the assumptions that guide its design to the long-term goals it seeks to achieve. 

From this the reader might conclude that a theory of change is the narrative of some social change process with goals appended.  But would any such narrative qualify?  After all, “describing a process of planned social change” can mean writing a rambling manifesto, penning the succession plan for a drug cartel, or developing marketing guidelines for a smokeless ashtray.   The authors acknowledge this ambiguity when, several paragraphs into their article, they add:

What does “theory of change” really mean, in practice?  Grant makers who use the term may be describing anything from a detailed map to a general storyline.  What they agree on is that a theory of change is valuable if it helps them and their grantees understand the relationship between the problems they’re addressing and the strategies they’re using to get the work done.

At this stage, the careful reader might have some inkling of what a theory of change does and why it might be useful.  He might still have little idea what it is.  If the essential mark of a theory of change is that it helps us better understand how our efforts do—or do not—lead to our goals, then it’s possible that a glass of Merlot can qualify as a theory of change, for as Heraclitus reminds us, “It is better to hide ignorance, but it is hard to do this when we relax over wine.”

On one popular website the authors tell us that a theory of change “defines all the building blocks required to bring about a given long-term goal,” while on another we’re informed that it’s “the product of a series of critical-thinking exercises that provides a comprehensive picture of the early—and intermediate—term changes in a given community that are needed to reach a long-term goal articulated by the community.” In some publications they’re equated with so-called “logic models,” in others they’re not. 

One of the claims frequently made for theories of change is that they’re essential for helping us evaluate the effects of our social interventions.  As Mr. Brest himself informs us, “most evaluations of social interventions require understanding the theories that mediate between inputs, activities, and outcomes.” (p. 48)  Interestingly, many of the formal evaluation textbooks I researched didn’t use the term “theory of change” at all.  The closest cognates I found were the terms “program impact theory,” used, for example, in the 7th edition of Evaluation: A Systematic Approach by Peter H. Rossi et al., a popular textbook on evaluation; and the more common “program theory,” used in a significant corpus of books and papers by such authors as Carol Weiss and others.  Generally, these scholars understand a program theory to be a sequence of causes and effects, in which our interventions function as the instigating causes and certain hoped-for social benefits function as their ultimate effects.

To take a very simple-minded example, suppose we diagram the program theory for a mentorship program as follows:

(1)   A child’s close relationship with an adult mentor --> An increase in the child’s sense of self worth --> A reduction in the child’s susceptibility to violence

Here we might read the arrow symbol (-->) as “causes” or “leads to.”  But scan the theories of change produced by professional and amateur evaluators and you’ll find a significant range of choices regarding which types of causal connections to include (those drawn from individual belief-desire psychology, from the psychology of social movements, from rational actor theory, etc.); the necessary conditions one should make explicit (an inclination on the part of subjects to cooperate, a robust economy, favorable weather; etc.); and what unit of analysis to focus on (the individual, the family, the neighborhood)—among many other variations.

Given the general lack of consensus about what a theory of change is or what it should include, it’s not surprising that the skeptics alluded to in Mr. Brest’s article might fail to agree over its utility.  If we are ever to make any progress in our debates about theories of change, we will need a common language for describing and evaluating them. 

Coming up next: A definition, some case studies, and some damning conclusions

-----

* Bill Schambra, Bill Somerville, Sean Stannard-Stockton, and Jeff Berndt.

** Mr. Brest's article is titled "The Power of Theories of Chnage."

Comments

[erasmus]

I was hoping somewhere, someday, someone would work Zippy the Pinhead into a post.

[Albert]

Zippy, as most fans know, wears styrofoam shoes, and is rarely violent. Coincidence? Unlikely: Wearing styrofoam shoes --> Laughter, in which the wearer gets caught up --> A feeling of well-being --> Decreased violence.

[Mary Ellen Caoek]

My theory of change is that two dimes and nickel = a quarter...at least in the US

[Enrique the Gay Philosopher]

Do the two dimes and the nickel need to be co-located? or can the nickel be in New Orleans, say, and the dimes in Warsaw? You want to be able to say, "Here's a quarter," pointing to the nickel in your hand, without also referring obliquely to the dimes' alleged location in Poland and apologizing for their absence.

I'm troubled by these lingering questions regarding your theory of change. You clearly haven't paid enough for it.

[a careless reader]

"an idiot savant cannot explain to you how he is able, in under a second, to calculate the square root of a 55-digit number."

Hmmm, the idiot savant explains that he sees all numbers as gradations of colors, and he is able to arrange those gradations in a manner that allows him to produce the correct result. This is an explanation. An explanation which accompanies a successful result.

Also, you propose 'theories of change' are 'narratives' and therefore lack scientific rigor. This sounds postmodern to me. Scientists don't do narrative?

[Albert]

Sure. The idiot savant can provide an explanation of her methods, perhaps one that only other idiots savants can understand. An executive director, even one who uses her "gut" to design her social interventions, can appeal to instincts that are (almost) universally shared. As for the question of narratives: yes, point well taken: all is narrative. But I would argue that there are distinctions we can and should make between certain kinds of scientific narrative and chapter 3 of Great Expectations. My greatest worry about treating all narratives equally in this postmodern age is that this gives an enormous advantage to the silver-tongued--the Jerry Falwells, the Sarah Palins, the Glen Becks. I'd rather err on the side of reductionism than on the side of those who subvert Truth to ideology.

[Daniel Ben-Horin]

This is a great piece. Thanks, Albert

[Albert]

Thanks, much appreciated.

[a careless reader]

"As for the question of narratives: yes, point well taken: all is narrative."

No, no, that wasn't the point at all.

More like, ok:

1. Postmodernism "prime" as it were is, in Lyotard's formulation, "incredulity towards meta-narratives." This is a philosophical skepticism.

2. Pop postmodernism is "all is narrative." This is a moral cynicism.

Disparaging the role of narrative within science, as narrative, because it is narrative, bears the stain of pop postmodernism because it's a reaction to it.

It is perfectly possible to approach narrative *within* science without referencing postmodernism at all, and without displacing its rightful role.

[Albert]

Points well taken, careless reader, insofar as I'm able to understand them. While I agree that the claim "all is narrative" might be wielded like a blunt weapon, I believe there's an important insight there, one I attribute to postmodernist theorists who, like Barthes, urged that we counter the notion of a "work" with a broader understanding of what a "text" can be (or rather, is). I didn't mean to sound dismissive by agreeing that all is narrative. As for narratives "within" science, you're right. These were studied and discussed long before Derrida soiled his first diaper.

Many social sciences lack the rigorous modes of explanation available to physics, say, or genetics, because there are few general laws that social scientists can appeal to. Theories of change, because they so often involve predicting the actions of human actors, inherit the weaknesses of explanations in the social sciences (the weaknesses of folk belief-desire psychology, say, or of rational actor theory). They rely for their strength on the plausibility of their cause-effect connections, and this plausibility, I would argue, has more to do with artfulness, with the ability to spin a good yarn, than with any general laws of human psychology we might invoke that would lead us from one link to another in the theory of change. This is not a bad thing, it's simply a far cry from what I believe Mr. Brest expects of his grantees.

[a careless reader]

Hmmm, well hard science can't "explain" things that haven't happened yet, either, but certainly has predictive value in projecting what will happen when a cue ball hits an eightball. Social science is stuck with having to delve into the uncertain result of an eightball hitting a human. But it certainly is of value in being able to describe the result of past instances of human/billiard ball interactions.

What is really important is it seems that by denigrating the the role of "theory" you seem to think you are somehow avoiding ideological complications, while at the same time your descriptions of "implicit theories of change", "gut feelings", and "intuitions" are all plausible synonyms for the "false consciousness" model of ideology.

In which case, it's easy to understand why the likes of Schambra would reject the Theory of Change crap.

[Albert]

I'm getting sloppy here. I started talking about explanation in physics and then, as you pointed out, I shifted to discussing our ability to predict human actions. I guess I have Hempel on the brain. He argued that the logical form of an explanation is essentially the same as that of a prediction. A prediction to Hempel was simply an explanation that used the future tense.

I do wonder, now that you bring it up, about those domains in which we restrict our use of intuition. Dentistry, certainly; belly dancing, not so much. The handlers of politicians running for national office worry when they go off script and use their gut feelings to address a crowd. Intuition in that case can drive a person toward a more genuine kind of expression.