Debunking Theories of Change, Part 2 of 2

       P   O   S   T   E   D       B   Y       A   L   B   E   R  T

This post is part 2 of 2 in a series devoted to questioning the utility of theories of change in nonprofit and philanthropic work.  To review part 1 of the series, click here.  You can download a full version of the argument by clicking here.

A Convention for Representing Theories of Change

Consider the program theory for a mentorship program:

(1)   A child’s close relationship with an adult mentor --> An increase in the child’s sense of self worth --> A reduction in the child’s susceptibility to violence

Here we read the arrow symbol (-->) as “causes” or “leads to.”  A theory of change, as outlined in (1), is essentially two linked explanations.  The first explains the increase in the child’s sense of self-worth on the basis of that child’s close relationship with an adult mentor; the second explains the reduction in the child’s susceptibility to violence on the basis of his increased sense of self-worth.  In this essay, I adopt, for various reasons I describe below, the conventions used by the great American philosopher of science Carl Hempel for representing explanations in the natural sciences.  Those conventions have us represent (1) as two linked syllogisms:

(2a)	1.	If I provide child A with a close relationship with an adult mentor, then child A's sense of self-worth will increase.
	2.	My program provides child A with a close relationship with an adult mentor.
	_______________________________
	3.	Therefore, child A's sense of self-worth will increase.

 

(2b)	1.	Child A's sense of self-worth will increase.
	2.	If child A's sense of self-worth increases, his susceptibility to violence will decrease.
	_______________________________
	3.	Therefore, child A's susceptibility to violence will decrease.

These two syllogisms  are linked in the sense that the conclusion of the first syllogism (“Child A’s sense of self-worth will increase”) becomes the initial premise of the second syllogism.

The syllogism outlined in (2a) is an example of what Hempel calls a “deductive-nomological explanation.”  Its first premise (“If I provide child A with …”) has the look and feel of a general law of human psychology: it tells us something about how normally constituted individuals tend to behave under certain ideal conditions.  Compare it, for example, with the following “lawlike” statement from physics: “If at time t = 0 we drop a ball from a height h above the earth, and we drop it in a perfect vacuum, and there is nothing to obstruct its path as it falls, then it will hit the ground at time t = the square root of (2h/g).”  The second premise in (2a) has a different character.  It’s an “observation” sentence that makes assertions about certain observable facts—viz., my program provides child A with a close relationship with an adult mentor; the mentor is well trained; etc.

Identifying each link in a theory of change with a deductive-nomological explanation as defined by Hempel has several advantages.  First, it provides a common language and framework for arguing about the utility of theories of change.  No longer need we suspect—as I often have—that people who disagree about theories of change are arguing past one another, that they understand “theories of change” in very different ways.  Second, Hempel’s formalism requires that we make explicit the laws and conditions that need to obtain in order for certain effects to follow from our social interventions.  The very schematic theory of change outlined in (1), for example, does not do this. By contrast, the theory of change described in (2a) is richer: it offers the evaluator a number of avenues for exploring the possible failure of a given social intervention.  If, for example, the mentored child’s sense of self-worth doesn’t increase, we can examine the reasons for this failure in two places: either adult mentorship (even under the best conditions) doesn’t tend to increase a child’s sense of self-worth—a question that can be explored empirically; or the program we’ve designed simply doesn’t instantiate the conditions necessary for the success of an adult-child mentorship relation.*

But enough theory.  Let’s look at some real cases …

 

Putting Theories of Change to the Test

Imagine that you are the executive director of an organization whose mission is to put a park within walking distance of every home in America.  You’ve done this work successfully in your city and now you’re approaching the XYZ Foundation for funding that will help expand your work to an adjacent town.  You send in your proposal confident that the XYZ Foundation will note your track record of success and fund your expansion.  You’re surprised when a program officer from the Foundation gives you a call and asks you to produce a theory of change for your program.  “You mean like our inputs, outputs, and expected outcomes—that kind of thing?” you ask.  “No, that’s a logic model,” he says.  “I want to see an articulation of the underlying beliefs and assumptions that guide your service delivery strategy and are believed to be critical for producing change and improvement.”   “Well, uh, our beliefs and assumptions are that our program will succeed in creating a park in Neighborhood X of Town Y and that this will induce the residents of Neighborhood X to use and enjoy it.  But we told you that in our proposal,” you answer him, somewhat testily.  “Do your best,” your program officer insists.

Although you suspect your program officer is deeply in the grip of an ideology, your need for funding drives you to comply.  You begin hesitantly:

(3)	1.	Every other time we've created a nice park within walking distance of residents' homes, they have used and enjoyed it.
	2.	Our effort in Neighborhood X will succeed in creating a nice park within walking distance of residents' homes.
	_______________________________
	3.	Therefore, residents of Neighborhood X will use and enjoy it.

You send this to your program officer and after several e-mail exchanges, you agree on this emendation of your original theory of change:

(4)	1.	If we create a park within walking distance of residents’ homes, and residents participate in the design of the park, and the city commits to its upkeep, then residents will use and enjoy it.
	2.	We will create a park within walking distance of the residents of Neighborhood X, and residents will participate in the design of the park, and we will get the city to commit to its upkeep.
	_______________________________
	3.	Therefore, the residents of Neighborhood X will use the park and enjoy it.

You receive funding from the XYZ Foundation and build the park in Neighborhood X, just as you described in your proposal, and just as outlined in your theory of change (which to you still appears completely superfluous—but never mind).  A year later, when you conduct an evaluation of your program you discover, to your great surprise, that the residents of Neighborhood X are not using the park.  After polling the residents, they tell you they feel unsafe in the park because drug dealers have started using it to conduct their business.  Soon after, you hear from your program officer.  “Your theory of change was wrong,” he e-mails you.  “Please refer to the attached document for the correct theory of change.”  You open it and find this:

(5)	1.	If we create a park within walking distance of residents’ homes, and residents participate in the design of the park, and the city commits to its upkeep, and residents feel safe in the park, then residents will use and enjoy it.
	2.	We will create a park within walking distance of the residents of Neighborhood X, and residents will participate in the design of the park, and the city will commit to its upkeep, and we’ll take steps to ensure that residents will feel safe in the park.
	_______________________________
	3.	Therefore, the residents of Neighborhood X will use the park and enjoy it.

So, OK, you now convince Town Y’s police department to station a police cruiser next to the park.  One year later, same problem: residents aren’t using the park.  It turns out that because of the rising levels of violence in Neighborhood X, residents don’t even feel safe walking out of their homes, much less to a nearby park.  You wonder if you’ll get another e-mail from your program officer with yet another version of your theory of change:

(6)	1.	If we create a park within walking distance of residents’ homes, and residents participate in the design of the park, and the city commits to its upkeep, and residents feel safe in the park, and residents feel safe walking from their homes to the park, then residents will use and enjoy it.
	2.	We will create a park within walking distance of the residents of Neighborhood X, etc.
	_______________________________
	3.	Therefore, the residents of Neighborhood X will use the park and enjoy it.

There are several things to be noted about this simple example:

1. The most relevant parts of your theory of change were already implicit in your original grant proposal.
2. Connected with this last point, if your program officer can’t “get” what your theory of change is simply by reading your proposal, he or she might simply be in the wrong business.
3. Even in a very simple example like this one, articulating the “underlying beliefs and assumptions that guide your service delivery strategy and are believed to be critical for producing change and improvement” can get ridiculously complicated.  In general, you cannot know ahead of time what the “relevant” beliefs and assumptions will be.
4. Your theory of change is not working for you: you are working for it.  It’s like a dead albatross around your neck that you occasionally remove and dip in formaldehyde to keep your funders happy.

Our problems are compounded when we look at more complicated examples.  In her article, “How Can Theory-Based Evaluation Make Greater Headway?”** Carol Weiss describes a program that gives tenants representation on the management board of their public housing project:

This program espouses multiple goals, such as improving the maintenance of the property, decreasing crime in and around the housing project, and encouraging residents to take greater responsibility for their lives in such areas as work and education.  These are ambitious goals and, to some, inflated hopes, but let us take the most realistic objective, improved maintenance of the project … (p. 503)

What’s interesting about the description that Ms. Weiss gives of this project is that many of us immediately share her intuition about which of  its goals is the most realistic.  Even without seeing a detailed proposal, or a theory of change, we can imagine ways that tenant representation on the management board of a public housing project might lead to improved maintenance of the property.  Ms. Weiss points out that there are multiple causal paths that can lead to this goal, and she sketches a program theory that I reproduce below as Figure 1:

 
Figure 1

What’s striking about this theory of change is its overwhelming complexity—and this in spite of the fact that it doesn’t begin to articulate all the underlying beliefs and assumptions that might be critical to the success of the project—e.g., How much power do residents wield on the management board compared with non-resident managers?  How constrained is the board by a lack of funding?  How much time can residents reasonably dedicate to the management of a complex property? etc.  Weiss argues that:

The small everyday steps in Figure 1 can be couched in social science language, such as sense of communal responsibility, access to authority, local knowledge, bureaucratic responsiveness, and so on.  In the conduct of the evaluation, the evaluator will need to operationalize these concepts and develop specific indicators that capture their essential meaning. (p. 506)

This sounds like an enormous undertaking for what starts out as a fairly simple project.  If, after a year, the condition of the property doesn’t improve, is it really practical to ask our evaluator to do as Weiss suggests?  How much would such an evaluation cost?  Would he or she test each of the branches in Figure 1?  And just as with our previous example, we can imagine a thousand reasons, not pictured in Figure 1, why the program might have failed.  How, in that case, would the theory of change we’ve so carefully crafted guide our evaluation efforts?

By the way, we can also imagine scenarios in which the property improves but for none of the reasons outlined in Figure 1!  One of the resident managers, for example, might have a brother-in-law whose contracting company does excellent work for half the price; or one of the resident-managers might be very successful at organizing resident work crews who dedicate one Saturday each month to repairs and upkeep; or the very presence of residents on the management board might inspire a local donor to endow and thereby double the public housing project’s maintenance budget; etc.  Real life is extremely complicated.  When human actors are involved, its twists and turns are almost always unpredictable.

It takes enormous effort to dream up a program theory like that pictured in Figure 1.  And as we’ve seen, it can easily turn out to be a completely fiction.  At this point we need to ask, whom does it serve and to what end?

Adding It All Up

To summarize:

1. An airtight, explicit theory of change, even for the simplest intervention, might be beyond the powers of any mortal known.  If you’re a funder, think twice before you require it.
2. What typically passes for a theory of change is a narrative of how certain causes can plausibly lead to certain effects.  If this is all you’re really after, then read the proposal.  Everything you want to know is likely already there.
3. Because a theory of change is more like a narrative than an explanation in the social sciences, it will be more compelling if it’s written by a good story teller on your staff.  This moves you from the domain of sound scientific practice to the realm of compelling literature. This raises for me serious questions about the weight we should assign to these narratives in grantmaking work.
4. Funders who do not require highly articulated, explicit theories of change from their grantees are not treating them as idiots savants.  They’re treating them as fully rational human beings who have had enough experience of the world to know which effects generally follow from which causes.

I am not here denying that theories of change, and the process of constructing them, have helped many people in the nonprofit world.  But while I myself have found it enormously useful to discuss my plans with my Jack Russell Terrier, I would never dream of requiring this of my grantees.  There are myriad ways that organizations can fruitfully plan their social interventions.  All will likely involve some anticipation of hoped-for effects, and most, I assume, will fall far short of fully developed theories of change.  Some interventions will be designed entirely on the basis of intuitions or gut feelings.  While this is not a style of program planning I practice or endorse, I’ve also not yet seen any evidence that its failure rate is higher than that of allegedly more “scientific” approaches to planning social interventions.

The acid test for me is this: Take a close look at your theory of change:  Does it tell you anything beyond what you’ve already included in your grant proposal?  Does it suggest to you ways that you might improve your work?  Given all the factors we’ve considered in this article, are you confident of its accuracy?  Is it absolutely required by your funders?  No?  Then, borrowing a suggestion from David Hume, commit it to the flames!  For it can do nothing but confound you and waste untold hours of your time.

-----

* Hempel’s formalism certainly has its limits, especially when one uses it to describe change processes that are not strictly linear.  And as Hempel himself acknowledged, special consideration must be given to so-called “probabilistic explanations.”  But these complexities only serve to underscore my point about the questionable utility of theories of change in nonprofit work.

** Weiss, Carol 1997. “How Can Theory-Based Evaluation Make Greater Headway?” Evaluation Review, Vol. 21 No. 4.



Comments

[Conches]

A very interesting screed. :) I am definitely a proponent of a theory of change but not in the way that you describe it as a stick wielded by a funder. First, I don't really distinguish a ToC from a logic model and second, a theory of change is a way for me to keep the "gravity" of my organization positioned correctly. A well implemented Theory of Change is what the staff and leadership of a NPO can use to manage their tasks to check that they are spending their time using inputs to create outputs that have strong logical connections to the change the organization wants to see. Similarly, it is a good way to check any proposals for new activity. Does the new activity use valuable inputs to create outputs that have strong logical connections to the change the organization wants to see? I see a ToC as very tactical and very iterative. I am very sympathetic to your argument that a ToC should not be over-complicated. A ToC should make an NPO both more effective AND more efficient. The efficient part is the tough one and it is what brought me to this work in the first place. In the process of helping NPO's to advance their missions I have found that one of the most effective tools is CRM or more generically, data and process management. If an organization can successfully facilitate the tasks that are most successful at advancing their organization then they can 1) gain efficiencies through automation, 2) gain efficacy through more, better, faster, 3) gain insight in to their impact because the system will incidentally be collecting data on the tasks that effective move the needle.

[Albert]

Thanks for your message, Conches. While you don't distinguish a theory of change from a logic model, many people in the field do—see, for example, Evaluation: A Systematic Approach, 7th edition, by Rossi et al., or the GrantCraft publication referenced earlier. That said, it's not a difficult thing to read or infer a theory of change (in fact, many theories of change) from a logic model. Your use of the words "inputs" and "outputs" makes me think you're most likely using something like a logic model in your work.

Would you mind sharing your logic model or theory of change by e-mailing it to courtesy_telephone (at) yahoo.com? I would like to bettter understand how it made your work more effective and efficient. This has not generally been the experience of the colleagues I've polled, so yours would be an interesting case study.