Have you ever wanted to get a score for how creative you really are? How about a rating for how susceptible you are to hypnosis?
Believe it or not, scientists have actually developed ways to measure both.1,2,3,4 These are just a few interesting examples of a wider effort across psychology, business, sociology, and other disciplines to quantify nebulous things.
For nonprofits, measuring complex outcomes isn’t just an academic exercise—it’s vital to their missions. They need to prove that their programs are working and that donor dollars are being put to good use. But how does one even begin to quantify something as subjective as creativity or the broader concept of social impact? Is it really possible to assign a numerical value to something as nuanced as improving someone’s quality of life?
Here at The Decision Lab, evidence-based social impact is at the core of our work. This article will dive into a few well-crafted approaches to measuring social impact, demonstrating how rigorous methodology and creative thinking make it possible to assess the impact of difficult-to-measure outcomes.
Calibrating the scales
If you’re measuring impact to compare the efficacy of different programs, you need to get down to the nitty-gritty while still maintaining a broad scope—and the scale can vary widely depending on what it is you want to measure. For instance, imagine you’re trying to measure the impact of a program training surgeons to operate on tumors. You could start by counting the number of surgeons trained, but the number of procedures they actually perform could more meaningfully demonstrate the impact your program has on treating cancer. To go a step further, you might look at the actual success rate of the surgeries and whether surgeons trained in your program performed better than the average surgeon trained in a standard program.
Zooming out, you could track improvements in patients’ quality of life, the money saved on medical expenses, and even the economic benefits of people who can return to the workforce because of this surgery.5 Choosing the right scale for the impact you want to measure is a crucial step.
The Trolley Problem rears its ugly head
Logistical concerns about measuring such things aside, measuring social impact can raise ethical questions. How do we define the relative value of different outcomes? How much more or less significant is the impact of increasing someone’s income compared to saving their life? By what numerical factor would we value saving a young person’s life versus an old person’s life?
It can feel unsettling to try and answer these questions. For nonprofits working in global health, there’s a fear of losing nuance or some essential human element when reducing impact to cold, hard numbers. Yet, without quantifying impact, how can we ensure that resources go where they’ll do the most good?
Of course, the answers to the questions outlined above will vary widely from person to person, and we want to trust that those evaluating impact would make similar judgments to ours. GiveWell, a nonprofit evaluator, approaches this issue with as much transparency as possible. They provide detailed explanations of the “moral weights” they apply to different outcomes—such as averting the death of a single child under five versus improving consumption for a hundred people.6,7 Moral weights are a crucial component of how they measure the comparative cost-effectiveness of different programs and thus consistently rank various charities and allocate funds.
How much does it cost to save a life?
Meaningful comparison of program efficacy also requires a pragmatic evaluation of costs. In the case of programs relating to healthcare, an important metric is how much money is spent per life saved.
Let’s follow an example calculation from GiveWell.8 The cost of one dose of preventative malaria treatment is just $1.37. But, not everyone who receives the treatment is guaranteed full protection. So, in this circumstance, how much does it cost to save one life?
Let’s say a donor contributes $3,000. At $1.37 per monthly dose, that’s around 2,000 monthly treatments. At four monthly treatments per child, about 400 children are treated. Population-wide data in Nigeria suggests that about four of these 400 children would contract and pass from malaria in a four-month period and that the treatment is efficacious enough to save one of those four children.
So, that $3,000 donation saves one life through this program. This may be surprising given the low cost of treatment, and it may be even more surprising to know that this is an extremely cost-effective program by GiveWell’s (very high) standards.
Understanding and applying this calculation has a huge influence on donor decision-making and funding priorities. This cost breakdown provides a powerful, unified metric that readily dispels any fears about where money might really go when a donation is made while also providing charities with a tool to showcase the efficiency and effectiveness of their work.
behavior change 101
Start your behavior change journey at the right place
Credit where credit is due
Another consideration for organizations involved in program design and execution is how much impact they can take credit for. If you work for a consulting organization that advises a non-profit, how can you quantify your contribution to the impact that was achieved through their implementation?
IDinsight, a social impact consultancy, uses the following formula to quantify this:
Impact Contribution = Depth of impact x Breadth of impact x IDinsight’s contribution to partner’s impact.9
- Breadth is the number of people affected by the program
- Depth (measured from 0-4) is the magnitude and longevity of the improvement
- Contribution (measured from 0-3) reflects how much IDinsight’s recommendations influenced the partner’s decisions and how evidence-based the recommendations were
Similarly, Global Innovation Fund uses the formula “Depth of impact x Breadth of impact x Probability of success”, where the final component is their calculation of the likelihood that the innovation will still succeed after 10 years.10,11
These relatively simple formulas effectively capture the broad strokes of impact and apply to many fields of study. It is a good starting place for any organization looking to take a holistic view of the impact they have helped create in a methodological way.
Your turn!
For any readers thinking of how to measure their own impact, there are a few guiding principles to keep in mind.
Accepting limitations
Any modeling system is going to be an imperfect representation of reality. This means that limitations are unavoidable and should be acknowledged outright. Awareness of this will allow anyone who consumes the output to mentally adjust for biases or gaps.
Transparency
Having documentation of how you choose to measure impact is vital. This should be publicly available and accessible to a general audience. This can increase confidence in your methodology and trust in your organization as an entity.
Balancing scalability and precision
Creating a framework that is broad enough to apply unilaterally across the sectors you work in while remaining nuanced enough to capture reality is tricky. Applying some of the techniques described in this article may help you create a measuring instrument that is simultaneously robust and delicate.
At TDL, social impact is at the heart of everything we do. In the past, we’ve helped organizations like Care Predictor use their own measurement of social impact (in their case, a therapist's propensity to form good relationships with their clients) to ensure their measures are both rigorous and actionable.
Whether you're measuring creativity, hypnosis susceptibility, or the impact of life-saving interventions, thoughtful design can help turn the unmeasurable into something tangible and translate human efforts into data that speaks.
