Posted on

The Cost of Confidence: Finding Evidence of Impact for Social Enterprises

seven levels of evidence pyramid

If you’re planning to launch a new social enterprise, or scaling a successful venture, you may be asked by investors and funders to demonstrate impact. But with product development and customer acquisition to worry about, you may not have the time or experience to track down the latest rigorously peer-reviewed study that proves your value proposition will work. How can you be informed by impact evidence while building something innovative and new?

The 7 Levels of Evidence

It starts with an understanding of what “evidence” means. Evidence is defined as “the available body of facts or information indicating whether a belief or proposition is true or valid.” Research and evaluation use a simple framework to rank the rigor of studies, corresponding to higher and higher levels of causality, or proof of impact.

The seven common levels of evidence are understood as:

  • Expert Opinion
  • Evidence from Single Study
  • Systematic Review/Lit Review of Similar Studies
  • Case-Control or Cohort Study
  • Quasi-Experimental (using a similar matched population as the control group)
  • Randomized Controlled Trials (known as the “gold standard” of evidence)
  • Meta-Analysis (weighted averages of many similar randomized controlled trials)

As you move up the ladder, there is usually a greater quality of evidence of causality and greater confidence in the key findings, due to greater scientific rigor and larger sample sizes.

The Cost of Confidence

Higher levels of evidence also tend to be associated with higher costs. Conducting a randomized controlled trial or extensive study can be expensive and time-consuming. That’s why it’s important to determine which level of evidence makes sense for your social enterprise, your budget, and where you are in the funding journey.

For a new idea, you probably don’t need to be 95% confident in your predicted outcome. If you can project your impact assumptions with a lower level of evidence, you can begin to shop your idea sooner and see the direct results. If you’re waiting for the highest level of evidence, you might end up taking months or even years to launch your program, while spending significant money on research.

We’re calling this idea “the cost of confidence.” Getting to 95% confidence in an intervention might cost twice as much as getting to 70% confidence. That cost is often a barrier to new ideas launching, particularly in pay-for-success-type programs where the taxpayer funding may require a higher level of confidence of causality. The burden of proof ends up taking too much time and money, and the program never gets off the ground.

The Right Level of Evidence at the Right Time

To address that problem, we advocate for using the levels of evidence “out front” to indicate where you’ve found sources, and what that indicates as a research baseline.

When considering the cost of confidence, meta-analysis could be the best benchmark for many social enterprises. Looking at meta-analyses of multiple studies not only results in a higher level of confidence — because it shows that results can be repeated under different circumstances — but it also tends to be quicker and less expensive. It bypasses the extensive costs of putting together a randomized controlled trial or other rigorous study, and offers strong evidence that an intervention will work.

There are more and more clearinghouses or sources of meta-analytic studies that are being used to inform public policy and grantmaking. These are a great place to start for social entrepreneurs.

Even if a meta-analysis is used for evidence up front, a randomized controlled trial may still be needed to show evidence of impact after launch. The level of evidence required often depends on the stage of the enterprise: Funders’ expectations and investments may increase as higher levels of evidence are possible. However, simply getting off the ground and attracting initial funding shouldn’t require a prohibitive cost of confidence.

As evaluation and data become more central to investments, public funding, and other decision-making, we think it’s important for social entrepreneurs to have a basic understanding of the cost of confidence and levels of impact evidence.

Further reading

Get the basics:

Evidence Based Practice Toolkit, Winona State University: http://libguides.winona.edu/c.php?g=11614&p=61584

Dig deeper:

More insights about the cost of confidence and the right time to evaluate impact in “Ten Reasons Not to Measure Impact — and What to Do Instead,” Mary Kay Gugerty & Dean Karlan, Stanford Social Innovation Review: https://ssir.org/articles/entry/ten_reasons_not_to_measure_impact_and_what_to_do_instead

Posted on

Using Stakeholder Theory to Understand Impact and Strategy

stakeholder map

The concept of stakeholders has become common across sectors as a contrast to the traditional, profit-driven “shareholder” view of organizations. Considering your stakeholders allows you to think more broadly and critically about impact and social responsibility.

More specifically, you can’t measure the outcomes of your work without understanding who that work affects. Accounting for impact requires identifying stakeholders.

You can start to pinpoint your organization’s stakeholders with a simple mapping exercise. First, draw a hub-and-spoke diagram with your organization at the center. Then think about the people and systems closest to the center, the ones most directly affected by what your organization does. Those probably include customers and/or beneficiaries of your programs, as well as employees and partners. Think about your organization’s mission: Who are the people you serve?

Now take a step further out from the center. How does your work impact the local communities where you operate, and which people and systems are directly affected? How does your work impact the environment, both locally (through your direct physical footprint) and globally (through choices such as your use of fossil fuels and where you source materials)? Be as specific as you can, and remember to consider unintended and indirect consequences of your operations.

Whether or not you’ve drawn out your hub-and-spoke map, you likely have thought about many of these stakeholders and how your organization affects them. The next step is deciding how your awareness of stakeholders will inform your business model — in other words, going from “Who are they?” to “How does this affect our strategy?”

For this next step beyond the hub-and-spoke map, you can create what is known as a materiality matrix. This matrix can help you determine how your stakeholders’ needs and wants intersect with your business goals and activities. On the matrix, you will map out issues and challenges that are relevant to you as well as your stakeholders. One axis of the matrix represents the importance to your organization. The other axis represents importance to your stakeholders. You can map out economic, social, and environmental issues on these axes, and identify areas that are of greatest importance to both you and your stakeholders.

Now imagine a regression line drawn through the issues you’ve mapped — where do the stakeholder and organizational goals line up? Are your priorities aligned with your stakeholders? Do you need to change your strategy, or can you safely set aside some of your stakeholders’ priorities as being outside the scope of your organization? Part of the goal of social enterprise is to align these priorities — to create shared value for a business as well as the environment and the communities where it operates.

Of course, you’ll want to test your assumptions about stakeholders through research: talking directly with people affected by your work; finding government and other data about stakeholder groups; and looking for previous studies that are relevant to your programs.

How do you think about your stakeholders? If you’ve used a hub-and-spoke stakeholder map and/or a materiality matrix to think about stakeholders, did anything surprise you or change your strategic approach?

Further reading

Get the basics:

Dig deeper:

Posted on

Using Logic Models to Predict Risk

logic models risk

If you’re starting a social enterprise or nonprofit, you’ll naturally focus on the impact you plan to have. You’ll predict the effects of your work based on research and past successes in your field, and you’ll imagine the positive outcomes that motivate you to succeed.

You’ll also, however, have to anticipate the potential negative or unexpected outcomes, and predict your ability to successfully execute the business plan and deliver your intended value. You’ll need to account for risk.

Failure, unforeseen circumstances, and unintended consequences are possible in every enterprise. Acknowledging and measuring those risks — both the likelihood that they will happen, and the impact they could have — allows entrepreneurs and investors to plan ahead more safely.

Entrepreneurs aiming for social and environmental impact need to consider the possibility that their work will have unintended or insufficient impact on the planet, on specific groups of people, and/or on society as a whole.

The Impact Management Project has a great resource on impact risk, including a graphic illustrating all of the ways risk can show up in an enterprise.

Graphic created by the Impact Management Project.

As you can see, the types and dimensions of risk break it down much more specifically than just saying, “Something could go wrong” (or naively saying, “We’re going to absolutely crush this”).

For example, an ice cream shop that plans to hire and train people with barriers to employment could face typical business risks — maybe there’s an unusually cool summer that drives down ice cream sales, or a mold problem that forces them to relocate.

But a social enterprise also faces specific risks related to their intended impact. The shop faces stakeholder participation risk if they put together a training curriculum without consulting people affected by the problems they’re trying to solve, resulting in a program that doesn’t effectively meet their employees’ needs. They face unexpected impact risk if they fail to account for the heavy electricity usage and carbon footprint of their industrial freezers and other equipment, leading to a negative environmental impact.

By anticipating those potential outcomes, whether they are unconsidered social or environmental externalities or consequences of the ice cream shop’s operations, the entrepreneurs can proactively minimize avoidable risk and become resilient in the face of unavoidable risk.

Social enterprises can look for and identify those specific types of risk as part of creating their logic models, which we discussed in our last post. As a social entrepreneur maps out their plan for using specific activities to achieve specific impacts, they can pinpoint risk at each stage and show how they will protect against it.

One of Ecotone’s goals is to help our clients recognize and account for risk as part of their business models, so that they can strategically minimize or offset it.

Further reading:

Posted on

Designing Your Impact: Logic Models for Entrepreneurs and Investors

network model

When social entrepreneurs launch businesses with the goal of creating impact, they have to determine how their business model connects to their mission. They must decide what the business does on a daily basis, how the business makes money, and how the operations and cost structure relate to the impact.

An organization’s logic model describes its intended impact, along with the intervention the organization’s leaders have chosen to achieve that intended impact. Entrepreneurs often use tools such as the Business Model Canvas to map out their activities, value proposition, and target audiences. However, an understanding of how costs and impact value are created is best developed using what is known as a logic model.

Logic models have traditionally been used by nonprofits and international aid organizations, but the convergence of social impact and business has seen them used by organizations such as the Impact Management Project to organize the costs and activities that make up a social enterprise.

Logic models lay out:

  • Inputs — the infrastructure, materials, and labor needed to make the organization run
  • Outputs — the products and/or services the organization creates, as well as the externalities, or side effects and unintended products of their work
  • Outcomes — the results the organization produces, from long-term stability for its employees to increased traffic and economic growth for its neighborhood

For example, a new organization might determine that their mission is to help people with developmental disabilities become independent. They look up research on the most important skills to teach people with disabilities, and review outcomes from similar programs around the country and the world. To achieve their mission, they choose as their social enterprise an ice cream shop, hiring and offering job skills training to adults with disabilities.

And now for one of Ted’s food service analogies…

In many ways, this ice cream shop is very similar to a conventional for-profit business. They rent a space that meets the needs of an ice cream shop and market to the public. They purchase large equipment such as mixers and freezers, and buy supplies including ingredients and packaging. They handle cash and follow health code requirements. They serve delicious ice cream to customers.

However, there are a few unique expenses that the social enterprise ice cream shop has to account for. Those include the cost of training materials, staff time and gas money to give employees rides home or to appointments, and other concerns specific to the population that the business serves.

One of the goals when creating a logic model — and our goal at Ecotone when we help social enterprises develop their logic models — is to account for those extra expenses and find sustainable ways to fund them. Laying out inputs, outputs, and outcomes in one blueprint allows organizations to attribute their impact to specific inputs. That, in turn, allows them to look for ways to fund those inputs so they can continue growing their impact.

Further reading

Get the basics:

Dig deeper:

Posted on

Defining Your Impact with the UN SDGs: Using the UN Sustainable Development Goals to Classify the Impacts of your Social Enterprise

You’re launching a social enterprise. You run a high-impact non-profit. Or you work at a for-profit corporation, but want to think more critically and strategically about proactive, positive impact.

You want to do “good” — have a positive impact and effect change. But how do you define what “good” means? How do you communicate the positive impact of your organization to investors that are aligned with your cause?

This set of 17 targets, also known as “the global goals” or just “the SDGs,” was agreed upon and adopted in 2015 as part of the United Nations’ 2030 Agenda for Sustainable Development. The SDGs offer a high-level categorization of impact, and outline the key targets and measures identified for investment in each area to achieve the global goals. Because it’s an internationally agreed-upon list, it’s one of the closest things we have as a society to a consensus definition of areas of positive impact and need.

At Ecotone, like many organizations and institutions around the world, we use the Sustainable Development Goals as a framework to help our clients classify their areas of impact. For example, let’s say there’s a bakery that offers employment and job skills training for formerly incarcerated people. That supports SDG 8, Decent Work and Economic Growth, and SDG 16, Peace, Justice and Strong Institutions. Tying their work to those SDGs helps the bakery quickly communicate their goals to the public, their customers, and potential investors. An investor or grantmaker who wants to invest in peace and justice can instantly identify the bakery as being aligned with their mission.

As the SDGs continue to be used by organizations around the world, we’d love to see it become a more widely understood shorthand for impact. That bakery hiring formerly incarcerated people could display the icons for SDGs 8 and 16 in their front window and on their website — almost like medieval shields displaying the icons of each clan.

Then, when that bakery is ready to measure the impact they’re having, they can refer to the SDGs to guide which metrics they should be paying attention to. Rather than saying, “We’re a nonprofit, of course we’re doing good,” organizations can — and should — provide evidence that their work is achieving progress in an area determined to be important to society as a whole. Using the SDGs to categorize social good is the first step to measuring and accounting for impact, and to using the evidence of impact to design programs and business models.

Our work to help organizations account for their impact — and create business models to fund, scale, and sustain that impact — starts with connecting their work to a global definition of good.

Have you thought about how your work connects to and fulfills the Sustainable Development Goals? Which of the 17 goals are most relevant for your organization?

Further reading

Get the basics:

Dig deeper:

SDGs usage:

There has been some misunderstanding about using them for fundraising. This is directly from the usage guidelines:

“The SDG logo and 17 SDG icons may be used for both informational (primarily illustrative) and fundraising purposes.”

“Fundraising uses are those that are intended to raise resources to cover costs of activities in support of the SDGs.”

http://www.un.org/sustainabledevelopment/wp-content/uploads/2017/12/UN-Guidelines-for-Use-of-SDG-logo-and-17-icons-December-2017.pdf