Lessons From the Social Innovation Fund - SAGE Journals

Forum

Lessons From the Social Innovation Fund: Supporting Evaluation to Assess Program Effectiveness and Build a Body of Research Evidence

American Journal of Evaluation 1-15 ª The Author(s) 2017 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/1098214017734305 journals.sagepub.com/home/aje

Lily Zandniapour1 and Nicole M. Deterding2

Abstract Tiered evidence initiatives are an important federal strategy to incentivize and accelerate the use of rigorous evidence in planning, implementing, and assessing social service investments. The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service, adopted a public–private partnership approach to tiered evidence. What was learned from implementing this ambitious program? How can large funding initiatives promote evaluation capacity in smaller organizations and evidence building in a sector broadly, increasing knowledge about how to address important social problems? And what can evaluators and evaluation technical assistance providers not working within a tiered evidence framework learn from the SIF? We provide an overview of the SIF model and describe how the fund operationalized “evidence building.” Materials developed to support SIF grantees represent practical, best practice strategies for successfully completing rigorous, relevant evaluations. Key lessons from overseeing over 130 evaluations—and their utility for other local evaluators—are discussed.

Keywords rigorous evaluation, capacity building, evidence-based grant making, Social Innovation Fund, tiered evidence The federal government spends more than US$600 billion per year on grants to state and local governments, educational institutions, and nonprofits to fund a wide range of social services, education, and health-care programs (Government Accountability Office [GAO], 2016). Evidence generated through evaluations of these dispersed efforts can be very important for responsible oversight of federal resources and allocation of funding (Maynard, Goldstein, & Nightingale, 2016). Indeed, within a broad climate of tightening public resources, there is growing interest in 1 2

Corporation for National and Community Service, Washington, DC, USA Institute for Research on Poverty, University of Wisconsin–Madison, Madison, WI, USA

Corresponding Author: Lily Zandniapour, Corporation for National and Community Service, 250 E Street SW, Washington, DC 20525, USA. Email: [email protected]

2

American Journal of Evaluation XX(X)

and demand for evidence-based solutions to pressing social problems (Haskins & Margolis, 2014; Nussle & Orszag, 2014). However, grant makers and nonprofit social service providers face twin challenges in implementing evidence-based programming: using existing research evidence to design promising interventions and producing rigorous evidence to advance the field of practice. If nonprofit social service providers are to contribute to our understanding of “what works,” systematic efforts to build the capacity of grant makers and service organizations are required. Both grant makers and service providers face new demands to use research evidence. On the funder side—including in the public sector—discretionary funding for social services is shrinking, increasing the pressure to demonstrate responsible stewardship and productive use of grant funds. On the programmatic side, a lack of evidence can be an existential threat for organizations unable to make a case for why they should receive scarce funding. In 2016, 92% of a national sample of 501c3 organizations reported engaging in evaluation activities in the prior year. However, only 28% of organizations possessed “promising” research capacity, defined as having some internal evaluation capacity, basic evaluation tools such as program logic model, and at least annually engaging in evaluation activities (Morariu, Athanasiades, Pankaj, & Grodzicki, 2016). How can we build evidence of effectiveness despite challenging circumstances including the varied maturity of the existing evidence base, realworld implementation constraints, scarce resources, and limited evaluation capacity? Tiered evidence initiatives are an important federal strategy for incentivizing the production and use of research evidence in the design and implementation of social services (GAO, 2016; Haskins & Baron, 2011; Haskins & Margolis, 2014; Lester, 2015). These initiatives use research evidence to inform which applicants receive funding, determine the amount, and employ rigorous evaluation to identify impact and inform future funding. Five federal tiered evidence programs had begun by 2009: The Department of Education’s Investing in Innovation (I3) Fund; Department of Health and Human Services’ Teen Pregnancy Prevention Program and Maternal, Infant, and Early Childhood Home Visiting program; Department of Labor’s Workforce Innovation Fund; and the Corporation for National and Community Service’s (CNCS) Social Innovation Fund (SIF; GAO, 2016). The SIF aimed to incentivize and support system-wide production of evidence about what works and the use of this evidence in program design and decision-making. The SIF was designed to foster innovative social programming to transform lives and communities, with a focus on three core program areas: youth development, economic opportunity, and healthy futures. After 7 years of grant making, what has been learned from the experience of implementing this tiered evidence program? How can larger funding initiatives promote evaluation capacity in smaller organizations? How can they promote evidence building in a sector more broadly, increasing knowledge about what works, where, and how to solve important social problems? And, what can evaluators and evaluation technical assistance (TA) providers not working within one of these initiatives learn from SIF?

Background and Principles of the SIF The SIF was a program of the CNCS that received funding from fiscal year 2010 to 2016. SIF was launched following the passage of Edward M. Kennedy’s Serve America Act in 2009. Beginning in 2009, the program’s annual earmarked Congressional appropriation was US$50 million, except for 2014, when the program received a US$70 million appropriation. Although CNCS made its last SIF intermediary awards in fiscal year 2016, SIF intermediaries will continue to administer their subgrant programs until their federal funding is exhausted. The SIF model is characterized by the six key programmatic elements: 1.

Intermediary grant-making organizations (i.e., SIF grantees) implement the program. These grantees find, select, monitor, support, evaluate, and report on the nonprofit organizations implementing community-based interventions.

Zandniapour and Deterding 2. 3. 4.

5.

6.

3

All funded programs or interventions are required to demonstrate at least preliminary evidence of effectiveness—in other words, SIF only funds what works. All programs implement an independent or third-party rigorous evaluation to build further evidence for the field of practice. Grantee organizations are charged with scaling evidence-based programs to increase impact within the communities they serve. This addresses a key challenge for evidence-based programming: How can we transplant and scale evidence-based programs successfully and efficiently? Through a unique leveraged funding model, public–private partnerships marshal resources toward large-scale community impact that could not be achieved by federal or philanthropic grants alone. The ultimate goal is to improve the effectiveness of nonprofits, funders, and other federal agencies by documenting best practices and promoting approaches shown to generate the greatest impact for individuals and communities.

Starting in 2010, the SIF awarded 5-year grants, ranging from US$1 to US$10 million per year, to high-performing intermediary grant makers. These SIF grantees are selected through an open, competitive process and must have a demonstrated track record of using evaluation findings for making programmatic decisions, providing training and TA to grantees, and possessing capacity to manage a federal grant and engage with third-party evaluators. SIF grantees then identify, select, and fund local nonprofit organizations through subgrants of US$100,000 or more per year. The SIF grantees may choose to fund one or multiple program models with their subawards. These subgrantees implement promising community solutions with evidence of successful outcomes in SIF’s focus areas. Examples of the desired outcomes that SIF-supported interventions seek include:

youth development—school readiness; school attendance; academic performance (K–12); graduation rates; and college access, persistence, and completion; economic opportunity—employment and job placements rates, training credentials, income earnings, hours worked and benefits, financial literacy and stability, access to housing, and pursuit of a college degree; and healthy futures—access to preventive and primary health care, promotion of physical activity and healthy eating habits, reduction in childhood obesity, and reduction in risky behaviors.

Once service providers enter the program, SIF grantees support them in increasing the impact of the interventions they implement on the ground. SIF has a highly leveraged public–private partnership model. SIF federal funds are matched one to one and in cash at both the grant and subgrant level, producing up to US$3 of total funding per US$1 of federal funds. The goal of the funding model is to facilitate program scale-up, sustainability, and increase return on taxpayer investments. In this sense, the program achieves community impact at a scale that is not easily achieved through traditional federal grant investment or a philanthropic grant investment alone. Figure 1 shows the structure of the SIF “Classic” Grant Program.1 Since the program’s inception, CNCS/SIF generated around US$1 billion in investments for evidence-based community solutions in 46 states and Washington, DC. Private and local funders multiplied the US$340 million federal investment, providing more than US$672 million in additional funding to expand evidence-based programming. These investments provided support to more than 500 nonprofit organizations and over 1,800 service locations across the country and benefited more than 857,000 individuals and their families.

4

Figure 1.


Social Innovation Fund’s “Classic” grant-making structure.

The SIF’s Tiered Evidence Initiative SIF’s grant-making structure includes an evidence framework, evaluation design guidelines, a supervised evaluation process, and TA to support evaluation. The goal of these tools is to place interventions on a continuum of evidence, identify important questions to be answered and next steps for improving the evidence base, and support organizations through the very real challenges of implementing rigorous evaluation in their local context. The grant program is designed to initiate systems change by providing both resources and accountability to participating grantees and service providers.

SIF’s Evidence Framework Research evidence plays a central role in SIF (and all tiered evidence) grant making.2 At the beginning of the SIF process, an intervention’s existing evidence base is categorized in one of the three tiers: preliminary, moderate, and strong. While there is not a one-to-one correspondence of evidence tiers across the federal innovation funds, the frameworks closely mirror one another at the “strong” level, where there is convincing causal evidence of effectiveness and the generalizability of findings. At lower levels, evidence tiers vary somewhat across innovation funds, due to their different histories and the starting state of the evidence base in each subject domain. SIF’s evidence review process and criteria were informed by and closely resemble What Works Clearinghouse’s approach to evidence review and assessment of elements of design, implementation, analysis, and results. A contribution of this framework and set of tools to the broader field of evaluation is that it provides language and criteria for defining the current maturity of the research base and suggests next steps to improve the rigor of evidence on solving important social problems. Preliminary evidence. Preliminary evidence is based on a reasonable theory of action supported by credible research findings. Typically, research has yielded promising results for either the program model itself or a similar program model. Examples of research that meet the threshold include (1) outcome studies that track participants through a program and measure participants’ responses at the end of the program and (2) third-party pre- and posttest research that determines whether participants have improved on an intended outcome. Moderate evidence. Moderate evidence comes from previous studies of the program, the designs of which can support causal conclusions (i.e., studies with high internal validity) but have limited generalizability (i.e., moderate external validity). The reverse—studies that support moderate causal conclusions but have broad general applicability—also meets the “moderate” threshold. The following constitutes moderate evidence: (1) at least one well-designed and well-implemented experimental or

Zandniapour and Deterding

5

Studies that establish causal impact and are generalizable to more diverse geographies and populaons also contribute to the strength of evidence.

Evaluaon Study Design Experimental (RCT)

Strong Internal Validity

Moderate

Nonexperimental (NE)

Preliminary External Validity

Figure 2.

Scope of Study Sites and Groups

Social Innovation Fund’s tiered evidence framework.

quasi-experimental study, with small sample sizes or other implementation or analysis that limits generalizability; (2) at least one well-designed and well-implemented experimental or quasiexperimental study where intervention and comparison groups are not equivalent at program entry, but there are no other major flaws related to internal validity; or (3) correlational research with strong statistical controls for selection bias and for discerning the influence of internal factors. Strong evidence. Strong evidence is drawn from previous studies on the program designed to support causal conclusions (i.e., studies with high internal validity), including enough of the range of possible participants and settings to suggest scaling up to the state, regional, or national level (i.e., studies with high external validity). The following are the examples of strong evidence: (1) more than one well-designed and well-implemented experimental or quasi-experimental study showing effectiveness or (2) one large, well-designed and well-implemented randomized controlled multisite trial supporting the effectiveness of the intervention. The three-tiered evidence framework provides a road map for funders and programs, demonstrating how to improve and strengthen the evidence base of their program model. Figure 2 offers a depiction of evidence strength in SIF’s tiered evidence framework. To be considered for an SIF subgrant, interventions must meet a minimum threshold of existing evidence. All interventions selected for funding by SIF grantees must have at least preliminary evidence of effectiveness. Once in the program, interventions or program models are expected to undergo rigorous independent evaluation to build upon the existing evidence base, implementing study designs that support either a moderate or strong level of evidence. Without significant investment in program evaluation, it is not always possible to demonstrate a program’s effectiveness in meeting its intended outcomes and move from promising to “proven.” Too often, skilled service providers lack the expertise, resources, or infrastructure to evaluate their efforts and produce credible evidence about program impacts. While many grant programs do not provide support or funding for evaluations, SIF provides both. The SIF was thus explicitly designed to help diverse organizations build rigorous evidence in otherwise-challenging circumstances.

SIF-Supported Evaluation Designs While the SIF evidence framework defines tiers of evidence strength, there is no one-size-fits-all evaluation design for SIF-funded programs. Grantees and subgrantees adopt a variety of study

6

American Journal of Evaluation XX(X) Experimental designs:

Quasi-experimental designs:

Nonexperimental designs:

SIF Evaluaon Study Designs of 2010-2014 Cohorts RCT (n=31)

29%

QED-Propensity Scores (n=43)

41%

Matched Comparison (n=6)

6%

QED-Interrupted Time Series (n=5)

5%

QED-Other Stascal Technique (n=3)

3%

QED-RDD (n=2)

2%

Short Interrupted Time Series (n=2)

2%

Lagged-Cohort (n=1)

1%

Pre-post (n=6)

6%

Other (n=6)

6%

Cost Studies (n=1)

1%

0

10

20

30

40

50

Number of Evaluaon Plans

Figure 3.

Range of study designs/approaches.

designs when evaluating impact. The type of study pursued depends on the program’s size; service approach; the stage of existing evidence and implementation; allocated budget; and other technical, logistical, and ethical considerations. Decisions around the design are made at the SIF grantee or local level with CNCS consultation and feedback. In all cases, independent third-party evaluators conduct evaluations. The first four cohorts of SIF grantees (funded in 2010, 2011, 2012, and 2014) and their funded organizations are in the process of conducting 106 evaluations that vary in design, scope, and outcomes. Most of these use a combination of designs including implementation and cost studies3 and include a comparison group to assess effectiveness. The majority of studies in the portfolio are implementing an experimental (29% or 31 studies) or quasi-experimental design (58% or 62 studies). Another 12% (13 studies) employ nonexperimental designs as shown in Figure 3. To document programmatic details that can aid in future expansion and replication of successful interventions, almost all SIF-funded programs examine the implementation process. Implementation studies document program participation, engagement, quality, and participant satisfaction. A subset of SIF grantees addresses issues of fidelity during implementation in a new situation or with a new target population. Studies may also assess the feasibility of conducting future experimental or quasiexperimental studies to improve the quality and strength of evidence. A number explore program costs, including cost per participant, cost–benefit analysis, or return on investment. In short, the SIF accepts a range of approaches to measure program effectiveness and establish causal impact. To accommodate a diversity of programmatic strategies while building an evidence base, research questions and evaluation strategies vary, generating rich data that help to improve existing programs, demonstrate outcomes or impacts of established interventions, and inform future programmatic decision-making (Spera, DiTommaso, Hyde, & Zandniapour, 2015). To increase the broad applicability of findings, evaluations also discuss how programs address social problems in the SIF’s three programmatic areas, measuring outcomes of interest within them. Although a number of studies within the portfolio employ single site or multisite randomized controlled trials (RCTs), SIF recognizes that these designs are not always technically or practically feasible or appropriate based on ethical considerations. In short, organizations are encouraged to pursue RCTs when they are most appropriate and potentially informative. When RCT is not the design option of choice, SIF encourages use of other research designs to assess impact, address questions of interest, and move evidence forward from where it stands. CNCS works with grantees,


7

Year 1

Years 2-5 2-step evaluaon planning process

Evaluaon Program Orientaon

Subgrantee Selecon

Select an External Evaluator

Dra a Porolio Evaluaon Strategy

Dra/ Redra SIF Evaluaon Plan (SEP)

CNCS/SIF SEP Approval

Implement Plan (Collect Data; Analyze Findings)/ Report/ Receive Feedback

CNCS and the SIF evaluaon TA team provide guidance; feedback; technical assistance to grantees, subgrantees, and their evaluaon partners; and monitor their acvies. (wrap-around services)

Figure 4.

Social Innovation Fund’s evaluation process.

subgrantees, and their respective evaluation partners to develop as robust an evaluation plan as possible, but SIF does not prescribe designs and methods to its stakeholders. The goal is to advance the evidence base from where it stands, not jump to the “gold standard” RCT before it is feasible or appropriate. Quite simply, evidence building is a field-wide challenge, and there are important practice and policy questions addressed by a range of methods. SIF receives evaluation results staggered over time, depending on the length of program and individual studies. The first results arrived in 2013, and they continue to become available as studies are completed. CNCS makes final evaluation reports publicly available through its website and Evidence Exchange.4 This repository of research and evaluation is available for public use and has already contributed to our broader understanding about promising interventions in the three SIFfunded program areas: how they work, under what conditions, and who benefits.

The SIF Evaluation Process The SIF evaluation process provides guidelines and oversights to ensure that local studies produce rigorous findings that contribute to the body of evidence about what works in community-based interventions. Figure 4 provides a visual overview of this process. First, a CNCS review team uses a review of prior research identified by the grantee or subgrant candidate and additional searches of evaluation studies to assess the incoming level of evidence, determining the evidence tier that appropriately describes the intervention. In determining the evidence tier, the review team considers prior studies’ evaluation design, methodology, and findings. Is the proposed intervention adapted for implementation at the local context directly, with modifications, or loosely based on prior studies? The CNCS review team developed The SIF Rubric to guide grantees’ and subgrantees’ assessment of the current evidence base. The Rubric also assists in the CNCS evidence review and makes the key elements of the criteria explicit.5 This tool is available to the public and can also be used to streamline the literature review process in non-SIF evaluations. Under SIF, evaluation plans are developed in a two-step process. First, SIF grantees submit a portfolio strategy plan and an evaluation plan for each intervention in the portfolio. Each evaluation plan must be approved by CNCS prior to the study’s start. One of the key challenges of the program is making sure that all parties—CNCS and the SIF evaluation team, grantees funded under the SIF, subgrantees, and their external evaluators—share an understanding of what is required for the

8


evaluation to demonstrate rigor in SIF. To address this challenge, CNCS and its evaluation TA provider developed the SIF Evaluation Plan Guidance.6 The guidance includes a detailed checklist for writing an impact evaluation plan; references and links to resources for each section of the plan; sample formats for logic models, timelines, and budgets; and a glossary of research terms. This guidance document has been used to structure and review evaluation plans and has proven highly valuable in helping evaluators, programs, and funders build a shared understanding of what evaluation plans should include. Based on its success, it could also be considered an effective tool for planning strong evaluations for non-SIF studies. Approved plans form the basis of rigorous evaluation conducted by third-party evaluators. In total, it takes between 9 months and a year following grant award for SIF grantees to assemble approved evaluation plans. This reflects the challenges of building shared understandings of the intervention model, its existing evidence base, and an evaluation plan that is rigorous but feasible. The result is that evaluation designs are high quality and appropriate to the questions most important to the interventions and their fields. Once an evaluation plan is approved, the evaluation moves to the second phase: implementation and reporting. The time frame for evaluation plan implementation and reporting is based on the length of the study and can vary. Current evaluations range from 1 to 7 years in length. On average, studies take just over 3.5 years to complete. During this period, grantees submit periodic evaluation reports, which are reviewed and compiled by CNCS. The evaluation reports are used to meet CNCS’s information needs and provide feedback to grantees and their partners. Based on what it has learned from reviewing implementation reports, CNCS has also developed an SIF Reporting Guidance document7 to clarify expectations. When the study is complete, the final evaluation reports are reviewed to assess the level of evidence produced, comparing that to the target level of evidence for the program. The set of instructions and guidelines developed as SIF implemented its evaluation process represents valuable resources that could be used by any evaluation professional operating in local contexts. These materials provide a common framework and shared understanding of what rigorous evaluation means, the elements and criteria against which local evaluations can be assessed, and best practices for implementation monitoring, results reporting, and dissemination.

SIF Evaluation TA Given the status of evaluation capacity in nonprofit settings, SIF offers evaluation TA to SIF grantees and their selected interventions throughout the funding cycle. TA is perhaps one of the more innovative components of the SIF model. Assistance is based on a coaching model, recognizing diverse local contexts and allowing programs to pursue evidence of impact in a way responsive to their local needs. SIF TA is provided “in time” and throughout the evaluation implementation process. TA recipients actively participate in troubleshooting, and their views and contributions are respected and acknowledged. This support is provided as guidance rather than as third-party evaluation itself, so that SIF grantees and their subgrantees maintain ownership of the evaluation. SIF’s coaching model was designed to build sustainable evaluation capacity—a consciously transparent approach helps ensure that CNCS, SIF grantees, and their partners achieve program’s evaluation and evidence goals. Evaluation TA is delivered by the SIF evaluation contractors in collaboration with CNCS staff members. TA provided by the agency has taken many forms:

Written feedback on evaluation plans and reports using standardized review forms; Individual TA consultations; Response to questions by phone or e-mail at any point; Evaluation Webinar Series (live and recorded) and group calls for “hot topics” of importance to the SIF evaluation work;


9

Development and rollout of guides, tip sheets, checklists, tools, and other resources such as self-paced modules (tutorials) to assist grantees in evaluation planning (http://sep-tutorial s.org/), implementation, and reporting; Support of quarterly meetings of the Evaluation Workgroup, peer learning, and evaluationfocused sessions at convenings; Access to resources and expert advice on the SIF Knowledge Network, the program’s platform for grantee community of practice.

By creating a TA feedback loop, lessons learned about the challenges faced by grantees, subgrantees, and their respective evaluation partners are used to strengthen and tailor evaluation capacity building for all participating programs. Similarly, solutions developed for addressing challenges are shared across grantees and their partners, which improve TA effectiveness. These tools are available to the community of grant makers and evaluation professionals more broadly. Within and beyond SIF grantees, SIF TA supports another mission-critical goal of the fund: sharing best practices and lessons learned from evidence-based grant making with the broader social services sector.

Discussion SIF is but one example of tiered evidence grant-making entities embraced by the U.S. federal government in the early 2000s (GAO, 2016). The process of operationalizing the program produced many resources and lessons that could be used to improve the process of evidence-based grant making more broadly. This discussion is organized in four sections: (1) SIF’s responses to common evaluation challenges encountered by grantees, (2) the importance of an intentional learning strategy for continued improvement, (3) lessons from SIF for the field of evaluation more broadly, and (4) a discussion of key takeaways and challenges faced by the SIF.

Resources Developed in Response to Common Evaluation Challenges SIF found that it is difficult for many nonprofits to meet its evaluation requirements without concerted support. Many local organizations do not have the resources, capacity, and expertise to conduct quality evaluations of their programs. SIF grantees, however, tend to be more evidenceand data-driven than typical philanthropic and nonprofit organizations, and many have experience sponsoring program evaluation and providing support in the form of TA to the nonprofits they fund. Still, subgrantees and their evaluators typically receive additional support from CNCS as described above. Even with substantial assistance, successfully completing rigorous evaluations requires effort on the part of grantees. Grantees and subgrantees typically struggle with tasks such as assessing the evidence base of program models/interventions, finding the right evaluation partner, budgeting for rigorous evaluation, assessing subgrantees’ evaluation readiness, recruitment of participants, identifying a counterfactual, accessing necessary administrative data, obtaining institutional review board (IRB) approval, and ensuring subgrantees’ investment in evaluation. These challenges are common to local evaluation, even beyond the SIF, and the SIF has developed resources that could be of broad use to evaluation professionals based on best practices learned across SIF’s 130þ evaluations. For example, the majority of SIF grantees and subgrantees struggled with how to appropriately budget for a rigorous evaluation. This challenge is not unique to SIF program stakeholders; many organizations in the social service sector grapple with budgeting. To provide accurate guidance, CNCS analyzed the evaluation budgets of the 70 interventions funded by the program in its first two cohorts of grantees (Zandniapour & Vicinanza, 2013). Based on the study, SIF developed guidance

10


and tips for developing a detailed evaluation budget.8 This resource has proven useful for grant applicants, grant recipients, and other nonprofits who are not directly involved with SIF. CNCS also learned that SIF grantees had difficulty assessing whether potential subgrantees were ready to engage in rigorous evaluation. To support this task, CNCS developed The Impact Evaluability Assessment Tool9 to provide criteria that funders can use to assess evaluation readiness and offer guidance on how to discuss a program’s capacity to participate in a rigorous impact evaluation, particularly for the quasi-experimental or experimental designs best equipped to capture program impact (CNCS, 2014). The evaluability tool is a checklist that any funder or evaluator can use for assessment, planning, and communication purposes. Based upon common questions raised in discussions with the evaluation TA providers, SIF developed additional tools addressing common research challenges, including Working with IRBs10 and screening an external evaluator (Evaluator Screening Tips11). These tools and resources identify best practices for common research tasks, but in most cases, grantees, subgrantees, and their evaluation partners required individual consultation and coaching to address specific challenges that arose in the course of evaluation. The evaluation TA team held oneon-one meetings with stakeholders to address their specific concerns and needs. A case example is described in Figure 5 to illustrate how the CNCS/SIF worked with the grantee and other stakeholders to build strong evidence despite real-world challenges.

Program Learning Strategy The SIF is, fundamentally, a learning initiative, an experiment in systemic change to encourage the growth of evidence about what works in community-based interventions, in order to enable future evidence-based grant making. As such, CNCS consciously planned efforts to build and share knowledge about the effectiveness of its activities. The SIF’s learning strategy was framed by three key knowledge-building and knowledge-sharing efforts: comprehensive local program evaluations, a knowledge initiative, and a national assessment. Comprehensive local program evaluations. SIF-sponsored evaluations have become a rich source of rigorous research evidence identifying which interventions are working, under what conditions, and for whom. Just as important, SIF-funded evaluations identify programs that have not yielded intended results, which provide opportunities for improvement through redesign or refinement of these program models. Together, the studies represent a wealth of information that benefits program operators, funders, and policy makers alike. SIF is only now beginning the pipeline of evaluation reports from local subgrantees. These reports offer credible evidence on many community-based interventions that address pressing social challenges. This work represents a real contribution to the broader field of community-based social services. Knowledge initiative. CNCS designed the SIF’s knowledge initiative to document and share resources, SIF lessons, and best practices to shape the field of evidence-based programming more broadly. Target audiences included both organizations in SIF portfolio and other nonprofits, funders, and government agencies. The knowledge initiative focuses on the sharing of many program and policyrelated topics. In addition to program and policy content, a core component of the program learning strategy is the assessment of the SIF program at the funding stream and portfolio level, which is key to the dissemination of lessons learned. SIF national assessment. The SIF national assessment was designed to facilitate program evaluation and management by CNCS and support the greater goal of learning from the federal government’s tiered evidence initiatives.12 Launched in 2013, the assessment employs a broad, mixed-methods


11

Meeting Common Evaluation Challenges: The United States Soccer Foundation The U.S. Soccer Foundation was a 2011 SIF grantee. Their Soccer for Success (SfS) program is an after-school exercise and health education program implemented in 13 cities across the country. The program uses soccer as tool to combat obesity, promote healthy eating and exercise habits, and foster positive youth development. During the 2012-2013 SIF evaluation, local subgrantees served over 12,000 children in grades K through 8. Participants were of diverse races and ethnicities in low income schools. U.S. Soccer Foundation—the grantee— possessed preliminary evidence from an implementation study of its own program, and promising small-scale RCT in a single location of a similar after-school soccer program. SfS wanted to build upon this evidence base, demonstrating their effectiveness in multiple settings serving diverse students. The resulting study would move their level of evidence from preliminary to strong. For the year prior to their impact evaluation, they studied program implementation across their 13 sites. SfS faced a number of challenges in designing and implementing this evaluation:

SfS did not have full control of which children or sites were selected to participate in their services. This meant that an RCT, either at the site or individual level, was infeasible. To attain a strong level of evidence, the study would require local, matched comparison students. Sites varied in the level of readiness to participate in evaluation. The outcome measures SfS wished to use included Body Mass Index (BMI), waist circumference and physical fitness. These data would need to be collected in person, a process that could be very labor intensive. The implementation study was highly informative, but the timeline meant the impact study did not begin until later in their grant, limiting their time to collect data and show change. Although SfS wanted to have as strong an evaluation as possible, their resources (both time and money) were limited.

The Social Innovation Fund worked with SfS to: Develop criteria for selecting an external evaluator who could meet their needs, within their timeframe and budget; Agree on the parameters of the external evaluation, including the sample sizes, appropriate processes for site selection, and the process for identifying site-level and studentlevel comparison cases for a strong quasi-experimental study; Identify data collection, matching, and analysis approaches that would produce the quality of evidence they desired; Identify criteria for determining whether their matching successfully eliminated pretest group differences, and develop a strong data analysis plan; and, Develop a final evaluation report that contained sufficient information to document the study’s internal and external validity. Despite the multiple real-world challenges SfS faced in designing and implementing its evaluation, their final report documented that the study’s site selection, matching, and analysis approaches helped to limit and account for pretest group differences, and the study ultimately produced a strong level of evidence. Figure 5.

Case example.

12


approach including a core impact assessment, targeted qualitative studies to capture lessons from SIF implementation, meta-analysis13 to synthesize outcomes and impact findings across diverse evaluations to demonstrate program effectiveness at the portfolio and focus-area level, and the design and pilot implementation of return on investment analysis as a performance measure for the program and lens for evaluating the effectiveness of SIF investment. The impact evaluation and the portfolio-level meta-analysis have produced promising evidence of the program’s effectiveness. The impact study was designed to answer: “Is SIF a transformative program in terms of improving organizational practices and policies of its grantee organizations?” and “What can be learned from the implementation and results of this program?” Findings from this evaluation showed that the program has been successful in meeting its goals and has indeed facilitated organizational change. The study defined 14 areas of capacity and practice influenced by the SIF such as SIF grantees’ adoption of evidence-based grant-making strategies, ability and willingness to build the evidence base for the service models they support, ability to support and scale the service models, and use of collaborative approaches to address local community needs (Zhang et al., 2015). The study found promising evidence of improved organizational capacities among the SIF grantees in almost all these areas. Impact findings were most pronounced and statistically significant in the domains of evaluation and evidence-driven decision-making: (1) conducting rigorous evaluations of the programs, (2) using evaluation findings to improve programs, and (3) using evaluation findings to demonstrate and communicate effectiveness of programs funded by the organization. The study found a combination of programs features were key to SIF’s impact. These features included financial investment (federal funds and matching funds); accountability (clear requirements and intensive ongoing monitoring of progress); and training, TA, and other “hands-on” support from the SIF program staff, CNCS researchers, and their evaluation contractor. The program also produced a community of professionals supporting one another in carrying out evaluation. Peer support among the SIF grantees and support from SIF grantees to the organizations they fund were important to successfully implementing the SIF program.

Conclusion: Lessons of the SIF The SIF evaluation design process provides a framework supporting the planning and implementation of increasingly rigorous local evaluations. Rather than prematurely forcing the gold standard of an RCT, grantees were expected to build upon the existing evidence base where it stands to capture how their interventions work and document how program participants benefit. By building capacity for local evaluation appropriate to the status of the existing evidence base and level of program maturity, SIF intended to accelerate the production of research evidence and increase the number of interventions with moderate and strong levels of evidence of improving important social problems. In implementing the SIF, CNCS has learned much about the challenges and opportunities of building an evidence base for social services. This experience has prompted revision of other programs within the agency and accelerated the integration of evaluation and evidence throughout our organization’s work. For example, CNCS increased the number of points awarded to evidence in its review and scoring of grant applications submitted to AmeriCorps, the agency’s flagship program. For example, the agency now conducts systematic external review of the evidence base of grant applicants’ programs in AmeriCorps and uses an expanded version of evidence tiers that includes prepreliminary evidence level. A recent GAO (2016) report on federal tiered evidence initiatives confirmed that all agencies engaging in this type of evidence-based grant making had a rich and generative experience operationalizing their program mandates. Experiments with tiered evidence grant making, such as SIF, clearly offer many opportunities for cross-program learning for participating agencies.


13

Given that the independent evaluation of SIF demonstrates that the program produced meaningful, positive change in the evaluation capacities, policies, and practices of participating organizations, it is worthwhile to consider whether it makes sense to expand the use of the SIF program model and approach—in whole or in part. It is our view that many programmatic and evidence elements of the SIF have wider utility and applicability beyond CNCS. Other federal agencies, grant making, and nonprofit organizations can incorporate various elements of the program to achieve programmatic goals and produce evidence of change. Operationalizing an evidence-based program is not an easy task; factors critical to the successful implementation of a tiered evidence model other organizational settings include but are not limited to:

a well-defined and objective evidence framework; a thoughtful and transparent evaluation process with relevant and achievable milestones; availability of agency/program staff with technical expertise in evaluation to oversee program implementation; the prioritization of evaluation and commitment of reasonable resources to it, as service providers do not typically have access to funds for research, and rigorous evaluation can be quite costly; investment in evaluation capacity building to support program implementation at the grant maker level and to support subgrantees; and flexibility to allow programs to use the evaluation to answer the questions most important to the program and to select evaluation designs that are appropriate and feasible at the local level.

While the impact of SIF on its grantees is demonstrated by the national evaluation, what can evaluation professionals and TA providers who are not part of a tiered evidence program learn from the SIF-evidence framework and evaluation process? First, it is prudent to begin the evaluation engagement by working with service providers and grant-making organizations to identify the program model or intervention’s existing evidence, locating the program model along an evidence continuum. Starting from where the program model is, the evaluator can provide recommendations on how to further build evidence for the intervention. Federal tiered evidence frameworks can be used as a guide for this purpose. By locating the program model along an evidence continuum before embarking on evaluation design, evaluators can identify research questions that are useful both to the immediate program and to the broader base of evidence on programs of its type. Second, evaluation is a broad interdisciplinary field, and evaluation professionals do not all possess the same expertise, skills, and experiences. There are many approaches to evaluation, and a qualitative implementation study requires a different set of skills than an RCT. It is important that the evaluator possesses the skill set necessary to answer the research questions most appropriate to the existing evidence base. There should be a good fit between the skills and experiences of the evaluator and the type of evidence that the program intends to build. Evaluators and their clients alike must find the right fit to ensure a rigorous evaluation is designed and competently implemented. Third, evaluation is not an end in itself; it becomes meaningful and advances the field when it is used. It is important that the evaluator promotes the use of research evidence for making programmatic decisions. Once service providers and funders fully understand the benefits and utility of rigorous evaluations—including how findings can help them strengthen their program and demonstrate its effects—there will be increased buy-in. Ultimately, evaluation and evaluative thinking can contribute toward a learning-oriented culture benefiting both immediate programs and fields of practice more broadly. But, this requires more than the delivery of a final report.

14


Opportunities and Challenges Despite the opportunities offered by a tiered evidence framework like SIF, central challenges must continue to be thoughtfully addressed. The challenges faced in implementing SIF are likely not unique to this program. A key consideration and potential challenge for any evidence-based grant making is the readiness of participating organizations to engage in rigorous impact evaluations. This is not a trivial matter, and many nonprofit organizations will continue to require assistance building their evaluation capacity and implementing rigorous studies. While we continue to build evidence of what works in community-based social interventions, many program models—particularly the most innovative among them—will still require documenting program design and implementation before they are well positioned for studies assessing their effectiveness in achieving outcomes and impacts. Directing focus on beginning where an intervention’s evidence base lies and taking the next logical steps to increase rigor, we can work to build a cumulative body of evidence despite challenging circumstances. Authors’ Note The authors prepared this work within the scope of their employment with the following U.S. federal government agencies: Corporation for National and Community Service and Department of Health and Human Services.

Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes 1. In 2014, 2015, and 2016, Corporation for National and Community Service (CNCS) was authorized to use up to 20% of the annual Congressional appropriations of Social Innovation Funds (SIFs) to test Pay for Success (PFS) approaches. PFS is an innovative model for tying the funding for an intervention to achievement of its outcomes. The goal of PFS is to leverage philanthropic and private-sector investments to deliver better outcomes, enabling government or other payers (such as school districts or hospitals) to pay only when outcomes are achieved—that is, to pay only for what worked. Following the launch of the SIF PFS program, the original SIF Grant Program is referred to as SIF “Classic,” in order to distinguish it from the PFS awards. 2. For more information on evidence and evaluation within SIF, please go to https://www.nationalservice. gov/programs/social-innovation-fund/evidence-evaluation 3. In addition to the evaluations shown here, further evaluations are underway or in development by cohorts of grantees and subgrantees entering the program in 2015 and 2016. 4. The CNCS Evidence Exchange can be accessed at https://www.nationalservice.gov/impact-our-nation/ evidence-exchange 5. The SIF rubric for assessing levels of evidence can be accessed at http://www.nationalservice.gov/sites/ default/files/resource/SIF-Rubric-Assessing-levels-of-evidence.12.19.14.pdf 6. SIF Evaluation Plan Guidance (Corporation for National and Community Service, n.d.) can be accessed at https://www.nationalservice.gov/sites/default/files/documents/SIF%20Evaluation%20guidance %208%205%202014.pdf 7. SIF Evaluation Reporting Guidance can be accessed at https://www.nationalservice.gov/sites/default/files/ resource/GP_SIF_Evaluation_Reporting_Gudiance_0.pdf


15

8. Evaluation Budgeting Guidance can be accessed at http://www.nationalservice.gov/sites/default/files/doc uments/Budgeting_for_Evaluation.pdf 9. The Impact Evaluability Assessment Tool can be accessed at http://www.nationalservice.gov/sites/default/ files/resource/FR_SIFImpactEvaluabilityAssessmentTool_Final_2016.pdf 10. Working with institutional review boards can be accessed at https://www.nationalservice.gov/sites/default/ files/resource/SIF_IRB_Tips_7_2014.pdf 11. Evaluator Screening Tips can be accessed at https://www.nationalservice.gov/sites/default/files/resource/ Evaluator-Screening-Tips.pdf 12. The SIF Classic National Assessment products suite can be accessed at http://www.nationalservice.gov/ programs/social-innovation-fund/knowledge-initiative/sif-classic-national-assessment 13. Meta-Analysis of Evaluations across the Social Innovation Fund Program: Final Report (Zhang & Sun, 2016) can be accessed at https://www.nationalservice.gov/sites/default/files/documents/FR_SIF_MetaAnalysis_Final_Report.pdf

References Corporation for National and Community Service. (n.d.). Social innovation fund: Evaluation plan guidance—A step-by-step guide to designing a rigorous evaluation. Washington, DC: Author. Corporation for National and Community Service, Office of Research and Evaluation. (2014). Impact evaluability assessment tool (Lead author, Lily Zandniapour). Washington, DC: Author. Government Accountability Office. (2016). Tiered evidence grants: Opportunities exist to share lessons from early implementation and inform future federal efforts. Retrieved from http://www.gao.gov/assets/680/ 679917.pdf Haskins, R., & Baron, J. (2011). Building the connection between policy and evidence: The Obama evidencebased initiatives. Washington, DC: Brookings Institution Press. Retrieved from https://www.brookings.edu/ wp-content/uploads/2016/06/0907_evidence_based_policy_haskins.pdf. Haskins, R., & Margolis, G. (2014). Show me the evidence. Washington, DC: Brookings Institution Press. Lester, P. (2015). Social innovation fund: Early results are promising. The Social Innovation Research Center. Retrieved from http://www.socialinnovationcenter.org/wp-content/uploads/2015/07/Social_Innovation_ Fund-2015-06-30.pdf Maynard, R., Goldstein, N., & Nightingale, D. S. (2016). Program and policy evaluation in practice: Highlights from the federal perspective. In L. R. Peck (Ed.), Social experiments in practice: The what, why, when, where, and how of experimental design & analysis. New Directions for Evaluation (pp. 109–135). Morariu, J., Athanasiades, K., Pankaj, V., & Grodzicki, D. (2016). State of evaluation 2016: Evaluation practice and capacity in the non-profit sector. Washington, DC: The Innovation Network. Nussle, J., & Orszag, P. (Eds.). (2014). Moneyball for Government. Washington, DC: Disruption Books. Spera, C., DiTommaso, A., Hyde, M., & Zandniapour, L. (2015). The social innovation fund: Pioneering an evidence-based investment model. Washington, DC: Corporation for National Community Service. Retrieved from http://www.nationalservice.gov/sites/default/files/documents/SIF-Pioneering-an-EvidenceBased-Investment-Model.pdf Zandniapour, L., & Vicinanza, N. (2013). Budgeting for rigorous evaluation: Insights from the social innovation fund. Washington, DC: Corporation for National Community Service. Retrieved from http://www. nationalservice.gov/sites/default/files/documents/Budgeting_for_Evaluation.pdf Zhang, X., Griffith, J., Sun, J., Malakoff, L., Pershing, J., Marshland, W., . . . Field, E. (2015). The SIF’s impact on strengthening organizational capacity (Prepared for the Corporation for National and Community Service, Office of Research and Evaluation). Fairfax, VA: ICF International. Zhang, X., & Sun, J. (2016). Meta-analysis of evaluations across the social innovation fund program: Final report (Prepared for the Corporation for National and Community Service, Office of Research and Evaluation). Fairfax, VA: ICF International.