Scaling up complex interventions: insights from a realist synthesis
© The Author(s). 2016
Received: 19 June 2016
Accepted: 22 November 2016
Published: 19 December 2016
Preventing chronic diseases, such as cancer, cardiovascular disease and diabetes, requires complex interventions, involving multi-component and multi-level efforts that are tailored to the contexts in which they are delivered. Despite an increasing number of complex interventions in public health, many fail to be ‘scaled up’. This study aimed to increase understanding of how and under what conditions complex public health interventions may be scaled up to benefit more people and populations.
A realist synthesis was conducted and discussed at an in-person workshop involving practitioners responsible for scaling up activities. Realist approaches view causality through the linkages between changes in contexts (C) that activate mechanisms (M), leading to specific outcomes (O) (CMO configurations). To focus this review, three cases of complex interventions that had been successfully scaled up were included: Vibrant Communities, Youth Build USA and Pathways to Education. A search strategy of published and grey literature related to each case was developed, involving searches of relevant databases and nominations from experts. Data extracted from included documents were classified according to CMO configurations within strategic themes. Findings were compared and contrasted with guidance from diffusion theory, and interpreted with knowledge users to identify practical implications and potential directions for future research.
Four core mechanisms were identified, namely awareness, commitment, confidence and trust. These mechanisms were activated within two broad scaling up strategies, those of renewing and regenerating, and documenting success. Within each strategy, specific actions to change contexts included building partnerships, conducting evaluations, engaging political support and adapting funding models. These modified contexts triggered the identified mechanisms, leading to a range of scaling up outcomes, such as commitment of new communities, changes in relevant legislation, or agreements with new funding partners.
This synthesis applies and advances theory, realist methods and the practice of scaling up complex interventions. Practitioners may benefit from a number of coordinated efforts, including conducting or commissioning evaluations at strategic moments, mobilising local and political support through relevant partnerships, and promoting ongoing knowledge exchange in peer learning networks. Action research studies guided by these findings, and studies on knowledge translation for realist syntheses are promising future directions.
KeywordsScaling up Complex interventions Chronic disease prevention Public health Population health
Over recent years, the field of implementation science has garnered much interest as a way to study and understand how to close the gap between what we know and what we do to optimise the health of populations [1, 2]. This interest has stemmed from an increasing recognition that, despite the development of innovative products, practices and programs, we have often fallen short of realising their full potential impact, due to widespread adoption and scaling up challenges . Literature and research on scaling up remains relatively new, and dominated by a focus on single or ‘simple’ interventions [4, 5]. The result is limited knowledge of how to increase the implementation and sustainability of complex interventions that are often used to address population health and other social issues . This study responds to calls from researchers and those involved in designing or delivering complex interventions, to understand how, why and under what conditions such interventions may be scaled up in order to meet the needs of the populations they serve.
Various definitions exist in the literature on what constitutes ‘scaling up’. The concept is often considered to refer to a series of processes to introduce innovations with demonstrated effectiveness through a programme delivery structure with the aim of improving coverage and equitable access to the innovation(s) . As noted by Mangham and Hanson, scaling up is often thought of as a process requiring “a strategy and implementation plan that considers the policy context, delivery mechanisms and resource requirements, as well as the pace of change, sequencing of activities, areas for prioritization and monitoring and evaluation” ( p. 87). Research on scaling up therefore finds many points of intersection with other implementation-oriented work, particularly that focused on the sustainability of interventions [6–9]. While conceptually distinct, many of the factors influencing sustainability resonate with those also thought to influence scaling up, including characteristics of the innovation or intervention itself, the organisational context, and the capacity (internal and external) and processes related to sustainability and scaling up . Research agendas focused on sustainability are also emerging, calling for greater exploration of key sustainability issues such as methods for sustainability research and infrastructure to support sustainability research .
Also consistent with these related domains, much scaling up research to date – particularly in health settings – has focused on understanding the scaling up of discrete and well-bounded interventions such as vaccines and fluoride in drinking water [4, 5, 10, 11]. These studies have yielded important insights into the implementation and adaptive requirements for simple interventions, yet provide little guidance on the challenges of scaling up interventions related to complex social problems such as poverty, the use of illicit substances and the rising prevalence of chronic diseases . As noted by Luke and Stamakis, these problems are characterised by a number of factors, “they are made up of a large number of heterogeneous elements; these elements interact with each other; the interactions produce an emergent effect that is different from the effects of the individual elements; and this effect persists over time and adapts to changing circumstances” ( p. 358). In response to these problems, many complex public health interventions have been developed, particularly for addressing pressing population health and social issues. Such complex interventions typically include a set of purposefully coordinated components that target multiple levels and sectors of a system , that operate both independently and inter-dependently , and that interact in the contexts in which they are implemented [16, 17]. Examples of these types of complex interventions include multi-level tobacco control strategies, comprehensive school health initiatives, and community-based poverty reduction activities [18, 19].
Despite the plethora of complex interventions for addressing health and social issues (included under the broad umbrella of public health), there is comparatively little understanding of the actions that can be taken to scale up these interventions in order to extend the breadth and depth of benefit to those in need. Moreover, there is little understanding about how these actions modify the contexts in which they are embedded, which in turn influence the outcomes of scaling up efforts.
This study addresses a major knowledge gap and responds to calls from key policy and practice partners who are responsible for scaling up complex interventions. These partners want to better understand what actions they may use to scale up effective, complex interventions, in what contexts and with what outcomes. Given this focus, this study had the aim of increasing understanding of how and under what conditions complex public health interventions may be scaled up in order to benefit more people and populations.
More specifically, this study addressed the following research question, which explores elements and their relationships central to a realist synthesis: What are the mechanisms by which scaling up outcomes for complex public health interventions are achieved, and what actions can be used to change contexts in order to activate these mechanisms?
To address this study’s research questions, a realist synthesis was selected. Realist syntheses aim to ‘unpack’ the mechanisms by which complex interventions work (or the reasons for their failure) in particular circumstances and settings . In doing so, realist syntheses view causality through critical linkages between contexts (e.g. characteristics of subjects and locality), mechanisms (hidden processes or entities – often contained in the minds of people – that are sensitive to variations in context and that generate outcomes), and outcomes (the range of effects that occur over time) [20–22]. Through a realist lens, actions change contexts, activating certain mechanisms leading to specific outcomes. These relationships are specified as context–mechanisms–outcomes (CMO) configurations.
For scaling up complex interventions, relevant actions are those that actors undertake in order to support the scale up of interventions, ultimately to effect positive change in population outcomes. Using insights from selected cases of scaled up interventions (see section on focusing the synthesis), our synthesis generated multiple CMO configurations for each case, which were aggregated across cases according to common mechanisms. We then interpreted these findings in light of relevant theory and practitioner experience.
A participatory team approach
The synthesis was led by a core team (n = 5) who were guided by a panel of knowledge users (n = 10) and a panel of international experts (n = 7). The purpose of the knowledge user panel was to anchor the synthesis in current practices and related questions for scaling up complex interventions, while the purpose of the panel of international experts was to ensure methodological rigour and consistency with current scientific practice. Given this study’s potential to inform scaling up theory and practices in public health, knowledge user panel members were identified from leading Canadian organisations active in scaling up complex interventions in public health settings. In contrast, members of the panel of international experts were invited with backgrounds in complex interventions, implementation science (including scaling up), realist synthesis methodology and systems change. Both panels were engaged virtually and in-person during the synthesis process, including for refining synthesis questions, focusing the synthesis on a subset of information rich initiatives, developing the literature search strategy and interpreting data.
Developing the programme theory
Comparison of outcomes, actions and contexts identified in the program theory and observed in case study documentation
Outcomes, Actions or Contexts
As identified in program theory
As observed in case study documentation
Initiative adapted to context
Initiative adapted to local community needs
Increased organizational/community capacity and readiness
Increased local organizational capacity
Finalized scaling up strategy
Intentional and explicit efforts to scale-up through planned processes
Increased demand for action
Increased participation of communities
Increased community participation
Develop and adapt funding models and partners involved
Develop mergers between existing organizations with shared objectives
Adapt funding model to engage community based organizations
Implement membership fee for peer learning community
Identify external resources
Develop competitive funding proposals
Assess potential readiness and demand for intervention
Conduct needs assessment
Engage leaders and other stakeholders: develop shared vision; align priorities; recruit champions
In-person engagement sessions for community groups and organizations in peer learning networks
Engage national leaders in cross jurisdictional summits
Identify and engage local representatives and leaders
Identify and engage local champions
Launch campaigns to secure local political support
Establish cross-sectoral and jurisdictional partners
Form new partnerships with key funding and social policy organizations
Develop coalitions of local and national organizations
Adapt the initiative design to changing contexts
Refine criteria by which communities/sites are selected
Adapt governance mechanism for initiative
Wind-up/ descale the initiative
Develop an action and implementation plan
Develop action plans by founding individuals
Hold strategic dialogue for developing action plans
Systematically evaluate implementation and outcomes
Commission external evaluations
Conduct evaluations of pilot settings
Conduct interim evaluations
Conduct end of campaign evaluations
Conduct data mining to derive maximal learning
Develop/implement shared measures
Conduct economic evaluation
Conduct social return on investment
Develop, implement and evaluate a knowledge-to-action strategy
Hold information session for community members
Share promising results from early evaluations through a range of products and mediums
Disseminate results widely through tailored information packages
Conduct site visits
National summits to encourage in-person knowledge exchange
Develop communities of practice
Publicly release evaluation results
Implement learning community for disseminating information
Create supportive amendments to policy
Identify and engage lobby groups
Encourage a systems perspective
Not observed in case study documentation
Integrate change into organizational culture
Not observed in case study documentation
Provide technical assistance and training to communities
Provide coaching support to communities
Review existing evidence
Review best practices
Advocated on behalf of community
Availability of resources: money, training, technical expertise, data systems, effective communication channels, human resources, managerial skills, evaluation capacity, leadership skills, existing relationships
Credibility of organization
Staff difficulty in implementing action plan
High turnover of staff
Scarcity of financial resources
Lack of staff skills and capacity
High quality leadership skills
History of collaboration
Lack of time among funders
Staff skilled in relevant activities
Degree of political support
Uncertain political support
Strong political support
Degree to which socio-cultural conditions and climates support scaling up
Persistent sociological problem
Social and economic hardship
Stalling of efforts
Tragedy in community
Degree of readiness for intervention
Recognized need for change
Available local funds
Evidence of impact of the intervention
Uncertainty/variability of intervention effectiveness
Promising early results
A history of intervention success
Degree of consistency in objectives and mandates between stakeholders
Organizational mandate supports solution
Alignment with user needs
Users value owned interventions
Degree of interest and demand for interventions (in communities and organizations)
Other comparable interventions
Unique needs identified
Growing support for issue
Interventions and models available
Interest from funding organizations
Variability in funding support
Focusing the synthesis: identifying case examples for study
The literature on scaling up is extensive and as a result, the synthesis needed to be focused. This is a well-recognised problem for realist synthesis and recommended in the quality standards for this approach . We decided to focus on understanding pathways for scaling up complex interventions using a subset of examples of scaling up. Potential case examples were identified through two sources: nominations from expert and knowledge user panellists and a scan of the 31 documents reviewed as part of developing the programme theory. Both processes identified examples of complex interventions (defined previously) that were scaled up by engaging more communities over time. This process yielded 11 potential case examples from the expert and knowledge user panels, and 12 potential cases from the review of documents included in the programme theory. To determine the suitability of these 23 cases, each was reviewed by the core team using the following inclusion criteria: (1) the case example focused on scaling up complex interventions in any field; (2) the case had been implemented in developed world contexts; and (3) details of scaling up the complex intervention were well documented (so as to be likely to contain sufficient relevant data) in the peer reviewed and/or grey literature. These criteria excluded 12 cases (one due to insufficient documentation and 11 due to non-complex interventions). From the 11 cases meeting inclusion criteria, this synthesis focuses on three that were deemed by panellists as being most information-rich in relation to this study’s research question. These case examples are Vibrant Communities (VC), Pathways to Education (PTE), and YouthBuild (YB).
The VC initiative aims to create and grow a movement of diverse leaders and communities from across Canada who are committed to exploring, challenging and testing ways to unleash the potential of communities to reduce poverty and ensure a good quality of life for all citizens . VC was established in 2002 through a partnership of three national sponsors (Tamarack, the Caledon Institute, and the McConnell Foundation) and a series of local communities across Canada . The principles underlying the VC approach have been informed by evolutions in the design of the initiative, from its initial implementation as Opportunities 2000 (in a single community in Ontario), through to its status as Canada’s largest poverty reduction initiative – now in more than 100 communities across Canada. VC is built around three core components: national sponsors, a pan-Canadian Learning Community, and Trail Builders. The three national sponsors (noted above) each bring a different set of competencies and are responsible for different aspects of advancing the VC agenda [25, 26]. The pan-Canadian Learning Community enables all local and national partners to learn together about the challenges and opportunities of the approach being explored, collectively building knowledge and know-how . Finally, Trail Builders are a series of urban collaboratives coordinating poverty reduction initiatives in their local settings, and which receive additional support to pilot new ideas and to track key lessons of value to the broader VC community [25–29]. More details on the VC initiative can be found at http://www.vibrantcanada.ca/.
Pathways to education
Like the VC case example, PTE aims to reduce poverty and its effects; however, it has a specific focus on increasing the high school graduation rate in low-income communities in Canada. The PTE initiative began in one neighbourhood in Toronto in 2001, which was experiencing high rates of crime, youth unemployment and low high school completion rates [30, 31]. The PTE initiative was designed to increase high school graduation rates through a programme involving four key supports, namely academic, social, financial, and one-to-one mentoring support . After securing an initial set of funds for local implementation in Toronto, a range of funding agencies joined the initiative, including non-government organisations and federal and provincial governments. As a result, the PTE programme has expanded from the initial neighbourhood in Toronto to more than 17 communities in Ontario, Quebec, Nova Scotia, Manitoba and British Columbia . Further details about the PTE initiative can be found at http://www.pathwaystoeducation.ca/.
YB aims to capitalise on the intelligence and energy of low-income people in order to rebuild their communities and their lives . YB grew from the early efforts of Dorothy Stoneman, a teacher working in east Harlem in the 1960s and 1970s. At the time, thousands of teenagers in the community were failing to complete school and were becoming involved in a cycle of violence . Youth consultations identified a desire to rebuild run-down houses and other community buildings that had been progressively overtaken by drug dealers . Soon after these consultations, the Youth Action Program was founded, which led to the development of YouthBuild USA . The YB initiative engages low-income young people in a 10-month programme, with time spent in academic settings and hands-on training building affordable housing and other community assets. From its beginning in one community in East Harlem, the YB initiative now consists of 260 programs in over 46 states that engage approximately 10,000 students each year, with funding from the US Department of Labor, local non-profit organisations, community colleges and public agencies (the initiative also has an international component not reviewed as part of this study) . For more information on the YB initiative see https://www.youthbuild.org/.
Data extracted into the Excel spreadsheets for each case were cleaned by two reviewers, with the aim to ensure consistent coding as per C, M and O categories. Within each case, two reviewers identified proximal scaling up outcomes, and the contextual details linked to each outcome. Where stated, the linked mechanisms were also extracted; where the mechanism was not stated, two reviewers agreed on an inferred mechanism. These data were extracted into a Word file as individual CMO configurations. Where outcomes could not be linked to contexts, neither these nor any linked data were extracted. Using the extracted CMO configurations, reviewers created narratives for each case example, which were sent to key informants from each case for verification of accuracy.
As per guidance on realist syntheses , we recognised the central role of mechanisms being the causal processes that link contexts to outcomes. As such, we used the identified mechanisms as the focus of the synthesis process across the case examples. Mechanisms identified in only a single case were removed from further analysis, resulting in a set of narratives based around a common set of mechanisms that occurred within more than one of the cases. Two reviewers then examined the activation of these mechanisms across the three cases, noting potentially meaningful patterns of activation. Findings presented from this analysis use CMO configurations from across the case examples to illustrate how different mechanisms may be activated.
Wider interpretation and links to substantive theory
Findings of realist syntheses are normally interpreted in relation to the programme theory and one or more substantive theories . We interpreted our findings alongside the programme theory, and diffusion of innovation theory, as suggested through consultation with the expert panel . Diffusion of innovation theory relates to the consistent pattern of adoption over time by a population of a new idea or innovation. Diffusion of innovation theory holds that innovations typically demonstrate a slow rate of initial diffusion, followed by a rapid increase in the rate of diffusion as more people and populations adopt it, before a reduction in uptake as laggards eventually adopt the innovation . We used diffusion theory to interpret and extend our understanding of identified CMO configurations and scaling up strategies [38, 39].
To further aid in interpretation of synthesis findings, a one day in-person workshop was held in July 2015 with a mix of 20 national and international research, policy and practice leaders, with experiences in scaling up complex interventions. The purposes of the meeting were to assist in interpreting preliminary study findings, and provide practical input into a research agenda on scaling up complex interventions. Preliminary synthesis results were explored through interactive presentations, small group work and facilitated large group discussions. In concert with connections to relevant theory, insights from the in-person meeting were used to refine synthesis findings and highlight future areas for further inquiry.
The following sections present the common mechanisms activated across the three case examples, followed by discussion of their role in scaling up with illustrative CMO configurations from the case examples.
Scaling up mechanisms
Evidence from the three case examples identified four common mechanisms that appear important for achieving proximal scaling up outcomes for complex interventions: awareness, confidence, commitment and trust. These mechanisms were observed to occur in at least two of the three case examples, at different time points in the scaling up of each intervention, and were activated by different contexts (which were changed through different actions), leading to a range of proximal scaling up outcomes.
Awareness is considered as knowledge or perception of a situation or fact . In the context of scaling up complex interventions, awareness was interpreted in a number of ways: awareness among community members and local stakeholders that past efforts had reached a plateau of effectiveness or impact; awareness among initiative leaders of the need to adapt scaling up activities to suit their context; or awareness by community leaders that collaborative actions and innovation were required in order to regain momentum.
Confidence is the feeling or belief that one can rely on someone or something; the state of feeling certain about the truth of something; a sense of self-assurance arising from one’s appreciation of one’s own abilities or qualities . While related to the mechanism of ‘trust’, confidence differs subtly in its more explicit focus on self-assurance. From the data reviewed, we identified confidence in three forms: the confidence of local community members that the initiative was effective for addressing the community’s problem; the confidence of the initiative leaders’ that a particular community had the capacity to implement the initiative; and the confidence among local stakeholders that the leaders of the initiative were capable of successfully implementing the initiative in their settings.
Commitment is the state or quality of being dedicated to a cause or activity [40, 41]. For scaling up complex interventions, commitment is required from a range of individuals and organisations, including those designing, implementing, evaluating or partnering on an initiative. From the case examples included in this synthesis, commitment was observed to primarily relate to a community’s commitment to being part of or joining an initiative; perceived commitment of an initiative to addressing the needs and concerns of a community; commitment of funders to supporting an initiative; and the commitment of champions (either locally or nationally) to supporting the initiative.
Trust is the firm belief in the reliability, truth, ability or strength of someone or something . As noted, trust is closely related with the mechanisms of confidence and commitment, yet is more of a relational characteristic that was found to precede mechanisms of confidence and commitment. In the included case examples, trust was found to principally relate to the trust among participating organisations (those operating at local and national levels), as well as the degree of trusting relationships between these different levels.
Activating the identified mechanisms
Insights from the VC, PTE and YB cases help reveal how each of the identified mechanisms may be activated by different changes in context, leading to different outcomes, and at different stages of scaling up. Activation of these mechanisms was observed to occur as part of efforts to renew and regenerate complex interventions, and to document scaling up successes. The following section illustrates different ways in which each of the identified mechanisms may be activated. Relevant text is highlighted as context (C), mechanism (M) or outcome (O).
Renew and regenerate
Throughout the multi-year histories of PTE, VC and YB, our analysis identified repeated moments of renewal and regeneration. These moments were signalled by official launches or relaunches of the initiative in a community, regenerated and redesigned initiatives in terms of partners engaged, staff and funding involved, or major transitions for initiatives in their structures, branding or processes. These key inflection points resulted from efforts to design, redesign and re-envisage an initiative, and appear distinct from other (related) activities such as recruitment of individuals, organisations or communities to participate or join an existing initiative, or gradual adaptations made to interventions over time.
While the mechanisms of awareness and commitment were both observed to be activated within efforts to renew and regenerate the YB, PTE and VC cases, this analysis suggests that these mechanisms may be triggered using different actions to change contexts. The following illustrations from PTE and VC highlight specific CMO configurations that reveal the activation of each key mechanism leading to renewal and regeneration of complex interventions. Additional information has been provided for each CMO configuration to enable readers to situate the configurations.
The early stages of the PTE initiative highlight how commitment may be activated to launch a complex intervention. Launching what would become the PTE initiative required the commitment of local leaders and community members to the concept of an intervention in the community. Proponents for an initiative engaged community change experts, helping to identify and articulate the value and design of a “community succession plan”, along with practical ways leaders of the Regent Park Community Health Centre (RPCHC) could participate . These actions took place in a context (C) characterised by longstanding poverty, an increasing rate of high school non-completion, awareness among local stakeholders of the need to break the cycle of poverty, and an innovative board of directors managing the RPCHC . The actions increased the ability of the RPCHC board of directors to recognise the importance of a programme for their community, activating the commitment (M) of the RPCHC, other local leaders and the broader community to a structural change in the centre’s direction. As a result, the ownership of the RPCHC was given to community members , allowing local Regent Park residents to lead the identification of obstacles and solutions, and to tailor the initiative to local conditions . After reviewing other programs  and widely engaging community members and other stakeholders the “community succession” project was officially launched in 1996 (O) .
While commitment of the RPCHC board and other local leaders was the main mechanism for enabling the launch of the PTE in its developmental phase, awareness was identified as a key mechanism for renewing and regenerating the VC initiative, 9 years after its initial implementation.
To ensure continued investment in poverty reduction efforts, VC leaders adopted a broader strategy of renewal and regeneration. Evaluation, including developing expertise in shared measurement approaches, was central to this strategy. End of campaign evaluations were used to raise awareness of the variable experiences of some communities with the VC model, and included data mining to gain maximal insight into the contributing factors influencing community closures and successes (C) [25, 43]. These data helped to raise awareness (M) among national staff of the challenges facing the VC approach, and the need to reconceptualise a national expansion programme. Through greater awareness of what such a national expansion programme may involve, activated by a data-rich context, VC leaders were able to secure transition funding for communities (O) , and to continue the engagement of local, regional and national partners (O) . As a result of these activities, the initial VC campaign was wound up in 2011 and re-launched as Vibrant Communities: Cities Reducing Poverty, which, as of 2014, included 43 cities and regions that have actively aligned poverty reduction strategies and multi-sector roundtables , with plans for an additional 57 communities in 2016 (O) .
The examples of PTE and VC provide insights into how commitment and awareness may be activated as part of broader strategies to renew and regenerate complex interventions, in these cases, through effective use of evaluative data (either from other established interventions or a local solution) and engagement with local and inter-jurisdictional partners and community members. Both examples highlight the uncertainty faced by leaders of complex initiatives: in the formative stages of designing an initiative for addressing a problem of local importance or when faced with plateauing effectiveness of an existing intervention. In the PTE example, residents were engaged around expert evidence from external sources to activate local commitment, while in the VC example, the intervention’s own evaluative data were used to raise awareness of the intervention’s successes and challenges, and to use these data to reposition and redesign the approach. In both examples, conditions were characterised by a willingness of organisational leaders to engage around a persistent social and economic problem and to use data to better understand available options.
The identified mechanisms were also observed to fire as part of efforts to demonstrate the success of the PTE, VC and YB initiatives. In particular, the cases reviewed highlight the role of confidence and trust within broader demonstration efforts, and the cyclical nature of their activation by changes in various contexts (influenced by a range of actions). Our analysis suggests that these mechanisms are key parts of documenting the success of initiatives, which is critical in retaining the participation of communities and recruiting new communities to a scaling up effort.
Despite YB’s 12 years of operation, by 1990, concern existed amongst potential funders that the initiative was unsuitable for scale up due to a potential reliance on the leadership of a single individual . To reduce this concern (and therefore to modify this context), leaders of the YB initiative sought foundation support for implementing a demonstration project outside New York City. This funding was reliant on the development and implementation of an external evaluation and knowledge translation (KT) strategy. Findings from the evaluation and KT activities brought new insights into the process of implementing the YB initiative in sites outside New York City (C), activated confidence (M) among potential funders that investments in the initiative would lead to successful and scalable outcomes (O) [34, 44]. This self-assurance or confidence spurred supporting philanthropic foundations to both use and share knowledge gained from the evaluations to widely disseminate positive messages about the initiative to new communities in new jurisdictions (C) . With the support from the Mott Foundation, Ford Foundation and, later, the DeWitt Wallace Reader’s Digest Fund, the success of demonstration projects allowed the initiative to expand to five additional sites, bringing the total number of sites to 15 by 1993 (O) .
In contrast to the YB initiative, insights from the PTE case example highlight how trust may be activated in the early stages of scaling up a complex intervention in new communities . To do so, leaders of the PTE initiative sought to demonstrate the success of the PTE initiative through intensive research and evaluation activities (C). Data from these activities documented the effectiveness of the programme, ensured accountability and programme improvement, and helped identify the replicable aspects of the programme in other communities (C) . To support systematic data collection and sharing, the PTE leadership secured external funding and formed a key partnership with the local school board , allowing the collection of baseline data in order to track changes over time (C) .
These activities yielded early implementation results of the PTE initiative that demonstrated a reduction in the high school drop-out rate from 56% to 12% [31, 46]. The research and evaluation programme also engaged community members in defining and describing the barriers faced in education and employment, and identifying potential best practices from other programs (C) . These actions were important for activating the trust of community members in the structures, processes and leaders of the PTE initiative (M), leading to their continued support of PTE in their communities (O). The capacity of the PTE initiative to provide support to the Regent Park community over a long period has been highlighted as a key characteristic of effective youth development programs (e.g. Partee GL and Halperin ), and noted as a critical vehicle for maintaining continuity in the relationships necessary for effectiveness .
The YB and PTE examples illustrate the different ways that commitment and trust may be activated as part of a strategy to document a complex initiative’s success. In both case examples, concern existed among stakeholders (i.e. funders and community members) in the suitability of the intervention for local contexts. The present analysis suggests that, in a long standing initiative (such as YB), an external evaluation and KT plan as part of an explicit demonstration effort, are important actions for activating the commitment of funding agencies. Similarly, the PTE example suggests that, for newly developed initiatives, early evaluation results may be used to activate the trust of community members in the design, implementation and leadership of interventions. In both cases, transparent evaluation and KT actions that engage key stakeholders emerged as essential.
Links to theory
The synthesis findings were interpreted in light of the original programme theory developed as part of this synthesis, as well as concepts contained in existing frameworks related to scaling up and diffusion of innovation theory.
While the programme theory did not identify specific mechanisms, it did contain relevant contexts and outcomes of scaling up complex interventions. For example, the programme theory identified that the availability of resources was as an important contextual factor for successful scaling up. From the literature reviewed, we found these resources to include financial resources, as well as non-financial resources such as leadership, evidence, relationships and specific skills. These enabling contexts were influenced by a range of actions, including specific efforts to gather useful data from research and evaluation, to develop and adapt funding models and partners, and to engage leaders and other stakeholders (Table 1). As noted in both the programme theory and our findings, these contexts were associated with a range of outcomes, such as increased local organisational capacity, prioritisation of an issue, or recruitment of additional communities. Of note, our findings suggested a number of additionally important outcomes for scaling up complex interventions, such as the launch of spin-off initiatives, or enactment of new legislation.
Findings from the synthesis were largely consistent with recommendations from the small number of scaling up frameworks in the literature, particularly in relation to scaling up actions and important contextual facilitators. As per the case examples reviewed, conducting evaluations was identified across multiple scaling up frameworks as an important facilitator of scaling up efforts [48–50]. In addition, engaging champions and investing in collaborative structures (e.g. networks and partnerships) were also noted to be present in both scaling up frameworks and the literature reviewed in this analysis [48, 50]. Yet, some scaling up frameworks also emphasised a careful, incremental approach to scaling up, which was not consistently evident in the case examples reviewed . This is not to say that planned approaches were not used; simply that the relative emphasis given to this aspect of scaling up was variable across the cases and less than that given to other activities and outcomes such as community engagement and evaluation.
Not surprisingly, the mechanisms identified in the present study (i.e. awareness, commitment, confidence and trust) are not explicitly profiled in scaling up frameworks. However, when interpreted alongside insights from diffusion theory, the synergy of these mechanisms with concepts from diffusion become more apparent. For example, the mechanism of awareness finds support in the ‘problem/opportunity identification’ and ‘communication’ phases of diffusion theory, during which insights from assessments, evaluation and monitoring may be used to raise awareness about the potential impact or utility of an innovation . In contrast, the phase of ‘adoption’ evokes the mechanism of commitment, as opinion leaders begin to demonstrate their support for an innovation in a particular setting. Finally, confidence and trust are woven within and across multiple phases of diffusion theory, as adopters gain experience with an innovation, develop the necessary skills for its implementation, reinvent and renew an innovation, and adapt it to changing conditions and technologies [38, 51].
Findings from this study also suggest potential places where diffusion theory may diverge from the practice of scaling up complex interventions. While greater flexibility has been introduced over time, diffusion theory largely focuses on linear dissemination of an essentially unchanging innovation . In contrast, results from this study highlight importance of renewing and regenerating complex interventions, and of documenting the success of the initiative at key moments. As a result, the complex intervention itself may change over time, in some cases becoming distinct from its initial conceptualisation. This notion of cyclical adaptation and renewal is consistent with other theories of sustaining innovations and implementation science and may warrant further investigation to explore potentially relevant theoretical perspectives [6, 52].
Interpretation by knowledge users
The data interpretation workshop involving practitioners responsible for scaling up complex interventions identified multiple points of connection between the practice of scaling up and findings from the synthesis. Practitioner experience and the synthesis both noted the value of multi-level partnership and engagement strategies, the importance of leadership and champions, and the utility of evidence and evaluation as scaling up actions to modify contexts. In contrast, workshop participants identified some areas of potential disconnect between the results of the synthesis and their experiences of scaling up complex interventions, such as the limited detail on the multi-level nature of observed mechanisms; the potential differences between urban and rural scaling up settings; and the need to know more about the role of serendipitous events (e.g. unexpected change in staff or team structures) that may help or hinder scaling up efforts.
The data interpretation workshop also identified priorities for a research agenda focused on scaling up complex interventions. Within the strategic areas of renewal and regeneration and documenting success, there are important topics for future exploration, such as how to build and sustain the relationships necessary for scaling up complex interventions; examining the dynamic and multi-level relationships among identified mechanisms; how to foster the necessary skills and other capacities for effective scale up; studying the processes by which interventions and contexts may be adapted over time to optimise ‘fit’; and exploring how evaluation can be used to serve accountability as well as learning and improvement purposes. As noted by participants at the data interpretation workshop, this action-research agenda would benefit from examining scaling up efforts that may have been less successful in order to better understand the nature of supportive or unsupportive conditions for scaling up.
Closing the gap between what we know and what we do requires insights into the nature of interventions and how they interact to influence contexts in which they are embedded . Complex interventions that address challenging health and social issues may target multiple levels in a socio-ecological system (e.g. individuals, organisations, communities and/or broader socio-political/economic/cultural environments) . Ensuring the most people, in the most places, benefit from these interventions requires careful and considerable scaling up effort through deliberate strategies of renewal and regeneration and continually documenting success. Through examining the scaling up experiences of three complex interventions, this study has generated new insights into how specific actions may be used to change context in such a way as to trigger the awareness, confidence, commitment and trust of key stakeholders.
Findings from this work contribute to the theoretical perspectives on scaling up complex interventions and processes for diffusion of innovations, the methods for conducting realist syntheses, and the practical approaches used in scaling up.
This study integrates theoretical concepts from diverse literatures. The interventions reviewed are collaborative in nature, requiring the coordinated action of individuals, organisations, communities and systems. These foci are consistent with contemporary literature on complex interventions in implementation science [15, 54], systems change [55, 56], and inter-organisational collaboration (including partnerships and networks) [57–59]. These findings also resonate with literature on sustainability research, which highlights the role of partnerships in furthering the theory and practice of intervention sustainability . Given this emphasis, it follows that the generative mechanisms identified in this synthesis are largely relational in nature, including concepts of trust, confidence and commitment (noting awareness as a possible exception). Consistent with realist perspectives and literature focused on collaborative action, these mechanisms provide explanations for how specific activities operate on contexts to influence scaling up outcomes. As noted in the illustrative CMO configurations from each case example, each mechanism may be activated in various ways, and by various actors, to achieve a variety of scaling up outcomes such as securing alternative financial resources, expanding the number of communities involved or bolstering political support. These relationships between contexts, mechanisms and outcomes within broader strategies of renewal and regeneration and documenting success provide useful starting points for further examinations using primary data to test and refine emerging theories of scaling up complex interventions.
This study also contributes to the growing use of realist approaches in evidence synthesis. Through guidance from an expert panel, and careful attention to recently published guidelines on realist synthesis, we sought to adhere as closely as possible to the realist synthesis approach. Yet, the consistent classification of contexts (and actions that change them), mechanisms and outcomes remains a challenge, particularly in highly iterative and dynamic fields such as scaling up . For example, given that mechanisms are often confused with the active ingredients of interventions, the expert panel in this study assisted in identifying and applying Jagosh et al.’s definition of a mechanism as “the cognitive or affective responses of participants to resources offered” . The participatory nature of our synthesis was therefore an important way for navigating complex definitional terrain, providing regular opportunity for debate, discussion and decision-making within the core team, as well as with regular input from the expert panel. As realist syntheses continue to gain interest and attention, greater sharing of experiences as to how these concepts and decisions are made will be paramount.
Finally, findings from this synthesis may help to advance the practice of scaling up complex interventions. Specific actions appear important for renewing and regenerating complex initiatives, as well as documenting success as a complex intervention is scaled up. These actions include adapting funding models in response to changing resource environments, conducting or commissioning evaluations at different time points throughout scaling up activities, developing and implementing data sharing/feedback processes, identifying and engaging community champions, and building strong foundations of political support. While these (and other) actions are known to practitioners, the present findings link these actions to mechanisms of change (i.e. building awareness, confidence, commitment and trust) and to specific outcomes that may serve as proximal goals for scaling up.
Examining study findings at the data workshop readily surfaced for scaling up practitioners that insights from the synthesis are extensive and nuanced. The relationships between context, mechanisms and outcomes resonated with their experiences and also allowed practitioners to see their experiences in new ways. They especially appreciated the concreteness of actions that can influence context and activate mechanisms, with much less interest in engaging deeply in the latter, which is more theoretical. As noted, the workshop identified some points of disconnect between review findings and practitioner experiences, particularly related to the multi-level nature of mechanisms, rural and urban differences in scaling up strategies, and the role of serendipitous events in enabling or constraining scaling up activities. Rarely are details related to these factors included in documentary evidence, yet workshop participants recognised their importance. One implication for those seeking to better understand scaling up is a more explicit focus on capturing the practitioner wisdom from scaling up experiences and including these insights along with other forms of evidence.
Knowledge user participants at the workshop (many of whom were from public health policy and practice settings), also recognised the need for new approaches to KT that would allow them and their colleagues to apply the insights in their scaling up work. KT approaches would need to set priorities for what findings to share, with whom and when, as well as how. Relevance and timeliness of findings was considered important, and how realist synthesis findings may be tailored for use by different audiences. Informed by knowledge user interests, research is now underway to explore what effective KT strategies for realist synthesis might involve, such as identifying key stakeholders to engage, exploring knowledge requirements, the utility of different modes of delivery, and potential evaluation strategies and feedback designs. The present synthesis provides an initial example for exploring these KT issues in partnership with existing knowledge user and expert panels.
Strengths and limitations
This study closely followed guidance for conducting realist syntheses, and engaged experts in the methodology. The highly participatory approach to this synthesis enabled broad input and involvement from a variety of knowledge users and experts, increasing the rigour of the synthesis, as well as the utility of findings. However, while focusing on a specific set of cases allowed detailed examination of scaling up, it may have missed important mechanisms evident in other scaling up initiatives.
Under the evidence synthesis funding requirements for this synthesis it was not possible to complete primary data collection as part of this study. As a result, findings from this study are limited to insights drawn from existing documentation relating to each case. The full depth of important contextual details and relationships within and between CMO configurations may therefore be lacking. Exploring these contexts and overall findings from this synthesis through primary data collection is therefore an important next step in advancing this work and our understanding of scaling up complex interventions. The study also focused on successful examples of scaled up interventions, while each contained challenges and barriers to scaling up, future studies investigating less successful scaling up cases may provide important and valuable insights.
This study responds to calls for understanding and improving scaling up efforts for complex interventions to enhance positive population outcomes. Through in-depth examination of three case examples of successful scaling up of interventions, this synthesis applies and advances relevant theoretical perspectives, contributes to the growing discourse on realist synthesis methodology, and offers insights into how the mechanisms of awareness, commitment, confidence and trust may be activated as part of key strategies to renew and regenerate initiatives and document success when scaling up complex interventions. Findings from this synthesis therefore provide potentially useful directions for future action-research studies to further the science and practice of scaling up.
Pathways to Education
Regent Park Community Health Centre
The authors wish to acknowledge all members of the Knowledge User and Expert Panels who contributed throughout this study.
This study was supported by the Canadian Institutes of Health Research (Grant Number KRS-132045). Support for the in-person event was provided by a Canadian Institutes of Health Research – Institute of Cancer Research Planning and Dissemination Grant (CIHR PCS-138841) with additional support from the Canadian Partnership Against Cancer and the Propel Centre for Population Health Impact. The Propel Centre for Population Health Impact is supported by a grant from the Canadian Cancer Society (grant 2011–701019). At the time of writing, contributions by CW were supported by The Australian Prevention Partnership Centre (TAPPC) through the National Health and Medical Research Council Partnership Centre grant scheme (grant GNT9100001) with the Australian Government Department of Health, the NSW Ministry of Health, ACT Health, HCF and the HCF Research Foundation. CW is also supported by an NHMRC Sidney Sax Fellowship (grant 1013165).
Availability of data and materials
The dataset(s) supporting the conclusions of this article is(are) available in figshare, https://figshare.com/articles/YouthBuild_CMO_Narratives_REVIEW_docx/3443756.
CW, BL, LS, AA and DZ made substantial contributions to conception and design, acquisition of data, and analysis and interpretation of data. GW, KR and AB made substantial contributions to conception and design as well as to analysis and interpretation of data. All authors were involved in drafting the manuscript and revising it critically for important intellectual content. All authors gave final approval of the version to be published. All authors agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work were appropriately investigated and resolved.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Not applicable to this manuscript.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Colditz GA, Wolin KY, Gehlert S. Applying what we know to accelerate cancer prevention. Sci Transl Med. 2012;4(127):127rv4. doi:10.1126/scitranslmed.3003218.View ArticlePubMedPubMed CentralGoogle Scholar
- LaRocca R, Yost J, Dobbins M, et al. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751. doi:10.1186/1471-2458-12-751.View ArticlePubMedPubMed CentralGoogle Scholar
- Mangham LJ, Hanson K. Scaling up in international health: what are the key issues? Health Policy Plan. 2010;25(2):85–96. doi:10.1093/heapol/czp066.View ArticlePubMedGoogle Scholar
- Edwards N. Scaling up Health Innovations and Interventions in Public Health: A Brief Review of the Current State of the Science. Commissioned Paper for A State-of-the-Art Conference to Advance the Science and Practice of Scale up and Spread in Health Care and Public Health. 2010. http://www.ihi.org/education/Documents/ProgramMaterials/ScaleUpBlog/7a_Commissioned_Paper%202_Public_Health.doc. Accessed 1 Oct 2014.
- Perla RJ, Bradbury E, Gunther-Murphy C. Large-scale improvement initiatives in health care: a review of the literature. J Healthc Qual. 2013;35(1):30–40.View ArticlePubMedGoogle Scholar
- Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. doi:10.1186/1748-5908-8-117.View ArticlePubMedPubMed CentralGoogle Scholar
- Hanusaik N, O’Loughlin JL, Kishchuk N, et al. Organizational capacity for chronic disease prevention: a survey of Canadian public health organizations. Eur J Public Health. 2010;20(2):195–201. doi:10.1093/eurpub/ckp140.View ArticlePubMedGoogle Scholar
- Stirman SW, Miller CJ, Toder K, et al. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65. doi:10.1186/1748-5908-8-65.View ArticlePubMedPubMed CentralGoogle Scholar
- Proctor E, Luke D, Calhoun A, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:88. doi:10.1186/s13012-015-0274-5.View ArticlePubMedPubMed CentralGoogle Scholar
- McCormick LK, Steckler AB, McLeroy KR. Diffusion of innovations in schools: a study of adoption and implementation of school-based tobacco prevention curricula. Am J Health Promot. 1995;9(3):210–9.View ArticlePubMedGoogle Scholar
- Paulussen T, Kok G, Schaalma H, et al. Diffusion of AIDS curricula among Dutch secondary school teachers. Health Educ Q. 1995;22(2):227–43.View ArticlePubMedGoogle Scholar
- Green LW, Glasgow RE, Atkins D, et al. Making evidence from research more relevant, useful, and actionable in policy, program planning, and practice slips “twixt cup and lip”. Am J Prev Med. 2009;37(6 Suppl 1):S187–91. doi:10.1016/j.amepre.2009.08.017.View ArticlePubMedGoogle Scholar
- Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annu Rev Public Health. 2011;33:357–76. doi:10.1146/annurev-publhealth-031210-101222.View ArticleGoogle Scholar
- Kickbusch I. Good Planets are Hard to Find. In: Honari M, Boelyn T, editors. Health Ecology: Health, Culture, and Human-environment Interaction. New York: Routledge; 1999.Google Scholar
- Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3. doi:10.1136/bmj.39569.510521.AD.View ArticlePubMedPubMed CentralGoogle Scholar
- Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3–4):267–76. doi:10.1007/s10464-009-9229-9.View ArticlePubMedGoogle Scholar
- Trickett EJ, Beehler S, Deutsch C, et al. Advancing the science of community-level interventions. Am J Public Health. 2011;101(8):1410–9. doi:10.2105/AJPH.2010.300113.View ArticlePubMedPubMed CentralGoogle Scholar
- National Cancer Institute. Greater Than the Sum: Systems Thinking in Tobacco Control. Tobacco Control Monograph No. 18. Bethesda: U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute; 2007.Google Scholar
- Schwartz M, Karunamuni ND, Veugelers PJ. Tailoring and implementing comprehensive school health: The Alberta Project Promoting active Living and healthy Eating in schools. Phys Health Educ Acad J. 2010;2(1):1–15.Google Scholar
- Pawson R. The Science of Evaluation: A Realist Manifesto. London: SAGE Publications Ltd.; 2013.View ArticleGoogle Scholar
- Astbury B, Leeuw F. Unpacking black boxes: mechanisms and theory building in evaluation. Am J Eval. 2010;31:363–81.View ArticleGoogle Scholar
- Wong G, Greenhalgh T, Westhorp G, et al. RAMESES publication standards: realist syntheses. BMC Med. 2013;11:21. doi:10.1186/1741-7015-11-21.View ArticlePubMedPubMed CentralGoogle Scholar
- FSG. Collective impact case study: Vibrant Communities. 2013. https://collectiveimpactforum.org/resources/case-study-vibrant-communities. Accessed 12 Nov 2014.
- Leviten-Reid E. Reflecting on Vibrant Communities: (2002–2006). Ottawa: The Caledon Institute of Social Policy; 2007.Google Scholar
- The JW. McConnell Family Foundation & Caledon Institute of Social Policy. Inspired Learning: An Evaluation of Vibrant Communities’ National Supports 2002–2012. Montreal: The J.W. McConnell Family Foundation; 2012.Google Scholar
- Vibrant Communities. Learning and evaluation for Trail Builder initiatives in Vibrant Communities. 2005. http://tamarackcci.ca/files/learning_for_vibrant_communities_trail_builders.pdf. Accessed 12 Nov 2014.
- Gerbrandt I, Makhoul A, Weaver L, et al. Creating shared prosperity: cities that lead, succeed. Vibrant Communities. 2014. http://vibrantcanada.ca/files/creating20shared20prosperity.compressed2.pdf. Accessed 5 May 2015.
- Leviten-Reid E. Opportunities 2000: Multisectoral collaboration for poverty reduction- final evaluation report. Caledon Institute of Social Policy. 2001. http://sectorsource.ca/resource/file/opportunities-2000-multisectoral-collaboration-poverty-reduction-final-evaluation. Accessed 12 Nov 2014.
- Cabaj M, Leviten-Reid E. Understanding the Potential & Practice of Comprehensive, Multisectoral Efforts to Reduce Poverty: The Preliminary Experiences of the Vibrant Communities Trail Builders. Waterloo: Tamarack Institute for Community Engagement; 2006.Google Scholar
- Kettle-Verleyen J, Finnie R. Is Pathways to Education an Effective Program? Proposed Program Evaluation of Ottawa-Ontario. Final major research paper. Ottawa: University of Ottawa; 2013.Google Scholar
- Acker C, Rowen N. Creating hope, opportunity, and results for disadvantaged youth. Part III. Can J Career Dev. 2013;12(1):63–79.Google Scholar
- Pathways to Education. The Pathways Model. 2015. http://www.pathwaystoeducation.ca/about-us/pathways-model. Accessed 5 Jun 2015.
- YouthBuild. YouthBuild: Mission and Philosophy. YouthBuild. 2015. https://www.youthbuild.org/mission-and-philosphy. Accessed 5 Jun 2015.
- Guclu A. The Growth of YouthBuild: A Case Study. Durham: Center for the Advancement of Social Enterpreneurship, Duke University Fugua School of Business; 2004.Google Scholar
- Stoneman D. Full Scale Ahead. Stanf Soc Innov Rev. 2009;7(1):27–8.Google Scholar
- YouthBuild. YouthBuild: Our Story. 2015. https://www.youthbuild.org/our-story. Accessed 5 Jun 2015.
- Booth A, Harris J, Croot E, et al. Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013;13:118. doi:10.1186/1471-2288-13-118.View ArticlePubMedPubMed CentralGoogle Scholar
- Rogers EM. Diffusion of Innovations. 4th ed. New York: The Free Press; 1995.Google Scholar
- Masuda JR, Robinson K, Elliott S, et al. Disseminating chronic disease prevention “to or with” Canadian public health systems. Health Educ Behav. 2009;36(6):1026–50. doi:10.1177/1090198109339276.View ArticlePubMedGoogle Scholar
- Oxford Dictionaries. Oxford Dictionary of English. Oxford: Oxford University Press; 2016.Google Scholar
- Taylor SM, Elliott S, Riley B. Heart health promotion: predisposition, capacity and implementation in Ontario Public Health Units, 1994–96. Can J Public Health. 1998;89(6):410–4.PubMedGoogle Scholar
- Oreopoulos P, Brown RS, Lavecchia AM. Pathways to Education: An Integrated Approach to Helping At-Risk High School Students. Cambridge: National Bureau of Economic Research; 2014.Google Scholar
- Loewen G, Makhoul A. When a Collaboration Stumbles: the Opportunities Niagara Story. Caledon Institute of Social Policy. 2009. http://www.caledoninst.org/Publications/PDF/742ENG.pdf. Accessed 12 Nov 2014.
- Ferguson RF, Clay PL, Snipes JC, et al. YouthBuild in Developmental Perspective. Cambridge: A Formative Evaluation of the YouthBuild Demonstration Project; 1996. http://files.eric.ed.gov/fulltext/ED413381.pdf. Accessed 12 Nov 2014.Google Scholar
- Rowen N, Gosine K. Final Report of the Pathways to Education Program to the Wellesley Urban Health Research Program. Toronto: The Pathways to Education Program of the Regent Park Community Health Centre; 2005.Google Scholar
- Rowen N. Proceedings of the Standing Senate Committee on Social Affairs, Science and Technology. Ottawa: Parliament of Canada; 2010.Google Scholar
- Partee GL, Halperin S. Preparing Youth for Careers, Lifelong Learning, and Civic Participation: Principles and Characteristics of Six Leading United States Youth Development Programs. American Youth Policy Forum. Washington, DC: American Youth Policy Forum; 2006. p. 48.Google Scholar
- Simmons RS, Fajans P, Ghiron L. Scaling up health service delivery: from pilot innovations to policies and programmes. Geneva: World Health Organization; 2007.Google Scholar
- Hartmann A, Linn J. Scaling up: A Framework and Lessons for Development Effectiveness from Literature and Practice. Wolfensohn Center for Development, Working Paper 5. Washington: The Brookings Institution; 2008.Google Scholar
- Cooley L, Kohl R. Scaling Up—From Vision to Large-scale Change: A Management Framework for Practitioners. Washington DC: Management Systems International; 2006.Google Scholar
- Green LW, Ottoson JM, Garcia C, et al. Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009;30:151–74. doi:10.1146/annurev.publhealth.031308.100049.View ArticlePubMedGoogle Scholar
- Estabrooks PA, Glasgow RE, Dzewaltowski DA. Physical activity promotion through primary care. JAMA. 2003;289(22):2913–6. doi:10.1001/jama.289.22.2913.View ArticlePubMedGoogle Scholar
- McLeroy KR, Bibeau D, Steckler A, et al. An ecological perspective on health promotion programs. Health Educ Q. 1988;15:351–77.View ArticlePubMedGoogle Scholar
- Hawe P, Potvin L. What is population health intervention research? Can J Public Health. 2009;100(1):Suppl I8–14.Google Scholar
- Foster-Fishman PG, Watson ER. The ABLe change framework: a conceptual and methodological tool for promoting systems change. Am J Community Psychol. 2012;49(3–4):503–16. doi:10.1007/s10464-011-9454-x.View ArticlePubMedGoogle Scholar
- Best A, Greenhalgh T, Saul J, et al. Large system transformation in health care: A realist review. Milbank Q. 2012;90(3):421–56.View ArticlePubMedPubMed CentralGoogle Scholar
- Provan KG, Lemaire R. Core concepts and key ideas for understanding public sector organizational networks: using research to inform scholarship and practice. Public Adm Rev. 2012;72(5):638–48.View ArticleGoogle Scholar
- Woulfe J, Oliver TR, Zahner SJ, et al. Multisector partnerships in population health improvement. Prev Chronic Dis. 2010;7(6):A119.PubMedPubMed CentralGoogle Scholar
- Fawcett S, Schultz J, Watson-Thompson J, et al. Building multisectoral partnerships for population health and health equity. Prev Chronic Dis. 2010;7(6):A118.PubMedPubMed CentralGoogle Scholar
- Lacouture A, Breton E, Guichard A, et al. The concept of mechanism from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation. Implement Sci. 2015;10:153. doi:10.1186/s13012-015-0345-7.View ArticlePubMedPubMed CentralGoogle Scholar
- Jagosh J, Bush PL, Salsberg J, et al. A realist evaluation of community-based participatory research: partnership synergy, trust building and related ripple effects. BMC Public Health. 2015;15:725. doi:10.1186/s12889-015-1949-1.View ArticlePubMedPubMed CentralGoogle Scholar