Embracing complexity and uncertainty to create impact: exploring the processes and transformative potential of co-produced research through development of a social impact model
Health Research Policy and Systems volume 16, Article number: 118 (2018)
The potential use, influence and impact of health research is seldom fully realised. This stubborn problem has caused burgeoning global interest in research aiming to address the implementation ‘gap’ and factors inhibiting the uptake of scientific evidence. Scholars and practitioners have questioned the nature of evidence used and required for healthcare, highlighting the complex ways in which knowledge is formed, shared and modified in practice and policy. This has led to rapid expansion, expertise and innovation in the field of knowledge mobilisation and funding for experimentation into the effectiveness of different knowledge mobilisation models. One approach gaining prominence involves stakeholders (e.g. researchers, practitioners, service users, policy-makers, managers and carers) in the co-production, and application, of knowledge for practice, policy and research (frequently termed integrated knowledge translation in Canada). Its popularity stems largely from its potential to address dilemmas inherent in the implementation of knowledge generated using more reductionist methods. However, despite increasing recognition, demands for co-produced research to illustrate its worth are becoming pressing while the means to do so remain challenging. This is due not only to the diversity of approaches to co-production and their application, but also to the ways through which different stakeholders conceptualise, measure, reward and use research. While research co-production can lead to demonstrable benefits such as policy or practice change, it may also have more diffuse and subtle impact on relationships, knowledge sharing, and in engendering culture shifts and research capacity-building. These relatively intangible outcomes are harder to measure and require new emphases and tools. This opinion paper uses six Canadian and United Kingdom case studies to explore the principles and practice of co-production and illustrate how it can influence interactions between research, policy and practice, and benefit diverse stakeholders. In doing so, we identify a continuum of co-production processes. We propose and illustrate the use of a new ‘social model of impact’ and framework to capture multi-layered and potentially transformative impacts of co-produced research. We make recommendations for future directions in research co-production and impact measurement.
Globally, factors inhibiting the uptake of scientific evidence and, hence, the ability of health research to influence healthcare policy and improve practice are increasingly acknowledged [1,2,3]. Consequently, recognition of research co-production as a means to generate, and apply, rich implementable knowledge for healthcare policy and practice is expanding rapidly. Nevertheless, its impact remains unclear , partly due to the range of approaches under the co-production banner and current emphases within impact measurement. This paper is informed by published and grey literature, analysis of the authors’ co-produced research and insights that draw on our collective research experiences in Canada and the United Kingdom, generated through six iterative author workshops. It explores (1) the emergence and processes of research co-production, (2) how the co-production of research can increase research impact, (3) the mechanisms involved and (4) how this impact can be captured. We explicitly chose this approach to give us an opportunity to re-experience and collectively explore the benefits and challenges of co-production. It enabled us to identify a continuum of co-production processes and investigate their various impacts using a new analytic framework and case studies from our research. In this paper, we consider the paradigmatic implications of co-production methods and their potential for securing wider, more sustainable returns on investments in research. We propose a ‘social model of impact’ as an adjunct to existing more economic measures. We conclude by making recommendations for future directions in research co-production and for optimising and capturing the impact of co-produced research.
Few contest that research has the potential to improve the quality, effectiveness and consistency of healthcare. However, despite vast amounts of energy and funds directed globally towards improving the research evidence base behind policy and practice, there are clear limitations to existing methods of knowledge generation, dissemination and uptake, and thus our ability to improve healthcare quality by means of research [1,2,3]. Indeed, in the United Kingdom alone, despite an annual expenditure on health research of approximately £8 billion , most research fails to have a significant or lasting effect on policy or practice. Within a global climate of increased demand and finite resources, this return on investment, both financial and intellectual, is unacceptably poor. This has led to considerable effort from numerous stakeholders, resulting in a proliferation of approaches to transform research evidence into implementable practices.
Over time, these dilemmas have resulted in changes to the way in which the ‘gap’ between research and practice (or policy), and the best means to span it, have been conceptualised and addressed . Earlier assumptions were that the passage of research evidence into practice was largely linear and rational, and all that was required was to teach practitioners how to critically assess research and build organisational support (i.e. sufficient push or pull) . This has been replaced with more complex, social and relational models that seek to address the messy contextual realities of real-world healthcare [7,8,9,10]. Simultaneously, debate has highlighted yawning gaps between academic and health service cultures, timelines, interests and rewards, and the resulting need for collaborative methods, linkage and bridging skills [11, 12]. Questions have emerged about the nature, and ownership, of knowledge required for effective healthcare, and the processes by which it is generated and modified [2, 13, 14]. Knowledge, it is clear, is not an objective immutable product that can be packaged and transferred between contexts, but is dynamic, changeable, contested and politically imbued .
Recognition of the need for a richer, more inclusive ‘evidence’ base for real-world healthcare (including service user and practitioner perspectives and stories), which engages with and better reflects the emotional, relational, organisational, practical and rational aspects of care and policy , is not new . However, the drivers for such a change have gained momentum in recent years. For example, ethnographic research shows that clinical decisions are informed by ‘clinical mindlines’ containing evidence from multiple sources (including tacit and experiential knowledge and research) [14, 18]. Mindlines are learned, modified and applied using social means within, for example, practitioner ‘communities of practice’ . They are tested in practice and equip practitioners with the necessary ‘contextual adroitness’ for clinical decision-making and to address healthcare’s multiple realities and demands . Recent extensions to this work show how different agents/agencies engaged in the creation, policy-setting, use, or outcomes of health research have their own individual and collective mindlines relating to their specific world [13, 19]. The challenge for research in improving the quality of healthcare is therefore to acknowledge and utilise, rather than attempt to control this complexity , and to create social contexts and research approaches in which knowledge, practice and policy can be interrogated, modified and learned. Knowledge mobilisation (KM) is evolving to meet these challenges, but its evolution and expression have taken different forms, as demonstrated below.
Knowledge mobilisation (KM): definition and approaches
KM (sometimes called knowledge translation) is an umbrella term, defined broadly as the sharing of knowledge. Advances in KM over the past 20 years have led to new ways of thinking, driving new research methods and organisational structures to promote knowledge sharing – each with its own, underpinning rationale and purported mechanism(s) of action . Consequently, Davis et al.  systematically mapped diverse KM strategies and structures employed in the English NHS and its international comparators against six conceptual domains, namely (1) purpose(s) and goals (implicit or explicit), (2) knowledge types used, (3) connections and configurations, (4) people, roles and positions, (5) actions and resources available, and (6) context of operation. This led to the identification of eight KM archetypes, described from A to H, which provide a useful platform for agencies or researchers to compare and inform their KM activities .. Archetypes A, F and G represent strategies at opposite ends of Davies et al.’s  conceptual map (Box 1).
Davies et al.  do not suggest these archetypes are mutually exclusive, which strategies are most likely to be effective or claim superiority of any one approach. However, activities that broadly conform to Archetypes F and/or G combine elements that appear to directly address many problems facing the uptake of evidence. These approaches also offer means to embrace the complexity and diversity of researcher and stakeholder mindlines and help in developing the ‘contextual adroitness’ required for real world policy and practice. In the remainder of this paper, we are therefore interested in KM activities that explicitly emphasise research co-production (or integrated knowledge translation (IKT)), network building, broad inclusive knowledge sources and context, i.e. those that broadly conform to Davies et al.’s  Archetypes F and/or G. We start with a discussion of the principles and practices of research co-production and IKT.
Principles and practices of research co-production
Co-production can be defined as “a process through which inputs from individuals who are not [generally] ‘in’ the same organisation are transformed into goods and services” (, p. 1073). In co-production, both ‘producers’ and ‘users’ aim to collaborate equitably in the co-production process . Knowledge users are active agents not passive recipients, and their knowledge is valued equally . Co-production literature frequently focusses on the co-production of services by policy-makers/practitioners and the public/service users. However, it is increasingly applied to the co-production of knowledge by researchers, policy-makers, managers, practitioners, and/or service users and their carers/families. The co-production of research is a type of KM in which a “plurality of knowledge sources are combined, usually to address specific problems” (, p. 221); together, they may achieve more than they can alone . Research co-production ideally adheres to the following key principles: sharing of power, including all perspectives and skills, valuing the knowledge of everyone, reciprocity and building relationships . Outputs of co-produced research can be transformed by knowledge-user participation; consequently, they may better meet users’ needs and support decision-making and implementation in the local setting . Research co-production starts from a different epistemological and ontological stance to traditional or reductionist approaches to knowledge generation and dissemination; to illustrate, Table 1 contrasts these approaches using Davies et al.’s  six domains.
However, research co-production is a complex social and political process  and not, as sometimes described, a simple panacea for the poor uptake of research evidence. The following section explores key elements or mechanisms and known challenges of research co-production.
Key elements or mechanisms, and challenges in research co-production
To begin a process of research co-production, problems need to be collaboratively identified. Key contributors to the co-production process need personal qualities, such as openness, tolerance and flexibility , and commitment to collaboration, communication, rapport building and negotiation . Co-production of knowledge requires time, resources, blurring of boundaries and methodological exploration . Knowledge brokers might also be implicated as key actors in collaborative processes as they can overcome barriers related to relationship development and staff turnover. There is evidence that knowledge brokers currently do enact mechanisms (e.g. meetings, dialogues, relationship-building) to support collaborations .
Challenges for co-production include conflicting values, institutional rigidity and risk aversion, ensuring accountability, and shortage of capacity and incentives . Valuing different forms of knowledge is vital [23, 27], alongside sharing power  and working towards an ideal of equal relations [22, 25]. This can be demanding, as power and politics need careful negotiation and navigation  and different stakeholders and groups have their own cultural values and language, which can reinforce hierarchies . Traditional power-holders may need to relinquish influence  and unequal power relations need to be identified and addressed to avoid reproducing gender, racial/ethnic and socioeconomic inequalities . For example, the power and privilege conferred on researchers by their university affiliations may potentially affect collaborative processes with other stakeholders and communities [29, 31]. Representatives of power-holding institutions need to take responsibility to work towards equitable partnership with patients, communities and the public .
In order to realise tangible impacts from co-produced research, collaborative processes should involve different stakeholders rather than only those with greater power . However, evidence also suggests that involving those who have the authority to implement change within organisational and policy systems is key, as they have specific expertise in the area, and understand the likely facilitators and barriers to implementation . Attempts at collective action in implementation might be determined by the deliberate alignment of several features, including foundational relationships, vision, values, structures and processes, and views about the nature of the collaboration and implementation .
Maintaining rigour in co-produced research
As discussed, research co-production is neither a simple nor unidimensional process. If one considers the key elements and challenges (above) of co-production, the inherent difficulties in achieving rigour and robustness in design, and thereby outcomes, are clear to see. Thus, assessing both rigour, relevance and flexibility at the proposal stage are critical if value for money as well as likely impact are to be obtained. In a move towards distinguishing between high quality and poorly conceived co-production research, the United Kingdom N8 partnership recently proposed an 11-area evaluative framework to enable funders (and others) to evaluate this type of research proposal . These criteria include the need to focus on partnerships rather than projects, have experience and understanding of participatory engagement and facilitation, see evidence of reflective learning, and understand how opportunities for translation to support effective change are to be enacted .
Research co-production therefore goes far beyond consultation. Its growing popularity and recognition reflect its ability to achieve both rigorous and relevant findings . It is also important to note that, while the term ‘research co-production’ is increasingly used, collaborative research is rooted within diverse traditions and rationales, including participatory, collaborative and community engaged research, participatory/action research, communities of practice, civil rights, feminist and disability rights, and open innovation approaches . Furthermore, there are global variations in its manifestation and in the terms used, for example, IKT  in Canada (see below).
Integrated knowledge translation (IKT)
IKT is an increasingly prominent form of co-production in Canada, which actively tackles the need for early KM and translation [34, 35]. IKT is defined as an approach to collaborative research, in which researchers work with knowledge users who identify a problem and have the influence, and sometimes authority, to implement the knowledge generated through research . Knowledge users “function as active partners to generate research from conceptualisation to implementation, rather than be passive recipients of research or research products” . Knowledge users go beyond influencing the stages of research – they are co-investigators who carry out the research process in partnership with researchers, starting with the selection of a research question [36, 37]. Both researchers and knowledge users bring their expertise (methodological, contextual, topic related) to the project to generate research findings. In emphasising the role of knowledge-users specifically selected for their “authority to invoke practice or policy change” [33, 34], IKT brings issues of power to the fore. However, recent scoping reviews of IKT strategies reveal that, alongside other forms of research co-production, the area is theoretically undeveloped, requires greater attention to processes of engagement, and needs to establish stronger evidence between IKT models and outcomes [34, 38].
A continuum of research co-production
Co-produced research allows research ‘users’ to influence the production, mobilisation and transformation of knowledge at different stages within the research process, e.g. during the development of research questions, methods, data collection and analysis, which may help to then influence its application, outputs and outcomes, as opposed to being passive end-point recipients. Ideally, co-production occurs at all stages of the knowledge generation and application process and with all stakeholders, but this may be difficult to achieve and is the subject of much debate. However, in their recent review of IKT studies, Gagilardi et al.  found that the involvement of stakeholders tends to be under-described, making it difficult to conclude whether ideal, full involvement leads to better outcomes compared to selective involvement at particular stages.
Our experience suggests that co-produced research is situated along a continuum in terms of the number of research stages, the way stakeholders are involved in co-production, the project scope and scale, and the degree of adherence to the principles and practice of co-production achieved (Fig. 1).
Note that, in developing the dimensions of this continuum, we chose ‘adherence to the principles of co-production’ after much deliberation as a means to capture the ‘authenticity’ of co-production and extent to which it incorporated the key principles of co-produced research . We suggest that reflecting on and evaluating the extent to which a research project has been co-produced may also be supported by using the N8 partnership evaluative framework . Models that are designed to evaluate public involvement in research may also be helpful to reflect on the extent to which people are involved and influential in co-produced research . Further work is needed to develop criteria to determine the extent of co-production within research and how we evaluate and assess co-produced research .
In this paper, we focus on researchers working with policy-makers, organisations, practitioners and/or service users or their carers to co-produce research knowledge at any point in the research process (i.e. at any point on our continuum). The remaining sections focus on capturing the impact of this type of research.
Issues in measuring the impact of co-produced research
To demonstrate impact, we need to understand the various terms used to describe impact (Table 2) and to be able to capture how and where it occurs. However, determining research impact is difficult and complicated by the demands of different target audiences for evidence of different sorts of impact. Consequently, research-to-impact measurement has mushroomed, resulting in “a confusing array of models that draw on different epistemological assumptions about the link between research and impact” (, p. xxii). Research co-production approaches are likely to be more aligned philosophically with impact models that are critical and participatory and embrace a range of impacts, such as capacity development or network building, in addition to traditional impacts focused on behaviour change or economic benefit. They need to emphasise the “non-linearity, messiness, and unpredictability of the collaborative knowledge production process” (, p. 59). Currently, effective means to systematically evaluate and capture these more multifaceted impacts remain unclear.
The emphasis on measurable, economic and quantifiable impacts and relative neglect of ‘productive interactions’ or social impacts that occur in complex health research systems results in a partial view of the contributory processes and potential impacts of co-produced research. This may reinforce the appeal, to funders and research institutions, of apparently more tangible direct impacts offered by more reductionist models of research. To establish the extent to which co-produced research can affect improvements in health systems and population health, it is imperative that we address the challenges of measuring diverse, positive and negative impacts of this type of research.
To account for these issues, new approaches to studying KM activities, such as co-production and research impact, include ‘complex systems’ approaches incorporating multi-stakeholder networks , public value mapping  and contribution analysis, which is based on narratives and a wider range of different evidence types . To capture non-linear impacts within co-produced research, we need to understand both processes and outcomes so that we can attribute impacts to the co-produced research [25, 40,41,42]. For example, as IKT highlights, if we involve decision-makers with authority to make changes, this may facilitate implementation as key stakeholders are already interested and involved. Thus, process and outcome measures to understand co-production within research collaborations are an important development . Other advances include tools such as Barwick’s Knowledge Translation Planning Template [43, 44], which provide a useful framework to measure different types of research impact, including relational. However, research impact is often diffuse, long-term and potentially difficult to track; this becomes more complex within co-produced research. Research impact methods therefore need to account for this complexity and to capture the partnerships and processes involved in the co-production of knowledge between academy, policy-makers, service providers and citizenry , public engagement, ‘conceptual impact’ and ‘capacity-building’ , and cultural shifts in research and practice institutions .
Capturing the breadth of impact in co-produced research clearly requires new emphases and tools. In the following section, we therefore propose, and illustrate the use of, an analytic multi-layered framework with the ability to capture the potential breadth of co-produced research impacts. We offer this as an adjunct to strengthen existing assessments, for example, those already undertaken by the Canadian Institutes for Health Research  of health and economic impacts, or sector assessments such as the United Kingdom’s assessment of performance in higher education institutions (the Research Excellence Framework)  or assessments by care providers through the adoption of findings into guidelines and policies and their use.
Towards a research co-production impact framework
In developing a framework for capturing the impact of co-produced research, we were drawn to advances in related fields; for example, in the context of implementation science, complexity and systems approaches highlight multiple levels of influence on implementation, and relationships within and across levels, which lead to different synergies and outcomes . Research implementation can be understood as a series of feedback loops, rather than as a linear process . This means that there may be multiple mechanisms and interactions [50, 51] occurring within an implementation process, taking place at different levels over time, with interdependent relationships between them [48, 52]. Mechanisms of action within research co-production may occur and cause impacts at different levels, these impacts having the potential to become future mechanisms of action, which may initiate further changes over time. Other models explore situational and relational outcomes throughout the life-time of the research [42, 48]. However, most impact frameworks still focus on the end stage of a project after peer-reviewed articles have been published and findings disseminated ; these assume changes start at a macro-level filtering through to a meso- and micro-level (i.e. research influences policy, which influences practice). However, the impacts of co-produced research may start at a micro-level involving local policy-makers and practitioners through the research process long before peer-reviewed articles have been published. Indeed, Pawson  advocates exploring interactions and events between these different levels over time, and understanding of historical trajectories.
Since co-produced research may have multi-layered nuanced impacts, we have adapted Pawson’s (, p. 36–37) notion of context (listed 1–4 below) to inform a preliminary framework for mapping micro to macro levels of impact that can ensue from co-produced research. We have combined this with Pfadenhauer et al.’s  conceptualisation of the micro, meso and macro levels to aid understanding.
Individual (micro-level) – characteristics of stakeholders, including biological and psychological aspects (i.e. improved mental or physical health, improved practice and skills for practitioners).
Groups/networks/interpersonal relations (micro-level) – stakeholder relationships within a system (researcher/practitioner partnerships), practice changes within teams/departments.
Organisational or institutional (meso-level) – organisations including rules, norms (culture), capacity-building and organisational structures, funding organisations, educational institutions.
Societal or infrastructure (macro-level) - wider social, economic, policy and political impacts. Multiple institutions at a national scale. National public engagement, different elements of social and public value such as justice and equality.
We propose that to understand co-produced research impacts we need to capture and analyse the different elements of 1–4 and how their interactions may create emergent properties. Here, emergence can be described as “a whole having properties that are more than the sum of its parts” . To understand and document how impacts are catalysed through co-productive research we need to analyse nonlinear chains of contribution  that reflect the dynamism of complex health research systems. We need to consider longer term developments, wider social changes, any unintended consequences and how co-produced research might affect and be affected by different power dynamics.
To develop this preliminary framework, we applied it to six case studies purposefully selected from our own co-produced research. These case studies, from Canada and the United Kingdom, were chosen to ensure maximum variation in terms of their placement on the co-design continuum (i.e. in terms of research stages co-produced, types of contributor, scale of their contribution, and adherence to the principles and practice of co-production). Selection was according to the following method: authors presented several potential co-produced case studies to the group at a face-to-face workshop, we interrogated each one in relation to these key dimensions and collectively chose those for inclusion based on the criteria above. Selection was also guided by an a priori decision to include at least one case study per author and examples from both the United Kingdom and Canada, since we explicitly aimed to generate ideas through past and real-time experience of the challenges and benefits of co-production. Our choices were also clearly limited to the types and scope of projects we as authors had engaged in. Box 2 below summarises the six case studies chosen (full case study summaries, including types of contributor, scale, method and impacts, are included in Additional files 1, 2, 3, 4, 5, and 6 to inform the following analysis and subsequent conclusions and recommendations).
Applying our impact framework
To analyse these six case studies, we created a grid based on the above framework (Additional file 7) to map (1) contributors and processes involved in our six co-produced research case studies; (2) their impacts (outputs, uses, outcomes); and (3) contributory mechanisms, at each of the four levels (individual, group, organisational, societal). Next, each author analysed their own case study and made notes on the grid; these were subsequently shared, discussed and refined within an extended face-to-face author workshop. This permitted us to combine and synthesise findings from our individual case grids. Finally, these merged findings were analysed to discern broad themes in terms of the relationship between co-produced processes, their impacts and key mechanisms. KB completed the initial phase of this broader impact level analysis, AlM provided secondary independent verification and their combined findings were iteratively questioned and corroborated by other members of the team at subsequent workshops. We found that the impact framework was practical and easy to use; it helped us to simultaneously explore processes, impacts and contributory mechanisms.
What we found
While our case studies exemplified different points on the co-production continuum and their impact varied in degree and timing, we found that two distinct impact ‘patterns’ could be distinguished within them all, namely (1) ‘specific level impacts’ and (2) ‘broad impacts’ occurring across all levels. However, as previously observed [50,51,52], we found the same phenomenon could be both mechanism and impact, e.g. a mechanism may cause an impact, this impact then becomes another mechanism, which causes another impact.
Specific level impacts
Specific impacts were found to re-occur in our case studies at some levels, e.g. individual, but not across all levels. Box 3 summarises these impacts.
Broad impacts were found to re-occur across case studies and across levels (individual, group, organisational and societal). Further analysis suggested these broad impacts, occurring at every level, could be categorised according to four overarching themes which we named (1) knowledge required for effective healthcare policy and practice; (2) research for healthcare policy and practice; (3) capacity for research; and (4) nature of impact. Our case studies’ broad impacts are illustrated under these themes in Box 4.
Paradigmatic impacts arising from co-produced research
The sections above suggest that, to succeed and realise impact research, co-production requires specific skills, time and resources. However, by extrapolating from our case studies within our workshops we also noticed that, where successful, the multi-level processes, impacts and momentum of co-production also combined to promote and sustain much broader change. Indeed, it became apparent that research co-production potentially leads to a fifth level of impact, which is more conceptual and discursive than the original four. We have named this level ‘paradigmatic’ as it has potential to modify ways of understanding the world and shift frames of reference. This may involve wider cultural struggles over what is considered ‘legitimate’ knowledge and challenging the ‘cultural hegemony of powerful groups’ , resulting in a culture shift and realignment of our relationship to knowledge, research and healthcare practice and policy. These significant effects are poorly captured with current impact frameworks and highlight the need for a ‘social model of impact’ to complement those already in use. Table 3 illustrates the paradigmatic implications of research co-production emerging from our case studies and deliberations.
The transformative potential of co-production
The insights drawn from the literature, our case studies and workshops show how research co-production engenders change within, during and beyond the research project as a result of multiple social processes and productive interactions; it is dynamic and cyclical rather than linear and finite. These changes can be subtle and covert, starting at the micro-level but combining to seed macro-level change and the emergence of new ideas. These in turn may lead to transformative synergies  at a broader macro scale where co-produced research combines with other interventions, wider policies or practice priorities to create dynamic synergies. For example, micro actions by stakeholders within co-produced research may produce ‘self-organising’ macro-level changes, as exemplified in Case Study 5 (Additional file 5), where co-produced indicators had a national influence, or researchers may involve policy-makers to lever changes (e.g. Case Study 1 (Additional file 1), where national policy was altered, having a subsequent national impact . Understanding interactions across different individual policy levels over time can help us reflect on what has changed, why and how. These reflections may then help feedback learning into future collaborations. However, the framework does not advocate any particular measurement instrument as impacts can be diverse, unpredictable, occur at different levels and be tangible or intangible. We propose that the cumulative effect of micro to macro multi-layered impacts of co-produced research can potentially lead to a virtuous cycle in which broader and more enduring transformation can occur (Fig. 2).
Case studies: factors that facilitated or challenged research co-production
By applying our framework to the six case studies we were also able to discern a range of factors that facilitated or hindered co-production. The collaborations and impacts described in our case studies did not emerge from a vacuum, participants needed time to learn, develop networks and trust. Our case studies’ life cycle started from an explicit position on co-production, collaboration, knowledge and implementation. While these case studies suggest significant and wide-ranging impacts from co-produced research (Boxes 3 and 4 and Table 3), our discussions and analysis also discerned key elements, activities and mechanisms that were commonly noted within our case studies as being essential to their achievement. Some of these elements appeared stable regardless of collaborator type, while others were affected by the context of the collaborator. These findings support the literature but also extend current knowledge by identifying those which facilitated co-production at specific levels (Table 4).
Our collective experience as researchers engaged in co-production (including our case studies) also highlighted challenges related to the process of working collaboratively. We found reconciling different stakeholder agendas and expectations and keeping projects within their scope could be difficult. Funders need to be aware that approaching research in this way requires additional resources (e.g. time to develop participant capabilities, funding for staff time to participate in research or backfill). Finding existing research evidence for the topic (e.g. originating from a practice/policy-making priority) can also be problematic, as relevant research may not be available . Common challenges were maintaining practitioner engagement, maintaining project relevance in the face of constantly changing practitioner and policy-maker priorities, balancing this with service provision demands, co-ordinating multiple ethics applications, meaningful data analysis and interpretation by multiple stakeholders. Inter-agency or institutional data sharing can present issues, especially with different IT systems and stances on data confidentiality and security. Co-production partners in a number of our case studies also expressed concern at their ability to maintain momentum and dedicate sufficient time to prioritise this work, especially after the project ended.
Strengths and limitations of our approach
Some members of the author team were known to each other before we set out to develop this paper and some were not; this ensured a wide spread of experiences, views and lively debate. Our choice of approach involved ‘walking the co-production walk and talking the talk’, meaning that it required time to understand each other’s positions, discuss ideas and gain consensus on our thoughts. Our ability to track, trace and capture multi-level impacts within and beyond our case studies was made possible by ongoing relationships nurtured in the co-production process.
We are all researchers (although KB and AlM also have clinical backgrounds) and our insights, though varied, all represent the researcher voice. We verified our case study summaries and impact grids with key co-production collaborators, but they did not contribute to this paper; thus, our inferences and conclusions may have benefited from these perspectives. In selecting our case studies, we gravitated towards co-produced research projects that had gone well, as these were more likely to generate micro to macro level impacts. This facilitated the development of the research co-production continuum and impact analysis framework. However, our choices were also informed by the techniques and philosophy of appreciative inquiry . Additional insights may have emerged from reflection on negative cases. Further framework development and application will need to include a more systematic examination of the negative instances of impact. Our case studies focus on topics that were amenable to and benefited from co-production; not all healthcare questions can be answered in this way. Finally, while service users were involved as ‘participants’ in three case studies and co-production ‘contributors’ in another three (mainly at later stages of the research cycle, e.g. intervention development), they were not involved in co-production at earlier stages or throughout the research cycle.
Recommendations and questions for future research
This concept paper proposes a continuum of research co-production, a social model of impact and a new framework for capturing the multi-layered impacts of this type of research. We offer it as a stimulus for debate, discussion and further research. The recommendations and research questions in Box 5 below are offered for research funders, policy-makers, managers and stakeholders involved in the co-production of knowledge and its application.
History suggests research methods that explicitly aim to control and reduce complexity and contextual uncertainty and employ linear methods with the purpose of generating objective facts need to be balanced with other rigorous approaches to generating knowledge to inform healthcare quality and efficacy in the real world. The principles of co-production embrace complexity and uncertainty, potentially leading to a virtuous cycle of research processes and micro to macro level impacts with the ability not only to generate useful knowledge, but also to transform it into usable knowledge and to broaden research capacity in the process. Within complex human systems, emphases on the economic impact or end-of-project research outputs neglect the potential for the research process and productive human interactions to affect much deeper and more enduring change; our social model of impact aims to address this gap.
Co-production is challenging; it demands flexibility, reflexivity and boundary crossing, but when it works it results in insights and actions far greater than the sum of its contributory parts. Co-production can actively support the democratisation of knowledge and incorporate and blur the boundaries between different forms and sources of knowledge. It can provide the rich evidence required for effective policy and practice and foster ‘contextually adroit’ research-informed decision-making . This may lead to more sustainable and wider impacts from intellectual and economic investment in research.
Following the initial phase of framework development described in this article, the authors presented and tested it further at a United Kingdom KM (http://knowledgemobilisation.net/) Forum 2018, workshop held in Bristol, United Kingdom. At this event, the authors facilitated workshop attendees in applying the framework to their own co-produced research, including projects where co-production was deemed to have been successful or those perceived of as having failed in some respect. This experience highlighted the need for guidance to assist others in using and testing it, which we subsequently developed (Additional file 8). This guidance is offered here as a preliminary means for co-production collaborators to operationalise the framework and capture impacts of their co-produced research. The authors anticipate that future work is likely to include further development of a Social Impact Framework tool; we welcome feedback to assist us in making it workable and accessible.
Our experience at the United Kingdom KM workshop also suggested the framework is applicable and useful for capturing impacts of projects where co-production was less successful, and/or the challenges involved impeded its completion or success. In one group discussion, they found that, by using the framework to reflect on micro-macro levels processes, impacts and mechanisms within a project that had been perceived as failing to achieve the expected outcomes, multiple impacts had actually occurred at all the levels, although they were not necessarily those initially anticipated or sought. Some of these impacts were significant and positive, especially at individual level, and had not been captured, or considered, before. The framework supported reflection on what had occurred, and highlighted that co-production had exerted a dynamic effect, akin to the scattering of billiard balls, and appeared to set in motion a range of unexpected processes and impacts. This warrants further investigation.
Integrated Knowledge Translation
Greenhalgh T, Wieringa S. Is it time to drop the ‘knowledge translation’ metaphor? A critical literature review. J R Soc Med. 2011;104(12):501–9.
Rycroft-Malone J, Seers K, Titchen A, Harvey G, Kitson A, McCormack B. What counts as evidence in evidence-based practice? J Adv Nurs. 2004;47(1):81–90.
Davies HT, Powell AE, Nutley SM. Mobilising knowledge to improve UK health care: learning from other countries and other sectors – a multimethod mapping study, Health Services and Delivery Research. Southampton: NIHR Journals Library; 2015.
Graham ID, Kothari A, McCutcheon C, on behalf of the Integrated Knowledge Translation Research Network Project Leads. Moving knowledge into action for more effective programmes, practice and policy: protocol for a research programme on integrated knowledge translation. Implement Sci. 2018;13:22.
Walshe K, Davies H. Research, influence and impact: deconstructing the norms of health services research commissioning. Policy Soc. 2010;9(2):103–11.
Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones A. Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme. Health Technol Assess. 2016;20(76):1–254.
Ward V, Smith S, House A, Hamer S. Exploring knowledge exchange: A useful framework for practice and policy. Soc Sci Med. 2012;74(3):297–304.
Fitzgerald L, Ferlie E, Hawkins C. Innovation in healthcare: how does credible evidence influence professionals? Health Soc Care Community. 2003;11(3):219–28.
May C. Towards a general theory of implementation. Implement Sci. 2013;8:18.
Wye L, Branagan E, Cameron A, Gabbay J, Klein J, Pope C. Evidence-based policy-making and the 'art' of commissioning - how English healthcare commissioners access and use information and academic research in ‘real life’ decision-making: An empirical qualitative study. BMC Health Serv Res. 2015;15:430. https://doi.org/10.1186/s12913-015-1091-x.
Boaz A, Locock L, Ward V. Whose evidence is it anyway? Evid Policy. 2015;11(2):145–8.
Lomas J. Using 'linkage and exchange' to move research into policy at a Canadian foundation. Health Aff. 2000;19(3):236–40.
Wieringa S, Greenhalgh T. 10 years of mindlines: a systematic review and commentary. Implement Sci. 2015;10:45.
Gabbay J, Le May A. Practice-based Evidence for Healthcare: Clinical Mindlines. Abingdon: Routledge; 2011.
Bannister J, O'Sullivan A. Knowledge mobilisation and the civic academy: the nature of evidence, the roles of narrative and the potential of contribution analysis. Contemporary Soc Sci. 2013;8(3):249–62.
Bozeman B, Sarewitz D. Public value mapping and science policy evaluation. Minerva. 2011;49(1):1–23.
Nutley SM, Walter I, Davies HTO. Using Evidence: How Research Can Inform Public Services. Bristol: Policy Press; 2007.
Gabbay J, le May A. Evidence based guidelines or collectively constructed "mindlines?" Ethnographic study of knowledge management in primary care. BMJ. 2004;329(7473):1013.
Oduro-Mensah E, Kwamie A, Antwi E, Amissah Bamfo S, Bainson HM, Marfo B, Coleman MA, Grobbee DE, Agyepong IA. Care decision making of frontline providers of maternal and newborn health services in the greater Accra region of Ghana. PLoS One. 2013;8(2):e55610.
Ostrom E. Crossing the great divide: Coproduction, synergy, and development. World Dev. 1996;24(6):1073–87.
Osborne SP, Radnor Z, Strokosch K. Co-production and the co-creation of value in public services: a suitable case for treatment? Public Manag Rev. 2016;18(5):639–53.
Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. 2016;11:20.
Rycroft-Malone J, Burton CR, Bucknall T, Graham ID, Hutchinson AM, Stacey D. Collaboration and Co-Production of Knowledge in Healthcare: Opportunities and Challenges. Int J Health Policy Manage. 2016;5(4):221–3.
Hickey G, Brearley S, Coldham T, Denegri S, Green G, Staniszewska S, Tembo D, Torok K, Turner K. Guidance on Co-producing a Research Project. Southampton: NIHR INVOLVE; 2018. http://www.invo.org.uk/wp-content/uploads/2018/03/Copro_Guidance_Mar18.pdf. Accessed 6 Nov 2018.
Campbell H, Vanderhoven D. N8/ESRC Research Programme Knowledge That Matters: Realising the Potential of Co-Production. Manchester: N8 Research Partnership; 2016. http://www.n8research.org.uk/research-focus/urban-and-community-transformation/co-production/. Accessed 9 Oct 2017
Kothari A, MacLean L, Edwards N, Hobbs A. Indicators at the interface: managing policymaker-researcher collaboration. Knowl Manag Res Pract. 2011;9(3):203–14.
Cooke J, Langley J, Wolstenholme D, Hampshaw S. “Seeing” the difference: the importance of visibility and action as a mark of “authenticity” in co-production; comment on “Collaboration and co-production of knowledge in healthcare: opportunities and challenges”. Int J Health Policy Manage. 2017;6(6):345–8.
Bornbaum C, Kornas K, Peirson L, Rosella LC. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implement Sci. 2015;10:162.
Williams BN, Kang S, Johnson J. (Co)-contamination as the dark side of co-production: public value failures in co-production processes. Public Manage Rev. 2016;18(5):692–717.
Rycroft-Malone J, Burton CR, Wilkinson J, Harvey G, McCormack B, Baker R, Dopson S, Graham ID, Staniszewska S, Thompson C, Ariss S, Melville-Richards L, Williams L. Collective action for implementation: a realist evaluation of organisational collaboration in healthcare. Implement Sci. 2016;11:17.
Muhammad M, Wallerstein N, Sussman AL, Avila M, Belone L, Duran B. Reflections on researcher identity and power: the impact of positionality on community based participatory research (CBPR) processes and outcomes. Crit Sociol. 2015;41(7–8):1045–63.
Kothari A, McCutcheon C, Graham ID. Defining integrated knowledge translation and moving forward: A response to recent commentaries. Int J Health Policy Manage. 2017;6(5):299–300.
Facer K, Enright B. Creating Living Knowledge: The Connected Communities Programme, Community-University Relationships and the Participatory Turn in the Production of Knowledge. Bristol: University of Bristol and AHRC Connected Communities Programme; 2014. https://connected-communities.org/wp-content/uploads/2016/04/Creating-Living-Knowledge.Final_.pdf. Accessed 9 Oct 2017.
Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38.
Gagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: what we know and do not yet know. J Epidemiol Community Health. 2017;71(2):105–6.
Kothari A, Wathen CN. A critical second look at integrated knowledge translation. Health Policy. 2013;109(2):187–91.
Kothari A, Wathen CN. Integrated knowledge translation: digging deeper, moving forward. J Epidemiol Community Health. 2017;71(6):619.
Camden C, Shikako-Thomas K, Nguyen T, Graham E, Thomas A, Sprung J, et al. Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disabil Rehabil. 2015;37:1390–400.
Gibson A, Welsman J, Britten N. Evaluating patient and public involvement in health research: from theoretical model to practical workshop. Health Expect. 2017;20(5):826–35.
Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study: achieving research impact through co-creation. Milbank Q. 2016;94(2):392–429.
Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ. Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med. 2017;14(8):e1002370.
Morton S. Progressing research impact assessment: A ‘contributions’ approach. Res Eval. 2015;24(4):405–19.
Barwick M. Building Scientist Capacity in Knowledge Translation: Development of the Knowledge Translation Planning Template. Technol Innov Manag Rev. 2016;6(9):9–15.
Barwick M. Knowledge Translation Planning Template. Ontario: The Hospital for Sick Children; 2017.
Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003;8(3):165–70.
McLean R, Tucker J. Evaluation of CIHR’s Knowledge Translation Funding Program Evaluation Report 2013. 2013. http://cihr-irsc.gc.ca/e/documents/kt_evaluation_report-en.pdf. Accessed 22 Sept 2017.
Stern N. Building on Success and Learning from Experience: An Independent Review of the Research Excellence Framework. London: IND/16/9. Department for Business, Energy & Industrial Strategy.; 2016. https://www.gov.uk/government/publications/research-excellence-framework-review. Accessed 9 Oct 2017.
Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, Wahlster P, Polus S, Burns J, Brereton L, Rehfuess E. Making sense of complexity in context and implementation: the Context and Implementation of Complex Interventions (CICI) framework. Implement Sci. 2017;12:21.
May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016;11(1):141.
Byrne D. Evaluating complex social interventions in a complex world. Evaluation. 2013;19(3):217–28.
Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10:21.
Pawson R. The Science of Evaluation: A Realist Manifesto. London: SAGE; 2013.
Room G. Evidence for agile policy makers: the contribution of transformative realism. Evid Policy. 2013;9(2):225–44.
Room G. Agile Actors on Complex Terrains: Transformative Realism and Public Policy (Complexity in Social Science). London: Routledge; 2016.
Wye L, Cramer H, Carey J, Anthwal R, Rooney J, Robinson R, Beckett K, Farr M, le May A, Baxter H. Knowledge brokers or relationship brokers? The role of an embedded knowledge mobilisation team. Evid Policy. 2017. https://doi.org/10.1332/174426417X15123845516148.
Kavanagh T, Stevensor B, Seers K, Sidani S, Watt-Watson J. Process evaluation of appreciative inquiry to translate pain management evidence into pediatric nursing practice. Implement Sci. 2010;5(90). https://doi.org/10.1186/1748-5908-5-90.
Nolan M, Brown J, Davies S, Nolan J, Keady J. The Senses Framework: Improving Care for Older People Through a Relationship-Centred Approach. Getting Research into Practice (GRiP), Report no 2. Project Report. Sheffield: University of Sheffield; 2006.
Andrews N, Gabbay J, le May A, Miller E, O’Neill M, Petch A. Developing Evidence-Enriched Practice in Health and Social Care with Older People. York: Joseph Rowntree Foundation; 2015. https://www.jrf.org.uk/report/developing-evidence-enriched-practice-health-and-social-care-older-people. Accessed 9 Oct 2017.
Morgan S, Andrews N. Positive risk-taking: from rhetoric to reality. J Ment Health Train Educ Pract. 2016;11(2):122–32. https://doi.org/10.1108/JMHTEP-09-2015-0045.
Care Council for Wales. Good Work - A Dementia Learning and Development Framework for Wales. Cardiff: Care Council for Wales; 2016. https://socialcare.wales/resources/good-work-dementia-learning-and-development-framework. Accessed 24 Jan 2018.
Salter K, Salvaterra RP, Antonello D, Cohen BE, Le Ber MJ, Kothari A, Lemieux S, Moran K, Rizzi K, Robson J, Wai C. Organizational level indicators to address health equity work in local public health agencies: a scoping review. Canadian J Public Health. 2017;108(3):e306–13. https://doi.org/10.17269/cjph.108.5889.
Salvaterra RP, Antonello D, Le Ber MJ, Kothari A, Lemieux S, Moran K, Robson J, Wai C, Salter K. Indicators to guide health equity work in local public health agencies: a locally driven collaborative project in Ontario. J Health Promot Chronic Dis Prev Canada 2018; 38(7–8):277–285.
Valaitis R, MacDonald M, Kothari A, O'Mara L, Regan S, Garcia J, et al. Moving towards a new vision: implementation of a public health policy intervention. BMC Public Health. 2016;16:412.
Regan S, MacDonald M, Allan DE, Martin C, Peroff-Johnston N. Public health human resources: a comparative analysis of policy documents in two Canadian provinces. Hum Resour Health. 2014;12:13.
Kothari A, Gore D, MacDonald M, Bursey G, Allan D, Scarr J, et al. Chronic disease prevention policy in British Columbia and Ontario in light of public health renewal: a comparative policy analysis. BMC Public Health. 2013;13:934.
Tomm-Bonde L, Schreiber RS, Allan DE, MacDonald M, Pauly B, Hancock T. Fading vision: knowledge translation in the implementation of a public health policy intervention. Implement Sci. 2013;8:59.
Pauly BB, MacDonald M, Hancock T, Martin W, Perkin K. Reducing health inequities: the contribution of core public health services in BC. BMC Public Health. 2013;13(1):550.
Pinto AD, Manson H, Pauly B, Thanos J, Parks A, Cox A. Equity in public health standards: a qualitative document analysis of policies from two Canadian provinces. Int J Equity Health. 2012;11(1):28.
Davies P, Wye L, Horrocks S, Salisbury C, Sharp D. Developing quality indicators for community services: the case of district nursing. Qual Prim Care. 2011;19(3):155–66.
Horrocks S, Cook JR, Ashton N, Wye L. Made to measure? Assessing feasibility of quality indicators for district nursing. Br J Community Nurs. 2012;17(1):28–34.
Cressey P, Milner S, Farr M, Abercrombie N, Jaynes B. Proving the Value of Advice: A Study of the Impact of Citizens' Advice Bureau Services: University of Bath; 2014. http://www.bath.ac.uk/publications/proving-the-value-of-advice-a-study-of-the-impact-of-citizens-advice-bureau-cab-services/. Accessed 9 Oct 2017
Farr M, Cressey P, Milner S, Abercrombie N, Jaynes B. Proving the value of advice: a study of the advice service of Bath and North-East Somerset Citizens Advice Bureau. South West Forum 2014. http://opus.bath.ac.uk/39052/. Accessed 9 Oct 2017.
Farr M, Cressey P: The social impact of advice during disability welfare reform: from social return on investment to evidencing public value through realism and complexity. Public Manag Rev. 2018. doi: https://doi.org/10.1080/14719037.2018.1473474.
The authors acknowledge all the contributors (co-producers and participants) in the six case studies and their host or funding organisations. We would also like to thank the IKT Research Network for giving us the opportunity to develop and write this article.
This paper was commissioned by the Integrated Knowledge Translation Research Network, which brings together knowledge users and researchers to advance the science and practice of IKT and train the next generation of IKT researchers. Honorariums were provided for completed papers. The iKTR Network is funded by a Canadian Institutes of Health Research Foundation Grant (FDN #143247).
MF’s time is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care West (CLAHRC West) at University Hospitals Bristol NHS Foundation Trust. KB and LW are supported by NIHR Knowledge Mobilisation Research Fellowships.
Availability of data and materials
All data generated and analysed during this study are included in this published article (and its supplementary information files).
The views expressed are those of the authors and not necessarily those of the case studies, the NHS, the NIHR or the Department of Health.
We have provided additional author information (see below) to help the reader understand our viewpoint.
KB currently holds an NIHR Knowledge Mobilisation Research Fellowship and works at the University of the West of England, United Kingdom. Her interests are knowledge mobilisation, mindlines, adult and child injury, post-injury psychological issues, and use of creative approaches to knowledge mobilisation such as forum theatre and film. Her background is in NHS nursing and midwifery.
MF is a Senior Research Associate at NIHR Collaborations for Leadership in Applied Health Research and Care West (NIHR CLAHRC West), Bristol, United Kingdom. Her research interests include co-production and co-design within public services and research, knowledge mobilisation and evaluation methodologies based on realism and complexity. Before her academic career, she worked within the voluntary sector on environmental community projects, within mental health services and with young people.
AK is an Associate Professor in the Faculty of Health Sciences at Western University, Canada. She has two main areas of research expertise. First, her work focuses on developing a greater understanding of integrated knowledge translation and identifying the factors that influence the successful execution of IKT initiatives. AK’s second area of research expertise is in public health systems research.
LW is a NIHR Knowledge Mobilisation Research Fellow working at the University of Bristol, United Kingdom. She previously led the Bristol Knowledge Mobilisation Team, which was an embedded researcher/policy-maker multi-disciplinary team working across the two communities (www.bristol.ac.uk/primaryhealthcare/km). She has worked in academic and policy institutions such as the Department of Health and the think tank the King’s Fund (both United Kingdom). Areas of interest include healthcare policy-making, community services, service evaluations, end-of-life care and quality measures.
AlM is Professor Emerita of Nursing at the University of Southampton, United Kingdom. Her research focuses on how knowledge is used in healthcare, how communities of practice influence learning and what skills frontline staff need for quality improvement. She is Co-editor-in chief of the Journal of Research in Nursing.
Ethics approval and consent to participate
All six case studies had appropriate ethical approvals in place; these are available on request from the corresponding author.
Consent for publication
Key investigators involved in the six case studies have confirmed their consent to use of this work in this paper. They have also reviewed and agreed to the case study summaries.
All authors were involved as researchers in the co-production of Case Studies 1–6. KB, MF, LW and AlM declare that they have no other competing interests. AK is a co-investigator and member of the Executive Committee for the research group that initiated this call for papers (the Integrated Knowledge Translation Research Network, iKTR network).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Case study 1. (DOCX 18 kb)
Case study 2. (DOCX 17 kb)
Case study 3. (DOCX 18 kb)
Case study 4. (DOCX 19 kb)
Case study 5. (DOCX 18 kb)
Case study 6. (DOCX 22 kb)
Case analysis grid. (DOCX 15 kb)
Guidance for using the Social Impact Framework. (PPTX 80 kb)
About this article
Cite this article
Beckett, K., Farr, M., Kothari, A. et al. Embracing complexity and uncertainty to create impact: exploring the processes and transformative potential of co-produced research through development of a social impact model. Health Res Policy Sys 16, 118 (2018). https://doi.org/10.1186/s12961-018-0375-0
- Knowledge mobilisation
- integrated knowledge translation
- knowledge translation
- impact framework
- case studies