- Open Access
- Open Peer Review
The dark side of coproduction: do the costs outweigh the benefits for health research?
Health Research Policy and Systemsvolume 17, Article number: 33 (2019)
Coproduction, a collaborative model of research that includes stakeholders in the research process, has been widely advocated as a means of facilitating research use and impact. We summarise the arguments in favour of coproduction, the different approaches to establishing coproductive work and their costs, and offer some advice as to when and how to consider coproduction.
Despite the multiplicity of reasons and incentives to coproduce, there is little consensus about what coproduction is, why we do it, what effects we are trying to achieve, or the best coproduction techniques to achieve policy, practice or population health change. Furthermore, coproduction is not free risk or cost. Tensions can arise throughout coproduced research processes between the different interests involved. We identify five types of costs associated with coproduced research affecting the research itself, the research process, professional risks for researchers and stakeholders, personal risks for researchers and stakeholders, and risks to the wider cause of scholarship. Yet, these costs are rarely referred to in the literature, which generally calls for greater inclusion of stakeholders in research processes, focusing exclusively on potential positives. There are few tools to help researchers avoid or alleviate risks to themselves and their stakeholders.
First, we recommend identifying specific motivations for coproduction and clarifying exactly which outcomes are required for whom for any particular piece of research. Second, we suggest selecting strategies specifically designed to enable these outcomes to be achieved, and properly evaluated. Finally, in the absence of strong evidence about the impact and process of coproduction, we advise a cautious approach to coproduction. This would involve conscious and reflective research practice, evaluation of how coproduced research practices change outcomes, and exploration of the costs and benefits of coproduction. We propose some preliminary advice to help decide when coproduction is likely to be more or less useful.
It is now widely recognised that research evidence requires ‘translation’, beyond the standard academic journal article into some format which can be easily absorbed by policy-makers, practitioners or other ‘users’, if it is to have influence outside the academy [1,2,3]. In addition to interventions to increase knowledge translation and uptake , there has been a re-evaluation of what ‘counts’ as evidence for practice and policy [5, 6]. Moving away from its medical roots, the evidence-informed policy and practice movement and debate increasingly value experiential and practical knowledge [7,8,9,10,11,12]. Many researchers also feel that their research skills should be deployed in the service of those members of society who are less empowered – through, for example, participatory action research  – and that science in general should be made more inclusive and democratic [14,15,16].
To achieve these diverse aims, collaborative research practices have been welcomed and promoted by many as the best way to answer these challenges [17,18,19,20,21,22,23,24,25]. There are many forms of collaborative research practices, including coproduction, co-design, co-creation, stakeholder and public engagement, participation/involvement and integrated knowledge translation all sit under this umbrella , reflecting a very diverse set of motivations, activities, identities and discourses about how research interacts with the rest of society . Coproduction originally referred to the development of more responsive, personalised public services through “the joint working of people who are not in the same organisation to produce goods or services” [27,28,29]. These principles are increasingly applied to the production of knowledge, and coproduction is now a mainstream term in health research . In addition, researchers are required and supported by universities, funders and publishers to create impactful research, with ‘engagement’ as a routine section on most grant proposals. Similarly, integrated knowledge translation approaches have gained traction, led by the example of Canadian researchers, governments and funders (e.g. Canadian Institutes of Health Research).
Given the confluence of ethical, practical and procedural reasons in support of coproduction, one might think that there is a good level of consensus about what coproduction is, how to do it, what the effects are, and an evidence-based set of techniques to achieve these effects. In fact, there are very few evaluations of how coproduction works in practice or on the impact of coproduction on research, practice, policy or population outcomes . Given the pressures on researchers to coproduce in order to create research impact, we feel it is timely to closely examine the arguments and evidence for this. The purpose of this paper is to present the motivations for coproduction, identify associated tensions and costs, and identify where coproduction appears warranted.
There are four main arguments for coproduction in the literature:
Substantive: where engagement is undertaken to improve the quality of the research . This may be through helping researchers and policy-makers develop a more holistic understanding of a context, an issue and/or a solution [32, 33], both epistemologically and ontologically [25, 34]. The mechanisms by which research quality may improve are not always articulated, but many commentators agree that engagement makes research more relevant by focusing on appropriate topics which need elucidating [24, 35], and that by talking with others, we discover ‘unknown unknowns’  and create new knowledge together.
Instrumental: these arguments are based on “a desire to see research findings utilized in effective ways” (, p. 230). Many commentators claim that co-designed and coproduced research is likely to be more impactful [23, 24, 35, 37]. The actual process of collaborative research can help identify practice-based research questions and outcomes that are related to the implementation setting . This includes practical mechanisms such as upskilling and creating capacity amongst non-academics [29, 35], and creating a sense of trust and empowerment amongst potential users, thus increasing the likelihood of research utilisation  and of evidence sharing [40, 41]. Innovative research designs and feasible ways to collect data can be identified through collaboration . Further, stakeholders can describe how receptive the practice or policy setting is for subsequent implementation of the research findings .
Normative: where engagement is simply “of intrinsic value” (, p. 153). This set of arguments connects to the discourse of the civic or public university/academic, focusing on accountability to (public) funders, and the conduct of research to serve public interests. Second, authors argue that sharing expertise (in its guise as power) is simply a way to be fairer and more ethical [37, 45]. Flinders characterises this view as the belief that coproduction can be “transformative not solely in research terms but in social terms: the engagement of citizens and social groups nourishes the renewal of democracy” (, p. 261). This often sits alongside a view that mutual and continual learning is a virtue of collaborative research practices – a clear shift from the paternalistic ‘science advice’ model still prevalent (see e.g. [46, 47]). Of course, individuals may also view coproduction as a satisfying and enjoyable way to practise – a rarely articulated but, in our view, valid justification for attempting it.
Political: Often less explicitly made, but equally important, is a set of arguments used to justify coproduction on political grounds. By involving non-researchers in the elite, de facto exclusive process of research, coproduction can make users feel empowered and included [48, 49], and increase a sense of ownership (a prerequisite for acting on research findings) . The integrated knowledge translation view specifies partnering with research users who have the authority to implement the findings . Greater collaboration can reduce negative stereotypes held by researchers and policy-makers about each other , and improve trust [36, 52], smoothing the way for research to be more impactful. Several commentators also claim that coproduced research is more credible and relevant to intended audiences , and that the legitimacy of the knowledge , goals and activities  are increased. The motivation for either approach – collaborating with the disempowered or the powerful – is political, in the sense that it is about changing the attitudes of different groups to attain a particular end, even to the extent of being explicitly framed as a political and strategic response to diminishing belief in public services .
Coproductive approaches are therefore used for a variety of reasons. In the spirit of evidence-based enquiry, we ask whether coproduction is likely to enable researchers to realise these four objectives and at what cost.
How to coproduce?
There is a multiplicity of modes by which researchers may interact with stakeholders . At very low, shallow levels of involvement, stakeholders may be the mere recipients of research findings . More active interaction types may include reaching out through soft interpersonal networks to seek advice about policy agendas, or to influence policy dialogues individually . Researchers may be employed by public policy institutes to do research commissioned by and directly applicable to policy issues. At the most engaged level, what many would regard as ‘truly’ coproductive approaches have been tried in specific, partnered project grant calls, research institutes or long-standing research programmes (such as the English NHS Collaborations for Leadership in Applied Health Care and Research; CLAHRCs) [58,59,60,61] implying an equality of status between all co-researchers. Often, a financial contribution by the stakeholder is indicative of an authentic coproduction partnership. These coproductive approaches often include elements such as provision of opportunities to co-learn about research and substantive topics, to work iteratively over the lifespan of a project to co-steer the questions and direction of research, or to develop interpretations of data and implications of these together. However, there are very few evaluations of such approaches or projects, and existing evaluations often point to a “lack of hard evidence as to its impact on the quality of research” (, p. 3) and a need to “determin[e] best practices for engaging patients” (, p. 3). We concur with these findings.
As originally intended, coproduction referred to a significant shift of power from researchers or decision-makers to service users (e.g. patients) [26, 27], implying deeply embedded collaborative practices. However, it has come to refer to activities as diverse as consultation on topics, co-designing research questions, co-interpreting results and recommendations, and embedded or in-house researchers within policy or service delivery agencies . These activities describe very different modes of interaction, and imply different practical resourcing needs, skills and processes, as well as outcomes. However, there is very limited evidence about the impact of each type of strategy [25, 60, 63], no consensus about the best theories to use to inform this work , and little knowledge as to which strategies will help achieve which aims. Yet, millions of pounds are invested into deliberative, coproductive, collaborative research processes, and to related initiatives supporting research uptake and research impact, without (1) looking at what works or (2) knowing the pros and cons of different approaches or (3) evaluating and monitoring these processes. We might expect to see changes to policy or practice, to population outcomes, to research practices or to research outputs, yet no evaluation of coproduction attempts to explore all the domains holistically [65, 66]; this matters, because coproduction is not free of risk or cost. As Flinders et al. note , coproduction carries significant risks for academics, who are required to adopt practices far from those traditionally taught, adopted, recognised or rewarded by the academy. They argue that coproduction is “time-consuming, ethically complex, emotionally demanding, inherently unstable, vulnerable to external shocks, subject to competing demands and expectation” (, p. 266). In the next section, we explore how coproductive approaches may generate tensions through the research process and identify costs to participants.
Challenges and costs to coproduction
Challenges can arise throughout the lifecycle of a research project, and more generally through the research process (see Table 1 for a non-exhaustive list based on our own experiences). From the framing of research questions to the development and dissemination of recommendations, coproductive research can cause conflict, consume resources and lead to misunderstandings. The very purpose of a collaboration may not always be clear to all, or be shared. The direction of a research project may not (or possibly cannot) reflect the values and priorities of all stakeholders, and team members may have different views about what the research findings show, or what to do with them.
Beyond the lifecycle of an individual research project, coproductive research can also mean (1) investing time and resources into relationships with no guaranteed concrete output; (2) calling on favours and being able to provide favours in return; and (3) in general, having the capacity to manage sometimes tense relationships. Each of these challenges is also associated with costs for researchers and stakeholders. Below, we list and describe these costs (see Fig. 1 for a summary).
Firstly, there are clear practical costs to doing research in this way. It is expensive, as it requires the presence or time of multiple actors who are often not on site, have other primary responsibilities, or need travel or other reimbursement. The administrative burden of arranging spaces and attendance is significant. In addition, it takes skill to recruit and manage mixed groups who are working together to create something new. Managing group dynamics, not letting the loudest dominate and maintaining rules of conduct require interpersonal skills which not all academics are trained in or endowed with. Thus, a set of professional skills and resources are needed – but this type of work is often added on (to ‘real research’) with little thought for how to properly resource it . A gendered aspect may feature in coproduction work; our own experiences suggest that coproduction conferences are attended by women and coproduced projects are often led by women. Senior men may be keener to manage coproduction work with senior managers, clinicians and policy advisers than with lay people, patients, carers, etc. How, why and the consequences of this possible imbalance require further thought.
Second, there are significant personal costs to researchers. Many of the tensions above play out through interpersonal conflict, difficult conversations and/or outright disagreement. Many people find this very difficult to handle in the workplace, and when research careers, outputs and funding are at stake, this only adds to the pressure. There are reputational risks associated with conflict, which, for many, is reason enough to avoid difficulties; yet, ignoring these challenges and going along with stakeholder views with little negotiation of mutual interests can lead to unnecessary duplicative research and perhaps less imaginative or derivative research, which can also compromise research(er) integrity. It is too easy to dismiss these tensions as being because of interpersonal tensions; rather, we argue that these tensions are how inherent power imbalances and conflicts in coproductive research are expressed. Badging them as being because of interpersonal difficulties dismisses the emotional labour of working collaboratively, and risks ignoring the need to find solutions for real challenges inherent in this type of research practice. Managing the engagement process takes personal qualities that are not hugely abundant amongst the academic population. These interpersonal conflicts can lead to – as can the additional work associated with coproduction – serious stress and burnout .
Third, there are professional costs to researchers. Taking time out of seeking research funding, writing publications in impactful journals, doing administration, and teaching is a huge ask for academics, and coproduction requires significant commitment of time and resources, often with little guarantee of success or recognition. Coproduction can lead to research outputs that are regarded as being of lower quality than ‘real’ or ‘pure’ research , or simply hard to publish. For engagement to occur successfully, researchers are often required to do professional favours (e.g. do extra presentations), be easily and readily available, and be able to engage with no expectation of a guaranteed impact as measured in academia. This is clearly not the case for all researchers, and may lead to the costs of coproduction being borne by those least able in terms of time or capacity – in our experience, often the most junior, temporary, female and/or of colour.
Additionally, researchers risk being seen as partisan and/or lacking in credibility [68, 69], by both their colleagues and by their stakeholders. Researchers may find they are being used to add legitimacy to political positions  and may find that adopting political or policy positions, or being seen to endorse political viewpoints, can hinder their ability to work with a range of stakeholders – even compromising personal safety . It can also lead to the view that the research itself is not a neutral representation of the data, but a cherry-picked narrative to support a political position – producing “policy-based evidence” . Finally, as well as being regarded as partisan and biased, coproductive researchers risk being regarded as an academic “lightweight” , producing nothing of substance.
Fourth, there are risks to the research itself. Under business-as-usual rules, researchers spend their time identifying genuine and novel gaps in the knowledge base, which have to be justified at length to colleagues and funders. However, the coproduction process can lead to researchers being asked to answer questions which are dull, not novel (little contribution to the scientific literature), or not generalisable (focused on local issues) – and therefore not easily publishable. As Kothari indicates, coproductive research can “encourage a reassertion of control and power by dominant individuals and groups” who possess the skills to engage with the project, or those whose contributions fit in with any preconceived ideas of the ‘right’ responses will be highlighted over those who “lack the skills to perform as required” (: p. 142). This can lead to research being conducted on unnecessary or repetitious topics, or on topics which serve the interests of some groups over others. Of course, this is often the case, but coproduction offers a vision of equality and fairness which can mask these dynamics.
The research process can be significantly affected, with delays caused by the additional work of recruiting and engaging with stakeholders; differences of opinion about the purpose and role of coproduction or the research itself; tensions around how to frame interpretations and recommendations that may result in research being co-opted or researchers being silenced (at a micro level, during conversations, or procedurally through being prevented from publishing); and in how and when to share the findings. These concerns are not new to implementation science or ethnographic researchers (see, for example [75,76,77]) and the tensions around the research process itself have long been a domain of inquiry for social researchers. Yet, in the current climate, the culture of hit-and-run research (get funding, do research, achieve impact, leave) does not easily allow researchers the time and space to reflect on their practice, and any proffered thoughts would be considered incidental to the main project, irrelevant or possibly unwanted by funders.
Fifth, there are costs to stakeholders. In addition to taking time and resources away from their primary responsibilities, coproduction requires that participants – especially policy-makers – make themselves vulnerable, by sharing uncertainties or sensitive information . This could include admitting the political uncertainty, lack of policy direction, or political mistakes – all of which could incur costs. More broadly, one can imagine scenarios where participants were asked for input, which was then cherry-picked to suit research agendas, or where their contributions were not treated transparently and systematically. Policy-makers, service users and professionals can feel ignored or used, may not understand why their voices are not represented in the way they had expected, or may feel that they were not accorded the usual respect they enjoy in the workplace. Far from empowering people politically, participation in research can lead to personal narratives and experiences being dominated by senior interests, leading to lack of motivation to engage again , and the ‘subjugation’ of participants [78, 79].
Finally, there are costs to the entire profession, or the ‘scientific endeavour’, by the reduction of trust in science and scientists. Doing coproduction recklessly, discourteously or without due attention to professional etiquette can cause significant ill-feeling about participating in research. This can lead to mistrust of researchers, and a lack of willingness to engage in other research processes. Furthermore, the issues about independence and credibility can call the value of academic research into question. If researchers are working so closely with parties with clearly vested interests in particular aims, how can anyone be confident that the findings have been represented truly and holistically? This can lead to research being viewed as simply one more lobby group, rather than holding a special status .
A cautious approach to coproduction: how to minimise the costs and maximise the benefits of coproduction
We recognise the transformative potential of coproduced research, and also feel that calls to do impactful research are unlikely to go away. If done well, coproduced research processes can – indeed must – manage and alleviate these tensions. Yet, there is a significant dearth of empirical evidence about coproduction processes or outcomes, which makes its widespread advocacy troubling. We simply do not know which strategies are the most promising for collaborative research. For example, which stages of the research process are critical for stakeholder involvement, and which stages are less essential? What are the best ways to keep stakeholders engaged with the process? Under what circumstances should coproduction be used, rather than other (e.g. consultative) approaches? What types of infrastructure need to be in place to support productive coproduction?
What then should be done? We suggest a two-step process when commissioning and/or initiating each research project, as follows:
First, considering whether coproduction is likely to be useful in helping the research meet its aims and selecting strategies accordingly, and
Second, considering whether other approaches are as or more likely to help achieve those aims.
This process is likely to be the responsibility not just of the researchers but also policy-makers, practitioners, funders and patient/user representatives.
Using the four arguments for coproduction above, researchers and others (that is, funders, policy-makers, managers, clinicians, the public, patients, carers, etc., depending on the topic) should identify specific potential aims for any coproduction process – for example, to identify or refine a research question, or to broker access to a political community – and decide whether coproductive strategies can best help them achieve those aims, and if so which ones. Having identified the purpose of the research and its desired outcomes in relation to the context and the interests potentially affected by the findings, the researchers and others should attempt to identify the best strategies to achieve these, and thus decide when and where coproduction might help, and when and where it might not.
Given the lack of much evidence in this area, we have to make judgements based on the circumstances of each project. Some early thinking along these lines is summarised in Table 2, which particularly relates to coproduction of evaluations with policy-makers and service managers.
We suggest that a cautious approach to coproduction would include examining the costs and benefits to all involved, recognising the significant costs and risks to investing time and resources into good facilitation and management of expectations, establishing ground rules and processes, and deciding on evidence-informed strategies to achieve established and shared aims and outcomes. Below, we summarise some key questions which we believe all those involved in coproduction should ask themselves prior to attempting any engagement.
Individual researchers, funders, commissioners and participants in the research process should ask themselves:
What is everyone bringing to the table? For example, policy-makers and funders bring money, knowledge of the political context, pressure for answers; researchers bring topic and methodological expertise; public and patients bring their experiences.
Under which circumstances are these needed, for what purpose, and at which stage of the research process? For example, when is it better to have patient representatives articulate the user perspective rather than derive understanding from a systematic review of patient experiences? Should policy-makers be involved all the way through, or only at specific stages?
What are the costs? How will the time, administrative, cultural and professional costs be borne and defrayed by those involved?
How are decisions taken, and how will responsibility and accountability be shared? Will group dynamics, market forces, authority or some other decision-making process control the process? How will this be governed and managed?
Research organisations and funders need to consider:
How to create (co-create) and support the infrastructure and leadership for coproduction
How to provide training in coproduction, and help researchers and funders take this seriously as a skill set
How to reward good practice, and to recognise the work coproduction may take even if it does not lead to impact
How to evaluate the potential impact(s) of coproduction
How to ensure that coproduction supports, rather than mitigates against diversity and quality in research and policy
Coproduction is an exciting approach to research that can generate truly novel, unexpected outputs . However, it takes investment, skills, time and courtesy. Engaging stakeholders should be done for the right reasons and in the right way , yet there is so little empirical evidence about how coproduction changes research, policy or practice, or how it may compare to alternatives. Coproduction as a way to generate knowledge can be seen as one way to make the resulting research more ‘implementation ready’ on the assumption that, through the coproduction process, the team has already anticipated research users’ needs, capacities and priorities. Yet, there may be alternative ways to achieve this outcome without risking the costs detailed above.
The main motivations of researchers to engage in coproductive work are of course positive, with genuine belief in the importance and potential transformational nature of working with those outside academia. However, the political arguments are often not made explicitly (see, e.g. ). To us, this suggests that, either some researchers engage naively (albeit in good faith), or that researchers appreciate the political nature of decision-making and research, but prefer not to openly and honestly explain their strategies to deal with these dynamics. We also note that most normative, substantive and instrumental justifications for coproduction can be achieved through other means. Thus, we argue that political reasons for engaging in coproductive research may be the least-discussed, yet most important rationale made by researchers. Given the costs, we argue that this requires serious attention by the research community, including those who commission, participate in and evaluate research.
Coproduction and participation may have a profound impact on the practice of research and the process of decision-making, and there are many promising and exciting examples of joint inquiry between researchers and stakeholders [30, 59, 81,82,83]. There are certainly many unanswered questions about the potential value of coproduction and collaborative research, and outlining the benefits and the mechanisms generating those benefits is an important task.
However, the tensions and challenges, costs and opportunities must also be described; they are unequally experienced and borne, strongly suggesting that mindful engagement is essential for the ethical practice of research. Many of these points of tension and costs are about power, and we recognise that all the implementation type tools, frameworks and processes cannot design away these tensions. There must be a more reflective, open discussion about when to do coproduction, the ethics of coproduction, and so on; we need more empirical evidence about its processes and outcomes; and we do not know which types are best suited for which problems, institutions or team configurations. As a community, it is imperative that we are more reflective about how coproduction influences the process of research, and the roles and responsibilities of everyone involved in collaborations .
Traynor R, DeCorby K, Dobbins M. Knowledge brokering in public health: a tale of two studies. Public Health. 2014;128(6):533–44.
Connelly S, Vanderhoven D. Translation across borders: exploring the use, relevance and impact of Academic Research in the Policy Process: Policy Press; 2015. p. 181–98.
Lane JP, Rogers JD. Engaging national organizations for knowledge translation: comparative case studies in knowledge value mapping. Implement Sci. 2011;6(1):106.
Boaz A, Baeza J, Fraser A. Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Research Notes. 2011;4:212.
Cairney P. The Politics of Evidence-Based Policy Making. London: Palgrave Macmillan UK; 2015.
Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12(1):34.
Fre Z. Knowledge Sovereignty among African Cattle Herders. London: UCL Press; 2018.
Elliott H, Popay J. How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Community Health. 2000;54(6):461–8.
Russ LW, Takahashi LM, Ho W, Tseng W, Ponce NA. Bridging academic-legislative divides: models of policy-relevant health research and practice by the university of california. Prog Community Heal Partnerships Res Educ Action. 2012;6(1):95–102.
Wieringa S, Greenhalgh T. 10 years of mindlines: a systematic review and commentary. Implement Sci. 2015;10:45.
Greenhalgh T. Narrative based medicine: narrative based medicine in an evidence based world. BMJ. 1999;318(7179):323–5.
Oliver S, Duncan S. Editorial: The challenges of sharing different ways of knowing. Res All. 2018;2(1):1–5.
Langley J, Wolstenholme D, Cooke J. ‘Collective making’ as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Serv Res. 2018;18(1):585.
Sabine M, Peter W, Maasen S, Weingart P. Democratization of Expertise? - Exploring novel forms of scientific advice in political decision-making: Springer; 2005. p. 238.
Carrozza C. Democratizing Expertise and Environmental Governance: Different Approaches to the Politics of Science and their Relevance for Policy Analysis. J Environ Policy Plan. 2015;17(1):108–26.
Stewart R. Terminology and tensions within evidence- informed decision-making in South Africa over a 15-year period. Res All. 2017;1(2):252–64.
Oliver K, Cairney P. The dos and don’ts of influencing policy: a systematic review of advice to academics. Palgrave Commun. 2019;5(1):21.
Durose C, Mangan C, Needham C, Rees J, Hilton M. Transforming local public services through co-production. Swindon: Arts and Humanities Research Council; 2013.
Filipe A, Renedo A, Marston C. The co-production of what? Knowledge, values, and social relations in health care. PLoS Biol. 2017;15(5):e2001403.
Traynor R, Dobbins M, DeCorby K. Challenges of partnership research: insights from a collaborative partnership in evidence-informed public health decision making. Evid Policy A J Res Debate Pract. 2015;11(1):99–109.
Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M, et al. Patient and service user engagement in research: A systematic review and synthesized framework. Health Expect. 2015;18(5):1151–66.
Boivin A, L’Espérance A, Gauvin F-P, Dumez V, Macaulay AC, Lehoux P, et al. Patient and public engagement in research and health system decision making: A systematic review of evaluation tools. Heal Expect. 2018;21(6):1075–84.
Oliver S, Liabo K, Stewart R, Rees R. Public involvement in research: making sense of the diversity. J Health Serv Res Policy. 2015;20(1):45–51.
Barber R, Beresford P, Boote J, Cooper C, Faulkner A. Evaluating the impact of service user involvement on research: a prospective case study. Int J Consum Stud. 2011;35(6):609–15.
Fransman J. Charting a course to an emerging field of ‘research engagement studies’: A conceptual meta-synthesis. Res All. 2018;2(2):1–49.
Flinders M, Wood M, Cunningham M. The politics of co-production: risks, limits and pollution. Evid Policy A J Res Debate Pract. 2016;12(2):261–79.
Ostrom E. Crossing the great divide: coproduction, synergy, and development. World Dev. 1996;24(6):1073–87.
Durose C, Needham C, Mangan C, Rees J. Generating “good enough” evidence for co-production. Evid Policy. 2017;13:135–51.
Iedema R, Carroll K. The “clinalyst.”. J Organ Chang Manag. 2011;24(2):175–90.
Gagliardi A, Kothari A, Graham I, Gagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: What we know and do not yet know. J Epidemiol Community Health. 2017;71(2):105–6.
Duncan S, Oliver S. Editorial: Motivations for engagement. Res All. 2017;1(2):229–33.
Walter I, Nutley S, Davies H. Research impact: a cross sector literature review. St. Andrews: Research Unit for Research Utilisation, Department of Management University of St. Andrews; 2003.
Oliver K, Rees R, Brady LM, Kavanagh J, Oliver S, Thomas J. Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews. Res Synth Methods. 2015;6(2):206–17.
Liabo K, Stewart R. Involvement in research without compromising research quality. J Health Serv Res Policy. 2012;17:248–51.
Goodyear-Smith F, Jackson C, Greenhalgh T. Co-design and implementation research: challenges and solutions for ethics committees. BMC Med Ethics. 2015;16:78.
Maybin J. How proximity and trust are key factors in getting research to feed into policymaking. LSE Impact Blog. 2016; http://blogs.lse.ac.uk/impactofsocialsciences/2016/09/12/how-proximity-and-trust-are-key-factors-in-getting-research-to-feed-into-policymaking/. Accessed 1 Aug 2018.
Nesta. Seven Principles for Public Engagement in Research and Innovation Policymaking: Nesta; 2018. https://www.nesta.org.uk/report/seven-principles-public-engagement-research-and-innovation-policymaking/. Accessed 15 Dec 2018.
Rycroft-Malone J, Burton C, Wilkinson J, Harvey G, McCormack B, Baker R. Collective action for knowledge mobilisation: a realist evaluation of the Collaborations for Leadership in Applied Health Research and Care. Health Services Delivery Research. Southampton: NIHR Journals Library; 2015.
Ghate D. Developing theories of change for social programmes: co-producing evidence-supported quality improvement. Palgrave Commun. 2018;4(1):90.
Dobbins M, Robeson P, Ciliska D, Hanna S, Cameron R, O’Mara L, et al. A description of a knowledge broker role implemented as part of a randomized controlled trial evaluating three knowledge translation strategies. Implement Sci. 2009;4:23.
Armstrong R, Waters E, Roberts H, Oliver S, Popay J. The role and theoretical evolution of knowledge translation and exchange in public health. J Public Health. 2006;28:384–9.
Bowen S, Martens P. Demystifying knowledge translation: learning from the community. J Health Serv Res Policy. 2005;10(4):203–11.
Gagliardi A, Dobrow MJ. Identifying the conditions needed for integrated knowledge translation (IKT) in health care organizations: qualitative interviews with researchers and research users. BMC Health Serv Res. 2016;16:256.
Conklin A, Morris Z, Nolte E. What is the evidence base for public involvement in health-care policy?: Results of a systematic scoping review. Health Expect. 2015;18(2):153–65.
Doubleday RRR, Wynne B. Despotism and Democracy in the United Kingdom. In: Jasanoff S, Caplan A, editors. Reframing Rights: Bioconstitutionalism in the Genetic Age; 2011. p. 239–61.
Gluckman P. The art of science advice to the government. Nature. 2014;507:163–5.
Sutherland WJ, Burgman MA. Policy advice: use experts wisely. Nature. 2015;526:317–8.
Muir Gray JA. Evidence based policy making. BMJ. 2004;329(7473):988–9.
Beresford P. Developing the Theoretical Basis for Service User/Survivor-Led Research. Epidemiol Psychiatr Sci. 2005;14(1):4–9.
Kothari A, McCutcheon C, Graham I. Defining integrated knowledge translation and moving forward: a response to recent commentaries. Int J Heal Policy Manag. 2017;6(x):1–2.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14(1):–2.
Boaz A, Hanney S, Borst R, O’Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Syst. 2018;16(1):60.
Cash DW, Clark WC, Alcock F, Dickson NM, Eckley N, Guston DH, et al. Knowledge systems for sustainable development. Proc Natl Acad Sci U S A. 2003;100(14):8086–91.
Di Giulio A, Defila R. Lessons from “Real-world Laboratories” about Transdisciplinary Projects, Transformative Research and Participation. https://i2insights.org/2018/03/22/lessons-from-real-world-laboratories/. Accessed 15 Dec 2018.
Iedema R, Merrick E, Piper D, Britton K, Gray J, Verma R, et al. Codesigning as a discursive practice in emergency health services: the architecture of deliberation. J Appl Behav Sci. 2010;46(1):73–91.
Martin S. Co-production of social research: strategies for engaged scholarship. Public Money Manag. 2010;30(4):211–8.
Oliver K, Faul MV. Networks and network analysis in evidence, policy and practice. Evid Policy. 2018;14(3):369–79.
Rycroft-Malone J, Seers K, Chandler J, Hawkes CA, Crichton N, Allen C, et al. The role of evidence, context, and facilitation in an implementation trial: Implications for the development of the PARIHS framework. Implement Sci. 2013;8(1):28.
Kislov R, Wilson PM, Knowles S, Boaden R. Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations. Implement Sci. 2018;13(1):111.
Oborn E, Barrett M, Prince K, Racko G. Balancing exploration and exploitation in transferring research into practice: A comparison of five knowledge translation entity archetypes. Implement Sci. 2013;8(1):104.
Rushmer R, Shucksmith J. AskFuse origins: system barriers to providing the research that public health practice and policy partners say they need. Evid Policy. 2018;14(1):81–101.
Canadian Institutes of Health Reseacrh. Evaluation of the Strategy for Patient-Oriented Research. http://www.cihr-irsc.gc.ca/e/49937.html. Accessed 18 Mar 2019
Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient. 2014;7(4):387–95.
Watermeyer R. Challenges for university engagement in the UK: towards a public academe? High Educ Q. 2011;65(4):386–410.
Camden C, Shikako-Thomas K, Nguyen T, Al E. Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disabil Rehabil. 2015;37(15):1390–400.
Gagliardi A, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38.
Graffy EA. Enhancing policy-relevance without burning up or burning out: A strategy for scientists. In: Science into Policy: Water in the Public Realm: Proceedings, Specialty Conference, June 30-July 2, 1999, Bozeman, Montana. Herndon: American Water Resources Association; 1999. p. 293–8.
Behrens TR, Gray DO. Unintended consequences of cooperative research: impact of industry sponsorship on climate for academic freedom and other graduate student outcome. Res Policy. 2001;30(2):179–99.
Lidskog R, Sundqvist G. From consensus to credibility: new challenges for policy-relevant science. Innovation. 2004;17(3):205–26.
Himmrich J. How should academics interact with policy makers? Lessons on building a long-term advocacy strategy. LSE Impact Blog. 2016; http://blogs.lse.ac.uk/impactofsocialsciences/2016/06/20/how-should-academics-interact-with-policy-makers-lessons-on-building-a-longterm-advocacy-strategy/. Accessed 10 Jul 2018.
Zevallos Z. Protecting Activist Academics Against Public Harassment. Other Sociologist. 2017; https://othersociologist.com/2017/07/06/activist-academics-public-harassment/. Accessed 18 Mar 2019.
Davey Smith G, Ebrahim S, Frankel S. How policy informs the evidence. BMJ. 2001;322(7280):184–5.
Maynard A. Is public engagement really career limiting? Times High Education. 2015; https://www.timeshighereducation.com/blog/public-engagement-really-career-limiting. Accessed 18 Mar 2019.
Kothari Uma. “Power, Knowledge and Social Control in Participatory Development”. In: Bill Cooke and Uma Kothari. Participation: The New Tyranny? London: Zed Books; 2001. pp. 142-143
Spano R. Potential sources of observer bias in police observational data. Soc Sci Res. 2005;34(3):591–617.
Kanuha VK. “Being” native versus “going native”: conducting social work research as an insider. Soc Work. 2000;45(5):439–47.
Bell C, Encel S. Inside the Whale: Ten Personal Accounts of Social Research. Oxford: Pergamon Press; 1978.
Aarts N, Leeuwis C. Participation and power: reflections on the role of government in land use planning and rural development. J Agric Educ Ext. 2010;16(2):131–45.
Cooke B, Kothari U. Participation: The New Tyranny? London: Zed Books; 2001.
Gluckman P. Policy: the art of science advice to government. Nature. 2014;507(7491):163–5.
DuMont K. Reframing Evidence-Based Policy to Align with the Evidence: William T. Grant Foundation; 2019. http://wtgrantfoundation.org/digest/reframing-evidence-based-policy-to-align-with-the-evidence. Accessed 28 Jan 2019.
Prainsack B. The “we” in the “me”: solidarity and health care in the era of personalized medicine. Sci Technol Hum Values. 2018;43(1):21–44.
Callard F, Rose D, Wykes T. Close to the bench as well as at the bedside: involving service users in all phases of translational research. Health Expect. 2012;15(4):389–400.
Callard F, Fitzgerald D. Rethinking Interdisciplinarity Across the Social Sciences and Neurosciences. Wellcome Trust–Funded Monographs and Book Chapters. Basingstoke: Palgrave Macmillan; 2015.
We thank all attendees of the UK Implementation Science conference 2018 who saw and commented on an early version of this paper.
This paper was not supported by any current funding.
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.