Skip to main content

Advertisement

We’d like to understand how you use our websites in order to improve them. Register your interest.
Download PDF

How to engage stakeholders in research: design principles to support improvement

Health Research Policy and Systems volume 16, Article number: 60 (2018) Cite this article

Abstract

Background

Closing the gap between research production and research use is a key challenge for the health research system. Stakeholder engagement is being increasingly promoted across the board by health research funding organisations, and indeed by many researchers themselves, as an important pathway to achieving impact. This opinion piece draws on a study of stakeholder engagement in research and a systematic literature search conducted as part of the study.

Main body

This paper provides a short conceptualisation of stakeholder engagement, followed by ‘design principles’ that we put forward based on a combination of existing literature and new empirical insights from our recently completed longitudinal study of stakeholder engagement. The design principles for stakeholder engagement are organised into three groups, namely organisational, values and practices. The organisational principles are to clarify the objectives of stakeholder engagement; embed stakeholder engagement in a framework or model of research use; identify the necessary resources for stakeholder engagement; put in place plans for organisational learning and rewarding of effective stakeholder engagement; and to recognise that some stakeholders have the potential to play a key role. The principles relating to values are to foster shared commitment to the values and objectives of stakeholder engagement in the project team; share understanding that stakeholder engagement is often about more than individuals; encourage individual stakeholders and their organisations to value engagement; recognise potential tension between productivity and inclusion; and to generate a shared commitment to sustained and continuous stakeholder engagement. Finally, in terms of practices, the principles suggest that it is important to plan stakeholder engagement activity as part of the research programme of work; build flexibility within the research process to accommodate engagement and the outcomes of engagement; consider how input from stakeholders can be gathered systematically to meet objectives; consider how input from stakeholders can be collated, analysed and used; and to recognise that identification and involvement of stakeholders is an iterative and ongoing process.

Conclusion

It is anticipated that the principles will be useful in planning stakeholder engagement activity within research programmes and in monitoring and evaluating stakeholder engagement. A next step will be to address the remaining gap in the stakeholder engagement literature concerned with how we assess the impact of stakeholder engagement on research use.

Peer Review reports

Background

Closing the gap between research production and research use is a key challenge for the health research system. Stakeholder engagement is being increasingly promoted across the board by health research funding organisations, and indeed by many researchers themselves, as an important pathway to achieving impact [1]. The literature is diverse, with a rapidly expanding but still relatively small number of papers specifically referring to ‘stakeholder engagement’, and a larger number of publications discussing issues that at least partly overlap with stakeholder engagement. Several of the papers explicitly analysing stakeholder engagement come from the field of environmental research (e.g. Jolibert and Wesselink [2], Phillipson et al. [3]). However, stakeholder engagement is also gaining traction in the health field. A recent supplement in this journal consolidated learning relating to tools and approaches to stakeholder engagement within the United Kingdom Department for International Development’s Future Health Systems research consortium [4]. In particular, in health, there is an important stream of analysis from North America. A review for the United States Agency for Healthcare Research and Quality drew on papers from a range of fields [5].

This opinion piece provides a short conceptualisation of stakeholder engagement, followed by ‘design principles’ that we put forward based on a combination of existing literature and new empirical insights from our recently completed longitudinal study of stakeholder engagement in research. We have drawn on a systematic literature search conducted to inform the wider study and in particular to conceptualise stakeholder engagement (Additional file 1).

Literature review

Conceptualising stakeholder engagement: What does the literature say?

Stakeholders have been defined as “individuals, organizations or communities that have a direct interest in the process and outcomes of a project, research or policy endeavor” ([6], p. 5). In seeking to conceptualise stakeholders, Concannon et al. [7] developed the 7Ps Framework to identify stakeholders in Patient-Centered Outcomes Research and Comparative Effectiveness Research in the United States of America. The 7Ps are patients and the public, providers, purchasers, payers, public policy-makers and policy advocates working in the non-governmental sector, product makers, and principal investigators. The seven categories signal an overlap with the large literature on patient and public involvement (PPI) in research. However, our focus here is on multi-stakeholder engagement, where diverse groups of stakeholders take part in the research process. Deverka et al. [6] define engagement as “an iterative process of actively soliciting the knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interest in a particular issue, for the dual purposes of: creating a shared understanding; making relevant, transparent and effective decisions” ([6], p. 5).

Roles, activities and phases of stakeholder engagement: What does the literature say?

There are additional issues about the definition of stakeholder engagement when the nature of the engagement activities is considered. For example, there are issues about how far co-creation/participatory action research approaches can be considered to be stakeholder engagement or something so far beyond the usual stakeholder engagement that they are really in a different category [8]. Similarly, there is a large and currently distinct literature on PPI in research [9], including the development of reporting guidelines such as GRIPP2 [10]. There are a number of parallels in the issues discussed in these literatures as well as some interesting differences (particularly in terms of power inequalities). However, herein, we conceptualise PPI as a subset of stakeholder engagement in-line with most of the literature, including Concannon et al. [7].

Most of the stakeholder engagement literature highlights the broad range of activities in which stakeholders can engage depending on their own skills and attributes and the capacity and wishes of the researchers conducting specific studies. At the broadest level of a research system, or research funding body, Lomas [11] claimed there were many activities in which stakeholders could be engaged in a ‘linkage and exchange’ approach for health services research. These were setting priorities, funding programmes, assessing applications, conducting research and communicating findings. The importance of engaging a wide range of stakeholders in priority-setting has often been emphasised. The pioneering study by Kogan and Henkel [12] analysed both the importance of engaging policy-makers in setting research agendas to meet their needs, and the obstacles to making the process work well. These obstacles included issues around how far the assessment of needs-based research should focus on the relevance and practical impact of the research as well as its scientific merit. Many of the more recent studies explicitly examining stakeholder engagement also set out a range of activities in which stakeholders may be involved. These are often related to phases of the research processes. Concannon et al. [7] provide a list of roles related to stages and used the identified roles in a subsequent review [13].

Knowledge translation (KT) is one of many terms used to describe efforts to ensure research evidence is used to inform decision-making [14]. Although the importance of engaging stakeholders in KT is recognised, it has been acknowledged that stakeholder engagement is often overlooked in favour of more conventional dissemination strategies [15]. Integrated KT has been developed as an approach to collaborative research in which researchers work with stakeholders who identify a problem and have the influence and sometimes authority to implement the knowledge generated through research [16, 17]. Grimshaw et al. [14] argue that different groups of stakeholders are likely to be engaged depending on the type of research that is being translated.

Assessing the impact of stakeholder engagement: What does the literature say?

A final consideration about the nature of the body of literature specifically on stakeholder engagement is that not only is it still quite limited in total, but there are also notable areas where authors claim it is particularly sparse. In particular, Hinchcliff et al. [18] examined the literature on multi-stakeholder health services research collaborations in an attempt to address the question of whether it was worth investing in them. They identified very few studies (Harvey et al’s. [19] 2011 evaluation of a Collaboration for Leadership in Applied Health Research and Care being one exception) and concluded that their generalisability was questionable. They therefore suggested that “The lack of reliable evidence compels implementers to rely largely on trial and error, risking variable success” ([18], p. 124).

The nature of engagement activity is less contentious than the arguments about its potential impact. Research impacts on non-academic audiences are defined by the United Kingdom Higher Education Funding Council as: “benefits to one or more areas of the economy, society, culture, public policy and services, health, production, environment, international development or quality of life, whether locally, regionally, nationally or internationally” [20]. Various studies have attempted to assess a range of impacts of research (especially health research) and/or attempted to identify facilitators and barriers of research use in policy-making. There are also a growing number of reviews of such studies [21,22,23,24,25,26,27]. While these are not explicitly studies of stakeholder engagement, many of them have identified some form of collaboration between researchers and users as one of the factors most likely to lead to the research making an impact. However, this wider range of literature does not go into detail in terms of analysing the nature of the processes of stakeholder engagement that leads to impact.

Studies specifically focusing on the impact of stakeholder engagement are less common, although it is a growing area of interest [28, 29]. Jolibert and Wesselink [2] found a few examples of impact, but suggested ways to increase impact through what they describe as sustained interactions. Concannon et al. concluded that approximately 20% of their study participants “reported that stakeholder engagement improved the relevance of research, increased stakeholder trust... enhanced mutual learning by stakeholders, and researchers about each other, or improved research adoption” ([13], p. 1697), whereas 6% reported improved transparency and 9% increased understanding of research processes. Also, while Forsythe et al. referred to a lack of evidence about impact, they also observed that “Commonly reported contributions included changes to project methods, outcomes or goals; improvement of measurement tools; and interpretation of qualitative data” ([30], p. 13). In the United States, the Center for Medical Technology Policy website makes a strong statement about the impact of stakeholder engagement: “Including the perspectives of all key stakeholders has powerful benefits, enhancing both the short- and long-term relevance of clinical research efforts” [31].

Assessing the impact of stakeholder engagement: a new study

Given the diversity of stakeholder engagement and the thin evidence base for its impact, our study set out to identify a set of indicators that might be used to identify stakeholder engagement with potential for impact. We identified a study called the European study on Quantifying Utility of Investment in Protection from Tobacco (EQUIPT) and then conducted our own study, Stakeholder Engagement in EQUIPT (SEE-Impact) as a prospective study of stakeholder engagement running alongside. EQUIPT, a major European Commission (EC) – funded project, aimed to achieve impact through extensive stakeholder engagement. Both studies are briefly described in Box 1.

The results of the EQUIPT study have now been published [32, 33] and a full account of the main methods from SEE-Impact have been submitted for publication. Papers on the full findings are being finalised. Herein, our aim is to address the statement in our original funding proposal in 2013 that it should be possible to identify aspects of the stakeholder engagement (and perhaps other features of the processes) that might be viewed as intermediate indicators of the eventual impact achieved.

Our analysis of the complex and nuanced process of stakeholder engagement has resulted not in a list of indicators, but in a set of design principles. We hope that these design principles will help to inform the future development of stakeholder engagement as a mechanism for promoting research impact. These principles, rooted in both the existing literature and in the findings from our prospective study of stakeholder engagement, are intended to inform the planning and delivery of stakeholder engagement activities. It is anticipated that they will also provide a structure for building a narrative account of stakeholder engagement as part of an evaluation of an individual project or programme. They might also provide a starting point for the development of future indicators.

Design principles for stakeholder engagement in research

The project team (comprising members of the SEE-Impact research team and collaborators from EQUIPT) met for a 2 day analysis workshop. One aim of the workshop was to begin to build a consensus among the team on what seemed to be the key design principles emerging from the SEE-Impact data and the on-going literature review. SEE-Impact data included observational data, interviews and document analysis. The research team continued to develop the principles through an ongoing period of deliberation, informed by the impact study and the literature. As part of this process, the principles were categorised into three groups, namely organisational, values and practices.

In this section, we first present empirical evidence from the SEE-Impact study that informed our development of the design principles. We then briefly summarise published evidence for each group of design principles in order to situate them in the wider literature.

Design principles and empirical evidence from the stakeholder engagement in EQUIPT for impact (SEE-impact) study

The stakeholder engagement study (SEE-Impact) and the project being studied (EQUIPT) are described in Box 1. In terms of the organisational level principles, the EQUIPT project objectives for stakeholder engagement were clear, as set out in the proposal, protocol and project documents [34]. The key aims of stakeholder engagement activity were to access the knowledge and skills (described in the protocol as co-creation innovation in the working space) and to increase influence and impact (described in the protocol as dissemination innovation in the transfer space through stakeholder engagement).

In terms of values, the commitment to stakeholder engagement was more clearly demonstrated by some of the EQUIPT project team members than others. For some team members, previous successful experience of an interactive form of working with stakeholders had built a commitment to this particular way of working. It also provided experience of practical elements of working with stakeholders, but perhaps most importantly lived experience of the practical benefits of engagement. For other members of the team, too, working with stakeholders fitted closely with their ethos and values. For example, the Hungarian team talked about their pragmatic approach to research and the need to conduct useful and usable research, with stakeholder engagement being a key component. However, a small group within the wider project team did not seem committed to ensuring stakeholder engagement remained a core element of the project. They favoured a particular, individualised approach to stakeholders and, over time, partially reshaped the stakeholder engagement activities to something more akin to research participation (that is, taking part in a research study as a means of generating specific data as determined by researchers, rather than as co-producers of research). Finally, not all stakeholders identified by the project team were interested in engaging with the project. In particular, the lack of policy priority given to smoking cessation (the focus of the return on investment (ROI) tool) made engagement of policy stakeholders in the Netherlands very difficult to achieve.

In terms of practices, while the EQUIPT project protocol did set out how the stakeholder engagement would operate [34], there was not as much flexibility as the investigators would have liked in terms of the project plan and this had an impact on the nature of the stakeholder engagement activities. In particular, time intensive methods of engagement originally proposed in the protocol (particularly the large number of face-to-face meetings) began to look unrealistic to members of the team. The lack of flexibility came in part from the funder. The EC told the project team at an early point that there was no scope for negotiation around the project end date. Thus, initial delays in the project put a strain on the project timetable and deliverables. Members of the team proposed a shift from face-to-face meetings with stakeholders to Skype meetings in an effort to ‘catch up’. The technical team producing the new version of the ROI tool for roll out in Europe added to a sense of urgency in ‘speeding up’ the stakeholder engagement work with their need for data to feed into their work. Nevertheless, despite the practical difficulties, in EQUIPT, a significant amount of consideration had been given to stakeholder engagement, including planning how the input provided by stakeholders might be gathered, collated, analysed and used. Vokó et al. highlight that it is important to “fully analyse several aspects of stakeholder engagement in research” ([32], p. 15) and note that there is a tendency to ignore the value of early stakeholder engagement when it comes to development and transferability in the work of economic evaluation. EQUIPT’s careful consideration and the methods adopted facilitated a much more rigorous approach to stakeholder engagement than is often experienced.

Design principles and supporting literature

The design principles for stakeholder engagement are organised into three groups, namely organisational, values and practices, albeit with some inevitable overlaps. We look at each category in turn, alongside a consideration of some of the relevant literature.

Organisational

  1. 1)

    Clarify the objectives of stakeholder engagement

  2. 2)

    Embed stakeholder engagement in a framework or model of research use

  3. 3)

    Identify the necessary resources for stakeholder engagement

  4. 4)

    Put in place plans for organisational learning and rewarding of effective stakeholder engagement

  5. 5)

    Recognise that some stakeholders have the potential to play a key role

Some examples from the literature

It is desirable to have a conceptual framework that situates stakeholder engagement as part of a plan for promoting research use in practice. Deverka et al. [6] proposed an ‘analytic-deliberative’ conceptual model for stakeholder engagement which “illustrates the inputs, methods and outputs relevant to CER [comparative effectiveness research]. The model differentiates methods at each stage of the project; depicts the relationship between components; and identifies outcome measures for evaluation of the process” ([6], p. 1). Furthermore, having a clear evaluation plan is considered critical. Concannon et al. recommended conducting “evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research” ([13], p. 1698). Esmail et al. argue that evaluations of stakeholder engagement should be “designed a priori as an embedded component of the research process” ([35], p. 142). They suggest that, where possible, evaluations should use predefined, validated tools. Jolibert and Wesselink [2] point out that linking stakeholders’ contributions with specific research objectives is important in order to establish when and how to engage and with whom. They argue that, at the recruitment stage, stakeholders should be made aware of, for example, their role/s, what they could contribute, costs in terms of time and effort, and benefits. Concannon et al. also conclude that funding is needed “to account for the costs of implementing meaningful engagement activities” ([7], p. 989).

In a Canadian study looking at stakeholder involvement in KT as a means of leading to more evidence-informed healthcare, Holmes et al. [36] identify a range of complexities which, they argue, need to be taken into account by funding schemes in order to meet funders’ and stakeholders’ expected ROI. Stakeholder involvement in research and implementing its findings is complex and time consuming, and the authors recommend an advocacy role where funders support a range of activities to address barriers to effective KT. These include carrying out an assessment of stakeholders’ KT needs “to identify gaps and opportunities and avoid duplication of efforts” ([36], p. 6). Kramer et al. [37] looked at the involvement of intermediary organisations as research partners on three interventions across four sectors, namely manufacturing, transportation, service and electrical utilities sectors. The authors describe the difficulties, benefits and challenges from the perspectives of both researchers and research partners and stress the importance of allowing the design of the protocol to be collaborative and flexible. Researchers need to honour, trust and respect their partners’ knowledge and expertise, and take into account their needs and priorities. Failure to meet these criteria will significantly dampen stakeholders’ enthusiasm. They also point out the importance of having a model of collaborative research with clear guidelines of how to conduct partnership research projects in order to further facilitate the use of research by practitioners. There would be an invested interest in “the research question, design and findings, and this would prove to be very valuable as a knowledge transfer strategy” ([37], p. 330).

The main literature on stakeholder analysis of policy-making is also useful for highlighting that some stakeholders have more potential to play a key role in the policy deliberations than others. For example, as part of their review of stakeholder analysis of health policy-making, Brugha and Varvasovszky [38] described an example in which the Hungarian Ministries of Finance and Industry were non-mobilised, high-influence, low-interest stakeholders in debates about public health interventions, but might, in some circumstances, become mobilised high-interest actors.

Values

  1. 6)

    Foster shared commitment to the values and objectives of stakeholder engagement in the project team

  2. 7)

    Share understanding that stakeholder engagement is often about more than individuals

  3. 8)

    Encourage individual stakeholders and their organisations to value engagement

  4. 9)

    Recognise potential tension between productivity and inclusion

  5. 10)

    Generate a shared commitment to sustained and continuous stakeholder engagement

Some examples from the literature

Concannon et al. [7] stress that researchers and stakeholders should be committed to the processes from the outset. Hinchcliff et al. [18] argue that it is important to define expectations and roles and provide time. Hering et al.’s [39] global study of water science and technology used stakeholder involvement in the objectives and approaches of the research for the co-production of knowledge as part of transdisciplinary research. Key aspects of particular value to the research included early identification and involvement of stakeholders, continuous engagement with stakeholders, and availability to stakeholders of supporting materials and in multiple languages. Mallery et al. recommend continuing to build trust with stakeholders “throughout the engagement process” ([5], p. 27).

Practices

  1. 11)

    Plan stakeholder engagement activity as part of the research programme of work

  2. 12)

    Build flexibility within the research process to accommodate engagement and the outcomes of engagement

  3. 13)

    Consider how input from stakeholders can be gathered systematically to meet objectives

  4. 14)

    Consider how input from stakeholders can be collated, analysed and used

  5. 15)

    Recognise identification and involvement of stakeholders is an iterative and ongoing process

Some examples from the literature

Forsythe et al. [30] highlight the importance of careful and strategic selection of stakeholders. As part of evidence and experience-based guidance to researchers and practice personnel about forming and carrying out effective research partnerships, Ovretveit et al. [40] developed a guide to categorise and describe types of partnerships or approaches to collaborative working. The guide sets out a framework for the roles and tasks, and the allocation of responsibilities for each partner involved. Roles and tasks are assigned to three main categories, namely questions, design and data, reporting and dissemination, and implementation and integration into organisation or policy. Concannon et al. [13] suggest the need to develop (and validate) stakeholder engagement tools to support engagement work. Forsythe et al. also stress the importance of “establishing ‘parameters and expectations for roles’, giving stakeholders guidance, and allowing time for stakeholders to ‘get comfortable with their roles’ as important tasks” ([30], p. 19).

The review of methods of stakeholder engagement conducted by Mallery et al. [5] identified a range of innovative methods and stressed the potential for engaging stakeholders at different points in the research process. The five methods highlighted for consideration were online collaborative forums, product development challenge contests, online communities, grassroots community organising and collaborative research. Jolibert and Wesselink [2] explored levels and types of stakeholder engagement in 38 EC-funded biodiversity research projects and the impacts of collaborative research on policy, society and science. They looked at how and when stakeholders were involved and the roles played, and argue that greater engagement throughout the whole of the research process, rather than, for example, at the dissemination stage, tends to lead to improved assessment of environmental change and effective policy proposals. Jolibert and Wesselink suggest, following Huberman’s [41] work in education, that it is desirable to have ‘sustained interactivity’ between researchers and users. Concannon et al. suggest that “General principles can be drawn from community-based participatory research, which underscores that engagement is a relationship-building process” ([7], p. 988). They found that, if bi-directional relationships are sustained over time, stakeholders can serve as ambassadors for high-integrity evidence even where the findings are contrary to generally accepted beliefs. Hinchcliff et al. point to the importance of “building respect and trust through ongoing interaction” ([18], p. 125). Forsythe et al. flag up the importance of continuous involvement and using in-person contact to build relationships [30]. They also stress the value in having a flexible approach that can adapt to the practical needs of stakeholders. A recent supplement of this journal edited by Paina et al. [4] also highlighted the importance of flexibility in making space for stakeholder engagement in research processes.

Based on the literature and the application of initial principles to our study, we have developed the elaborated design principles presented in Box 2.

Conclusions

There is a growing interest in stakeholder engagement as a potentially promising approach to promoting research impact. There is also a developing literature mapping out who potential stakeholders might be (the ‘who’), considering approaches to stakeholder engagement (the ‘how’) and identifying rationales for stakeholder engagement (the ‘why’). In this paper, evidence from the literature around these dimensions has been combined with the findings from our study of stakeholder engagement in an EC-funded project to develop a set of design principles to inform future stakeholder engagement in research. The design principles encompass organisational factors, values and practices. We hope that the principles will be useful in planning stakeholder engagement activity within research programmes and in monitoring and evaluating stakeholder engagement. Active engagement of stakeholders may well shift our understanding of what research use looks like [39]. A next step will be to address the remaining gap in the stakeholder engagement literature concerned with how we assess the impact of stakeholder engagement on research use.

Abbreviations

EC:

European Commission

EQUIPT:

European-study on Quantifying Utility of Investment in Protection from Tobacco

KT:

Knowledge translation

PPI:

Patient and public involvement

ROI:

Return on investment

SEE-Impact:

Stakeholder Engagement in EQUIPT for Impact

References

  1. 1.

    Kok M, Gyapong J, Wolffers I, Ofori-Adjei D, Ruitenberg J. Which health research gets used and why? An empirical analysis of 30 cases. Health Res Policy Syst. 2016;14:36. https://doi.org/10.1186/s12961-016-0107-2.

  2. 2.

    Jolibert C, Wesselink A. Research impacts and impact on research in biodiversity conservation: The influence of stakeholder engagement. Environ Sci Policy. 2012;20:100–11.

  3. 3.

    Phillipson J, Lowe P, Proctor A, Ruto E. Stakeholder engagement and knowledge exchange in environmental research. J Environ Manag. 2012;95:56–65. https://doi.org/10.1016/j.jenvman.2011.10.005.

  4. 4.

    Paina L, Ekirapa-Kiracho E, Ghaffar A, Bennett S. Engaging stakeholders in implementation research: tools, approaches, and lessons learned from application. Health Res Policy Syst. 2017;15(Suppl 2):104.

  5. 5.

    Mallery C, Ganachari D, Fernandez J, Smeeding L, Robinson S, Moon M, Lavallee D, Siegel J. Innovative Methods in Stakeholder Engagement: An Environmental Scan. Prepared by the American Institutes for Research under contract No. HHSA 290 2010 0005 C. AHRQ Publication NO. 12-EHC097-EF. Rockville: Agency for Healthcare Research and Quality; 2012.

  6. 6.

    Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1(2):181–94.

  7. 7.

    Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise JM, Santa J, Conway PH, Daudelin D, Morrato EH, Leslie LK. A new taxonomy for stakeholder engagement in patient centered outcomes research. J Gen Intern Med. 2012;27:985–91. https://doi.org/10.1007/s11606-012-2037-1.

  8. 8.

    Greenhalgh T, Jackson C, Shaw S, Janaiman T. Achieving research impact through cocreation in community-based health services: literature review and case study. Milbank Q. 2016;94(2):392–429.

  9. 9.

    Boaz A, McKevitt C, Biri D. Rethinking the relationship between science and society: has there been a shift in attitudes to patient and public involvement and public engagement in science in the United Kingdom? Health Expect. 2016;19(3):592–601. https://doi.org/10.1111/hex.12295.

  10. 10.

    Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453.

  11. 11.

    Lomas J. Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff. 2000;19(3):236–40. https://doi.org/10.1377/hlthaff.19.3.236.

  12. 12.

    Kogan M, Henkel M. Government and Research: The Rothschild Experiment in a Government Department. London: Heinemann Educational Books; 1983. (2nd edition. Dortrecht: Springer; 2006).

  13. 13.

    Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, Lau J. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med. 2014;29:1692–701. https://doi.org/10.1007/s11606-014-2878-x.

  14. 14.

    Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50. https://doi.org/10.1186/1748-5908-7-50.

  15. 15.

    Bowen S, Graham ID. Integrated knowledge translation. In: Straus SE, Tetroe J, Graham ID, editors. Knowledge Translation in Health Care: Moving from Evidence to Practice. Chichester: John Wiley & Sons Ltd; 2013. https://doi.org/10.1002/9781118413555.ch02.

  16. 16.

    McCutcheon C, Graham ID, Kothari A. Defining integrated knowledge translation and moving forward: a response to recent commentaries. Int J Health Policy Manage. 2017;6(5):299–300.

  17. 17.

    Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2016;11:38. https://doi.org/10.1186/s13012-016-0399-1.

  18. 18.

    Hinchcliff R, Greenfield D, Braithwaite J. Is it worth engaging in multi-stakeholder health services research collaborations? Reflections on key benefits, challenges and enabling mechanisms. Int J Qual Health Care. 2014;26:124–8. https://doi.org/10.1093/intqhc/mzu009.

  19. 19.

    Harvey G, Fitzgerald L, Fielden S, McBride A, Waterman H, Bamford D, Kislov R, Boaden R. The NIHR collaboration for leadership in applied health research and care (CLAHRC) for greater Manchester: combining empirical, theoretical and experiential evidence to design and evaluate a large-scale implementation strategy. Implement Sci. 2011;6(1):96. https://doi.org/10.1186/1748-5908-6-96.

  20. 20.

    Higher Education Funding Council for England (HEFCE). 2014 REF: Assessment Framework and Guidance on Submissions. Panel A Criteria. London: HEFCE; 2012.

  21. 21.

    Innvær S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44. https://doi.org/10.1258/135581902320432778.

  22. 22.

    Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2. https://doi.org/10.1186/1478-4505-1-2.

  23. 23.

    Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policymaking. J Health Serv Res Policy. 2005;10 Suppl 1:35–48. https://doi.org/10.1258/1355819054308549.

  24. 24.

    Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: a literature review. Sci Public Policy. 2009;36:255–70. https://doi.org/10.3152/030234209X436545.

  25. 25.

    Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26. https://doi.org/10.1186/1478-4505-9-26.

  26. 26.

    Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2. https://doi.org/10.1186/1472-6963-14-2.

  27. 27.

    Hanney S, Greenhalgh T, Blatch-Jones A, Glover M, Raftery J. The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews. Health Res Policy Syst. 2017;15:26. https://doi.org/10.1186/s12961-017-0191-y.

  28. 28.

    Nyström ME, Karltun J, Keller C, Andersson Gäre B. Collaborative and partnership research for improvement of health and social services: researcher’s experiences from 20 projects. Health Res Policy Syst. 2018;16:46. https://doi.org/10.1186/s12961-018-0322-0.

  29. 29.

    Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Watts C, Morley R. Stakeholder involvement in systematic reviews: a protocol for a systematic review of methods, outcomes and effects. Res Involv Engage. 2017;3:9. https://doi.org/10.1186/s40900-017-0060-4.

  30. 30.

    Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, Frank L. Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned. J Gen Intern Med. 2016;31:13–21. https://doi.org/10.1007/s11606-015-3450-z.

  31. 31.

    Center for Medical Technology Policy. Stakeholder Engagement. http://www.cmtpnet.org/our-work/stakeholder-engagement/. Accessed 31 Aug 2017.

  32. 32.

    Vokó Z, Cheung KL, Józwiak-Hagymásy J, Wolfenstetter S, Jones T, Muñoz C, Evers S, Hiligsmann M, de Vries H, Pokhrel S. On behalf of the EQUIPT study. Similarities and differences between stakeholders’ opinions on using health technology assessment (HTA) information across five European countries: results from the EQUIPT survey group. Health Res Policy Syst. 2016;14:38. https://doi.org/10.1186/s12961-016-0110-7.

  33. 33.

    Evers SMAA, Hiligsmann M, Vokó Z, Pokhrel S, Jones T, et al. Understanding the stakeholders’ intention to use economic decision-support tools: a cross-sectional study with the tobacco return on investment tool. Health Policy. 2016;120(1):46–54.

  34. 34.

    Pokhrel S, Evers S, Leidl R, Trapero-Bertran M, Kalo Z, De Vries H, et al. EQUIPT: protocol of a comparative effectiveness research study evaluating cross-context transferability of economic evidence on tobacco control. BMJ Open. 2014;4(11) https://doi.org/10.1136/bmjopen-2014-%20006945.

  35. 35.

    Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45. https://doi.org/10.2217/cer.14.79.

  36. 36.

    Holmes B, Scarrow G, Schellenberger M. Translating evidence into practice: the role of health researcher funders. Implement Sci. 2012;7:39.

  37. 37.

    Kramer D, Wells R, Bigelow P, Carlan N, Cole D, Hepburn CG. Dancing the two-step: collaborating with intermediary organizations as research partners to help implement workplace health and safety interventions. Work. 2010;36(3):321–32. https://doi.org/10.3233/WOR-2010-1033.

  38. 38.

    Brugha R, Varvasovszky Z. Stakeholder analysis: a review. Health Policy Plan. 2000;15:239–46.

  39. 39.

    Hering J, Hoffmann S, Meierhofer R, Schmid M, Peter A. Assessing the societal benefits of applied research and expert Consulting in Water Science and Technology. Gaia. 2012;21(2):95–101. https://doi.org/10.14512/gaia.21.2.6.

  40. 40.

    Ovretveit J, Hempel S, Magnabosco JL, Mittman BS, Rubenstein LV, Ganz DA. Guidance for research-practice partnerships (R-PPs) and collaborative research. J Health Organ Manage. 2014;28(1):115–26. https://doi.org/10.1108/JHOM-08-2013-0164.

  41. 41.

    Huberman M. Research utilization: the state of the art. Knowl Policy. 1994;7(4):13–33.

Download references

Acknowledgements

The authors would like to acknowledge the contribution of the EQUIPT team, in particular, the Principal Investigator Subhash Pokhrel.

Funding

The SEE-Impact study (Stakeholder Engagement in EQUIPT for Impact), received funding from the United Kingdom Medical Research Council to explore the engagement of stakeholders in the EQUIPT project.

The funding body had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Author information

Affiliations

  1. Faculty of Health, Social Care and Education, a partnership between Kingston University and St George’s, University of London, London, United Kingdom
    • Annette Boaz
    •  & Alison O’Shea
  2. Health Economics Research Group, Brunel University London, Uxbridge, United Kingdom
    • Stephen Hanney
  3. Erasmus School of Health Policy & Management, Rotterdam, The Netherlands
    • Robert Borst
  4. VU University Amsterdam, Amsterdam, The Netherlands
    • Maarten Kok
Authors
  1. Annette Boaz
    View author publications
    You can also search for this author in
  2. Stephen Hanney
    View author publications
    You can also search for this author in
  3. Robert Borst
    View author publications
    You can also search for this author in
  4. Alison O’Shea
    View author publications
    You can also search for this author in
  5. Maarten Kok
    View author publications
    You can also search for this author in

Contributions

AB and SH conceived of the study. All authors contributed to the design, data collection and analysis. An initial draft of the paper was produced by AB, with all authors contributing significantly to its development and revision. The final version has been approved by all authors.

Corresponding author

Correspondence to Annette Boaz.

Ethics declarations

Ethics approval and consent to participate

Ethical approval for the study entitled ‘Stakeholder Engagement in EQUIPT for Impact (SEE-IMPACT)’ was gained from the Research Ethics Committee, the Faculty of Health, Social Care and Education, St George’s University of London and Kingston University, on 18th March 2014.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

SEE-Impact study literature search. (DOCX 17 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Boaz, A., Hanney, S., Borst, R. et al. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Sys 16, 60 (2018). https://doi.org/10.1186/s12961-018-0337-6

Download citation

Health Research Policy and Systems

ISSN: 1478-4505

Contact us