The project team (comprising members of the SEE-Impact research team and collaborators from EQUIPT) met for a 2 day analysis workshop. One aim of the workshop was to begin to build a consensus among the team on what seemed to be the key design principles emerging from the SEE-Impact data and the on-going literature review. SEE-Impact data included observational data, interviews and document analysis. The research team continued to develop the principles through an ongoing period of deliberation, informed by the impact study and the literature. As part of this process, the principles were categorised into three groups, namely organisational, values and practices.
In this section, we first present empirical evidence from the SEE-Impact study that informed our development of the design principles. We then briefly summarise published evidence for each group of design principles in order to situate them in the wider literature.
Design principles and empirical evidence from the stakeholder engagement in EQUIPT for impact (SEE-impact) study
The stakeholder engagement study (SEE-Impact) and the project being studied (EQUIPT) are described in Box 1. In terms of the organisational level principles, the EQUIPT project objectives for stakeholder engagement were clear, as set out in the proposal, protocol and project documents . The key aims of stakeholder engagement activity were to access the knowledge and skills (described in the protocol as co-creation innovation in the working space) and to increase influence and impact (described in the protocol as dissemination innovation in the transfer space through stakeholder engagement).
In terms of values, the commitment to stakeholder engagement was more clearly demonstrated by some of the EQUIPT project team members than others. For some team members, previous successful experience of an interactive form of working with stakeholders had built a commitment to this particular way of working. It also provided experience of practical elements of working with stakeholders, but perhaps most importantly lived experience of the practical benefits of engagement. For other members of the team, too, working with stakeholders fitted closely with their ethos and values. For example, the Hungarian team talked about their pragmatic approach to research and the need to conduct useful and usable research, with stakeholder engagement being a key component. However, a small group within the wider project team did not seem committed to ensuring stakeholder engagement remained a core element of the project. They favoured a particular, individualised approach to stakeholders and, over time, partially reshaped the stakeholder engagement activities to something more akin to research participation (that is, taking part in a research study as a means of generating specific data as determined by researchers, rather than as co-producers of research). Finally, not all stakeholders identified by the project team were interested in engaging with the project. In particular, the lack of policy priority given to smoking cessation (the focus of the return on investment (ROI) tool) made engagement of policy stakeholders in the Netherlands very difficult to achieve.
In terms of practices, while the EQUIPT project protocol did set out how the stakeholder engagement would operate , there was not as much flexibility as the investigators would have liked in terms of the project plan and this had an impact on the nature of the stakeholder engagement activities. In particular, time intensive methods of engagement originally proposed in the protocol (particularly the large number of face-to-face meetings) began to look unrealistic to members of the team. The lack of flexibility came in part from the funder. The EC told the project team at an early point that there was no scope for negotiation around the project end date. Thus, initial delays in the project put a strain on the project timetable and deliverables. Members of the team proposed a shift from face-to-face meetings with stakeholders to Skype meetings in an effort to ‘catch up’. The technical team producing the new version of the ROI tool for roll out in Europe added to a sense of urgency in ‘speeding up’ the stakeholder engagement work with their need for data to feed into their work. Nevertheless, despite the practical difficulties, in EQUIPT, a significant amount of consideration had been given to stakeholder engagement, including planning how the input provided by stakeholders might be gathered, collated, analysed and used. Vokó et al. highlight that it is important to “fully analyse several aspects of stakeholder engagement in research” (, p. 15) and note that there is a tendency to ignore the value of early stakeholder engagement when it comes to development and transferability in the work of economic evaluation. EQUIPT’s careful consideration and the methods adopted facilitated a much more rigorous approach to stakeholder engagement than is often experienced.
Design principles and supporting literature
The design principles for stakeholder engagement are organised into three groups, namely organisational, values and practices, albeit with some inevitable overlaps. We look at each category in turn, alongside a consideration of some of the relevant literature.
Clarify the objectives of stakeholder engagement
Embed stakeholder engagement in a framework or model of research use
Identify the necessary resources for stakeholder engagement
Put in place plans for organisational learning and rewarding of effective stakeholder engagement
Recognise that some stakeholders have the potential to play a key role
Some examples from the literature
It is desirable to have a conceptual framework that situates stakeholder engagement as part of a plan for promoting research use in practice. Deverka et al.  proposed an ‘analytic-deliberative’ conceptual model for stakeholder engagement which “illustrates the inputs, methods and outputs relevant to CER [comparative effectiveness research]. The model differentiates methods at each stage of the project; depicts the relationship between components; and identifies outcome measures for evaluation of the process” (, p. 1). Furthermore, having a clear evaluation plan is considered critical. Concannon et al. recommended conducting “evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research” (, p. 1698). Esmail et al. argue that evaluations of stakeholder engagement should be “designed a priori as an embedded component of the research process” (, p. 142). They suggest that, where possible, evaluations should use predefined, validated tools. Jolibert and Wesselink  point out that linking stakeholders’ contributions with specific research objectives is important in order to establish when and how to engage and with whom. They argue that, at the recruitment stage, stakeholders should be made aware of, for example, their role/s, what they could contribute, costs in terms of time and effort, and benefits. Concannon et al. also conclude that funding is needed “to account for the costs of implementing meaningful engagement activities” (, p. 989).
In a Canadian study looking at stakeholder involvement in KT as a means of leading to more evidence-informed healthcare, Holmes et al.  identify a range of complexities which, they argue, need to be taken into account by funding schemes in order to meet funders’ and stakeholders’ expected ROI. Stakeholder involvement in research and implementing its findings is complex and time consuming, and the authors recommend an advocacy role where funders support a range of activities to address barriers to effective KT. These include carrying out an assessment of stakeholders’ KT needs “to identify gaps and opportunities and avoid duplication of efforts” (, p. 6). Kramer et al.  looked at the involvement of intermediary organisations as research partners on three interventions across four sectors, namely manufacturing, transportation, service and electrical utilities sectors. The authors describe the difficulties, benefits and challenges from the perspectives of both researchers and research partners and stress the importance of allowing the design of the protocol to be collaborative and flexible. Researchers need to honour, trust and respect their partners’ knowledge and expertise, and take into account their needs and priorities. Failure to meet these criteria will significantly dampen stakeholders’ enthusiasm. They also point out the importance of having a model of collaborative research with clear guidelines of how to conduct partnership research projects in order to further facilitate the use of research by practitioners. There would be an invested interest in “the research question, design and findings, and this would prove to be very valuable as a knowledge transfer strategy” (, p. 330).
The main literature on stakeholder analysis of policy-making is also useful for highlighting that some stakeholders have more potential to play a key role in the policy deliberations than others. For example, as part of their review of stakeholder analysis of health policy-making, Brugha and Varvasovszky  described an example in which the Hungarian Ministries of Finance and Industry were non-mobilised, high-influence, low-interest stakeholders in debates about public health interventions, but might, in some circumstances, become mobilised high-interest actors.
Foster shared commitment to the values and objectives of stakeholder engagement in the project team
Share understanding that stakeholder engagement is often about more than individuals
Encourage individual stakeholders and their organisations to value engagement
Recognise potential tension between productivity and inclusion
Generate a shared commitment to sustained and continuous stakeholder engagement
Some examples from the literature
Concannon et al.  stress that researchers and stakeholders should be committed to the processes from the outset. Hinchcliff et al.  argue that it is important to define expectations and roles and provide time. Hering et al.’s  global study of water science and technology used stakeholder involvement in the objectives and approaches of the research for the co-production of knowledge as part of transdisciplinary research. Key aspects of particular value to the research included early identification and involvement of stakeholders, continuous engagement with stakeholders, and availability to stakeholders of supporting materials and in multiple languages. Mallery et al. recommend continuing to build trust with stakeholders “throughout the engagement process” (, p. 27).
Plan stakeholder engagement activity as part of the research programme of work
Build flexibility within the research process to accommodate engagement and the outcomes of engagement
Consider how input from stakeholders can be gathered systematically to meet objectives
Consider how input from stakeholders can be collated, analysed and used
Recognise identification and involvement of stakeholders is an iterative and ongoing process
Some examples from the literature
Forsythe et al.  highlight the importance of careful and strategic selection of stakeholders. As part of evidence and experience-based guidance to researchers and practice personnel about forming and carrying out effective research partnerships, Ovretveit et al.  developed a guide to categorise and describe types of partnerships or approaches to collaborative working. The guide sets out a framework for the roles and tasks, and the allocation of responsibilities for each partner involved. Roles and tasks are assigned to three main categories, namely questions, design and data, reporting and dissemination, and implementation and integration into organisation or policy. Concannon et al.  suggest the need to develop (and validate) stakeholder engagement tools to support engagement work. Forsythe et al. also stress the importance of “establishing ‘parameters and expectations for roles’, giving stakeholders guidance, and allowing time for stakeholders to ‘get comfortable with their roles’ as important tasks” (, p. 19).
The review of methods of stakeholder engagement conducted by Mallery et al.  identified a range of innovative methods and stressed the potential for engaging stakeholders at different points in the research process. The five methods highlighted for consideration were online collaborative forums, product development challenge contests, online communities, grassroots community organising and collaborative research. Jolibert and Wesselink  explored levels and types of stakeholder engagement in 38 EC-funded biodiversity research projects and the impacts of collaborative research on policy, society and science. They looked at how and when stakeholders were involved and the roles played, and argue that greater engagement throughout the whole of the research process, rather than, for example, at the dissemination stage, tends to lead to improved assessment of environmental change and effective policy proposals. Jolibert and Wesselink suggest, following Huberman’s  work in education, that it is desirable to have ‘sustained interactivity’ between researchers and users. Concannon et al. suggest that “General principles can be drawn from community-based participatory research, which underscores that engagement is a relationship-building process” (, p. 988). They found that, if bi-directional relationships are sustained over time, stakeholders can serve as ambassadors for high-integrity evidence even where the findings are contrary to generally accepted beliefs. Hinchcliff et al. point to the importance of “building respect and trust through ongoing interaction” (, p. 125). Forsythe et al. flag up the importance of continuous involvement and using in-person contact to build relationships . They also stress the value in having a flexible approach that can adapt to the practical needs of stakeholders. A recent supplement of this journal edited by Paina et al.  also highlighted the importance of flexibility in making space for stakeholder engagement in research processes.
Based on the literature and the application of initial principles to our study, we have developed the elaborated design principles presented in Box 2.