Evaluation of priority-setting processes
Stakeholder engagement
All key stakeholders were involved in the decision-making process. Consumers and consumer group representatives, health policy-makers, health professionals, health service staff and research funders were all included in the steering group (n = 11), the online survey (n = 151) and the workshop (n = 28). The steering group was involved in the decision-making process by influencing key directions in the methods used, whereas workshop participants were involved in decisions by voting on the top research priorities to go forward as Cochrane Reviews. However, as steering group members and workshop participants pointed out, participation in the online survey by Indigenous people and people from culturally and linguistically diverse backgrounds was low. We subsequently used a sampling frame to guide workshop recruitment and approached specific organisations representing these groups in the recruitment process, but participation rates in the workshop remained low (3/28 people from culturally and linguistically diverse backgrounds and 1/28 Indigenous people).
Across the project, we used multiple techniques to identify stakeholders and offered stakeholders multiple ways to contribute. We recruited stakeholders for the survey and workshop using networks of the project team and steering group. For the online survey and workshop, we also contacted a range of health organisations, inviting their members to forward an email invitation to participate. One steering group member encouraged face-to-face methods for the online survey to facilitate participation for particular groups, such as older people, but this was discounted on account of limited resources. To make participation more accessible, online survey participants could, on request, take part by phone or post (2/151 people accepted this offer).
We were committed to genuine engagement at the outset, following the principles of coproduction [16]. Examples of how this was operationalised included partnering with stakeholders in all project stages, giving the steering group decision-making powers and using multiple strategies in the workshop to facilitate true participation. These workshop strategies included having 50% consumer participation (mitigating power imbalances) and reimbursing time and transport costs. One steering group member said, “I appreciated the respectful and rigorous way in which consumer perspectives were sought and incorporated in this project”.
All workshop participants agreed with the statements “I feel that my contribution was valued and heard” and “the materials and resources used during the day helped me understand my role and make a contribution”, which suggests that they were satisfied with their participation. In addition, of the 64 free-text comments received in the workshop feedback survey, 43 included positive feedback and 21 offered suggestions for improvement. Of these, 5/25 workshop participants suggested that the workshop structure may not have allowed quieter participants to contribute and 3/25 suggested the time constraints limited discussion.
Use of explicit process
The priority-setting methods were pre-determined and made transparent to stakeholders as they were planned prior to the first steering group meeting and made available on the project webpage. In addition, participants in the online survey and workshop were provided with information about the CCCG along with a summary of the methods and how their contribution would be included in project. With respect to including internal and external stakeholders, we did not specifically seek to include CCCG staff and editors but instead focussed on probing external stakeholders for their research priorities.
The methods used to set priorities in this project were understandable, transparent and relevant for different stakeholders. The steering group reviewed and approved our methods and all workshop participants surveyed agreed with the statements that the information they received helped them understand what was expected of them before they came, and that the materials and resources used on the day helped them understand their role and make a contribution. A workshop participant stated in the final report, “thank you so much for all your help to allow me to participate in the workshop. It was one of the best experiences of my life, I felt ‘heard’ and I hope I was able to help in some way”.
Communication with stakeholders was well coordinated, systematic and well planned. The steering group were given agendas and meeting documents approximately a week prior to meetings and the minutes were circulated within a week. Similarly, workshop participants were provided with a pre-reading pack several days prior to the workshop (see online appendices in Synnot et al. [32]) while online survey reminders were sent weekly to all email addresses on the recruitment list. During the project, information was communicated using the project webpage [4] and at project completion with a professionally formatted final report [30] and in academic publications [31, 32].
Information management
The primary information used to determine the five priority Cochrane Reviews (namely the online survey and the workshop) was pre-determined and made explicit to all stakeholders throughout the project. However, the intended final step, generating evidence maps of the priority topics for final selection by the steering group, was not undertaken because the complexity of the CCCG scope and the broad categories of interventions within many areas makes this task too abstract and not helpful for editorial decision-making in terms of specific topics for reviews. In its place, the project team undertook a more pragmatic step of mapping the priority topics selected at the workshop against the CCCG review portfolio and then sought steering group and editorial approval. While the steering group endorsed the new approach, the final step in the priority-setting process involved no stakeholders in the decision-making.
Consideration of values and context
The Centre’s mission (where CCCG is located), is to conduct research to “to improve people’s health and wellbeing through the generation and promotion of evidence-informed strategies for consumer communication and participation in health [and] to strengthen the active involvement of consumers in health care, policy and research” [5]. This mission was included in the project plan which was shared with the steering group and made available on the project webpage. It also informed the methods used, specifically giving stakeholders the opportunity to shape, identify and select the priority topics. The final selection of five priority Cochrane Reviews was informed by a set of editorial criteria reflecting CCCG’s more review-specific priorities (for example, that the review could be completed in a timely manner). One of these review-specific criteria was that the author team was agreeable to consumer/stakeholder engagement in their review. It was included because it was a priority research topic identified by stakeholders in the survey and workshop, reflecting the way in which stakeholders’ values were included in the decision-making process.
Revision or appeals mechanism
There were two ways in which decisions were formally reviewed by stakeholders. Steering group meetings provided three opportunities for members to review and contribute to project decisions. For example, at one meeting, members urged further recruitment efforts to increase the diversity of participants in the online survey. We also gave workshop participants an opportunity to refine and contribute new ideas to the priorities generated in the online survey. As a result, one new priority topic was added to the list of priorities, which were voted on at a later stage in the workshop. However, once the five priority topics were announced, we had no formal appeals mechanism for stakeholders.
Evaluation of priority-setting outcomes
Improved stakeholder understanding
Based on the workshop feedback survey, stakeholders understood the nature of the priority-setting process since they all agreed with the statements that they understood what was expected of them and felt able to make a contribution. In addition, at least six workshop participants left feedback or contacted the team after the workshop to express their interest in being involved in the priority Cochrane Reviews, which suggests they had gained insight into ways of being involved in CCCG/Cochrane and understood what they were contributing to. Stakeholders in the steering group also demonstrated improved understanding given one steering group member subsequently initiated a priority-setting exercise in her workplace and another became a co-author on one of the priority Cochrane Reviews.
Shifted priorities and/or reallocation of resources
At project completion, the five Cochrane Review titles that were selected as priority topics directly reflected the top priority areas as identified and voted on by stakeholders. One of the priority titles [13], an existing Cochrane Review needing updating, later became two protocols for new reviews as a result of a new coproduction process with stakeholders [18, 21]. In response to stakeholders’ desire to be involved in the priority Cochrane Reviews and to reflect the degree to which consumer engagement in health was a priority topic, CCCG required the priority review author teams to involve stakeholders. To support the timely publication of the priority reviews, the CCCG reallocated editorial resources to provide additional guidance and support to the priority review teams, for example, providing rapid assistance with methodological queries and prioritising search development. In addition, the CCCG sought and received funding to support the production of one review conducted by the CCCG editorial team and one review conducted by an external author team. Despite these achievements, more than 3 years after the prioritisation process (at the time of writing) no reviews have yet been published (all reviews have published protocols, two reviews have been submitted for editorial assessment, one rejected (for reasons unrelated to the engagement process), and the remainder should be completed within 2020).
Improved decision-making quality
The priority-setting project resulted in a shift in CCCGs editorial policies and its organisational values. For example, the CCCG changed its editorial policy on title proposals and updates, requesting they align with stakeholder priorities, with more invited or commissioned title proposals. With regards to its organisational values, since the priority-setting project, CCCG now conducts its in-house reviews with stakeholder input, with one co-produced review and one with stakeholder input now underway. Subsequently, the involvement of three editorial staff in prioritisation and coproduction activities led to the initiation of a coproduction network for Australian researchers and consumers, and two new collaborations with international researchers involved in stakeholder engagement activities.
Stakeholder acceptance and satisfaction
Stakeholders across the project stages expressed satisfaction with the process by showing a willingness to contribute to subsequent project stages. For example, at least one participant took part in the online survey and then accepted an invitation to participate in the workshop. Five workshop participants and steering group members accepted an invitation to contribute to the final report and five joined author teams or stakeholder panels for the priority reviews. In addition, workshop participants gave overwhelmingly positive comments in the feedback survey, with 14/25 participants praising the structure and/or facilitation of the day, with typical comments including a “great day” that was well planned/communicated/organised.
All the priority reviews involve stakeholders in sometimes novel ways. In some reviews, there are consumer co-authors and others have a stakeholder advisory group. One of the priority reviews involved stakeholders in the peer review process, with a group of health policy-makers being briefed on the review in its early stages and subsequently providing peer review of the protocol.
Positive externalities
The results of the priority-setting project were shared with stakeholders via a professionally designed (and coproduced) reports and promoted via health-sector aimed newsletters, blogs, policy-maker seminars and meetings, and academic presentations and publications.
While we are not aware of research funders or research institutes including the priorities as part of their research agenda or strategic planning, it has influenced the work of others. For example, an Australian state health department replicated our methods to inform the development of their state-wide policy on consumer engagement [17]. In addition, the project’s findings have influenced other systematic reviewers, for example, Selman [27].
The priority-setting project has also indirectly influenced the work of Cochrane. As a result of this project, CCCG staff contributed to Cochrane’s guidance on priority-setting [10], with the CCCG priority-setting project described in considerable detail as an example of good practice. Finally, CCCG successfully advocated for a change to Cochrane’s now discontinued priority review funding programme, whereby it allowed a longer funding period for complex reviews such as those being produced with stakeholders.