Table 3 ‘Nine Common Themes of Good Practice’ conceptual framework and its adaption for the evaluation of a research priority setting process

Context

Articulating the contextual factors that underpin the process

1. Will the established goals, underlying values and principles continue to be relevant the next time the program facilitates priority setting? 2. Are there changes to the number of resources available for the next priority setting cycle?

i) Extent to which findings from original stakeholder engagement processes remain relevant for next research priority setting process ii) Availability of ongoing financial and human resources

i) Consultation with senior leadership within PHO ii) Program documents

Inclusiveness

Deciding who should be involved in setting research priorities

3. Did appropriate stakeholders participate in the most recent priority setting cycle, and was there balanced representation?

i) Number and representativeness of health units who submit phase 1 survey ii) Percentage of workshop participants who are front-line staff, managers, and senior decision-makers iii) Percentage of workshop participants who agree or strongly agree that they had opportunity to express opinions and ideas

i) Health unit demographic characteristics obtained from Phase 1 survey ii) Workshop registration list iii) Workshop evaluation iv) Informal feedback from program participants

Information gathering

Choosing what information should be gathered to inform the process

4. Was the most recent priority setting exercise appropriately informed? 5. Did the provided information sources support decision making?

i) Types of technical information provided to workshop participants ii) Perceived usefulness of technical material provided to workshop participants to aid decision-making

i) Workshop facilitation materials ii) Informal feedback

Planning for implementation

Establishing plans for translating research priorities into projects

6. In previous cycles, were there challenges to translating the research priorities into research?

i) Challenges reported relative to implementing LDCP project proposals ii) Quality and amount of support available from LDCP program staff

i) Interim and final progress reports ii) Consultation with program staff

Criteria

Selecting relevant criteria to focus discussion

7. In the most recent priority setting cycle, were the criteria effective for decision making, and will the criteria continue to be relevant for the next cycle?

i) Percentage of workshop participants who agree or strongly agree that the process helped them successfully apply the criteria ii) Alignment of criteria with overarching goals of research priority setting process

i) Workshop evaluation ii) Consultation with program advisory committee

Methods for deciding on priorities

Choosing a method for deciding on priorities

8. In the most recent priority setting cycle, were the methods for deciding on priorities appropriate and effective for decision making?

i) Percentage of workshop participants who agree or strongly agree that the process used to select priorities helped to build consensus ii) Percentage of workshop participants who agree or strongly agree that the workshop was an effective way to help health units discuss and prioritize topics for research and evaluation projects

i) Workshop evaluation

Use of a comprehensive approach

Assessing whether a comprehensive approach is necessary or if a tailored process and methods are required

9. Are there elements in comprehensive approaches and priority setting methods? Specifically, the Listening Model, COHRED, Child Health and Nutrition Research Initiative, Essential National Health Research, Combined approach matrix and Delphi technique, which are transferable to the LDCP priority setting process

i) Extent to which elements of established comprehensive approaches and methods can be transferred to or would strengthen LDCP priority setting processes

i) Review of comprehensive approaches and methods described in the Checklist

Transparency

Communicating the approach that was used to set priorities

10. Did all stakeholders receive information about the process and outcomes of the most recent priority setting process?

i) Types of communication strategies used to share information about the LDCP priority setting process with stakeholders ii) Timeliness of communications about the priority setting process

i) Program document review

Evaluation

Defining when and how evaluation of process and outcome will occur

11. Are further evaluation activities required to assess the delivery and outcomes of the priority setting process?

i) Perceived usefulness of current evaluation activities for informing quality improvements to the LDCP priority setting process

i) Consultation with program advisory committee