Table 2 Summary of the strengths of the five priority-setting frameworks

From the literature

As identified by the workshop participants in addition to/support of those in the literature

ENHR

Step 1: How big is the health problem?

Step 2: Why does the disease burden persist?

Step 3: Is there sufficient knowledge about the problem to consider potential interventions?

Step 4: How cost-effective will these interventions be? How soon can they be developed at a reasonable cost?

Step 5: What are the current investments and available resources in this research area?

• Inclusiveness, involvement of a broad range of multidisciplinary, cross-sectoral stakeholders, e.g. experts, researchers, healthcare providers and representatives of the community

• Transparent and systematic, involves analyses of health needs, and societal and professional expectations

• Strong guiding principles, includes putting country priorities first, working for equity in development, and linking research to action for development

• Provides a detailed listing of priority options

• The situation analysis provides opportunities to identify benchmarks for evaluation, is well aligned with already existing systems within the country

• Provides chance to categorise the different service delivery opportunities

• Creation of a consultative group or TWG based on the situation analysis

CHNRI

Step 1: PS framework managers or initiators identify and convene a group of experts or TWG

Step 2: TWG members systematically create a long list of competing research options

Step 3: Members of the TWG independently score the list of research options using specific criteria

Step 4: PS framework managers and the TWG identify a Larger Reference Group to assess the importance of the criteria by attributing weights to the criteria

Step 5: After applying the weights, the average scores for each research option are often, though not always, calculated to obtain the RPS

Step 6: In some cases, the TWG then uses the derived RPS score to perform Program Budgeting Marginal Analysis with regards to research funding and the impact of the different options

• Reliability: the process is well documented and listed priorities are reproducible, challengeable, revisable

• The process is systematic and reduces the impact of self-interest when deriving the initial list

• Incorporates the consideration of values of a wider group of stakeholders and the public

• It is transparent

• Individual ranking in the technical working groups reduces any undue individual influence on the process and outcome

• Methods can simultaneously evaluate different kinds of research, e.g. health systems research, intervention research

Participatory

• TWGs bring experts together to discuss existing evidence, fosters a sense of ownership

• Method recognises the extreme importance of multi-stakeholder engagement, makes the final priorities credible and acceptable to stakeholders

• Face-to-face engagement of experts is very valuable, you would not get the same feeling virtually

The process

• Situation analysis ensures that current efforts/actors/evidence are part of the discussion

• Standardised process using explicit criteria

• Has the potential to offer a rapid assessment platform

• Contextual applicability Priorities can be focal, i.e. research questions, or broad, i.e. research issues

• Tried and tested in the local context and in other LICs

• Uses most commonly used PS criteria

• Aligned with the Ministry of Health policy-making process, and can be integrated in the system

JLA

Step 1 – Initiation: Establish a PSP of clinicians, patients and caregivers responsible for identifying, prioritising and publicising (the methods and results) identified priorities to the general public and research funders

Step 2 – Gather uncertainties: The PSP then identifies and gathers a long list of the treatment uncertainties as perceived by patients, carers and clinicians, using an open or broad prompt such as “What questions about treating X would you like to be answered by research?”

Prioritise the uncertainties: This involves two stages:

Step 3 – Interim PS: This interim measure provides a short list of the uncertainties, and can be done by the steering committee or the PSP itself

Step 4 – Final PS: This is often a face-to-face meeting or workshop, providing participants with an opportunity to express their views, hear those of others and broaden their thinking, often using the Nominal Group Technique in small groups or alternatively with larger group discussions, the aim is to develop an agreed-upon list of the top 10 priorities

Step 5: Disseminate the final priorities to funders

• Integrating quantitative and qualitative methods (where applied) enables researchers to gather many validated uncertainties and to understand the rationale behind them

• In 2009, the JLA underwent a formal review

• Employs innovative and participatory approaches to involve patients in decision-making

• Generates a list of priorities

• Detailed description of how to engage the various stakeholders

Participatory

• Engaging various stakeholders, e.g. the ‘public’ (specifically patients)

• Proposes specific strategies for gathering input from the major stakeholders (e.g. electronically, face-to-face)

The process

• Streamlined and straightforward process

• Clear idea of which stakeholders ought to participate

• Explicitly define the use of the Nominal group technique or Delphi method to determine between competing priorities

• Provides opportunities to revise and refine priorities

CAM

Step 1: Provide the best available information to participants who are populating the matrix; a comprehensive lack of information may indicate a research gap

Step 2: Start by completing the public health dimension, e.g. magnitude of the problem, and work with the best available data (which may be national or global) before proceeding to the determinants row (for the different levels)

Step 3: Discuss the entries; once all participants are in agreement, proceed to completing the third dimension

Step 4: Review the entries in the two dimensions using the lens of the equity stratifier; participants should look at disaggregated data to assess if the problem is experienced differently by the stratified groups, and if so, identify the determinants; complete these cells once in agreement

Step 5: Repeat this step for all the stratifiers

Step 6: Distribute the final report to all relevant stakeholders; the 3DCAM should be applied in a PS process based on three equally important pillars: the PS process, the tools and the context

• Flexibility: can be applied in diverse contexts, for diverse issues, and by people of differing expertise

• Practical and standardised way through which data can be presented and summarised, improving the transparency of the PS process

• Organises, summarises and presents the best available information on one disease, risk factor, group or condition, and facilitates comparisons between the likely benefits of different types of intervention at different levels; this ensures that decisions are based on the best available evidence rather than participants’ views and knowledge

• Draws attention to various domains where interventions are possible and desirable (from the household to global macroeconomic policies)

• Explicit consideration of equity as a major dimension

Participatory workshops

• This is already a strong feature of the health system in Zambia

• TWGs and inclusion in the process is a part of the Zambian health system

The process

• Selection of priorities is a continuous and cyclical activity, this may facilitate institutionalisation

• Ongoing structured tools for data collection to help inform the process of PS

• Sampling the representativeness in terms of performance of health indicators at all levels (community, district, provincial, national)

• Cost effectiveness and financial flows are considered

• Information gaps in the matrix are flagged as research priorities

Alignment

• Would fit in the existing tools for performing joint annual reviews within the Ministry of Health, e.g. the Social Economic Status, and cultural context already existing in the multi-sectoral initiatives, public health, institutional capacity, equity

• The emphasis on equity is key, since it is an issue in the health sector in Zambia

• Creates demand for better systems at different levels

L4D

Step 1: Stakeholder identification and assembling of background information needed for the consultation

Step 2: Consultation workshops: These identify priority research issues (as determined by decision-makers) in both the short and longer term; this could also involve discussing the availability or lack of studies on the issues identified

Step 3: Translation and sorting; identifying health research priorities and priorities that require synthesis of evidence

Step 4: Validate the identified priority issues against similar exercises

Step 5: Translate priority issues into research themes; research experts translate the identified research issues into research questions that should and can be answered in order to address the priority issue

Step 6: Validate priority research themes with stakeholders; this step ensures that the above translation into research areas correlates to the desires/opinions of stakeholders involved in the consultation of Step 3 – ensuring that the priorities “truly reflect their expressed views”

• A strong qualitative/interpretive framework designed to gather and listen above all, not slowing the process down with, for instance, criteria application details

• Participatory and transparent

• Allows for using combined qualitative and quantitative research techniques

• In the Canadian case, the priorities that emerged have received considerable acceptance and funding

Participatory

• Involvement of decision-makers is a key component

• Promotes clarity in decision-making for policy-makers

• Process involves people with knowledge (TWGs) and people with power (decision-makers)

• Validation at beginning and end of process

• Emphasis on KT – making priorities accessible to different groups within the population

The process

• The focus on research themes rather than narrower questions allows for flexibility in PS

• Situation analysis is important

• Two-step validation process, – at the beginning of the process and at conclusion – streamlines M&E processes

• KT is important, it makes the process accessible for different populations

• Produces both short- and long-term plans

• Focus on the future

• Can be integrated into existing infrastructure