Improving the use of research evidence in guideline development: 2. Priority setting

Background The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the second of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. Objectives We reviewed the literature on priority setting for health care guidelines, recommendations and technology assessments. Methods We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. Key questions and answers There is little empirical evidence to guide the choice of criteria and processes for establishing priorities, but there are broad similarities in the criteria that are used by various organisations and practical arguments for setting priorities explicitly rather than implicitly, What criteria should be used to establish priorities? • WHO has limited resources and capacity to develop recommendations. It should use these resources where it has the greatest chance of improving health, equity, and efficient use of healthcare resources. • We suggest the following criteria for establishing priorities for developing recommendations based on WHO's aims and strategic advantages: • Problems associated with a high burden of illness in low and middle-income countries, or new and emerging diseases. • No existing recommendations of good quality. • The feasibility of developing recommendations that will improve health outcomes, reduce inequities or reduce unnecessary costs if they are implemented. • Implementation is feasible, will not exhaustively use available resources, and barriers to change are not likely to be so high that they cannot be overcome. • Additional priorities for WHO include interventions that will likely require system changes and interventions where there might be a conflict in choices between individual and societal perspectives. What processes should be used to agree on priorities? • The allocation of resources to the development of recommendations should be part of the routine budgeting process rather than a separate exercise. • Criteria for establishing priorities should be applied using a systematic and transparent process. • Because data to inform judgements are often lacking, unmeasured factors should also be considered – explicitly and transparently. • The process should include consultation with potential end users and other stakeholders, including the public, using well-constructed questions, and possibly using Delphi-like procedures. • Groups that include stakeholders and people with relevant types of expertise should make decisions. Group processes should ensure full participation by all members of the group. • The process used to select topics should be documented and open to inspection. Should WHO have a centralised or decentralised process? • Both centralised and decentralised processes should be used. Decentralised processes can be considered as separate "tracks". • Separate tracks should be used for considering issues for specific areas, populations, conditions or concerns. The rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration. • Updating of guidelines could also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.

What processes should be used to agree on priorities?: • The allocation of resources to the development of recommendations should be part of the routine budgeting process rather than a separate exercise.
• Criteria for establishing priorities should be applied using a systematic and transparent process.
• Because data to inform judgements are often lacking, unmeasured factors should also be considered -explicitly and transparently.
• The process should include consultation with potential end users and other stakeholders, including the public, using well-constructed questions, and possibly using Delphi-like procedures.
• Groups that include stakeholders and people with relevant types of expertise should make decisions. Group processes should ensure full participation by all members of the group.
• The process used to select topics should be documented and open to inspection.
Should WHO have a centralised or decentralised process?: • Both centralised and decentralised processes should be used. Decentralised processes can be considered as separate "tracks".
• Separate tracks should be used for considering issues for specific areas, populations, conditions or concerns. The rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration.
• Updating of guidelines could also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.

Background
The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the second of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. In this paper we address the following questions: • What criteria should be used to establish priorities?
• What processes should be used to agree on priorities?
• Should WHO have a centralised or decentralised process?
Questions related to group processes for committees developing guidelines and recommendations and priority setting for systematic reviews are addressed in other papers in this series [1,2].

What WHO is doing now
WHO does not have a centralised process specifically for setting priorities for the development of recommendations. A report of the Director-General to the Executive Board on WHO's strategic budgeting and planning process had this to say about priority setting in general (without specific reference to priorities for recommendations): Specific global priorities were included in the procedural guidance for 2002-2003, and measures were provided to ensure a shift of resources to those areas. With regard to regional and country health issues, the team received diverging views. Some staff expressed concern about the little room for specific regional or country priorities, which would not relate directly to the global priorities. Others expressed the need for flexibility during the operational planning phase, which should contribute to the achievement of global priorities in terms of reducing a health problem or improving the health status of the population. It is important to create a monitoring and evaluation system, in which such flexibility can be taken into account.
The report offered these two recommendations regarding priority setting in general: Criteria and parameters for rationalizing the setting of programme priorities should be re-examined with the view to achieving more objectivity.
Exercises to shift resources to priority areas should be an integral part of the programme budgeting process, and not taken up as a separate exercise.
The Guidelines for WHO Guidelines recommends the following [3]: Guideline development is a process which consumes resources (see Sec VII). They could be developed on almost every health topic or intervention so it is necessary for WHO to decide which topics should be given priority. It is suggested that the following areas be given priority: interventions that will require system changes (feasibility concerns) as opposed to those dealing solely with provider/ patient interactions. WHO has greater comparative advantage in dealing with governments, for interventions which require inputs and coordination at different levels of the system. It has less comparative advantage on purely provider/patient interactions.
cost-effective interventions that address a disease burden which is still causing major health losses, implying under-utilization of the technology (population perspective).
interventions that are of limited or questionable effectiveness but are being used widely (opportunity costs). The Health Evidence Network (HEN), based at the European Regional Office of WHO [4], collects policy concerns and questions from several sources and through both a proactive and a reactive approach:

What other organisations are doing
About 30% of respondents to an international survey of producers of clinical practice guidelines or health technology assessments reported using explicit methods of setting priorities, including the use of explicit criteria, formal consensus processes, and burden of disease [4]. The US Community Preventive Services Task Force, for example, chooses broad topics (e.g., tobacco use, cancer, diabetes, the social environment) for review on the basis of the public health burden of the problem; how preventable it is; how it relates to other public health initiatives; and the current level of research and practice activity in public health, clinical, and other settings. The agenda-setting process incorporates input from interested others.
The process of selecting specific interventions for review within those topics involves developing a candidate list of interventions, and setting priorities using a voting procedure among the team and the consultants. The Task Force approves or modifies the resulting priorities. Priority-setting criteria that are adapted for the reviews include perceived potential to reduce the burden of disease and injury; potential to increase healthy behaviors and reduce unhealthy behaviors; potential to increase the implementation of effective but not widely used interventions; potential to phase out widely used, less effective interventions in favor of more effective or more cost-effective options; and current level of interest among providers and decision makers. Other priority-setting criteria may be added as relevant and appropriate. Occasionally, review teams have engaged in formal scoring and weighting of the criteria. One or more rounds of this process results in a prioritized list of interventions.
Other respondents to the survey reported selecting topics based on consultations with their constituencies, requests from end-users, or decisions made by expert panels or a steering group without explicit methods. Seventeen of 67 respondents (25%) reported involving target users in the groups that set priorities.

Methods
The methods used to prepare this review are described in the introduction to this series [5]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct a full systematic review. We searched PubMed and three databases of methodological studies (the Cochrane Methodology Register [6], the US National Guideline Clearinghouse [7], and the Guidelines International Network [8]) for existing systematic reviews and relevant methodological research that address these questions. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.
For this review we searched PubMed using (clinical practice guidelines or public health guidelines) and (priority setting or setting priorities) and related articles for selected references [9,10]. We searched the Cochrane Methodology Register using priority or priorities. We reviewed the website of the 5 th International Conference on Priorities in Health Care [11] and references that we had in our files [12][13][14][15].

What criteria should be used to establish priorities?
The US Institute of Medicine's (IOM) Committee on Methods for Setting Priorities for Guidelines Development in its study of setting priorities for clinical practice guidelines published in 1995 argued that the priority setting process should be open and defensible [14] They recommended six general criteria: prevalence, burden of illness, cost of managing the problem, variability in practice, potential of a guideline to improve health outcomes, and potential of a guideline to reduce costs. Because data used to make these judgements is often lacking, they suggested explicit opportunities for important unmeasured factors to be considered. They further suggested separate "tracks" for considering issues for specific populations, conditions or concerns. They argued that the rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration. They suggested that updating of guidelines should also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.
Oortwijn identified 25 criteria used to prioritise health technology assessments and categorised these into four broad categories: burden of disease, potential effects, potential costs, and uncertainty regarding application of the technology [15].
In a more recent selective review for the New Zealand Guidelines Group, the following criteria were identified as indicating that a topic is suitable for guideline development [16]: 1. The topic is clinically important affecting large numbers of people with substantial morbidity or mortality (the burden of illness).

The topic is complex enough to initiate debate about the recommendations.
3. There is evidence of variation between actual and appropriate care. 4. There are no existing valid guidelines available to use. 5. There is an adequate amount of existing evidence available. 6. The recommendations will be acceptable to the potential users.
7. Implementation of the guideline is feasible, will not exhaustively use the communities' resources, and barriers to clinical change are not so high that they cannot be overcome.
While burden of disease is commonly used as a criterion for priority setting, it should be noted that the use of summary burden of disease measures, such as disability adjusted life years (DALYs) has been criticised for focusing on disease rather than resource use and interventions, because of the assumptions about values inherent in such measures, and because of the technical limitations of such measures (see for example references [17] and [18]).

What processes should be used to agree on priorities?
Batista and Hodge in a review conducted 10 years ago found only three articles pertinent to priority setting for clinical practice guidelines [10]. They suggested the following framework for priority setting: 1. Consult with end users and other stakeholders before selecting topics.
2. Consider feasibility during the consultation.
3. Document the process used to select guideline topics.
The IOM suggested the following procedures [14]: • the use of Delphi-like procedures for obtaining expert judgments or topic rankings through correspondence • the use of questions that are specific, explicit and consistent with standard methods for questionnaire construction • experimentation with more formal procedures to arrive at group judgments They also suggested there is a need to define more narrowly and precisely topics for guideline development. They argued that this would result in more efficient organization of panels and their work, resolution of some apparent controversies, more responsive guidelines, and easier implementation.
Oortwijn identified six steps in the development of practical procedures for setting priorities [15]:

Clarifying goals and responsibilities:
2. Choosing a general approach, method, and criteria for prioritisation; 3. Establishing advisory mechanisms and relations with external bodies; 4. Establishing arrangements to support and manage the procedure;

Defining a time table and cycle of activity; and
6. Evaluating and developing the procedure.
She further identified the following ways in which approaches to priority setting can vary: • the extent to which the procedure is explicit and systematic • the extent to which external input and advice is accepted or actively sought

• the relative weight given to the views of decision-makers, researchers, and others
• the extent to which the procedure is transparent • the effort and resources devoted to the procedure Her main conclusion was that explicit and transparent priority setting for health technology assessment is feasible, but that some important methodological issues need to be addressed to ensure that the procedure used is valid, reliable, consistent and useful for policy making.
There is some debate, variation in practice, and limited data regarding involvement of the public in priority setting. There is limited evidence from a small survey in Australia that the public overwhelmingly want their preferences to inform priority-setting decisions [19].

Should WHO have a centralised or decentralised process?
There are two ways in which priority setting is currently decentralised: geographically (across headquarters, regional offices and countries), and across technical departments. There are limited findings in the literature to inform decisions about how this might best be handled. The IOM noted, "that it is unreasonable -indeed impossible -to expect nationally developed guidelines to cover every operational issue for every kind of setting". "Yet guidelines that leave too much to be decided at the local level or during implementation run the risk of being ignored, misused, and modified in ways detrimental to patients." This is even more so for internationally developed guidelines. Priority setting at each level should draw on the strengths and minimize the limitations of international, national and local organizations. Thus, both centralised and decentralised processes that take account of these different strengths and limitations, as well as needs, are necessary.

Discussion
WHO has limited resources and limited technical capacity for developing recommendations. It is essential that it should set priorities for how best to use the resources and capacity it has. We did not find an empirical basis for deciding how best to set priorities. However, the use of explicit criteria and systematic processes are more likely than implicit criteria and non-systematic processes to ensure open and defensible priority setting. Based on the experience of other organisations, logic and the aims and strategic advantages of WHO we suggest that the following criteria should be used to set priorities: • Problems associated with a high burden of illness in low and middle-income countries, or new and emerging diseases.
• No existing guidelines or recommendations of good quality.
• The feasibility of developing recommendations that will improve health outcomes, reduce inequities or reduce unnecessary costs if they are implemented.
• Implementation is feasible, will not exhaustively use available resources, and barriers to change are not likely to be so high that they cannot be overcome.
• Additional priorities for WHO include interventions that will likely require system changes and interventions where there might be a conflict in choices between individual and societal perspectives.
The application of these criteria requires judgements. Appropriate processes are needed, in addition to explicit criteria, to ensure that these judgements are made openly, that they are taken account of in how WHO uses its resources, and that they reflect the priorities of WHO's member states, particularly those of low and middleincome countries. We suggest that the following processes be used for these reasons: • The allocation of resources to the development of recommendations should be part of the routine budgeting process rather than a separate exercise.
• Criteria for establishing priorities should be applied using a systematic and transparent process.
• Because data to inform judgements are often lacking, unmeasured factors should also be considered -explicitly and transparently.
• The process should include consultation with potential end users and other stakeholders, including the public, using well-constructed questions, and possibly using Delphi-like procedures.
• Groups that include stakeholders and people with relevant types of expertise should make decisions. Group processes should ensure full participation by all members of the group.
• The process used to select topics should be documented and open to inspection.
Both centralised and decentralised processes should be used to take account of different strengths, limitations and needs within WHO across headquarters, regions and countries; and across different technical areas. Drawing on the suggestion of the IOM for having different tracks for considering issues for specific populations, conditions or concerns [14], we suggest: • Both centralised and decentralised processes should be used. Decentralised processes can be considered as separate "tracks".
• Separate tracks should be used for considering issues for specific areas, populations, conditions or concerns. The rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration.
• Updating of guidelines could also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.

Further work
Many organisations are now using explicit and systematic priority setting processes for practice guidelines and health technology assessments. A more comprehensive and systematic survey of this experience could inform decisions about processes WHO should use to set priorities for recommendations. Because there is uncertainty about the best ways to set priorities, the processes that are used should be evaluated. When feasible and relevant, alternative processes should be directly compared with respect to the priorities that are generated and the resources that are used.