Skip to main content

Improving the use of research evidence in guideline development: 14. Reporting guidelines

Abstract

Background

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 14th of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

Objectives

We reviewed the literature on reporting guidelines and recommendations.

Methods

We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.

Key questions and answers

There is little empirical evidence that addresses these questions. Our answers are based on logical arguments and standards put forward by other groups.

What standard types of recommendations or reports should WHO use?

  • WHO should develop standard formats for reporting recommendations to facilitate recognition and use by decision makers for whom the recommendations are intended, and to ensure that all the information needed to judge the quality of a guideline, determine its applicability and, if needed, adapt it, is reported.

  • WHO should develop standard formats for full systematically developed guidelines that are sponsored by WHO, rapid assessments, and guidelines that are endorsed by WHO.

  • All three formats should include the same information as full guidelines, indicating explicitly what the group preparing the guideline did not do, as well as the methods that were used.

  • These formats should be used across clinical, public health and health systems recommendations.

How should recommendations be formulated and reported?

  • Reports should be structured, using headings that correspond to those suggested by the Conference on Guideline Standardization or similar headings.

  • The quality of evidence and strength of recommendations should be reported explicitly using a standard approach.

  • The way in which recommendations are formulated should be adapted to the specific characteristics of a specific guideline.

  • Urgent attention should be given to developing a template that provides decision makers with the relevant global evidence that is needed to inform a decision and offers practical methods for incorporating the context specific evidence and judgements that are needed.

Peer Review reports

Background

The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 14th of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.

Guidelines are formal advisory statements that should be robust enough to meet the unique circumstances and constraints of the specific situation to which they are being applied [1]. The basic nature and intent of guidelines have been variously labelled as guidance, guides, guiding principles, recommendations, policies, protocols, best practice, algorithms, consensus statements, expert committee recommendations, integrated care pathways, manuals, tool kits, handbooks, model lists, technical updates and principles [1, 2]. Whatever they are called, rigorously developed guidelines, can translate complicated research findings into actionable recommendations. They are an important step in moving from research to action and ensuring that the best available research evidence informs decisions and actions [3]. However, for users of guidelines to be able to apply criteria to assess whether guidelines have been rigorously developed and are likely to be valid and applicable [36], the information needed to make these judgements must be reported [7]. Unfortunately, critical information is often absent from published guidelines [5, 7, 8].

In this paper we address the following questions:

  • What standard types of recommendations or reports should WHO use?

  • How should recommendations be formulated and reported?

We address questions about reporting systematic reviews [9] and dissemination [10] in other papers in this series.

What WHO is doing now

Although the Guidelines for WHO Guidelines recommends "that a uniform, readily-recognizable printing format be developed for WHO guidelines," there are, as yet, no standard formats for WHO policies, recommendations or guidelines. A survey of WHO guidelines published in 2005 found that WHO publishes a large number of recommendations of many different types, in many different formats [2], and a review of WHO documents did not find any standards for reporting WHO recommendations [11].

What other organisations are doing

In a recent survey of organisations that produce clinical practice guidelines, all 31 organisations that responded (response rate 86%), and 46 of 57 (81%) of units that support the use of research evidence by governments in developing health policy, reported producing full versions of guidelines with references and notes [12]. Several organisations use different formats for different types of recommendations, and a majority produce different versions of guidelines, such as executive summaries, summaries of take-home messages, separate versions for different target users, and tools for application (e.g., algorithms or flow charts). Many guideline producers have standard formats that they use and some organisations, such as the U.S. National Guidelines Clearing House, have developed standard formats for reporting guidelines produced by other organisations [13].

The UK National Center for Health and Clinical Excellence (NICE) states that recommendations should be clear and concise, but should contain sufficient information that they can be understood without reference to other supporting material [14]. This is particularly important where recommendations are published in isolation from the background details in the full guideline. Any terminology included in the recommendations therefore needs to be clearly defined and unambiguous.

Methods

The methods used to prepare this review are described in the introduction to this series [15]. Briefly, the key questions addressed in this paper were vetted amongst the authors and the ACHR Subcommittee on the Use of Research Evidence (SURE). We did not conduct a full systematic review. We searched PubMed and three databases of methodological studies (the Cochrane Methodology Register [16], the US National Guideline Clearinghouse [17], and the Guidelines International Network [18]) for existing systematic reviews and relevant methodological research that address these questions. The answers to the questions are our conclusions based on the available evidence, consideration of what WHO and other organisations are doing, and logical arguments.

For this review we searched PubMed using [the MeSH terms 'Documentation/standards' and 'Practice Guidelines/standards'] and related articles; the Cochrane Methodology Register using [the key words 'CMR: Review methodology – applicability & recommendations' and the text words (format or reporting)] and ['Levels of evidence and strength of recommendations']; the National Guidelines Clearinghouse annotated bibliography using the terms format, reporting and structure; and checked the reference lists of retrieved articles.

Findings

What standard types of recommendations or reports should WHO use?

Given the wide variety of different types of recommendations that are made by WHO, there is likely to be a need for several standard types of recommendations. Systematically developed clinical practice guidelines can take 18 months or more and as many as 15 meetings [19]. Systematically developed public health guidelines also require substantial resources and time [20]. Given the time and resources required to produce guidelines, many organisations, particularly HTA organisations, have developed rapid assessment processes [2125]. There is variation in the scope, methods, time to complete assessments, and the formats used to report rapid assessments. Another type of recommendation or guideline that is receiving increasing attention, also because of the resources and time required to develop guidelines systematically, are guidelines developed by other organisations that have been adapted or endorsed [2628]. Another approach being taken in several countries is to create databases or clearinghouses of clinical practice guidelines with the aim of facilitating their evaluation and adaptation for local use by health care organizations [29].

Systematically developed clinical recommendations, public health recommendations, and health systems recommendations all require similar processes to ensure their quality. Decision makers also require similar types of information to be able to critically appraise whether guidelines have been rigorously developed and are likely to be valid [3, 4, 7, 20, 30, 31].

How should recommendations be formulated and reported?

The Conference on Guideline Standardization (COGS) used a two-stage modified Delphi process to develop standards for reporting clinical practice guidelines [7]. Representatives of 22 organisations active in guideline development reviewed the proposed items and commented favourably. The items were consolidated into 18 topics (Table 1) to create the COGS checklist, which provides a framework to support comprehensive documentation of guidelines. While it is possible that some guideline developers may not include content for every item, it is suggested that they should address explicitly whether the guideline development team considered that item.

Table 1 The COGS checklist for reporting clinical practice guidelines (from Shiffman et al. [7])*

While many organisations have their own standard formats for reporting guidelines, this is the only consensus standard for reporting guidelines across organisations. We have, however, summarised the key items included in guidelines for guidelines, which provides the basis for a similar, but more comprehensive checklist for conducting or reporting guidelines [32]. In addition, there are a number of instruments for evaluating clinical practice guidelines that can also be used as checklists for reporting [1, 46, 33]. The content used in the National Guideline Clearing house also represents a standard for reporting imposed on organisations that want their guidelines included in that database [13]. It includes 52 items under the following headings: scope, methodology – including rating scheme and cost analysis, recommendations, evidence supporting the recommendations, benefits/harms of implementing the recommendations, contraindications, qualifying statements, implementation of the guideline, Institute of Medicine (IOM) national healthcare quality report categories, identifying information and availability, and disclaimer; in addition to indexing attributes.

Similarly, some journals have standard formats for reporting clinical practice guidelines, including structured abstracts with the following headings [34]:

Objective

a succinct statement of the objective of the guideline, including the targeted health problem, the targeted patients and providers, and the main reason for developing recommendations concerning this problem for this population.

Options

principal practice options that were considered in formulating the guideline.

Outcomes

significant health and economic outcomes identified as potential consequences of the practice options.

Evidence

methods used to gather, select, and synthesize evidence, and the date of the most recent evidence obtained.

Values

persons and methods used to assign values (relative importance) to potential outcomes of alternative practice options.

Benefits, harms, and costs

the type and magnitude of the main benefits, harms, and costs that are expected to result from guideline implementation.

Recommendations

a brief and specific list of key recommendations.

Validation

the results of any external review, comparison with guidelines developed by other groups, or clinical testing of guideline use.

Sponsors

key persons or groups that developed, funded, or endorsed the guideline.

While many organisations have standards for how recommendations are formulated, we are not aware of any consensus standards for how recommendations should be formulated. Most guidelines development groups now grade the quality of evidence and the strength of recommendations, but a variety of different grading systems are used [12, 35].

Shekelle and colleagues evaluated the effect of different levels of specificity of recommendations on clinicians test ordering behaviour using clinical vignettes [36]. They found that clinicians receiving the non-specific recommendations ordered fewer indicated tests for appropriate clinical vignettes than did physicians receiving specific recommendations. Furthermore, compared to physicians receiving non-specific guidelines, physicians receiving specific guidelines ordered significantly more appropriate tests for corresponding vignettes and significantly fewer tests for inappropriate vignettes. The authors concluded that the clarity and clinical applicability of a guideline might be important attributes that contribute to the effects of practice guidelines. We did not find any other comparisons of different ways of formulating recommendations, and it is likely that the way in which recommendations are formulated may need to be adapted to the specific characteristics of a guideline. However, there is a consensus that recommended actions should be stated precisely [7].

Both checklists for evaluating guidelines and for reporting guidelines include items that may be dependent on the specific setting in which a guideline is developed or used. There are a number of international organisations, including WHO, that develop guidelines that are intended to be used in a variety of settings around the world. We did not find any published papers that addressed methods for taking into account setting specific factors in international guidelines, although several groups are working on methods for adapting guidelines developed in one setting for use in another [27, 28, 3739].

Discussion

While the content of WHO guidelines, recommendations and policies will vary, depending on the topic, it would be desirable to have standard formats across different topics to facilitate recognition and use by decision makers and ensure that all the information needed to judge the quality of a guideline, determine its applicability and, if needed, adapt it is reported. There is likely a need for at least three standard formats: full systematically developed guidelines that are sponsored by WHO, rapid assessments, and guidelines that are endorsed by WHO. Standards such as those advocated by COGS should form the basis for developing a uniform format for full guidelines developed by WHO. Although the COGS standards were developed for clinical practice guidelines, the same considerations are relevant to public health and health systems recommendations. Further consideration is needed regarding the inclusion of additional items that need to be considered in WHO guidelines, such as applicability to different settings, equity, and scaling up. In particular, work is needed to develop a template for decision-making frameworks when different conditions are likely to lead to different decisions in different settings [40, 41]. In addition, different versions of guidelines should be developed for different target audiences, including a structured executive summary and key messages [31, 34].

A different format for rapid assessments would help to distinguish these from full guidelines and could be designed to reduce the work and time necessary to complete a report. Rapid assessments should, nonetheless, include the same information as full guidelines, indicating explicitly what the group preparing the guideline did not do, as well as the methods that were used.

WHO has limited resources and capacity for developing guidelines. At the same time, low and middle-income (LMIC) countries also have limited resources and capacity, and a core function of WHO is to provide its member states, particularly LMIC, with technical advice that is informed by the best available research evidence. Through collaborating with other organisations and establishing standards for reporting, and possibly endorsing guidelines developed by other organisations, WHO may be able to expand its potential for supporting access to guidelines that are appropriate for LMIC or can easily be adapted to conditions in LMIC.

Given WHO's role as the world's leading public health agency and its mandate to provide evidence-informed guidance, it is of major importance for WHO to address and develop methods for formulating and reporting recommendations that are developed internationally, but need to be adapted and implemented in specific settings.

Further work

As part of the methodological work that is needed to addresses the challenges of developing international guidelines, specific attention should be given to the development of standard templates that provide decision makers with the relevant global evidence that is needed to inform a decision and offers practical methods for incorporating the context specific evidence and judgements that are needed [40, 41].

In addition to ensuring that standard formats are used for WHO guidelines to ensure complete reporting, attention should be paid to ensuring that the format that is used, and derivative versions, are understandable and useful to the intended target audiences.

References

  1. Global Programme on Evidence for Health Policy: Guidelines for WHO Guidelines. Geneva: World Health Organisation, 2003 (EIP/GPE/EQC/2003.1)

  2. Panisett U: A review of WHO recommendations published in 2005.

  3. Guyatt G, Yayward R, Richardson WS, Green L, Wilson M, Sinclair J: Moving from evidence to action. Users' Guides to the Medical Literature. Edited by: Guyatt G, Rennie D. 2002, Chicago: AMA Press, 175-99.

    Google Scholar 

  4. Institute of Medicine: Guidelines for Clinical Practice: From Development to Use. 1992, Washington, DC: National Academy Pr

    Google Scholar 

  5. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer reviewed medical literature. JAMA. 1999, 281: 1900-5. 10.1001/jama.281.20.1900.

    Article  CAS  PubMed  Google Scholar 

  6. The AGREE Collaboration: Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003, 12: 18-23. 10.1136/qhc.12.1.18.

    Article  Google Scholar 

  7. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM: Standardized Reporting of Clinical Practice Guidelines: A Proposal from the Conference on Guideline Standardization. Ann Intern Med. 2003, 139: 493-8.

    Article  PubMed  Google Scholar 

  8. Grilli R, Magrini N, Penna A, Mura G, Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet. 2000, 355: 103-6. 10.1016/S0140-6736(99)02171-6.

    Article  CAS  PubMed  Google Scholar 

  9. Oxman AD, Schünemann HJ, Fretheim A: Improving the Use of Research Evidence in Guideline Development: 8. Synthesis and presentation of evidence. Health Res Policy Syst. 2006

    Google Scholar 

  10. Fretheim A, Schünemann HJ, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 15. Disseminating and implementing guidelines. Health Res Policy Syst. 2006

    Google Scholar 

  11. Oxman AD, Fretheim A, Schünemann HJ: A review of WHO documents that guide the development, dissemination and implementation of recommendations by WHO.

  12. Moynihan R, Oxman AD, Lavis JN, Paulsen E: Evidence-Informed Health Policy: Using Research to Make Health Systems Healthier. A review of organizations that support the use of research evidence in developing guidelines, technology assessments, and health policy, for the WHO Advisory Committee on Health Research. 2006, Oslo: Norwegian Knowledge Centre for the Health Services

    Google Scholar 

  13. Guideline Summary Sheet. National Guideline Clearinghouse. [http://www.guideline.gov/about/CompleteSummaryDescrip.aspx]

  14. What are the key components of a template for recommendations could be the question here?. National Instutites for Health and Clinical Excellence Handbook. [http://www.nice.org.uk/pdf/GDM_Allchapters_0305.pdf]

  15. Oxman AD, Schünemann HJ, Fretheim A: Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst. 2006

    Google Scholar 

  16. Cochrane Methodology Register. [http://www.cochrane.org/access_data/cmr/accessDB_cmr.asp]

  17. National Guideline Clearinghouse. [http://www.guidelines.gov]

  18. Guidelines International Network (GIN). [http://www.g-i-n.net]

  19. Raine R, Sanderson C, Black N: Developing clinical practice guidelines: a challenge to current methods. BMJ. 2005, 331: 631-3. 10.1136/bmj.331.7517.631.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Briss PA, Zaza S, Pappaioanou M: Developing an evidence-based guide to community preventive services-methods. Am J Prev Med. 2000, 18 (1S): 35-43. 10.1016/S0749-3797(99)00119-1.

    Article  CAS  PubMed  Google Scholar 

  21. Stevens A, Colin-Jones D, Gabbay J: 'Quick and clean': authoritative health technology assessment for local health care contracting. Health Trends. 1995, 27: 37-42.

    CAS  PubMed  Google Scholar 

  22. Hailey D, COrabian P, Harstall C, Schneider W: The use and impact of rapid health technology assessments. Int J Technol Assess Health Care. 2000, 16: 651-6. 10.1017/S0266462300101205.

    Article  CAS  PubMed  Google Scholar 

  23. Perleth M, Luhmann D, Givis B, Droste S: Rapid Assessments – quick evaluation of medical technology. Gesundheitswesen. 2001, 63 (Suppl 1): S79-84. 10.1055/s-2001-12121.

    Article  PubMed  Google Scholar 

  24. Canadian Coordinating Office for Health Technoloty Assessment: Rapid response services in health technology assessment: information from INAHTA members and additional information. 2004, Ottawa: CCOHTA

    Google Scholar 

  25. Milne R, Clegg A, Stevens A: HTA responses and the classic HTA report. J Public Health Med. 2003, 25: 102-6. 10.1093/pubmed/fdg023.

    Article  PubMed  Google Scholar 

  26. Schünemann H, Fretheim A, Oxman AD: Adaptation, applicability, transferability.

  27. Graham ID, Harrison MB, Brouwers M, Davies BL, Dunn S: Facilitating the use of evidence in practice: evaluating and adapting clinical practice guidelines for local use by health care organizations. J Obstet Gynecol Neonatal Nurs. 2002, 31: 599-611. 10.1111/j.1552-6909.2002.tb00086.x.

    Article  PubMed  Google Scholar 

  28. Graham ID, Harrison MB, Lorimer K, Piercianowski T, Friedberg E, Buchanan M, Harris C: Adapting national and international leg ulcer practice guidelines for local use: the Ontario Leg Ulcer Community Care Protocol. Adv Skin Wound Care. 2005, 18: 307-18. 10.1097/00129334-200507000-00011.

    Article  PubMed  Google Scholar 

  29. Guideline Resources. National Guideline Clearinghouse. [http://www.guideline.gov/resources/guideline_resources.aspx]

  30. Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EAF, Millson ME: An approach to the development of practice guidelines for community health interventions. Can J Public Health. 1994, 85 (Suppl 1): S8-13.

    PubMed  Google Scholar 

  31. Lavis JN, Davies HTO, Oxman AD, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform healthcare management and policymaking. Journal of Health Services Research and Policy. 2005, 10 (Suppl 1): 35-48. 10.1258/1355819054308549.

    Article  PubMed  Google Scholar 

  32. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006

    Google Scholar 

  33. Graham ID, Calder LA, Hebert PC, Carter AO, Tetroe JM: A comparison of clinical practice guideline appraisal instruments. Int J Technol Assess Health Care. 2000, 16: 1024-38. 10.1017/S0266462300103095.

    Article  CAS  PubMed  Google Scholar 

  34. Hayward RS, Wilson MC, Tunis SR, Bass EB, Rubin HR, Haynes RB: More informative abstracts of articles describing clinical practice guidelines. Ann Intern Med. 1993, 118: 731-7.

    Article  CAS  PubMed  Google Scholar 

  35. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 9. Grading evidence and recommendations. Health Res Policy Syst. 2006

    Google Scholar 

  36. Shekelle PG, Kravitz RL, Beart J, Marger M, Wang M, Lee M: Are nonspecific practice guidelines potentially harmful? A randomized comparison of the effect of nonspecific versus specific guidelines on physician decision making. Health Serv Res. 2000, 34: 1429-48.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. INCLEN Knowledge 'Plus' Program (K+). The INCLEN Trust. [http://www.inclentrust.org/index.php?option=content%26task=view%26id=182%26Itemid=223#KPP]

  38. Adapting existing HTAs from one country to other. European Network for Health Technology Assessment – EUnetHTA. [http://www.sst.dk/upload/planlaegning_og_behandling/cemtv/eunethta/wp5description.pdf]

  39. Fervers B, Graham I: Guideline Guideline adaptation: the way forward?. The Guidelines International Network. [http://www.g-i-n.net/download/files/Plenary_2__Fervers_Graham.pdf]

  40. Oxman AD, Fretheim A, Schünemann HJ: Improving the Use of Research Evidence in Guideline Development: 7. Deciding what evidence to include. Health Res Policy Syst. 2006

    Google Scholar 

  41. Schünemann HJ, Fretheim A, Oxman AD: Improving the Use of Research Evidence in Guideline Development: 13. Adaptation, applicability and transferability. Health Res Policy Syst. 2006

    Google Scholar 

Download references

Acknowledgements

We gratefully acknowledge WHO and the Norwegian Knowledge Centre forthe Health Services for financial support, the other authors of these reviews for their contributions, the other members of the ACHR SURE, and all of those who have provided us with feedback on earlier versions of this paper and the others in this series.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Andrew D Oxman.

Additional information

Competing interests

ADO and AF work for the Norwegian Knowledge Centre forthe Health Services, an agency funded by the Norwegian government that produces systematic reviews and health technology assessments. All three authors are contributors to the Cochrane Collaboration. ADO and HJS are members of the GRADE Working Group. HJS is documents editor and chair of the documents development and implementation committee for the American Thoracic Society and senior editor of the American College of Chest Physicians' Antithrombotic and Thrombolytic Therapy Guidelines.

Authors' contributions

ADO prepared the first draft of this review. HJS and AF contributed to drafting and revising it.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Oxman, A.D., Schünemann, H.J. & Fretheim, A. Improving the use of research evidence in guideline development: 14. Reporting guidelines. Health Res Policy Sys 4, 26 (2006). https://doi.org/10.1186/1478-4505-4-26

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1478-4505-4-26

Keywords