Clinical practice guidelines within the Southern African development community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases

Prioritising guidelines

We considered two issues: firstly, priority diseases should be representative of the following key components: adult & paediatric conditions; communicable & non-communicable diseases; chronic and acute onset diseases; maternal health; hospital-based & primary health care conditions. Secondly, time and feasibility limited our selection to five diseases. The following conditions were prioritised by the SARPAM study team: HIV in adults; malaria in children and adults; essential hypertension in primary care; pre-eclampsia; and diarrhoea in children.

Search strategy for guidelines

AGREE II instrument

Two reviewers (TK, AG) independently evaluated the global quality of the five CPGs for each of the SADC states and the reference guidelines using AGREE II. This tool is a recently revised and validated version of the AGREE instrument [14–18]. AGREE II contains 23 key quality items categorised in six domains scored with a 7-point Likert scale. Standardised guideline domain scores were calculated by summing scores of individual items and standardising the total as a percentage of the maximum possible score for that domain. The six domain scores are independent and were not aggregated into a single quality score. Uncertainties in the application of AGREE II were resolved in consultation with a third investigator (NS). We used Microsoft™ Excel to record the scores. As the data was nonparametric, we calculated a median (range) for each domain across countries to provide overall results.

Alignment with reference standard guidelines and expert opinion

Search results

Summary of AGREE II findings

We present the results according to diseases (Table 4). Matrices are shown which report the intersection of AGREE II by domain and alignment with best evidence (Figures 1, 2, 3, 4 and 5). AGREE II evaluation of the reference guidelines is shown in Figure 6.

Priority diseases	Domains of AGREE II
	Scope and purpose	Stakeholder involvement	Rigor of development	Clarity and presentation	Applicability	Editorial independence
HIV (n = 10)	57(19-78)	43(22-58)	16(6-30)	88(33-94)	22(0-58)	4(0-29)
Malaria (n = 11)	71(19-89)	38(6-53)	20(6-32)	88(17-97)	15(0-52)	0(0-25)
Pre-eclampsia (n = 8)	58(31-83)	36(22-61)	14(5-20)	75(36-89)	10(0-27)	0(0-25)
Diarrhoea in children (n = 7)	58(25-83)	36(22-58)	14(6-20)	83(42-100)	10(0-27)	4(0-25)
Hypertension (n = 9)	75(39-92)	42(22-64)	11(6-44)	81(53-97)	10(0-42)	4(0-50)
Overall	58(19-92)	39(6-75)	14(5-44)	83(17-100)	10(0-58)	0(0-50)

Alignment of CPGs with reference standards

Key recommendations from the reference guidelines were identified with input from experts in the respective fields.

Summary of main findings

HIV and malaria were most likely to have disease-specific guidelines which may reflect the global funding streams and political impetus targeting these conditions. The other priority diseases occupied sections within larger primary care CPGs. Our review found that the sections within other manuals that we evaluated were not comprehensive and provided incomplete guidance and were less likely to be up-to-date. Overall, the 'scope and purpose' and 'clarity and presentation' domains of the AGREE II tool were reported most comprehensively. However, the purpose, health question and target users were not explicitly described. Rather, the information was incorporated within the introduction and foreword sections and required extraction in order to identify the scope and objectives.

Most documents employed clear and consistent methods for identifying key recommendations, such as flow diagrams, tables and highlighted text, making the documents accessible for end-users and resulting in good scores in 'clarity and presentation'. The value of the clarity and presentation has been questioned as it does not strictly reflect the internal and external validity of a guideline document [28]. However, the usability of a guideline may impact on the applicability of the document. Target-users are not necessarily trained to discriminate on the quality of the guideline, but may be encouraged to use it if simple to navigate and apply [29].

The remaining four domains scored poorly across all diseases. The guidelines described the 'stakeholder involvement' of multidisciplinary professional groups; however, little was reported about the contribution of primary-care doctors and target patients. This is increasingly recognised as important for assuring that guidelines represent the needs of both the target users and patients. Involving these groups in the guideline development process, for example by pre-testing the guideline, or evaluating and incorporating preferences and values, may aid in securing the successful implementation of the guideline [30, 31].

In our study, as in previous studies, 'rigour of development' scored poorly [12]. A minority of guidelines provided references to the primary data and despite this many guidelines were highly aligned with current evidence (Figures 1, 2,3, 4 and 5). A plausible explanation is that the data required to evaluate this domain may exist in supporting documentation, such as appendices, which our search failed to locate. In addition, many of the SADC guidelines base their recommendations on other reference guidelines, such as WHO publications. Within SADC there may be members that have the capacity to appraise, synthesise and apply current evidence but generally it is accepted that few SADC countries are currently equipped with the necessary technical and financial resources. Despite this, had the guidelines we assessed clearly referenced their source guideline, they should have scored higher in this domain. In future similar evaluations, it may be prudent to augment the 'rigour of development' domain to clearly interrogate the source of the guidelines document, including whether it is based on another reference guideline. Many of the guideline documents we evaluated indicated that there would be a process for updating but none were explicit in their methods or the timing of updates. An important finding from this report is the lag between revisions of some of the guidelines with the result that the recommendations are no longer informed by current evidence potentially posing a risk to public health.

The methods necessary to successfully implement the guidelines, were not clearly delineated, hence the low scoring 'applicability domain'. Facilitators and barriers to applying the guidelines should be described to support implementation. The process of defining facilitators and barriers to application should be integrated early in the guideline development process and include professionals proficient in implementation strategies [29].

The low score in the 'editorial independence' domain reflects the poor reporting of potential conflicts of interest of stakeholders and the potential influence of funders in the guideline development process. Although the absence of these declarations does not necessarily imply that inappropriate influences guided the final recommendations, the presence of such declarations ensures that a guideline can be considered trustworthy [8, 13, 32].

Higher alignment scores were attained when guidelines were dedicated to a specific illness as seen with the malaria, HIV and hypertension guidelines. Gaps in the key recommendations occurred when the guidelines were out-of-date, occurring more frequently in the primary care CPGs. Pre-eclampsia scored poorly for alignment-indicating that the primary care CPGs we evaluated did not reflect current evidence. This condition requires hospital-based care and we did not identify any secondary or tertiary hospital guidelines during this review. Good alignment was achieved in guidelines despite poor scores in the 'rigour of development' domain-indicating a possible mismatch of the tool with the local practice of basing guidelines on WHO or equivalent high quality guidelines.

Agreement with previous research

A systematic review evaluating 42 guideline appraisal studies, including 626 guidelines, between 1988 and 2007 using the AGREE tool found similar distributions of low and high scores, supporting the notion that the domains within the guidelines that require improvement are similar despite disease or region [12]. Our scores for rigour of development, editorial independence and applicability were substantially lower than those described-indicating areas that require particular attention for future guideline development within SADC. Our study further highlights the need for support to improve the quality of guidelines by implementation of current normative standards of reporting within guidelines such as those developed by established guideline developers [33–35].

Strengths and limitations

We were mandated by the SADC Secretariat Pharmaceuticals Programme and therefore received cooperation from the ministries within the member states to assist with locating the relevant guidelines. We have attempted to address the research-knowledge gap by communicating a technical summary of the results to the SADC secretariat with specific recommendations for improving the availability, content and quality of CPGs within SADC. Although the AGREE II tool has been adopted widely as the reference tool to be used to evaluate guideline quality, this is the first time, to our knowledge, that it has been systematically applied across several diseases in a number of resource-constrained countries in Africa. This study can therefore contribute to a validation of the AGREE II tool and support uptake in our setting. The assessment of the alignment of the contents of the CPGs in this review was conducted with both published normative standard guidelines, such as WHO guidelines, and the input from experts in the respective fields.

We did not locate all relevant documents given the absence of a central or country-level repository. The outstanding documents would be required to provide a representative baseline analysis for SADC. The guidelines we evaluated included a combination of disease-specific guidelines and sections within larger primary care manuals. These guidelines may not lend themselves to be pooled in analysis, but do provide a true reflection of current guidance of the management of the included diseases. AGREE collaborators recommend that increasing the number of reviewers increases validity [11, 19]. Cost and time constraints dictated the feasible number of content experts and reviewers for this evaluation. There is currently no validated method for assessing alignment with evidence; therefore we used a method that will need future review to assess validity. Lack of timely translation prevented us from reviewing the guidelines from French- and Portuguese-speaking countries. This should be addressed in a future evaluation. The overall appraisal of quality of the guidelines would be enhanced by supplementary consultation and interview-based data collection with ministries, giving particular attention to guideline development processes and strategies and the roles of various members of the ministries of health, scientists and technical experts in formulating the guidelines [10]. Although the AGREE II tool may be applied across regions and settings, our experience suggests that 'rigour of development' domain may have scored more poorly than warranted, as the majority of SADC guidelines rely on guidance from the WHO, and therefore do not reference primary research as the domain requires. For this reason we recommended that this domain be amended for future evaluations for use in our setting.

What have we learned?

It is important that gaps in the availability of CPGs be identified and addressed. A repository of all guidelines in an accessible database will facilitate access for all SADC member states. It will facilitate cataloguing of guidelines and enable identification of those that may be relevant but missing or out-of-date. This could be done in collaboration with organisations such as the Guideline International Network [36].

There may be value in creating a SADC guideline support committee, through the SADC Pharmaceutical Programme, to assist all member states to adapt, maintain and update in-country guidelines of high standard [37]. This will facilitate that expertise in guideline development be shared. This committee may enlist expertise in reviewing current evidence with regards its applicability and generalisability to local healthcare needs. Such a committee should include, amongst others, the following relevant stakeholders-content experts; funders; policy makers; public health professionals; physicians, nurses and pharmacy staff; patient representative groups and an external review committee. Collaboration with experts in the field of guideline development could support capacity development and aid the process of bridging research and practice.

The value of this review has been to identify specific gaps in the quality and content of the guidelines within SADC. There is increasing awareness that the transfer of research evidence into policy and practice is a complex issue, sensitive to the context of each country. In order to inspire confidence in the quality and evidence-base of guideline recommendations and in the transparency of the development process, each newly developed or updated guideline should adhere to the recommended reporting norms currently in use globally.