Skip to main content

Table 2 Interpreting the results of systematic reviews of effects

From: SUPPORT Tools for evidence-informed health Policymaking (STP) 8: Deciding how much confidence to place in a systematic review

The following questions can help to guide policymakers in interpreting the findings of systematic reviews of effects (adapted from [33, 47, 48])*:

• What estimate of effect is presented? Many reviews present an average estimate of effect across the included studies. This is often in the form of a risk ratio, odds ratio, or standardised mean difference

• Is an average estimate of effect across studies appropriate? Reviews use statistical methods to summarise and combine outcome data from the studies included in the review. To ensure that the combining of outcome data is appropriate, it is useful to consider whether the included studies were sufficiently similar in terms of population, intervention, comparison, and the outcomes measured. Where an average estimate of effect is not possible, reviews usually present a narrative overview of the available data

• Are confidence limits for the estimate of effect presented? The review should present confidence intervals around the average estimate of effect. The wider the confidence interval the less certain we can be about the true magnitude of the effect

• If the results of subgroup analyses are reported, are these appropriate? A review may present findings for a particular subgroup of participants across all trials or for a subgroup of studies [49]. For example, a review of interventions to reduce diarrhoeal diseases in children less than 5 years of age might also consider the effects of the interventions on children less than 1 year of age. Similarly, a review may include a subgroup analysis of studies judged as having a low risk of bias. A subgroup analysis should make sense in relation to both the overall review question and prior knowledge of factors that may have influenced or moderated the effects of the intervention. For example, it might be anticipated that a higher intensity intervention may produce larger effects. Subgroup analyses should be planned before a review is undertaken and less confidence should be placed in these particular results. This is because they are less reliable than analyses based on all of the included trials and because multiple statistical analyses may produce positive findings by chance alone

• If there is 'no evidence of effect' is caution taken not to interpret this as 'evidence of no effect'? 'No evidence of effect' is not the same as 'evidence of no effect'. The former suggests that insufficient evidence is available to draw conclusions regarding the effects of the intervention in question. The latter suggests that there is clear evidence from the included studies that the intervention does not have the anticipated effects [50]

• Do the conclusions and recommendations (if any) flow from both the original review question and the evidence that is presented in the review? It is important to consider whether the conclusions presented by the review authors emerge directly from the data gathered from the review and do not go beyond this evidence

• Is the evidence applicable to the policy question under consideration? Differences in health systems can mean that a programme or intervention that works in one setting may not work the same way in another. Policymakers need to assess whether the research evidence from a review applies in their setting. Guidance on this is presented in Article 9 in this series [28]

* There is some overlap between the questions listed here and those intended to guide assessment of the reliability of systematic reviews. This is because reliability is an important element in assessing and understanding the results of a systematic review