Table 2 Guidelines for interpreting subgroup analyses

Is the magnitude of the difference important?

If the magnitude of a difference between subgroups will not result in different decisions for different subgroups, then the overall results can be used.

Is the difference between subgroups statistically significant?

To establish whether a policy or programme has a different effect in different situations, the magnitudes of effects in different subgroups should be compared directly with each other. The statistical significance of the results within separate subgroup analyses should not be compared, as this is likely to be misleading. For example, if a subgroup analysis showed that the effect of a policy or programme was not statistically significant for women but was statistically significant for men, it is likely that this could simply be because few women were included in the studies. It does not answer the question of whether the difference between the size of the effect in women and men was greater than would otherwise have been expected if this had occurred by chance. If there is both an important difference in effects and that difference is statistically significant (i.e. it is unlikely to have occurred by chance), then serious consideration should be given to basing a decision on the subgroup analysis rather than on the overall analysis.

Is there indirect evidence in support of the findings?

Indirect evidence is research that has not directly compared the options in which we are interested in the populations in which we are interested, or measured the important outcomes in which we are interested. For differences between subgroups to be convincing, they should be plausible and supported by other external or indirect evidence. For example, research that has measured intermediary outcomes (not the ones in which we are interested) can provide evidence of a plausible mechanism for differential effects. For subgroup analyses for disadvantaged groups, there should be a similarly plausible reason - supported by indirect evidence - to anticipate differential effects.

Was the analysis pre-specified or post hoc?

Researchers should state whether subgroup analyses were pre-specified or undertaken after the results of the studies had been compiled (post hoc). Greater reliance may be placed on a subgroup analysis if it formed part of a small number of pre-specified analyses. Performing numerous post hoc subgroup analyses could be seen as data dredging, a process that is inherently unreliable. This is because it is usually possible to find an apparent - but false - explanation for differences in effects when considering many different characteristics.

Are analyses looking at within-study or between-study relationships?

Differences in subgroups that are observed within studies are more reliable than analyses of subsets of studies. If such within-study relationships are replicated across studies then this will add confidence to the findings.