- Open Access
SUPPORT Tools for evidence-informed health Policymaking (STP) 7: Finding systematic reviews
Health Research Policy and Systems volume 7, Article number: S7 (2009)
This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.
Systematic reviews are increasingly seen as a key source of information in policymaking, particularly in terms of assisting with descriptions of the impacts of options. Relative to single studies they offer a number of advantages related to understanding impacts and are also seen as a key source of information for clarifying problems and providing complementary perspectives on options. Systematic reviews can be undertaken to place problems in comparative perspective and to describe the likely harms of an option. They also assist with understanding the meanings that individuals or groups attach to a problem, how and why options work, and stakeholder views and experiences related to particular options. A number of constraints have hindered the wider use of systematic reviews in policymaking. These include a lack of awareness of their value and a mismatch between the terms employed by policymakers, when attempting to retrieve systematic reviews, and the terms used by the original authors of those reviews. Mismatches between the types of information that policymakers are seeking, and the way in which authors fail to highlight (or make obvious) such information within systematic reviews have also proved problematic. In this article, we suggest three questions that can be used to guide those searching for systematic reviews, particularly reviews about the impacts of options being considered. These are: 1. Is a systematic review really what is needed? 2. What databases and search strategies can be used to find relevant systematic reviews? 3. What alternatives are available when no relevant review can be found?
This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The series is intended to help such people ensure that their decisions are well-informed by the best available research evidence. The SUPPORT tools and the ways in which they can be used are described in more detail in the Introduction to this series . A glossary for the entire series is attached to each article (see Additional File 1). Links to Spanish, Portuguese, French and Chinese translations of this series can be found on the SUPPORT website http://www.support-collaboration.org/. Feedback about how to improve the tools in this series is welcome and should be sent to: STP@nokc.no.
Scenario 1: You are a senior civil servant and will be submitting a brief report to the Minister regarding evidence about a high-priority problem, options to address the problem, and implementation considerations. You are concerned about whether the current draft of the report profiles research evidence that has been synthesised in a systematic and transparent way. You want to ensure that your staff have found the most relevant systematic reviews in the limited time available to them.
Scenario 2: You work in the Ministry of Health and have been given a few hours to prepare a brief report about a problem, options to address it, and implementation considerations. All that you have been told is that the report should draw on any relevant systematic reviews that can be found within this time frame.
Scenario 3: You work in an independent unit that supports the Ministry of Health in its use of evidence in policymaking. You are preparing a detailed research report for the Ministry of Health about what is known and not known about a problem, options to address it, and implementation considerations. You have been told to find all relevant systematic reviews and you have been given two weeks to do this, but you want guidance on how to do this in a thorough and efficient way.
This article suggests a number of questions that policymakers (Scenario 1) might ask their staff to consider when preparing a brief report regarding the evidence about a high-priority problem, options to address the problem, and implementation considerations. For those who support policymakers (Scenarios 2 and 3), this article suggests a number of questions to guide the search for systematic reviews, particularly reviews about the impacts of options being considered. This article is the first of four articles in this series about finding and assessing systematic reviews to inform policymaking (see also Articles 8-10 [2–4]). Figure 1 outlines the steps involved in finding and assessing systematic reviews to inform policymaking.
Systematic reviews are increasingly seen as a key source of information for policymaking, particularly in assisting with framing options and describing their impacts . Systematic reviews offer four key advantages over single studies in characterising the impacts of an option:
They reduce the likelihood that policymakers will be misled by research (by being more systematic and transparent in the identification, selection, appraisal and synthesis of studies)
They increase confidence among policymakers about what can be expected from an option (by increasing the number of units for study)
They allow policymakers to focus on assessing the applicability of the findings of systematic reviews to their own setting (instead of also having to find and synthesise the available research evidence on their own). The reviews also allow policymakers to focus on collecting and synthesising other types of evidence, such as local evidence about technical feasibility, the fit with dominant values and the current provincial/national mood, and the acceptability of potential options in terms of budget workability and their likely degree of political support or opposition, and
They allow stakeholders, including public interest groups and civil society groups to contest research evidence constructively because it is arranged in the reviews in a more systematic and transparent way [5, 6]
The first two advantages listed above - namely, the reduction of bias and increase in precision (to use the terminology of researchers) - apply only to systematic reviews of impacts, some of which will include the statistical synthesis of findings as a final step. In these instances, the reviews are referred to as meta-analyses .
Although not the principal focus of this article, systematic reviews are also increasingly used as key sources of information in the clarification of problems and providing complementary perspectives on options. Systematic reviews can also be conducted for:
Administrative database studies and community surveys that help to place problems in comparative perspective
Observational studies that help to describe the likely harms of an option, and
Qualitative studies that help to understand the meanings that individuals or groups attach to a problem, how and why options work, and stakeholder views about (and experiences with) particular options
These issues are discussed further in Article 4 in this series (which focuses on the process of clarifying a problem) and Article 5 (which focuses on framing options to address a problem) [8, 9]. There are many methodological approaches that can be used in systematic reviews of qualitative research evidence (or in systematic reviews of both qualitative and quantitative research evidence within the same review) including a narrative summary, thematic analysis, grounded theory, meta-ethnography, a meta-study, realist synthesis, cross-case techniques, content analysis, a case survey, qualitative comparative analysis, and Bayesian meta-analysis [10, 11].
Several constraints have hindered the wider use of systematic reviews in policymaking. The first key constraint is the limited awareness of their value. Policymakers require synthesised research evidence and systematic reviews are able to provide this in a way that is both systematic and transparent. Many policymakers and researchers with influence in health systems initially believed that systematic reviews could only include randomised controlled trials and required some form of statistical synthesis . For them, the value of these reviews lay only in assessing the effectiveness of healthcare interventions. This belief was underpinned by misconceptions. A second key constraint relates to the retrievability of systematic reviews. Policymakers need timely access to relevant high-quality systematic reviews that are retrievable using the terminology of policymakers. A systematic review of the factors that influence the use of research evidence in policymaking found that timing/timeliness increased the likelihood of research being used by policymakers [6, 13]. In the past, policymakers have not been able to search databases using terms familiar to them  but this, as we discuss below, has now changed.
A third key constraint relates to the degree to which systematic reviews can be easily understood and interpreted. Policymakers need access to user-friendly summaries of systematic reviews that are written in ways that highlight what they need to know to clarify a problem or describe the costs and consequences of options to address the problem. In the past, even if searches were successful, they may have retrieved structured abstracts and full reviews that had been written in a way that failed to highlight the types of information that policymakers were seeking . Again, as we explain below, this situation has also changed.
Questions to consider
The following questions can guide policymakers in the process of finding systematic reviews to inform policymaking:
Is a systematic review really what is needed?
What databases and search strategies can be used to find a relevant systematic review?
What alternatives are available when no relevant review can be found?
1. Is a systematic review really what is needed?
Before conducting a search for systematic reviews it is first necessary to confirm whether a systematic review is really what is needed. Systematic reviews may be appropriate if, for example, a policy question that is posed addresses a specific health system arrangement or a specific programme, service or drug. They may also be useful for specific implementation strategies that target consumers (e.g. citizens and healthcare recipients) or healthcare providers (with or without some specification of the people, comparisons and outcomes of interest). Article 5 in this series addresses how to structure questions related to the impacts of options .
But an overview of systematic reviews could provide helpful information if the question at hand relates to a broad category (or several broad categories) of health system arrangements, programmes, services or drugs, or implementation strategies. A particular overview of systematic reviews, for example, was found to be helpful by many policymakers because it examined the impacts of a full array of options that could be used to improve the supply, distribution, efficient use and performance of healthcare providers . A policy brief that draws on a range of systematic reviews could also prove to be helpful. This would be the case if the question posed by policymakers addresses a spectrum of concerns ranging from the clarification of a problem, the framing of options and the description of their costs and consequences, through to key implementation considerations. The Program in Policy Decision-Making/Canadian Cochrane Network and Centre (PPD/CCNC) database described below could prove helpful in finding both overviews of systematic reviews and policy briefs, as well as systematic reviews. Policy briefs are described in further detail in Article 13 in this series .
Systematic reviews are likely to be unhelpful if a question pertains to local evidence, such as local evidence about on-the-ground realities and constraints, the values and beliefs of citizens, interest group power dynamics, institutional constraints, and donor funding flows. Article 11 addresses considerations related to finding and using local evidence to inform policymaking .
2. What databases and search strategies can be used to find a relevant systematic review?
When it has been decided that a systematic review is needed, and when the question that the review needs to address relates to the impacts of (or more generally what is known about) health system arrangements, the PPD/CCNC database can be prioritised as a search tool. This is because it is accessible without charge, it has a particular focus on health system arrangements, and it provides links to user-friendly summaries (and, in their absence, scientific abstracts) (see Table 1 for a description of this and other databases). The database captures both systematic reviews that address questions about impacts and systematic reviews that address other types of questions.
However, if the question that a review should answer relates to the description of the impacts of programmes, services or drugs, or of implementation strategies targeting consumers and healthcare providers, then policymakers can access two databases used more commonly by healthcare providers. (The 'Resources' section later in this paper provides links to the databases mentioned.) The Cochrane Library - and specifically the Cochrane Database of Systematic Reviews and the Database of Reviews of Effects contained within it - only captures systematic reviews that address questions about impacts (see Table 1). PubMed captures systematic reviews that address many types of questions. Hedges (i.e. validated search strategies) are available to assist with finding systematic reviews in PubMed. Hedges are also used to find systematic reviews in three other databases: CINAHL, EMBASE, and PsycINFO (see Additional File 2 later in this article).
Two additional points are important to consider. Firstly, within any of the databases, policymakers who are interested in describing impacts but are pressed for time, may want to give priority to reviews produced by the Cochrane Collaboration (otherwise known as Cochrane reviews). These reviews have been found to be of higher quality and are updated more frequently than reviews produced by other groups . Secondly, while health technology assessments (or HTAs) should typically include a range of economic, social, ethical and legal considerations, as well as a review of the research evidence about the effectiveness of a technology, some HTA reports contain a systematic review that can be applied in contexts other than the one for which the report was produced.
Table 2 provides an example of how groups of policymakers and those who support them can work together to find reviews to address a high-priority issue.
3. What alternatives are available when no relevant review can be found?
Despite improvements in the ease with which policymakers can search and find systematic reviews in available databases, there will be occasional instances when no review can be found. If policymakers are able to wait between 6 and 18 months (depending on the complexity of the question being asked) and have the necessary resources, one option could be to commission a systematic review from an experienced research group . If, however, the available timeline is shorter than this or resources are limited, policymakers can search for single studies instead. In doing so they are essentially conducting a review themselves, and the more systematically this is done the better. In such situations, policymakers can take issues related to ensuring the quality of reviews into consideration. A web-based tool to support such 'rapid evidence assessments' is described later in this paper, while a further discussion of the quality of reviews is provided in Article 8 .
Particular databases can also be prioritised when looking for single studies. PubMed, which includes over 20 million records, is often a good starting point. When searching PubMed, hedges can be used to restrict searches to the types of studies most relevant to a particular type of question. Hedges are also available for other databases. (Please refer to the 'Resources' section of this paper for a list of links to hedges that are particularly relevant to policymakers.)
Some policymakers will only require this basic level of detail related to finding systematic reviews or single studies if they have access to subscription databases and are able to rely on the expertise of librarians (Please see Additional File 2 for a list of subscription access databases). This may be either within their own organisation or through colleagues in other universities and settings. We have summarised additional details about high-priority databases in which to search for systematic reviews, including their content, how they can be searched, and what information is returned from search results (see Table 1). This is particularly useful for policymakers who want to gain access to additional information in order to establish clear expectations among those who support them, as well as for policymakers and librarians who will be conducting searches on their own.
Two additional points are worth noting. Firstly, there has been a steady growth in the number of groups and organisations providing user-friendly summaries highlighting the decision-relevant information contained in systematic reviews. Such summaries are usually an excellent place for policymakers to start (Article 13 provides additional detail about these summaries ). Secondly, terms have been identified for PubMed in order to help with the identification of systematic reviews and studies focused on low- and middle-income countries. This is particularly useful for policymakers based in these countries. (Additional File 3 at the end of this article provides a list of terms that can be used in searches for systematic reviews or studies focused on these countries.)
While many of the prioritised databases above provide free online access, such access often does not include full-text systematic reviews. In such cases, it will be necessary for policymakers and those who support them (and librarians) to make use of the mechanisms that have been created to allow for the free or low-cost retrieval of the full-text systematic reviews they have identified through their database searches (see Table 3 for a list of these mechanisms).
Three key options are available:
The Health Inter Network Access to Research Initiative (HINARI) which provides institutions in low-income countries with free access to many published reviews and studies
The Cochrane Library which provides free access to Cochrane reviews in low-income countries and in countries with a national subscription, and
Journals that make their content available free of charge either as soon as they are published or after a defined period of time (e.g. one year)
Three additional methods warrant mention. It may be worthwhile identifying the institution where the authors of a review are based in case they have made it available free of charge on their institution's website. It may also be possible to contact the authors directly by email. Finally, Google Scholar may be used to track down a full-text review if the review is in the public domain and the correct citation is known.
Systematic reviews are increasingly seen as a key source of information to inform policymaking, particularly in assisting with framing options and describing their impacts. They are also used to assist with a range of questions about a problem, options to address the problem, and implementation considerations. The PPD/CCNC database is a good source for finding systematic reviews that address a range of questions about health system arrangements, as well as overviews of systematic reviews and policy briefs. The Cochrane Library (particularly the Cochrane Database of Systematic Reviews and the Database of Reviews of Effects) and PubMed are both good sources of systematic reviews that address questions about the impacts of programmes, services and drugs. When systematic reviews cannot be found and timelines and resources permit, policymakers could commission a systematic review or conduct their own rapid evidence assessment.
Useful documents and further reading
McKibbon A, Wyer P, Jaeschke R, Hunt D. Finding the evidence. In Guyatt G, Rennie D, Meade MO, Cook DJ (Editors). Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. Second Edition. New York: McGraw Hill Medical, 2008; pp. 29-58.
Links to websites
Program in Policy Decision-making/Canadian Cochrane Network and Centre (PPD/CCNC) database: http://www.researchtopolicy.ca/search/reviews.aspx - Source of systematic reviews of studies about health system arrangements (benefits, harms, key features, and the views and experiences of stakeholders).
Cochrane Library's Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effects (DARE): http://www.thecochranelibrary.com and http://www.york.ac.uk/inst/crd/signup_form.htm (to sign up for electronic updates from DARE) - Source of systematic reviews of programmes, services and drugs (including benefits and possibly harms), as well as health technology assessments, which sometimes contain systematic reviews.
PubMed: http://www.ncbi.nlm.nih.gov/pubmed and http://www.ncbi.nlm.nih.gov/corehtml/query/static/clinical.shtml#reviews (to use the 'hedge' for reviews) - Source of systematic reviews addressing a range of questions, as well as single studies.
Health Information Research Unit: http://hiru.mcmaster.ca/hiru/hiru_hedges_home.aspx - Source of 'hedges' (i.e. validated search strategies) to find systematic reviews and a variety of types of studies.
Rapid Evidence Assessment Toolkit: http://www.gsr.gov.uk/professional_guidance/rea_toolkit/index.asp - Web-based toolkit to assist policymakers and those who support them to find and use research evidence as comprehensively as possible within tight time constraints, which includes a summary of the differences between a rapid evidence assessment and a systematic review and when a rapid evidence assessment might be used.
Lavis JN, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). Introduction. Health Res Policy Syst. 2009, 7 (Suppl 1): I1-10.1186/1478-4505-7-S1-I1.
Lewin S, Oxman AD, Lavis JN, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 8. Deciding how much confidence to place in a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S8-10.1186/1478-4505-7-S1-S8.
Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 9. Assessing the applicability of the findings of a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S9-10.1186/1478-4505-7-S1-S9.
Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 10. Taking equity into consideration when assessing the findings of a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S10-10.1186/1478-4505-7-S1-S10.
Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking. Lancet. 2004, 364: 1615-21. 10.1016/S0140-6736(04)17317-0.
Lavis JN, Davies HTO, Oxman A, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform healthcare management and policymaking. J Health Serv Res Policy. 2005, 10: 35-48. 10.1258/1355819054308549.
Egger M, Smith GD, O'Rourke K: Rationale, potentials, and promise of systematic reviews. Systematic Reviews in Health Care: Meta-Analysis in Context. Edited by: Egger M, Smith GD, Altman DG. 2001, London: BMJ Books, 3-19.
Lavis JN, Wilson MG, Oxman AD, Grimshaw J, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 5. Using research evidence to frame options to address a problem. Health Res Policy Syst. 2009, 7 (Suppl 1): S5-10.1186/1478-4505-7-S1-S5.
Lavis JN, Wilson MG, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 4. Using research evidence to clarify a problem. Health Res Policy Syst. 2009, 7 (Suppl 1): S4-10.1186/1478-4505-7-S1-S4.
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005, 10: 45-53. 10.1258/1355819052801804.
Mays N, Pope C, Popay J: Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005, 10 (Suppl 1): 6-20. 10.1258/1355819054308576.
Petticrew M: Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ. 2001, 322: 98-101. 10.1136/bmj.322.7278.98.
Innvaer S, Vist G, Trommald M, Oxman A: Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002, 7: 239-44. 10.1258/135581902320432778.
Lavis JN, Davies HT, Gruen RL, Walshe K, Farquhar CM: Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthc Policy. 2006, 1: 21-33.
Chopra M, Munro S, Lavis JN, Vist G, Bennett S: Effects of policy options for human resources for health: an analysis of systematic reviews. Lancet. 2008, 371: 668-74. 10.1016/S0140-6736(08)60305-0.
Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 13. Preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 2009, 7 (Suppl 1): S13-10.1186/1478-4505-7-S1-S13.
Lewin S, Oxman AD, Lavis JN, Fretheim A, García Martí S, Munabi-Babigumira S: SUPPORT Tools for evidence-informed health Policymaking (STP). 11. Finding and using research evidence about local conditions. Health Res Policy Syst. 2009, 7 (Suppl 1): S11-10.1186/1478-4505-7-S1-S11.
Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007, 4: e78-10.1371/journal.pmed.0040078.
Montori VM, Wilczynski NL, Morgan D, Haynes RB: Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005, 330: 68-10.1136/bmj.38336.804167.47.
Aaserud M, Austvoll-Dahlgren A, Sturm H, Kosters JP, Hill S, Furberg CD: Pharmaceutical policies: Effects of rational drug use, an overview of 13 reviews (Protocol). Cochrane Database of Systematic Reviews. 2006
Lagarde M, Palmer N: Evidence From Systematic Reviews to Inform Decision Making Regarding Financing Mechanisms that Improve Access to Health Services for Poor People: A Policy Brief Prepared for the International Dialogue on Evidence-Informed Action to Achieve Health Goals in Developing Countries (IDEAHealth). 2006, Geneva, World Health Organization, [http://www.who.int/rpc/meetings/HealthFinancingBrief.pdf]
Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, Grilli R, Harvey E, Oxman AD, O'Brien MA: Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001, 39: II2-45. 10.1097/00005650-200108002-00002.
Please see the Introduction to this series for acknowledgements of funders and contributors. In addition, we would like to acknowledge Andrew Booth and Julie Glanville for helpful comments on an earlier version of this Article.
This article has been published as part of Health Research Policy and Systems Volume 7 Supplement 1, 2009: SUPPORT Tools for evidence-informed health Policymaking (STP). The full contents of the supplement are available online at http://www.health-policy-systems.com/content/7/S1.
The authors declare that they have no competing interests.
JNL prepared the first draft of this article. ADO, JG, MJ, JAB, SL and AF contributed to drafting and revising it.
Electronic supplementary material
About this article
Cite this article
Lavis, J.N., Oxman, A.D., Grimshaw, J. et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 7: Finding systematic reviews. Health Res Policy Sys 7, S7 (2009). https://doi.org/10.1186/1478-4505-7-S1-S7
- Systematic Review
- Research Evidence
- Qualitative Comparative Analysis
- Local Evidence
- Implementation Consideration