The global stock of research evidence relevant to health systems policymaking

Abstract

Background

Policymakers and stakeholders need immediate access to many types of research evidence to make informed decisions about the full range of questions that may arise regarding health systems.

Methods

We examined all types of research evidence about governance, financial and delivery arrangements, and implementation strategies within health systems contained in Health Systems Evidence (HSE) (http://www.healthsystemsevidence.org). The research evidence types include evidence briefs for policy, overviews of systematic reviews, systematic reviews of effects, systematic reviews addressing other questions, systematic reviews in progress, systematic reviews being planned, economic evaluations, and health reform and health system descriptions. Specifically, we describe their distribution across health system topics and domains, trends in their production over time, availability of supplemental content in various languages, and the extent to which they focus on low- and middle-income countries (LMICs), as well as (for systematic reviews) their methodological quality and the availability of user-friendly summaries.

Results

As of July 2013, HSE contained 2,629 systematic reviews of effects (of which 501 are Cochrane reviews), 614 systematic reviews addressing other questions, 283 systematic reviews in progress, 186 systematic reviews being planned, 140 review-derived products (evidence briefs and overviews of systematic reviews), 1,669 economic evaluations, 1,092 health reform descriptions, and 209 health system descriptions. Most systematic reviews address topics related to delivery arrangements (n = 2,663) or implementation strategies (n = 1,653) with far fewer addressing financial (n = 241) or governance arrangements (n = 231). In addition, 2,928 systematic reviews have been quality appraised with moderate AMSTAR ratings found for reviews addressing governance (5.6/11), financial (5.9/11), and delivery (6.3/11) arrangements and implementation strategies (6.5/11); 1,075 systematic reviews have no independently produced user-friendly summary and only 737 systematic reviews have an LMIC focus. Literature searches for half of the systematic reviews (n = 1,584, 49%) were conducted within the last five years.

Conclusions

Greater effort needs to focus on assessing whether the current distribution of systematic reviews corresponds to policymakers’ and stakeholders’ priorities, updating systematic reviews, increasing the quality of systematic reviews, and focusing on LMICs.

“Congress has provided vital funding for research that compares the effectiveness of different treatments, and this should help reduce uncertainty about which treatments are best. But we also need to fund research that compares the effectiveness of different systems of care – to reduce our uncertainty about which systems work best for communities. These are empirical, not ideological questions”. Atul Gawande, The New Yorker, 1 June 2009, p. 44

Background

Policymakers and stakeholders need immediate access to many types of research evidence to make informed decisions about the full range of questions they may have regarding health system arrangements and implementation strategies (with the latter including those aimed at supporting the use of research evidence at the level of citizens, providers, organizations, and policymakers). While policymakers need to consider many factors for any given decision about health systems (e.g., institutional constraints, interests of stakeholders affected by decisions, and the values and preferences of the public), research evidence can also help support and inform their efforts to strengthen or reform health systems or get cost-effective programs, services, and drugs to those who need them. Atul Gawande got it mostly right: “these are empirical, not just ideological questions”.

The timeliness of research evidence was one of two factors that emerged with some consistency in a systematic review of the factors that increased the prospects for research use in policymaking [1]. When the research literature has already been identified, selected, appraised and synthesized in a systematic and transparent way, health system policymakers can move directly to assessing how much confidence they can place in the review (i.e., its quality), the local applicability of the review’s findings, and what the findings mean for their setting [2]. Stakeholders, such as professional associations and citizen groups, also need timely access to many types of research evidence to inform their advocacy efforts focused on health systems. Researchers and research funding agencies need systematic reviews to identify gaps in knowledge about health systems (both primary studies and systematic reviews) and domains that could benefit from overviews of systematic reviews, as well as to put the findings of any new health systems research in the context of existing research [1].

Questions about the comparative effectiveness of one health system arrangement over another (such as using nurses rather than doctors to deliver certain forms of care) are one of the types of questions for which policymakers and stakeholders may turn to systematic reviews [3]. The likelihood of them being misled by research evidence about comparative effectiveness is lower and confidence in what effects they can expect from a health system arrangement is higher with a systematic review than with an individual study [4]. However, what the quotation from Atul Gawande fails to point out is that policymakers and stakeholders can also turn to systematic reviews to address questions best answered using qualitative and mixed-methods studies, such as questions regarding patients’ views about and experiences with problems encountered in health systems and with options for addressing these problems [5]. Furthermore, given often constrained resources, policymakers also need to consider value for money in any decision they make, which requires access to locally applicable economic evaluations about the various policy levers at their disposal. To further support their decisions, policymakers may also wish to turn to descriptions of health reforms undertaken in other jurisdictions to better understand what was done, how, and why, as well as to descriptions of other health systems to help determine the local applicability of research evidence generated in these systems to their own.

Policymakers and stakeholders are relatively well served by existing databases such as Medline (particularly when validated search strategies are employed) [6, 7] and the Cochrane Library (which contains both the Cochrane Database of Systematic Reviews and the Database of Reviews of Effects) if their questions about comparative effectiveness pertain to clinical programs and services or to drugs [3]. They are also well served by Health Evidence™ (http://www.healthevidence.org) if their questions about comparative effectiveness pertain to public health programs and services [8]. More recently, if their questions pertain to the governance, financial and delivery arrangements within which programs, services and drugs are provided, and to implementation strategies for these programs, services and drugs, they are well served by Health Systems Evidence (HSE) (http://www.healthsystemsevidence.org). HSE provides a comprehensive inventory of nine types of research evidence, namely evidence briefs for policy (i.e., a document that summarizes how the findings from a number of systematic reviews pertain to a pressing problem, select options for addressing the problem, and key implementation considerations), overviews of systematic reviews, systematic reviews of effects, systematic reviews addressing other types of questions, systematic reviews in progress, systematic reviews being planned, economic evaluations, health reform descriptions, and health system descriptions [9, 10].

All of these existing databases are further strengthened by significant efforts from a large and growing number of groups to package, quality appraise, and facilitate assessments of local applicability of systematic reviews [5]. In addition, recent efforts have drawn on HSE to document the types of study designs included in reviews addressing health-system interventions [11]. To our knowledge, there have been no efforts to develop a comprehensive profile of the available research evidence addressing topics related to governance, financial and delivery arrangements within health systems, and implementation strategies that can support change in health systems, and the products derived from them.

Methods

We drew on the global stock of the nine types of research evidence related to health systems contained in HSE as of July 2013. HSE draws its content from all of the major sources of each of the nine types of research evidence (e.g., Cochrane Library for systematic reviews of effects and protocols for such reviews, Economic Evaluation Database for economic evaluations, and Health Policy Monitor for health reform descriptions) [12],Unpublished data]. Each record contained in HSE is categorized according to a taxonomy of health system topics (e.g., remunerating providers) and domains (e.g., primary care), coded according to key features (e.g., systematic reviews are coded according to their methodological quality and the countries in which included studies were conducted) [13], and made available in seven languages. All of the eligibility assessments and coding are done by two independent raters. The methods underpinning the development and continuous updating of HSE are described in a separate manuscript [Unpublished data].

The data related to, and coding categories associated with, each of the records contained in HSE were used to calculate descriptive statistics that profile: 1) the distribution of research evidence, particularly systematic reviews, across health system topics and domains; 2) trends over time in how recently the literature was searched for systematic reviews and in the volume of publication for all types of research evidence; 3) the distribution of systematic reviews according to their methodological quality (assessed using AMSTAR (A MeaSurement Tool to Assess the quality of systematic Reviews) [13], as well as trends over time in the distribution of average quality scores; 4) the availability of user-friendly summaries (for systematic reviews) and links to additional content (for all types of research evidence) by source, type of systematic review, and language; and 5) the distribution of research evidence according to the type of low- and middle-income country (LMIC) focus. When summarizing our findings we often focus on systematic reviews, either because the data only apply to this type of research evidence (e.g., AMSTAR ratings) or because this type of research evidence is likely to be of greatest immediate relevance to health system policymakers and stakeholders.

Results and Discussion

HSE contains (as of July 2013) 6,613 documents addressing topics related to health systems, which include:

• 140 review-derived products (94 evidence briefs and 46 overviews of systematic reviews);

• 2629 systematic reviews of effects

  • ∘501 Cochrane reviews of which one in five have been produced by the Effective Practice and Organization of Care (EPOC) review group (111/501, 22%);

• 614 systematic reviews addressing other types of questions (e.g., reviews of observational studies that often assess the scale of problems or associations between variables and reviews of qualitative studies that often assess the nature of problems and/or how and why interventions work);

• 283 systematic reviews in-progress (i.e., systematic review protocols);

• 186 systematic reviews being planned (i.e., systematic review titles that have been registered and for which a protocol is being prepared);

• 1669 economic evaluations;

• 1092 health reform descriptions; and

• 209 descriptions of health systems.

Topics and domains addressed

Most systematic reviews address topics related to delivery arrangements (n = 2,663) or implementation strategies (n = 1,653) in whole or in part, whereas much smaller numbers of systematic reviews address financial arrangements (n = 241) or governance arrangements (n = 231) (Table 1). Within the delivery arrangements category, ‘by whom care is provided’ (i.e., human resources) represents the largest sub-category (n = 1,426, of which 342 addressed multi-disciplinary teams). This is followed by “how care is designed to meet consumers’ needs” (n = 972, of which 562 addressed packages of care/care pathways/disease management) and “where care is provided” (n = 720, of which 427 addressed the site of service delivery). Within the implementation strategies category, consumer-targeted strategies represent the largest sub-category (n = 1,208), followed by provider-targeted strategies (n = 622) and organization-targeted strategies (n = 65). Within the governance arrangements category, organizational authority (i.e., what decisions can organizations like hospitals make and how) represents the largest sub-category (n = 89) and “consumer and stakeholder involvement” the second largest (n = 71). Within the financial arrangements category, remunerating providers (i.e., how providers are paid) represents the largest sub-category (n = 106), followed by “incentivizing consumers” (n = 85) and “financing systems” (i.e., how revenue is raised for health systems and services) (n = 56).

Table 1 Number (and %) of documents addressing particular health system topics, by type of document*

Many systematic reviews address particular diseases (n = 1,819), sectors (n = 1,634), providers (n = 1,582), and technologies (n = 667) (Table 2). Within the groupings of diseases indexed by HSE, the top five categories of diseases addressed by reviews are mental health and addictions (n = 962), cardiovascular disease (n = 343), diabetes (n = 276), maternal and child health (n = 284), and cancer (n = 222). While we found that approximately half (n = 1,582, 49%) of the systematic reviews in HSE address topics related to one or more types of providers, most within this grouping are focused on physicians (n = 945) and nurses (n = 692) and, to a lesser extent, allied health professionals (n = 391) as compared to pharmacists (n = 154) and lay/community health workers (n = 137). For the grouping of health system sectors, a relatively large number of reviews address primary care (n = 569), hospital care (n = 555), home care (n = 626), public health (n = 393), and rehabilitation (n = 195), but few address long-term care (n = 64). The majority of reviews addressing health system arrangements and implementation strategies related to the technologies category address drugs (n = 404) and surgery (n = 151) with far fewer addressing diagnostics (n = 90) and devices (n = 65).

Table 2 Number (and %) of documents addressing particular domains, by type of document*

Trends in production over time

We found that 2,893 of the 3,243 systematic reviews (89%) reported the last year the literature was searched and, based on the available data, 1,584 (49%) were conducted within the last five years (i.e., since 2008) (Figure 1). Of the 350 reviews for which we do not have the last year the literature was searched, 199 (57%) were published within the last five years. Our sub-analysis of Cochrane reviews revealed that most of the 501 reviews (n = 493, 98%) reported the last year the literature was searched and 332 (66%) of these were conducted within the last five years, while only 153 (30%) were up to date within the last two years, which is the Cochrane Collaboration’s recommended timeframe for updating reviews [14]. For overviews of systematic reviews, 37 of 46 (80%) reported the last year the literature was searched, and 26 of these (56%) were conducted within the last five years (Figure 1).

Figure 1

Last year literature was searched for systematic reviews and overviews of systematic reviews.

Regarding the year of publication, we found that 90 of the 94 (96%) evidence briefs for policy, 273 of the 283 (96%) systematic reviews in progress (i.e., protocols that had not yet been turned into a full review), all of the 186 systematic reviews being planned, 1,275 of the 1,669 (76%) economic evaluations (note that only those conducted in the last 10 years are indexed in HSE), 423 of the 1,092 (39%) descriptions of health reforms, and 101 of the 209 (48%) descriptions of health systems were published within the last five years (Figure 2).

Figure 2

Publication dates for documents.

Methodological quality of systematic reviews

Most systematic reviews (n = 2,928, 90%) indexed in HSE have been quality appraised using AMSTAR and their methodological quality is (on average) moderate (i.e., between 4 and 7 out of 11), which has remained relatively consistent over time (Table 3 and Figure 3). The average methodological quality of reviews addressing delivery arrangements (6.3/11) and implementation strategies (6.3/11) was somewhat higher than those addressing financial (5.9/11) and governance arrangements (5.6/11). However, methodological quality differs substantially according to the type of systematic review. Those produced by the Cochrane Collaboration are of high quality (i.e., between 9 and 11 out of 11) across all second-level topic domains (with none falling outside this range). In contrast, systematic reviews of effects (which include Cochrane reviews) are of moderate quality (with no second-level category averages falling outside the range of 4.8–7.4) and quantitative reviews addressing other types of questions were low to moderate quality (with no averages falling outside the range of 3.6–5.4).

Table 3 Mean quality rating* of systematic reviews addressing particular health system topics, by focus and type of systematic review

Figure 3

Mean quality ( AMSTAR ) score.

Availability of user-friendly summaries and links to additional content

The majority (n = 2,168, 67%) of systematic reviews now have at least one user-friendly (English-language) summary available from one of the eight groups in the world that package, quality appraise and/or facilitate assessments of the local applicability of systematic reviews (Table 4). The largest producers of these structured decision-relevant summaries for reviews about health system arrangements are the DARE (n = 1,962, 90% of those with at least one summary) and the Cochrane Library through its plain language summaries of reviews (n = 505, 23%). However, these review summaries are not targeted specifically at health system policymakers and stakeholders. The three groups representing the next largest shares of summaries of reviews about health system arrangements – Rx for Change (n = 211, 10%), Health Evidence™ (n = 231, 11%), and Australasian Cochrane Centre (n = 85, 4%) – are more targeted at these groups. A number of reviews also have one or more summaries available in Spanish (n = 474) and French (n = 391), but fewer summaries are available in the four remaining languages (14 for each of Arabic and Russian, 36 for Chinese and 1 for Portuguese).

Table 4 Number (and %) of user-friendly summaries available for systematic reviews, by type of systematic review*

Across all documents contained in HSE (n = 6,613), more than half (n = 3,893, 59%) have no links to one or more independently produced structured decision-relevant summaries (Table 5). Most documents have links to a scientific abstract from PubMed (n = 4,549), the Cochrane Library (n = 1,994, of which 299 have been translated into Spanish and 28 have been translated into French) or from another publisher (n = 4,281, of which 55 have been translated into French). Two thirds of the documents have a link to a full-text report that can be accessed without a subscription (n = 4,638, 70%); however, countries that have a national license to the Cochrane Library can access full Cochrane reviews (n = 501) and protocols of Cochrane reviews (n = 283) for free as well.

Table 5 Number of links available for all types of document, by language*

Low- and middle-income country (LMIC) focus

We found 1,228 (19%) documents in HSE that have an LMIC focus (Table 6), half of which are systematic reviews of effects (n = 614, 50%) and a lower percentage of which are economic evaluations (n = 280, 23%), systematic reviews addressing other types of questions (n = 123, 10%), and health system descriptions (n = 92, 7%). The 737 systematic reviews with an LMIC focus (which are the subset of documents described in Table 6 that we focus on here) were most often the result of including at least one study conducted in an LMIC setting (n = 712, 97%) with far fewer identified as having an LMIC setting as the target of the document (n = 193, 26%) or having at least one author from an LMIC (n = 118, 16%).

Table 6 Number (and %) of documents addressing health system topics, by type of low- and middle-income country (LMIC) focus

Similar to the full set of reviews contained in HSE, the systematic reviews with an LMIC focus most often addressed topics related to delivery arrangements (n = 590) and implementation strategies (n = 394) in whole or in part, with far fewer addressing financial (n = 74) and governance arrangements (n = 55). The most common sub-categories under delivery arrangements containing systematic reviews with an LMIC focus were “by whom is care provided” (n = 319), “how care is designed to meet consumers’ needs” (n = 236), “where care is provided” (n = 171), and “with what supports is care provided” (n = 161). The majority of systematic reviews within the implementation strategies category address consumer-targeted strategies (n = 310), while fewer addressing provider-targeted strategies (n = 132). Within the governance arrangements category, policy authority (n = 20) represents the largest sub-category and within the financial arrangements category, “incentivizing consumers” (n = 29), “remunerating providers” (n = 27) and “financing systems” (n = 26) represent the largest sub-categories.

The systematic reviews identified as having an LMIC focus also have a similar profile to the global stock of reviews in terms of how recently they were conducted and their quality. As outlined in Figure 4, slightly more than half of the reviews (n = 403, 54%) were conducted within the last five years (since 2008) and of the 12 overviews of systematic reviews with an LMIC focus 6 (50%) were conducted within the last five years. In addition, we found that all of the systematic reviews in progress (n = 25), systematic reviews being planned (n = 17), and health reform descriptions (n = 3), as well as 237 of 280 (85%) economic evaluations, 58 of 60 (97%) evidence briefs for policy and 42 of 92 (46%) health system descriptions, with an LMIC focus, were published within the last five years. In terms of quality, almost all of the reviews with an LMIC focus (n = 696, 94%) have been appraised for methodological quality using AMSTAR and (on average) have a moderate quality score (range = 4.2–7.7 out of a possible score of 11) across topic domains (Table 3).

Figure 4

Last year literature was searched for systematic reviews and overviews of systematic reviews with an LMIC focus.

Conclusions

We have provided a detailed examination of the global stock of research evidence relevant to health systems policymaking. Systematic reviews of effects are the most common of the nine types of research evidence in HSE (n = 2,629, 40%). Most systematic reviews address topics related to delivery arrangements (n = 2,663) or implementation strategies (n = 1,653) in whole or in part, which may reflect a tendency among more clinically oriented researchers to first examine the effects of an intervention and then (faced with feasibility and other concerns raised by policymakers and stakeholders) to examine the effects of alternative ways of delivering the intervention or implementing it on a large scale. Half (49%) of the systematic reviews presented findings based on searches conducted within the last five years (since 2008), quality ratings ranged from 5.6 (for governance arrangements) to 6.3 (for delivery arrangements and implementation strategies) out of a possible 11, nearly one in four reviews (n = 737, 23%) contain at least one study from a LMIC, and 1,075 reviews have no independently produced structured decision-relevant summary.

This study has one key strength and one potential limitation. Our analysis represents the first effort (at least to our knowledge) to systematically assess the global stock of nine types of research evidence relevant to health systems policymaking. Moreover, we have developed an approach to real-time reporting for all of the tables and figures included in our analysis, which will allow us to update the tables and figures at any time and identify changes over time. The potential limitation of our study is that, by relying on one data source for our analysis, we have missed including relevant systematic reviews. However, we believe this possibility to be remote given that HSE systematically culls relevant documents from a number of databases (Medline, Cochrane Database of Systematic Reviews, three databases – DARE, PROSPERO, Economic Evaluations Database – from the Centre for Reviews and Dissemination, Rx for Change, and the Cochrane Qualitative and Implementation Methods Group’s reference database for qualitative reviews) and many other sources such as listservs (e.g., EvidenceUpdates and PAHO EQUIDAD) and websites (e.g., AHRQ, Campbell Collaboration, EPPI-Centre and several more). As a result, searching other sources would be unlikely to identify many documents. For those interested in reviewing the full list of sources and methods used to build and continuously update HSE we provide these details in a separate manuscript [Unpublished data].

Based on the profile of research evidence provided here, particularly the profile of systematic reviews, we believe that greater effort should be placed on several priorities to enhance the usefulness of research evidence about health systems and support its use by policymakers and stakeholders. First, there is a need to support efforts to regularly update systematic reviews, as doing so will ensure that policymakers and stakeholders have access to the most recent synthesized research evidence. As outlined in our findings, half of the systematic reviews in HSE were conducted five or more years ago and a large proportion of those produced by the Cochrane Collaboration have also not been updated within their recommended timeline of two years. Such efforts could be supported (at least partially) by funders of systematic reviews providing resources both for new systematic reviews and for updating existing reviews (e.g., by identifying reviews that address timely policy questions but are out of date).

Second, there is a need to increase the average quality of systematic reviews. This is particularly evident for systematic reviews addressing questions other than effectiveness and/or for those addressing topics related to governance and financial arrangements (many of which are reviews addressing other types of questions), which generally scored lower for quality as compared to reviews addressing questions of effectiveness. Registering titles and protocols for systematic reviews and implementing specific quality standards as part of the registration process is a promising mechanism that may help increase the overall quality of reviews. This requirement from the Cochrane Collaboration contributes (at least in part) to the average level of quality for Cochrane reviews being significantly higher than the rest of the reviews in HSE. In addition, PROSPERO [15] now offers an international prospective register of systematic reviews requiring minimum methodological standards based on the PRISMA statement [16]. These efforts could be further supported by journals requiring systematic reviews to be registered in order to be considered for publication.

It should be noted that the AMSTAR tool was designed originally for reviews of effects. However, the fundamental methodological requirements for reviews that include quantitative evidence remain the same and where they differ there is an option to mark a question as “not applicable” to the score. As noted in our results, we did not conduct quality appraisals of reviews that draw exclusively on qualitative research evidence given that there is currently no quality appraisal tool available (at least to our knowledge) that takes into account the unique methods and approaches to data analysis used in these types of reviews (although two of us – MGW and JNL – are currently finalizing a tool that will be able to be used for this purpose).

Third, there is a need to use the global stock of systematic reviews to start developing global guidance to support evidence-informed policies about health systems [17–19]. However, the utility of such guidelines is dependent upon the availability of systematic reviews addressing the full range topics related to governance, financial and delivery arrangements within health systems, and about implementation strategies that can support change in health systems. This will require diversifying the types of topics addressed in systematic reviews (particularly for governance and financial arrangements or for increasingly important topic-specific domains such as long-term care, which are relatively underserved in the current global stock) as well as ensuring relevance to LMICs (for which there are comparatively few reviews). Addressing this could also include conducting assessments of whether the current distribution of systematic reviews corresponds to policymakers’ and stakeholders’ priorities.