Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

The global stock of research evidence relevant to health systems policymaking

Health Research Policy and Systems201311:32

https://doi.org/10.1186/1478-4505-11-32

Received: 15 February 2013

Accepted: 10 August 2013

Published: 4 September 2013

Abstract

Background

Policymakers and stakeholders need immediate access to many types of research evidence to make informed decisions about the full range of questions that may arise regarding health systems.

Methods

We examined all types of research evidence about governance, financial and delivery arrangements, and implementation strategies within health systems contained in Health Systems Evidence (HSE) (http://www.healthsystemsevidence.org). The research evidence types include evidence briefs for policy, overviews of systematic reviews, systematic reviews of effects, systematic reviews addressing other questions, systematic reviews in progress, systematic reviews being planned, economic evaluations, and health reform and health system descriptions. Specifically, we describe their distribution across health system topics and domains, trends in their production over time, availability of supplemental content in various languages, and the extent to which they focus on low- and middle-income countries (LMICs), as well as (for systematic reviews) their methodological quality and the availability of user-friendly summaries.

Results

As of July 2013, HSE contained 2,629 systematic reviews of effects (of which 501 are Cochrane reviews), 614 systematic reviews addressing other questions, 283 systematic reviews in progress, 186 systematic reviews being planned, 140 review-derived products (evidence briefs and overviews of systematic reviews), 1,669 economic evaluations, 1,092 health reform descriptions, and 209 health system descriptions. Most systematic reviews address topics related to delivery arrangements (n = 2,663) or implementation strategies (n = 1,653) with far fewer addressing financial (n = 241) or governance arrangements (n = 231). In addition, 2,928 systematic reviews have been quality appraised with moderate AMSTAR ratings found for reviews addressing governance (5.6/11), financial (5.9/11), and delivery (6.3/11) arrangements and implementation strategies (6.5/11); 1,075 systematic reviews have no independently produced user-friendly summary and only 737 systematic reviews have an LMIC focus. Literature searches for half of the systematic reviews (n = 1,584, 49%) were conducted within the last five years.

Conclusions

Greater effort needs to focus on assessing whether the current distribution of systematic reviews corresponds to policymakers’ and stakeholders’ priorities, updating systematic reviews, increasing the quality of systematic reviews, and focusing on LMICs.

“Congress has provided vital funding for research that compares the effectiveness of different treatments, and this should help reduce uncertainty about which treatments are best. But we also need to fund research that compares the effectiveness of different systems of care – to reduce our uncertainty about which systems work best for communities. These are empirical, not ideological questions”. Atul Gawande, The New Yorker, 1 June 2009, p. 44

Background

Policymakers and stakeholders need immediate access to many types of research evidence to make informed decisions about the full range of questions they may have regarding health system arrangements and implementation strategies (with the latter including those aimed at supporting the use of research evidence at the level of citizens, providers, organizations, and policymakers). While policymakers need to consider many factors for any given decision about health systems (e.g., institutional constraints, interests of stakeholders affected by decisions, and the values and preferences of the public), research evidence can also help support and inform their efforts to strengthen or reform health systems or get cost-effective programs, services, and drugs to those who need them. Atul Gawande got it mostly right: “these are empirical, not just ideological questions”.

The timeliness of research evidence was one of two factors that emerged with some consistency in a systematic review of the factors that increased the prospects for research use in policymaking [1]. When the research literature has already been identified, selected, appraised and synthesized in a systematic and transparent way, health system policymakers can move directly to assessing how much confidence they can place in the review (i.e., its quality), the local applicability of the review’s findings, and what the findings mean for their setting [2]. Stakeholders, such as professional associations and citizen groups, also need timely access to many types of research evidence to inform their advocacy efforts focused on health systems. Researchers and research funding agencies need systematic reviews to identify gaps in knowledge about health systems (both primary studies and systematic reviews) and domains that could benefit from overviews of systematic reviews, as well as to put the findings of any new health systems research in the context of existing research [1].

Questions about the comparative effectiveness of one health system arrangement over another (such as using nurses rather than doctors to deliver certain forms of care) are one of the types of questions for which policymakers and stakeholders may turn to systematic reviews [3]. The likelihood of them being misled by research evidence about comparative effectiveness is lower and confidence in what effects they can expect from a health system arrangement is higher with a systematic review than with an individual study [4]. However, what the quotation from Atul Gawande fails to point out is that policymakers and stakeholders can also turn to systematic reviews to address questions best answered using qualitative and mixed-methods studies, such as questions regarding patients’ views about and experiences with problems encountered in health systems and with options for addressing these problems [5]. Furthermore, given often constrained resources, policymakers also need to consider value for money in any decision they make, which requires access to locally applicable economic evaluations about the various policy levers at their disposal. To further support their decisions, policymakers may also wish to turn to descriptions of health reforms undertaken in other jurisdictions to better understand what was done, how, and why, as well as to descriptions of other health systems to help determine the local applicability of research evidence generated in these systems to their own.

Policymakers and stakeholders are relatively well served by existing databases such as Medline (particularly when validated search strategies are employed) [6, 7] and the Cochrane Library (which contains both the Cochrane Database of Systematic Reviews and the Database of Reviews of Effects) if their questions about comparative effectiveness pertain to clinical programs and services or to drugs [3]. They are also well served by Health Evidence™ (http://www.healthevidence.org) if their questions about comparative effectiveness pertain to public health programs and services [8]. More recently, if their questions pertain to the governance, financial and delivery arrangements within which programs, services and drugs are provided, and to implementation strategies for these programs, services and drugs, they are well served by Health Systems Evidence (HSE) (http://www.healthsystemsevidence.org). HSE provides a comprehensive inventory of nine types of research evidence, namely evidence briefs for policy (i.e., a document that summarizes how the findings from a number of systematic reviews pertain to a pressing problem, select options for addressing the problem, and key implementation considerations), overviews of systematic reviews, systematic reviews of effects, systematic reviews addressing other types of questions, systematic reviews in progress, systematic reviews being planned, economic evaluations, health reform descriptions, and health system descriptions [9, 10].

All of these existing databases are further strengthened by significant efforts from a large and growing number of groups to package, quality appraise, and facilitate assessments of local applicability of systematic reviews [5]. In addition, recent efforts have drawn on HSE to document the types of study designs included in reviews addressing health-system interventions [11]. To our knowledge, there have been no efforts to develop a comprehensive profile of the available research evidence addressing topics related to governance, financial and delivery arrangements within health systems, and implementation strategies that can support change in health systems, and the products derived from them.

Methods

We drew on the global stock of the nine types of research evidence related to health systems contained in HSE as of July 2013. HSE draws its content from all of the major sources of each of the nine types of research evidence (e.g., Cochrane Library for systematic reviews of effects and protocols for such reviews, Economic Evaluation Database for economic evaluations, and Health Policy Monitor for health reform descriptions) [12],Unpublished data]. Each record contained in HSE is categorized according to a taxonomy of health system topics (e.g., remunerating providers) and domains (e.g., primary care), coded according to key features (e.g., systematic reviews are coded according to their methodological quality and the countries in which included studies were conducted) [13], and made available in seven languages. All of the eligibility assessments and coding are done by two independent raters. The methods underpinning the development and continuous updating of HSE are described in a separate manuscript [Unpublished data].

The data related to, and coding categories associated with, each of the records contained in HSE were used to calculate descriptive statistics that profile: 1) the distribution of research evidence, particularly systematic reviews, across health system topics and domains; 2) trends over time in how recently the literature was searched for systematic reviews and in the volume of publication for all types of research evidence; 3) the distribution of systematic reviews according to their methodological quality (assessed using AMSTAR (A MeaSurement Tool to Assess the quality of systematic Reviews) [13], as well as trends over time in the distribution of average quality scores; 4) the availability of user-friendly summaries (for systematic reviews) and links to additional content (for all types of research evidence) by source, type of systematic review, and language; and 5) the distribution of research evidence according to the type of low- and middle-income country (LMIC) focus. When summarizing our findings we often focus on systematic reviews, either because the data only apply to this type of research evidence (e.g., AMSTAR ratings) or because this type of research evidence is likely to be of greatest immediate relevance to health system policymakers and stakeholders.

Results and Discussion

HSE contains (as of July 2013) 6,613 documents addressing topics related to health systems, which include:

  • 140 review-derived products (94 evidence briefs and 46 overviews of systematic reviews);

  • 2629 systematic reviews of effects

    • 501 Cochrane reviews of which one in five have been produced by the Effective Practice and Organization of Care (EPOC) review group (111/501, 22%);

  • 614 systematic reviews addressing other types of questions (e.g., reviews of observational studies that often assess the scale of problems or associations between variables and reviews of qualitative studies that often assess the nature of problems and/or how and why interventions work);

  • 283 systematic reviews in-progress (i.e., systematic review protocols);

  • 186 systematic reviews being planned (i.e., systematic review titles that have been registered and for which a protocol is being prepared);

  • 1669 economic evaluations;

  • 1092 health reform descriptions; and

  • 209 descriptions of health systems.

Topics and domains addressed

Most systematic reviews address topics related to delivery arrangements (n = 2,663) or implementation strategies (n = 1,653) in whole or in part, whereas much smaller numbers of systematic reviews address financial arrangements (n = 241) or governance arrangements (n = 231) (Table 1). Within the delivery arrangements category, ‘by whom care is provided’ (i.e., human resources) represents the largest sub-category (n = 1,426, of which 342 addressed multi-disciplinary teams). This is followed by “how care is designed to meet consumers’ needs” (n = 972, of which 562 addressed packages of care/care pathways/disease management) and “where care is provided” (n = 720, of which 427 addressed the site of service delivery). Within the implementation strategies category, consumer-targeted strategies represent the largest sub-category (n = 1,208), followed by provider-targeted strategies (n = 622) and organization-targeted strategies (n = 65). Within the governance arrangements category, organizational authority (i.e., what decisions can organizations like hospitals make and how) represents the largest sub-category (n = 89) and “consumer and stakeholder involvement” the second largest (n = 71). Within the financial arrangements category, remunerating providers (i.e., how providers are paid) represents the largest sub-category (n = 106), followed by “incentivizing consumers” (n = 85) and “financing systems” (i.e., how revenue is raised for health systems and services) (n = 56).
Table 1

Number (and %) of documents addressing particular health system topics, by type of document*

Health system topics

All documents (n = 6,613)

Evidence briefs for policy (n = 94)

Overviews of systematic reviews (n = 46)

Systematic reviews of effects** (n = 2,629)

Systematic reviews addressing other types of questions (n = 614)

Systematic reviews in progress (n = 283)

Systematic reviews being planned (n = 186)

Economic evaluations (n = 1,669)

Health reform descriptions (n = 1,092)

Governance arrangement

1360

21%

46

49%

10

22%

117

4%

114

19%

15

5%

12

6%

80

5%

966

88%

• Policy authority

890

13%

24

26%

5

11%

26

1%

31

5%

8

3%

5

3%

23

1%

768

70%

• Organizational authority

648

10%

20

21%

4

9%

37

1%

52

8%

2

1%

3

2%

28

2%

502

46%

• Commercial authority

280

4%

8

9%

3

7%

17

1%

17

3%

1

0%

1

1%

9

1%

224

21%

• Professional authority

441

7%

16

17%

3

7%

33

1%

34

6%

2

1%

1

1%

25

1%

327

30%

• Consumer & stakeholder involvement

335

5%

21

22%

4

9%

35

1%

36

6%

4

1%

2

1%

6

0%

227

21%

Financial arrangement

1240

19%

47

50%

14

30%

171

7%

70

11%

23

8%

17

9%

153

9%

745

68%

• Financing systems

604

9%

22

23%

6

13%

32

1%

24

4%

6

2%

2

1%

23

1%

489

45%

• Funding organizations

230

3%

8

9%

4

9%

22

1%

9

1%

4

1%

0

0%

5

0%

178

16%

• Remunerating providers

400

6%

23

24%

11

24%

73

3%

33

5%

7

2%

1

1%

22

1%

230

21%

• Purchasing products & services

454

7%

9

10%

3

7%

20

1%

11

2%

3

1%

2

1%

91

5%

315

29%

• Incentivizing consumers

487

7%

16

17%

7

15%

72

3%

13

2%

10

4%

12

6%

46

3%

311

28%

Delivery arrangement

5590

85%

71

76%

40

87%

2145

82%

518

84%

225

80%

128

69%

1581

95%

882

81%

• How care is designed to meet consumers’ needs

2849

43%

32

34%

16

35%

808

31%

164

27%

84

30%

45

24%

1078

65%

622

57%

• By whom care is provided

2767

42%

58

62%

23

50%

1097

42%

329

54%

107

38%

59

32%

586

35%

508

47%

• Where care is provided

1677

25%

31

33%

13

28%

579

22%

141

23%

61

22%

33

18%

417

25%

402

37%

• With what supports is care provided

1792

27%

27

29%

23

50%

756

29%

152

25%

69

24%

25

13%

339

20%

401

37%

Implementation strategy

3031

46%

50

53%

25

54%

1425

54%

228

37%

145

51%

64

34%

633

38%

461

42%

• Consumer-targeted strategy

2297

35%

35

37%

16

35%

1078

41%

130

21%

114

40%

52

28%

489

29%

383

35%

• Provider-targeted strategy

1103

17%

38

40%

18

39%

502

19%

120

20%

34

12%

15

8%

203

12%

173

16%

• Organization-targeted strategy

104

2%

2

2%

5

11%

37

1%

28

5%

12

4%

2

1%

18

1%

0

0%

* This table does not provide data related to health system descriptions as they are not categorized in Health Systems Evidence according to health system topics. As a result, the total number of documents listed here is 6,613 but the total including the 209 health system descriptions indexed in Health Systems Evidence is 6,822.

** Systematic reviews of effects include 501 Cochrane reviews of which 111 (22%) have been produced by the Effective Practice and Organization of Care (EPOC) review group.

Many systematic reviews address particular diseases (n = 1,819), sectors (n = 1,634), providers (n = 1,582), and technologies (n = 667) (Table 2). Within the groupings of diseases indexed by HSE, the top five categories of diseases addressed by reviews are mental health and addictions (n = 962), cardiovascular disease (n = 343), diabetes (n = 276), maternal and child health (n = 284), and cancer (n = 222). While we found that approximately half (n = 1,582, 49%) of the systematic reviews in HSE address topics related to one or more types of providers, most within this grouping are focused on physicians (n = 945) and nurses (n = 692) and, to a lesser extent, allied health professionals (n = 391) as compared to pharmacists (n = 154) and lay/community health workers (n = 137). For the grouping of health system sectors, a relatively large number of reviews address primary care (n = 569), hospital care (n = 555), home care (n = 626), public health (n = 393), and rehabilitation (n = 195), but few address long-term care (n = 64). The majority of reviews addressing health system arrangements and implementation strategies related to the technologies category address drugs (n = 404) and surgery (n = 151) with far fewer addressing diagnostics (n = 90) and devices (n = 65).
Table 2

Number (and %) of documents addressing particular domains, by type of document*

Domains

All documents (n = 6,613)

Evidence briefs for policy (n = 94)

Overviews of systematic reviews (n = 46)

Systematic reviews of effects** (n = 2,629)

Systematic reviews addressing other types of questions (n = 614)

Systematic reviews in progress (n = 283)

Systematic reviews being planned (n = 186)

Economic evaluations (n = 1,669)

Health reform descriptions (n = 1,092)

Diseases

3833

58%

49

52%

21

46%

1590

60%

229

37%

172

61%

83

45%

1279

77%

410

38%

Infectious diseases

808

12%

18

19%

4

9%

217

8%

35

6%

23

8%

22

12%

423

25%

66

6%

• HIV

295

4%

5

5%

2

4%

123

5%

19

3%

13

5%

16

9%

100

6%

17

2%

• Tuberculosis

99

1%

4

4%

2

4%

31

1%

9

1%

2

1%

2

1%

42

3%

7

1%

• Malaria

60

1%

8

9%

0

0%

11

0%

7

1%

7

2%

1

1%

26

2%

0

0%

• Diarrhoeal disease

52

1%

1

1%

0

0%

14

1%

0

0%

2

1%

1

1%

33

2%

1

0%

• Lower respiratory infections

71

1%

0

0%

0

0%

25

1%

2

0%

1

0%

2

1%

39

2%

2

0%

Non-communicable diseases

1884

28%

6

6%

9

20%

742

28%

111

18%

78

28%

30

16%

658

39%

250

23%

• Cancer

584

9%

3

3%

3

7%

183

7%

39

6%

18

6%

3

2%

209

13%

126

12%

• Cardiovascular disease

661

10%

2

2%

2

4%

306

12%

37

6%

26

9%

11

6%

204

12%

73

7%

• Diabetes

493

7%

2

2%

4

9%

234

9%

42

7%

14

5%

8

4%

119

7%

70

6%

• Alzheimer and other dementias

97

1%

1

1%

0

0%

46

2%

8

1%

7

2%

3

2%

12

1%

20

2%

• Chronic obstructive pulmonary disease

91

1%

1

1%

1

2%

40

2%

6

1%

7

2%

5

3%

24

1%

7

1%

Other

1833

28%

31

33%

13

28%

898

34%

129

21%

84

30%

37

20%

413

25%

228

21%

• Maternal and child health

678

10%

17

18%

5

11%

242

9%

42

7%

35

12%

12

6%

207

12%

118

11%

• Accidents

190

3%

1

1%

0

0%

84

3%

10

2%

8

3%

9

5%

52

3%

26

2%

• Mental health and addictions

1089

16%

15

16%

10

22%

637

24%

83

14%

42

15%

16

9%

155

9%

131

12%

Technologies

1805

27%

25

27%

9

20%

562

21%

105

17%

39

14%

15

8%

634

38%

416

38%

• Drugs

1047

16%

19

20%

9

20%

345

13%

59

10%

16

6%

11

6%

323

19%

265

24%

• Devices

174

3%

1

1%

0

0%

59

2%

6

1%

4

1%

1

1%

54

3%

49

4%

• Diagnostics

464

7%

2

2%

1

2%

76

3%

14

2%

9

3%

1

1%

200

12%

161

15%

• Surgery

379

6%

4

4%

0

0%

119

5%

32

5%

11

4%

3

2%

110

7%

100

9%

Sectors

3689

56%

44

47%

20

43%

1368

52%

266

43%

138

49%

51

27%

1025

61%

777

71%

• Primary care

1138

17%

23

24%

13

28%

460

17%

109

18%

38

13%

9

5%

206

12%

280

26%

• Home care

604

9%

6

6%

3

7%

287

11%

39

6%

26

9%

4

2%

134

8%

105

10%

• Hospital care

1537

23%

5

5%

3

7%

445

17%

110

18%

49

17%

24

13%

375

22%

526

48%

• Rehabilitation

345

5%

3

3%

0

0%

180

7%

15

2%

11

4%

3

2%

56

3%

77

7%

• Long-term care

252

4%

3

3%

3

7%

53

2%

11

2%

11

4%

3

2%

22

1%

146

13%

• Public health

1085

16%

20

21%

9

20%

349

13%

44

7%

47

17%

12

6%

395

24%

209

19%

Providers

3098

47%

33

35%

22

48%

1264

48%

318

52%

88

31%

41

22%

645

39%

687

63%

• Physician

1951

30%

14

15%

13

28%

767

29%

178

29%

50

18%

21

11%

345

21%

563

52%

• Nurse

1259

19%

17

18%

8

17%

526

20%

166

27%

40

14%

15

8%

284

17%

203

19%

• Pharmacist

338

5%

5

5%

4

9%

133

5%

21

3%

13

5%

7

4%

93

6%

62

6%

• Allied health professional

828

13%

9

10%

9

20%

334

13%

57

9%

27

10%

10

5%

170

10%

212

19%

• Lay/community health worker

322

5%

10

11%

3

7%

114

4%

23

4%

23

8%

8

4%

77

5%

64

6%

* This table does not provide data related to health system descriptions as they are not categorized in Health Systems Evidence according to topic domains. As a result, the total number of documents listed here is 6,613 but the total including the 209 health system descriptions indexed in Health Systems Evidence is 6,822.

Trends in production over time

We found that 2,893 of the 3,243 systematic reviews (89%) reported the last year the literature was searched and, based on the available data, 1,584 (49%) were conducted within the last five years (i.e., since 2008) (Figure 1). Of the 350 reviews for which we do not have the last year the literature was searched, 199 (57%) were published within the last five years. Our sub-analysis of Cochrane reviews revealed that most of the 501 reviews (n = 493, 98%) reported the last year the literature was searched and 332 (66%) of these were conducted within the last five years, while only 153 (30%) were up to date within the last two years, which is the Cochrane Collaboration’s recommended timeframe for updating reviews [14]. For overviews of systematic reviews, 37 of 46 (80%) reported the last year the literature was searched, and 26 of these (56%) were conducted within the last five years (Figure 1).
Figure 1

Last year literature was searched for systematic reviews and overviews of systematic reviews.

Regarding the year of publication, we found that 90 of the 94 (96%) evidence briefs for policy, 273 of the 283 (96%) systematic reviews in progress (i.e., protocols that had not yet been turned into a full review), all of the 186 systematic reviews being planned, 1,275 of the 1,669 (76%) economic evaluations (note that only those conducted in the last 10 years are indexed in HSE), 423 of the 1,092 (39%) descriptions of health reforms, and 101 of the 209 (48%) descriptions of health systems were published within the last five years (Figure 2).
Figure 2

Publication dates for documents.

Methodological quality of systematic reviews

Most systematic reviews (n = 2,928, 90%) indexed in HSE have been quality appraised using AMSTAR and their methodological quality is (on average) moderate (i.e., between 4 and 7 out of 11), which has remained relatively consistent over time (Table 3 and Figure 3). The average methodological quality of reviews addressing delivery arrangements (6.3/11) and implementation strategies (6.3/11) was somewhat higher than those addressing financial (5.9/11) and governance arrangements (5.6/11). However, methodological quality differs substantially according to the type of systematic review. Those produced by the Cochrane Collaboration are of high quality (i.e., between 9 and 11 out of 11) across all second-level topic domains (with none falling outside this range). In contrast, systematic reviews of effects (which include Cochrane reviews) are of moderate quality (with no second-level category averages falling outside the range of 4.8–7.4) and quantitative reviews addressing other types of questions were low to moderate quality (with no averages falling outside the range of 3.6–5.4).
Table 3

Mean quality rating* of systematic reviews addressing particular health system topics, by focus and type of systematic review

Domains**

All reviews (n = 2,928)†

All LMIC-focused reviews (n = 696)†

All types of systematic reviews†

Cochrane systematic reviews of effects (n = 484)†

Systematic reviews of effects (n = 2408)†

(Quantitative) Systematic reviews addressing other types of questions*** (n = 520)†

Governance arrangement (n = 197)

5.6

5.4

10.3

6.6

4.5

• Policy authority (n = 46)

5.1

5.4

9.9

5.7

4.5

• Organizational authority (n = 69)

5.5

7.0

10.3

6.6

4.7

• Commercial authority (n = 33)

4.6

4.1

10.5

5.4

3.6

• Professional authority (n = 50)

5.4

4.8

10.3

6.4

4.3

• Consumer & stakeholder involvement (n = 60)

5.7

5.3

10.3

6.5

4.7

Financial arrangement (n = 219)

5.9

6.6

9.9

6.4

4.4

• Financing systems (n = 50)

5.1

5.7

10.5

6.1

3.7

• Funding organizations (n = 25)

4.7

5.5

9.3

5.0

3.5

• Remunerating providers (n = 97)

5.5

5.8

9.4

5.6

5.4

• Purchasing products & services (n = 27)

4.4

4.2

10.0

4.8

3.9

• Incentivizing consumers (n = 83)

7.0

7.7

10.0

7.4

4.5

Delivery arrangement (n = 2,395)

6.3

6.8

9.9

6.7

4.5

• How care is designed to meet consumers' needs (n = 883)

6.5

6.7

10.0

6.7

5.0

• By whom care is provided (n = 1,276)

6.2

6.6

9.9

6.6

4.5

• Where care is provided (n = 642)

6.6

7.3

9.9

7.0

4.9

• With what supports is care provided (n = 807)

6.1

6.8

10.0

6.4

4.5

Implementation strategy (n = 1,519)

6.3

6.8

9.8

6.5

4.9

• Consumer-targeted strategy (n = 1,118)

6.5

6.9

10.0

6.7

4.9

• Provider-targeted strategy (n = 562)

6.0

6.5

9.5

6.2

5.0

• Organization-targeted strategy (n = 53)

5.9

6.4

9.7

6.2

5.4

†The numbers refer to the number of reviews in each category with a completed quality appraisal. The total numbers for each column are as follows: All reviews = 3,243; All LMIC-focused reviews = 737; Cochrane systematic reviews of effects = 501; Systematic reviews of effects = 2,629; Systematic reviews addressing other types of questions = 614.

* Due to the fact that the denominators for quality appraisal scores can vary using the AMSTAR tool when a question is deemed to be 'Not applicable', we standardized the mean quality score by: 1) converting each score into a percentage, 2) calculating the average of the percentage scores for each cell of the table, and 3) using the average percentage score to calculate the mean score out of 11.

** The numbers beside each category refer to the number of quality appraisals available.

*** This excludes systematic reviews of qualitative evidence that require a different approach to quality appraisal.

Figure 3

Mean quality ( AMSTAR ) score.

Availability of user-friendly summaries and links to additional content

The majority (n = 2,168, 67%) of systematic reviews now have at least one user-friendly (English-language) summary available from one of the eight groups in the world that package, quality appraise and/or facilitate assessments of the local applicability of systematic reviews (Table 4). The largest producers of these structured decision-relevant summaries for reviews about health system arrangements are the DARE (n = 1,962, 90% of those with at least one summary) and the Cochrane Library through its plain language summaries of reviews (n = 505, 23%). However, these review summaries are not targeted specifically at health system policymakers and stakeholders. The three groups representing the next largest shares of summaries of reviews about health system arrangements – Rx for Change (n = 211, 10%), Health Evidence™ (n = 231, 11%), and Australasian Cochrane Centre (n = 85, 4%) – are more targeted at these groups. A number of reviews also have one or more summaries available in Spanish (n = 474) and French (n = 391), but fewer summaries are available in the four remaining languages (14 for each of Arabic and Russian, 36 for Chinese and 1 for Portuguese).
Table 4

Number (and %) of user-friendly summaries available for systematic reviews, by type of systematic review*

Source of summary

All reviews (n = 3,243)**

Systematic reviews of effects (n = 2,629)

Systematic reviews addressing other types of questions (n = 614)

Australasian Cochrane Centre Policy Liaison Initiative

85

3%

85

3%

0

0%

Cochrane Library (plain language summaries)

505

16%

504

19%

1

0%

Database of Review of Effects

1,962

60%

1,821

69%

141

23%

Effective Health Care Research Programme Consortium

14

0%

14

1%

0

0%

Health Evidence™

231

7%

228

9%

3

0%

Reproductive Health Library

21

1%

21

1%

0

0%

Rx for Change

211

7%

201

8%

10

2%

SUPPORT

34

1%

34

1%

0

0%

* This table only provides counts of links available in English.

** 1,075 systematic reviews have no independently produced user-friendly summary.

Across all documents contained in HSE (n = 6,613), more than half (n = 3,893, 59%) have no links to one or more independently produced structured decision-relevant summaries (Table 5). Most documents have links to a scientific abstract from PubMed (n = 4,549), the Cochrane Library (n = 1,994, of which 299 have been translated into Spanish and 28 have been translated into French) or from another publisher (n = 4,281, of which 55 have been translated into French). Two thirds of the documents have a link to a full-text report that can be accessed without a subscription (n = 4,638, 70%); however, countries that have a national license to the Cochrane Library can access full Cochrane reviews (n = 501) and protocols of Cochrane reviews (n = 283) for free as well.
Table 5

Number of links available for all types of document, by language*

Links

Language

English

Arabic

Chinese

French

Portuguese

Russian

Spanish

No. of user-friendly summaries

• 0

3,893

7,855

7,835

7,479

7,868

7,855

7,411

• 1

3,260

14

32

375

1

14

438

• 2

507

0

2

15

0

0

19

• 3

141

0

0

0

0

0

1

• 4

44

0

0

0

0

0

0

• 5+

24

0

0

0

0

0

0

Abstracts

• PubMed

4,549

0

0

0

0

0

0

• Cochrane Library

1,994

0

0

28

0

0

299

• Publisher

4,281

0

0

55

0

0

0

Full-text

4,638

63

63

731

2

62

152

* Includes IberoAmerican (for Spanish) and BVS Salud (for Portuguese).

Low- and middle-income country (LMIC) focus

We found 1,228 (19%) documents in HSE that have an LMIC focus (Table 6), half of which are systematic reviews of effects (n = 614, 50%) and a lower percentage of which are economic evaluations (n = 280, 23%), systematic reviews addressing other types of questions (n = 123, 10%), and health system descriptions (n = 92, 7%). The 737 systematic reviews with an LMIC focus (which are the subset of documents described in Table 6 that we focus on here) were most often the result of including at least one study conducted in an LMIC setting (n = 712, 97%) with far fewer identified as having an LMIC setting as the target of the document (n = 193, 26%) or having at least one author from an LMIC (n = 118, 16%).
Table 6

Number (and %) of documents addressing health system topics, by type of low- and middle-income country (LMIC) focus

Health system topics

Type of LMIC focus

Any category* (n = 1,228)

Target of document (n = 648)

At least one LMIC author (n = 392)

At least one LMIC study included (n = 781)

Governance arrangement

117

10%

77

12%

46

12%

84

11%

• Policy authority

57

5%

46

7%

25

6%

40

5%

• Organizational authority

36

3%

25

4%

14

4%

29

4%

• Commercial authority

18

1%

16

2%

7

2%

14

2%

• Professional authority

26

2%

13

2%

6

2%

22

3%

• Consumer & stakeholder involvement

25

2%

15

2%

12

3%

19

2%

Financial arrangement

166

14%

120

19%

82

21%

108

14%

• Financing systems

62

5%

51

8%

25

6%

46

6%

• Funding organizations

18

1%

12

2%

5

1%

14

2%

• Remunerating providers

51

4%

33

5%

22

6%

42

5%

• Purchasing products & services

42

3%

33

5%

27

7%

14

2%

• Incentivizing consumers

59

5%

44

7%

27

7%

41

5%

Delivery arrangement

944

77%

472

73%

341

87%

618

79%

• How care is designed to meet consumers' needs

485

39%

318

49%

220

56%

248

32%

• By whom care is provided

445

36%

180

28%

141

36%

340

44%

• Where care is provided

287

23%

165

25%

120

31%

193

25%

• With what supports is care provided

202

16%

52

8%

40

10%

168

22%

Implementation strategy

516

42%

207

32%

154

39%

407

52%

• Consumer-targeted strategy

390

32%

151

23%

112

29%

318

41%

• Provider-targeted strategy

195

16%

85

13%

63

16%

147

19%

• Organization-targeted strategy

17

1%

7

1%

3

1%

12

2%

* Of these documents, 737 are systematic reviews.

Similar to the full set of reviews contained in HSE, the systematic reviews with an LMIC focus most often addressed topics related to delivery arrangements (n = 590) and implementation strategies (n = 394) in whole or in part, with far fewer addressing financial (n = 74) and governance arrangements (n = 55). The most common sub-categories under delivery arrangements containing systematic reviews with an LMIC focus were “by whom is care provided” (n = 319), “how care is designed to meet consumers’ needs” (n = 236), “where care is provided” (n = 171), and “with what supports is care provided” (n = 161). The majority of systematic reviews within the implementation strategies category address consumer-targeted strategies (n = 310), while fewer addressing provider-targeted strategies (n = 132). Within the governance arrangements category, policy authority (n = 20) represents the largest sub-category and within the financial arrangements category, “incentivizing consumers” (n = 29), “remunerating providers” (n = 27) and “financing systems” (n = 26) represent the largest sub-categories.

The systematic reviews identified as having an LMIC focus also have a similar profile to the global stock of reviews in terms of how recently they were conducted and their quality. As outlined in Figure 4, slightly more than half of the reviews (n = 403, 54%) were conducted within the last five years (since 2008) and of the 12 overviews of systematic reviews with an LMIC focus 6 (50%) were conducted within the last five years. In addition, we found that all of the systematic reviews in progress (n = 25), systematic reviews being planned (n = 17), and health reform descriptions (n = 3), as well as 237 of 280 (85%) economic evaluations, 58 of 60 (97%) evidence briefs for policy and 42 of 92 (46%) health system descriptions, with an LMIC focus, were published within the last five years. In terms of quality, almost all of the reviews with an LMIC focus (n = 696, 94%) have been appraised for methodological quality using AMSTAR and (on average) have a moderate quality score (range = 4.2–7.7 out of a possible score of 11) across topic domains (Table 3).
Figure 4

Last year literature was searched for systematic reviews and overviews of systematic reviews with an LMIC focus.

Conclusions

We have provided a detailed examination of the global stock of research evidence relevant to health systems policymaking. Systematic reviews of effects are the most common of the nine types of research evidence in HSE (n = 2,629, 40%). Most systematic reviews address topics related to delivery arrangements (n = 2,663) or implementation strategies (n = 1,653) in whole or in part, which may reflect a tendency among more clinically oriented researchers to first examine the effects of an intervention and then (faced with feasibility and other concerns raised by policymakers and stakeholders) to examine the effects of alternative ways of delivering the intervention or implementing it on a large scale. Half (49%) of the systematic reviews presented findings based on searches conducted within the last five years (since 2008), quality ratings ranged from 5.6 (for governance arrangements) to 6.3 (for delivery arrangements and implementation strategies) out of a possible 11, nearly one in four reviews (n = 737, 23%) contain at least one study from a LMIC, and 1,075 reviews have no independently produced structured decision-relevant summary.

This study has one key strength and one potential limitation. Our analysis represents the first effort (at least to our knowledge) to systematically assess the global stock of nine types of research evidence relevant to health systems policymaking. Moreover, we have developed an approach to real-time reporting for all of the tables and figures included in our analysis, which will allow us to update the tables and figures at any time and identify changes over time. The potential limitation of our study is that, by relying on one data source for our analysis, we have missed including relevant systematic reviews. However, we believe this possibility to be remote given that HSE systematically culls relevant documents from a number of databases (Medline, Cochrane Database of Systematic Reviews, three databases – DARE, PROSPERO, Economic Evaluations Database – from the Centre for Reviews and Dissemination, Rx for Change, and the Cochrane Qualitative and Implementation Methods Group’s reference database for qualitative reviews) and many other sources such as listservs (e.g., EvidenceUpdates and PAHO EQUIDAD) and websites (e.g., AHRQ, Campbell Collaboration, EPPI-Centre and several more). As a result, searching other sources would be unlikely to identify many documents. For those interested in reviewing the full list of sources and methods used to build and continuously update HSE we provide these details in a separate manuscript [Unpublished data].

Based on the profile of research evidence provided here, particularly the profile of systematic reviews, we believe that greater effort should be placed on several priorities to enhance the usefulness of research evidence about health systems and support its use by policymakers and stakeholders. First, there is a need to support efforts to regularly update systematic reviews, as doing so will ensure that policymakers and stakeholders have access to the most recent synthesized research evidence. As outlined in our findings, half of the systematic reviews in HSE were conducted five or more years ago and a large proportion of those produced by the Cochrane Collaboration have also not been updated within their recommended timeline of two years. Such efforts could be supported (at least partially) by funders of systematic reviews providing resources both for new systematic reviews and for updating existing reviews (e.g., by identifying reviews that address timely policy questions but are out of date).

Second, there is a need to increase the average quality of systematic reviews. This is particularly evident for systematic reviews addressing questions other than effectiveness and/or for those addressing topics related to governance and financial arrangements (many of which are reviews addressing other types of questions), which generally scored lower for quality as compared to reviews addressing questions of effectiveness. Registering titles and protocols for systematic reviews and implementing specific quality standards as part of the registration process is a promising mechanism that may help increase the overall quality of reviews. This requirement from the Cochrane Collaboration contributes (at least in part) to the average level of quality for Cochrane reviews being significantly higher than the rest of the reviews in HSE. In addition, PROSPERO [15] now offers an international prospective register of systematic reviews requiring minimum methodological standards based on the PRISMA statement [16]. These efforts could be further supported by journals requiring systematic reviews to be registered in order to be considered for publication.

It should be noted that the AMSTAR tool was designed originally for reviews of effects. However, the fundamental methodological requirements for reviews that include quantitative evidence remain the same and where they differ there is an option to mark a question as “not applicable” to the score. As noted in our results, we did not conduct quality appraisals of reviews that draw exclusively on qualitative research evidence given that there is currently no quality appraisal tool available (at least to our knowledge) that takes into account the unique methods and approaches to data analysis used in these types of reviews (although two of us – MGW and JNL – are currently finalizing a tool that will be able to be used for this purpose).

Third, there is a need to use the global stock of systematic reviews to start developing global guidance to support evidence-informed policies about health systems [1719]. However, the utility of such guidelines is dependent upon the availability of systematic reviews addressing the full range topics related to governance, financial and delivery arrangements within health systems, and about implementation strategies that can support change in health systems. This will require diversifying the types of topics addressed in systematic reviews (particularly for governance and financial arrangements or for increasingly important topic-specific domains such as long-term care, which are relatively underserved in the current global stock) as well as ensuring relevance to LMICs (for which there are comparatively few reviews). Addressing this could also include conducting assessments of whether the current distribution of systematic reviews corresponds to policymakers’ and stakeholders’ priorities.

Abbreviations

DARE: 

Database of abstracts of reviews of effects

HSE: 

Health systems evidence

LMICs: 

Low- and middle-income countries.

Declarations

Acknowledgments

This work was funded by grants from the Canadian Institutes of Health Research to the Pushing Useful Science to Healthcare Managers and Policymakers (PUSHMaP) study (MOP84546) and the Canadian Cochrane Centre (13278A and CON105529) and by a grant from the Cochrane Collaboration’s Opportunities Fund (RFP20061101). We thank Amanda Hammill, Jennifer Boyko, Jeremy Grimshaw and Signe Flottorp for their contributions to the methods that led to the development of Health Systems Evidence. We also thank the many staff and students who contributed to building the content of Health Systems Evidence. Lastly, we would like to thank the various organizations that produce the resources on which Health Systems Evidence depends.

Authors’ Affiliations

(1)
McMaster Health Forum, McMaster University
(2)
Centre for Health Economics and Policy Analysis, McMaster University
(3)
Department of Clinical Epidemiology and Biostatistics, McMaster University
(4)
Health Policy PhD Program, McMaster University
(5)
Department of Political Science, McMaster University
(6)
Department of Global Health and Population, Harvard School of Public Health

References

  1. Lavis JN, Davies HTO, Oxman AD, Denis J-L, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005, 10 (Suppl. 1): 35-48.View ArticlePubMedGoogle Scholar
  2. Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking. Lancet. 2004, 364: 1615-1621. 10.1016/S0140-6736(04)17317-0.View ArticlePubMedGoogle Scholar
  3. Lavis JN, Davies HTO, Gruen RL: Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy. 2006, 1: 21-33.PubMedPubMed CentralGoogle Scholar
  4. Egger M, Smith GD, O'Rourke K: Rationale, potentials, and promise of systematic reviews. Systematic Reviews in Health Care: Meta-Analysis in Context. 2001, ondon: BMJ Books: 2nd edition. Egger M, Smith GD, Altman DG (Eds.), 3-19.View ArticleGoogle Scholar
  5. Lavis JN: How can we support the use of systematic reviews in policymaking?. PLoS Medicine. 2009, 6 (11): e1000141-10.1371/journal.pmed.1000141.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Montori VM, Wilczynski NL, Morgan D, Haynes RB, for the Hedges Team: Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2004, 330: 68-View ArticlePubMedGoogle Scholar
  7. Wilczynski NL, Haynes RB: for the Hedges Team: EMBASE search strategies achieved high sensitivity and specificity for retrieving methodologically sound systematic reviews. J Clin Epidemiol. 2007, 60: 29-33. 10.1016/j.jclinepi.2006.04.001.View ArticlePubMedGoogle Scholar
  8. Dobbins M, DeCorby K, Robeson P, Husson H, Tirilis D, Greco L: A knowledge management tool for public health: health-evidence.ca. BMC Publ, Health. 2010, 10: 496-10.1186/1471-2458-10-496.View ArticleGoogle Scholar
  9. McMaster Health Forum: Health Systems Evidence. 2013, [http://www.healthsystemsevidence.org/] Accessed 15th Aug 2013Google Scholar
  10. Moat KA, Lavis JN, Wilson MG, Røttingen J-A, Bärnighausen T: Twelve myths about systematic reviews for health system policymaking. J Health Serv Res Policy. 2013, 18 (1): 44-50. 10.1258/jhsrp.2012.011175.View ArticlePubMedGoogle Scholar
  11. Rockers PC, Feigl A, Røttingen J-A, Fretheim A, De Ferranti D, Lavis JN, Melberg HO, Bärnighausen T: Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: A meta-review. Health Policy. 2012, 104: 206-214. 10.1016/j.healthpol.2011.12.007.View ArticlePubMedGoogle Scholar
  12. Moat KA, Lavis JN: Supporting the use of Cochrane Reviews in health policy and management decision-making: Health Systems Evidence [editorial]. The Cochrane Library. 2011, 8: ED000019-Google Scholar
  13. Shea B, Grimshaw J, Wells G, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 10-10.1186/1471-2288-7-10.View ArticlePubMedPubMed CentralGoogle Scholar
  14. Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. 2013, C, [http://www.cochrane-handbook.org] Accessed 29th January 2013Google Scholar
  15. Centre for Reviews and Dissemination: PROSPERO. 2013, International Prospective Register of Systematic Reviews, [http://www.crd.york.ac.uk/prospero/] Accessed 15th Aug 2013Google Scholar
  16. Moher D, Liberati A, Tetzlaff J, Altman DG: The PRISMA group: preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Lavis JN, Røttingen J-A, Bosch-Capblanch X, Atun RA, El-Jardali F, Gilson L, Lewin S, Oliver S, Ongolo-Zogo P, Haines A: Guidance for evidence-informed policies about health systems: Linking guidance development to policy development. PLoS Medicine. 2012, 9 (3): e1001186-10.1371/journal.pmed.1001186.View ArticlePubMedPubMed CentralGoogle Scholar
  18. Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen J-A, Dröschel D, Beck L, Abalos E, El-Jardali F, Gilson L, Oliver S, Wyss K, Tugwell P, Kulier R, Pang T, Haines A: Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012, 9 (3): e1001185-10.1371/journal.pmed.1001185.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Lewin S, Bosch-Capblanch X, Oliver S, Akl EA, Vist GE, Lavis JN, Ghersi D, Røttingen JA, Steinmann P, Gulmezoglu M, Tugwell P, El-Jardali F, Haines A: Guidance for Evidence-Informed Policies about Health Systems: Assessing How Much Confidence to Place in the Research Evidence. PLoS Med. 2012, 9 (3): e1001187-10.1371/journal.pmed.1001187.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© Wilson et al.; licensee BioMed Central Ltd. 2013

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.