If health organisations and staff engage in research, does healthcare improve? Strengthening the evidence base through systematic reviews

Abstract

Background

There is an often-held assumption that the engagement of clinicians and healthcare organizations in research improves healthcare performance at various levels. Previous reviews found up to 28 studies suggesting a positive association between the engagement of individuals and healthcare organizations in research and improvements in healthcare performance. The current study sought to provide an update.

Methods

We updated our existing published systematic review by again addressing the question: Does research engagement (by clinicians and organizations) improve healthcare performance? The search covered the period 1 January 2012 to March 2024, in two phases. First, the formal updated search ran from 1 January 2012 to 31 May 2020, in any healthcare setting or country and focussed on English language publications. In this phase two searches identified 66 901 records. Later, a further check of key journals and citations to identified papers ran from May 2020 to March 2024. In total, 168 papers progressed to full-text appraisal; 62 were identified for inclusion in the update. Then we combined papers from our original and updated reviews.

Results

In the combined review, the literature is dominated by papers from the United States (50/95) and mostly drawn from the Global North. Papers cover various clinical fields, with more on cancer than any other field; 86 of the 95 papers report positive results, of which 70 are purely positive and 16 positive/mixed, meaning there are some negative elements (i.e. aspects where there is a lack of healthcare improvement) in their findings.

Conclusions

The updated review collates a substantial pool of studies, especially when combined with our original review, which are largely positive in terms of the impact of research engagement on processes of care and patient outcomes. Of the potential engagement mechanisms, the review highlights the important role played by research networks. The review also identifies various papers which consider how far there is a “dose effect” from differing amounts of research engagement. Additional lessons come from analyses of equity issues and negative papers. This review provides further evidence of contributions played by systems level research investments such as research networks on processes of care and patient outcomes.

Background

There is an often-held assumption that the engagement of clinicians and healthcare organizations in research improves healthcare performance at various levels. This assumption contributed to policy documents from various health organizations promoting research engagement by healthcare providers as a way of improving healthcare, for example, in the United Kingdom [1,2,3]. Therefore, it was believed that policy-makers who make relevant decisions, such as on the allocation of resources for health and health research systems, should have access to evidence on the validity of the assumption. In the United Kingdom, two programmes of the National Institute for Health Research (now called the National Institute for Health and Care Research) (NIHR) decided to commission reviews of the global evidence on this [1,2,3].

The wide-ranging brief provided for the second review, which was the original review by the authors of this present paper (published in full as Hanney et al. in 2013 [3] and more succinctly as Boaz et al. in 2015 [2]), included the additional aim of conducting a theoretically grounded synthesis to explore the mechanisms by which research engagement might improve healthcare [3]. The protocol for that study considered pertinent global literature, including on accelerating the adoption of evidence in health systems, and ways to enhance the relevance of the research conducted to the needs of health systems. The final protocol published as part of the Hanney et al. report [3] then used these ideas to identify possible mechanisms that would be worth analysing to help understand the processes that might be at work when research engagement leads to improved health. Among these was the idea that engaging in conducting research increases the ability and willingness of clinicians to use research findings from the global pool of knowledge, and here the concept of “absorptive capacity” was expected to be useful [3].

Some analyses focussed on the importance of exploring the relationship between research engagement and improved healthcare to contribute towards understanding of the benefits for healthcare performance in the context of a strong research culture. These papers were reviewed in Australia by Harding et al. in 2017 [4].

As far as we are aware, these three systematic literature reviews published in the second decade of this century provided the first analyses of the empirical evidence available to support the assumption of improved healthcare from research engagement [1,2,3,4]. Their differing scopes and approaches are summarized briefly in Table 1.

Table 1 Summary of the three systematic reviews conducted prior to 2020 on whether research engagement by healthcare organizations and staff improves healthcare

All three of the reviews reported some evidence of a positive association between research engagement and healthcare performance, but the available evidence was not mature enough to support statements about causality [2]. Our review [2, 3] had the widest scope of the three, reflecting the broad brief given by our NIHR funder. It included an extensive initial mapping exercise, a formal focussed review, and a wider review which drew on the earlier stages to explore, as noted above, the mechanisms by which research engagement might improve healthcare [3]. Our review identified 33 papers from 9 countries (15 from the United States), 28 of which reported positive findings [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. Even our review concluded, however, that there did not appear to be a well-structured, steadily accumulating body of knowledge about the benefits associated with research engagement.

In the succeeding years, we have identified a continuing and growing interest in this general topic, therefore an updated review seemed desirable to gather more evidence about how far research engagement might lead to improved healthcare and the mechanisms involved. In addition to these general questions, our original review had identified two specific issues that could usefully be considered further. These were research networks as potentially important mechanisms through which research engagement might improve healthcare, and whether greater amounts of research engagement would have a larger beneficial effect. Our original review also covered some aspects of a third issue (health equity) that has subsequently become increasingly important [2, 3].

The growing development of research networks has been associated with efforts to move towards more formalized attempts to boost the role of health research systems in accelerating science and facilitating the translation of research into practice [2, 3]. However, at the time of our original review, the evidence was still emerging and its availability was heavily skewed by the different timing of the establishment of formal research networks in different countries.

In the United States, various research networks had been set up in the second half of the last century, and most of the early papers on networks and their role came from there [2]. These networks in the United States are described in the “Glossary of the United States of America and United Kingdom Research Organizations and Networks Discussed in the Papers” (see Additional file 1). They include the National Cancer Institute (NCI)-funded Cancer Community Oncology Program (CCOP), established to encourage outreach and improve equity by bringing the advantages of clinical research to cancer patients in their own communities [37]. In the United Kingdom in contrast, national research networks were not formally created until this century, too late for any potential benefits to patient outcomes to be fully researched and reported prior to our review which started in 2011. However, even in our original review we were aware of concurrent United Kingdom work to measure those outcomes and to improve patient access to clinical research, and identified a need for further evaluations (see Hanney et al. [3], pp. 48, 83).

Subsequently, we also became increasingly aware of new studies on the effects of the developing research networks, especially in the United Kingdom, and Boaz et al. identified a promising approach in statistical analysis that could help further analysis [2]. As set out in the Glossary, there have been policy shifts and organizational changes in the United States and the United Kingdom, and there have been further ones elsewhere, which are designed to promote research networks to address the time lag between the production of research and its use in practice, including various efforts to strengthen links between academic centres and community services. There has also been an increasing emphasis, including within research networks, on the potential research contribution of healthcare professionals other than medical professionals.

Our original review had also noted a partly related second issue as worthy of further attention. This is the question of whether the association between research engagement by healthcare providers and improved healthcare outcomes increases with greater amounts of research participation. There was early evidence that it did. This came, in particular, from the 2008 paper by Majumdar et al. [26] that compared outcomes for patients with angina in hospitals in the United States having a high level of angina research activity with hospitals with low research activity, and those with no research activity. Other papers compared centres with different levels of research activity within a research network [23]. However, there was little certainty about extent and implications around this issue at that time, although it has become increasingly important with the development of the comprehensive research networks that we summarize in the Glossary. It also has theoretical implications for the exact nature of the association between research engagement and improved healthcare: in our original review we argued that further data on this effect, and on the time an institution was research active, “are needed to provide evidence of causation” (p. 12) [2].

These findings also have implications for health equity, the third unresolved issue. More outreach by research networks means more access to clinical research and its benefits for more patients. The United States CCOP has been rightly lauded for achieving this [37], but can that be squared with the emerging finding that higher levels of research participation in specific provider institutions bring greater benefit to the patients in those centres?

Reflection on these uncertainties further strengthened the argument that with all the developments since our original review, it seemed timely in 2020 to revisit this topic to explore and collate what additional understanding had been gained. While conducting the resulting update, we became aware of some more recent developments. A United Kingdom qualitative systematic review was published in 2021 that explored the impact of research activity by healthcare professionals other than medical professionals [38], and another UK review published in 2023 focussed on research engagement by allied health professionals (AHPs) [39]. With few exceptions, the papers specifically on nursing and AHPs in these reviews were typically smaller scale than the papers included in our formal review, and/or usually did not include the quantifiable comparisons that featured in most of our included papers. Nevertheless, these reviews usefully illustrate the growing interest in the contribution of these healthcare professionals in countries such as Australia, Canada and the United Kingdom.

In addition, we identified a large-scale study from the United States by Shahian et al. [40] that was published in 2022 and examined the link between research engagement and improved healthcare performance in 5 major medical fields across 1604 Medicare-participating hospitals. A noticeable facet of the paper by Shahian et al. was their referencing of a large number of papers that we had identified either in our original review, or in the first phase of our updated review [40].

To ensure our updated review adequately reflected all such developments since May 2020, we conducted a further search in March 2024. The review presented here is based on papers identified in both phases of the updated review, the findings of which are then combined with those from our original review.

Review question

To identify studies, the primary research question used the same approach as Boaz et al. [2, 3].

• Does research engagement (by clinicians and organizations) improve healthcare performance?

By research engagement, we mean, as in our original review, engagement in research rather than the broader concept of engagement with research, and we are referring to participation in research by healthcare organizations and staff rather than patient participation in trials. Engagement in research is taken to mean, “a deliberate set of intellectual and practical activities undertaken by healthcare staff (including conducting research and playing an active role in the whole research cycle) and organizations (including playing an active role in research networks, partnerships or collaborations)” (p. 2) [2].

Methods

Design

The 2020 decision to complete an update of the previous review [2, 3] was informed by a published decision framework for updating systematic reviews [41]. After completion in 2024 of the comprehensive initial phase of the updated review, including the two searches and considerable subsequent analysis, we recognized, as noted above, that while we had been conducting the review some important further papers had been published. We wanted to incorporate such papers, and so decided to conduct a further search for papers. The design of this final phase (which included a third search) was informed both by the fact that we had already identified a considerable number of papers for the updated review, and by the way new papers in this field were by now much more likely to cite earlier papers, with Shahian et al. [40] being a prime example. Therefore, we thought it was reasonable to rely to a much greater extent on checking citations to the papers already identified, as explained below.

Search strategy and information sources

Search 1 (update)

The first step in syntax development used the Medline Ovid strategy published by Boaz et al. [2].

Initial diagnostic testing indicated issues preventing code execution. Due to the syntax comprising several nested terms and Boolean operators, it was rebuilt using recommendations for “single-line” optimization for debugging complex code [42].

Search 2 (modified)

The syntax for Search 2 was a term modification to capture papers that more explicitly indexed research networks and collaborations. Search 2 necessitated a deeper dive into the full-text content of papers. The decision to search full-text articles reflected observations that the sensitivity of Search 1 was potentially affected by the variable quality (and relevance for our review) of abstracts, a consistent challenge for reviewers [43]. As a second search also adapted published syntax, the Preferred Reporting Items for Systematic reviews and Meta-Analyses Literature Search—Extension Checklist (PRISMA-S) reporting protocol was followed [44]. (The full text for search strategies is provided in “Search Strategy and Syntax Sensitivity”; see Additional file 2).

Electronic databases

Nine electronic records collections were used in Search 1: Medline (OVID and EBSCO), EMBASE, PsycInfo (OVID and EBSCO), CINAHL, Web of Science, Health Management and Information Consortium and British Nursing Institute. The mix provided parity with previous reviews and mitigated risk of missed papers by combining general and specialized databases. Different interfaces (e.g. OVID, EBSCO) for the same collection were also included to offset variations due to platform [45]. Grey literature was not searched: these collections failed to uniquely identify papers in previous reviews on this topic. Search 2 was restricted to the Medline EBSCO Full Text records, which was the collection which yielded the highest hit ratio for relevant papers (see Additional file 2).

Other sources

Manual and snowball searching were used in three ways. Firstly, a range of search engines (Google Scholar, PubMed, ProQuest Central, Scopus, the Web of Science Cited Reference Search) were used to track citations for (a) prior reviews as whole papers, (b) the individual studies within these reviews and (c) article reference lists. Secondly, key journals that published studies shortlisted in the previous reviews were hand-checked, including: Implementation Science, PLOS One, BMJ Open and BMC Health Services Research. Thirdly, topic experts suggested papers for consideration.

Search 3 (final phase)

As explained above, we subsequently conducted a further search covering May 2020–March 2024. This consisted of: a hand-search of three of the journals in which papers from the first phase of the updated review had been published (Health Research Policy and Systems, Implementation Science and Medical Care); a check of papers in the two reviews published in this period [38, 39]; and a check of citations in this period to all the papers identified both in our original review and in the update’s initial phase.

Eligibility criteria

The following limiters were applied:

• Timeframe: 1 January 2012 to 20 March 2024 (inclusive of eprint)

• Population: Human (any setting)

• Language: English (any country)

• Paper type: Academic Journals (scholarly works). Conference papers were admitted as flags for accessible peer-reviewed works (e.g. pre-print) or key teams.

Three criteria were defined, guided by definitions from the original review [3].

Criterion A: study design

Empirical studies using method/s aligned with health services research, including clinical trials, retrospective cohort and survey methods. Studies with only patient reported outcomes (e.g. satisfaction) were excluded.

Criterion B: healthcare performance

Studies must report an outcome indexing performance assessment for a care process or healthcare improvement. The following were excluded: staff-specific reports alone, (e.g. job satisfaction or morale), policy impacts alone (no flow through to healthcare), descriptions of networks without outcomes data.

Criterion C: research engagement

Explicit demonstration of engagement in research including: agenda-setting, conducting research, participation in action research or in networks where the research involvement is noted. This criterion also allowed engagement implicitly through research network membership, even if a specific study was not recorded, but there was a comparison of healthcare between member and non-member settings. More details about examples that were in scope can be found in Hanney et al. [3, p. 2].

Records management

To efficiently manage the export of the large records for the first two searches, Endnote X9 (Clarivate) was used to combine downloads from different databases and discard software detected duplicates. The endnote library was imported into Rayyan, a free multi-collaborator online screening tool [46]. Study selection procedures for Searches 1 and 2 followed the same screening/eligibility check sequence.

Screening and eligibility/quality checking

In Rayyan, titles were scanned to exclude papers that were irrelevant, did not meet criteria or were non-exact duplicates. Abstracts of retained records were then screened and classified as “include”, “exclude” or “maybe”. A third screening of “maybe” classifications forced a binary coding of “include” or “exclude”, with comment flags on issues. A final records’ sweep with the Rayyan query function checked for misclassified studies. This four-step screening process was completed by a single reviewer (BG).

Full-text for each provisionally included study was uploaded into Rayyan. The initial eligibility check was completed by three experts who were involved in article screening for Hanney et al. [3]. As a criterion check and to orient reviewers to the Rayyan platform, a practice phase used 10 randomly sampled records. The abstract was the primary source for expert reviewers, with full-text also available. After the practice task and consensus discussion of criteria, a batch of records (alphabet determined) was assigned to each expert reviewer, to rate each paper as “include”, “exclude” or “maybe” (ratings were unblinded). If the rating pair (i.e. B.G. and an expert from the original review) were both “include”, the paper was progressed to full-text appraisal. If there was disagreement, papers rated as “maybe” were reassigned to another expert reviewer for an opinion, and those rated as “exclude” by an expert reviewer were marked for discard. If consensus for a “maybe” paper could not be reached by discussion, it was progressed to a full text appraisal, conducted by a single reviewer (B.G.) using all available information sources and reviewer ratings.

A final review of all potential “includes” was jointly conducted by team members, including a few papers identified by other sources such as continued manual snowballing from key papers. The study selection procedures for Search 3 mirrored this final step, and so consisted of a review of all potential “includes” conducted jointly by team members.

Study quality was assessed using the mixed-methods appraisal tool (MMAT v2018), on a scale of 1 (low) to 5 (high) [47]. The MMAT accommodated all designs in the paper set. The majority of the papers have a design which fitted into the MMAT category of quantitative non-randomized. All papers scored good to high quality on the five questions in their relevant MMAT subscale. The lower end of ratings (good) was typically due to lack of information in the article, such as whether and/or how confounding factors may have been identified or managed. Quality ratings were not used to exclude papers, but formed part of the discussion about the quality and contribution of the papers.

Data extraction, coding and ethics

As Rayyan is only a screening platform, a data extraction sheet was created in Excel (v2016) for each included paper. A university research ethics committee deemed the project as not requiring formal ethical approval, due to secondary data mining on anonymized aggregated records.

Analysis

A large and methodologically diverse mix of papers was identified with a range of different outcomes and outcome measures. The papers were combined through a process of critical interpretive synthesis inspired, as in our original review, by the approach outlined by Dixon-Woods et al. [48]. This involves adopting an iterative approach to refining the research question, searching the literature and defining and applying codes and categories. It enables the generation and development of theory with strong explanatory power and uses relevance as one measure of quality. Following analysis of the papers in the updated review, we collated the results from the updated review with those from our original review to create one combined set of papers for overall analysis.

Results

Figure 1 summarizes the review literature flow. The two formal searches identified 66 901 records, with 68 further papers coming from other sources, including the March 2024 extension. From these, 168 papers progressed to full-text appraisal, and 62 were identified for inclusion [40, 49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109].

Fig. 1

Flow diagram for literature search

This review updates the previous review conducted by the team [2]. Table 2 outlines the 95 papers in our combined review: the 62 additional papers in the updated review along with the 33 papers in our original review. The latter 33 papers are shown in italics in Table 2, which includes details about the study characteristics of all 95 included papers as well as key dimensions of the findings. To complement Table 2, brief notes on the development and scope of key United States and United Kingdom research networks/organizations discussed in the papers are provided in the “Glossary of the United States of America and United Kingdom Research Organizations and Networks Discussed in the Papers” (see Additional file 1).

Table 2 Combined review’s 95 papers on whether research engagement by healthcare organizations and staff improves healthcare

Study characteristics

Across the 95 papers, 12 countries are either the location for the research engagement described in a single-country study, or the location from which a multi-country study was led, with one paper led from South Africa having authors from a range of African countries (and Yemen) [88]. The 12 countries are: United States (50 papers), United Kingdom (17), Canada (7), Spain (5), Germany (4), the Netherlands (3), Australia (2), Denmark (2), South Africa (2), China (1), Finland (1) and Sweden (1).

Cancer was the most common field, with 32/95 papers overall. Next came hospital care in general/multi-field/acute care with 16 papers, cardiovascular/stroke (12), substance use disorder (7), dentistry (3), mental health/psychiatry (3) and obstetrics (3).

Main findings

As presented in Table 2, the key findings from the combined review are presented in terms of the four pairs of binary options, though inevitably some papers did not neatly fit into one category. The first categorization is in terms of the level of analysis explored in different papers; 23 papers compare clinicians, but 72 compare organizations. There is an even higher proportion in the updated review at the organizational level (50/62, 81%) than in our original review (22/33, 67%).

A total of 86 of the 95 papers report positive results, of which 70 are purely positive and 16 are positive/mixed meaning that there are some key negative elements in their findings, that is, important parts of the analysis where a lack of healthcare improvement is identified. Nine papers are negative, of which four are negative-mixed.

The final two pairs of binary options consider just the 86 positive papers. In total, 37/86 report improved health outcomes in terms of reduced mortality or morbidity. A higher proportion of the positive papers in the updated review (30/58, 52%) than in our original review (7/28, 25%) describe such improved health outcomes. There is a corresponding reduction from three quarters (21/28) to a half (28/58) in the proportion of papers solely describing improved processes in terms such as applying proven interventions.

Finally, in terms of the type of impact, 55/86 of the papers describe research engagement leading to a broader impact on healthcare performance. Broad impacts arise when the improved healthcare goes more widely than just being linked to clinicians or healthcare organizations implementing the findings, or processes, from their own research more rapidly/extensively than do others. When the improved healthcare is linked to the results or processes of their own research, that is categorized as specific impact, which is the case in 31/86 papers. Using these various categories, Fig. 2 outlines the findings from the combined review, alongside the findings from our original review, and the updated review. This highlights various trends in terms of the main findings.

Fig. 2

Results from Boaz et al. systematic reviews of whether research engagement by health organizations and staff improves healthcare: analysis of original; updated; and combined reviews (and of the 86 positive papers). Green rows (top): original review; Brown rows (middle): updated review; blue rows (bottom): combined review

One further trend in terms of the type of analysis is seen in the 11/95 papers that used bibliometric analysis as an indicator of the extent, and/or quality, of research engagement compared with some measure of the healthcare performance, in terms of processes and/or outcomes [30, 40, 50, 51, 56, 65, 76, 95,96,97, 100]; 10 of these papers are in the updated review, with just 1 [30] from our original review. All these 11 papers are positive, but various types of bibliometric analysis are used. The broad categories of academic indicators applied include publication volume [95, 96], publication “quality” (for example, as measured by citations) [51] and a combination of volume and “quality” [30, 40, 50, 56, 65, 76, 97, 100]. Of the latter, five relatively small studies suggest that the association with “quality” was stronger than with volume. The bibliometric studies also illustrate the varying levels of analysis at which the included studies in the review are conducted; 4 of the 11 papers compare the academic outputs of clinicians [50, 56, 96, 97] and 7 make comparisons at an organizational level [30, 40, 51, 65, 76, 95, 100], focussing variously on academic outputs at ward, department or hospital/trust level.

The combined review allows for a range of issues to be analysed more thoroughly than they had been in our original review. These include issues highlighted in the background such as the role of networks and the “dose effect”. These are examined in turn below, followed by consideration of how far the included studies have addressed various aspects of health equity, and finally an analysis of lessons from the overall portfolio of positive and negative studies.

The role of research networks

The full significance of papers on research networks is seen in the combined review. Using the inclusive definition developed by Laliberte et al. [24], we have applied the term to various arrangements that, however loosely, give some measure of commonality to the research of multiple healthcare organizations that not only enhance science production, but also share a concern to transfer research findings into clinical practice. About half the papers in the combined review analysed research activity by clinicians or healthcare organizations who were part of research networks of various types.

In the United States, the NCI cancer research networks include the NCI-designated Comprehensive Cancer Centres, the NCI Cooperative Groups and collaborative groups of community hospitals affiliated to the NCI’s CCOP- see the Glossary for its new name. In various ways these networks all include outreach and the engagement of community physicians in their brief; see the Glossary for more details. Their potential was recognized early in the 2005 study by Laliberte et al. [24] that looked at these networks and concluded that network membership may influence compliance with treatment guidelines, and should therefore be taken into account in predictive models of compliance.

Seven included papers illustrated various aspects of this issue by comparing the processes and outcomes for patients treated at NCI-designated (comprehensive) cancer centres with those treated elsewhere, six of these studies showed better outcomes for patients treated at NCI centres [52, 64, 80, 84, 86, 106], while one paper suggested that despite better processes, patient outcomes were worse at NCI centres. This paper is considered in the section on negative papers below [81]. Of the positive papers, Paulson et al. showed how the NCI designation was “associated with lower risk of postoperative death and improved long-term survival” (p. 675) [86], identified possible factors such as better adherence to guidelines, and demonstrated that the better outcomes at NCI-designated centres remained even when compared with non-NCI designated centres with a similar high volume of cases [86]. Wolfson et al. identified the requirements that underpin the positive association between high-quality research and high-quality care [106]. These included the mandate NCI centres have to “lead clinical trials, exchange ideas, disseminate findings” (p. 3892), which showed how the centres could act as part of a network. Wolfson et al. continued: “The NCI operates on the belief that a culture of discovery, scientific excellence, transdisciplinary research, and collaboration yields tangible benefits extending far beyond the generation of new knowledge” [106].

Building on Laliberte et al. [24], Carpenter et al. demonstrated an association between CCOP membership and accelerated innovation adoption but added the important codicil that it was not possible to “definitively ascertain whether there is a direct causal relationship between the two” [54].

Improved healthcare has also been associated with membership of the United States practice-based research networks (PBRNs). These networks cover family practice/primary care, dentistry, mental health and substance abuse. Like the CCOP and its affiliates, PBRNs involve practising clinicians in the community who conduct research. The combined review includes seven PBRN papers covering primary care and dentistry, all of which are positive [32, 36, 66, 78, 83, 92, 108] and one of which describes an international dental PBRN led from the United States that includes three Scandinavian countries [66].

A total of seven papers from another PBRN, the National Institute on Drug Abuse’s Clinical Trials Network (CTN), also provided evidence of accelerated translation, identified mechanisms through which this might work, and discussed the theoretical frameworks within which those operated [5, 14, 23, 49, 63, 90, 91]. Thus, Ducharme et al. [14] and Knudsen et al. [23] explored Rogers’ notion of the “trialability” [110] of innovations, that is, how far an innovation may be experimented with on just a limited basis, and Abraham et al. [5] discussed the role of absorptive capacity [111, 112], which they summarized as an organization’s ability to assess and use information [5]. Rieckmann et al. noted that although the mechanisms involved were not fully understood they appeared “to be influenced by core experiences from network participation” (p. 894) [91], and Fields et al. [63] used insights from implementation science to explore the influence of a set of organizational characteristics (including network membership) on innovation adoption [113].

In an analysis of data on 12 993 transplants conducted in 162 US centres, the 32 centres in the Bone Marrow Transplant trials network were found to have significantly better survival rates than others [77]. Marmor et al. reported that there was not an association between procedure volume and survival. Rather, they suggested, the better outcomes for those treated in centres in this network could be linked to the nature of trials that required “higher levels of national clinical collaboration and standardization of protocols”, and such collaboration was “likely to generate higher levels of innovation and excellence among clinical colleagues” (p. 92) [77].

In Germany, one team produced three papers on the improved healthcare performance of hospitals that were part of clinical trials organizations [13, 34, 94]. Two papers described the improved outcomes for patients with ovarian cancer if they were treated in a hospital that belonged to one of two German ovarian cancer clinical trials organizations, in effect research networks [13, 34]. They noted that the improved outcomes were not related to patient volume, suggesting instead that possible factors may include hospitals’ participation in the study group’s quality assurance programs and team members attending regular and scientific and educational meetings [13]. In a follow-up study, the data were analysed in more detail using mediation analysis that showed not just that the research participation of a hospital contributed to superior patient survival, but also began to unpick how it happened, including through better use of surgery and chemotherapy [94].

Downing et al. noted that, following the 2006 establishment of the NIHR in the United Kingdom, the increase in research activity in networks throughout the English NHS also increased the scope for analysing the benefits of research engagement [58]. The role of NIHR networks in boosting research engagement, which is then linked to improved healthcare, also covers clinicians such as nurses and AHPs who had traditionally had limited research opportunities. Studies are now showing how they can play an important role by engaging in research because, according to Trusson et al. reporting on a research network for nurses and AHPs, people working in such roles “have opportunities to explore possible solutions to issues that they encounter in their clinical role through academic study” (p. 1) [101]. Such opportunities can also enhance their clinical skills. More broadly, Downing et al. claimed that, in relation to the NIHR’s clinical trials network, “this natural experiment, presented by the rapid expansion of trial activity across a whole national health system, is perhaps the best opportunity to address the subject though outcomes research” (p. 95) [58]. This development is discussed in the next section.

The “dose effect” of the extent of research engagement

Evidence indicating a link between the extent of research engagement and the degree of improved healthcare has been accumulating for some time. In the United States, the 1996 study by Brown and Griffiss found that the average acute length of stay (LoS) in Department of Veteran Affairs hospitals was inversely related to the size of research programmes [53]. Majumdar et al. [26] used a tertile approach to show that in-hospital mortality decreased as the rate of trial participation increased in the area of unstable angina. In the substance abuse field, early CTN studies also contributed: thus Knudsen et al. [23] noted that the adoption of buprenorphine therapy by practitioners within the trials’ network was much greater in those programmes in the network that participated in the specific buprenorphine trial than those that had not. In a 2006 study of a sexual health trial in Australia, Morton et al. [28] identified improved post-trial clinical practice by high-recruiting clinicians, but not by low-recruiting ones.

In our combined set of papers the first use of the specific term “dose effect” to describe the effects of differing amounts of research engagement occurred in Downing et al., who tested the hypothesis that for colorectal cancer (CRC) “high, sustained hospital-level participation in interventional clinical trials improves outcomes for all patients with CRC managed in those research-intensive hospitals” (p. 89) [58]. They found that high participation in such clinical trials was independently associated with better outcomes and that these effects were not restricted to academic centres or large institutions but were seen across all the NHS Trusts that conducted research on and treated patients with colorectal cancer. They extended their analysis to look at the effects of different levels of research participation and found that the highest levels of participation led to the highest levels of improved outcomes. However, in relation to these findings, Downing et al. were careful to say that, in the absence of the possibility of an RCT, caution was needed if attempting “to infer a causal contribution” (p. 89) from participation in research activity to improved healthcare [58].

Other United Kingdom database studies support the findings of Downing et al. For example, Ozdemir et al. [85] compared mortality with research funding per hospital bed in hospitals with high, medium and low levels of research funding and showed that not only was mortality lower in high-funded research hospitals than in other hospitals, but also, on average, hospitals in the middle category had a lower mortality rate than ones with the least research funding. In two studies using NIHR research study activity data from different years, Jonker and Fisher [68, 69] showed an inverse correlation between the number of clinical trials/patient participation levels in United Kingdom hospitals and the mortality rate. Lin et al. [73] used retrospective data to examine the survival rate of the 465 patients (recruited by 60 hospitals) who had participated in an RCT in the NIHR Clinical Research Network (CRN). While they identified a significant association between low trial recruitment and lower survival rates, looking at the volume of patients treated in the disease area by the respective hospitals they report that “no significance was found between hospital throughput and outcomes” (p. 40) [73].

Further support for the “dose effect” concept comes from the United States and elsewhere. According to Abraham et al., in the substance abuse field “treatment programs participating in a greater number of CTN protocols had significantly higher levels of treatment quality, an association that held after controlling for key organizational characteristics” (p. 232) [49]. Similarly, Gilbert et al. [66] reported that members of a dental PBRN who fully participated in the network were more likely to move evidence-based care into everyday practice than members who only partially participated. Seaburg et al. [96] showed an association between the quantity of resident physicians’ publications and their clinical performance scores during training, and García-Romero et al. claimed that increases in the scientific output of Spanish hospitals made a significant contribution to a reduction of hospital LoS [65].

In Canada, Tsang et al. [103] conducted a pre-planned observational study nested within a clinical trial to test how well traditionally non-research active community hospitals could participate in an RCT alongside the traditional RCT sites in academic hospitals. However, while that aspect of the study did show that, in terms of adherence to trial metrics, the community hospitals could successfully participate in studies, outcomes for patients in the trial were significantly better in the traditional research hospitals, although the full reasons for this will need further exploration [103].

Health equity

Various aspects of health equity are considered in the included papers, and some of these report attempts to improve health equity. Some population groups are particularly vulnerable. In the United States, for example, Wolfson et al. listed the following groups: “underrepresented minorities, those with low socio-economic status (SES), those with public or no insurance, and those with a significant distance to care” (p. 3886) [106]. On the basis of its long-held assumption that patient access to research active healthcare providers is beneficial, the NCI has attempted to reduce geographic inequalities in access. In a 1995 paper, Warneke et al. noted that the CCOP was established by the NCI in 1983 with the deliberate intention of spreading the benefits of the clinical research conducted in NCI centres: “The program was designed with the assumption that by participating as equals in the research process, community physicians would be more likely to accept and implement the results in their practices with non-protocol patients” (p. 336) [37].

Similar moves to encourage wider participation in clinical trials have recently been made in Canada in the nested study described above [103]. A recent analysis showing higher levels of research activity within the English healthcare system were associated with lower mortality, noted that although the NIHR CRN was established to promote research participation across England, there was still some way to go to ensure greater geographical equity [69].

Other initiatives, such as the United States minority-based CCOPs described in the Glossary, addressed racial inequalities in relation to access to research engagement and timely evidence-based healthcare. These sometimes overlap with geographic inequalities. Some of the papers on the NCI-designated cancer centres observed with concern that the proportion of certain racial/ethnic groups, including African Americans, who received treatment at these centres compared with non-NCI centres, was lower than for other racial groups [64, 80, 106]. Having noted that African Americans with colon cancer experienced worse outcomes than Caucasian Americans, and suggested that this was partly due to differential treatment, a study by Penn et al. found evidence that African Americans receiving treatment from CCOP providers had benefitted from a seemingly deliberate attempt to boost early access to a recently recommended innovative treatment [87]. In Australia, Young et al. [109] reported that the health services, and health research system, of the Aboriginal community work together to try to ensure health research is embedded into activities that improve health, and described a specific example in relation to ear, nose and throat surgery and speech-language pathology services.

Lessons from the overall collection of studies: positive and negative

A wide variety of papers contribute to the combined review’s overall finding that the included studies are overwhelmingly positive. As the section on the “dose effect” illustrates, throughout the time covered by the combined review, individual papers have contributed to a wider understanding that goes beyond specific issues about research networks. Many papers contribute to the analysis of both the strength of the association between research engagement and improved healthcare, and the mechanisms involved. For example, a 2019 US positive study by Fanaroff et al. [60] identified improved care and outcomes for patients with acute myocardial infarction who were treated at research active hospitals, even after accounting for potential confounders. The authors encapsulated some of the key thinking on research engagement with their conclusion that participation in clinical trials by hospitals “may be emblematic of a culture that embraces novel therapeutics, engages both clinicians and patients, and incentivizes continuous improvement in care” (p. 191) [60].

While overall the 95 studies included in the combined review are positive, about 10% are categorized as negative. These nine negative papers also provide important insights [7, 11, 15, 20, 25, 67, 79, 81, 99]. For example, existing widespread use of one proven intervention prior to a company-sponsored clinical trial exploring physicians’ adherence to international treatment recommendations meant that the trial had no significant impact on that adherence, although it did increase use of the trial sponsor’s drug [7]; physicians adopted another trial intervention before it was proven one way or another [11]; more positively, a unique policy and regulatory environment governing the adoption of another intervention ensured that all hospitals benefitted, not just those in the trial [79]. Two teams with negative results later conducted further, more comprehensive studies with positive conclusions [25, 26, 67,68,69]. Six of the seven papers examining whether NCI-designated cancer centres provided patients with better healthcare processes and outcomes are positive [52, 64, 80, 84, 86, 106]. However, one paper suggested that outcomes were worse in these accredited hospitals despite the better healthcare and, in seeking to explain this, drew attention to the factors considered in the accreditation processes used by different organizations and how far they accurately captured the most relevant data [81].

Discussion

Our original review set out to find whether there was empirical evidence that supported the often-held assumption that engagement by clinicians and healthcare organizations in research improves healthcare performance at various levels. It concluded that there was some positive evidence but that systematic analysis of the data related to this engagement was in its infancy [2]. The 62 papers in the updated review, 58 of which are positive, provide further empirical evidence to support the positive conclusions of the original review.

When the papers from both reviews are considered together, they provide a more complete dataset than previously available [1,2,3,4], and an updated picture of this literature in which the trends identified in our initial analyses [3] become more apparent. With more than a third of the papers in the combined review (32/95) focussing on aspects of cancer, this is the field overall in which there is the most comprehensive analysis of the link between research engagement and improved healthcare. While the individual cancer papers differ in the strength of the association identified, and most of the papers focus one or other of the main cancer sites, many of the cancer papers analyse the role of research networks – one of the main mechanisms through which it is claimed research engagement improves healthcare.

The combined review reflects policy shifts and organizational changes that occurred first in the United States and later in the United Kingdom and elsewhere, and were designed to address the time lag between the production of research and its use in practice. These include the development of research networks and their associated databases over several decades (accompanied by an improved understanding of their strengths and limitations [54, 64, 77, 106, 108]) and efforts to strengthen links between academic centres and community services [61, 87]. More recent developments, especially in the United Kingdom, encouraged further deliberate attempts to identify and explore the impacts of research engagement. Research teams were, for example, better able to study the real-world impacts of system-level mechanisms such as research networks as they became more formalized and embedded in national health and science structures [58, 67,68,69, 85, 93, 101].

Across the board, within and beyond networks, there is also further evidence about the mechanisms by which research engagement might improve healthcare, including the ones identified in our original review. The role of strong evidence-based protocols developed for RCTs, but contributing to improved healthcare more widely in research active healthcare sites, was highlighted in various studies [77, 98, 105]. Papers also identified the importance of providing evidence-based/guideline consistent care, which could also be linked to a culture of discovery, excellence and collaboration [40, 60, 62, 64, 77, 84, 86, 87, 106]. There were also more nuanced mechanisms at the speciality and clinician levels, such as the use of multi-disciplinary coordination of care in radiation therapy treatment [107] and practitioner skill development in substance abuse work [90]. Similar practitioner skill development was also reported among nurses and AHPs, including in the wider literature [31, 38, 39, 62, 70, 101].

In the combined review it also became easier to see connections across this diverse literature. It was possible to identify research teams that had worked together on multiple studies and to explore the extent of cross referencing. In the United States, for example, the CTN of the drug abuse institute had been created to emulate the CCOP, and a centre was established to assess the CTN’s impact [114]. Analysis of this research network highlighted its role both in conducting research that was relevant to the “real-world” needs of clinical settings, and in enhancing evidence-adoption by healthcare organizations and staff [114]. Many of the papers from this substance abuse CTN [23, 49, 90] referenced each other and also cross-referenced key cancer papers [8, 24, 54], and there was common use of the same early sources [110, 112, 115, 116]. These interactions prompted ongoing methodological development, strengthened understanding of theoretical concepts, and supported shared learning across the specialities. Additionally, themes that had been recognized in the original review, including concepts such as absorptive capacity [5, 111], were further explored and tested in new contexts, even if the same literature was not always drawn upon [40, 65].

In the combined review, the nature and strength of the association found between research engagement and improved health varies enormously among the 86 positive papers, even among those that describe the role of research networks. One approach that begins to identify where evidence might be strongest was noted in the original review as being the important concept of the “dose effect”, even if it was not specifically labelled as such [26]. However, the combined review can now more fully consider the concept because evidence about this greatly increased as the scope of the papers included has increased. There are many more studies where all the clinicians or organizations compared are engaged in research but to varying extents and/or with different levels of resources, for example within a trial [28, 50, 59, 73, 93, 103, 107] or within a network [23, 33, 49, 51, 66, 68, 69, 85, 95]. The inclusion of papers regarding differences within trials, and the emergence of the importance of the “dose effect”, have implications for both (a) how the issue of research engagement is analysed and (b) how far efforts to enhance research engagement should be concentrated or spread widely across a system.

In relation to the first of these issues, when considering how research engagement is analysed, the key question morphs somewhat: it is no longer simply whether research engagement improves healthcare performance compared with no research engagement, rather, it is whether a larger amount of research engagement improves healthcare performance by more than a smaller level of engagement (and, if so, by how much). Answers to these questions could then feed back to strengthen the evidence for a positive association between research engagement and improved healthcare performance.

In relation to the second question, about the concentration or wide distribution of research funding, analyses might have to consider the context and trade-offs in terms of benefits for improved health and health equity. The widespread distribution of research funding across the health system could maximize the number of patients who might benefit, but a more concentrated approach, with a higher dose of research engagement in a smaller number of hospitals, could maximize the benefit for patients in such centres.

Research infrastructures in countries such as the United States and United Kingdom have been developed to enhance the relationship between health and health research systems, and the evidence from our combined review suggests that these changes have been positive. In both systems, but particularly in the United Kingdom, there have been deliberate attempts to fund major centres of research in leading healthcare facilities, as well as to spread research funding more widely to healthcare organizations across the country, but this impetus needs to be maintained if the full benefits of research engagement are to be realized.

Such an argument is reinforced by the conclusions of a major recent analysis of progress in the United Kingdom in engaging healthcare staff in research and building research capacity. The findings from the study suggest that many healthcare staff in the United Kingdom are interested in being involved in research, there are supportive national policies and strategies in place and there has been some important progress. However, achieving widespread involvement “will only be possible by focusing more on how healthcare organizations embed and support research activity through organizational policies which are supported by the wider research support and funding infrastructure. This is an essential part of a system-based approach to developing and supporting research engagement” (p. 356) [117]. The progress possible, and the potential benefits of trying to build a health research system embedded into a healthcare system, but also the full range of substantial challenges, have also recently been explored in a hospital and regional healthcare system in northern Queensland, Australia [118, 119]. Studies such as these indicate that this combined review could provide timely evidence to further the challenging task of improving healthcare by boosting engagement in health research.

Strengths and limitations

The combined review contains a considerable number of papers from diverse perspectives, but the literature is drawn predominantly from the United States and the Global North, thus the conclusions may not be appropriate in different contexts, including in the Global South. This, perhaps, partly reflects the inclusion criteria of papers in English only. While the increasing use of bibliometrics as an indicator of research engagement has widened the range of positive studies available, differing claims as to the most appropriate measure of research publications challenge consistent interpretation of the data and indicate there is more work to do. Furthermore, it is important to recognize that the national policy, noted in one paper, of attaching promotion and bonuses for clinicians to publish in journals with an impact factor of at least three [97] runs contrary to the internationally widely endorsed Declaration on Research Assessment [120].

The complexity of this literature (with many generic terms such as “research” and “engagement”), and the tangential approach of some papers to the broad question of whether research engagement improves performance, posed considerable challenges. It helped enormously that this time around, we were able to build on our experience in the original review. We adopted a somewhat more extensive approach to the formal search in the updated review, and we identified some papers that we had missed in the original review. We were aided by the generally greater clarity in later papers. We are now able, therefore, to present a more nuanced understanding of this field, building on our experience in the original review. In particular, we have found considerably more evidence on two topics identified as important in our original review, and on their implications for health equity: the role of research networks and consideration of how far there is a dose effect with regard to the degrees of research engagement. On both topics the combined review has strong papers showing important healthcare improvements even after considering potential confounders such as patient volume [8, 13, 26, 40, 58, 73, 77, 84,85,86]. However, the failure of some papers to address such confounders [59, 107] means some weaknesses in the overall analysis remain, and we are still not able to undertake any meta-analysis as the included literature remains very diverse.

We have now included a significant range of largely positive papers in the combined review. However, lack of resources meant we were not able to replicate our original review’s [2, 3] structured analysis of the wider range of papers identified as making many relevant and illuminating points related to the topic, but not meeting the review’s inclusion criteria. For example, while the combined review does include some consideration of health equity issues, there were papers taken to full paper review that were not in the end included but which provide considerably more evidence [121, 122].

Future possible work

The system-based approaches for expanding the amount of research in healthcare systems that are mentioned above continue to provide important opportunities for further work on exploring the relationship between research engagement and improved healthcare, including the implications for health equity. Likewise, improvements in the identification and collection of relevant data and developments in statistics have prompted increasingly sophisticated analyses, sometimes using approaches developed in other fields, and could continue to do so [65, 90, 94]. There has also been increasingly sophisticated use of bibliometrics, and there are likely to be continuing opportunities to apply such approaches to more countries. However, the warning from Downing et al. that caution is needed if attempting “to infer a causal contribution” from research participation to improved health outcomes [58], as well as frequent mention of similar disclaimers in other papers [8, 40, 54, 59, 69, 74], is a reminder that more work is needed.

While some of our papers have claimed that the costs of research engagement are broadly covered by the associated reduced LoS [53, 65], further research might be useful around the costs associated with research engagement and how these relate to reported benefits. Such studies could add to the existing large-scale studies showing the considerable monetary value of the health and economic gains resulting from health research [123].

The insights revealed by the negative papers, particularly in relation to the contexts in which research and research networks operate [99], could usefully be further explored. Merkow et al. [81], the one negative paper out of seven papers included on the NCI-designated centres, raises issues about the accuracy, or perhaps appropriateness, of the measurement used by various organizations to accredit cancer centres. These issues have also been explored by various teams [122, 124, 125] but could perhaps be worth further examination because the findings from Merkow et al. are so starkly different from those of other papers included in our review.

Finally, there are increasing opportunities, as well as a growing need, to address the limitations identified above (and also noted in the review by Chalmers et al. [39]) and go beyond the formal inclusion criteria of this review. A major area that could usefully be incorporated into an overall analysis of the field relates to the impact of the growing interest in research engagement strategies [126]. This includes the efforts to enhance research roles for healthcare professionals other than medical professionals [38], and the increasing number of organizational arrangements within health and health research systems for partnerships that seek to boost the production and use of relevant evidence [127, 128].

Conclusion

Previous reviews [1,2,3,4] have investigated the association between research engagement and improvements in healthcare performance. This study updates and extends the most comprehensive of these reviews [2, 3], and combines its findings with those from that original review to produce a more substantial pool of studies, which are largely positive in terms of the impact of research engagement on processes of care and patient outcomes. Of potential mechanisms, the combined review highlights the important role played by research networks and further identifies the various ways the research engagement facilitated by them operates to improve healthcare. The review also draws together a set of papers which consider how far there is a research engagement “dose effect”. Given the difficulty of conducting randomized controlled trials of large-scale research engagement initiatives, studies of the dose effect offer another approach to understanding the potential contribution and complexities of research engagement, including the implications for health equity. This review provides further evidence of the important contribution played by systems-level research investments such as research networks on processes of care and patient outcomes.