- Open Access
Impact of NIHR HTA Programme funded research on NICE clinical guidelines: a retrospective cohort
Health Research Policy and Systems volume 13, Article number: 37 (2015)
It is vitally important that there is a connection between health research and clinical practice. Indications as to the impact of the research on evidence-based practice and policy can be obtained by tracking the use of outputs of health research, especially its use in clinical guidelines (CGs). This study aims to assess the proportion of National Institute for Health and Care Excellence (NICE) CGs citing National Institute for Health Research Health Technology Assessment (NIHR HTA) studies and the impact of evidence from those studies on the included NICE CGs.
This is a retrospective cohort study assessing the proportion of NICE CGs from all NICE CGs issued between April 2001 and April 2012, which cited evidence from studies funded by the NIHR HTA Programme and the impact of those studies on the CGs as the primary and secondary outcome measures.
Of the cohort of NICE CGs (n = 122), 3 (2%) CGs were based on previous NIHR HTA reports and would not have been issued in that form without those NIHR HTA studies, 90 (74%) included evidence from NIHR HTA studies, and 29 (24%) did not include evidence from NIHR HTA studies. The impact of NIHR HTA evidence on NICE CGs varied in the type and quantity of data used.
Findings suggest that NIHR HTA funded research impacts on clinical guidance from NICE and hence is well connected to both clinical practice and policy.
It is vitally important that there is a connection between health research and clinical practice [1‐3]. It is also of great importance to research funders to assess the impact of the research they fund on clinical practice and policy. The National Institute for Health and Care Excellence  (NICE) publishes several types of publication , including clinical guidelines (CGs) which recommend appropriate treatment and care for people with specific conditions or diseases. The organisation has developed an international reputation, its guidance frequently being consulted (as judged by high numbers of visits to the website) or adopted by those in other countries . The recommendations in NICE CGs are prepared by groups of experts (clinicians, lay members, and patients) who examine the available evidence on particular diseases or clinical areas. The evidence used may include systematic reviews, technology assessment reports, and findings from clinical trials, and is moderated by the clinical and patient experience of members of the guideline groups. Use of this evidence exemplifies the concept of evidence-based practice and aims to encourage practitioners to employ similar high standards of care .
The National Institute for Health Research Health Technology Assessment (NIHR HTA) Programme  produces independent research information about the effectiveness, costs, and broader impact of healthcare treatments and tests, for those who plan, provide, or receive care in the National Health Service (NHS). As part of the NIHR journals library , each NIHR HTA funded project publishes an open access report in the NIHR journal Health Technology Assessment . This journal is peer reviewed, indexed on Medline, and is freely available online via the NIHR journals library website . Additionally, researchers funded by NIHR HTA are also strongly encouraged to disseminate their findings through other peer reviewed journals  and non-academic routes aimed at both professional communities and the general public. Research funded by the NIHR HTA programme includes primary research, systematic reviews, evidence syntheses, and technology assessment reports, and acts as a bridge between evidence and policymaking, providing NICE with accessible and evidence-based information to guide their decisions. Conversely, the NIHR HTA programme uses knowledge gaps identified by NICE to inform areas of future research. It should be noted, however, that the two organisations are separate and each operates independently of the other.
It is extremely useful and important to track the output of health research [11,12], especially its use in clinical guidelines [13‐15]. There has been much debate about how the impact of research may be defined or measured [16,17]. Citation in clinical guidelines can provide a broad indication that the research has been included and therefore has made a contribution, and so may be considered as having had impact on policy and evidence-based practice . It is widely recognised that an important element of impact of research is influence on policymakers and clinical guidelines , and it is important for research funders to know the degree to which their funded research is used in order to help shape future funding strategies.
A study undertaken by Alderson and Tan in 2011  investigated the number of Cochrane Reviews cited in NICE CGs and found that 81% of NICE CGs cited these, but that not all Cochrane Review Groups were represented in the reviews cited. Similarly, this study seeks to investigate the number of NICE CGs citing NIHR HTA funded studies and to assess the impact of those studies on NICE CGs.
We identified all NICE CGs issued between April 2001 and April 2012 through an online search of the NICE website . We excluded all CGs which had been withdrawn or superseded by subsequent CGs. The reference section for each included NICE CG (including appendices if they contained reference sections) was independently searched by two reviewers (ST and SB) to identify NIHR HTA research cited. Search terms used to identify NIHR HTA research included ‘HTA’, ‘Health Technol Assess’, ‘Health Technology Assessment’, ‘NCCHTA’, and ‘technology appraisal’. We defined NIHR HTA research outputs as publications in the NIHR HTA journal or other peer reviewed publications arising from projects funded by the NIHR HTA. Investigators funded by NIHR HTA are required to inform the funders of all output from their projects, including articles in peer reviewed journals (other than publications in the NIHR journals library). Notifications of these publications are recorded in an internal database. We identified peer reviewed journal publications arising from NIHR HTA funded studies by searching this internal database and matched these titles to citations in the CGs. However, not all investigators inform the NIHR HTA when they publish peer reviewed articles, so we also searched the reference sections of the CGs using author names and key words from relevant NIHR journals, and checked articles identified for information as to which organisation had funded the study, recording those funded by NIHR HTA. We then calculated the percentage of NICE CGs citing NIHR HTA funded studies.
In the second phase of this study we classified the cohort of NICE CGs according to the extent to which they were influenced by NIHR HTA studies using the criteria described in Table 1. Category 1 included CGs which were based on a previous NIHR HTA journal publication(s) and could not have been issued in the current form in absence of evidence from the NIHR HTA study(s). Category 2 included CGs which either reviewed NIHR HTA study(s) in detail, reproduced evidence table(s) from NIHR HTA study(s), or had one or more sections drawing evidence from NIHR HTA study(s). Examples which clarify the categorisations are given in the results section. Category 3 included all the CGs where no NIHR HTA study(s) were cited.
Assessment of the level of impact was performed independently by two researchers (ST and SB). Where there was disagreement, consensus was reached in consultation with a third researcher (AC). The agreement between the two researchers was assessed using a Cohen’s Kappa score.
We identified a total of 122 NICE CGs issued between April 2001 and April 2012. Of these, 93 (76%) had referenced NIHR HTA funded studies (Table 2). We found inconsistency in the referencing style of the NIHR HTA journal publications making it difficult to accurately record all NIHR HTA journal publications cited. In order to identify peer reviewed articles cited in CGs we used an internal Evaluation, Trials and Studies Coordinating Centre (NETSCC) database which relies on the researchers informing the NIHR HTA programme when their articles are accepted for publication. We know that this does not always happen, so it is very likely that is an underestimation of the number of studies cited.
The total number of citations identified was 321. Of these, 284 were publications from the NIHR HTA journal series and 37 were publications in other peer reviewed journals, of which 5 were not in the NETSCC database and were identified by searching authors’ names and keywords. We were unable to identify any citations in 29 out of the 122 CGs. Figure 1 shows the numbers of citations of publications in the NIHR HTA journal and other publications in the CGs within the cohort.
We assessed the impact on NICE CGs of NIHR HTA evidence and found that, of the 122 CGs, 3 (2%; CGs 7, 19, and 29) were included in category 1; 90 (74%) in category 2; and 29 (24%) in category 3 (Table 3). Agreement between the two researchers assessing impact was high (Kappa score = 0.980; weighted Kappa score = 0.980). Between the two researchers there was only disagreement over one CG, which was referred to the third researcher, and was then placed in category 2.
In category 1, the CG was largely based on a previously published NIHR HTA journal publication or publications. For example, text in CG7 states that “In April 2001, an HTA review was published on pressure-relieving devices for the prevention and treatment of pressure ulcers” . This review updated the earlier Cochrane systematic review, “Beds, mattresses and cushions for pressure sore prevention and treatment”. For the purposes of this guideline, the HTA review by Cullum et al.  was then updated by the Cochrane Wounds Group and National Collaborating Centre for Nursing and Supportive Care staff to provide the most up-to-date and rigorous source of clinical effectiveness evidence. For CG19, the methods used included “Systematic review of the literature – to ‘update’ the previous Health Technology Assessment review on the clinical effectiveness and cost-effectiveness of routine dental checks” .
Within category 2, the impact of NIHR HTA evidence on NICE CGs varied in the type and quantity of data used. The guideline could include a large volume of text about a particular study or studies; one or more tables of data; or contribute to meta-analysis. The evidence included may influence one particular section of the CG or relate to the whole CG. For example, in CG 95  (Chest Pain of Recent Onset) the chapter on people presenting with acute chest pain was heavily influenced by NIHR HTA studies, tables of data from the original NIHR HTA sources were presented in the CG. In other cases, such as CG 137  (The epilepsies), evidence from the SANAD trial  was used extensively throughout the CG, similarly in CG 108  (Chronic Heart Failure) a systematic review funded by the NIHR HTA programme  provided data which had substantial impact on the CG as a whole. Initially we had planned to achieve a greater level of granularity, breaking down the data on impact into more categories; however, we found the ways in which the data was used by NICE was so heterogeneous (use of text, tables, trial data, meta-analyses in varying amounts and combinations) that this couldn’t be done. We therefore decided to include only 3 categories as described in Table 1, amalgamating the heterogeneous data into category 2.
This study assessed the percentage of NICE CGs informed by NIHR HTA studies and the impact of evidence from those studies on the included NICE CGs. The results show that NIHR HTA funded studies were referenced in 76% of NICE CGs and had an impact on 76% of the 122 NICE CGs. The study suggests that NIHR HTA research has a considerable contribution towards NICE CGs, and hence is well connected to both clinical practice and policy. Demonstrating or quantifying impact of health research is a complex issue  and not all the factors involved are readily quantifiable ; however, citation of health research studies in clinical guidelines has previously been used as an indicator of impact . In this study, we found great heterogeneity in the way NIHR HTA evidence is used by NICE in CGs. Trying to quantify the impact of the different scenarios is both challenging and subjective.
It might be asked why NIHR HTA studies are not referenced in all CGs. It would be surprising, however, if there were a complete overlap in these portfolios as the remits of the NIHR HTA Programme and NICE are different. Both organisations were set up to support the NHS and patients in clinical decision-making – either by identifying important tractable research questions and hence filling evidence gaps (the NIHR HTA Programme) or by identifying areas of clinical uncertainty or inconsistency, and producing guidelines based on (among other things) available evidence to aid clinical decision-making (NICE). NICE commonly produces guidelines in areas where clinical practice is variable but the evidence base is clear. Once the evidence base has been established, the NIHR HTA Programme, by comparison, would not have so much to contribute in an area such as this. The NIHR HTA Programme will commission research in clinical areas which NICE may consider too small to devote the resources required to produce a guideline, but where there is clinical uncertainty resulting in possible sub-optimal patient care. A similar study conducted by Alderson and Tan  found that some Cochrane Review Groups did not have any Cochrane reviews cited by NICE CGs. It should also be remembered that the NICE guideline programme is not the NIHR HTA Programme’s only customer. NIHR HTA commissions work informed by research suggestions, and direct requests from other areas of NICE (such as the Interventional Procedures Advisory Committee) and other national bodies (such as the National Screening Committee).
The strengths of this study include its consideration of data over a long period of time, of more than 10 years, and its comparison of data from a major UK research funder with data from an organisation with an international reputation responsible for issuing national guidelines. Thus, this study gives an insight into the impact on clinical practice and policy, of the research which has been funded by the NIHR HTA Programme.
The limitations of this study include the time-limited nature of CGs, given that this is a snapshot view and some of the cohort considered herein have already been superseded, as well as the problem of accurately identifying all NIHR HTA sources used by NICE. NIHR HTA journal publications were not always cited in a consistent referencing style so we may have failed to identify some of these in the searches of the reference lists of CGs. Further, also we know that the data from our internal database used to identify other peer reviewed publications of NIHR HTA studies is incomplete. Additionally, this study does not include NIHR HTA studies which have been published after the relevant CG was issued; therefore, the contribution of NIHR HTA funded research to relevant clinical areas will have been underestimated.
This discussion of the interaction between the NIHR HTA Programme and NICE’s guidelines has, however, neglected the final common pathway. The NIHR HTA Programme strives to provide the NHS with the best possible evidence. NICE strives to provide it with the best possible guidance. We can see from this study that the interactions of these processes seem to work. What is missing is information on how new evidence or guidance [1,2,15,27,28] impacts on patient care and what factors influence the implementation of CGs [29‐32]. These relationships are not nearly so well understood and could be rich areas for future research. Additionally, the information from NICE has enabled us to assess how many publications were consulted in the preparation of the CGs; what we do not know is how this information was used in the guideline development process and what it was about the research that was most valued. It would be most informative to conduct qualitative research investigating the process of evidence use in guideline development.
It is vitally important that there is a connection between health research and clinical practice and it is important to track the output of health research, especially its use in clinical guidelines. More than 75% of NICE CGs issued between April 2001 and April 2012 utilised evidence from NIHR HTA funded research. Findings suggest that evidence from NIHR HTA funded research impacts on NICE CGs and hence is well connected to both clinical practice and policy.
Health technology assessment
Evaluation, Trials and Studies Coordinating Centre
National Health Service
National Institute for Health and Care Excellence
National Institute for Health Research
Boaz A, Baeza J, Fraser A. Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4:212.
Holmes B, Scarrow G, Schellenberg M. Translating evidence into practice: the role of health research funders. Implement Sci. 2012;7:39. doi:10.1186/1748-5908-7-39.
Lewison G, Sullivan R. The impact of cancer research: how publications influence UK cancer clinical guidelines. Br J Cancer. 2008;98:1944–50.
National Institute for Health and Care Excellence. http://www.nice.org.uk/. Accessed February 2014.
Rawlins MD. The decade of NICE. Lancet. 2013;374:351–2.
Rawlins MD. NICE work – providing guidance to the British National Health Service. N Engl J Med. 2004;351:1383–5.
NIHR Health Technology Assessment Programme. http://www.nets.nihr.ac.uk/programmes/hta. Accessed February 2014.
NIHR Journals Library. http://www.journalslibrary.nihr.ac.uk/hta. Accessed July 2015.
Turner S, Wright D, Maeso R, Cook A, Milne R. Publication rate for funded studies from a major UK health research funder: a cohort study. BMJ Open. 2013;3:e002521.
Wright D, Young A, Iserman E, Maeso R, Turner S, Haynes RB, et al. The clinical relevance and newsworthiness of NIHR HTA-funded research: a cohort study. BMJ Open. 2014;4, e004556.
Hanney S, Frame I, Grant J, Buxton M, Young T, Lewison G. Using categorisations of citations when assessing the outcomes from health research. Scientometrics. 2005;65:357–79.
Jones TH, Donovan C, Hanney S. Tracing the wider impacts of biomedical research: a literature search to develop a novel citation categorisation technique. Scientometrics. 2012;93:125–34.
Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings: the Cochrane Effective Practice and Organization of Care Review Group. BMJ. 1998;317:468.
Grant J, Cottrell R, Cluzeau F, Fawcett G. Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study. BMJ. 2000;320:1107–11.
Kryl D, Allen L, Dolby K, Sherbon B, Viney I. Tracking the impact of research on policy and practice: investigating the feasibility of using citations in clinical guidelines for research evaluation. BMJ Open. 2012;2, e000897.
Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc. 2010;98:17–23.
Moscone F, Tosetti E, Costantini M, Ali M. The impact of scientific research on health care: evidence from the OECD countries. Econ Model. 2013;32:325–32.
Alderson P, Tan T. Use of Cochrane Reviews in NICE clinical guidelines. Cochrane Database Syst Rev. 2011;12, ED000032.
Callum N, Nelson EA, Fleming K, Sheldon T. Systematic reviews of wound care management: (5) beds; (6) compression; (7) laser therapy, therapeutic ultrasound, electrotherapy and electromagnetic therapy. Health Technol Assess. 2001;5:1–221.
Davenport C, Elley K, Salas C, Taylor-Weetman CL, Fry-Smith A, Bryan S, et al. The clinical effectiveness and cost-effectiveness of routine dental checks: a systematic review and economic evaluation. Health Technol Assess. 2014;7:1–127.
NICE. Chest pain of recent onset: Assessment and diagnosis of recent onset chest pain or discomfort of suspected cardiac origin. NICE clinical guideline 95. Available at www.nice.org.uk/guidance/cg95. 2010.
The epilepsies: the diagnosis and management of the epilepsies in adults and children in primary and secondary care. NICE clinical guideline 137. Available at www.nice.org.uk/guidance/cg137. 2012.
Marson AG, Appleton R, Baker GA, Chadwick DW, Doughty J, Eaton B, et al. A randomised controlled trial examining the longer-term outcomes of standard versus new antiepileptic drugs The SANAD trial. Health Technol Assess. 2007;11:1–134.
Chronic heart failure: Management of chronic heart failure in adults in primary and secondary care. NICE clinical guideline 108. Available at www.nice.org.uk/guidance/cg108. 2010.
Mant J, Doust JA, Roalfe AK, Barton P, Cowie MR, Glasziou P, et al. Systematic Review and Individual Patient Data meta-Analysis of Diagnosis of Heart Failure, with Modelling of Implications of Different Diagnostic strategies in Primary Care. Health Technology Assessment 2009, 13.
Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst. 2015;13:18. doi:10.1186/s12961-015-0003-1.
Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it”. J Am Board Fam Med. 2005;18:541–5.
Dougherty M. Clinical guidance. Where NICE leads, can commissioners follow? Health Serv J. 2008;21:16–7.
Francke AL, Smit MC, de Veer AJ, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;12:38. doi:10.1186/1472-6947-8-38.
Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30.
Ploeg J, Davies B, Edwards N, Gifford W, Miller PE. Factors influencing best-practice guideline implementation: lessons learned from administrators, nursing staff, and project leaders. Worldviews Evid Based Nurs. 2007;4:210–9.
Cotter D, Thamer M, Narasimhan K, Zhang Y, Bullock K. Translating epoetin research into practice: the role of government and the use of scientific evidence. Health Aff. 2013;25:1249–59.
This research was supported by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) through its Research on Research programme. The views and opinions expressed are those of the authors and do not necessarily reflect those of the Department of Health, or of NETSCC.
This study did not require study ethics approval; it did not involve patients or clinical data.
The authors have no competing financial interests; however, all of the authors are employed by the University of Southampton to work at least part time for NETSCC and NETSCC’s reputation rests substantially on having managed the HTA Programme for the Department of Health for over 15 years. AC has worked for the HTA Programme since 2005; ST worked for the HTA Programme from 2005 to 2008.
The study was conceived and designed by SB, AC, and ST, and undertaken by SB and ST with advice from AC; ST led the writing guided by AC and SB. All authors read and approved the final manuscript.
About this article
Cite this article
Turner, S., Bhurke, S. & Cook, A. Impact of NIHR HTA Programme funded research on NICE clinical guidelines: a retrospective cohort. Health Res Policy Sys 13, 37 (2015). https://doi.org/10.1186/s12961-015-0025-8
- Health Technology Assessment
- Clinical guidelines