Table 2 Examples of primary studies assessing the impact of cancer research
Author(s) | Title | Year | Location | Description | Unit of analysis | Main reason for assessment | Categories assessed | Methods used | Framework | Main findings | Strengths | Limitations |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Brown ML, Nayfield SG, Shibley LM | Adjuvant therapy for stage III colon cancer: economic returns to research and cost-effectiveness of treatment | 1994 | USA | Evaluation of cost-effectiveness of adjuvant treatment for colorectal cancer and assessment of the return on investment to conduct clinical trial research | Cancer clinical trial | Accountability | Based on principles of economic evaluation. Economy, health | Cost effectiveness analysis (CEA) followed by evaluation of the social return on research investment. Human capital approach used to value return on investment | Nil | Cost effectiveness of treatment estimated at $2094 US dollars per year of life saved. The net present value of the return on the $10.84 million investment in the trial estimated at $1.66 billion dollars | The cost of conducting research incorporated into analysis of the value of the research investment. Time-period for assessment of the costs and benefits extended to 2020 to capture downstream effects | Assumptions made about adoption of trial results into practice rather than an assessment of actual practice change |
Ugolini D, Bogliolo A, Parodi S, Casilli C, Santi L | Assessing research productivity in an oncology research institute: the role of the documentation centre | 1997 | Italy | Estimation of the academic impact of departments within a cancer research institution | Cancer research centre | Allocation | Academic impact | Bibliometric assessment using an journal impact factor based metric | Nil | Most publications from the research centre that were identified and analysed scored highly (8–10/10) on the normalised journal impact factor score | An early (1997) attempt to evaluate the academic impact of a cancer institute | Metric, quantitative based approach only looking at a narrow interpretation of impact. Using the impact factor of a journal to assess the quality of individual research articles and for allocation of resources |
Ugolini D, Casilli C, Mela GS | Assessing oncological productivity: Is one method sufficient? | 2002 | Italy | Estimation of the contribution to academic cancer publications from different European countries using bibliometric methods | European countries | Analysis | Academic impact | Bibliometric method using the number of occurrences of an oncological publication in a journal (by author country of origin) compared to country population/GDP and the "mean impact factor" of the occurrences of the publication by author | Nil | The UK made the highest contribution to European cancer publication output (21.12%), whereas Sweden performed best in the metric of number of publication occurrences versus country population and the Netherlands was ranked first for the mean impact factor of the occurrences | Comparison of two methods to understand how to assess the impact of cancer researchers' work | Quantitative metric used that does not assess the quality of the individual articles or the contribution of the authors to the work. The authors acknowledge the limitations of using bibliometrics alone |
Coyle D, Grunfeld E, Wells G | The assessment of the economic return from controlled clinical trials | 2003 | Canada | Evaluation of the potential payback of conducting a clinical trial to test the optimal follow up for patients with colorectal cancer. Two trial designs were compared: an equivalence trial and an effectiveness trial (looking for a survival benefit from more intensive follow up) | Cancer clinical trial | Allocation | Economic and health impact (potential) | Assessment of “time to payback” for two alternative hypothetical cancer trials. This is the number of years until the returns from conducting a clinical trial outweigh the costs | Nil | An effectiveness clinical trial would be worthwhile to look for a 5% improvement in survival from more intensive follow up (if a 5% improvement is considered likely). An equivalence trial would not be a worthwhile investment | Ex-ante evaluation shows how impact assessment can prevent investment in cancer trials that are unlikely to be worthwhile | An evaluation of potential rather than realised impact. Any error in the assumptions made and results obtained in these types of analysis means that some potentially impactful trials will not be performed |
Lewison G, Sullivan R | The impact of cancer research: how publications influence UK cancer clinical guidelines | 2008 | UK | Evaluation of the cancer research that are cited in UK clinical guidelines | Any type of cancer research as cited in guideline | Analysis | Policy/guidelines | Identification of 43 UK guidelines from three guideline series (NICE, SIGN and Clinical Evidence) and bibliometric software used to analyse the guideline citations. Comparison of the citations versus those in the world oncology literature over 3-year period | Nil | UK papers were cited more frequently in cancer clinical guidelines than expected from their presence in the world oncology literature. The publications were generally more clinical than basic | Outlines and executes a method for evaluating the impact of cancer research on guidelines | The authors highlight that small clinical trials with negative outcomes are unlikely to be cited in guidelines, and discuss the limitations of using clinical guideline impact as a surrogate for practice change |
Lewison G, Tootell S, Roe P, Sullivan R | How do the media report cancer research? A study of the UK's BBC website | 2008 | UK | Evaluation of the impact of cancer research on the BBC news website (accessed by 13.2 million people annually in the UK (2008 figures)) | Any cancer research as cited in the media | Analysis | Media impact (dissemination) | Search of BBC archive health Section 1998–2006. Percentage of BBC stories focusing on different cancer sites were compared with the UK's burden of disease (WHO 2002). Research level (basic versus clinical) of cited papers was determined. The potential citation rate (PCI), actual citation rate (ACI) and funding of any cited paper compared to the global oncology research papers | Nil | Research on breast, cervical and skin cancer are over-reported in the media compared to their burden of disease whereas lung cancer is under-reported. New and improved drugs are the research topic most cited. UK research was over-cited in the UK media compared to its place in world oncology research | Novel attempt to investigate the impact of research in the media. Methodology clearly explained | Only one media website archive used for analysis |
Saad et al. | The geography of clinical cancer research: analysis of abstracts presented at the American Society of Clinical Oncology Annual Meetings | 2009 | Brazil | Evaluation of cancer research disseminated at an international conference | Individual research projects presented in abstract form at a conference | Analysis | Academic impact and dissemination | Bibliometric analysis of a sample of abstracts | Over 50% of abstracts were from the USA. Clinical trials were more likely than "other" types of research to be presented in poster or oral form (vs publication only) | Large, international cancer conference and several years of submissions chosen for the analysis | Analysed only a sample (10%) of all abstracts, no comparison between the two time periods sampled performed and no information on the abstracts that were submitted but rejected | |
Lewison G, Markusova V | The evaluation of Russian cancer research | 2010 | UK | Evaluation of the extent of cancer research in Russia, if research meets the needs of the country, and how the impact of research compares on a world scale | Cancer research performed by one country | Analysis | Academic/scholarly impact | Quantification of number of cancer publications compared to the national wealth and disease burden of cancer in Russia compared to other countries. Citation scores for Russian cancer publications compared with the citations to cancer papers worldwide in the same years. Analysis of collaboration on Russian cancer research papers | Nil | Russia publishes one sixth as many cancer papers as its wealth and disease burden would suggest. Clinical cancer research papers receive more citation than basic ones (the reverse of what is seen elsewhere). Russian cancer research scores well for the percentage of reviews and collaboration in cancer research is dominated by former socialist states. Russian cancer research is incorporated into UK clinical guidelines, but rarely into UK media | Multiple indicators used to map some important impacts of cancer research | Mainly quantitative indicators used. No assessment of broader aspects of societal impact. Only looked at the impact on UK clinical guidelines and media rather that in Russia (likely to be much higher) |
Lakdawalla DN, Sun EC, Anupam BJ, Reyes CM, Goldman DP, Philipson TJ | An economic evaluation of the war on cancer | 2010 | USA | Evaluation of the returns on investment from the USA government into cancer research from 1971 | National cancer research | Accountability | Economic impact from a societal perspective | Quantification of gains in cancer survival using willingness to pay estimates derived from the literature and a comparison with the cost in cancer spending | Nil | Cancer survival increased by 3.9 years from 1988–2000 which equates to 23 million additional life years. The authors estimate that an average life year is worth 82,000 dollars and therefore the value of this survival is monetised at 1.9 trillion dollars in social value. Uses an 18-year time lag from research investment to survival outcome attributable to that research | Attempts to monetise a complex concept of willingness to pay for cancer survival. Detailed description of analysis | Complex methods, willingness to pay values from previous literature. The actual investment costs in different cancer types and different research types (basic versus trials) not outlined |
Montague S, Valentim R | Evaluation of RT&D: from "prescriptions for justifying" to "user-orientated guidance for learning" | 2010 | Canada | Evaluation of the impact of a cancer RCT (MA17), which reported that an aromatase inhibitor called letrozole reduced rates of disease recurrence when given to patients after radical treatment for breast cancer | One cancer trial | Analysis | Multiple types of impact (based on CAHS framework) | Canadian Academy of Health Sciences Framework (CAHS) used to guide impact evaluation. Indicators of impact from CAHS framework combined with Bennett’s hierarchy | CAHS model, Bennett hierarchy | Impacts in all categories of the CAHS can be identified for this trial. The authors produce an impact timeline to show the pathway to impact in chronological order | Use of a conceptual framework means that the impact of this cancer trial is communicated in a transparent and organised fashion. The use of the hierarchy of events helps to demonstrate the processes through which impact has occurred | The theory of action hierarchy makes assumptions that prior events influence events higher up the chain |
Sullivan R, Lewison G, Purushotham AD | An analysis of research activity in major UK cancer centres | 2011 | UK | Evaluation of the impact of research from UK major cancer centres on knowledge production (citation impact), clinical management, and the general-public (via media citation) | Research from cancer centres in the UK | Analysis | Academic/scholarly, clinical management, general public/media | Academic papers linked with cancer centres in the UK (via location or author) identified via database review. The journals of the papers were categorised by research level (basic versus clinical). Potential citation impact (PCI) and actual citation impact (ACI) calculated for the included papers and any international collaboration on the papers documented. Calculation of the number of papers from UK cancer centres that are reported in UK guidelines and in the media | Nil | UK cancer centres are heterogeneous in terms of their overall research output and the type of research performed. Overall, there was more focus on basic/fundamental research being published by UK cancer centres compared to applied research. There is heterogeneity in the proportion of papers from UK cancer centres that are cited in guidelines and the media that does not correlate with the size of the centre or the conventional citation impact of the papers | The authors combine publication citation impact with other impacts (on guidelines and media) to give a broader overview of cancer centre research impact | Mainly quantitative measures of impact. The authors mention the potential future use of the Research Impact Framework but they do not use this or any other framework in their current analysis |
Donovan C, Butler L, Butt AJ, Jones TH, Hanney SR | Evaluation of the impact of National Breast Cancer Foundation-funded research | 2014 | UK | Evaluation of the academic and wider impacts resulting from funding by a specific cancer charity in Australia (National Breast Cancer Foundation NBCF) | Programme of cancer research projects | Accountability | Multiple (based on Payback framework) | Desk analysis of data held by the charity, survey of chief investigators, 16 case studies, bibliometrics and international benchmarking using bibliometrics. The case studies used document and archival analysis, citation analysis, searching for citations in guidelines and interviews with principal investigators of projects | Payback | 153 responses to a survey sent to 242 NBCF-funded researchers showed that the research they performed had impacts on drug development, higher degree attainment, capacity building for future research, policy, and health gain. Findings showed differences in impact between basic and applied cancer research, for example, basic research was more likely to lead to product development whereas applied research was more likely to impact on policy, behaviour, and practice | Transparent methodology using a recognised framework and a mixture of quantitative and qualitative methods to gain in depth insight into impact of the research | Impact from the perspective of the researchers, no other stakeholders approached to comment on perceived impact of the funded research |
Glover M, Buxton M, Guthrie S, Hanney S, Pollitt A, Grant J | Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes | 2014 | UK | Evaluation of the economic value of cancer research in the UK | National cancer research | Accountability | Economic | Economic analysis (bottom up approach used). Five main steps: Estimated spending on research between 1970–2009 (£15 billion), estimated the net monetary benefit using a monetized value for a QALY, estimated the cost of delivering the benefit by identifying a list of the most important interventions that had contributed to this benefit (via discussions with experts), estimated the proportion of NMB attributable to UK research and the time lapse between funding and health gain (using bibliometric analysis of clinical guidelines) and finally, the internal rate of return from cancer research on health benefits | Nil | Time lag between research spending and impact on health gain estimated as 15 years. Overall return on public spending on cancer research estimated as 10.1% | Considered the time lag between research spending and impact on health gain. Sensitivity analysis performed where possible and acknowledged areas of uncertainty. Acknowledged the purely quantitative nature of this assessment and accompanying case studies were performed (Guthrie et al.) | Multiple assumptions made to perform this economic analysis (recognised and outlined by the authors). The authors outline the difficulty in differentiating the impact of smoking cessation in their calculation of the impact of cancer research overall |
Guthrie S, Pollitt A, Hanney S, Grant J | Investigating time lags and attribution in the translation of cancer research. A case study approach | 2015 | UK | Evaluation of the attribution of health gains due to investment in cancer research and the time lags between research investment and health gain being recognised using a case study approach | Six examples of cancer research topics which included clinical trials | Accountability | Health gain | Case study approach using mainly desk/documentary analysis | Nil | Individual narratives for each case study | The use of case studies enables the reader to understand the process through which impact has occurred and the time lines involved | Requires in depth documentary analysis to contextualise and explain case study specific impacts |