How do we define the policy impact of public health research? A systematic review
Health Research Policy and Systems volume 15, Article number: 84 (2017)
In order to understand and measure the policy impact of research we need a definition of research impact that is suited to the task. This article systematically reviewed both peer-reviewed and grey literature for definitions of research impact to develop a definition of research impact that can be used to investigate how public health research influences policy.
Keyword searches of the electronic databases Web of Science, ProQuest, PubMed, EMBASE, CINAHL, Informit, PsycINFO, The Cochrane Database of Systematic Reviews and Google Scholar were conducted between August 2015 and April 2016. Keywords included ‘definition’ and ‘policy’ and ‘research impact’ or ‘research evidence’. The search terms ‘health’, public health’ or ‘mental health’ and ‘knowledge transfer’ or ‘research translation’ were used to focus the search on relevant health discipline approaches. Studies included in the review described processes, theories or frameworks associated with public health, health services or mental health policy.
We identified 108 definitions in 83 publications. The key findings were that literature on research impact is growing, but only 23% of peer-reviewed publications on the topic explicitly defined the term and that the majority (76%) of definitions were derived from research organisations and funding institutions. We identified four main types of definition, namely (1) definitions that conceptualise research impacts in terms of positive changes or effects that evidence can bring about when transferred into policies (example Research Excellence Framework definition), (2) definitions that interpret research impacts as measurable outcomes (Research Councils UK), and (3) bibliometric and (4) use-based definitions. We identified four constructs underpinning these definitions that related to concepts of contribution, change, avenues and levels of impact.
The dominance of bureaucratic definitions, the tendency to discuss but not define the concept of research impact, and the heterogeneity of definitions confirm the need for conceptual clarity in this area. We propose a working definition of research impact that can be used in a range of health policy contexts.
The measurement of research impact is a contested research and political agenda that poses a complex academic question. Differences in the ways in which evidence might inform policy, and its political underpinnings, highlight key challenges in understanding the policy impacts of research.
The quest to measure and investigate research impact has multiple drivers. Researchers, practitioners and policymakers continue to promote the need for, and benefits of, evidence-informed practice and policies in public health and medicine more generally [1,2,3]. Government and other funders of research increasingly demand that researchers track the impact of their research to justify research expenditure by showing economic benefits, policy uptake, improved health and community outcomes, industry application and/or positive environmental effects [2, 4,5,6]. Accountability for research impact is typically embedded in the requirements of grant applications and project reports, in which researchers are required to anticipate the measurable outcomes arising from their proposed research [5, 6]. Within health research, there is an expectation that evidence-based policies and practices may improve the efficiency and effectiveness of health services [2, 6].
Definitions of research impacts are informed (explicitly or implicitly) by what we think knowledge is, what we value about it, and our understanding of the ways in which research evidence can contribute to policy . There is a lack of consensus on how to define research impact [8, 9] and on the terminology used to describe it . Various terms associated with the concept of ‘research impact’, such as knowledge or research ‘translation’, ‘uptake’, ‘diffusion’, ‘utilisation’, ‘payback’, ‘valorisation’, ‘benefits’ and ‘outcomes’, are often used interchangeably [11,12,13]. Boaz et al. suggest that the “different terms have a shared interest in change that lies beyond the research process and its primary outputs” (, p. 256), as well as an “appreciation of the complexity and diversity of research use” (, p. 266). In contrast, other authors note that the lack of standard terminology reflects a deficiency in the literature and call for a clear definition of research impact [14,15,16].
Many have argued that using a ‘narrow’ approach to measuring research impact results in acknowledging only those types of impact that are easily measured and overlooking those that are hard to measure [2, 17, 18], for example, narrowly defining research impact through the number of citations in the literature (see for example ). In contrast, Milat et al. argue that “The emerging literature on research impact highlights its complex, non-linear, unpredictable nature, and the propensity, to date, to count what can be easily measured, rather than measuring what ‘counts’ in terms of significant, enduring changes” (, p. 2). Greenhalgh and Fahy argue that “the unenhanced ‘logic model’ of impact, comprising inputs (research funding)→activities (research)→outputs (e.g. papers, guidelines)→outcomes (e.g. changed clinician behaviour, new service models)→impacts (e.g. reduced mortality), is increasingly viewed as over simplistic” (, p. 3). Similarly, Haynes et al. contend that research impact constitutes “a contested bundle of concepts subject to interpretation and tactical use” (, p. 1047) (see also [22,23,24]).
This study explores different definitions of research impact, with a specific focus on the applicability of definitions to advancing an academic understanding of how evidence informs health policy. The article presents the findings of a systematic review of the literature to assess how research impact is currently defined in the health literature. On the basis of this review, we propose a working definition of research impact relevant for health policy.
This review uses the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodological review framework  to guide a systematic data collection and critique of the literature defining research impact. The review was conducted to answer three research questions. (1) How is research impact defined in the health literature? (2) What are some of the key constructs underpinning different definitions of research impact? (3) What are some of the implications for research, policy and theory of different ways of defining research impact in the health policy field? The review used mental health research and policy as a case study.
The research questions resulted from discussions among all authors. The search strategies were developed with input from all authors supported by the expertise of a specialist librarian. KA conducted the database searches, assessed the literature against the review criteria, and undertook data extraction, synthesis and analysis of the literature. All authors provided input into findings and conclusions and edited drafts of the article.
An initial scoping review was undertaken to determine the feasibility of research questions and keywords for the search strategy. The initial focus on mental health policy was expanded to include health policy broadly in the absence of a research impact literature specific to mental health. Heterogeneity of concepts in the research impact literature and the challenge in finding studies that explicitly defined research impact motivated a more expansive search strategy and broader criteria for a definition of ‘research impact’. The peer-reviewed literature was found to be too limited since the majority of definitions were generated within government, not academic, contexts. As Sibbald et al. argue, “the exclusion of grey literature can skew the results of research syntheses” (, p. 49). The review criteria were expanded to include definitions of impact in all health research found in the grey literature. Grey literature was sourced through (1) Google Scholar, (2) examining reference lists in the included articles, (3) consulting with members of the research team, and (4) contacting external experts.
Electronic databases were searched for peer-reviewed article abstracts and other literature that were evaluated against the review aims and scope. The databases Web of Science, ProQuest, PubMed, EMBASE, CINAHL, Informit, PsycINFO, The Cochrane Database of Systematic Reviews and Google Scholar were searched using a desktop research method tailored to individual databases. Filters were applied to include only articles written in English without time limits. The search of full texts was conducted between August 2015 and April 2016. Keywords included ‘definition’ and ‘policy’ and ‘research impact’ or ‘research evidence’. The search terms ‘mental health’ or ‘public health’ or ‘health’ and ‘knowledge transfer’ or ‘research translation’ were used to focus the search on relevant health discipline approaches. Studies included in the review described processes, theories or frameworks associated with public health, health services or mental health policy.
Relevant full texts were retrieved and assessed for inclusion against the review criteria. The systematic literature search and review was conducted in the stages depicted in Fig. 1. Reference lists were explored for further relevant resources. Project team members and external experts provided recommendations on the websites of additional key organisations. Data about research impact definitions and constructs were extracted, classified into themes, discussed and synthesised.
The review imposed no restrictions on the study design apart from excluding unpublished dissertations (n = 3). The sample included theoretical and opinion pieces, case studies, descriptive studies, frameworks and systematic reviews describing processes, methods and conceptual models for assessing research impact. Inclusion criteria were (1) studies addressing public health, mental health, political science or health services disciplines; (2) expressly addressing policy impacts of research as focus or aim; and (3) including an explicit definition of research impact.
A definition of research impact was considered to be present when there was a statement in the article explaining the meaning of research impact (or ‘impact of research’ or ‘policy impact of research’) and there was an explicit effort made to define the term, i.e. an explanatory statement was given such as ‘definition’ or ‘term’ to indicate that research impact ‘is’, ‘denotes’ or ‘is understood as’. In some cases, ‘impact’ rather than ‘research impact’ was defined. When it could be reasonably deduced that the definition referred to research impact, then the definition was included in the study. Texts that described constructs related to research impact but did not define the term(s) were excluded from analysis. All definitions were included from sources that discussed several definitions.
Data extraction and analysis
Publications were recorded in the reference management software Endnote. Excel spreadsheets were used to record (1) information on the method used in the study; (2) research impact definition(s); (3) source reference(s) of the definition; and (4) constructs extracted from the research impact definitions.
Results were analysed and synthesised in two ways; first, definitions were ordered into types – an ordering that was based largely on the source cited. Second, underlying features of these definitions, based on keywords and constructs evident in definitions, were identified using an inductive comparative method and then categorised into definition types and domains.
As given in Fig. 1, the search identified 866 sources. Supplemental searches, including reference list searches and expert recommendations, yielded a further 20 publications. After duplicates were removed, 661 titles were screened against the inclusion criteria; 350 articles were excluded during abstract screening because they were not on research impact, health or policy and 135 sources were excluded during full-text screening because they did not contain a definition of research impact. A total of 83 sources were included in this review, including 45 peer-reviewed journal articles, 13 books, 7 conference papers and 18 websites or online reports (Additional file 1). Grey literature comprised 29% of included publications.
Only 23% of peer-reviewed journal articles that were on research impact (45 out of 200 that underwent full-text screening) actually defined the term.
The majority of all sources (76%) were published during or after 2011. Half (51%) of the definitions of research impact were from the United Kingdom, 22% from Australia, 16% from other European countries (Germany, Netherlands, Spain, Italy, Austria, Sweden, France and Finland), 10% from the United States of America and 2% from Canada.
A total of 108 definitions were provided. Most publications (60%) referred to a single research impact definition, while the remainder presented two to four definitions. The majority of definitions (76%) were from, or cited, research organisations and funding institutions (i.e. grey literature). The remainder provided original (i.e. unreferenced) definitions of research impact (16%) or cited other peer-reviewed literature (9%). The most highly cited definitions were provided by the United Kingdom Research Excellence Framework (REF), the Higher Education Funding Council for England (HEFCE), the Research Councils UK (RCUK), the Australian Research Quality Framework (RQF) and the Australian Research Council (ARC) (Table 1).
Several commonalities were evident in the four main types of definitions identified. These were (1) research impact defined as a demonstrable contribution to society and economy (definition provided by the RCUK); (2) research impact defined as an effect, change or benefit to society and economy (REF and HEFCE); (3) bibliometric definitions; and (4) use-based definitions.
The first two types – the RCUK and the REF/HEFCE – can also be classified as research governance definitions. The research governance group also includes the RQF and ARC definitions, which were hybrids of the RCUK and REF/HEFCE definitions.
The RCUK definition
The two central definitions used in scientific journals and policy books were contributed by the RCUK and HEFCE (the REF). The RCUK defines research impact as “the demonstrable contribution that excellent research makes to society and the economy” . The RCUK defines research impact using the adjective ‘demonstrable’, emphasising that the contribution must be provably linked to an impact (i.e. that the societal impact of research cannot be assumed) whilst the adjective ‘excellent’ equates impact with research quality. In this definition, impact is restricted to the contribution of research to the domains of ‘society’ and ‘economy’. The emphasis on contribution (input) makes this definition neutral with respect to having an expectation of a specific outcome or change.
While the RCUK does not explicitly reference policy impacts, others have expanded upon it to encompass research impacts on policy in two slightly different ways; first, is the impact of research to an area policy (as in ), i.e. the ‘policy benefits’ of research  and second is the contribution that research can make to policy and good governance, i.e. to improving the effectiveness of public services and policy [30, 31].
The REF/HEFCE definition
The HEFCE manages the REF, which is used to assess research quality and allocate research funding in the United Kingdom . HEFCE and the REF research impact definitions are equivalent and referenced interchangeably.
The HEFCE/REF guidelines define research impact as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” . These two core definitions – research impact as a ‘demonstrable contribution’ (by RCUK) versus ‘effect on, change or benefit’ to society, policy and economy – differ in the inclusivity of their concept of impact, on whether the process (contribution) or outcome (effect) is emphasised, and whether research impact can be readily measured.
The HEFCE/REF is conceptually more nuanced than that provided by the RCUK insofar as it emphasises a broader range of areas of influence. Chandler  adds to the core definition that research impact enables the development of new products, services and policies – in other words, research impact can be defined through its capacity to facilitate innovation. Similarly, Donovan  adds industry and government to the list of ‘beneficiaries’ of research impact. Pragmatic and person-centred interpretations of research impact require that research impact translates into ‘real-world outcomes’  and some, such as Chandler , see research impact as pertaining to economic, social and cultural ‘lives’, and thus referencing (individual) human activities.
Multiple authors cite the second part of the REF/HEFCE definition, which includes a list of impact foci, namely “activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding of an audience, beneficiary, community, constituency, organisation or individuals in any geographic location whether locally, regionally, nationally or internationally" (, p. 5; , p. 45). This definition broadens the spheres of possible impact to include psychosocial impacts and impacts at numerous organisational and geographical scales.
REF departs substantially from the RCUK definition insofar as it includes within the definition the role of research in the prevention of harms and reducing risks, costs or negative impacts . The normative rendering of research impact as a benefit or a positive return (rather than the value-neutral ‘change’ and ‘effect’) is the focus of Ovseiko et al.’s  definition. This extends the HEFCE definition to include ‘social value’ and specify positive returns from research in terms of social cohesion, social welfare and investments, public engagement with science, and sustainable development. Reed  specifies that research evidence can be useful in preventing the adoption of harmful legislation and products.
The ARC and the Australian RQF definitions
The third most frequently cited definition, that of the research funding body the ARC, is a hybrid of the REF and RCUK definitions and so cannot be classified as a distinct type. This is the broadest core definition included within this review with respect to the areas of potential impact that the definition encompasses. The ARC  defines research impact as “demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia” (for example , p. 158; , p. 32). National security is a unique feature of the ARC cited definitions. The ARC definition includes policy impacts of research within its core definition and regards a wide range of different types and levels of impact that are left open for further inclusion.
The common unifying elements between the ARC  and the Australian RQF  definitions are (1) a reference to the social, economic, cultural and environmental benefits of research and (2) extension of impact scope beyond academia. Some definitions phrased these four contributions as ‘outcomes’ instead of benefits [14, 34]. They differed insofar as publications citing the RQF definition of research impact were more heterogeneous, narrative and interpretative than those referring to the ARC. Policy impacts were explicitly mentioned in all five publications that used the ARC definition. However, only one source that cited the RQF mentioned policy impacts.
Bibliometric definitions, some of which arise out of the field of economics, focus on demonstrable and measurable research impacts in the form of quantifiable data. Some authors, such as Tonta et al. , approach research impacts quite narrowly and define research impact quantitatively as citation frequency in literature. However, others are more inclusive and list other forms of quantifiable impacts as part of their definition. Research impact for the London School of Economics and Political Science Public Policy Group (PPG) is a “recorded or otherwise auditable occasion of influence from academic research on another actor or organization” (, p. 310; , p. 7). The PPG website  adds to this definition by indicating that “impact is usually demonstrated by pointing to a record of the active consultation, consideration, citation, discussion, referencing or use of a piece of research”. This approach considers research impact assessment methods beyond citations by attempting to capture oral communication, but it demands a record of impacts. The Association of Commonwealth Universities  cites the PPG definition and states that research impact establishes the influence of research knowledge, rather than its consequences.
Similarly, Hannemann-Weber et al.  explain research impact through quantifiable influence and draw direct links between activities and research outputs referred to as ‘impact of publications’. While focussed on bibliometrics, this conceptualisation acknowledges broader social processes that underpin research impact as measured bibliometrically, such as the acceptability and visibility of research, the status (reputation) of research producers and the actions of researchers in the promotion of research findings. The explanation of research impact through quality, visibility and reputation of research outputs thus provides a definition that interestingly overlaps with the RCUK’s alignment of research impact with research quality. Moed et al. (, p. 132) formulate a definition in which they clarified the relationship between research outputs (‘the extent to which the research creates a body of scientific results’) and impact (‘the actual influence of the research output on surrounding research activities’).
Qin  agrees with these ideas in defining research impact by the extent to which outputs are diffused across disciplinary and geographical boundaries (measured by citations), the extent to which these have been adopted (measured by intellectual property purchases and licences), and benefits established (measured quantitatively and qualitatively). Harland , citing Korhonen et al. , expands on the list of research outputs that constitute evidence of impact by adding the concept of ‘pathways’, notably international and cross-national platforms, that can improve impact, albeit still defining research impact narrowly, in terms of dissemination in academic circles. Nightingale and Marshall  expressed the idea that citations exhibit the extent of academic significance, noting, however that this is not the same thing as research impact.
The Australian National Health and Medical Research Council  defines citation tracking as one expression of research impact in terms of the impact of ideas and methods within academia. However, the National Health and Medical Research Council definition acknowledges that there are also less easily measurable forms of research impact such as research that improves patient care, guides policymakers to adopt health prevention strategies or translates into systems level change. Hartwell et al.  suggested that only research that affects practice has impact regardless of how highly cited it is. Cohen et al.  agree that policy impacts of research have broad effects, and can result from pro-health campaigns and from organisational and funding changes. For Cohen et al.  policy impacts must be tangible, measurable and manifest in a specific time frame, namely after research had been produced without feeding back into research production.
Many academic articles define research impact by distinguishing between research impact, research use and research outputs. Unlike the instrumentalist definitions found in the grey literature, these definitions tend to be more theoretical, policy and practice oriented, and focussed on the influence of research findings on the activities and knowledge of researchers and policymakers.
Walter et al.  defined research impacts in terms of the uses to which it is put, namely, conceptual use versus instrumental use. An extended form of this definition is provided by Nutley et al.:
“Broadly, instrumental use refers to the direct impact of research on policy and practice decisions. It identifies the influence of a specific piece of research in making a specific decision or in defining the solution to a specific problem, and represents a widely held view of what research use means. Conceptual use is a much more wide-ranging definition of research use, comprising the complex and often indirect ways in which research can have an impact on the knowledge, understanding and attitudes of policy makers and practitioners. It happens where research changes ways of thinking, alerting policy makers and practitioners to an issue or playing a more general ‘consciousness-raising’ role” (, p. 36).
Meagher et al.  emphasise that instrumental research impact deals with the attribution of particular policy decisions to specific research whereas conceptual impact embodies the significance of diffusion of research impacts. Their definition is different to the outcomes/benefits-based definitions. Instrumental use is understood in terms of the metaphor of a ‘hammer’. Research ‘hits’ policy and practice to cause a decision or directive. Impact, here, is causal but not necessarily linked to outcomes (beneficial or otherwise).
Jones and Cleere  reference the European Science Foundation in defining research impacts in terms of both their contributions to specific fields and in terms of how they are enacted. This included health impacts (‘contribution to public health, life expectancy, prevention of illnesses and quality of life’) and policy impacts of research (‘contribution to how policymakers act and how policies are constructed and to political stability’). Thus, research impacts on policy can be manifested through contributions to the political culture, the policy development process and the stability of the political regime.
Brewer  argues that policy-specific impacts are demonstrated in research use by policymakers, research uptake into policies, and by improved effectiveness of policies and health services. Wilkinson et al.  also stress that the policy impacts of research extend to private and non-governmental sectors. Their broad definition encompasses the processes of knowledge exchange and relationships that facilitate research impact.
Comparison of domains found in definitions
Definitions each varied on one of four domains of meaning, namely contribution, change, avenues and levels of impacts (Table 2).
Research impact was most often defined in terms of the contribution that research made to different areas of influence, including among others the economy, society, environment, culture, policies and health. Just over half (52%) of definitions explicitly mentioned policy as an object of research impact.
Research impact definitions also varied concerning the types of avenues of impact, i.e. the mechanisms or processes by which research could be said to have impact. This was the second predominant construct found in definitions. Effects on knowledge, understanding, awareness and/or attitudes (for example, of practitioners and policymakers) were included in 59% of research impact definitions. References to ‘activities’ (22% of definitions) and ‘processes’ (16%) were also frequent. One-third (33%) of definitions distinguished research impacts as being those evident beyond academia. Many publications defined impact in terms of ‘outcomes’ achieved (14%) and ‘outputs’ (13%). Research impact was defined in terms of ‘demonstrable’ or ‘measurable’ outcomes in 28% of definitions. Two main aspects emerged from definitions, namely (1) research has impact by changing knowledge, understanding, awareness and attitudes, or through creating products (effects on different avenues of impact); and (2) research has impact through scholarly activities or excellent research (through effects on quality and behaviour).
Another important element was the variety of synonyms for impact that existed, i.e. as an effect, change or benefit to areas of influence including any positive and negative impacts that research may have. A clear ‘positivity bias’ was evident in these definitions, indicating their origins in bureaucratic documents. Importantly, over half (58%) of all definitions interpreted research impact as leading to positive gains or the reduction in societal harms; no definitions mentioned that research use may also lead to negative outcomes.
Finally, the research impact construct was also commonly defined through a range of levels of impacts that research evidence can have (i.e. international, national, local and individual impacts). References to the individual and national levels of research impact received most attention and were respectively mentioned in 19% and 18% of research impact definitions. Global and regional research impacts were mentioned in less than 10% of definitions.
The evolution, diffusion and use of research impact definitions
This review confirmed the heterogeneous and recombinant nature of research impact definitions indicating, perhaps, struggles to find an acceptable definition for this complex term [8, 9, 59]. However, our review also highlighted that most of the research impact literature discusses this concept without explicitly defining it, with only one-fifth of peer-reviewed journal articles doing so. Attempts to define research impact were more common in the grey than in the peer-reviewed literature, confirming the extent to which impact is a bureaucratic rather than academic term [60, 61]. Our findings confirm previous research showing that research impact definitions, and the research impact ‘agenda’, have emanated from research funding bodies in the United Kingdom and been diffused to other countries, such as Australia [60, 62].
The dominance of research governance definitions indicated by our review, reflects the political history of the impact agenda [62, 63]. However, their continued dominance may limit the pursuit of academic understandings of research impact by restricting it to demonstrable returns from research investments. Our review found evidence of this type of focus in nearly a third of research impact definitions. While some authors, like Kenyon , express scepticism as to whether it is in fact possible to define such a complex term in an adequate manner, we echo the views of authors such as Tinkler  and Bornmann , who advocate for the need to include ‘the diversity and richness’ of research impacts into definitions; we argue that this entails the need for researchers to exercise reflexivity in relation to research impact definitions, being mindful of the origins of the different definitions, their purposes and limitations.
Debates over definitions
Most definitions of research impact emphasised positive returns. While some definitions used more neutral language, negative impacts of research were rarely mentioned. The issue from a research governance standpoint is whether a focus on impacts defined in terms of non-academic benefits creates an incentive to skew results to demonstrate benefit, even where there is none. This may create perverse incentives to implement ideas before they have been properly tested or their implications fully thought through. Most definitions interpreted research impact as leading to positive gains or the reduction in societal harms. However, there are several examples of research that has had negative or, at least, contested impacts (e.g. drugs such as thalidomide or weapons of mass destruction). Researchers may be encouraged to conduct research in favour of short-term ‘impacts’ with the result that research that is critical of prevailing governing paradigms is not pursued and not funded, resulting in longer term negative effects on innovation and advancement through research. Furthermore, a definition that encompasses a clear ‘positivity bias’, as is evident in these definitions, may be limited in pursuing academic understanding of how evidence impacts policy.
A related concept to that of ‘research impact’ is that of ‘knowledge valorisation’. Knowledge valorisation is gaining significant traction in the European Union research funding and dissemination discourse. Valorisation is a process by which academic knowledge is transformed into social and economic value [65, 66]. Valorisation focusses on the process of value creation from academic research through commercial activities and industry associations with academia; in other words, it is closely associated with the commercialisation of academic research .
Valorisation is a concept that is linked to, but not the same as, a definition of research impact. Valorisation and impact are linked through their combined focus on the usefulness of research, and the ability to produce commercial and/or social returns from academic knowledge. Perhaps due to its focus on commercialisation, the literature on valorisation has paid less attention to policy impacts of research. Furthermore, Benneworth  has critiqued the conceptualisation of knowledge valorisation for being more applicable to the physical and life sciences than to the humanities and social sciences. In contrast, ‘research impact’ definitions provide a broader and more abstract conceptualisation concerned with the longer term application of knowledge to more complex societal problems.
How policy features in research impact definitions
Around half of the research impact definitions included a consideration of how research impacts on policy, mostly by mentioning policy as one of several impact foci. The complexities involved in the conceptualisation of research impact on policy have been acknowledged by many authors [7, 10, 13]. There is a recursive issue here, among the main challenges of defining research impact on policy are the uncertainties regarding how exactly research evidence brings about policy changes, and also how those policy changes link to ‘real-world’ outcomes [55, 63]. However, these uncertainties exist, in part, due to a lack of agreed upon definitions of research impact that can facilitate a research agenda.
There are several recognised difficulties in attributing a policy impact to a specific piece of research [5, 69, 70]. The original piece of research may be re-interpreted in the policy process in ways that are incorrect or not consistent with its intent, or it may be adapted to particular contexts and transformed in the process. Multiple influences at different stages of research and policy translation may also function to diffuse knowledge. Policy change, as suggested by Thomas , is dynamic and the product of a web of decisions that may reflect competing values that result in political compromises. A policy relevant definition of research impact should take account of the fact that there is not always a direct pathway from evidence to policy and that impact can be more, or less, directly identifiable – depending on whether that impact is conceptual or instrumental [22, 24].
Consequently, for a research impact definition to adequately capture the complexities of policy impacts, it must include elements that relate to two different phenomena – policy content and policy processes. We need a definition that is clear about the different, both direct and indirect, ways in which research can impact on policy and thus help us investigate it for academic purposes.
Proposed definition of research impact for (mental) health policy
Based on this review, we propose the following definition for research impact on health policy that can be tailored for use in health disciplines, including public and mental health. A definition specific to mental health is given:
Research impact is a direct or indirect contribution of research processes or outputs that have informed (or resulted in) the development of new (mental) health policy/practices, or revisions of existing (mental) health policy/practices, at various levels of governance (international, national, state, local, organisational, health unit).
This definition tailors core constructs that were identified in the literature to the field of health policy. It includes the constructs of contribution (but not demonstrable), change, research outputs, policies, practices, various avenues and levels of impacts and encompasses impacts that may be said to occur at different time points. For example, immediate impact might be the use of research processes and outcomes to increase policymakers’ knowledge and inform attitudes, medium-term impact may be an impact of research on the development and revision of policy, and a long-term outcome may be the multilevel impact of research through the implementation and evaluation of policy and practice.
The definition overcomes some of the limitations of existing definitions. It does not restrict research impact to its measurable qualities and includes both desirable and undesirable impacts, allowing for its use in different contexts to capture the full range of possible research impacts. Of the definitions available, the proposed definition is perhaps most similar to that of the ARC  definition.
Strengths, limitations and future research
The key strength of this review is its comprehensiveness and wide coverage of both peer-reviewed and grey literature, the latter having been neglected in previous reviews. The use of a systematic search methodology allowed us to identify the prevalence and reach of different types of definition and research impact definitions overall. The review confirmed that the two most common definitions in both peer-reviewed and grey literature originated from the grey literature, supporting the need to include the grey literature in future reviews of research impact studies.
This review is limited by its conservative search term selection, as only publications that explicitly used the term ‘research impact’ or its close derivatives were included. It is possible that relevant literature that failed to use this terminology was excluded, for example, economic literature on payback models [72, 73]. That said, economic models such as the payback model arguably represent operational definitions rather than conceptual definitions.
The study focus is limited to literature that was in English. Thus, it may not have captured relevant discourses from European or non-English speaking literature.
Additionally, five unique definitions were identified in this review that fell outside the typologies constructed [61, 74,75,76,77]. These definitions all drew distinctions between the ideas of impacts, outputs and/or outcomes, and shared some of the features of the aforementioned definitions. Future research will take account of feedback from relevant stakeholders (e.g. researchers and policymakers) on different ways of defining research impact and on the definition proposed here; academics, policymakers, bureaucrats, clinicians, patients and the general community are likely to hold different views on this topic.
Facilitating the effective translation of health research to policy and practice requires a dedicated research agenda. The dominance of bureaucratic definitions, the tendency to discuss but not define the concept of research impact, and the heterogeneity of definitions confirm the need for conceptual clarity in this area. Without wanting to impose a reductive imperative within debates around research impact definitions, we pose a definition of research impact that is primarily for the purposes of academic study of the impact of research on health policy but that could be adapted for use in other specific contexts.
Australian Research Council
Higher Education Funding Council for England
London School of Economics and Political Science Public Policy Group
Research Councils UK
United Kingdom Research Excellence Framework
Australian Research Quality Framework
Greyson DL, Cunningham C, Morgan S. Information behaviour of Canadian pharmaceutical policy makers. Health Info Libr J. 2012;29:16–27. doi:10.1111/j.1471-1842.2011.00969.x.
Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst. 2015;13:1–7. doi:10.1186/s12961-015-0003-1.
Nutbeam D. Achieving ‘best practice’ in health promotion: improving the fit between research and practice. Health Educ Res. 1996;11:317–26. doi:10.1093/her/11.3.317.
Donovan C, Butler L, Butt AJ, Jones TH, Hanney SR. Evaluation of the impact of national breast cancer foundation-funded research. Med J Aust. 2014;200:214–8. doi:10.5694/mja13.10798.
Kuruvilla S, Mays N, Pleasant A, Walt G. Describing the impact of health research: a research impact framework. BMC Health Serv Res. 2006;6:134. doi:10.1186/1472-6963-6-134.
Zardo P, Collie A. Type, frequency and purpose of information used to inform public health policy and program decision-making. BMC Public Health. 2015;15:381. doi:10.1186/s12889-015-1581-0.
Straus SE, Tetroe J, Graham ID, Zwarenstein M, Bhattacharyya O, Shepperd S. Monitoring use of knowledge and evaluating outcomes. Can Med Assoc J. 2010;182:E94–8. doi:10.1503/cmaj.081335.
Brewer JD. The public value of the social sciences: an interpretive essay. London: Bloomsbury; 2013.
Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99:1576–83. doi:10.2105/AJPH.2008.156224.
Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9:26. doi:10.1186/1478-4505-9-26.1-10.
Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, Kinsman L. 'Making evidence count': a framework to monitor the impact of health services research. Aust J Rural Health. 2012;20:51–8. doi:10.1111/j.1440-1584.2012.01256.x.
Reed MS. The research impact handbook. St Johns Well: Fast Track Impact; 2016.
Boaz A, Fitzpatrick S, Shaw B. Assessing the impact of research on policy: a literature review. Sci Public Policy. 2009;36:255–70. doi:10.3152/030234209X436545.
Duryea M, Hochman M, Parfitt A. Measuring the impact of research. Research Global. 2007;1:8–9.
Kenyon T. Defining and measuring research impact in the humanities, social sciences and creative arts in the digital age. Knowledge Org. 2014;41:249–58.
Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: a review. Res Eval. 2014;23:21–32. doi:10.1093/reseval/rvt021.
Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004;26:75–106. doi:10.1177/1075547004267491.
Greenhalgh T, Fahy N. Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK research excellence framework. BMC Med. 2015;13:232. doi:10.1186/s12916-015-0467-4.
Tonta Y, Ünal Y, Al U. The research impact of open access journal articles. Vienna, Austria: 11th International Conference on Electronic Publishing; 2007. Proceedings. http://yunus.hacettepe.edu.tr/~tonta/yayinlar/tonta-unal-al-vienna-2007.pdf. Accessed 20 Feb 2016.
Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C, Bauman AE, Redman S, Bennie J. Policy and practice impacts of applied research: a case study analysis of the New south Wales health promotion demonstration research grants scheme 2000-2006. Health Res Policy Syst. 2013;11:5. doi:10.1186/1478-4505-11-5.
Haynes AS, Derrick GE, Chapman S, Redman S, Hall WD, Gillespie J, Sturk H. From “our world” to the “real world”: Exploring the views and behaviour of policy-influential Australian public health researchers. Soc Sci Med. 2011;72:1047–55. doi:10.1016/j.socscimed.2011.02.004.
Weiss CH. Policy research: data, ideas or argument? In: Wagner P, Weiss CH, Wittrock B, Wollman H, editors. Social sciences and modern states: national experiences and theoretical crossroads. Cambridge: Cambridge University Press; 1991.
Lingard B. The impact of research on education policy in an era of evidence-based policy. Crit Stud Educ. 2013;54:113–31. doi:10.1080/17508487.2013.781515.
Nutley SM, Walter I, Davies HT. Using evidence: how research can inform public services. Bristol: Policy Press; 2007.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg. 2010;8:336–41. doi:10.1016/j.ijsu.2010.02.007.
Sibbald SL, MacGregor JCD, Surmacz M, Wathen CN. Into the gray: a modified approach to citation analysis to better understand research impact. J Med Lib Assoc. 2015;103:49–54. doi:10.3163/1536-5050.103.1.010.
Research Councils UK. Excellence with Impact. http://www.rcuk.ac.uk/innovation/impact. Accessed 1 Oct 2014.
Weitkamp E. Between ambition and evidence. Sci Commun. 2015;14:1–5.
Halse C, Mowbray S. The impact of the doctorate. Stud High Educ. 2011;36:513–25. doi:10.1080/03075079.2011.594590.
Jones A, Cleere L. Furthering the Research Impact of UCD: Report of the Beyond Publications Committee. http://irserver.ucd.ie/bitstream/handle/10197/7292/Furthering_Impact_May_2014.pdf?sequence=1. Accessed 20 Feb 2016.
SOAS University of London. Impact Acceleration Fund: Definition of Impact. https://www.soas.ac.uk/researchoffice/iaf/. Accessed 20 Feb 2016.
Higher Education Funding Council from England. REF Impact. http://www.hefce.ac.uk/rsrch/REFimpact. Accessed 27 Feb 2016.
Chandler C. What is the meaning of impact in relation to research and why does it matter? a view from inside academia. In: Denicolo P, editor. Achieving impact in research. Los Angeles: Sage; 2013.
Donovan C. The Australian research quality framework: a live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Direct Eval. 2008;2008:47–60. doi:10.1002/ev.260.
University of York. What is Research Impact? https://www.york.ac.uk/staff/research/research-impact/impact-definition. Accessed 14 Oct 2015.
Khazragui H, Hudson J. Measuring the benefits of university research: impact and the REF in the UK. Res Eval. 2015;24:51–62. doi:10.1093/reseval/rvu028.
Parker J, van Teijlingen E. The research excellence framework (REF): assessing the impact of social work research on society. Practice. 2012;24:41–52. doi:10.1080/09503153.2011.647682.
Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: a study using research excellence framework pilot impact indicators. BMC Health Serv Res. 2012;12:478. doi:10.1186/1472-6963-12-478.
Australian Research Council. Research impact principles and framework. http://www.arc.gov.au/research-impact-principles-and-framework#Definition. Accessed 14 Oct 2015.
Birks M, Mills J. Grounded theory: a practical guide. London: Sage; 2015.
Harris R. The impact of research on development policy and practice: this much we know. In: Chib A, May J, Barrantes R, editors. Impact of information society research in the global south. Singapore: Springer Open; 2015. p. 21–44.
Commonwealth of Australia. Research Quality Framework: Assessing the Quality and Impact of Research in Australia. The Recommended RQF. https://research.vu.edu.au/ordsite/management/Recommended_RQF_Dec2006.pdf. Accessed 2 Jan 2016.
Drummond R. RIMS revisited: the evolution of the research impact measurement service at UNSW library. Aust Acad Res Libr. 2014;45:309–22. doi:10.1080/00048623.2014.945065.
Kelly U. The ‘Impact Analysis System’: Project Report and Guide to the Underpinning Conceptual Framework. http://www.viewforthconsulting.co.uk/sitebuildercontent/sitebuilderfiles/finalrep.pdf. Accessed 10 Feb 2016.
LSE Public Policy Group. Maximising the Impacts of Your Research: A Handbook for Social Scientists. http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf. Accessed 27 June 2016.
Association of Commonwealth Universities. Defining, Understanding and Measuring Impact. https://www.acu.ac.uk/membership/acu-insights/acu-insights-2/defining-understanding-and-measuring-impact. Accessed 3 June 2016.
Hannemann-Weber H, Kessel M, Schultz C. Research performance of centers of expertise for rare diseases – the influence of network integration, internal resource access and operational experience. Health Policy. 2012;105:138–45. doi:10.1016/j.healthpol.2012.02.008.
Moed HF, Burger WJM, Frankfort JG, Van Raan AFJ. The use of bibliometric data for the measurement of university research performance. Res Policy. 1985;14:131–49. doi:10.1016/0048-7333(85)90012-5.
Qin J. Empirically assessing impact of scholarly research. In iConference 2010; Urbana-Champaign, IL, USA. 2010. https://www.ideals.illinois.edu/bitstream/handle/2142/14924/qin.pdf. Accessed 3 June 2016.
Harland CM. Supply chain management research impact: an evidence-based perspective. Supply Chain Manag. 2013;18:483–96. doi:10.1108/SCM-03-2013-0108.
Korhonen P, Tainio R, Wallenius J. Value efficiency analysis of academic research. Eur J Oper Res. 2001;130:121–32. doi:10.1016/S0377-2217(00)00050-3.
Nightingale JM, Marshall G. Reprint of "citation analysis as a measure of article quality, journal influence and individual researcher performance". Nurse Educ Pract. 2013;13:429–36. doi:10.1038/465860a.
National Health and Medical Research Council. Measuring the Impact of Research – Not Just a Simple List of Publications. https://www.nhmrc.gov.au/media/newsletters/ceo/2014/measuring-impact-research-not-just-simple-list-publications. Accessed 26 Nov 2016.
Hartwell H, van Teijlingen E, Parker J. Nutrition: effects of the research excellence framework (REF). Nutr Food Sci. 2013;43:74–7. doi:10.1108/00346651311295941.
Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, Bauman AE, Redman S, Chapman S. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2015;13:3. doi:10.1186/1478-4505-13-3.
Walter I, Davies H, Nutley S. Increasing research impact through partnerships: evidence from outside health care. J Health Serv Res Policy. 2003;8:58–61. doi:10.1258/135581903322405180.
Meagher L, Lyall C, Nutley S. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Res Eval. 2008;17:163–73. doi:10.3152/095820208X331720.
Wilkinson H, Gallagher M, Smith M. A collaborative approach to defining the usefulness of impact: lessons from a knowledge exchange project involving academics and social work practitioners. Evid Policy. 2012;8:311–27. doi:10.1332/174426412X654040.
Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrative review. BMC Med. 2016;14:78. doi:10.1186/s12916-016-0620-8.1-16.
Ferguson M. The Research Impact Agenda: Defining, Demonstrating and Defending the Value of the Social Sciences. Australian Review of Public Affairs Digest. 2014. http://www.australianreview.net/digest/2014/08/ferguson.html. Accessed 3 June 2016.
Gooch D, Vasalou A, Benton L. Impact in interdisciplinary and cross-sector research: opportunities and challenges. J Assoc Inf Sci Technol. 2017;68(2):378–91. doi:10.1002/asi.23658.
Ferguson M. The evolution of ‘research impact’. In Australian Political Studies Association Conference (APSCA 2016). Sydney, Australia. 2016. https://apsa2016.arts.unsw.edu.au/node/65/paper/1513. Accessed 1 July 2017.
Bornmann L. Measuring the societal impact of research. Sci Soc. 2012;13:673–6. doi:10.1038/embor.2012.99.
Tinkler J. Rather than Narrow Our Definition of Impact, We Should use Metrics to Explore Richness and Diversity of Outcomes. http://blogs.lse.ac.uk/impactofsocialsciences/2015/07/28/impact-metrics-and-the-definition-of-impact-tinkler. Accessed 2 June 2016.
Innovation Exchange Amsterdam. Valorisation Guide. http://www.ixa.nl/fileadmin/user_upload/Documenten/ValorisatiegidsVU-UvA_Web_ENG.pdf. Accessed 1 July 2017.
Lokhorst L. What Drives Valorization in the Humanities, Arts and Social Sciences? https://dspace.library.uu.nl/handle/1874/342406. Accessed 1 July 2017.
Benneworth P, Jongbloed BW. Who matters to universities? a stakeholder perspective on humanities, arts and social sciences valorisation. High Educ. 2010;59:567–88. doi:10.1007/s10734-009-9265-2.
Calle E. P, Parnall L. Knowledge Exchange and Valorization Workshop. https://www.norface.net/wp-content/uploads/2015/12/Summary-joint-knowledge-exchange-workshop.pdf. Accessed 1 July 2017.
Kuruvilla S, Mays N, Walt G. Describing the impact of health services and policy research. J Health Serv Res Policy. 2007;12:23–31. doi:10.1258/135581907780318374.
Rispel LC, Doherty J. Research in support of health systems transformation in South Africa: the experience of the centre for health policy. J Public Health Pol. 2011;32:S10–29. doi:10.1057/jphp.2011.33.
Thomas P. The challenges of governance, leadership and accountability in the public services. In: Wallace M, Fertig M, Schneller E, editors. Managing change in the public services. Malden: Blackwell Publishing; 2007. p. 116–35.
Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: development and implementation of the Alberta innovates – health solutions impact framework. Res Eval. 2012;21:354–67. doi:10.1093/reseval/rvs027.
Kalucy E, Jackson-Bowers E, McIntyre E, Reed R. The feasibility of determining the impact of primary health care research projects using the payback framework. Health Res Policy Syst. 2009;7:11. doi:10.1186/1478-4505-7-11.1-10.
Drew CH, Pettibone KG, Finch FO, Giles D, Jordan P. Automated research impact assessment: a new bibliometrics approach. Scientometrics. 2016;106:987–1005. doi:10.1007/s11192-015-1828-7.
Eisenberg JM. Putting research to work: reporting and enhancing the impact of health services research. Health Serv Res. 2001;36:x–xvii.
Canadian Institute of Health Research. Developing a CIHR framework to measure the impact of health research: a framework for measuring the impact of health research. Ottawa: CIHR; 2005.
Thonon F, Boulkedid R, Delory T, Rousseau S, Saghatchian M, van Harten W, O'Neill C, Alberti C. Measuring the outcome of biomedical research: a systematic literature review. Plos One. 2015;10:1–14. doi:10.1371/journal.pone.0122239.
We thank Dr Florin Oprescu from the University of the Sunshine Coast for his input into the research impact definition and Ms Sarah Yeates from the Centre for Youth Substance Abuse Research, The University of Queensland, for her assistance with database searches.
The project was funded by the National Health and Medical Research Council (NHMRC) Centre for Research Excellence in Mental Health Systems Improvement (APP1041131). This research was supported by the Australian Government Research Training Program Scholarship from the University of Queensland to Kristel Alla.
Availability of data and materials
All data generated or analysed during this study are included in this published article and its Additional file.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Definitions of research impact included in the study. Table S1. RCUK definitions. Table S2. REF/HEFCE definitions. Table S2a. Research impacts as benefits, effects or changes (REF/HEFCE not cited). Table S3. ARC/RQF definitions. Table S4. Bibliometric definitions. Table S5. Use-based definitions. Table S6. Original definitions. (DOCX 58 kb)
About this article
Cite this article
Alla, K., Hall, W.D., Whiteford, H.A. et al. How do we define the policy impact of public health research? A systematic review. Health Res Policy Sys 15, 84 (2017). https://doi.org/10.1186/s12961-017-0247-z