New directions in evidence-based policy research: a critical analysis of the literature
Health Research Policy and Systems volume 12, Article number: 34 (2014)
Despite 40 years of research into evidence-based policy (EBP) and a continued drive from both policymakers and researchers to increase research uptake in policy, barriers to the use of evidence are persistently identified in the literature. However, it is not clear what explains this persistence – whether they represent real factors, or if they are artefacts of approaches used to study EBP. Based on an updated review, this paper analyses this literature to explain persistent barriers and facilitators. We critically describe the literature in terms of its theoretical underpinnings, definitions of ‘evidence’, methods, and underlying assumptions of research in the field, and aim to illuminate the EBP discourse by comparison with approaches from other fields. Much of the research in this area is theoretically naive, focusing primarily on the uptake of research evidence as opposed to evidence defined more broadly, and privileging academics’ research priorities over those of policymakers. Little empirical data analysing the processes or impact of evidence use in policy is available to inform researchers or decision-makers. EBP research often assumes that policymakers do not use evidence and that more evidence – meaning research evidence – use would benefit policymakers and populations. We argue that these assumptions are unsupported, biasing much of EBP research. The agenda of ‘getting evidence into policy’ has side-lined the empirical description and analysis of how research and policy actually interact in vivo. Rather than asking how research evidence can be made more influential, academics should aim to understand what influences and constitutes policy, and produce more critically and theoretically informed studies of decision-making. We question the main assumptions made by EBP researchers, explore the implications of doing so, and propose new directions for EBP research, and health policy.
Introduction: the evidence-based policy movement
Although sceptics can be found, few researchers or policymakers would publicly disagree that evidence-based policy (EBP) is a goal for both academe and government. Reports describing the importance – and difficulty of – achieving this goal are published with regularity [1–3]. Perhaps unlike other disciplines, EBP has staunch advocates who contribute to an ever-increasing body of commentary around the subject.
EBP is sometimes said to have derived from evidence-based medicine (EBM), which dates back at least to 1972, with Archie Cochrane’s seminal work on effectiveness and efficiency . Since the early 1970s, both practitioners and academics have also considered how policy – in the sense of larger-scale decisions about the delivery and management of services at a population level – could be based on, or informed by, evidence [5, 6]. For Cochrane and his heirs, the goal of EBM was to bring about the abandonment of harmful and ineffective interventions, and the adoption of interventions shown to be effective for clinical outcomes. This was to be achieved by implementing findings from systematic reviews and meta-analyses of robust outcome evaluations, ideally randomised trials [7–10].
However, this straightforward narrative of evaluation-based EBP modelled on medicine has long existed in a broader landscape of initiatives to foster closer and more effective links between research and policy (or between researchers and policy-makers). In the UK, for example, these ideas can be traced back at least to the Rothschild experiment, evaluated by Kogan and Henkel in 1983 . The aim of this funding initiative was to enable the health research system to respond to policymakers’ priorities for research . The experiment was largely abandoned, which Kogan and Henkel ascribed to cultural differences between researchers and policymakers, the need for interaction, and other barriers [12, 13]. Recent publications, again in the UK, from the Government Office for Science have provided guides for both academics and policymakers who wish to engage with the other group [14, 15], and there is increasing interest within academic and higher education bodies generally about how academics may be able to increase their impact, the nature of this impact, and implications of the ‘impact agenda’ [16–18]. However, the tenor of these publications still tends to focus on promoting the use of academic research, rather than studying the practices of knowledge production and policymaking and implementation.
As a result, specific questions about the use of research within policy have become divorced from a broader perspective on policy-making. Many researchers in what may be termed ‘applied research’ fields use terms like ‘knowledge translation’, ‘knowledge exchange’, or ‘evidence use’ without providing clear definitions of what knowledge is part of which decision-making process [19–21]. The result has been a loss of clarity with respect to how evidence is supposed to improve decision-making, what constitutes and defines ‘evidence’ and ‘policy’, and which processes and outcomes are targeted by political or academic efforts. Nevertheless, both EBM and EBP have achieved substantial financial and political support, and substantial weight by the creation of a collection of organisations dedicated to producing evidence-based policy and practice recommendations, guidelines, and best practice statements, such as the What Works centres . For health policy and health management in particular, there remains a prominent discourse about moral, ethical, political, and often financial imperatives to use evidence to make the best (value) decisions  – without, ideally, disenfranchising non-expert publics . The study of the use of evidence in policy varies from negative to positive advocacy, from simplistic to complex understandings of the processes involved, from uncritical technical approaches to highly cynical commentary. Broadly, practitioners and academics have focused on facets of the use of evidence by policymakers and practitioners, have written polemics encouraging colleagues to do so [23, 25–29], identified barriers and facilitators into evidence use [30, 31], and designed interventions to increase the use of evidence by policymakers . Evidence-based policy and practice, knowledge translation, and related concepts have become touchstones across a vast range of disciplines – almost sub-disciplines in their own right, with canons and conceptual toolkits of their own.
While much of this work remains mainly theoretical (e.g., ), there is a rapidly growing empirical evidence base on barriers and facilitators of evidence use. Some idea of the extent and nature of this literature can be gained by looking at systematic reviews. Three reviews are particularly relevant here [30, 31, 34]; their methods, findings, and conclusions are summarised in Table 1.
Common findings across all three include the importance of personal relationships and contacts between decision-makers and researchers, and the need for research to be clearly and accessibly presented. Cultural and practical barriers to the use of evidence by policy-makers are identified. Finally, all three make the point that policymakers’ definitions of evidence do not match academic constructions of ‘evidence’. All three reviews also point to gaps in the literature as priorities for further research, but differ in their identification of these gaps. Orton focuses on the need for evaluation research of strategies to increase the uptake of research evidence , while Innvær et al.  and Oliver et al.  are both more circumspect, pointing out that despite the size of the evidence base, much about policy-makers’ attitudes to research evidence remains unclear. Innvær et al. show how the limited available evidence mainly describes policymakers’ beliefs and attitudes, rather than actual behaviours, and hence cannot be used as a basis to make strong recommendations . Perhaps more importantly, both reviews note that there are few grounds by which to make firm recommendations or conclusions about the process, impact, or effectiveness of research in policy. Only rarely is enough detail known about the policy process to be able to comment usefully: for example, who are the main actors, where are decisions made, and how evidence fits into the process.
The structure of this paper is as follows. First, drawing on a dataset from a recent systematic review of barriers and facilitators , we offer a high-level overview of the literature on evidence use in health policy, drawing out broad theoretical and definitional commitments. This paper does not primarily consider the findings of those studies, which are summarised above and set out in more detail elsewhere . We also draw on similar findings about policymakers in non-health fields . We set these data in the context of the wider theoretical literature about evidence and policy, drawing insights from policy sciences. Secondly, we aim to identify and challenge some of the more normative assumptions which are widely prevalent (if often implicit) in the EBP literature, particularly the following: that the policy-evidence ‘gap’ needs ‘bridging’; that policy is usually not based on any data; that policy requires research evidence, preferably evaluative intervention research evidence; and that if more evidence were used in policy making, policy would be better. Finally, we make suggestions for a new agenda for EBP research.
We are not advocating against EBP. On the contrary, we believe that better policy decisions would be a desirable outcome, and that evidence ought to play a role in those decisions. Rather, our conclusions from this critical review of the literature are about EBP research, rather than EBP itself. Researchers have directed their attention at how to increase the impact of their own outputs, rather than on understanding the processes behind policy change. The support for EBP is not as single-minded or vociferous as it was. Reflecting on this historical trend, and reasons behind any such shift, we present our novel contribution to the literature: an illumination of the discourse around EBP by comparing theories, methods, and substantive approaches with those from other fields, and on this basis propose new directions for EBP research.
Review: evidence-based policy research
Below, we describe some of the underlying concepts and approaches available to researchers attempting to understand the relationship between research and policy processes.
Innvær et al. described how the literature at the time fell into two camps ; those supporting the ‘two-communities’ hypothesis, which explores whether barriers to research utilisation are mainly driven by cultural or institutional differences between researchers and decision-makers, and those drawing on Carol Weiss’s typology of ways of using evidence [36, 37]. Policy and academic actors were conceptualised as opposing sets of actors, with different priorities, languages, practices, and priorities. For proponents of this perspective, ‘bridging’ this gap becomes a priority. It seems likely, however, that insisting on the existence of this ‘gap’ may polarize previously neutral actors; and indeed this debate fails to recognise that this may be a UK-specific problem (cf. to Dutch studies).
Models of the research and policy processes are, as noted by previous theorists of EBP, rarely made explicit in evaluations of applied research. Where implicit, a simple ‘pipeline’ model is usually assumed (i.e., that the more research is carried out and the higher the quality, the bigger the effect on policy and practice) [38–41].
A large new theoretical strand within health policy focuses on knowledge brokerage/translation as a framework for understanding use of evidence [42–46]. This model can be seen as an extension of Weiss’s ‘instrumental use’ model, or ‘enlightening’ and ‘strategic use’ of evidence, describing the influence of research on policy, and is linked to ideas about ‘coproduction’ and ‘user involvement’ .
Weiss and Caplan are still major influences on the field of EBP research, but the degree to which these contributions are appropriately exploited is debatable. The linear (direct use) model is usually perceived to be superior, quite contrary to Weiss’ argument that policymakers thought enlightening (indirect) use could offer more. This message is often overlooked by research (and researchers) in the field, namely that researchers need to reflect on the common view that policymakers are interest-oriented and indifferent to evidence. Barriers to use of research are equated with barriers to direct use of research, while the broader concept of enlightening (indirect) use is rarely seen as equally relevant and useful.
While this is still widespread, theoretical learning from other fields is filtering into the debate. Researchers in policy studies have long seen the policy process itself as a contested arena of negotiation [48, 49]. The messy, complex, and serendipitous nature of policymaking is described by Kingdon , Weiss , Simon , and Lindblom , amongst others. These scholars have contributed ideas such as punctuated equilibrium , policy ‘windows’ and ‘streams’  or ‘stages’ , bounded rationality , and incrementalism . However, the degree to which these models are used in planning and conducting empirical research is debateable. As Lomas notes, one reason why these may have been resisted by EBP researchers is because they offer little help and no tools to help the aspiring policy-advisor .
Of course, many of these are theories about policy, rather than analysis tools to assist advisors . Cairney argues that policy analysts tend to use concepts, such as the policy cycle, which have been rejected by policy scholars. We would qualify this statement by restricting the rejection of such theories to within science and technology studies and policy sciences – these ideas are still common currency within health policy amongst other fields .
The influence of these ideas on the health field is perhaps increasing, with applications of ethnographic methods  and actor-network theory applied to EBP . However, the majority of studies still use over-simple theoretical models . Asserting the existence of and describing and prescribing interventions to close the ‘research-policy gap’ is a stance which, we argue, is likely to perpetuate and even create gaps between the professions.
Focus of the research
The main focus of much theoretical and empirical work in EBP has traditionally been, implicitly or explicitly, on research evidence uptake, primarily peer-reviewed research carried out by university-based academics [30, 31]. However, a third of studies included in our review examined non-research data, for example, public health surveillance data, strategic needs assessments and other impact assessments, geographic information systems , or other non-research evidence [63–65]. This suggests that policymakers interpret and use ‘evidence’ in a broad sense, which is usually not acknowledged by academic commentators [7, 10].
The reviews described above included a minority of studies which aimed to describe the policy process in detail, often using a case-study approach. Frequently, these focused on the use of a particular type of evidence such as economic evaluations  or the role of a specific piece of evidence [60, 67]. Few attempt a descriptive contextualisation or ethnographic understanding of the policy process, with exceptions, e.g., [68–71]. Virginia Berridge’s seminal work on the NHS and comparative health policy development uses historical methods to understand these processes and develop theory around them . Others have taken empirical ethnographic approaches to understand, for instance, how health decision-makers conceptualise and use evidence [73, 74]. With these exceptions, very few studies have, as yet, taken anthropological or historical approaches to understanding the role of evidence or research in policy.
Research in the area thus focuses primarily on how to increase uptake of research, on designing and evaluating interventions aiming to increase research use, and on identifying barriers and facilitators of research use by policymakers [74–76]. This pattern of research on evidence use skews the debate by focusing on exceptional cases of research use in policy-making, rather than the normal discharging of statutory business. As Kogan and Henkel noted, attempts to improve use of evidence can “fail to note how in those areas of policy where data are diffuse, and analyses most likely to be strongly influenced by value preferences, problems must be identified collaboratively between policy-maker and scientist. It failed to acknowledge that policy makers have to work hard to identify problems, to specify research that might help solve them, and to receive and use the results of research” . While a comment evaluating the Rothschild experiment, the suggestion that EBP research does not reflect the range of knowledge-producing or policy-making processes can also be levied at much of the academic work subsequent to this important study.
Focusing on the use of research evidence also allows researchers to sidestep the rest of the policy process and avoid the context of decision-making more widely. Most studies focus on single elements of the policy-making process – dissemination of evidence, sources, and types of source, knowledge transfer, and priority setting – rather than trying to characterise the process as a whole. EBP research could draw here on an extensive research literature in policy analysis from political science, starting with studies of what is called ‘the policy cycle’ or ‘a stages approach’ [39, 77]. Such studies can analyse who is making which decisions, about what, and when; the distinction between practice, management, governance, and policy is rarely spelt out . Clear definitions of policy, use of research, and decision making has been encouraged for over 60 years, but remains evasive .
Considerable theoretical work has gone into producing taxonomies of factors influencing the utilisation of evidence, e.g., [36, 80]. However, these rich theoretical discourses are not reflected in the bulk of the empirical literature, which, despite its breadth with respect to the sectors and categories of evidence types studied, finds fairly consistently that the main factors affecting use of evidence are (a) access to relevant and clear information and (b) good relationships between researchers and research users. This may be due to the methods used in the studies: most use only interviews or surveys to ask researchers and policy-makers about their perceptions about evidence use; very few use methods such as participant observation to observe how evidence is actually used in practice, or attempt to find documentary proof of research use (with exceptions; ). These lists, while important, cannot on their own lead us to an improved understanding of the role of evidence in the jigsaw of the policy process.
Another noticeable feature of the evidence base is the emphasis given to researchers’ own views of research utilisation, for example, a study aiming to look at everyday working practice, such as that of Taylor-Robinson et al. , samples primarily health inequalities/public health academics and practitioners rather than decision-makers themselves (such as councillors or executives), even though they themselves describe lack of contact between academics and policymakers as a barrier to use of evidence. The majority of academic studies in EBP research are, unsurprisingly, written by and for academics, with little involvement of policy-makers as co-authors; indicating that policy-makers are not involved in developing or carrying out relevant research.
Underlying assumptions of this research and critical reflections
This overview of the literature provides a starting point for a more critical engagement with the empirical literature on EBP. We provide a broad-brush characterisation of parts of the literature which helps to draw out common assumptions across the field as a whole and enable critical reflection on them. We focus on three such assumptions: 1) that the policy-evidence ‘gap’ needs ‘bridging’; 2) that policy is usually not based on any data, and policy requires research evidence, preferably evaluative intervention research; and 3) that greater use of evidence in policy-making will produce better outcomes at a population level. It is by no means the case that these assumptions are universally shared among EBP researchers, or that we are the first to identify these issues [61, 82, 83]. Nonetheless, a large proportion of the available research still rests on an uncritical acceptance of these assumptions. Below, we describe the effect of these assumptions, justify our rejection of them, and discuss the implications of taking a more critical approach.
Assumption 1: that a policy-evidence ‘gap’ exists
The ‘evidence-policy gap’ is a widely-acknowledged construction in policy-related research, asserting the existence of two separate communities with their own ecosystems and languages [61, 83–85]. Much of the ‘knowledge translation’ literature, which attempts to take a broader perspective on EBP, fails to question the assumption that knowledge and practice are two separate practices, and the ‘joining’ or ‘bridging’ of these (depending on the authors’ preferred metaphor) is the task of EBP researchers; see, e.g., [20, 61, 83, 84, 86]. However, as Choi recognizes, evidence – and those associated with evidence – is just one voice among many . We do not yet know how to make that voice more helpful nor more influential.
Recently, a shift away from this dichotomous debate has been made, with concepts such as ‘knowledge translation’ or ‘transfer’ being replaced by ideas about ‘learning’, ‘contribution’, and co-production . These ideas frame the relationship between research and policy as a two-way negotiation in which both partners learn from the other – pragmatically and politically a step towards an equality of prioritisation and experience. Certainly, this represents a greater openness to seeing a broader range of data types as relevant – including, for example, contextual, descriptive data as well as evaluations and other forms of research evidence. Too often, however, this debate is hijacked by methodologists from opposing camps wishing to defend their own method in the face of criticism – whether real or perceived, e.g., [89, 90]. Without clear definitions of ‘evidence’ and ‘impact’ or ‘learning’, these studies contribute to negative stereotypes on both sides, and perpetuate the gap they aim to bridge.
Assumption 2: that policy is usually not based on evidence
Despite the increased literature in the area, there is a surprising lack of evidence about how much evidence policymakers use. Studies have largely reported policymakers’ perceptions of their usage (e.g., ), acknowledge that it is impossible to tell how much evidence was used by policy participants , or rely on self-reported measures . EBP researchers have tended to interpret this absence of any contradictory evidence as a confirmation of their belief that policymakers do not use evidence. This is both grossly unfair to policymakers who have been shown to draw on a wide range of information sources , hugely over-simplifies the relationship between evidence and policy, and, of course, contradicts the avowed principles of EBP researchers; viz, that beliefs ought to be based on evidence .
Implicit in the ‘barriers and facilitators’ approach is an assumption that if these factors were alleviated, research uptake would increase. However, this is to miss the key point, which is that most research in the area studies the use of research evidence by policymakers, not what knowledge or information policymakers use. This subtle shift in emphasis opens up new avenues of enquiry. Other information than research evidence might be more relevant and timely, two factors seen as top facilitators for policymakers’ use of evidence . Policymakers may prefer to use local information or intelligence such as patient or practice level data, or that held by local councils (e.g., datasets of rent, crime, and transport) . It seems likely that these sources of information have been undervalued by evaluation methodologists, who often value trial data above other types. A more naturalistic approach using empirical methods to study policymakers in vivo, would conceive of evidence as one of many influences on a decision.
Assumption 3: that use of more research evidence by policymakers would lead to ‘better’ policy
We are not the first to note that “[t]he assumption that the use of evidence would improve the outcome of the policy process remains relatively untested by any form of empirical analysis” [96–98]. The absence of robust evaluation evidence showing that evidence utilisation actually leads to better outcomes is widely admitted. The bulk of the intervention evidence in EBP uses only research utilisation or uptake as an outcome (or, in some cases, merely attitudes and intentions regarding research use). Nonetheless, it is still widely claimed that decisions made in partnership between “politicians and researchers & lay people are more likely to result in positive health outcomes”  and many researchers continue to advocate for increased use of research evidence .
Such claims, where they are not treated as automatically self-evident, are usually supported either by anecdotal cases of increased evidence use leading to better outcomes, or by studies of the impacts of evidence use on process measures such as transparency of decision-making . However, the value of process-oriented goals is surely questionable, if they cannot be shown to lead to improved health, wellbeing, social, or other outcomes for the putative beneficiaries of the policy in question. The typologies of ‘research impact’ which have dominated much work in this area (e.g., ) are of limited value without a more open debate about the correct metrics for evaluating EBP, based on a realistic view of the currently existing evidence base . Moreover, much of the commentary around the ‘impact agenda’ focuses on the aspect of academic performance management, without wider examination of its connection to policy and knowledge practices and theories .
Even in the absence of robust evaluation data, it is clear that many of the existing theoretical rationales for how evidence utilisation is supposed to improve outcomes are inadequate. If the pipeline model of research use were correct, it would be possible to demonstrate the impact of research on policy and the value of research would be judged on its contribution to policy and its quantifiable impact . This model “fails the practitioner because the literature on which guidelines are based constitutes an unrepresentative sample of the varied circumstances and populations in which the intervention might be usable or unusable” .
New directions for EBP research
Above, we describe three assumptions commonly found in the EBP literature. We critically discuss the reasons we believe these assumptions are flawed, and show how the existing research conducted on the basis of these assumptions is likely to fail to answer the most pressing problems in EBP. Here, we describe how these assumptions can form the basis for a new programme of research aiming to understand the relationship between science and policy. We explore the implications of re-framing future studies in a new direction, and suggest more explorative perspectives and participative methods.
Firstly, the assumption that the policy-evidence ‘gap’ needs ‘bridging’. Most of the studies identified perpetuate the division between the ‘two communities’ in one way or another. Approaching researchers and policy-makers separately and asking them for their accounts of evidence use may be likely to produce these conflicting accounts; similarly, asking researchers about their perceptions of what policy-makers do may not be the most sensitive way of exploring policy processes. It would be more interesting, and more novel, to approach policymakers from an unprejudiced stance, to describe their activities, and to identify how they populate policy areas and steer policies through . Of course, we are not the first to suggest this [56, 97] – but these studies are generally the exception rather than the rule.
Secondly, the assumption is often held that policy is usually not based on any data, and policy requires research evidence, preferably evaluative intervention research. By concerning themselves with questions such as “how [can] the tension between scientific rigour and timely relevance to policy-making be handled” , EBP researchers often fail to acknowledge their lack of knowledge about forms and models of the impact and contributions of evidence to policy processes, which can lead to the creation of unhelpful straw men.
Finally, the unspoken corollary to both these assumptions is that greater use of evidence in policy-making will produce ‘better policy’ and better outcomes at a population level. Leaving aside the question of what constitutes ‘better’ in a self-evidently political and therefore value-driven terrain, for researchers to convincingly argue for the increased use of evidence in policy making, they must be able to demonstrate the benefits of doing so. The growth of the ‘applied research’ sector claims to address this, but often restricts output to vague and untested policy and research recommendations, about which there is no evidence of effectiveness . If the effects of these policy and research recommendations are not evaluated these could be misguided at best .
Therefore, we argue that the following issues are of outstanding importance and could form the basis for a new agenda of EBP research:
Refocus research on influences on and processes of policy rather than how to increase the amount of evidence used. Researchers in political and policy studies, anthropology and history of policy, and science and technology studies have provided a wealth of insights and rich empirical data on the functioning (or otherwise) of the policy-making process [13, 53, 56, 68–70, 88, 105–108]. Understanding the daily lives and activities of policy actors can bring fresh insights into how ‘evidence’ is conceptualised, the potential roles it may play, and how it fits with the other drivers and triggers which affect policy [95, 109]. Understanding the roles of exceptional individuals, such as policy entrepreneurs, and networks in the policy process is also recognised as a key research area [3, 110, 111].
Dialogue between these fields has not been very extensive or productive so far, largely due to scepticism on the part of policy scientists, e.g., , and a lack of engagement with this body of theory and empirical data, with recent exceptions [112, 113]. To take only one example, policy researchers observe that policies rarely have a consistent and well-defined goal or aim; rather, “solutions become joined to problems” in a provisional and largely haphazard way, with the same policy taking on different goals at different times or in different contexts [50, 114]. If this is the case, the question of how researchers can evaluate whether or not the policy has attained its goal, and use the results of this evaluation to inform future policy development, is largely moot.
Determine what information and evidence is normally used as part of policy processes. As we point out above, it is likely that the needs and practices of policymakers are rarely the subject of rigorous study, and are certainly more complex and nuanced than can be captured in surveys. Using ‘research’ and ‘policy’ as one-size-fits-all concepts underestimates the variety of activities and outputs involved in each type of process. A legislative manoeuvre is very different from a local tailoring of licensing hours; applied health research aiming to develop interventions about a specific condition is worlds away from contemplative studies of models of theories of change. Elucidation of the relationships at both ends of the spectra ought to be a research priority, in place of the dichotomising of activities as ‘research’ or ‘policy’. Furthermore, attention to context is vital; pressures faced by researchers and policymakers in low- and middle-income countries may be very different from those in Western settings. Finally, as argued above, although similar barriers and facilitators are often identified across policy settings , there is variation of practices and processes across policy areas. Analysis of these variations would be a whole research agenda in itself. A subtler interpretation of context, ‘policy’, and ‘research’ are necessary to understand processes, influences and impacts, and indeed to develop any meaningful abstractions and generalisations – should these be possible, which is by no means certain. A compromise between the complexity of policy making and development of useful frameworks needs to be found.
These questions are likely to require a broader range of methodologies than usually applied. Experimental, ethnographic, and conceptual studies all need to be applied to understand the impact of evidence on policy and policy processes more generally. Novel theoretical approaches could be phenomenological, psychosocial, or interpretive [115–117].
Develop conceptual clarity around and metrics to evaluate ‘impact’ of research on policy and populations. Without clear methods to understand how policy works and how it changes in response to information, it will be impossible for researchers to know whether they have had an effect on policy. Research usage could take many forms, from agenda-setting, to provision of policy options, to challenging debates, or refuting arguments. Impact could therefore be a change in policy, consistency in policy, changes in population level outcomes. Measures proposed thus far tend to focus on citations or mentions of work by policymakers. We feel this addresses only a narrow aspect of potential impacts, namely awareness of research, which may not translate into action. Furthermore, this metric-led approach tends to ignore the indirect means by which research and evidence of all kinds fit into policy – whether by sustaining the status quo, or by leading to decisions for change. More attention to the variety of impacts and effects leading from research may help to develop this debate.
Finally, our analysis of the literature suggests that new methods and organisations aiming to bring the processes of research and policy closer together are likely to further our understanding of the relationship between these two types of activities. Co-creation and co-production of knowledge are lauded as a more democratic, and potentially more useful, type of learning activity than many other knowledge exchange events . If universities were to provide assistance for local policymakers in the analysis of existing data, a relationship of mutual benefit could start to develop – an end in itself, according to reviews of barriers and facilitators of evidence use [30, 34]. Moreover, such organizations would provide natural laboratories for studying the role of institutional and organizational factors on the practices of policy formation and implementation, noted in our review as likely to affect evidence use.
Perhaps due to the political, financial, and ethical pressures on health policymakers to make good decisions, health policy leads the way in forming collaborative organisations to conduct research. Funders in the UK and Netherlands have developed specific types of collaborative engagement organisations (e.g., Collaborations for Leadership in Applied Health Research and Care, CLARHCs, or the health funder ZonMw), which aim to bring practitioners and researchers together for mutual benefit. There are examples of institutions which specifically allow researchers and policymakers to learn about each other’s priorities and ways of working [56, 119]. In general, however, the literature above suggests that there are insufficient opportunities and incentives to form links with policymakers directly.
The existing literature on EBP has certainly contributed to the desirable outcome of better policy decisions and acceptance that evidence ought to play a role in those decisions. Now, we believe it is time for researchers to reflect on their assumptions, develop new perspectives, and use other methods to tackle the problems of EBP. There is a common lack of clarity about what researchers understand as ‘policy’, which can encompass decision making, project implementation and evaluation, and service reconfiguration. There is, in general, little evidence about management and organisation, despite these being potentially major factors affecting the policy process . The assumptions governing the design and outcomes of research into policy will probably be significantly different in different institutional areas. It is often not clear what constitutes ‘a decision’, nor who is involved in it, or whether research evidence is relevant or timely. This muddles and prevents any engagement with discussions about what constitutes good and bad policy; or indeed how evidence ought to be used.
Rather than attempting to develop a one-size-fits-all (pipeline) model, research in this area should revert to observational methods with in-depth descriptions of practices and their inherited processes and provide an empirical basis for theoretical development which can inform future activities. Instead of repeating studies of perceptions of barriers and facilitators of use of research evidence, appropriate methods must be used to answer questions about when, why, how, and who finds what type of knowledge sound, timely, and relevant at different stages of the policy cycle.
The position of researchers who wish to influence policy is untenable unless there is engagement with the questions outlined above. We would also argue that researchers who advocate for change (for example, via policy and practice recommendations) without evaluating the (likely) impact of these changes may have a limited effect at best. At worst, they create distance between policy and research by demonstrating an understanding of the context within which they would like their newly-generated knowledge to be used. Without understanding the complex processes of policy and knowledge mobilisation, researchers who make policy and practice recommendations may simply be ignored. Ultimately, the role of researchers is not to judge the ‘quality’ of policy making on the basis of how much of their research is used. This stance is both unhelpful and divisive, blinding researchers to the important questions raised above regarding the types of information used, by whom, for what purpose, and under which circumstances. Rather, our role as scientists ought to be to investigate the processes surrounding the use of evidence and policy activities more widely and to disseminate findings in order to help others make informed decisions, of all kinds.
Evidence based medicine
Evidence based policy.
Cooksey D: A Review of UK Health Research Funding. 2006, London: Stationery Office
Fox DM: Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Affairs. 2005, 24: 114-122.
Mitton C, Adair C, McKenzie E, Patten S: Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007, 85: 729-768.
Cochrane A: Effectiveness and Efficiency: Random Reflections on Health Services. 1972, London: Nuffield Provincial Hospitals Trust
Champagne F, Lemieux-Charles L: Using Knowledge and Evidence in Health Care: Multidisciplinary Perspectives. 2008, Toronto: University of Toronto Press
Guyatt G, Cairns J, Churchill D: Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA. 1992, 268: 2420-2425.
Barton S: Which clinical studies provide the best evidence?. BMJ. 2000, 321 (7256): 255-256.
Chalmers I: If evidence-informed policy works in practice, does it matter if it doesn’t work in theory?. Evidence Policy. 2005, 1: 227-242.
Glasziou PP, Irwig LM: An evidence based approach to individualising treatment. BMJ. 1995, 311: 1356-1359.
Guyatt GH: Users’ guides to the medical literature. JAMA. 1995, 274: 1800-
Kogan M, Henkel M, Britain G: Government and Research: The Rothschild Experiment in a Government Department. 1983, Portsmouth, NH: Heinemann Educational Books
Hanney SR, González-Block MA: Evidence-informed health policy: are we beginning to get there at last. Health Res Policy Syst. 2009, 7: 30-
Duffy MP: The Rothschild experience: health science policy and society in Britain. Sci Tech Hum Val. 1986, 11: 68-78.
Government Office for Science: Engaging with Government: Guide for Academics, BIS/11/1360. 2011, [https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/283118/11-1360-guide-engaging-with-government-for-academics.pdf]
Government Office for Science: Engaging with Academics: How to Further Strengthen Open Policy Making – A Guide For Policy Makers, BIS/13/581. 2013, [https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/283129/13-581-engaging-with-academics-open-policy-making.pdf]
Martin BR: The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster?. Res Eval. 2011, 20: 247-254.
Pettigrew AM: Scholarship with impact. Br J Manag. 2011, 22: 347-354.
HEFCE: Joint Statement on Impact by HEFCE, RCUK and UUK. 2011, [http://www.hefce.ac.uk/news/newsarchive/2011/news62296.html]
Armstrong R, Waters E, Roberts H, Oliver S, Popay J: The role and theoretical evolution of knowledge translation and exchange in public health. J Public Health. 2006, 28: 384-389.
Dobbins M, Robeson P, Ciliska D, Hanna S, Cameron R, O'Mara L, DeCorby K, Mercer S: A description of a knowledge broker role implemented as part of a randomized controlled trial evaluating three knowledge translation strategies. Implement Sci. 2009, 4: 23-
Ward V, House A, Hamer S: Knowledge brokering: exploring the process of transferring knowledge into action. BMC Health Serv Res. 2009, 9: 12-
Cabinet O, Treasury HM, Open Public S, Rt Hon D, Alexander MP, Rt Hon O, Letwin MP: What Works: Evidence Centres for Social Policy. 2013, London: HM Government
Walshe KWTR: Evidence-based management: from theory to practice in health care. Milbank Q. 2001, 79: 429-457.
Marston G, Watts R: Tampering with the evidence: a critical appraisal of evidence-based policy-making. The Drawing Board: Aust Rev Pub Affairs. 2003, 3: 143-163.
Black N, Donald A: Evidence based policy: proceed with care commentary: research must be taken seriously. BMJ. 2001, 323: 275-279.
Lavis J, Robertson D, Woodside J, McLeod C, Abelson J: How can research organizations more effectively transfer research knowledge to decision makers?. Milbank Q. 2003, 81: 221-248.
Muir Gray J: Evidence-based policy making: is about taking decision based on evidence and the needs and values of the population. BMJ. 2004, 329: 988-989.
Nutbeam D: Getting evidence into policy and practice to address health inequalities. Health Promot Int. 2004, 19: 137-140.
Smith G: How policy informs the evidence. BMJ. 2001, 322: 184-185.
Oliver K, Innvaer S, Lorenc T, Woodman J, Thomas J: A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014, 14: 2-
Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S: The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011, 6: e21704-
Boaz A, Baeza J, Fraser A, European Implementation Score Collaborative Group (EIS): Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011, 4: 212-
Pawson R: Evidence-based Policy: A Realist Perspective. 2006, Thousand Oaks, CA: Sage Publications Ltd
Innvaer S, Vist G, Trommald M, Oxman A: Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002, 7: 239-244.
Lorenc T, Tyner E, Petticrew M, Duffy S, Martineau F, Philips G, Lock K: Cultures of evidence across policy sectors: systematic review of qualitative evidence. Eur J Public Health. 2014, doi:10.1093/eurpub/cku038
Weiss CH: The many meanings of research utilization. Publ Admin Rev. 1979, 39: 426-431.
Weiss CH, Bucuvalas MJ: Social Science Research and Decision-Making. 1980, New York: Columbia University Press
Green LW, Ottoson JM, Garcia C, Hiatt RA: Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annu Rev Public Health. 2009, 30: 151-174.
Hanney S, Gonzalez-Block M, Buxton M, Kogan M: The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Pol Syst. 2003, 1: 2-
Sabatier PA, Jenkins-Smith H: Policy Change and Learning: An Advocacy Coalition Approach. 1993, San Francisco: Westview Press
Dror Y: Public Policy Making Re-Examined. 1983, New Jersey: Transaction Publishers
Aaserud M, Lewin S, Innvaer S, Paulsen E, Dahlgren A, Trommald M, Duley L, Zwarenstein M, Oxman AD: Translating research into policy and practice in developing countries: a case study of magnesium sulphate for pre-eclampsia. BMC Health Serv Res. 2005, 5: 68-
Hivon ML: Use of health technology assessment in decision making: coresponsibility of users and producers?. Int J Tech Assess Health Care. 2005, 21: 268-275.
Jack SM: Knowledge transfer and exchange processes for environmental health issues in Canadian Aboriginal communities. Int J Environ Res Publ Health. 2010, 7: 651-674.
Priest N: Engaging policy makers in action on socially determined health inequities: developing evidence-informed cameos. Evid Policy. 2009, 5: 53-70.
Vingilis E, Hartford K, Schrecker T, Mitchell B, Lent B, Bishop J: Integrating knowledge generation with knowledge diffusion and utilization: a case study analysis of the Consortium for Applied Research and Evaluation in Mental Health. Canadian J Pub Health. 2003, 94: 468-471.
Sebba J: Enhancing impact on policy-making through increasing user engagement in research. Educational Research and Policy-Making: Exploring the Border Country between Research and Policy. 2007, London & NY: Routledge, 127-143.
Sabatier PA: Theories of the Policy Process. 2007, Boulder, CO: Westview Press
Shore C, Wright S, Per D: Policy Worlds: Anthropology and Analysis of Contemporary Power. 2011, Oxford: Berghahn
Kingdon JW: Agendas, Alternatives, and Public Policies. 2010, Boston: Little, Brown, 3
Weiss CH: Nothing as practical as good theory: exploring theory-based evaluation for comprehensive community initiatives for children and families. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. 1995, Queenstown, MD: The Aspen Institute, 65-92.
Simon HA: Models of Man; Social and Rational. 1957, New York: John Wiley and Sons, Inc.
Lindblom CE: The science of “muddling through”. Publ Admin Rev. 1959, 19: 79-
True JL, Jones BD, Baumgartner FR: Punctuated-equilibrium theory: explaining stability and change in American policymaking. Theories of the Policy Process. 1999, Colorado: Westview Press Boulder, 97-115.
Simon HA: Models of Bounded Rationality: Empirically Grounded Economic Reason. 1982, Cambridge, MA: MIT press, 3
Lomas J, Brown A: Research and advice giving: a functional view of evidence-informed policy advice in a Canadian ministry of health. Milbank Q. 2009, 87: 903-926.
Cairney P: How can policy theory have an impact on policy making? The role of theory-led academic-practitioner discussions. Teach Pub Adminis. 2014, doi:10.1177/0144739414532284
Buse K, Mays N, Walt G: Making Health Policy. 2012, New York: McGraw-Hill International
Stevens A: Telling policy stories: an ethnographic study of the use of evidence in policy-making in the UK. J Soc Policy. 2011, 40: 237-255.
Burris H, Parkhurst J, Du-Sarkodie Y, Mayaud P: Getting research into policy - Herpes simplex virus type-2 (HSV-2) treatment and HIV infection: international guidelines formulation and the case of Ghana. Health Res Pol Syst. 2011, 9 (Suppl 1): S5-
Smith K, Joyce K: Capturing complex realities: understanding efforts to achieve evidence-based policy and practice in public health. Evid Policy. 2012, 8: 57-78.
Weitkamp G, Van den Berg AE, Bregt AK, Van Lammeren RJA: Evaluation by policy makers of a procedure to describe perceived landscape openness. J Environ Manage. 2012, 95: 17-28.
Aoki-Suzuki C, Bengtsson M, Hotta Y: International comparison and suggestions for capacity development in industrializing countries. J Ind Ecol. 2012, 16: 467-480.
Cerveny LK, Blahna DJ, Stern MJ, Mortimer MJ, Predmore SA, Freeman J: The use of recreation planning tools in U.S. Forest Service NEPA assessments. Environ Manag. 2011, 48: 644-657.
Jenkins RAR: Bridging data and decision making: development of techniques for improving the HIV prevention community planning process. AIDS Behav. 2005, 9: S41-S53.
Williams I, McIver S, Moore D, Bryan S: The use of economic evaluations in NHS decision-making: a review and empirical investigation. Health Tech Assess. 2008, 12 (7): 1-175. iii, ix-x
Colon-Ramos U, Lindsay A, Monge-Rojas R, Greaney M, Campos H, Peterson K: Translating research into action: a case study on trans fatty acid research and nutrition policy in Costa Rica. Health Pol Plann. 2007, 22: 363-374.
Hunsmann M: Limits to evidence-based health policymaking: policy hurdles to structural HIV prevention in Tanzania. Soc Sci Med. 2012, 74: 1477-1485.
Kok MO, Schuit AJ: Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Pol Syst. 2012, 10: 1-16.
Wehrens R, Bekker M, Bal R: The construction of evidence-based local health policy through partnerships: research infrastructure, process, and context in the Rotterdam ‘Healthy in the City’ programme. J Publ Health Pol. 2010, 31: 447-460.
Wehrens R, Bekker M, Bal R: Coordination of research, policy and practice: a case study of collaboration in the field of public health. Sci Public Policy. 2011, 38: 755-766.
Berridge V: Health and Society in Britain since 1939. 1999, Cambridge: Cambridge University Press, 38
Elliott H, Popay J: How are policy makers using evidence? Models of research utilisation and local NHS policy making. J Epidemiol Community Health. 2000, 54: 461-468.
McDonough JE: Experiencing Politics: A Legislator’s Stories of Government and Health Care. 2000, Oakland, CA: University of California Press, 2
Haynes A, Derrick G, Redman S, Hall W, Gillespie J: Identifying trustworthy experts: how do policymakers find and assess public health researchers worth consulting or collaborating with?. PLoS ONE. 2012, 7: e32665-
Innvaer S: The use of evidence in public governmental reports on health policy: an analysis of 17 Norwegian official reports (NOU). BMC Health Serv Res. 2009, 9: 177-
Buse K, Mays N, Walt G: Making Health Policy. 2005, Maidenhead: Open University Press
Oliver K, Kislov R: Policy, evidence and theory: contextualising a glossary of policymaking. J Epidemiol Community Health. 2012, Response to A glossary of theories for understanding policymaking. Smith, Katikireddi 67:2 198-202 doi:10.1136/jech-2012-200990
Lasswell HD: Politics: Who Gets What, When, How. 1950, New York: P. Smith
Gold M: Pathways to the use of health services research in policy. Health Serv Res. 2009, 44: 1111-1136.
Taylor-Robinson D, Milton B, Lloyd-Williams F, O'Flaherty M, Capewell S: Planning ahead in public health? A qualitative study of the time horizons used in public health decision-making. BMC Public Health. 2008, 8: 415-
Exworthy M, Hunter D: The Challenge of joined-up government in tackling health inequalities. Int J Publ Admin. 2011, 34: 201-212.
Sanderson I: Evaluation, policy learning and evidence-based policy making. Public Administration. 2002, 80: 1-22.
Dopson S, Locock L, Gabbay J, Ferlie E, FitzGerald L: Evidence-based medicine and the implementation gap. Health (London). 2003, 7: 311-330.
Gorissen WHM, Schulpen TWJ, Kerkhoff AHM, van Heffen O: Bridging the gap between doctors and policymakers: The use of scientific knowledge in local school health care policy in the Netherlands. Eur J Public Health. 2005, 15: 133-139.
Armstrong R, Doyle J, Lamb C, Waters E: Multi-sectoral health promotion and public health: the role of evidence. J Public Health. 2006, 28: 168-172.
Choi BCK, Pang T, Lin V, Puska P, Sherman G, Goddard M, Ackland MJ, Sainsbury P, Stachenko S, Morrison H, Clottey C: Can scientists and policy makers work together?. J Epidemiol Community Health. 2005, 59: 632-637.
Jasanoff S: States of Knowledge: The Co-Production of Science and the Social Order. 2013, London: Routledge
Farrington : What works in preventing crime? Systematic reviews of experimental and quasi-experimental research. Ann Am Acad Pol Soc Sci. 2001, 578: 8-
Ritchie J, Spencer L: Qualitative data analysis for applied policy research. Analyzing Qualitative Data. Edited by: Bryman A, Burgess R. 1999, London: Routledge
Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M: Evidence for public health policy on inequalities: 1: The reality according to policymakers. J Epidemiol Community Health. 2004, 58: 811-816.
Ettelt S, Mays N: Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011, 16 (Suppl 2): 48-60.
Coleman PN: Influence of evidence-based guidance on health policy and clinical practice in England. Qual Health Care. 2001, 10: 229-237.
Ritter A: How do drug policy makers access research evidence?. Int J Drug Pol. 2009, 20: 70-75.
Oliver K, de Vocht F: Defining ‘evidence’: a survey of public health policy makers’ needs and preferences. Eur J Public Health. 2013, In press
Bullock H, Mountford J, Stanley R, Britain G: Better policy-making. 2001, London: Cabinet Office
Boaz A, Grayson L, Levitt R, Solesbury W: Does evidence-based policy work? Learning from the UK experience. Evid Policy. 2008, 4: 233-253.
Cameron A, Lart R, Salisbury C, Purdy S, Thorp H, Stewart K, Peckham S, Calnan M, Purdy S, Thorp H: Policy makers’ perceptions on the use of evidence from evaluations. Evid Policy. 2011, 7: 429-448.
Macintyre S, Chalmers I, Horton R, Smith R: Using evidence to inform health policy: case study. BMJ. 2001, 322: 222-225.
Banks G: Evidence-Based Policy Making: What is it? How do we get it?. 2009, Canberra: Australian Government Productivity Commission
Davies TO, Nutley S, Walter I: Approaches to Assessing Research Impact. 2005. Report of the ESRC Symposium on Assessing the Non-Academic Impact of Research. 2005, [http://www.esrc.ac.uk/_images/non-academic_impact_symposium_report_tcm8-3813.pdf]
Penfield T, Baker MJ, Scoble R, Wykes MC: Assessment, evaluations, and definitions of research impact: A review. Res Eval. 2014, 23: 21-32.
Van Kammen J, de Savigny D, Sewankambo N: Using knowledge brokering to promote evidence-based policy-making: the need for support structures. Bull World Health Organ. 2006, 84: 608-612.
Coe R, Fitz-Gibbon C, Tymms P: Promoting Evidence-Based Education: The Role of Practitioners. 2000, Cardiff, Wales: Roundtable presentation to the British Educational Research Association’s Annual Conference
Berridge V: Passive smoking and its pre-history in Britain: policy speaks to science?. Soc Sci Med. 1999, 49: 1183-1195.
Bijker WE, Bal R, Hendriks R: The Paradox of Scientific Authority: The Role of Scientific Advice in Democracies. 2009, Cambridge, MA: MIT Press
Cairney P: A ‘multiple lenses’ approach to policy change: the case of tobacco policy in the UK. British Politics. 2007, 2: 45-68.
Jasanoff S: The Fifth Branch: Science Advisers as Policymakers. 1994, Cambridge, MA: Harvard University Press
Lipsky M: Street-Level Bureaucracy. 2010, New York: Russell Sage Foundation
Checkland K, Snow S, McDermott I, Harrison S, Coleman A: Animateurs and animation: what makes a good commissioning manager?. J Health Serv Res Policy. 2012, 17: 11-17.
Oliver K, de Vocht F, Money A, Everett MG: Who runs public health? A mixed-methods study combining network and qualitative analyses. J Public Health. 2013, 35: 453-459.
Head BW: Three lenses of evidence - based policy. Aust J Publ Admin. 2008, 67: 1-11.
Head BW: Reconsidering evidence-based policy: key issues and challenges. Pol Soc. 2010, 29: 77-94.
Cohen M, March J, Olsen J: A garbage can model of organisational choice. Admin Sci Q. 1972, 17: 1-25.
Fotaki M: Why do public policies fail so often? Exploring health policy-making as an imaginary and symbolic construction. Organization. 2010, 17: 703-720.
Yanow D: A Policy ethnographer’s reading of policy anthropology. Policy Worlds: Anthropology and the Analysis of Contemporary Power. 2011, New York: Berghahn Books, 300-313.
Yin RK: Case Study Research: Design and Methods. 2014, Thousand Oaks, CA: Sage Publications
Bourke A: Universities, civil society and the global agenda of community-engaged research. Globalisation, Soc Educ. 2013, 11: 498-519.
Dobrow MJ, Goel V, Lemieux-Charles L, Black NA: The impact of context on evidence utilization: a framework for expert groups developing health policy recommendations. Soc Sci Med. 2006, 63: 1811-1824.
Wendt C: Mapping European healthcare systems: a comparative analysis of financing, service provision and access to healthcare. J Eur Soc Pol. 2009, 19: 432-445.
We thank colleagues who have read and commented on the draft version, and the editor and reviewers for their extremely helpful comments.
The authors declare no competing interests. This study was not supported by any funding agency.
All authors contributed to the conception, design, and analysis of the data which underpins this critical review. All authors contributed to the development of the argument and the final manuscript. All authors read and approved the final manuscript.
About this article
Cite this article
Oliver, K., Lorenc, T. & Innvær, S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Sys 12, 34 (2014). https://doi.org/10.1186/1478-4505-12-34
- Critical analysis
- Evidence-based policy
- Knowledge utilization
- Science and technology studies