Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy?

The health sciences are well represented in studies of the ‘evidence-policy gap’, in which academics describe their attempts to overcome ‘barriers’ between the production of evidence by scientists and its use by policymakers. Oliver et al.’s [1, 2] systematic reviews draw insights from 126 health policy studies published from 2000–2012. The most frequently-reported barriers relate to problems with disseminating high quality information effectively, namely the lack of time, support, resources and incentives for scientists to engage in dissemination. These studies suggest that scientific evidence is often not presented at the correct time and scientists are unable to anticipate a demand for information to solve a very specific problem quickly. Further, policymakers lack the research skills to understand scientific evidence. More generally, there are different scientific and policymaker cultures [3]. These insights generally come from a small number of countries, including the United States, United Kingdom, Canada, Australia and New Zealand.

There are two key problems with this literature. First, very few studies draw on policy theory or other forms of knowledge of the policy process [4]. Most are based on personal practitioner perspectives generated through, for example, surveys, focus groups and academic-policymaker workshops. They generally have a particular reference point, namely evidence-based medicine (EBM) [2]. The EBM agenda is (1) to gather the best evidence on health interventions, based on a hierarchy of methods, in which randomised control trials and their systematic review are at the top, and (2) to ensure that it has a direct impact on practice, to exhort practitioners to replace bad interventions. Individual clinicians have high discretion on how to use the evidence, but based on a common ideal and understanding of evidence quality. Consequently, the studies present limited findings because (1) their source material is restricted primarily to interviewees with minimal knowledge of policymaking, and (2) potential solutions are based on a misperception of the policy process, built on EBM assumptions, not policy theory [5].

The most-frequently stated solutions to ‘barriers’ highlight the limits to such a theoretical analysis. For example, to address the problem of supply, studies highlight the need for ‘improved dissemination’ to ensure that policymakers pay attention to, and understand, the best evidence [1]. Too few studies recognise that policymakers will not share the sense that there is a hierarchy of evidence. Too many assume that better dissemination will prompt policymakers to think, like scientists, that the scientific evidence alone is persuasive or self-evident.

To address demand, the most-proposed solution is to develop scientific competence in government. Too many studies assume that it is realistic to produce a captive audience of policymakers willing to invest the time to prioritise and understand scientific evidence. This approach is at odds with the less rigid ways in which policymakers use many forms of evidence [3, 6–9].

Second, too few studies recognise the role of values in politics. Instead, an often implicit and untested assumption is that policymaking should be as ‘evidence based’ as medicine, which is at odds with the more common starting point, in the study of politics, to produce a democratic system which translates competing societal values and preferences into policy solutions. One may desire a political system based on value judgements and evidence, but should recognise and address the trade-offs between these aims, and that the production of evidence is also an inherently value-driven process.

To develop new perspectives on these questions, we identify insights from secondary data, namely systematic reviews [1], critical analysis [2] and policy theories relevant to evidence-based policymaking [3, 10, 11]. The studies are drawn primarily from countries such as the United States, United Kingdom, Canada, Australia and New Zealand. We combine empirical and normative elements to identify the ways in which scientists can, do and should try to influence policy. Our case study draws partly from an ESRC-funded study based on 30 anonymous semi-structured interviews with civil servants in the Scottish and United Kingdom governments. We use this this data to produce models of evidence-based policymaking (described in more depth in [12]), then analyse the dilemmas that they present for scientists.

To provide context for the second question, we place the possible responses that scientists could make on a spectrum, from ‘purist’, or the insistence on the primacy of RCT evidence and fidelity to successful interventions during policy implementation, to the ‘diplomatic’, treating evidence as part of a success story, encouraging the autonomy of practitioners to ‘co-produce’ and modify policy solutions to meet the values and specific circumstances of service users. We examine the implications of these practices on scientific research, outlining pragmatic responses to combine scientific evidence with governance principles.

Most policy theories explore the implications of two basic insights, namely that policymakers are limited by ‘bounded rationality’ [13] and share power with many actors in complex policymaking systems [3, 10, 14]. In part, bounded rationality relates to the fact that policymakers do not have the ability to gather and consider all evidence relevant to policy problems. Instead, they employ two shortcuts – ‘rational’, pursuing clear goals and prioritising certain sources of information, and ‘irrational’, drawing on emotions, gut feelings, beliefs and habits to make decisions quickly.

The key problem with many health studies is that they focus on the first short cut. They identify the problem of uncertainty and incomplete information, seeking to solve it by creating hierarchies of evidence and improving the supply of information to policymakers. They ignore the role of manipulation and persuasion to reduce ambiguity by establishing a dominant way to ‘frame’ policy problems. The latter can determine the demand for evidence.

From such discussions, we can provide a first key message for scientific advocates, namely to recognise the tendency of policymakers to base judgements on their well-established beliefs and shortcuts based on their emotions and familiarity with information. On that basis, consider how to reduce ambiguity, to persuade policymakers to frame a problem primarily in one particular way and, therefore, to demand scientific evidence to help solve that problem.

Persuasion or ‘framing’ strategies are effective because they appeal to the emotions and the familiar, combining facts with emotional appeals to prompt lurches of attention; telling simple and easily understood stories which manipulate people’s biases, apportioning praise and blame and highlighting the moral and political value of solutions; and recognising the importance of interpreting new scientific evidence through the lens of the beliefs and knowledge of influential actors.

Without this focus on the ways in which policymakers understand and respond to problems, scientists will be unable to exert influence, responding only to sudden policymaker demand for evidence-based solutions to a pre-defined problem [10, 15–27].

However, this first message also produces the first practical or ethical dilemma – how far should scientists go to persuade policymakers to act on their evidence? This knowledge allows us to decide how far we should go to secure influence, from the ‘pure scientist’ providing evidence with little thought for its application and the ‘honest broker’ prepared to engage with stakeholders to define policy problems [28], towards engaging in persuasion and/or manipulation to generate attention for problems and evidence-based solutions [29].

There is no definitive answer to this question. Instead, there are factors to consider, including the trade-off between (1) being manipulative and hyperbolic, to generate attention in the short term, and (2) maintaining an image of objectivity, to generate strong relationships in government based on reliability, expertise, trust and ‘independence’. Further, there are many compromise solutions, including a strategy built on framing the implications of evidence according to the stated beliefs of policymakers and building up a reputation for reliability within many policymaking venues.

Any competition to frame issues takes place in a complex policymaking environment or system [61]. This context is not described well by the ‘policy cycle’ – an understanding built on the idea that policymaking can be separated into discrete stages (agenda setting, policy formulation, implementation, legitimation, evaluation and succession) – which remains popular within health scholarship despite being rejected by policy scholars as a poor explanation of policymaking [2, 3, 10, 14, 62–66].

In its place are theories which identify a policy ‘environment’ which relates to the interaction between five factors, namely the many policy actors operating in multi-level systems, pursuing ideas (using their beliefs and knowledge to influence how policymakers think), adapting to institutions (the formal and informal rules of political systems and organisations), networks (between policymakers and influential actors), and the impact of socioeconomic context and events [3, 27, 67–71]. Further, some theories identify complex policymaking systems in which, for example, the same inputs of evidence can receive no, or disproportionate attention, and policy outcomes often ‘emerge’ in the absence of central government control, which makes it difficult to know how, and to whom, to present evidence [3, 72].

This analytical shift from cycles to environments and systems helps us reject the notion that there is a core group of policymakers at the heart of the process, analysing and making policy from the ‘top down’, and that scientists promoting evidence can focus their efforts at a single point of problem definition or policy formulation [3].

Rather, scientists compete with many influential actors to present evidence to secure a policymaker audience at many levels of government. Support for evidence-based solutions varies according to which organisation takes the lead and how its rules encourage policymakers to understand the problem. Some networks are close-knit and difficult to access because bureaucracies have operating procedures that favour some sources of evidence and participants over others. There is a language in institutions and networks that takes time to learn. It indicates which ways of thinking are in good ‘currency’, including a dominant way to ‘frame’ a problem and limit discussion to certain solutions. Well-established beliefs provide the context for policymaking – new evidence on the effectiveness of a policy solution has to be accompanied by a shift of attention and successful persuasion. Social or economic ‘crises’ can prompt lurches of attention from one issue to another, and some evidence can be used to encourage that shift. However, most policy change is minor; it is rare for events to combine with new evidence to produce quick and radical policy change [10].

From such discussions, we can provide a second key message for scientific advocates – learn ‘where the action is’, and be prepared to engage in a long-term strategy to be in a position to influence policy. In other words, identify ‘the action’ at several levels of government, learn the ‘rules of the game’ in institutions and networks (or the ‘venues’ in which policymaking takes place), form coalitions with like-minded actors, and work with the ‘policy entrepreneurs’ possessing the skills to exploit ‘windows of opportunity’ to give policymakers the motive and opportunity to adopt new solutions [3, 8, 23, 24, 27, 73–78].

In most cases, these coalitions will be with many actors with different ways of thinking about policymaking, engaging in politics to turn their beliefs and values into policy, often unaware or unsupportive of EBM’s hierarchy of evidence, and likelier to refer to governance principles other than evidence-based policymaking.

Consequently, this second message produces a second practical and ethical dilemma – how far should you go to defend a hierarchy of evidence to deliver policy solutions? It involves the extent to which policy actors should accept and adapt to policymaking complexity and the diffusion of power, or seek to change some aspects and assert their claim for greater power.

For example, central governments have some incentives to try to reassert central control because they are held accountable in elections and parliaments for policy outcomes, regardless of their responsibility [14]. Yet, they also recognise the need to share decision-making with actors at multiple levels of government, including at the point of policy delivery. This produces a form of government in which the co-production of policy solutions is based partly on evidence and partly on the principles underpinning local governance, including ‘localism’ (central governments recognise the role of partly autonomous local authorities or partnerships) and service user-driven services (based on their values and preferences) [12].

In that context of multi-level and delegated policymaking, scientists may support phrases such as ‘co-production of research’ superficially (who would not want to involve all relevant actors in the production of knowledge?), but also recognise the need for tough choices to ensure the application of evidence-based policy solutions. They need to engage with two debates simultaneously, on EBM and localism.

Second, central governments face the trade-off between national and local policymaking [91]. Their national strategies can include reference to uniform delivery standards, to avoid a ‘postcode lottery’, and encourage local autonomy and policy flexibility, to recognise the legitimacy of local policymaking and encourage community or service user participation. For some interventions, they may favour uniformity of delivery and fidelity to a programme. For others, they express a preference for localism built on arguments, including one size does not fit all, local authorities have their own electoral mandates, a policy will not succeed without local ‘ownership’, and/or local public bodies need to adapt to quickly changing local circumstances.

Consequently, there is an inextricable link between debates on the selection of evidence-based solutions/interventions and the correct choice of governance arrangements. This involves a two-part decision in which policymakers combine (1) their chosen method to identify and gather knowledge of successful policy solutions, with (2) their judgement on how to work with other actors to ‘scale up’ success.

In this context, consider the scale of the task to ensure the primacy of RCTs and/or a hierarchy of evidence in policymaking. It involves (1) persuading many actors in many central and local venues that (2) there is a well-established best way to produce scientific evidence, and therefore (3) that scientific forms of knowledge should be prioritised in policymaking. This is a two-pronged dilemma – should you devote necessarily high resources to this line of persuasion, and should you go to great lengths to dismiss competing understandings of good policymaking?

To demonstrate the issues central to dilemma 2, we outline a spectrum of approaches to answer the ‘how do you scale up success?’ question (see [12] for a summary table). It allows us to demonstrate the implications for evidential hierarchies when power is spread across multiple levels of government. We show the constraining effect of a rigid attachment to ‘fidelity’ to policy interventions on the ability of local communities to modify policy solutions to reflect their values and preferences. This outcome may only be appropriate in some cases. We compare it with an approach that favours local flexibility and governance principles at the expense of scientific evidence. The contrast allows us to highlight the political choices with which advocates of evidential hierarchies need to engage. We use examples of initiatives pursued by the United Kingdom and Scottish central governments, each of which enact or negotiate the delivery of policy interventions with a variety of elected local authorities and unelected health and social care organisations.

We identify two implications of policy theory for scientists, namely to address ambiguity and complexity. From these, we identify key messages for policy actors. First, be realistic about how policy decisions are made, and adapt persuasion techniques accordingly. This requires scientists to decide how far they should go to persuade policymakers: when does persuasion tip into cynical manipulation? Second, be prepared to engage in long-term relationships to build coalitions for change. This prompts scientists to decide when to defend the ‘hierarchy’ or primacy of evidence and when to accept the value of other sources of ‘good’ policymaking. Further, while improvement science appears to offer a happy marriage between these approaches, in practice it offers only one step towards negotiating the trade-offs between RCT-led and story-telling approaches.

Our discussion takes scientists away from the lazy assumption that elected policymakers are the villains of this piece because they, for example, do not understand RCTs and the need for RCT-driven evaluations, do not recognise a hierarchy of evidence in which the systematic review of RCTs represents the gold standard, and/or are unwilling to overcome ethical dilemmas about who gets to be in/out of the control group of a promising intervention. There are also academics and professions who remain sceptical of the value of RCTs in policymaking, have different views on the hierarchy of evidence and/or who promote different ways to gather and use comparable policy-relevant evidence [83].

Evidence-based policymaking is not just about the need for policymakers to understand how evidence is produced and should be used. It is also about the need for academics to reflect on the assumptions they make about the best ways to gather evidence and put the results into practice, in a political environment where other people may not share, or even know about, their understanding of the world; and the difference between the identification of evidence on the success of an intervention, in one place and one point in time (or several such instances), and the political choice to roll it out, based on the assumption that national governments are best placed to spread that success throughout the country.

We have largely discussed extremes or ideal types to highlight key issues. In practice, people may be willing to compromise or produce pragmatic responses to the need to adapt research methods to real world situations. In that context, this kind of discussion should help clarify why scientists may need to work with policymakers or practitioners to produce a solution that each actor can support and why that solution may not be ‘evidence based’ in the way that scientists understand that phrase.