A major task for the NOKC as an independent government organization is to provide knowledge that will underpin health policy decisions [19]. Institutional strategies are clearly influenced by EBM values and methods [17], installing SRs as the most valued format for knowledge delivery [32]. From a total sample of 151 SRs published by the NOKC in the period 2004–2013, our purposive subsample served to demonstrate that most of these reports advised major caution in relation to their conclusions owing to the quality or relevance of the underlying documentation. Although our case study did not include a systematic investigation of uptake and policy consequences, it became apparent that SRs are not appropriate universal tools for health policy decisions. Below, we discuss the impact of these findings.
EBM in clinical decision-making
EBM was developed as a strategy to inform patient care as well as health policy decisions. We are not the first to suggest that the evidence hierarchy of EBM, which gives precedence to quantifiable and supposedly universal knowledge, may be too confined to meet the needs of clinical decision-making in respect of the individual patient [33,34]. Resistance to this confined conception of evidence was expressed by members of clinical disciplines where experientially based, tacit knowledge is required, as distinct from the formal knowledge offered by science [35]. Similar arguments regarding the nature of knowledge had previously been articulated from general practice researchers, with regard to the limitations of universal knowledge and the impact of particular and individualized knowledge in clinical practice [33,36-40]. Approaching clinicians, the NOKC has invested considerable resources in implementation, with courses, websites and organized access to research literature for healthcare professionals. The centre has conceptualized a platform for knowledge-based practice in which mainstream EBM is integrated with the experience-based and user-based knowledge in a local context [17].
Pope described EBM as a contemporary social movement with a high profile, although not overly successful [35]. After a decade of operation, the values of the EBM movement and the corresponding evidence hierarchy are still embraced by the NOKC [17]. The original idea of integrating individual clinical expertise with best evidence [6] seems to have gradually vanished in the shadow of meta-analyses, RCTs and algorithmic rules [34,41]. Procedures for the assessment and synthesis of qualitative evidence have recently been included among EBM standards and tools by the Cochrane Collaboration [42], but such approaches were not prominent in the sample of SRs we explored.
We find it plausible that EBM principles applied in SRs that synthesize current research knowledge in a standardized and transparent format may well enhance clinical decision-making. However, the tool seems most appropriate and capable of providing a clear answer when the research question fits the hierarchy of evidence and the PICO formula [9,12]. A typical example would be whether medication X is a better treatment than medication Y for patients suffering from a single and well-defined disease, as in the prevention of eye infection after cataract surgery [43]. Another example of the tool’s appropriateness would be to ask whether test Z as applied to a defined population will have an impact on morbidity or mortality (disease-specific or all-cause, respectively) for tests and conditions such as prostate-specific antigen and prostate cancer [44].
In clinical practice, however, such clear-cut decision spaces rarely exist, especially in the primary healthcare setting, where multimorbidity is commonplace [45]. This is probably one important reason for the limited compliance among general practitioners to EBM-based guidelines [46]. The challenges of EBM-based clinical decision-making become more apparent in complex and behavioural interventions in more heterogeneous populations, as in most of the sample we examined.
EBM as a tool to guide health policy decisions
Our exploration of EBM indicates its shortcomings as a tool for clinical practice, in that its universal and standardized evidence base fails to address the particular question and context for decision-making. These concerns are no less valid in assessing EBM as a tool for policy decision-making (even though the current Cochrane terminology refers to evidence-informed health decision-making) [47]. Dobrow et al. [48] emphasize current conceptual deficiencies and the limited attempts to acknowledge the role of context in evidence-based decisions. To understand the complex adaptive nature of health systems, the context, acceptability and feasibility of interventions must be highlighted [49]. Mixed-methods approaches have been proposed as a means of studying complex health systems at patient, provider, institutional and system levels [26,42,50,51]. To date, however, our sample of SRs from 2012 indicates little attention to contextual or qualitative matters in NOKC knowledge deliveries.
Even with better research studies and more solid conclusions than those of our sample from 2012, policy decision-making processes demand much more than knowledge translation, defined by WHO as “the synthesis, exchange and application of knowledge by relevant stakeholders to accelerate the benefits of global and local innovation in strengthening health systems and advancing people’s health” [52]. This conception represents the ‘know-do’ gap to be bridged between scientific facts and policymaking as a simple pipeline model, in which incoming evidence underpins decisions [53,54]. Our analysis of the available empirical data suggests that, if such a pipeline were operating, the flow of conclusions and corresponding policy decisions would be very low.
Knowledge deliveries that repeatedly communicate a broad lack of quality evidence across most tasks may indicate the inadequacy of standard tools for a government unit commissioned to enhance decision-making. One hypothesis arising from our empirical analysis is that the NOKC (and, probably, comparable institutions within the EBM movement) addresses too broad a range of inquiries, using too confined a range of tools. The main problem with the EBM approach is its restricted and simplistic approach to scientific knowledge [55]. The standardizing imperative that underpins EBM strategies is neither compatible with all questions of relevance for policymaking nor with scientific reasoning. It would be interesting to look more closely at the decision-making processes and the role of the authorities in how assignments are accepted, modified or dismissed. Knowledge is always situated [56]. The NOKC SRs are no exception, informing both the question asked and the consequences of the answer.
In a critical analysis of evidence-based policy research, Oliver et al. [54] concluded that many approaches within this area are naïve in neglecting how policy is influenced and constituted. They point to a loss of clarity about what constitutes and defines ‘evidence’ and ‘policy’, neglecting “the policy process itself as a contested area of negotiation” and “the messy, complex, and serendipitous nature of policymaking” [54]. Comparing three SRs on EBP research, Oliver et al. [54] argue that, without an understanding of the complex processes of policy and knowledge mobilization, researchers who make policy and practice recommendations will simply be ignored. Orton et al.’s [1] SR of the use of research evidence in public health decision-making processes also concluded that the impact of research evidence was often indirect and had to compete with other influences.
We believe that current EBM-based NOKC studies seek to provide a legitimate basis for health policy decision-making in Norway, and we endorse the idea that certain domains of inquiry may be well suited to such approaches. However, given the large proportion of recent NOKC deliveries that lack substantial conclusions, we must also reflect on the potential policy consequences of ‘empty’ knowledge deliveries. As a broad range of complex interventions cannot be adequately evaluated by the EBM format, there will be limited availability of ‘authorized’ research documentation, and SRs will be unable to offer strong conclusions about positive effects. This is not the same as documentation of negative or zero intervention effects. Often, the issues leading to SR commissions are controversial, and the delivery is expected to recommend a direction or choice. The present findings highlight the possibility of policy decision-making in which political interests (as in “this government does not want to support such a reform”) may be legitimized by SRs that draw weak or deficient conclusions, which are then interpreted as evidence-based warnings against the intervention in question. In this way, the EBM evidence hierarchy may actually contribute to concealing the foundation of policy decision-making rather than providing transparency.
