- Open Access
- Open Peer Review
Evidence-based medicine: is it a bridge too far?
Health Research Policy and Systemsvolume 13, Article number: 66 (2015)
This paper aims to describe the contextual factors that gave rise to evidence-based medicine (EBM), as well as its controversies and limitations in the current health context. Our analysis utilizes two frameworks: (1) a complex adaptive view of health that sees both health and healthcare as non-linear phenomena emerging from their different components; and (2) the unified approach to the philosophy of science that provides a new background for understanding the differences between the phases of discovery, corroboration, and implementation in science.
The need for standardization, the development of clinical epidemiology, concerns about the economic sustainability of health systems and increasing numbers of clinical trials, together with the increase in the computer’s ability to handle large amounts of data, have paved the way for the development of the EBM movement. It was quickly adopted on the basis of authoritative knowledge rather than evidence of its own capacity to improve the efficiency and equity of health systems. The main problem with the EBM approach is the restricted and simplistic approach to scientific knowledge, which prioritizes internal validity as the major quality of the studies to be included in clinical guidelines. As a corollary, the preferred method for generating evidence is the explanatory randomized controlled trial. This method can be useful in the phase of discovery but is inadequate in the field of implementation, which needs to incorporate additional information including expert knowledge, patients’ values and the context.
EBM needs to move forward and perceive health and healthcare as a complex interaction, i.e. an interconnected, non-linear phenomenon that may be better analysed using a variety of complexity science techniques.
Over the past 20 years or more, the concept of evidence-based medicine (EBM) has increasingly been accepted as the gold standard for decision making in medical/health practice and policy.
EBM provides a standard procedure for using evidence in clinical decision making. It is framed as “…the conscientious, explicit and judicious use of best evidence in making decisions about the care of individual [sic] patients. The practice of evidence-based medicine means integrating individual clinical experience with the best available external clinical evidence from systematic research” . Muir Gray regarded this definition as too doctor-centric and expanded it to emphasize the importance of the patient perspective and proposed that, “…evidence based clinical practice is an approach to decision making in which the clinician uses the best scientific evidence available, in consultation with the patient, to decide upon the option which suits the patient best.” . In their respective papers, both Sacket and Gray described the stages of EBM decision making as (1) assessment and synthesis of external evidence using clinical epidemiology, systematic search and meta-analysis, and other techniques such as cost analysis and modelling; (2) use of probabilistic reasoning, taking into account, clinical expertise, and patients’ values and preferences. Remarkably, this broad but sensitive approach to rational clinical decision making was actually followed when applied to guideline development, but reduced the evidence in a skewed manner. Only evidence from explanatory randomized controlled trials (RCTs) was admitted as ‘reliable evidence’.
Whilst the value of EBM has been staunchly defended by its proponents, it has been widely criticized by many disciplines including clinical practice [3–8], epistemology [9–14], health sociology [15, 16], and implementation science . Moreover, in recent years, previously supportive EBM researchers argue for a ‘renaissance’ of the movement that follows and applies their original broad principles and multidisciplinary values, specially regarding the components of EBM related to shared decisions with patients and to expert judgment, built of evidence and experience [18, 19]. The main argument is that, in spite of its benefits, EBM could have also had important negative consequences for healthcare delivery, policy and financing. Examples of this include (1) failing to manage complexity, the individual’s needs, and the person’s context and issues such as multi-morbidity; (2) the quantity of research studies and the variable quality, which has become impossible to manage and in some cases lack clinical significance; and (3) the medicalization of life, namely creating new diseases for non-specific complaints and the use of the evidence-based ‘quality markers’ to widely promote drugs and medical devices [20–22].
This paper contributes to the descriptive rational reconstruction of EBM by analysing its historical development and controversies , as well as its limitations in the current healthcare context. We approach this analysis from a complex adaptive systems science perspective with its focus on the relational interactions of health and healthcare variables  and the unified approach to the philosophy of science as suggested by Schurz . A complex adaptive view of health as a balanced state between the person’s physical, social, emotional and cognitive experiences and its consequences for shaping complex adaptive healthcare and healthcare systems as highly responsive to the person’s unique needs as well as a complex adaptive understanding of medical knowledge have been described in detail elsewhere [25–27]. The unified approach to the philosophy of science provides a systematization of the basic assumptions of scientific knowledge and revises the role of values in science. It provides a new framework for understanding the differences between the phases of discovery, corroboration and implementation in science. Its importance for defining new areas of scientific knowledge and the role of different logic inferences in each phase have been reviewed elsewhere .
The present paper is structured in four sections: in the first, we review the origins, principles and actors who contributed to the rise of EBM, whilst in the second, we discuss why this movement evolved so rapidly and was so broadly accepted. The third describes a ‘restricted’ approach to EBM and its use in designing standard methods for developing practice guidelines, and finally, we comment upon the current challenges faced by the EBM movement in the context of systems thinking and implementation sciences.
Where does EBM come from?
There were three factors at the beginning of the 20th century that predated the development of EBM, namely (1) the transformation of hospitals in the USA, from a shelter for the sick, to prestigious organizations, where medical care was based on scientific principles ; (2) the reform of medical education , and (3) the birth of clinical epidemiology . The transformation of hospitals was accompanied by a process of standardization of healthcare provision through guidelines, which was also closely related to the efforts of the American Medical Association to establish its position as the reference accreditation body in medicine . Standardization included the regulation of the medical profession, which ensured surgeons were well trained; the development of procedural standards in hospitals, which reduced variability and improved quality; and inclusion, for the first time, of the patient record file, allowing hospital managers to control what the physicians were doing . As Timmermans and Berg suggested  the use of standards and guidelines, together with the emerging scientific knowledge and technologies enabled the growth of professional autonomy. However, standards and guidelines also became major triggers for the decline in clinical autonomy by the late 20th century .
The subject of clinical epidemiology was progressively introduced into medical programs based on the Enlightenment idea that progress was achievable through objectivity and rationality, so medicine has to be a science, not an art . In 1968, McMaster University (Canada) was the first to offer an integrative ‘problem-based learning’ curriculum, combining the studies of basic sciences, clinical epidemiology and clinical medicine resulting from clinical problems [31, 33]. The ‘father’ of EBM, David Sackett, directed this department. The publication of a series of recommendations by the Canadian Task Force on Periodic Health Examination that was led by Sackett [34, 35] in 1979, underscored the rationale for using insights from clinical epidemiology to inform clinical practice. The findings supported recommendations to abandon routine annual check-ups in favour of selective approaches based on the patient’s age and sex. It was the first time that recommendations were made according to the ‘levels of evidence’ and exclusively based on ‘grading study designs’, i.e. RCTs provide good evidence (level I), cohort studies and case–control studies provide fair evidence (level II), and expert opinion arising from clinical experience provides poor evidence (level III). Unsurprisingly the same basis for grading treatment recommendations was applied from level A: to apply the intervention to level E: not to apply the intervention.
A prerequisite for the widespread adoption of EBM required clinicians to be more critical when appraising the scientific literature. In 1981, Sackett et al.  published a series of articles in the Canadian Medical Association Journal that explained the criteria for assessing the internal validity of study designs as RCTs providing the gold standard for treatment, cohort studies for diagnosis, and case–control studies for etiology or harm. However, as Zimmerman indicated , this simplification was one of the most important weaknesses of EBM. Indeed, the major resistance to EBM relates to the specification of the knowledge base of medicine as something rational/technical/linear/predictable rather than contingent/experiential/non-linear/unpredictable [16, 26, 36].
Managing the vast amount of research literature became possible with wider availability of computers, in particular the personal computer on the doctor’s desktop . This enabled Iain Chalmers, director of the National Perinatal Epidemiology Unit in Oxford (United Kingdom) in the mid 1980s, to establish an electronic database of perinatal trials which made this information readily accessible to clinicians [38, 39]. The concepts and creation of electronic databases and increasing computing power facilitated the democratisation of knowledge management, something previously confined to only a few experts . Some years later, The Cochrane Collaboration emerged as an organisation that systematically combed, reviewed and synthesised the vast amount of research literature to make it accessible to the clinician at the time of the patient consultation.
Another contextual factor to explain the development of the EBM was the rising concern about the sustainability of health systems during the 1970s. This concern resulted in the emergence of new disciplines, such as health economics, that influenced the development of major approaches to healthcare reform such as managerialism  and outcomes management , in addition to EBM. These three approaches focused on the ‘specific’ to achieve measurable objectives; continuous evaluation of performance against defined objectives, outputs and standards; and rationing of resources by effectiveness criteria to make the work of physicians more transparent through control and surveillance.
Closely related to the development of Health Maintenance Organizations was Outcomes Management (OM) in the United States, which adopted the principles of quality improvement to facilitate physicians’ autonomy and control of their clinical practice. OM follows four major principles : (1) appropriateness, which relies on standards and guidelines; (2) routine outcome assessment based on routine and systematic measures of patients’ functioning and wellbeing, along with disease-specific clinical outcomes at appropriate time intervals; (3) the link to data mining to pool clinical and outcome data on a massive scale; and (4) a focus on dissemination and impact analysis to take into account the segment of the database most appropriate to the concerns of each decision maker. OM differs from EBM in its emphasis on ‘real data’ in contrast with EBM’s ‘experimental data’, while both OM and EBM aimed to empower clinicians to improve their clinical decision making capacity through the new tools on offer. This contrasts markedly with the view of managerialism, or neo-liberal approaches, where the power of decision making is shifted from clinicians to managers and auditors [41, 43].
The need for standardization, the development of clinical epidemiology, and concerns about the economic sustainability of health systems together with the increased capacity of computers to handle large amounts of data paved the way to the development of the EBM movement, officially founded in 1991 . The publication of “Evidence-based medicine. A new approach to teaching the practice of medicine” by the Evidence-Based Working Group in JAMA  rapidly spread the concept and principles of EBM allowing the Evidence-Based Working Group to pronounce EBM to be a ‘new paradigm’. It would change the ‘old way’ of ‘solely’ practicing subjectivity-based medicine predicated on intuition, clinical experience and pathophysiological rationale with an objective approach based on ‘scientific’ evidence. Whilst they advocated the addition of evidence as a key consideration, they also clearly rejected the role of experts with ‘authoritative opinions’ as guiding clinical decision making.
Why was EBM so widely accepted?
From 1992 until September 2015, the PubMed database revealed over 20,000 papers with ‘evidence-based’ in their title. Evidence-based practice guidelines are the norm for the majority of official agencies and professional organizations, and EBM approaches are at the core of today’s scientific thinking. The RCT is regarded as the fundamental research response underpinning the ‘perceived new paradigm’ of EBM for healthcare , and these ideas have now expanded far beyond the realm of medicine (consider, for instance, the debate if Conservation Science needs to include RCTs in the same way medicine does: http://blog.nature.org/science/2013/08/15/debate-randomized-control-trials-in-conservation/).
As Pope suggested , EBM evolved as a social movement that started with agitation (i.e. we need to change the current paradigm based on experience). It was crystalized by the shared experience of the group at McMaster University and the development of an enduring sense of purpose, disseminated in a series of position papers, declarations, and guidelines published in influential medical journals by key opinion leaders in clinical epidemiology. So, ironically, the adoption of EBM by the scientific community was not based on evidence but on authoritative knowledge, precisely the type of approach EBM was meant to replace, a point recently acknowledged by one of its key proponents, Sackett himself .
Use of authoritative knowledge in EBM
We can identify three factors related to authoritative knowledge that could have played a major role in the success of EBM: reputation, the Matthew effect, and the invisible college.
The first ever paper on EBM written by an almost anonymous EBM Working Group a appeared in JAMA and provided the movement with instant grounds of credibility. Publishing under the authorship of a working group raised its status to that of an authoritative consensus paper. However, as Zimmerman suggested , the EBM Working Group used a language closer to a political manifesto, calling for far-reaching changes in the practice of medicine, in the process creating an ‘enterprise of scientific objectivity’. This working group, together with Drummond Rennie, deputy editor at JAMA, remained the main advocates of the EBM movement for the first critical years: out of 22 articles on EBM published in the first 3 years, 12 were published by JAMA, reflecting Rennie’s and JAMA’s remarkable commitment to the new approach [31, 46]. This new movement was not only sustained by JAMA, it also found the British Medical Journal to be its keen European supporter [1, 47].
Within 3 years, the movement was threatened by an equally prestigious journal, The Lancet, which took a critical position of EBM. In 1995, an anonymous editorial stating that although “The Lancet applauds practice based on the best available evidence – bringing critically appraised news of such advances to the attention of clinicians is part of what peer-reviewed medical journals do – but we deplore attempts to foist evidence-based medicine on the profession as a discipline in itself” . The Lancet has since remained one of the most critical journals about the EBM movement. For instance, in 2005, it published a paper entitled “External validity of randomized controlled trials: To whom do the results of this trial apply?” criticizing the hierarchy of evidence as its focus is internal validity, neglecting the critical issue of external validity/generalizability of those results .
The reputations of the EBM movement’s key proponents and authors were well established. David Sackett and Iain Chalmers were renowned clinical epidemiologists and worked in highly regarded institutions – McMaster and Oxford University, respectively. Gordon Guyatt, as co-founder of the Medical Reform Group, a Canadian medical group composed of young doctors and nurses based in Toronto , notably agitated against the practice of medicine guided by senior doctors’ opinions. Personal experience gained under extreme conditions shaped the views of Archie Cochrane, a doctor and prisoner of war, and Iain Chalmers, a doctor in Gaza; they realized that in many cases new expensive treatments were no better than older ones . From any point of view, the leading professionals were clearly well motivated but in practice their recommendations resulted in an over-simplified approach to ‘the clinical care of patients’.
A related reputational effect is gained from the Matthew effect  – raising the credibility of a viewpoint and an author group by excessive cross-citation amongst its proponents,b a practice utilized by scientists since the 17th century. As a result, a group is highly likely to gain influence and power to set future research, practice and policy agendas (through grants, publications, conference presentations, etc.), made easier by the current state of publication policies and quality assessment procedures .
The extraordinary ability of the major EBM players to promote, implement and expand collaborative groups and networking resulted in what is known as an ‘invisible college’ . The invisible college consists of a group of scientists or professionals who may live in separate locations but attend the same conferences, publish in the same journals, and invite each other to give keynote lectures to share the same ideas. An invisible college emerged from the collaboration between the groups at McMaster University and the Cochrane Collaboration. One could argue that the Cochrane Collaboration over time has morphed into a form of ‘visible college’. Indeed, the Cochrane Collaboration’s initiative of a series of small workshops started an international social network of EBM supporters.
From a broad model to a narrow version of EBM
The historical and philosophical basis for EBM started with a broad health system’s perspective. In the 1930s, the then medical student, Cochrane, demanded on a protest placard that “All effective treatment must be free” [53, p. 1]. This call was about demonstrating a cost/benefit perspective, predicated on measuring “the effect of a particular medical action in altering the natural history [sic] of a particular disease for the better” [53, p. 2]. Cochrane argued that the RCT would remove bias and subjective opinion from managing disease, and indeed RCTs demonstrated important but limited gains in understanding therapeutic interventions. He clearly distinguished between ‘effectiveness’ and ‘efficiency’ and observed that, while the RCT as a scientific method could demonstrate ‘effectiveness’ in the trial population, this would not equate to greater ‘efficiency’ in healthcare, i.e. the same outcomes would generally not be achieved in routine practice due to the “complexities within the health system” [53, p. 2]. In addition, Cochrane was much more interested in the aspects of care crather than cure, alluding to the often neglected concern of ‘equality’ within the health system. As he stated: “In particular I believe that cure is rare while the need for care is widespread [sic], and that the pursuit of cure at all costs may restrict the supply of care, but the bias has at least been declared” [53, p. 7].
The beginnings of the EBM approach were clearly focused on understanding the complexities of the ‘workings of the healthcare system’ and its relationship to making the ‘best possible decision’s for the care of patients’. However, these complexities have rapidly been reduced to a narrow focus on standardised and typically single disease management guidelines.
Managing scientific knowledge for practice and the guideline development movement
One of the main objectives of EBM is to make large amounts of scientific knowledge more accessible, and developing clinical guidelines with recommendations to support clinical decisions seemed the obvious way to proceed.
Although clinical guidelines are useful they are also limiting if, for instance, they only draw on one source of information (i.e. the explanatory RCT). These guidelines will also restrain the freedom of professionals to use other sources of knowledge in their clinical decision making, like knowing patients’ preferences and clinical experiences . Evidence-based guidelines for a specific area of practice are typically seen by clinicians as the penultimate and authoritative practice pathway, reinforced by adverse litigation and clinical review committee outcomes . Consequently, many practitioners see clinical guidelines as the main threat to adapting clinical decisions to individual patients’ needs and contexts, i.e. interfering with their necessary clinical autonomy. Indeed, EBM supporters like JR Hampton, 32 years ago, asked for the death of clinical freedom as they saw ‘clinical judgment’ as the major obstacle to advancing medicine ; only recently they realized that clinical autonomy is needed so practitioners can use their ‘expert knowledge’ in the best interest of their patients .
The preoccupation with the quality of the studies used to develop clinical guidelines most likely explains the transformation of the broader EBM framework into its narrower RCT-driven form. The difficulties in translating the recommendations contained in EBM guidelines into practice and policy and the consecutive process of revision of the reductionist EBM approach to guidelines has been reviewed by others [49, 57, 58].
The initial problems with translating evidence-based guidelines into practice were attributed to the difficulties in properly rating the supporting body of research. Hence, a detailed analysis of the grading of RCTs was suggested . The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) working group analyzed six grading systems used by 51 organizations and found, to its surprise, poor reliability in the assessment of the quality of studies. None of the systems were usable for all user groups (professionals, patients and policymakers). This study probably was the first major criticism of EBM by one of its leading groups, concluding that the systems for grading levels and strength of evidence had important shortcomings for proposing clinical recommendations . The same group later recognized that the early systems of grading, which focused almost exclusively on randomized trials, were inadequate. The group realized that observational studies had features that could both decrease or increase the quality of the supporting body of research used for recommendations, and that high quality observational studies could contribute to better clinical decision making . The factors considered when rating evidence include the overall quality of studies, the coherence of studies, uncertainties about the balance of benefits versus harms, and uncertainties in values and opportunity costs – reconnecting with Cochrane’s much broader vision.
Recognition of the need to separate assessment of quality of evidence from the strength of recommendations was a major step forward – “high quality evidence doesn’t necessarily imply strong recommendations, and strong recommendations can arise from low quality evidence” . However, a new question arose: who should make these recommendations? Paradoxically, recommendations in clinical guidelines developed by ‘EBM experts’ use the opinion of their group of experts to work through and agree on the wording of their recommendations, grade the body of evidence, and utilize their values and background, i.e. apply their ‘prior expert knowledge’. However, how the information, expert opinion and contextual factors are balanced in these deliberations often remains unclear.
In recognition of the need for systematic and explicit approaches in the development and grading of recommendations, guideline development standards now require a summary of findings [63, 64] or evidence statements  for each recommendation and these are included in the guideline’s technical report.
Various ways of knowing
The EBM community is starting to respond to criticism of limiting evidence and is attempting to incorporate and value other sources of knowledge, such as observational studies and the findings of qualitative research. Nevertheless, the fundamental flaws inherent in the grading systems remain such as the assumption that: study designs and quality can be arranged in a systematic but simplistic linear structure, when, in reality, the use of different and highly relevant information such as patient preferences, and applicability to local or practice contexts would lead to completely different grading systems ; and that studies graded according to epidemiological principles will diminish the risk of bias. This criterion may make sense in the discovery of new knowledge but is not the key consideration if the main aim is to implement research into local practice.
It is important to consider that scientific knowledge is divided in three major areas: discovery, corroboration and implementation, and that the information of one domain cannot be directly applied to the others [23, 28]. The value-neutrality principle that guides discovery and corroboration requires further – and complex – clarification in the implementation phase. According to Schurz , the value-neutrality requirement “implies that the scientist separates her scientific knowledge from fundamental value assumptions which she assumes in means-end inferences”. Means-end inferences and abduction are used in EBM guideline development without an adequate formalization of their contribution to the construction of the guideline recommendations . The roots of this philosophical debate are far beyond the scope of this paper and, unfortunately, the philosophy of science principles necessary to support the underpinnings of EBM have as yet not been properly explored.
In the fields of discovery and corroboration the important criterion is internal validity (observed variation can be interpreted as a causal relationship, therefore, the study design needs to guarantee that the risk of bias is low). In the field of implementation, the important criterion is the degree of external validity of the results applicability to the local context and acceptability of the intervention/s to the patient. External validity is important as it means that the results can be generalized to different persons, settings and times. There is an inverse relationship between internal and external validity. If the final purpose of EBM was to improve the health of real people in real settings, external validity should be emphasized and strengthened. It is not only crucial to know if a treatment is effective in controlled situations (i.e. internal validity), but also that it is going to be effective in the real world (i.e. external validity). While the grading systems for developing clinical guidelines used by EBM are systematic and reliable, they often prioritize internal validity and therefore are not ‘fit for purpose’. The emphasis on internal validity has contributed to the failure of EBM, as recommendations – being based on experimental designs where variables and confounders are controlled (RCTs) – often fail to be translatable into practice because the research context does not reflect real world clinical practice/reality [67, 68].
The EBM and guideline communities have also recognized the limited implementation of guidelines jeopardized by their current static and unfriendly structure [18, 54, 69]. New proposals, such as the development of dynamic wiki-based clinical guidelines, might eventually resolve this problem and enable the participation of all stakeholders (e.g. patients, clinicians and decision makers), in a collaborative effort that may result in greater transparency and acceptability [70, 71].
The leap from discovery/corroboration to implementation was partly reflected in the criticism made in The Lancet in 2005  and in the position of other EBM experts such as ER Epstein, who developed the disease management approach, superseded by Wagner’s chronic/integrated care model . Even though he adhered to EBM, Epstein’s vision of health knowledge was clearly beyond RCTs and much closer to Paul Ellwood’s OM approach. He considered EBM as one of several tools to improve quality of care: “The new paradigm is population-based risk and disease assessment, systems of disease prevention and health promotion, community-based intervention and provider contacts within a framework of automated information, evidence-based medicine, and defined protocols of care, with explicit collection of outcomes information” . Epstein and Sherwood, although subscribing to the gold-standard of RCTs, mention the difficulty of using them as the main source of information in outcome management/implementation and mention ‘prospective effectiveness trials’ as the alternative to RCTs [72, 73].
The limitation of RCTs to assess real world outcomes
A more fundamental question would be, can real world outcomes be achieved/evaluated with randomized controlled trials? In short, the answer is no, if we only use explanatory randomized trials as preferred by its proponents. However, pragmatic controlled trials that, by definition, are conducted under usual conditions offering practitioners considerable freedom in deciding how to apply the intervention to be tested, are not obtrusive (i.e. there is no special effort to improve compliance by patients or practitioners), and use administrative databases for the detection of outcomes, can offer a valid alternative. While explanatory RCTs will be linked to discovery and corroboration and will aspire to removing variability, pragmatic controlled trials (even including randomization) fit in the area of implementation and embrace variability as the norm [74, 75]. They take into account the local context and are mostly valued when driven by theory and complemented by other sources of knowledge .
In conclusion: the challenges facing EBM
Most likely, EBM grew too fast to effectively incorporate its original propositions: evidence, expert knowledge, and patients’ preferences . The reliance of EBM on the RCT was useful for acute (mostly single disease) conditions treated with simple interventions, but this approach is not suitable in the current epidemiological context characterized by chronicity and multimorbidity in complex health systems. In particular, EBM has largely disregarded the importance of social determinants of health and local context – hence the nicknames ‘cookbook approach’ or ‘MacDonaldization’ of medicine [29, 77]) – and its real impact on the ‘effectiveness’ and ‘efficiency’ of healthcare on the ‘equality’ of needed healthcare services.
As an a priori, evidence is context sensitive, and therefore to some extent tacit , and both global and local evidence need to be combined in the development of usable recommendations for clinical decision making . Local evidence includes the presence of modifying factors in the specific settings, magnitude of needs (prevalence, baseline risk or status), patient values, costs (to the patient and the system), and the availability of resources in the system . This local evidence needs to be combined with ‘expert knowledge’, which should be differentiated from ‘expert opinion’ and valued in a different way. By ‘expert knowledge’ we mean the implicit knowledge that professionals have that helps them to better understand the local conditions. It is based on data (their accumulated experiences) and thus different to simple opinions or feelings about something [81, 82]. There is on-going debate of the relevance of ‘colloquial evidence’ in the development of guidelines . This reflects a worrying lack of a basic understanding by authors and reviewers of the fundamentals of scientific knowledge and the differences between expert knowledge and evidence.
There is an imperative to explore and then learn from other disciplines on how to use research evidence and incorporate it with local context and expert knowledge to achieve best possible patient outcomes. For example, in other areas of science, e.g. conservation science and artificial intelligence, expert knowledge is routinely incorporated in the analysis. Expert-Based Collaborative Analysis is a systematic procedure to incorporate expert knowledge into data analysis; such an approach has been proven to be useful when dealing with complex issues and can be seen as a powerful tool in the current health context characterized by an increase in the number of patients with multiple conditions, resulting from heterogeneous genomic/pathophysiological pathways and diverse personal needs .
In future, the inclusion of ‘expert knowledge’ in the analysis of research data might produce more usable evidence for clinical decision-making. In this sense, EBM needs to go beyond the sole use of the RCT and acknowledge that scientific knowledge is multidimensional and cannot be arranged in only one hierarchical system. Knowledge coming from studies using different methodological approaches is complementary . Hence, to have a complete picture, information coming from explanatory RCTs has to be complemented and contrasted with information coming from pragmatic RCTs evaluating effectiveness in routine practice. This implies some loss of ‘internal validity’ and an increase in the uncertainty of the results, but ‘gains in representativeness’.
The most important challenge facing the EBM-movement is the provision of a detailed description of its methods for scientific reasoning. This requires an analysis of its taxonomic principles, including formal definitions of ‘scientific knowledge’, ‘evidence’, and ‘decision making’ in health, as well as the different types of logic inferences used in the scientific reasoning process . As others have highlighted, we believe that this academic exercise is crucial to clarify the confusion between ‘good’ evidence  and scientific ‘truth’. Apart from systems thinking [85, 86], healthcare researchers, clinicians and policymakers could benefit from greater knowledge of the philosophy of science to design and interpret research, and their use in guiding decision making processes – beyond the classical experimental/deductive approach favoured by the EBM movement [18, 23, 28, 87].
It should also be highlighted that health systems research involves different disciplines (including social ones) with different perspectives, epistemologies, and ways of conceptualizing and conducting research. Health systems research, as intimated by Cochrane, is broader than identifying ‘clinical effectiveness’ – ‘efficiency’ and ‘equality’ are equally important considerations for achieving successful implementation of health system improvement; therefore, all stakeholders’ fundamental value assumptions should be explicit.
Ethics approval was not required as this manuscript is a narrative review of published papers.
aThey signed as a group but in a footnote they included their names.
bIt refers to the fact that even though the majority of papers have little impact on the scientific community (the hath not), a small number of them attracts the attention of the community and receive a lot of citations (the hath abundance).
cAs an aside, Cochrane’s observations working as a doctor in a prisoners of war camp provided him with the empirical evidence of the great benefits of care, its scientific basis has been untangled by the study of psychoneuroimmunology.
Randomized controlled trials
Sackett D, Rosenberg W, Gray J, Haynes R, Richardson W. Evidence based medicine: what it is and what it isn’t. Br Med J. 1996;312(7023):71–2.
Muir JG. Evidence-based healthcare. London: Churchill Livingston; 1996.
Carrhill R. Welcome to the Brave-New-World of evidence based medicine. Soc Sci Med. 1995;41(11):1467–8.
Miles A, Loughlin M. Continuing the evidence-based health care debate in 2006. The progress and price of EBM. J Eval Clin Pract. 2006;12(4):385–98.
Wilson K. Evidence-based medicine. The good, the bad and the ugly. A clinician’s perspective. J Eval Clin Pract. 2010;16(2):398–400.
Sturmberg JP. EBM: a narrow and obsessive methodology that fails to meet the knowledge needs of a complex adaptive clinical world: a commentary on Djulbegovic, B., Guyatt, G. H. & Ashcroft, R. E. (2009) Cancer Control, 16, 158–168. J Eval Clin Pract. 2009;15(6):917–23.
Silva SA, Wyer PC. Where is the wisdom? II – Evidence-based medicine and the epistemological crisis in clinical medicine. Exposition and commentary on Djulbegovic, B., Guyatt, G. H. & Ashcroft, R. E. (2009) Cancer Control, 16, 158–168. J Eval Clin Pract. 2009;15(6):899–906.
Miles A. Evidence-based medicine: requiescat in pace? A commentary on Djulbegovic, B., Guyatt, G. H. & Ashcroft, R. E. (2009) Cancer Control, 16, 158–168. J Eval Clin Pract. 2009;15(6):924–9.
Newton W. Rationalism and empiricism in modern medicine. Law Contemp Probl. 2001;64(4):299–316.
Harari E. Whose evidence? Lessons from the philosophy of science and the epistemology of medicine. Aust N Z J Psychiatry. 2001;35(6):724–30.
Wyer PC, Silva SA. Where is the wisdom? I–a conceptual history of evidence-based medicine. J Eval Clin Pract. 2009;15(6):891–8.
De Vreese L. Evidence-based medicine and progress in the medical sciences. J Eval Clin Pract. 2011;17(5):852–6.
Sehon SR, Stanley DE. A philosophical analysis of the evidence-based medicine debate. BMC Health Serv Res. 2003;3(1):14.
Holmes D, Murray SJ, Perron A, Rail G. Deconstructing the evidence-based discourse in health sciences: truth, power and fascism. Int J Evid Based Healthc. 2006;4(3):180–6.
De Vries R, Lemmens T. The social and cultural shaping of medical evidence: case studies from pharmaceutical research and obstetric science. Soc Sci Med. 2006;62(11):2694–706.
Pope C. Resisting evidence: the study of evidence-based medicine as a contemporary social movement. Health. 2003;7(3):267–82.
Porter S, O’Halloran P. The use and limitation of realistic evaluation as a tool for evidence-based practice: a critical realist perspective. Nurs Inq. 2012;19(1):18–28.
Greenhalgh T, Howick J, Maskrey N, Evidence Based Medicine Renaissance Group. Evidence based medicine: a movement in crisis? BMJ. 2014;348:g3725.
Greenhalgh T. Why do we always end up here? Evidence-based medicine’s conceptual cul-de-sacs and some off-road alternative routes. Int J Prosthodont. 2013;26(1):11–5.
Moynihan R, Heath I, Henry D. Selling sickness: the pharmaceutical industry and disease mongering. Br Med J. 2002;324:886–91.
Gøtzsche P. Deadly medicines and organised crime: how big pharma has corrupted healthcare. Oxford: Radcliffe; 2013.
Ioannidis J. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294(2):218–28.
Schurz G. Philosophy of science. A unified approach. New York: Routledge; 2014.
Sturmberg JP, Martin CM. Handbook of systems and complexity in health. New York: Springer; 2013.
Sturmberg JP. The personal nature of health. J Eval Clin Pract. 2009;15(4):766–9.
Sturmberg JP, Martin CM. Knowing–in medicine. J Eval Clin Pract. 2008;14(5):767–70.
Sturmberg JP, O’Halloran DM, Martin CM. People at the centre of complex adaptive health systems reform. Med J Aust. 2010;193(8):474–8.
Salvador-Carulla L, Fernandez A, Madden R, Lukersmith S, Colagiuri R, Torkfar G, et al. Framing of scientific knowledge as a new category of health care research. J Eval Clin Pract. 2014;20(6):1045–55.
Timmermans S, Berg M. The gold standard: the challenge of evidence-based medicine and standardization in health care. Philadelphia: Temple University Press; 2003.
Flexner A. Medical Education in the United States and Canada. A Report to the Carnegie Foundation for the Advancement of Teaching. New York: Carnegie Foundation; 1910.
Zimerman AL. Evidence-based medicine: a short history of a modern medical movement. Virtual Mentor. 2013;15(1):71–6.
Rosenberg CE. The tyranny of diagnosis: specific entities and individual experience. Milbank Q. 2002;80(2):237–60.
Sur RL, Dahm P. History of evidence-based medicine. Indian J Urol. 2011;27(4):487–9.
The periodic health examination. Canadian task force on the periodic health examination. Can Med Assoc J. 1979;121(9):1193–254.
Bell N, Connor Gorber S, Tonelli M, Pottie K, Singh H, Joffres M, et al. From ABCs to GRADE: Canadian Task Force on Preventive Health Care’s new rating system for clinical practice guidelines. Can Fam Physician. 2013;59(12):1282–9.
Malterud K. The social construction of clinical knowledge - the context of culture and discourse. Commentary on Tonelli (2006), Integrating evidence into clinical practice: an alternative to evidence-based approaches. J Eval Clin Pract 12, 248–256. J Eval Clin Pract. 2006;12(3):292–5.
Bakken S. An informatics infrastructure is essential for evidence-based practice. J Am Med Inform Assoc. 2001;8(3):199–201.
Chalmers I, Hetherington J, Newdick M, Mutch L, Grant A, Enkin M, et al. The Oxford Database of Perinatal Trials: developing a register of published reports of controlled trials. Control Clin Trials. 1986;7(4):306–24.
Chalmers I. A register of controlled trials in perinatal medicine. WHO Chron. 1986;40(2):61–5.
Georgiou A. Data, information and knowledge: the health informatics model and its role in evidence-based medicine. J Eval Clin Pract. 2002;8(2):127–30.
Beck M, Sara M. Quality management and managerialism in healthcare. A critical historical survey. New York: Palgrave Macmillan Publishing; 2014.
Ellwood PM. Outcomes management. N Engl J Med. 1988;318(23):1549–56.
Davies B. Death to critique and dissent? The policies and practices of new managerialism and of evidence-based practice. Gend Educ. 2003;15(1):91–103.
Evidence-Based Medicine Working Group. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268(17):2420–5.
Montori VM, Guyatt GH. Progress in evidence-based medicine. JAMA. 2008;300(15):1814–6.
Smith R, Rennie D. Evidence-based medicine—an oral history. JAMA. 2014;311(4):365–7.
Ham C, Hunter DJ, Robinson R. Evidence based policymaking. BMJ. 1995;310(6972):71–2.
Evidence-based medicine, in its place. Lancet. 1995;346(8978):785.
Rothwell PM. Treating individuals 1 - external validity of randomised controlled trials: “To whom do the results of this trial apply?”. Lancet. 2005;365(9453):82–93.
Hill GB. Archie Cochrane and his legacy – an internal challenge to physicians’ autonomy? Commentary. J Clin Epidemiol. 2000;53(12):1189–92.
Blashfield RK. Feighner et al., invisible colleges, and the Matthew effect. Schizophr Bull. 1982;8(1):1–12.
Pendlebury DA. The use and misuse of journal metrics and other citation indicators. Arch Immunol Ther Exp. 2009;57(1):1–11.
Cochrane A. Effectiveness and efficiency: random reflections on health services. London: The Nuffield Provincial Hospitals Trust; 1972.
Zuiderent-Jerak T, Forland F, Macbeth F. Guidelines should reflect all knowledge, not just clinical trials. BMJ. 2012;345:e6702.
Hampton JR. The end of clinical freedom. Br Med J (Clin Res Ed). 1983;287(6401):1237–8.
Hampton J. Commentary: the need for clinical freedom. Int J Epidemiol. 2011;40(4):849–52.
Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet. 1995;345(8953):840–2.
Ansari MT, Tsertsvadze A, Moher D. Grading quality of evidence and strength of recommendations: a perspective. PLoS Med. 2009;6(9), e1000151.
Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess. 1999;3(12):i–iv. 1–98.
Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, Hill S, et al. Systems for grading the quality of evidence and the strength of recommendations - I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res. 2004;4:38.
Brozek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW, et al. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Allergy. 2009;64(5):669–77.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. 2008;336:924.
Cluzeau F, Burgers J, Brouwers M, Grol R, Makela M, Littlejohns P, et al. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12(1):18–23.
Makarski J, Brouwers MC. The AGREE Enterprise: a decade of advancing clinical practice guidelines. Implement Sci. 2014;9:103.
National Health and Medical Research Council. NHMRC levels of evidence and grades for recommendations for developers of guidelines. Sydney: NHMRC; 2009.
Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126–53.
Metge CJ. What comes after producing the evidence? The importance of external validity to translating science to practice. Clin Ther. 2011;33(5):578–80.
Steckler A, McLeroy KR. The importance of external validity. Am J Public Health. 2008;98(1):9–10.
Francke AL, Smit MC, de Veer AJ, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:38.
Archambault PM, Turgeon AF, Witteman HO, Lauzier F, Moore L, Lamontagne F, et al. Implementation and evaluation of a Wiki involving multiple stakeholders including patients in the promotion of best practices in Trauma care: the WikiTrauma interrupted time series protocol. JMIR Res Protoc. 2015;4(1):e21.
den Breejen EM, Nelen WL, Knijnenburg JM, Burgers JS, Hermens RP, Kremer JA. Feasibility of a wiki as a participatory tool for patients in clinical guideline development. J Med Internet Res. 2012;14(5):e138.
Epstein RS, Sherwood LM. From outcomes research to disease management: a guide for the perplexed. Ann Intern Med. 1996;124(9):832–7.
Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff (Millwood). 2001;20(6):64–78.
Relton C, Torgerson D, O’Cathain A, Nicholl J. Rethinking pragmatic randomised controlled trials: introducing the “cohort multiple randomised controlled trial” design. BMJ. 2010;340:c1066.
Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464–75.
Marchal B, Westhorp G, Wong G, Van Belle S, Greenhalgh T, Kegels G, et al. Realist RCTs of complex interventions - an oxymoron. Soc Sci Med. 2013;94:124–8.
Timmermans S, Mauck A. The promises and pitfalls of evidence-based medicine. Health Aff. 2005;24(1):18–28.
Polanyi M. Personal knowledge. Towards a post-critical philosophy. London: Routledge; 1958.
Remme JH, Adam T, Becerra-Posada F, D’Arcangues C, Devlin M, Gardner C, et al. Defining research to improve health systems. PLoS Med. 2010;7(11):e1001000.
Oxman AD, Schunemann HJ, Fretheim A. Improving the use of research evidence in guideline development: 7. Deciding what evidence to include. Health Res Policy Syst. 2006;4:19.
Gibert K, Garcia-Alonso C, Salvador-Carulla L. Integrating clinicians, knowledge and data: expert-based cooperative analysis in healthcare decision support. Health Res Policy Syst. 2010;8:28.
Martin TG, Burgman MA, Fidler F, Kuhnert PM, Low-Choy S, McBride M, et al. Eliciting expert knowledge in conservation science. Conserv Biol. 2012;26(1):29–38.
Sharma T, Choudhury M, Kaur B, Naidoo B, Garner S, Littlejohns P, et al. Evidence informed decision making: the use of “colloquial evidence” at nice. Int J Technol Assess Health Care. 2015;31(3):138–46.
McCaughey D, Bruning NS. Rationality versus reality: the challenges of evidence-based decision making for health policy makers. Implement Sci. 2010;5:39.
Sturmberg JP, Martin CM, Katerndahl DA. Systems and complexity thinking in the general practice literature: an integrative, historical narrative review. Ann Fam Med. 2014;12(1):66–74.
Peters DH. The application of systems thinking in health: why use systems thinking? Health Res Policy Syst. 2014;12:51.
Sturmberg JP, O’Halloran DM, Martin CM. Health care reform – the need for a complex adaptive systems approach. In: Sturmberg JP, Martin CM, editors. Handbook of systems and complexity in health. New York: Springer; 2013. p. 827–53.
Other participants at some of the sessions of the DOCAHS were Abdolvahab Baghbanian, Di O’Halloran and Andrew Wilson. AF and LSC also thank the students of the Unit of Study HSBH3005 “Evidence-based Healthcare”, Faculty of Health Sciences, University of Sydney, 2013, for their critical contribution to this debate.
The authors declare that they have no competing interests.
This review paper is the result of the discussions made by the multidisciplinary expert group “Dialogues On Complexity And Health Systems” (DOCAHS), to advance the understanding of complexity issues in health sciences, particularly in public health research held at the Menzies Centre for Health Policy, University of Sydney, in 2013 and 2014. AF was the rapporteur of the sessions, and, together with LSC drafted the paper. AF, JS, RM, SL, GT, RC and LSC reviewed the literature and contributed intellectually to the debate and reviewed the different drafts. LSC guided the discussion. All authors approved the final version of this manuscript.
About this article
- Complexity of knowledge
- Evidence-based medicine
- Evidence-based practice
- External validity
- Internal validity
- Randomized controlled trial