What are the challenges of evidence-informed policymaking?

Evidence-informed policy decisions are essential to ensure that health intervention programs and service plans are likely to be effective and offer value for money. However, barriers to the use of evidence to inform decision-making remain [7] and the use of published research to inform policy development is often limited [8]. Descriptive evidence and analytical studies are used to describe issues and inform priorities; however, evidence on the implementation and impact of interventions is less commonly used to inform program planning decisions and strategic actions [7]. In some cases, program decision-making can be driven by “informed guesswork, expert hunches, political and other imperatives” [9].

To address this, evidence provided to policymakers needs to be in a form that is useful to them [10–12]. Policymakers require synthesised and localised data that contrasts and prioritises policy options, demonstrates effectiveness of interventions, demonstrates the need for a policy response, demonstrates cost effectiveness of actions, reflects the level of public support for a particular issue and personalises the problem [12, 13]. In addition, policy and program decision-making processes are rarely linear. They are frequently iterative processes and are influenced by a range of inputs such as political environment, budget constraints, resources, values, available expertise and ethics [7, 12, 14, 15].

Even when research evidence is considered, as in public health policy development for the prevention of chronic disease [2], this evidence often points to a large range of risk factors that contribute to the problem, including broader social determinants of health. Our lack of understanding about how these risk factors interact, and which are the most important, have resulted in the development of more comprehensive, cross-sectoral strategies to tackle complex or ‘wicked’ problems [5]. However, this approach may not represent the most efficient or effective approach to reducing disease burden at the population level. Rather, it may act to spread finite resources less intensively over a greater number of programs and initiatives, diluting the potential impact of investment [5].

Knowledge mobilisation to support evidence-based decision-making

The term knowledge mobilisation (KM) is used to refer to a range of active approaches deployed to encourage the creation and sharing of research-informed knowledge [2]. The number of terms used to describe KM activities is large [16] and have been widely debated. These terms include knowledge translation, knowledge transfer, knowledge to action, knowledge exchange, knowledge interaction, etc. [2]. This multiplicity of terms can be a barrier to clear communication in this field [2]. In this research, the term KM is preferred as it reflects that the process of producing and applying knowledge in the health sector is non-linear and iterative. KM can involve a number of activities, including capacity building, advocacy, implementation, research and evaluation [17]. Not all of these activities are applied in every KM project [17] and they can be applied in different orders; however, they share the common function of generating and sharing research-informed knowledge [2].

KM strategies have been applied to a range of issues, including the quality and effectiveness of health services, addressing policy questions (for example, mapping health inequity and healthcare disparities), and addressing managerial and organisational issues such as the composition of multidisciplinary teams and the costs and consequences of different service models [2, 18]. A key strategy of KM is the production of good quality, synthesised evidence [9] such as scoping reviews, systematic reviews, meta analyses and research summaries highlighting key findings for decision-makers [9, 10].

Traditional methods of KM via evidence synthesis have made a valuable contribution; however, they have a number of characteristics that limit their utility as decision support methods for complex policy questions. Firstly, systematic reviews and meta-analyses focus on clear and specific questions and therefore have a narrow focus of investigation and limited potential to examine complex questions [11, 19]. Secondly, these methods frequently exclude qualitative evidence, and when qualitative evidence is included it is not used to answer the primary research question but only to answer supportive questions such as whether an intervention was acceptable to consumers [19]. Thirdly, these methods produce static overviews of the evidence and policy options that are passively provided to decision-makers, leaving them to interpret that evidence in their localised context and to navigate complexity and uncertainty as they weigh up options for responding to the problem [20].

While there are many KM approaches and techniques, the evaluation of their use is still in its infancy [2, 21]. The limited focus on evaluation of the effectiveness of KM methods, including systems-based ones, has been attributed to the challenges associated with the evaluation task [2], including the methodological challenges of conducting rigorous evaluations. It can be difficult to measure impact, to attribute impact to different strands of the activity in a complex environment, and to minimise the evaluation reporting burden on stakeholders [2].

Systems approaches to knowledge mobilisation

There are acknowledged synergies between KM and systems science [21]. Systems science methods have emerged as an effective analytical approach with the capacity to examine both complex health problems and the context in which they are embedded [6, 22, 23]. Systems science can be used to map health system components and their interactions; synthesise evidence, examine and compare the potential outcomes of interventions; and guide more efficient investment and conscientious disinvestment of resources [5]. As practical systems-based KM tools and strategies emerge, their efficacy needs to be evaluated and this knowledge to be shared [2, 21].

Systems approaches recognise the highly contextualised nature of health services and communities and, therefore, evidence to inform decision-makers is unlikely to be in the form of prescriptive statements of ‘what works’ [24]. Rather, evidence from a systems-thinking perspective will suggest the range of strategies that will have different types of effects for different groups under certain conditions. Building this type of evidence base will involve undertaking diverse methodologies, including the use of case studies investigating the efficacy of using systems techniques to inform decision-making [24].

Research methods in prevention science have traditionally employed a reductionist approach focusing on the detail of each component of the system. For example, many studies focus on the design, measurement and analysis of specific interventions for specific target groups. These studies have contributed and will continue to contribute significantly to understanding the effectiveness of prevention interventions, gaining knowledge about direct causal relationships and understanding components of complex systems [6, 25]. However, this approach can result in a failure to achieve understanding of the broader system behaviour influencing prevention problems and can hinder insights that may be critical for effective policy and program decision-making [25]. Traditional statistical methods have difficulty accounting for delays between cause and effect, non-linear relationships and unanticipated consequences of interventions [23].

Applying a systems approach through dynamic simulation modelling can provide a method to map, visualise and quantify a complex system, to promote discussion among stakeholders [26], and to identify points of high leverage for intervening. Leverage points are those places in a system where a small shift can create a large impact [27]. Leverage points are difficult to identify in complex systems using traditional reductionist research methods which examine relationships between specific elements of the system in isolation [28–30]. It is also difficult to identify the direction of shift required to obtain the desired outcome without comprehensive analysis and understanding of the system and its behaviour [27, 31]. Unanticipated consequences of interventions can have profound and negative impacts [31, 32], and can lead to policy resistance in which the intended positive impact of the intervention is counteracted by system responses to the intervention itself [32].

Dynamic simulation models allow for rapid integration and use of new evidence for policy analysis, make trade-offs of policy options explicit, and act as a vehicle for advancing controversial, contested and value-laden debates [5, 31, 33]. Their use to explore the implications of policy options can give rise to policy scenarios that have not previously been considered [5].

System dynamics modelling has been used as a tool to represent disease prevalence, risk factors and local context and to simulate the health outcomes of interventions, thus facilitating the alignment of prevention efforts by a range of community stakeholders [34]. For example, Loyo et al. [35] used a stakeholder engagement process to develop a system dynamics model to simulate the impact of various interventions in chronic disease outcomes. The model was used to illustrate which interventions were most effective leverage points in the local context/system and therefore to align and mobilise prevention efforts of community stakeholders [35].

Participatory modelling processes, such as the one described by Loyo et al. [35], provide an opportunity to understand and develop efficient solutions in the health sector [36, 37]. Participatory modelling, firstly, helps community stakeholders understand how multiple variables, factors and interventions interact. Secondly, simulation modelling can test the potential impact of programs and policies in the ‘safety’ of a virtual environment before they are implemented, saving time, effort, costs and resources. Thirdly, modelling demonstrates potential secondary and tertiary effects (and even unintended consequences) of intervention strategies. Fourthly, modelling can guide and prioritise data collection and facilitate dialogue among stakeholders [36].

The process of participatory simulation modelling involves engaging multidisciplinary stakeholders in a group model-building process and can be used in conjunction with a number of modelling methods [31, 37, 38]. The value of this engagement is the development of a shared mental model of the causal pathways and potential intervention points in the system [39]. A participatory modelling approach enhances stakeholder knowledge and understanding of the system and its dynamics in varying conditions. It identifies and clarifies complex and contested real world problems [33] and the impact of solutions, therefore facilitating the development of action statements based on the evidence [39, 40]. The involvement of key decision-makers in the model development and validation increases their sense of ownership and confidence that the model is valid for their local context. They are therefore more likely to draw on the outputs to inform decisions about priority interventions and policies [23, 37, 39, 41].

Important gaps in knowledge

The application of systems thinking to health improvement is acknowledged as an ongoing challenge [42, 43]. Stakeholder engagement and involvement in the modelling process has been particularly lacking, resulting in unsuccessful projects [42] and a reluctance from ‘non-researchers’ to use models as a decision support tool [33]. A systematic review of the use of simulation modelling to inform surgical patient flow processes found that only half of publications stated that they had produced a model to inform policymakers and health service managers and only 26% actually included policymakers and health service managers in the simulation modelling process [44]. Where policymakers have been included in the simulation modelling process there remains an absence of rigorous analysis of their perspectives on the utility of the model, their learning relating to the development and use of the model, and their commitment to implement the findings of the model [5, 37].

Relationships and collaborations are routinely identified as a key factor in systems approaches [45] and this is particularly true for participatory modelling processes. Important elements for implementing successful systems thinking to address complex issues include the formation of networks and teams, distributed leadership, and strong and effective communication and feedback mechanisms [17]. Understanding the role of participants within the system as well as in the participatory modelling process and bridging professional cultures [45] is key to understanding the factors that will impact on the uptake of simulation modelling as an evidence synthesis tool. Participatory modelling approaches aim to combine multidisciplinary stakeholder perspectives to tackle the social complexity of problems and recognise that different types of knowledge contribute alternative and valuable perspectives to the problem discourse [33].

Evaluation of the participatory simulation modelling process in the health sector has been lacking [5, 41] despite assessment of its efficacy being essential to inform decision-making [5, 37]. Understanding the intricacies of the participatory process [33] and evaluating methods and tools to facilitate participatory modelling is necessary to improve modelling outcomes [4, 31, 37] and further research is required to develop and refine rigorous evaluation methods [39]. The Challenge and Reconstruct Learning (CHaRL) Framework has been proposed by Smajgl and Ward [46] to evaluate participatory modelling processes. This framework can be used for deliberative approaches [47] and involves assessing formalised and facilitated learning among decision-makers and decision influencers at varied policy levels. The key component of the CHaRL framework is the change in perception or belief about assumed causality within the system. In other words, participants’ mental models are challenged by the presentation of different perspectives, scientific evidence and system interactions through the modelling process. The change in mental model can be measured using individual value and attitude/belief orientations recorded by participants pre- and post- the modelling process [46].

Study objectives

Using GDM as a case study

GDM is a complication of pregnancy that is defined as carbohydrate intolerance resulting in hyperglycaemia (abnormally high blood sugar) of variable severity with onset or first recognition during pregnancy [48]. GDM defined in this way includes women with undiagnosed pre-existing diabetes, as well as those for whom the first onset is during pregnancy (especially during the third trimester of pregnancy). The prevalence of GDM is increasing both in Australia and internationally [49].

Identified risk factors for GDM include maternal body mass index of at least 30 kg/m² [50–52], increasing maternal age [52], physical inactivity [50, 52], increasing parity, and ethnicity [53]. Women are also at increased risk if they have a history of GDM [52], previously had a macrosomic baby (birthweight greater than 4000 g), a family history of diabetes [52], polycystic ovary syndrome [52], or a diet low in fibre [54, 55].

Perinatal risks associated with GDM include macrosomia, shoulder dystocia, other birth injuries, hypoglycaemia and perinatal mortality [53, 56]. Long-term risks for the infant from GDM include sustained impairment of glucose tolerance [57], subsequent obesity [58] (although not when adjusted for size) [59], and impaired intellectual achievement [60]. For women, gestational diabetes is a strong risk factor for the development of diabetes later in life [61, 62].

Although the risks associated with gestational diabetes are well recognised, debate remains as to whether screening and treatment to reduce maternal glucose levels reduce these risks [53, 63]. Given this uncertainty, professional groups disagree on whether to recommend routine screening, selective screening based on risk factors for gestational diabetes, or no screening [53]. There is also debate over the efficacy of using a single raised blood glucose result to diagnose GDM [63].

The Australian diagnostic threshold for GDM was changed to be consistent with WHO criteria from January 1, 2015. The WHO report from which the criteria were obtained acknowledges that the evidence for the threshold chosen is weak. However, they argue that the benefits of treatment, i.e. reduction of risk for macrosomia, shoulder dystocia and pre-eclampsia is sufficient justification. Treatment of gestational diabetes once diagnosed is generally medicalised (insulin treatment) and involves intense use of health services, mostly in the third trimester. Investigations of the cost implications of using the lowered diagnostic threshold concluded that cost effectiveness will only be achieved if treatment reduces the risk of caesarean section birth and developing Type 2 diabetes mellitus [64, 65].

Pregnancy has been identified as a point in the life cycle where individuals have increased motivation to commit to health improving behaviours, for example, in smoking cessation [66]. A diagnosis of GDM (or even a glucose tolerance test result that approaches the diagnostic cut-off) may provide a powerful leverage point for multidisciplinary health interventions promoting lifestyle change to reduce the risk of developing diabetes later in life. Almost all women (95%) with a diagnosis of borderline GDM in an Australian study identified that managing their borderline GDM was important or very important for the health of their baby and themselves [67]. Enablers identified by women to implement lifestyle change during pregnancy include family support [66, 67], physical access to programs, knowledge (about diet, exercise and GDM), and motivation levels [67].

Previous models of GDM developed to investigate the cost effectiveness of screening and treatment regimens [64, 65, 68, 69] have provided valuable evidence to inform decision-making. However, these models focussed on an economic evaluation of specific treatments and did not analyse the wider outcomes of policy and program decisions, including the intended and unintended consequences and resource implications of interventions delivered in the health system [70]. Dynamic simulation modelling has been used to investigate the intergenerational impact of GDM on the development of Type 2 diabetes mellitus among First Nations and other population groups in Canada [71]. This model included representations of factors contributing to the development of diabetes mellitus, including changes in behaviour regarding diet and physical activity over time and found that GDM disproportionately contributed to the development of Type 2 diabetes mellitus in First Nations populations compared with other population groups [71].

Dynamic simulation modelling provides an opportunity to explore and compare the implications of health intervention options for GDM services in the Australian Capital Territory (ACT) and to inform policy and program decision-making. The simulations derived from the model can be used to explore the dynamic interaction of risk factors such as maternal weight and weight gain (pre and during pregnancy); the impact of screening earlier or later in pregnancy; the impact of universal or selective screening; the impact of lowering the diagnosis threshold on the number of women diagnosed, health outcomes and health system impacts; the implications of intervention options for prevention and treatment of GDM with different target groups and with different timings (e.g. at the start of pregnancy, during pregnancies, between pregnancies); GDM diagnosis and risk of later development of Type 2 diabetes in the ACT; and the short- and long-term outcomes for mother and baby following treatment for GDM.

The current research project will contribute to knowledge on the application of systems thinking to a localised health system case study by undertaking, validating and evaluating a participatory simulation modelling process focusing on GDM.