Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling

Health Research Policy and Systems201715:83

https://doi.org/10.1186/s12961-017-0245-1

Received: 31 March 2017

Accepted: 5 September 2017

Published: 2 October 2017

Abstract

Background

Evidence-based decision-making is an important foundation for health policy and service planning decisions, yet there remain challenges in ensuring that the many forms of available evidence are considered when decisions are being made. Mobilising knowledge for policy and practice is an emergent process, and one that is highly relational, often messy and profoundly context dependent. Systems approaches, such as dynamic simulation modelling can be used to examine both complex health issues and the context in which they are embedded, and to develop decision support tools.

Objective

This paper reports on the novel use of participatory simulation modelling as a knowledge mobilisation tool in Australian real-world policy settings. We describe how this approach combined systems science methodology and some of the core elements of knowledge mobilisation best practice. We describe the strategies adopted in three case studies to address both technical and socio-political issues, and compile the experiential lessons derived. Finally, we consider the implications of these knowledge mobilisation case studies and provide evidence for the feasibility of this approach in policy development settings.

Conclusion

Participatory dynamic simulation modelling builds on contemporary knowledge mobilisation approaches for health stakeholders to collaborate and explore policy and health service scenarios for priority public health topics. The participatory methods place the decision-maker at the centre of the process and embed deliberative methods and co-production of knowledge. The simulation models function as health policy and programme dynamic decision support tools that integrate diverse forms of evidence, including research evidence, expert knowledge and localised contextual information. Further research is underway to determine the impact of these methods on health service decision-making.

Keywords

Participatory dynamic simulation modelling Decision support Knowledge mobilisation Childhood obesity Alcohol Diabetes in pregnancy

Key messages

  • Participatory dynamic simulation modelling is being implemented as a knowledge mobilisation strategy in Australian health policy settings.

  • Key elements of this knowledge mobilisation approach has included:
    • Moving beyond evidence synthesis to providing dynamic decision support tool to compare policy options.

    • Embedding deliberative methods to build shared understanding of complex issues and intervention outcomes.

    • Emphasising stakeholder participation in the co-production of knowledge.

  • Operationalising participatory simulation modelling relied on:
    • Effective and equal partnerships.

    • Active participation of all stakeholders in the modelling process.

    • Transparency and trust in model outputs to facilitate consensus for action.

Background

The utilisation of evidence has come a long way since the advent of evidence-based medicine – a term coined in the early 1990’s [13] when the leading proponents were described as radicals [4]. Evidence-based healthcare captured the zeitgeist and coalesced into a mainstream movement built on decades of population-based research, clinical epidemiology, critical appraisal and systematic review methods [5]. Interventions aimed at supporting the use of evidence in policy and practice have spawned new theories and frameworks, translation research, and an evolving lexicon [6]. A recent scoping review identified 51 different taxonomies to categorise research translation interventions [7].

Yet, despite great progress and mainstream acceptance of evidence-informed decision-making there remain many operational challenges. Researchers who understand the scientific evidence are often not engaged, or unheard, when important policy decisions are made [8, 9]. Similarly, practitioners familiar with the local context and those who are considered the ‘end users’ of the research are often not engaged in the research process [10]. A common dilemma is the apparent mismatch between the information priorities of policy decision-makers and programme or service funders, and the research priorities of investigators and research funders [11]. To be policy relevant, research must reflect an understanding of decision-making environments, be responsive to end-user needs, and be supported with stakeholder engagement and strategic communication [11, 12]. Contemporary thinking also suggests locally co-created knowledge, derived from researcher and end-user partnerships, preferably on the location where it is to be applied, is particularly useful for addressing policy and practice questions [13, 14]. This is akin to Senge’s description of the art and practice of collective learning [15].

The conceptual and empirical developments of contemporary strategies to support evidence-informed action are reflected in the evolving terminology. While knowledge translation focuses on the transfer of investigator-driven research to policy and practice settings [6, 11, 16], knowledge exchange has emphasised the relational two-way communication required for research uptake to be effective and useful [17, 18]. The most recent adoption of knowledge mobilisation further highlights organisational structures and system design requirements, and more explicitly values the ‘co-creation’ of knowledge [19, 20]. It is also the broadest term to encompass all activities that involve generating, sharing and using research [19]. Best et al. [21, 22] described these developments as the three generations of translation, namely linear, relational and systems-based approaches. Whichever terminology one may prefer, however, it is widely acknowledged that mobilising knowledge for policy and practice is an emergent process, and one that is highly relational, often messy and profoundly context dependent [2327].

Systems thinking and systems science have growing influence on many aspects of public health discourse and research [28, 29]. Important elements of systems thinking include more conscious attention to how new forms of knowledge are “gained, managed, exchanged, interpreted, integrated and disseminated”, and an emphasis on “transdisciplinary, translational and network-centric” science [28]. There are natural synergies between knowledge mobilisation methods and systems science methods. Knowledge mobilisation refers to a range of active approaches deployed to encourage the creation and sharing of research-informed knowledge [30]. Systems science methods encompass a family of approaches that can be used to elucidate the behaviour of complex systems, inform efforts to address one or more system problems [31], and have the capacity to examine both complex health problems and the context in which they are embedded [29, 32, 33]. Key elements of a systems science approach include synthesising diverse knowledge and evidence, exploring the potential for non-linear relationships between contributing factors and unanticipated emergent behaviour of the complex systems (including policy resistance) [31, 34, 35]. The value of systems thinking for conducting reviews of evidence and integrating other forms of knowledge are well described [26, 36]. However, applying systems thinking to knowledge mobilisation is conceptually challenging and difficult to operationalise [37]. A recent multi-method review of knowledge mobilisation across health and other sectors concludes that the most fruitful lessons about the future role of systems thinking will come from natural experiments and case studies [19].

In this paper, we describe our experience of implementing a systems-based approach of participatory dynamic simulation modelling as a knowledge mobilisation strategy in Australian real-world policy settings. We describe how this approach combined both systems science methodology and some of the core elements of knowledge mobilisation best practice using three case studies (two published [38, 39] and one as yet unpublished). We describe the strategies adopted to address both technical issues (e.g. synthesising diverse evidence into a quantifiable model) and socio-political issues (e.g. user engagement and trust), and compile the experiential lessons derived. Finally, we consider the implications of these knowledge mobilisation case-studies and provide evidence for the feasibility of this approach in policy development settings.

Participatory dynamic simulation modelling draws on many elements of knowledge mobilisation best practice

Dynamic simulation modelling is a systems science method that recreates complex systems and human behaviours in a virtual world. These models can answer ‘what if’ questions about the likely impacts over time of different policy and intervention options and combinations so that they can then be deliberated and considered more broadly before implementation in the real world [40, 41]. Dynamic simulation modelling has been used to map health system components and their interactions, synthesise evidence, examine and compare the potential outcomes of interventions, and guide more efficient investment and conscientious disinvestment of resources [41]. This is important for preventive health policy and practice, where decision support tools must have the capacity to steer a course through the complexity of interactions that give rise to real-world public health problems such as the global epidemic of chronic disease [4042].

The concept of ‘evidence-informed decisions’ is challenging in population health policy and practice interventions that require engagement and partnership with sectors outside of health. Many factors, including types of information, opinion and experience, timing, the political cycle, local norms, the influence of external players, and the availability of funds, all influence decision-making [9, 43]. Many of the current ‘big questions’ in public health are complex and not easy to address. These problems have multiple interacting causal factors with competing possible courses of action for decision-makers to choose between, each course of action potentially resulting in complex and unintended consequences [40, 44].

However, to date, the potential of participatory simulation modelling as a knowledge mobilisation tool in the health sector has not been adequately explored. In particular, stakeholder engagement and involvement of end-users in health-related simulation model development has been lacking [41]. This has limited the use of simulation modelling across the range of potential applications, hindered the implementation of model findings [45, 46] and led to a reluctance among ‘non-researchers’ to use models as decision support tools [46, 47]. A systematic review of the use of simulation modelling to inform surgical patient flow processes found that only half of publications stated that the goal of the model was to inform policymakers and health service managers, and only 26% actually included these end-users in the simulation modelling process [10].

Below, we discuss how participatory simulation modelling can build on contemporary knowledge mobilisation approaches to offer a tool for timely and dynamic policy decision support, both by embedding deliberative methods and emphasising the co-production of knowledge in the modelling process. We then reflect on the experience and learnings drawn from three Australian case studies of participatory dynamic simulation modelling conducted in collaboration with jurisdictional health departments [38, 39].

Moving from evidence synthesis to timely and dynamic decision support

Evidence-informed policy and practice has traditionally relied on systematic reviews, evidence summaries and policy briefs to provide decision-makers with rigorous, timely and concise information [4850]. While their inherent value is acknowledged, there are limitations in their use and utility for health policy decision-making [12].

Systematic reviews and meta-analyses synthesise the available evidence to answer the question ‘what do we know about this issue?’ They focus on clear and specific questions and usually have a narrow scope of investigation with limited potential to examine complex questions [51, 52]. These methods produce static reports that rely on decision-makers to navigate the complexity and uncertainty of translating the evidence for their local context and weigh up the options for responding to their problem [53]. Many systematic reviews fail to address the policy implications of their findings [12] in a timely way to inform decision-making [54].

More recently, there has been a shift towards rapid reviews investigating policy questions. Here, the focus is on providing immediate value to addressing the problem at hand. For example, rapid reviews like Evidence Check from the Sax Institute [55, 56] commence with a collaborative process where policymakers and a knowledge broker develop a structured review proposal that describes the policy issue or decision for which the evidence review is required, and articulate the review questions and scope. The process aims to ensure the review will provide policymakers with information specific to their decision and context in a timely way. This collaborative approach has been shown to be well suited for assisting in planned policymaking processes and choosing between specific policy options [55, 57]. The use of knowledge brokers is integral to organising the interactive process between researchers and policymakers so that they can co-produce feasible and research-informed policy options [56].

Policy briefs begin with a policy issue and present evidence to answer the question ‘What should we do?’ A policy brief provides a rationale for choosing a policy alternative or course of action based on the synthesised research findings. They are more practical, flexible and timely in supporting evidence-informed decision-making [49] and can also consider how the evidence fits with prevailing values, beliefs and political context [49]. However, the final product is still a static assessment that is unable to adequately account for changes over time or test the prevailing real-world hypotheses and assumptions [53, 58].

However, participatory dynamic simulation modelling processes go further, providing a platform for explicit synthesis of empirical evidence, local data, expert- and practice-based knowledge, conceptual models and theory to construct, quantify and test a detailed representation of causal factors and the mechanisms of intervention effects [40, 41, 58, 59]. The resulting dynamic model becomes a decision support tool that can step beyond comprehensive approaches, for example, in the prevention of chronic disease, to be used as a ‘what if’ tool to simulate various policy and practice scenarios, and systematically explore the trade-offs of a range of intervention options [41, 58].

Embedding deliberative methods

An important strategy in knowledge mobilisation theory and practice is the incorporation of deliberative methods. The value of the deliberative process is that it increases understanding of the evidence, and of the competing issues and values, through the engagement and contribution of participants with different perspectives [60]. By deliberating on a problem and its potential solutions, participants strengthen their capacity to address a policy issue and gain confidence in influencing the policy agenda [60].

Processes such as deliberative dialogues involve group interactions that integrate and interpret multiple forms of evidence to inform policy development [61]. Key elements of a deliberative dialogue process include a meeting environment that is conducive to open deliberation about a policy issue, bringing together a mix of participants that ensures fair representation of all relevant interests, and fostering a more equal knowledge base among participants through the presentation of research evidence [60].

Deliberative approaches tend to emphasise the rigour and fairness of the process and try not to anticipate or pre-determine the outcomes of the deliberation. They rely on skilled and neutral facilitation and, while consensus building may be achieved, it is not the primary aim [62]. This requires flexibility and acceptance that the boundaries and scope may be changed as people reflect and discuss the problem, and sometimes modify the questions they want to address.

Thus, translation of research has progressed from managed and controlled dissemination initiatives with pre-defined targets [63]. For knowledge mobilisation, the social, relational and contested nature of true deliberative dialogue or ‘exchange’ relies on negotiated meanings and less predictable outcomes [62, 64, 65].

Participatory dynamic simulation modelling incorporates a deliberative process where stakeholders articulate and develop their understanding of how multiple variables, factors and interventions interact [66, 67], and provides a neutral platform for engaging stakeholders with conflicting views [59]. The participatory model development necessitates in depth deliberation to map a shared mental model of the causal pathways for the focus issue, and the mechanisms by which interventions have an effect on outcomes [68]. The map is then quantified, drawing on research evidence and other data sources through an iterative process of theory testing and building in collaboration with participants. Model outputs are compared with real-world historic data patterns across a range of indicators to establish the validity of the model as an accurate representation of the real-world system. The resulting model becomes a decision support tool that can be used to consider and compare alternative policy options [66, 67]. The model can be refined, updated and customised through ongoing dialogue. Both the process of model development and the results produced by the model enhance stakeholder knowledge and understanding of the system and its dynamics in varying conditions. The process identifies and clarifies complex and contested real world problems [47] and the impact of solutions, and facilitates the implementation of actions based on the available evidence [68, 69].

Emphasising co-production of knowledge

Co-creation and co-production are two terms used to refer to the process of individuals from different sectors working together to produce an output or outcome such as goods, services or research [70]. Co-production of evidence aims to overcome the often described disconnect between researchers and research end-users, such as health policymakers and programme planners [14]. This concept has been applied to social service design and delivery [71] and increasingly to health research [7274].

Research translation is embedded in the co-production and partnership approach as the end users are active participants in, and in some cases the drivers for, all phases of the research project [14]. The key elements of co-creation include involving participants as active and equal partners from beginning to end, encouraging reciprocity and sharing of resources and knowledge, and aiming for a ‘transformative’ outcome, i.e. where the research builds capacity and/or has a practical impact on decision-making [14, 71]. The establishment of effective co-production partnerships is an iterative journey, where structures, boundaries and even the purpose of the project are re-negotiated throughout the project dialogue [75].

Relationships and collaborations are routinely identified as key factors in systems approaches [76]. Participatory dynamic modelling provides a structure to facilitate multidisciplinary partnerships, co-learning and co-production. The participatory approach adopts co-production as its driving principle and places the end-user decision-makers at the centre of the process. The decision-makers define the model scope and purpose, and engage multidisciplinary expert stakeholders in the model design and parameterisation (and contribute the identification of data to be used in the model).

Participatory dynamic simulation modelling involves engaging multidisciplinary stakeholders in a deliberative group model-building process where participants discuss evidence and share knowledge about the causal mechanism of the issue being modelled and where and how interventions have their effect within the articulated mechanism. Participatory modelling approaches aim to combine diverse perspectives to tackle the social complexity of problems and recognise that different types of knowledge contribute alternative and valuable perspectives to the problem discourse [47, 59]. The involvement of decision-makers as participants in the model development and validation increases their sense of ownership and confidence that the model is valid for their local context; they are therefore more likely to draw on the model’s outputs to inform decisions about priority interventions and policies [68, 77, 78].

Reflections on process and early learning from participatory dynamic simulation modelling as a knowledge mobilisation approach

The Australian Prevention Partnership Centre (http://preventioncentre.org.au/), in collaboration with jurisdictional governments, has pioneered the co-production of sophisticated, multiscale dynamic simulation models to support policy and practice. In developing these models, researchers partnered with health departments, clinicians and regional planners in collaboration with a multidisciplinary group of stakeholders using a participatory process [38, 39]. The case studies are described in Box 1.

Box 1. Case study descriptions

Case Study 1. Model behaviour: A systems approach to reducing alcohol-related harm

 This project was implemented as a collaboration between The Australian Prevention Partnership Centre, the New South Wales Ministry of Health (NSW Health), and local and national alcohol researchers, clinicians and programme planners to inform strategies for reducing alcohol-related harms in NSW.

 Alcohol misuse is a complex, systemic problem. Globally, alcohol has been estimated to cause 3.3 million deaths each year, and the costs of alcohol-related harms amount to more than 1% of gross national product in high-income countries. In Australia, alcohol accounts for approximately 3.2% of the total burden of disease and injury, and is estimated to cost AU$15.3 billion each year [79, 80].

 The design of effective responses to this problem has been challenged by a lack of clarity on the mechanisms driving alcohol misuse and its associated harms, differing views of stakeholders regarding the most appropriate and effective intervention approaches, a lack of evidence supporting commonly implemented and acceptable intervention approaches, and strong evidence for less acceptable interventions. As a consequence, political considerations, community advocacy and industry lobbying contribute to a hotly contested debate on what is the most appropriate course of action.

 The developed model uniquely captures the heterogeneity of drinking behaviours across the NSW population, the dynamics of those drinking behaviours across the life course, the acute and chronic harms that arise from those behaviours, and the differential effects of interventions across subgroups in the population. Testing of the model demonstrated its ability to reproduce a range of real world data patterns, which provides confidence that the model can produce robust forecasts of the comparative impacts of interventions into the future. The model is currently being used to engage with broader policy stakeholders to demonstrate the value of such models in informing effective and acceptable strategies for reducing alcohol-related harms [38].

Case Study 2. Premier’s Priority Project – reducing childhood overweight and obesity by 5%

 In September 2015, the NSW Premier unveiled 30 State priorities to grow the economy, deliver infrastructure, protect the vulnerable and improve health, education and public services across NSW. Included in these areas of focus were the 12 Premier’s Priorities, including an ambitious target to reduce childhood overweight and obesity in children by 5% over 10 years.

 Based on population projections and the anticipated impact of enhancing the existing suite of interventions delivered by NSW Health, it was estimated that additional strategies, or combinations of strategies, would be required to achieve the Premier’s target. However, the complexity of the problem and uncertainty about where best to target resources and efforts presented a challenge to decision-makers. To address this, the Australian Prevention Partnership Centre in partnership with NSW Health undertook to co-develop a system dynamics model of childhood overweight and obesity.

 The model development process engaged a broad range of multidisciplinary stakeholders working in the area of childhood obesity spanning the fields of academia, service delivery, policy, planning and infrastructure. Through a series of participatory workshops the problem was collaboratively mapped and interventions to be included in the model prioritised. The map was conceptualised as a computational model, quantified, tested and validated against historic data, and iteratively refined through feedback sought during and between workshops.

 The model is being used by NSW Health and their stakeholders to test the likely impacts of a range of policies and programmes, and to inform the combination of interventions that might achieve the Premier’s target.

Case Study 3. Simulation modelling for Diabetes in Pregnancy (DIP) in the Australian Capital Territory (ACT)

 This project was implemented as a collaboration between The Australian Prevention Partnership Centre, ACT Health Directorate (ACT Health), local and national researchers, clinicians and policymakers. DIP is increasing both in the ACT and Australia, and diabetes services are having difficulty meeting demand with existing resources. The increase in DIP is associated with increasing prevalence of risk factors such as overweight and obesity, older maternal age and increasing numbers of women from high-risk ethnic groups. Changes to diagnostic screening has resulted in women being diagnosed with DIP earlier in their pregnancy and therefore requiring services for a longer period of time. Women are also more frequently presenting with a number of risk factors resulting in more complex care needs.

 A dynamic simulation model focusing on DIP from an ACT perspective was developed. The national context was considered in the model development, with the model being considered a proof of concept with the potential to expand more broadly.

 The model considers the short, intermediate and long-term implications of the increasing prevalence of risk factors for DIP. Prevention of risk factors was prioritised in the model as small delays in the development of diabetes will have large implications for the longer term burden of disease and costs to the health system.

 Alternative models of care for DIP were considered in the model. The rising prevalence of DIP is having a significant impact on health service demand and resources, and the need to ‘do things differently’ was identified by participants. The model informs the investments for intervention in DIP, including both clinical and population health interventions. Workload and resource use has been incorporated into the model to enable it to act as a resource allocation decision support tool. At the time of publication, this model was being finalised.

The participatory simulation modelling processes and activities utilised in these case studies have been described in detail elsewhere [38, 39]. However, a brief overview of the process and examples of activities are provided in Box 2 to give context for the discussion below.

Box 2. Overview of the process and examples of activities

Project planning and engagement (Fig. 1 )

 Early engagement with stakeholders for each case study was undertaken to identify a priority problem, and determine and define policy priorities requiring decision support methods. A domain expert, preferably from the primary partner organisation (partner), was identified to be a lead collaborator in the project (lead domain expert). This role included supporting the engagement of stakeholders and co-facilitating workshops.

 Project planning meetings were held to clearly define the aspects of the problem to be modelled and its scope and boundaries, as well as to identify key outputs of interest and intervention options to be included and tested by the model.

 Experts and key participants with an important ‘stake’ in the topic were identified and invited to participate in the model development group (participants). Group composition was purposefully considered to ensure inclusion of a diverse range of views and identification of participants who were considered reliable and reputable representatives of broader stakeholder groups (stakeholders). Background reading material regarding simulation modelling and the topic of interest was sent to participants prior to the workshop to provide a platform of common understanding.

Model building and validation

 Through a series of participatory workshops, the model building group, informed by collated evidence and data, collaboratively identified and mapped the key risk factors and likely causal pathways leading to outcomes of interest for the focus topic of the model.

 The proposed model architecture was presented at the first workshop, and then subsequent versions of the model were developed to reflect participant language, input and feedback as well as providing increased detail and maturity.

 Participants were familiarised with the model infrastructure using paper-based physical representations. For example, during one activity, participants built a physical representation of the model, with model components represented in card and tape. Participants worked collaboratively to document factors that contribute to the problem being modelled and mapped these directly onto the card and tape representation (Fig. 2).

 Similar activities were conducted to involve participants in mapping the mechanisms through which interventions would impact the model (Fig. 3).

 The interim conceptual map or model was tested and validated in collaboration with the model building group during each workshop.

 The workshop structure was flexible to account for differences in group size and incorporated a range of activities with the whole group or smaller sub-groups as appropriate to allow participants to raise issues, negotiate perspectives and build consensus. For activities where the group was split, the modelling team allocated participants to ensure each sub-group included a range of perspectives and areas of expertise, and to encourage productive group dynamics.

Consensus building for policy actions

 Final half-day workshops and follow-up webinars were conducted where the model was presented back to the model building group for verification, discussion, consensus, feedback of results and further input on preferred visualisation of model outputs.

 Outputs from modelled scenarios were presented to participants to facilitate the development of new insights and knowledge about the likely impact of interventions and discussion about potential policy actions.

 Examples of the user interface and model outputs are presented in Figs. 4 and 5. These figures illustrate how model users create scenarios to test and compare the outcomes for different combinations of selected interventions. Figure 4 includes the user interfaces from the Alcohol-related harm (top image) and the Premier’s Priority (bottom image) case studies.

 The model outputs take the form of dynamic visualisations and graphs that represent model outcomes for created scenarios, e.g. for variations of intervention effectiveness and reach. These can be compared against benchmark or ‘business as usual’ model outputs. Figure 5 presents example outputs from the Alcohol-related harm case study.

Fig. 1

Participatory simulation modelling phases used in the case studies, from Atkinson et al., 2017 [40]

Fig. 2

Causal factor mapping activity – Alcohol-related harm case study workshop

Fig. 3

Intervention mapping to model architecture activity – Diabetes in Pregnancy case study workshop

Fig. 4

User interfaces from Alcohol-related harm (top image) and Premier’s Priority Project (bottom image) case studies demonstrating the facility for participants to choose intervention combinations and vary parameters to generate unique scenarios

Fig. 5

Example model output from Alcohol-related harm case study demonstrating the output visualisations for participants generated scenarios and comparison against baseline

Key aspects of operationalising participatory dynamic simulation modelling

In the remainder of this paper we draw on our experience of operationalising participatory dynamic simulation modelling to support chronic disease prevention policy and practice in Australia. We consider three key aspects of the process, (1) establishing partnerships with stakeholders, (2) engaging participants actively in the modelling process and (3) using co-production methods to build trust in the model and its outputs. We then discuss some of the lessons and implications for adopting these approaches in contemporary knowledge mobilisation practice.

Establishing effective partnerships with stakeholders

In each of the case studies, the focus of the model was proposed by Australian Capital Territory Health or New South Wales Health as a priority area with current local concern, complex causal risk factors and as issues where previous policy responses had limited impact. The health jurisdictions were therefore coming to the process with a view that they needed to do things differently and were motivated to work on innovative solutions.

Engagement with these primary partner organisations (partners) continued throughout the modelling process, from identifying relevant subject matter experts to be involved (participants), to soliciting input on relevant data and literature sources, negotiating the model purpose, scope and structure, and encouraging involvement in the facilitation of modelling workshops.

Identifying and including a lead domain expert for each case study, e.g. a public health practitioner or clinician, who was well respected and associated with the partner, increased engagement, solidified the partnerships and built trust in the modelling process. These lead domain experts acted as co-facilitators for model development workshops, along with the project leader from the modelling team. The combination of domain and modelling expertise allowed workshop co-facilitators to navigate interdisciplinary participation through a process of developing a common language and understanding to facilitate model development.

Engaging participants actively in the modelling process

Facilitated workshop activities were designed to involve participants actively in the modelling process (examples in Box 2). These methods supported participant engagement and investment in the model as they deliberated and negotiated with each other to prioritise causal factors, their interactions, and interventions and outcomes to be captured in the model. Significant learning occurred through these deliberative dialogues, with participants reporting that their ‘interaction was key’ to the modelling process. The mapping activities provided an interactive opportunity for participants to synthesise their collective knowledge and expertise with quantitative evidence.

The practical hands-on mapping activities used during workshops (Box 2) also familiarised participants with the model architecture. The model architecture (the diagrammatic representation of the computer model) physically represented how the identified causal factors, interventions and resulting outputs were incorporated into the model. This allowed for two-way learning as increased familiarity and confidence in understanding the model architecture enabled participants to critique and provide feedback to modellers to ensure the model accurately represented their shared understanding. Taking an iterative approach facilitated collective learning, and the demonstration of the evolving model at the second and subsequent workshops further validated and improved model design.

At times, the priorities of policy partners (in terms of interventions and outputs to be included in the model) differed from those of subject matter experts. The participatory process of negotiation helped to build consensus on what to prioritise in the model and enhanced each participant group’s understanding of the others’ knowledge and research or policy. Expert facilitation skills were necessary to draw out diverse contributions, maintain engagement in the process and negotiate compromises where necessary. Explicit processes, including voting, were used to democratically resolve disagreements and to clarify priorities in model development.

Co-production built trust in model outputs and facilitated consensus for action

The strong partnerships and active engagement of partners and participants throughout the iterative model development were critical for building trust in model outputs and providing the best opportunity for impact on policy and programme decisions.

The use of co-production methods as described above increased transparency in the model building process. Demonstrating the model conceptualisation to participants at each workshop and highlighting their contributions increased participants’ understanding of the model. This transparency reinforced the value of their participation and their ownership of the model, and provided an opportunity to establish the face validity of models against expert and local knowledge.

Another important aspect to building trust in model outputs was to encourage discussion about the limitations and assumptions in the model design and available data sources. Documentation of data sources and assumptions built into the models was shared with participants to critically evaluate and provide feedback.

Building participants’ trust in the model was necessary for its acceptance by stakeholders/experts who were not involved in its development (stakeholders). Involvement of key opinion leaders in the model development groups brought credibility to the models as participants acted as ambassadors for the model within their broader stakeholder groups. Diversity of expertise within the participant group was also important so different stakeholder groups felt their perspective had been represented.

When some model outputs did not confirm long held beliefs about likely effects of interventions and their combinations, there was robust debate about the implications and caution in using such results to inform decision-making. In these situations, it was particularly useful to invite stakeholders to interact with the model, challenge their assumptions, provide alternative data and test their expectations against model outputs. Iterative, open and non-defensive communication was critical to facilitating these interactions, advancing understanding of the complex problem and building trust in the decision support tool.

Model validation is an essential stage of all model development, including models developed using participatory approaches [77]. Demonstrating to partners, participants and stakeholders that the models reproduced historic data patterns across a range of indicators confirmed their validity, and built confidence that the models would produce robust forecasts into the future.

Collaborative processes were also used to maximise the potential usability of the model. Participant engagement with the model was encouraged to test and refine the user interface. This user testing ensured that the interface was intuitive and accessible for a diverse range of users.

Conclusion

The participatory dynamic simulation modelling processes utilised in these case studies built on knowledge mobilisation best practice and produced dynamic decision support tools that integrated diverse forms of evidence, including research evidence, expert knowledge and localised contextual information. The participatory approach placed end-users at the centre of the process and embedded deliberative methods and co-production of knowledge. Policymakers, researchers, scientists, clinicians, consumers and modellers collaborated and explored policy and health service scenarios for priority public health topics.

An important element of co-production in these case studies was equal partnering with key stakeholders to negotiate the priority issue to be modelled. These were ‘hot topics’ that were current, locally relevant, had complex causal mechanisms, and for which decision-makers needed to decide between competing courses of action. It was in these circumstances that participatory dynamic modelling provided an opportunity for policy and programme options and combinations to be tested within a safe, simulated environment before being implemented in the real world. The case studies revealed valuable lessons for the participatory dynamic simulation modelling process in health policy settings. The case study topics were complex and multi-faceted, and the diverse representation of stakeholders in the modelling groups was essential as no one individual could be an expert on all aspects of the issue. An aspect of the process was thus to emphasise the need for knowledge sharing among stakeholders and to develop a common understanding of the issue, and of the potential interventions to address it. Differential participation did occur in some of the workshops, e.g. participants sometimes deferred to those they perceived to have greater authority or expertise for particular aspects of the content. However, the workshop facilitators promoted the value of diverse perspectives in building a robust model and regularly sought to draw out those who were less vocal.

Consistent with other knowledge mobilisation approaches, the participatory process was time consuming and required ongoing efforts to maintain and coordinate diverse engagements. However, the challenges were outweighed by the positive outcomes of effective collaborative networks, co-production of knowledge, and capacity to integrate diverse evidence and expert opinion. In our case study settings, many participants had limited or no prior experience with dynamic simulation models or the modelling process. An important dimension of the knowledge mobilisation process was the translation that occurred between disciplines, e.g. clinicians, computer scientists and population health professionals, to ensure that everyone understood each other’s perspectives and were working toward a common goal.

The processes of data gathering and synthesis commonly highlighted gaps in local programme outcome data, as well as in the published literature. For example, locally available evaluation data was frequently limited to process and participation measures, rather than programme outcomes and effectiveness. Often, the local contextually relevant data was utilised and triangulated with other potentially more reliable but less locally relevant sources to inform the models. Whilst evidence gaps are an ongoing challenge in all settings, the process of uncovering these gaps through the participatory process, and prioritising data needs through sensitivity testing in the model, provided important information for prioritising future research and guiding refinement of local programme evaluations and routine data collection.

An important challenge of knowledge mobilisation using this participatory modelling approach was the building of trust in the model outputs. This was less of an issue when the model outputs confirmed existing preconceptions of underlying causal mechanisms, but significantly more challenging when model outputs were contrary to participants’ long held beliefs. This could be particularly challenging for subject matter experts who were required to reassess their prior expectations. However, the dynamic, interactive nature of the models as decision support tools facilitated ongoing dialogue and negotiation with stakeholders and developed understanding and trust.

Our analysis of participatory model building methods is ongoing. Further activities in this programme of research involve evaluating the perceived value of the participatory process; the commitment and confidence of partners and participants to implement policy and programme decisions identified through the modelling process; and the impact of the process, i.e. how model outputs will be used to inform policy and programme decisions in the local public health settings.

Declarations

Acknowledgements

The authors acknowledge the valuable contributions of Ms Jocelyne Aldridge, NSW Ministry of Health, who made early contributions to this manuscript; and Professor John Wiggers, University of Newcastle, and Associate Professor Alison Kent, Australian National University, who co-facilitated the model development workshops described in the case studies.

Funding

This research was funded through The Australian Prevention Partnership Centre, which receives funds from the National Health and Medical Research Council of Australia (NHMRC) through its partnership centre grant scheme (Grant ID: GNT9100001) with NSW Health, ACT Health, The Australian Government Department of Health and the Hospitals Contribution Fund of Australia Research Foundation. The contents of this paper are solely the responsibility of the individual authors and do not reflect the views of the NHMRC or funding partners. The University of Notre Dame provided the following financial support for Case Study 3 in the form of an Australian Postgraduate Award scholarship and CRN top-up scholarship for supervision travel expenses.

Availability of data and materials

Not applicable.

Authors’ contributions

LF, LR and JA conceptualised the manuscript and LF wrote the first draft. All authors made important intellectual contributions to multiple draft revisions. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Case Study 3 was reviewed and approved as low risk by the ACT Health Human Research Ethics Committee (ACTHLR.15.150) and the University of Notre Dame Human Research Ethics Committee (0151195).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
ACT Government, Health Directorate
(2)
The Australian Prevention Partnership Centre, Sax Institute
(3)
School of Medicine, University of Notre Dame
(4)
Sydney Medical School, University of Sydney
(5)
The Australian National University
(6)
Adaptive Care Systems
(7)
NSW Ministry of Health

References

  1. Guyatt GH. Evidence-based medicine. ACP J Club. 1991;114(2):A16.Google Scholar
  2. Eddy DM. Practice policies: where do they come from? JAMA. 1990;263(9):1265.View ArticlePubMedGoogle Scholar
  3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71–2.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Eddy DM. The origins of evidence-based medicine—a personal perspective. Virtual Mentor. 2011;13(1):55–60.View ArticlePubMedGoogle Scholar
  5. Green LW. Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Fam Pract. 2008;25 Suppl 1:i20–24.View ArticlePubMedGoogle Scholar
  6. Rychetnik L, Bauman A, Laws R, King L, Rissel C, Nutbeam D, Colagiuri S, Caterson I. Translating research for evidence-based public health: key concepts and future directions. J Epidemiol Community Health. 2012;66:1187–92.View ArticlePubMedGoogle Scholar
  7. Lokker C, McKibbon K, Colquhoun H, Hempel S. A scoping review of classification schemes of interventions to promote and integrate evidence into practice in healthcare. Implement Sci. 2015;10:27.View ArticlePubMedPubMed CentralGoogle Scholar
  8. Alexander C. Outine the invisible academic: how researchers can get the hearing they deserve. In: Issue Paper Series (No 5). Melbourne: University of Melbourne, Melbourne Sustainable Society Institute; 2015.Google Scholar
  9. Brownson RC, Royer C, Ewing R, McBride TD. Researchers and policymakers: travelers in parallel universes. Am J Prev Med. 2006;30(2):164–72.View ArticlePubMedGoogle Scholar
  10. Sobolev BG, Sanchez V, Vasilakis C. Systematic review of the use of computer simulation modeling of patient flow in surgical care. J Med Syst. 2011;35(1):1–16.View ArticlePubMedGoogle Scholar
  11. Woolf SH, Purnell JQ, Simon SM, Zimmerman EB, Camberos GJ, Haley A, Fields RP. Translating Evidence into Population Health Improvement: Strategies and Barriers. Ann Rev Public Health. 2015;36:463–82.View ArticleGoogle Scholar
  12. Kite J, Indig D, Mihrshahi S, Milat A, Bauman A. Assessing the usefulness of systematic reviews for policymakers in public health: a case study of overweight and obesity prevention interventions. Prev Med. 2015;81:99–107.View ArticlePubMedGoogle Scholar
  13. Bowen S, Graham ID. Integrated knowledge translation. In: Knowledge Translation in Health Care. London: Wiley Blackwell, BMJ Books; 2013. p. 14–23.View ArticleGoogle Scholar
  14. Heaton J, Day J, Britten N. Inside the “black box” of a knowledge translation program in applied health research. Qual Health Res. 2015;25(11):1477–91.View ArticlePubMedPubMed CentralGoogle Scholar
  15. Senge PM. The fifth discipline: The art and practice of the learning organization. New York: Broadway Business; 2006.Google Scholar
  16. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24.View ArticlePubMedGoogle Scholar
  17. Lomas J. Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff (Millwood). 2000;19(3):236–40.View ArticleGoogle Scholar
  18. Lee RG, Garvin T. Moving from information transfer to information exchange in health and health care. Soc Sci Med. 2003;56(3):449–64.View ArticlePubMedGoogle Scholar
  19. Davies HTO, Powell AE, Nutley SM. Mobilising knowledge to improve UK health care: learning from other countries and other sectors - a multimethod mapping study, Health Services and Delivery Research. 3rd ed. Southampton: NIHR Journals Library; 2015.Google Scholar
  20. Fitzgerald L, Harvey G. Translational networks in healthcare? Evidence on the design and initiation of organizational networks for knowledge mobilization. Soc Sci Med. 2015;138:192–200.View ArticlePubMedGoogle Scholar
  21. Best A, Terpstra JL, Moor G, Riley B, Norman CD, Glasgow RE. Building knowledge integration systems for evidence-informed decisions. J Health Organ Manag. 2009;23(6):627–41.View ArticlePubMedGoogle Scholar
  22. Best A, Holmes B. Systems thinking, knowledge and action: towards better models and methods. Evid Policy. 2010;6(2):145–59.View ArticleGoogle Scholar
  23. Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88(4):444–83.View ArticlePubMedPubMed CentralGoogle Scholar
  24. Greenhalgh T, Wieringa S. Is it time to drop the ‘knowledge translation’ metaphor? A critical literature review. J R Soc Med. 2011;104(12):501–9.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Ward V, Smith S, House A, Hamer S. Exploring knowledge exchange: a useful framework for practice and policy. Soc Sci Med. 2012;74(3):297–304.View ArticlePubMedGoogle Scholar
  26. Riley B, Norman CD, Best A. Knowledge integration in public health: a rapid review using systems thinking. Evid Policy. 2012;8(4):417–31.View ArticleGoogle Scholar
  27. Kontos PC, Poland BD. Mapping new theoretical and methodological terrain for knowledge translation: contributions from critical realism and the arts. Implement Sci. 2009;4:1.View ArticlePubMedPubMed CentralGoogle Scholar
  28. Leischow SJ, Best A, Trochim WM, Clark PI, Gallagher RS, Marcus SE, Matthews E. Systems thinking to improve the public’s health. Am J Prev Med. 2008;35(2):S196–203.View ArticlePubMedPubMed CentralGoogle Scholar
  29. Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annu Rev Public Health. 2012;33:357–76.View ArticlePubMedPubMed CentralGoogle Scholar
  30. Davies HTO, Powell AE, Nutley SM. Mobilising knowledge to improve UK health care: learning from other countries and other sectors–a multimethod mapping study. Health Serv Deliv Res. 2015;3(27):1–190.View ArticleGoogle Scholar
  31. Hassmiller Lich K, Ginexi EM, Osgood ND, Mabry PL. A call to address complexity in prevention science research. Prev Sci. 2013;14(3):279–89.View ArticleGoogle Scholar
  32. Huang TT, Drewnosksi A, Kumanyika S, Glass TA. A systems-oriented multilevel framework for addressing obesity in the 21st century. Prev Chronic Dis. 2009;6(3):A82.PubMedPubMed CentralGoogle Scholar
  33. Burke JG, Lich KH, Neal JW, Meissner HI. Enhancing dissemination and implementation research using systems science methods. Int J Behav Med. 2015;22(3):283–91.View ArticlePubMedPubMed CentralGoogle Scholar
  34. Mabry PLP, Marcus SEP, Clark PIP, Leischow SJP, Méndez DP. Systems science: a revolution in public health policy research. Am J Public Health. 2010;100(7):1161–3.View ArticlePubMedPubMed CentralGoogle Scholar
  35. Meadows DH. Leverage Points: Places to Intervene in a System. Hartland: The Sustainability Institute; 1999.Google Scholar
  36. Best A, Greenhalgh T, Lewis S, Saul JE, Carroll S, Bitz J. Large-system transformation in health care: a realist review. Milbank Q. 2012;90(3):421–56.View ArticlePubMedPubMed CentralGoogle Scholar
  37. Holmes B, Noel K. Time to shift from systems thinking-talking to systems thinking-action. Int J Health Policy Manag. 2015;4(4):245–7.View ArticlePubMedPubMed CentralGoogle Scholar
  38. Atkinson J-A, O’Donnell E, McDonnell G, Mitchell J, Freebairn L, Indig D, Rychetnik L. Dynamic simulation modelling of policy responses to reduce alcohol-related harms: rationale and procedure for a participatory approach. Public Health Res Pract. 2017;27(1).Google Scholar
  39. Freebairn L, Atkinson J, Kelly P, McDonnell G, Rychetnik L. Simulation modelling as a tool for knowledge mobilisation in health policy settings: a case study protocol. Health Res Policy Syst. 2016;14:71.View ArticlePubMedPubMed CentralGoogle Scholar
  40. Atkinson J-A, Page A, Wells R, Milat A, Wilson A. A modelling tool for policy analysis to support the design of efficient and effective policy responses for complex public health problems. Implement Sci. 2015;10:26.View ArticlePubMedPubMed CentralGoogle Scholar
  41. Atkinson J-A, Wells R, Page A, Dominello A, Haines M, Wilson A. Applications of system dynamics modelling to support health policy. Public Health Res Pract. 2015;25(3):e2531531.View ArticlePubMedGoogle Scholar
  42. World Health Organization. Global Status Report on Noncommunicable Diseases. 2014. http://www.who.int/nmh/publications/ncd-status-report-2014/en/. Accessed 16 Sept 2017.
  43. Armstrong R, Pettman TL, Waters E. Shifting sands - from descriptions to solutions. Public Health. 2014;128(6):525–32.View ArticlePubMedGoogle Scholar
  44. Kelly P. Epidemiologically-led public health action. Keynote address. Canberra: Australian Epidemiological Association Annual Scientific Meeting; 2016.Google Scholar
  45. Naseer A, Eldabi T, Young TP. RIGHT: A toolkit for selecting healthcare modelling methods. J Simulation. 2010;4:2–13.View ArticleGoogle Scholar
  46. Jahangirian M, Naseer A, Stergioulas L, Young T. Simulation in health-care: lessons from other sectors. Oper Res. 2012;12(1):45–55.Google Scholar
  47. Seidl R. A functional-dynamic reflection on participatory processes in modeling projects. Ambio. 2015;44(8):750–65.View ArticlePubMedPubMed CentralGoogle Scholar
  48. Lavis J, Posada F, Haines A, Osei E. Use of research to inform public policymaking. Lancet. 2004;364:1615–21.View ArticlePubMedGoogle Scholar
  49. Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 13: Preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S13.View ArticlePubMedPubMed CentralGoogle Scholar
  50. Lavis JN. How can we support the use of systematic reviews in policymaking? PLoS Med. 2009;6(11):e1000141.View ArticlePubMedPubMed CentralGoogle Scholar
  51. Woolf SH, Purnell JQ, Simon SM, Zimmerman EB, Camberos GJ, Haley A, Fields RP. Translating evidence into population health improvement: strategies and barriers. Annu Rev Public Health. 2015;36:463–82.View ArticlePubMedGoogle Scholar
  52. Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Syst Rev. 2015;4(1):36.View ArticlePubMedPubMed CentralGoogle Scholar
  53. Rychetnik L, Redman S, Atkinson J-A, Overs M. Knowledge and policy dialogues discussion paper. Sydney: The Australian Prevention Partnership Centre; 2015.Google Scholar
  54. Lavis J, Davies H, Oxman A, Denis J-L, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10 Suppl 1:35–48.View ArticlePubMedGoogle Scholar
  55. Moore GM, Redman S, Turner T, Haines M. Rapid reviews in health policy: a study of intended use in the New South Wales’ Evidence Check programme. Evid Policy. 2015;12(4):505–19.View ArticleGoogle Scholar
  56. Campbell D, Donald B, Moore G, Frew D. Evidence check: knowledge brokering to commission research reviews for policy. Evid Policy. 2011;7(1):97–107.View ArticleGoogle Scholar
  57. Moore G, Redman S, D’Este C, Makkar S, Turner T. Does knowledge brokering improve the quality of rapid review proposals? A before and after study. Syst Rev. 2017;6(1):23.View ArticlePubMedPubMed CentralGoogle Scholar
  58. O'Donnell E, Atkinson J-A, Freebairn L, Rychetnik L. Participatory simulation modelling to inform public health policy and practice: rethinking the evidence hierarchies. J Public Health Policy. 2017;38(2):203–15.View ArticlePubMedGoogle Scholar
  59. Voinov A, Gaddis E. Values in participatory modeling: Theory and practice. In: Gray S, Paolisso M, Jordan R, Gray S, editors. Environmental Modeling with Stakeholders. Thery, Methods and Applications. Cham: Springer; 2017. p. 47–63.View ArticleGoogle Scholar
  60. Boyko JA, Lavis JN, Abelson J, Dobbins M, Carter N. Deliberative dialogues as a mechanism for knowledge translation and exchange in health systems decision-making. Soc Sci Med. 2012;75(11):1938–45.View ArticlePubMedGoogle Scholar
  61. Culyer AJ, Lomas J. Deliberative processes and evidence-informed decision making in healthcare: do they work and how might we know? Evid Policy. 2006;2(3):357–71.View ArticleGoogle Scholar
  62. Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S14.View ArticlePubMedPubMed CentralGoogle Scholar
  63. Spyridonidis D, Hendy J, Barlow J. Leadership for knowledge translation: the case of CLAHRCs. Qual Health Res. 2015;25(11):1492–505.View ArticlePubMedGoogle Scholar
  64. Williams C. Transfer in context: replication and adaptation in knowledge transfer relationships. Strateg Manag J. 2007;28(9):867–89.View ArticleGoogle Scholar
  65. Nutley SM, Walter I, Davies HTO. Using Evidence: How Research Can Inform Public Services. Bristol: The Policy Press; 2007.View ArticleGoogle Scholar
  66. Hovmand PS. Community Based System Dynamics. New York: Springer; 2014.View ArticleGoogle Scholar
  67. Hovmand PS, Andersen DF, Rouwette E, Richardson GP, Rux K, Calhoun A. Group model‐building ‘scripts’ as a collaborative planning tool. Syst Res Behav Sci. 2012;29(2):179–93.View ArticleGoogle Scholar
  68. Rouwette EAJA, Korzilius H, Vennix JAM, Jacobs E. Modeling as persuasion: the impact of group model building on attitudes and behavior. Syst Dyn Rev. 2011;27(1):1–21.Google Scholar
  69. Voinov A, Bousquet F. Modelling with stakeholders. Environ Model Softw. 2010;25(11):1268–81.View ArticleGoogle Scholar
  70. Ostrom E. Crossing the great divide: coproduction, synergy, and development. World Dev. 1996;24(6):1073–87.View ArticleGoogle Scholar
  71. Boyle D, Harris M. The challenge of co-production. London: New Economics Foundation; 2009.Google Scholar
  72. Reeve J, Cooper L, Harrington S, Rosbottom P, Watkins J. Developing, delivering and evaluating primary mental health care: the co-production of a new complex intervention. BMC Health Serv Res. 2016;16:470.View ArticlePubMedPubMed CentralGoogle Scholar
  73. Lehmann U, Gilson L. Action learning for health system governance: the reward and challenge of co-production. Health Policy Plan. 2015;30(8):957–63.View ArticlePubMedGoogle Scholar
  74. Gladman JRF, Conroy S, Ranhoff A, Gordon A. New horizons in the implementation and research of comprehensive geriatric assessment: knowing, doing and the ‘know-do’ gap. Age Ageing. 2016;45(2):194–200.View ArticlePubMedGoogle Scholar
  75. Wehrens R. Beyond two communities - from research utilization and knowledge translation to co-production? Public Health. 2014;128(6):545–51.View ArticlePubMedGoogle Scholar
  76. Willis CD, Best A, Riley B, Herbert CP, Millar J, Howland D. Systems thinking for transformational change in health. Evid Policy. 2014;10(1):113–26.View ArticleGoogle Scholar
  77. Happach R, Veldhuis GA, Vennix JAM, Rouwette EAJA. Group Model Validation. The 30th International Conference of the System Dynamics Society. St. Gallen, Switzerland; 2012. https://www.systemdynamics.org/conferences/2012/proceed/index.html. Accessed 16 Sept 2017.
  78. Voinov A, Kolagani N, McCall MK, Glynn PD, Kragt ME, Ostermann FO, Pierce SA, Ramu P. Modelling with stakeholders – Next generation. Environ Model Softw. 2016;77:196–220.View ArticleGoogle Scholar
  79. Mathers C, Vos T, Stevenson C. The Burden of Disease and Injury in Australia. Bruce: Australian Institute of Health and Welfare; 1999.Google Scholar
  80. Australian Institute of Health and Welfare. National Drug Strategy Household Survey detailed report 2013. Drug statistics series no. 28. Cat. no. PHE 183. Canberra: AIHW; 2014.Google Scholar

Copyright

© The Author(s). 2017