Table 1 Features of models that provide utility to policy-makers in times of crisis

Instrumental utility (the model works and is fit for purpose)

Data inputs

•Data inputs used to drive the model are robust, verified (where possible) and adequate for the purpose of model outputs

•The model inputs include ranges of uncertainty

•The model is of appropriate scale to capture the problem faced by decision-makers

•The model strikes an adequate balance between sophistication and simplicity (i.e. abstraction) across model elements and representation of phenomena

•The model appreciates differences between individuals and groups (e.g. age vulnerability differences) that are important for understanding significant differences in outcomes at the population level

•The model has been specified correctly across variables of importance to model outcomes

Mechanics

•The mechanics of the model are formally defined and explainable at a technical level by its authors (i.e. it is not a “black box”)

•The model has had interdisciplinary input and/or does not come from a single person or source

•The model structure and/or its components have been formally tested and verified, as well as by other independent experts

•The model is open—i.e. its authors are happy to share its mechanics with the world and have those open to scrutiny

•Interactions of variables and features in the model are reflective of—or analogous to—real-world interactions

•Understanding the mechanics of the model enables insight into how outcomes and output data are generated

•Data and assumptions are adjusted iteratively to take account of new evidence

Data outputs

•The model estimates outputs over time frames of relevance to policy-makers

•The model provides clear policy direction and guidance based on agreed system performance metrics (e.g. health, economic/financial costs, public acceptance of measures) [32]

•The model appreciates trade-offs of implemented policies across associated domains

•The model outputs can be validated against historical data

•The model outputs include ranges of uncertainty

•The modellers can explain which variables have the greatest influence on model outcomes and overall system performance (i.e. can provide sensitivity analyses where appropriate)

•The authors of the model can explain their confidence in the outputs of the model, which is a separate consideration from the confidence intervals produced by the model outputs

•The model produces results that are broadly consistent with other like models or representations but also adds unique insight that other models might not

General

•The model framework, including assumptions and outcome measures, has been developed in collaboration with policy-makers (as far as possible)

•The model is fast enough to provide guidance in the time frame required by policy-makers

•The model is of adequate scope to capture and reflect the problem faced by policy-makers

•The model is being used for the purposes it was designed

•The authors can clearly articulate what the model is missing, what level of detail it cannot capture, what it can’t tell the user, and what it should not be used for, and the possibility of the model being affected by “off-model” events

Conceptual utility (the model is understood)

•The model is transparent—each aspect of it is explainable in plain language to a naïve audience—it is not a “black box” that neither the model authors nor outside experts can explain

•The model looks and sounds credible to a naïve audience and/or the audience (e.g. general public) who will be subject to its recommendations

•The model authors and contributors are suitably qualified and experienced

•The model authors are independent and/or there is no apparent conflict of interest

•The model authors’ institution is suitably qualified and experienced, and their institution is independent from political decision-making

•The model appeals to common sense but is sophisticated enough to extend the boundaries of people’s ability to conceptualize multiple future scenarios

•The model results are at times surprising but remain logical and explainable when surprising results emerge, demonstrating insight that might not otherwise have been gained through informal, implicit modelling

•Model results are presented plainly and implications are self-evident

•Model authors and/or their institution are presentable and can defend the validity of their work to the public

•The model has a public interface and/or can be manipulated by the public and/or other end-users to aid understanding

•The range of uncertainty in estimates is made clear to policy-makers

Political utility (the policy implications of the model are supported)

•The implications of the model can be woven into an acceptable, consistent political narrative by policy-makers

•The model has adequate instrumental utility as described above

•The model has adequate conceptual utility as described above

•Decision-makers (ministers, public servants and health authorities) have input into the modelling and its assumptions as early as possible in the build process

•The model outputs and recommendations are accepted as robust by policy-makers

•If policy-makers plan to use the model and/or its authors to prosecute public health initiatives, they are prepared to implement recommendations as per the model design and/or clearly articulate which aspects of the model they are taking recommendations from

•The model is/is proving to be accurate

•The model authors are/are proving to be reliable communicators and support its use in the way it is being used

•The relationship between model authors and policy-makers remains collaborative and productive

•The public continue to support the policy implications of the model’s findings