Skip to main content
Download PDF
Download ePub

Using natural experiments to improve public health evidence: a review of context and utility for obesity prevention

Health Research Policy and Systems volume 18, Article number: 48 (2020) Cite this article

Abstract

Background

Natural experiments are increasingly valued as a way to assess the health impact of health and non-health interventions when planned controlled experimental research designs may be infeasible or inappropriate to implement. This study sought to investigate the value of natural experiments by exploring how they have been used in practice. The study focused on obesity prevention research as one complex programme area for applying natural experiment studies.

Methods

A literature search sought obesity prevention research from January 1997 to December 2017 and identified 46 population health studies that self-described as a natural experiment.

Results

The majority of studies identified were published in the last 5 years, illustrating a more recent adoption of such opportunities. The majority of studies were evaluations of the impact of policies (n = 19), such as assessing changes to food labelling, food advertising or taxation on diet and obesity outcomes, or were built environment interventions (n = 17), such as the impact of built infrastructure on physical activity or access to healthy food. Research designs included quasi-experimental, pre-experimental and non-experimental methods. Few studies applied rigorous research designs to establish stronger causal inference, such as multiple pre/post measures, time series designs or comparison of change against an unexposed group. In general, researchers employed techniques to enhance the study utility but often were limited in the use of more rigorous study designs by ethical considerations and/or the particular context of the intervention.

Conclusion

Greater recognition of the utility and versatility of natural experiments in generating evidence for complex health issues like obesity prevention is needed. This review suggests that natural experiments may be underutilised as an approach for providing evidence of the effects of interventions, particularly for evaluating health outcomes of interventions when unexpected opportunities to gather evidence arise.

Peer Review reports

Background

Many public health issues are complex, requiring preventive health actions targeted at multiple upstream social and environmental determinates to improve population-level outcomes [1]. Despite this, the published literature is almost entirely focused on short-term individual-level research outcomes and lacking complex, multi-level, population-level intervention evidence [2]. Obesity is now recognised as a complex health issue, driven by multiple interrelated factors, including environmental, social and cultural determinants beyond individual-level determinants of behaviour [3,4,5]. Today, over 700 million adults and children are obese [6]. Effective population-wide prevention strategies implemented at-scale are needed to combat obesity; individually targeted strategies, such as health education and behavioural skills, have largely been found to be ineffective and unsustainable [2, 7]. Recommended prevention strategies focus on environmental interventions and policies to promote healthy eating and physical activity, such as taxation and restrictions on advertising of unhealthy food, interventions to increase healthy food availability, and environmental changes to the built environment [8]. However, the effectiveness of these interventions or policies remains limited by a lack of evaluation evidence.

Public Health evidence has been somewhat restricted to individualised prevention and treatment interventions from randomised controlled trial (RCT) studies. The key challenges for assessing complex policy and environmental interventions is that RCT studies are rarely appropriate, or even possible, in most situations [9]. In these complex interventions, the intervention is unlikely to be investigator initiated and the researcher is unlikely to have direct control over the study environment or wider policy influences [10, 11]. Other factors related to the complexity of populations and context make it unrealistic to apply controlled study environments such as the long time over which health behaviours change or outcomes are established [12, 13]. These issues have led to calls for natural experiment studies to improve the evidence base for public health interventions [11, 14].

The Medical Research Council (MRC) in the United Kingdom describes natural experiments as evaluating health or other outcomes where “exposure to the event of intention of interest has not been manipulated by the researcher” [15]. A number of other definitions exists and contributes to widespread confusion that a natural experiment is a type of study design (synonymous with or distinct from a quasi-experiment) rather than what is usually an unplanned ‘opportunity’ for research [15]. There is also some debate as to whether a natural experiment should refer only to studies which are ‘experimental’, i.e. where variation in the outcomes are analysed using methods that attempt to make causal inference. This position is held by the MRC and would include quasi-experimental studies but would exclude observational study designs as insufficient for determining causality [16]. Others contest that even weak study designs may be better than no evidence at all [17, 18]. Hence, in evaluating the contribution of natural experiment studies to the evidence base, it is important to consider both the strength of their designs and their potential value to the existing evidence base.

The purpose of this paper is two-fold: firstly, given the increasing advocacy for using natural experiments, we sought to investigate how natural experiments have been defined and used in practice. Specifically, we describe and assess the characteristics of natural experiments conducted in the area of obesity prevention to reveal the strengths, gaps, and weaknesses and help inform future research practice. Secondly, we explore the value of natural experiments in evaluating real-world interventions by considering the extent to which a planned experiment might be possible or whether this knowledge could only have been generated by a natural experiment.

Methods

Literature search strategy

A literature review was conducted on published peer-reviewed studies that self-described as natural experiments and focused on obesity prevention through improving nutrition or physical activity. The purpose of the review was to explore the use and utility of natural experiments [19], not to determine the outcome effects of interventions, which has been reported elsewhere [20, 21]. A systematic search in Scopus, Web of Science and CINAHL databases identified potential studies by combining two main topics (natural experiment AND population health) with the areas of interest: physical activity, nutrition or obesity (for full search criteria details, see Additional file S1). The search results were then limited by language (English only), article type (original research published/in-press and full-articles only), and dated from 1 January 1997 to 22 December 2017. Articles involving non-human research or not related to obesity prevention or improving nutrition or physical activity were excluded.

Figure 1 depicts the study selection stages. The initial search results were combined (n = 117 articles) and duplicate studies (n = 21 articles) were removed. Titles were scanned and 14 additional articles were removed as non-human studies, resulting in 82 articles. Two authors (MC and EG) read the abstracts to confirm that the articles met the eligibility criteria. A further 37 studies were removed as a commentary or opinion piece (n = 6), a protocol or methods article (n = 7), a review or meta-analysis (n = 8), or unrelated to obesity prevention (n = 12); an additional 4 four studies were removed as they did not self-identify as natural experiments (Additional file S2). The reference lists of the review articles were also searched for natural experiment studies contained within and 1 additional study was identified. The full text of all 46 articles was evaluated to confirm all inclusion criteria were met.

Fig. 1
figure 1

Study selection

Full size image

Critical review of studies

To investigate use, identified studies were described according to the nature of the intervention (type/setting, theme and scale) and of the evaluation methodology (study design, the exposure, characteristics of the exposed group and inclusion of a control group, and data collection measures and period). We then conducted a qualitative synthesis on the value of the studies to the field (Box 1). This includes consideration of the purpose of the study, and whether it could be answered from a planned, researcher-driven study, following the typologies developed by Dawson and Sim [22]. We discuss whether a planned controlled experiment (specifically an RCT or cluster RCT) would be realistic, considering practical issues such as time and population constraints, the potential for ethical conflict, and other limitations of RCTs [12, 13, 23].

MC and EG independently assessed each of the included studies and met to identify and resolve any coding and data extraction discrepancies. Where the authors disagreed, the other authors were consulted and a consensus reached.

Results

The literature search resulted in 46 studies directly related to the topic of diet, physical activity and obesity prevention, and are described in Table 1. Among the natural experiments conducted in this area over the last 20 years, none were published before 2007, 7 were published prior to 2012 (when the MRC guidance was published) and 4 more were published the same year. The largest annual publication volume occurred in 2017 with 14 published studies, at least double the numbers of any previous year (Additional file S3). The studies were predominantly from North America (22 from the United States of America and 6 from Canada). The remaining studies were from Europe (n = 7), Australia (n = 8) and South America (n = 3).

Table 1 Studies self-identifying as natural experiments which met inclusion criteria
Full size table

Intervention context

Setting and scale

The majority of the natural experiments identified were policy interventions [27, 28, 31, 32, 37, 39, 42, 43, 48, 51, 56, 58, 59, 65, 66, 71, 72], followed by environmental interventions [30, 34,35,36, 38, 40, 41, 47, 49, 53, 55, 60,61,62, 64, 68, 69] and community-based interventions [29, 44, 63, 67, 70]. Fewer studies were of economic interventions [33, 50, 54], individual behaviour interventions [52] and anthropological studies [45, 46] (Table 2). In terms of the scale of implementation, the majority of interventions were conducted at the neighbourhood scale. All but two [31, 42] of the 19 policy interventions were implemented at a national or state level.

Table 2 Summary of study characteristics
Full size table

Intervention theme

The natural experiments related to diet assessed exposure to the introduction of, or changes to, food labelling [27, 42, 48, 66], or food regulation and taxation [28, 31, 37], and access to, or affordability of, healthy food options such as fruit and vegetables [39, 41, 43, 57, 58, 61]. Natural experiments related to physical activity included exposure to workplace physical activity programmes [49], schools [32, 65], park settings [29, 67], community-wide programmes [30, 44, 63], and built environment interventions, including infrastructure for active travel and changes in street design [30, 40, 53, 62], residential/school development [34, 35, 47, 55, 59, 60, 68] and green spaces [36, 38, 69]. Other experiments were conducted to assess physical activity outcomes associated with disruption to transport services [64], daylight saving [72] and social habitat [70]. Obesity-related outcomes more broadly were evaluated in relation to exposure to clinical procedures [52, 56], migration [45, 46], education [51] or economic events [33, 50, 54, 71].

Evaluation characteristics

Study design

A variety of study designs were used, with several quasi-experiments, comparing longitudinal pre/post intervention changes in exposure of a cohort against a control or comparison group [28, 31, 34, 36, 40, 42, 47, 48, 52, 61, 63, 67, 69, 72]. The remaining cohort studies were observational (non-experimental), comparing pre/post exposure without a control group [35, 44, 57, 64, 68, 71]. Four studies conducted interrupted time-series on sales data [27, 37, 43, 66], two without a control comparator (observational only) [43, 66]. Observational studies included repeat cross-sectional surveys [30, 33, 38, 43, 49,50,51, 56, 62], some of which incorporated mixed methods (qualitative interviews or geospatial mapping) [38, 43, 48, 56], or used a cross-sectional single time point only [29, 32, 39, 41, 45, 55, 58,59,60, 65, 70]. One study pooled repeat cross-sectional data as a result of data availability and was therefore unable to assess time-related effects [32]; another [46] was a feasibility study anticipating a larger, longitudinal investigation. Two studies were qualitative only [53, 54].

Data source

Natural experiment studies used a variety of sources. In total, four studies collected both primary and secondary data [43, 48, 53, 64] and 27 studies only collected primary data. Fifteen studies used secondary routinely collected surveillance data from national or state censes and health surveys [31,32,33, 51, 56, 71, 72], hospital administrative data [50, 52], pre-existing cohort studies [39, 70] and food supply [28] or sales data [27, 37, 43, 66]. Three studies used simulated data to establish a hypothetical unexposed group [27, 28, 44].

Exposure

Exposure samples comprised residential populations in a defined neighbourhood or region, consumers of food outlets (e.g. supermarkets or restaurant chain) or physical activity space (e.g. park-setting). A clearly defined unexposed ‘control’ group was used by 15 studies [28, 31, 34,35,36, 40, 42, 46,47,48, 52, 61, 63, 67, 69, 72]. More often, exposure was assessed in the one population group or area without a defined unexposed group (n = 18) [27, 30, 33, 38, 43, 44, 47, 49,50,51,52, 56, 57, 62, 64, 68, 70, 71] or comparisons made between two groups, one of which was exposed, at a single time point (n = 10) [29, 32, 39, 41, 45, 55, 58,59,60, 65].

Not all studies used a dichotomous definition of exposure. Graded levels of exposure between groups, areas or individuals over time were used in a small number of studies [33, 43, 51, 71]. These studies had no similar intervention characteristics that might suggest a pattern or typology of where and when graded exposure may be necessary.

Evaluation measures

A variety of evaluation measures were used to evaluate obesity prevention interventions, including subjective and objective measures of physical activity (walking, active travel, metabolic equivalents), diet and body mass index, and health behaviour from surveys, physical and manual counters (i.e. accelerometers), as well as secondary data sources listed above. One study used geospatial information systems to identify exposed and unexposed groups [35].

Evaluation methods varied across similar interventions. For example, two studies [36, 38] evaluated the impact of new outdoor gym equipment on physical activity. Only one compared changes in the exposed group (park users/residents) against users of parks without gym equipment. Three studies used different methods to assess sales data to determine the effectiveness of food labelling policies in supermarkets. One of these compared data across three supermarket chains, with one chain acting as the control [55], one compared variation in exposure across a variety of supermarket types [66], and the third study simulated exposure because the labelling policy in question was voluntary and haphazardly implemented [27]. These differences suggest a variety of evaluation measures are applied to natural experiment opportunities.

Evaluation timeframe

Of the studies that included pre and post data collection, studies conducted over a short (less than 3 year) evaluation period, such as community or environmental interventions, predominantly used original sourced data from surveys or observational measures. Studies with a longer evaluation period relied on secondary data collection as could be expected given the time constraints on primary data collection.

Utility

The value of natural experiments for providing real-world population evaluation evidence was appraised in terms of whether the studies could have been conducted as planned researcher-driven experiments. We determined the purpose of the study and found that the majority evaluated the effectiveness of an intervention or impact of a policy (see Additional file 4 Table S2 for full list). In terms of utility, we first found that planned researcher-driven research was not always feasible to answer the research questions raised by the authors and, as such, were opportunities that could only ever be investigated as natural experiments. For example, Jones-Smith et al. [50] sought to establish whether Native American economic resources (from casino ownership) influenced the likelihood of childhood obesity; Zick et al. [72] aimed to assess whether daylight saving time is associated with increased time spent in moderate-to-vigorous physical activity. The purpose of these studies was to examine social or environmental determinants or inequalities in the population, which would not be amenable to/appropriate for researcher manipulation. In some instances, we recognised it would be possible for a researcher to have some involvement in the decisions regarding when and where an intervention occurs in order to obtain baseline data; for example, evaluation of the impact of a new policy taxation [27, 37]. However, if a policy was enacted quickly, baseline data collection would be restricted. Some of the studies involved relocation, such as those for new housing developments in Australia [34, 35, 47, 68], where researcher control over the planning, timing or conditions of the relocation, even if working with the authorities, would likely be unrealistic.

We note that studies aimed at assessing inequalities in the population or determining intervention effectiveness would be unethical as a planned experiment where there was risk of potential harm from intentionally restricting access to medical care [52], economic support [50], educational opportunity [51] or randomising individuals/groups to social benefits [33, 57, 71], freedom to migrate [45, 46] or health services [57, 65].

In other cases, researcher manipulation of study components may have been possible, but the researcher would have been constrained by practical considerations such as time, population sample size or location of exposure [61]. As obesity takes time to develop and tends to reverse, some evaluation questions necessitate a long lag time between exposure and outcome, which negate short-time planning. In one study, historical data was used to investigate the effect of academic schooling on obesity-related health behaviours in adults [51]. A planned study of this nature would be unrealistic, involving long follow-up from schooling to adult years to establish causality; it would also be unethical to control exposure to schooling. Bernatchez et al. [30] evaluates awareness of a new bicycle share programme rather than use; this may be because evaluation occurred too early to measure behaviour change as measured in another similar study [40]. In other cases, a planned experiment would be unpractical because the nature of the intervention necessitates a whole-of-population approach, such as the effect of tariffs applied to certain energy dense foods on unhealthy food consumption [28]. The point of time at which a researcher engages may preclude pre-intervention data collection. For example, in one study, researchers could only collect retrospective survey data on commute mode from parents whose children transferred to a new school [55]; other researchers serendipitously had a public transit strike occur in the midst of measuring spatial behaviour among undergraduate students, allowing a spontaneous pre/post examination on changes to student travel patterns to university [64]. In some situations, a natural experiment may be the only realistic option available despite the absence of a control group because the intervention is so unique that a suitable control group is not feasible. For example, in the study by Cranney et al. [38], the park setting in which equipment was installed may have had unique features specific to its local environment, precluding a suitable control. Similarly, in the study by Barradas et al. [29], some parks were already receiving the intervention programme and intervention settings were pre-determined by another body.

Discussion

This study has characterised the use and value of natural experiments, particularly focusing on the area of obesity prevention and its complex aetiology. Although the number of studies self-described as natural experiments has increased over the past two decades, the body of research, at least in obesity prevention, remains small.

Obesity prevention is a complex issue. Research evidence about interventions implemented in real-world conditions and the impact of policies represent a key gap in the knowledge. Thus, there is a need for greater generation of evidence about the impact and effectiveness of policy strategies, and natural experiments could be better utilised to provide this evidence. In assessing how natural experiment studies have been used, we found that the majority of the studies reviewed were designed to evaluate the impact of a policy or the effectiveness of an environmental intervention. The policy evaluation studies were almost all at the national or state level. Geographically, the studies in this review were located in English-speaking countries, which may reflect the high prevalence rates of obesity in these countries, and the search methods precluded studies in non-English speaking countries; however, it may also be due to the low use of evaluation approaches across other countries and within research groups within these countries. For this reason, WHO Europe recently held a workshop to support and facilitate public health practitioners from participating European countries in the use of natural experiment methods [14].

Strengths and weaknesses of natural experiments

The strengths of natural experiments are in their ability to evaluate the process and outcomes of implementation of policies and interventions within the real-world complex social and political conditions they naturally operate in. The response to the obesity epidemic has required a broad range of policy, environmental and individual behaviour change interventions – necessarily complex interventions able to function within a complex socio-political system [2, 23]. Evaluation research designs need to be flexible and able to measure the interaction between multiple factors [73].

Natural experiments offer opportunistic evidence where a researcher-driven study may be impossible for reasons of intervention timing or exposure. Nevertheless, the ability to make causal inferences from natural experiments depends on optimising the research study design [74]. We identified a variety of designs, including interrupted time-series, cross-sectional and longitudinal cohort designs. Few of these studies were experimental/quasi-experimental, including both pre and post measures for an exposed and a comparator group. However, a surprising number of the studies used a single data collection point to evaluate an intervention and thus could not attribute any observed changes to exposure to the intervention. However, some used mixed methods designs to strengthen study findings. Two studies presented only baseline data (pre-experimental) [41, 60] and may have further intentions for collecting follow-up data; these may be premeditated natural experiments. This evidence represents a weakness in study design for evaluating natural experiment studies rather than a weakness of natural experiments in general, and something the MRC has tried to address through detailed guidance on measurement and statistical methodologies [26].

Another strength of natural experiments is their flexibility. Evaluation periods ranged from a single time point to spanning decades and unsurprisingly the evaluation period determined whether primary or secondary data was used to assess exposure. A variety of outcome measures was used and these sometimes varied across similar interventions. While this flexibility is an advantage of the method, it increases the difficulty for comparisons in evidence reviews [21]. Most studies adapted the research design to fit the existing intervention context. Harnessing resources before policy interventions is a research planning challenge posed by natural experiments. To overcome this, some studies strengthened their findings by the use of multiple data sources. In such situations, both primary and secondary data contribute to the evidence, and largely depend on the level of control the researcher has on the intervention occurring, the group exposed or the timeframe in which the intervention or event occurs.

A further strength of natural experiments is their delivery of scale, allowing for exploration of a wider range of research questions to be investigated at the population level, to enable generalisation [15]. Natural experiments operate under circumstances where randomisation is not possible, there are ethical considerations, identifying suitable controls may problematic, and timing (both researcher timing relative to the intervention and the time length of the intervention) may make the collection of ideal data infeasible. These natural experiments fill a void not otherwise met by traditional designs, and may yield insights into exposure–outcome relationships, which are nonetheless informative for obesity prevention.

Natural experiments are often criticised for their inability to eliminate bias. Benton et al. [75] assessed the risk of bias across a number of natural experiments using an acceptability score based on confounding, participant selection, exposure and outcome measures, and missing data, concluding that evidence on the effects of these interventions was too problematic to be useful. Bias, particularly due to confounding, is a concern for natural experiments [18]. We observed that few studies in this review had a defined control and exposed group – given that these studies are unplanned evaluations, a clearly defined control group may be challenging to establish. Within the studies reviewed, a clearly defined unexposed group was more likely in environmental interventions where the researcher could establish a study design with original data. In contrast, establishing a clearly defined unexposed group becomes more difficult when evaluating policy interventions, as these are generally implemented acutely, and at the population level. Modelling simulations to create control groups was one approach employed [27, 44] to circumnavigate the lack of control group comparator. Others employed a step-wedge design [63] or time-series data to help overcome some limitations of not having a suitable control comparator condition. Simulations may be especially useful when counterfactuals are difficult to establish or create [26]. Leatherdale [18] provides some other suggestions for facilitating greater evaluation evidence of health policies, suggesting the need to improve the ability to evaluate government policy. For this to happen, capacity needs to be built around practitioners either to conduct natural experiments or to work closely with academics so that more robust quasi-experimental methods of evaluation can be employed [76].

Despite these weaknesses, we caution against devaluing natural experiments based on a simple hierarchy of evidence. Applying the same standards of study design quality to natural experiments ignores the contribution they can make to overall evidence generation [17], particularly in regard to the complexity of real-world interventions and policy evidence. In response to the complexity of the obesity epidemic, obtaining policy and intervention impact evidence is critical. Indeed, natural experiments may provide innovative research translation evidence that has been lacking on obesity policy interventions [77, 78]. Therefore, despite the limitations of natural experiments, they provide valuable information on public health efforts to prevent obesity as, otherwise, any effects might remain unknown.

The limitations of this review include our search criteria for identifying studies. It is possible that the confusion around natural experiments and quasi-experiments limits the number of collected studies to those we have discussed. For example, some protocol papers extracted from the initial search identified as a natural experiment, yet subsequent articles by the authors of these papers did not self-identify their intervention as a natural experiment. This suggests that those studies that want to identify as this type of research are not meeting an agreed definition. The synthesis of the utility of studies was limited to the information provided in each article; political, socio-cultural and practical obstacles that affect natural experiments and limit the potential for planned interventions is thus speculative.

Conclusion

This review examined natural experiments in an effort to improve the public health evidence for obesity prevention where a controlled experimental design would be inappropriate. Research using natural experiments has increased over the last few years; however, it remains overlooked in the context of the wider research evidence despite the importance of these interventions taking place in real-world settings. Our findings highlight the strength of natural experiments in improving our understanding of the effectiveness of complex population interventions and providing informed evidence of the impact of policies and novel approaches to understanding health determinants and inequalities. The studies reviewed also reveal that there is need to strengthen research designs to enhance their utility.

Availability of data and materials

Data generated or analysed during this study are included in this published article and its additional files. Please contact the corresponding author for any additional queries.

Abbreviations

MRC:

Medical Research Council

RCT:

Randomised controlled trial

References

  1. Crane M, Bauman A, Lloyd B, McGill B, Rissel C, Grunseit A. Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program. Health Prom J Austrl. 2019;30(3):422..

    Article  Google Scholar 

  2. Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4.

    Article  PubMed  Google Scholar 

  3. Butland BJS, Kopelman P, McPherson K, Thomas S, Mardell J, Parry V. Tackling Obesities: Future Choices. Project Report. 2nd edition: Government Office for Science. London: Department of Innovation, Universities and Skills; 2007.

  4. Roberto CA, Swinburn B, Hawkes C, Huang TTK, Costa SA, Ashe M, et al. Patchy progress on obesity prevention: emerging examples, entrenched barriers, and new thinking. Lancet. 2015;385(9985):2400–9.

    Article  PubMed  Google Scholar 

  5. Swinburn BA, Sacks G, Hall KD, McPherson K, Finegood DT, Moodie ML, et al. The global obesity pandemic: shaped by global drivers and local environments. Lancet. 2011;378(9793):804–14.

    Article  PubMed  Google Scholar 

  6. GBD 2015 Obesity Collaborators. Health Effects of Overweight and Obesity in 195 Countries over 25 Years. New Engl J Med. 2017;377(1):13–27.

    Article  Google Scholar 

  7. Huang TT-K, Glass TA. Transforming research strategies for understanding and preventing obesity. JAMA. 2008;300(15):1811–3.

    Article  CAS  PubMed  Google Scholar 

  8. Gortmaker SL, Swinburn BA, Levy D, Carter R, Mabry PL, Finegood DT, et al. Changing the future of obesity: Science, policy, and action. Lancet. 2011;378(9793):838–47.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Bauman AE, King L, Nutbeam D. Rethinking the evaluation and measurement of health in all policies. Health Promot Int. 2014;29:i143–i51.

    Article  PubMed  Google Scholar 

  10. McKinnon RA, Orleans CT, Kumanyika SK, Haire-Joshu D, Krebs-Smith SM, Finkelstein EA, et al. Considerations for an obesity policy research agenda. Am J Prev Med. 2009;36(4):351–7.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Sallis JF, Story M, Lou D. Study designs and analytic strategies for environmental and policy research on obesity, physical activity, and diet. Recommendations from a meeting of experts. Am J Prev Med. 2009;36(2 Suppl):S72–S7.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Sanson-Fisher RW, Bonevski B, Green LW, D’Este C. Limitations of the randomized controlled trial in evaluating population-based health interventions. Am J Prev Med. 2007;33(2):155–61.

    Article  PubMed  Google Scholar 

  13. Victora CG, Habicht J-P, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94(3):400–5.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Snell A, Reeves A, Rieger M, Galea G, Mauer-Stender K, Mikkelsen B, et al. WHO Regional Office for Europe's Natural Experiment Studies Project: an introduction to the series. Eur J Pub Health. 2018;28(2):1–3.

    Article  Google Scholar 

  15. Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new medical research council guidance. J Epidemiol Commun Health. 2012;66(12):1182–6.

    Article  Google Scholar 

  16. Petticrew M, Roberts H. Evidence, hierarchies, and typologies: Horses for courses. J Epidemiol Commun Health. 2003;57(7):527–9.

    Article  CAS  Google Scholar 

  17. Humphreys DK, Panter J, Sahlqvist S, Goodman A, Ogilvie D. Changing the environment to improve population health: A framework for considering exposure in natural experimental studies. J Epidemiol Commun Health. 2016;70(9):941–6.

    Article  Google Scholar 

  18. Leatherdale S. Natural experiment methodology for research: a review of how different methods can support real-world research. Int J Soc Res Methodol. 2019;2(22):19–35.

    Article  Google Scholar 

  19. Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Kärmeniemi M, Lankila T, Ikäheimo T, Koivumaa-Honkanen H, Korpelainen R. The built environment as a determinant of physical activity: a systematic review of longitudinal studies and natural experiments. Ann Behav Med. 2018;52(3):239–51.

    Article  PubMed  Google Scholar 

  21. Macmillan F, George ES, Feng X, Merom D, Bennie A, Cook A, et al. Do natural experiments of changes in neighborhood built environment impact physical activity and diet? A systematic review. Int J Environ Res Public Health. 2018;15(2):217.

    Article  PubMed Central  Google Scholar 

  22. Dawson A, Sim J. The nature and ethics of natural experiments. J Med Ethics. 2015;41(10):848–53.

    Article  PubMed  Google Scholar 

  23. Ogilvie D, Cummins S, Petticrew M, White M, Jones A, Wheeler K. Assessing the evaluability of complex public health interventions: five questions for researchers, funders, and policymakers. Milbank Q. 2011;89(2):206–25.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Ogilvie D, Adams J, Bauman A, Gregg E, Panter J, Siegel K, et al. Using natural experimental studies to guide public health action: turning the evidence-based medicine paradigm on its head. J Epidemiol Commun Health. 2020;74:203–8.

    Article  Google Scholar 

  25. Petticrew M, Cummins S, Ferrell C, Findlay A, Higgins C, Hoy C, et al. Natural experiments: an underused tool for public health? Public Health. 2005;119(9):751–7.

    Article  CAS  PubMed  Google Scholar 

  26. Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Ann Rev Public Health. 2017;38:39–56.

    Article  Google Scholar 

  27. Allais O, Etilé F, Lecocq S. Mandatory labels, taxes and market forces: an empirical evaluation of fat policies. J Health Econ. 2015;43:27–44.

    Article  PubMed  Google Scholar 

  28. Barlow P, McKee M, Basu S, Stuckler D. Impact of the North American free trade agreement on high-fructose corn syrup supply in Canada: a natural experiment using synthetic control methods. CMAJ. 2017;189(26):E881–E7.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Barradas SC, Finck Barboza C, Sarmiento OL. Differences between leisure-time physical activity, health-related quality of life and life satisfaction: Al Ritmo de las Comunidades, a natural experiment from Colombia. Global Health Promot. 2019;26:5–14.

    Article  Google Scholar 

  30. Bernatchez AC, Gauvin L, Fuller D, Dube AS, Drouin L. Knowing about a public bicycle share program in Montreal, Canada: Are diffusion of innovation and proximity enough for equitable awareness? J Transp Health. 2015;2(3):360–8.

    Article  Google Scholar 

  31. Brandt EJ, Myerson R, Perraillon MC, Polonsky TS. Hospital Admissions for Myocardial Infarction and Stroke Before and After the Trans-Fatty Acid Restrictions in New York. JAMA Cardiol. 2017;2(6):627–34.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Cawley J, Meyerhoefer C, Newhouse D. The impact of state physical education requirements on youth physical activity and overweight. Health Econ. 2007;16(12):1287–301.

    Article  PubMed  Google Scholar 

  33. Cawley J, Moran J, Simon K. The impact of income on the weight of elderly Americans. Health Econ. 2010;19(8):979–93.

    Article  PubMed  Google Scholar 

  34. Christian H, Knuiman M, Bull F, Timperio A, Foster S, Divitini M, et al. A new urban planning code's impact on walking: The Residential Environments Project. Am J Public Health. 2013;103(7):1219–28.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Christian H, Knuiman M, Divitini M, Foster S, Hooper P, Boruff B, et al. A longitudinal analysis of the influence of the neighborhood environment on recreational walking within the neighborhood: Results from RESIDE. Environ Health Perspect. 2017;125(7):077009.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Cohen DA, Marsh T, Williamson S, Golinelli D, McKenzie TL, Cohen DA, et al. Impact and cost-effectiveness of family Fitness Zones: a natural experiment in urban public parks. Health Place. 2012;18(1):39–45.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Cornelsen L, Mytton OT, Adams J, Gasparrini A, Iskander D, Knai C, et al. Change in non-alcoholic beverage sales following a 10-pence levy on sugar-sweetened beverages within a national chain of restaurants in the UK: interrupted time series analysis of a natural experiment. J Epidemiol Commun Health. 2017;71(11):1107–12.

    Google Scholar 

  38. Cranney L, Phongsavan P, Kariuki M, Stride V, Scott A, Hua M, et al. Impact of an outdoor gym on park users' physical activity: A natural experiment. Health Place. 2016;37:26–34.

    Article  PubMed  Google Scholar 

  39. Datar A, Nicosia N. The effect of state competitive food and beverage regulations on childhood overweight and obesity. J Adolesc Health. 2017;60(5):520–7.

    Article  PubMed  Google Scholar 

  40. Dill J, McNeil N, Broach J, Ma L. Bicycle boulevards and changes in physical activity and active transportation: findings from a natural experiment. Prev Med. 2014;69:S74–8.

    Article  PubMed  Google Scholar 

  41. Dubowitz T, Ncube C, Leuschner K, Tharp-Gilliam S. A natural experiment opportunity in two low-income urban food desert communities: research design, community engagement methods, and baseline results. Health Educ Behav. 2015;42:87–96.

    Article  Google Scholar 

  42. Elbel B, Gyamfi J, Kersh R. Child and adolescent fast-food choice and the influence of calorie labeling: a natural experiment. Int J Obes. 2011;35(4):493–500.

    Article  CAS  Google Scholar 

  43. Ferguson M, O'Dea K, Holden S, Miles E, Brimblecombe J. Food and beverage price discounts to improve health in remote Aboriginal communities: mixed method evaluation of a natural experiment. Aust N Z J Public Health. 2017;41(1):32–7.

    Article  PubMed  Google Scholar 

  44. Frew EJ, Bhatti M, Win K, Sitch A, Lyon A, Pallan M, et al. Cost-effectiveness of a community-based physical activity programme for adults (Be Active) in the UK: an economic analysis within a natural experiment. Br J Sports Med. 2014;48(3):207–12.

    Article  PubMed  Google Scholar 

  45. Fu H, VanLandingham M. Disentangling the effects of migration, selection and acculturation on weight and body fat distribution: results from a natural experiment involving vietnamese americans, returnees, and never-leavers. J Immigr Minority Health. 2012;14(5):786–96.

    Article  CAS  Google Scholar 

  46. Gee GC, de Castro AB, Wang MC, Crespi CM, Morey BN, Kaori F. Feasibility of conducting a longitudinal, transnational study of Filipino migrants to the United States: a dual-cohort design. J Health Care Poor Underserved. 2015;26(2):488–504.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Giles-Corti B, Bull F, Knuiman M, McCormack G, Van Niel K, Timperio A, et al. The influence of urban design on neighbourhood walking following residential relocation: longitudinal results from the RESIDE study. Soc Sci Med. 2013;77:20–30.

    Article  PubMed  Google Scholar 

  48. Hobin E, Bollinger B, Sacco J, Liebman E, Vanderlee L, Zuo F, et al. Consumers' response to an on-shelf nutrition labelling system in supermarkets: evidence to inform policy and practice. Milbank Q. 2017;95(3):494–534.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Jancey JM, McGann S, Creagh R, Blackford KD, Howat P, Tye M. Workplace building design and office-based workers' activity: a study of a natural experiment. Austr N Z J Public Health. 2016;40(1):78–82.

    Article  Google Scholar 

  50. Jones-Smith JC, Dow WH, Oddo VM. Association between Native American-owned casinos and the prevalence of large-for-gestational- age births. Int J Epidemiol. 2017;46(4):1202–10.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Jürges H, Reinhold S, Salm M. Does schooling affect health behavior? Evidence from the educational expansion in Western Germany. Econ Educ Rev. 2011;30(5):862–72.

    Article  Google Scholar 

  52. Kapinos KA, Yakusheva O, Weiss M. Cesarean deliveries and maternal weight retention. BMC Pregnancy Childbirth. 2017;17:343.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Kesten JM, Cohn S, Ogilvie D, et al. BMC Public Health. 2014;14:482.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Kodish SR, Gittelsohn J, Oddo VM, Jones-Smith JC. Impacts of casinos on key pathways to health: qualitative findings from American Indian gaming communities in California. BMC Public Health. 2016;16:621.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Lee C, Yoon J, Zhu X. From sedentary to active school commute: multi-level factors associated with travel mode shifts. Prev Med. 2017;95:S28–36.

    Article  PubMed  Google Scholar 

  56. Madsen KA. School-based body mass index screening and parent notification: a statewide natural experiment. Arch Pediatr Adolesc Med. 2011;165(11):987–92.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Odoms-Young AM, Kong A, Schiffer LA, Porter SJ, Blumstein L, Bess S, et al. Evaluating the initial impact of the revised Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) food packages on dietary intake and home food availability in African-American and Hispanic families. Public Health Nutr. 2014;17(1):83–93.

    Article  PubMed  Google Scholar 

  58. Ovrum A, Bere E, Øvrum A. Evaluating free school fruit: results from a natural experiment in Norway with representative data. Public Health Nutr. 2014;17(6):1224–31.

    Article  PubMed  Google Scholar 

  59. Pollack CE, Green HD Jr, Kennedy DP, Griffin BA, Kennedy-Hendricks A, Burkhauser S, et al. The Impact of public housing on social networks: a natural experiment. Am J Public Health. 2014;104(9):1642–9.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Ram B, Nightingale CM, Hudda MT, Kapetanakis VV, Ellaway A, Cooper AR, et al. Cohort profile: Examining Neighbourhood Activities in Built Living Environments in London: the ENABLE London – Olympic Park cohort. BMJ Open. 2016;6(10):e012643.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Sadler RC, Gillil JA, Arku G. A food retail-based intervention on food security and consumption. Int J Environ Res Public Health. 2013;10(8):3325–46.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Schultz CL, Wilhelm Stanis SA, Sayers SP, Thombs LA, Thomas IM. A longitudinal examination of improved access on park use and physical activity in a low-income and majority African American neighborhood park. Prev Med. 2017;95:S95–S100.

    Article  PubMed  Google Scholar 

  63. Simões EJ, Hallal PC, Siqueira FV, Schmaltz C, Menor D, Malta DC, et al. Effectiveness of a scaled up physical activity intervention in Brazil: a natural experiment. Prev Med. 2017;103:S66–72.

    Article  Google Scholar 

  64. Stanley K, Bell S, Kreuger LK, Bhowmik P, Shojaati N, Elliott A, et al. Opportunistic natural experiments using digital telemetry: a transit disruption case study. Int J Geogr Inf Sci. 2016;30(9):1853–72.

    Article  Google Scholar 

  65. Stone MR, Faulkner GE, Zeglen-Hunt L, Bonne JC. The Daily Physical Activity (DPA) policy in Ontario: is it working? an examination using accelerometry-measured physical activity data. Can J Public Health. 2012;103(3):170–4.

    Article  PubMed  PubMed Central  Google Scholar 

  66. Sutherland LA, Kaley LA, Fischer L. Guiding Stars: the effect of a nutrition navigation program on consumer purchases at the supermarket. Am J Clin Nutr. 2010;91(4):1090S–4S.

    Article  CAS  PubMed  Google Scholar 

  67. Torres A, Díaz MP, Hayat MJ, Lyn R, Pratt M, Salvo D, et al. Assessing the effect of physical activity classes in public spaces on leisure-time physical activity: “Al Ritmo de las Comunidades” A natural experiment in Bogota, Colombia. Prev Med. 2017;103:S51–S8.

    Article  Google Scholar 

  68. Tudor-Locke C, Giles-Corti B, Knuiman M, McCormack G. Tracking of pedometer-determined physical activity in adults who relocate: results from RESIDE. Int J Behav Nutr Phys Act. 2008;5:39.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Veitch J, Ball K, Crawford D, Abbott GR, Salmon J. Park improvements and park activity: a natural experiment. Am J Prev Med. 2012;42(6):616–9.

    Article  PubMed  Google Scholar 

  70. Wagner B, Jiang L, Hexuan L, Guang G. Gene–environment correlation: difficulties and a natural experiment-based strategy. Am J Public Health. 2013;103(S1):S167–73.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Watson B, Osberg L, Phipps S. Economic insecurity and the weight gain of Canadian adults: a natural experiment approach. Can Public Policy. 2016;42(2):115–31.

    Article  Google Scholar 

  72. Zick CD. Does daylight savings time encourage physical activity? J Phys Act Health. 2014;11(5):1057–60.

    Article  PubMed  Google Scholar 

  73. Moore GF, Evans RE, Hawkins J, Littlecott H, Melendez-Torres GJ, Bonell C, et al. From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation. Evaluation. 2019;25(1):23–45.

    Article  PubMed  Google Scholar 

  74. Medical Research Council (MRC). Using Natural Experiments to Evaluate Population Health Interventions: Guidance for Producers and Users of Evidence. London: Medical Research Council; 2011.

    Google Scholar 

  75. Benton J, Anderson J, Hunter R, French D. The effect of changing the built envoionment on physical activity: a quantitative review of the risk of bias in natural experiments. Int J Behav Nutr Phys Act. 2016;13(1):107.

    Article  PubMed  PubMed Central  Google Scholar 

  76. Green J, Roberts H, Petticrew M, Steinbach R, Goodman A, Jones A, et al. Integrating quasi-experimental and inductive designs in evaluation: a case study of the impact of free bus travel on public health. Evaluation. 2015;21(4):391–406.

    Article  Google Scholar 

  77. Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.

    Article  PubMed  PubMed Central  Google Scholar 

  78. Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99(9):1576–83.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

Not applicable.

Funding

The work was funded by the National Health and Medical Research Council of Australia (NHMRC) through its Partnership Centre grant scheme [grant number GNT9100001]. NSW Health, ACT Health, the Australian Government Department of Health, the Hospitals Contribution Fund of Australia and the HCF Research Foundation have contributed funds to support this work as part of the NHMRC Partnership Centre grant scheme. The contents of this paper are solely the responsibility of the individual authors and do not reflect the views of the NHMRC or funding partners.

Author information

Authors and Affiliations

  1. The Australian Prevention Partnership Centre, Sydney School of Public Health, Charles Perkins Centre, The University of Sydney, Camperdown, NSW, Australia

    Melanie Crane, Erika Bohn-Goldbaum, Anne Grunseit & Adrian Bauman

Authors
  1. Melanie Crane
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erika Bohn-Goldbaum
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anne Grunseit
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adrian Bauman
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AB and MC conceived the study. MC led the design, analysis and manuscript writing. EG conducted the literature search and contributed to analysis and manuscript writing. AG contributed to the design and manuscript writing and provided critical feedback. AB contributed to the design and provided critical feedback on the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Melanie Crane.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: Table S1.

Search strategy.

Additional file 2: Table S2.

Describes studies excluded through abstract review.

Additional file 3: Figure S1.

Showing the use of natural experiment studies surrounding nutrition and physical activity interventions on obesity.

Additional file 4: Table S3.

Describes the research aims of included studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Crane, M., Bohn-Goldbaum, E., Grunseit, A. et al. Using natural experiments to improve public health evidence: a review of context and utility for obesity prevention. Health Res Policy Sys 18, 48 (2020). https://doi.org/10.1186/s12961-020-00564-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-020-00564-2

Keywords

Health Research Policy and Systems

ISSN: 1478-4505

Contact us