Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Utilization of a population health survey in policy and practice: a case study

  • Rachel Laws1,
  • Lesley King1,
  • Louise L Hardy1Email author,
  • Andrew Milat1,
  • Chris Rissel1,
  • Robyn Newson1,
  • Lucie Rychetnik2 and
  • Adrian E Bauman1
Contributed equally
Health Research Policy and Systems201311:4

DOI: 10.1186/1478-4505-11-4

Received: 16 August 2012

Accepted: 16 January 2013

Published: 30 January 2013

Abstract

Background

There is growing interest by funding bodies and researchers in assessing the impact of research on real world policy and practice. Population health monitoring surveys provide an important source of data on the prevalence and patterns of health problems, but few empirical studies have explored if and how such data is used to influence policy or practice decisions. Here we provide a case study analysis of how the findings from an Australian population monitoring survey series of children’s weight and weight-related behaviors (Schools Physical Activity and Nutrition Survey (SPANS)) have been used, and the key facilitators and barriers to their utilization.

Methods

Data collection included semi-structured interviews with the chief investigators (n = 3) and end-users (n = 9) of SPANS data to explore if, how and under what circumstances the survey findings had been used, bibliometric analysis and verification using documentary evidence. Data analysis involved thematic coding of interview data and triangulation with other data sources to produce case summaries of policy and practice impacts for each of the three survey years (1997, 2004, 2010). Case summaries were then reviewed and discussed by the authors to distil key themes on if, how and why the SPANS findings had been used to guide policy and practice.

Results

We found that the survey findings were used for agenda setting (raising awareness of issues), identifying areas and target groups for interventions, informing new policies, and supporting and justifying existing policies and programs across a range of sectors. Reported factors influencing use of the findings were: i) the perceived credibility of survey findings; ii) dissemination strategies used; and, iii) a range of contextual factors.

Conclusions

Using a novel approach, our case study provides important new insights into how and under what circumstances population health monitoring data can be used to influence real world policy and practice. The findings highlight the importance of population monitoring programs being conducted by independent credible agencies, researchers engaging end-users from the inception of survey programs and utilizing existing policy networks and structures, and using a range of strategies to disseminate the findings that go beyond traditional peer review publications.

Keywords

Government Policy Population health Research

Background

Public funds are expended through health research to lead to improvements in policy [1, 2], practice, resource allocation and, ultimately, the health of the community [3]. This can only occur if the evidence derived from the research is used to inform practice and policy decisions. There is growing interest by both funding bodies and researchers in measuring the impact of research. Over the past two decades many theoretical frameworks and approaches to measuring research impacts have been proposed [4], but there have been few empirical studies exploring how and why research is used [5, 6].

Population health monitoring surveys form one component in the public health research cycle, providing a key source of information about the prevalence and patterns of public health problems [7]. Such information may assist in guiding appropriate interventions, track changes over time, and support evaluation processes. That is, population monitoring surveys potentially provide valuable information to support policy and practice, although further research is generally required to test and disseminate effective interventions [8].

Despite the high investment in monitoring, little is known about if and how this type of population health data is used to inform policy and practice or the key factors influencing its use. We identified only one such study by de Goede and colleagues, exploring the use of epidemiological research in the development of local public health policy in the Netherlands [9, 10]. This study found that survey data was more often used in a conceptual way to improve understanding of the health problem or issue, rather than in a specific and direct way (instrumental use) or to justify a position or particular course of action (symbolic use) [10]. Research use was influenced by interaction between researchers and local health officials, the personal belief systems of the actors involved, and a range of contextual factors [9].

Further research is required to explore if, and in what ways, population health monitoring data is used in other contexts and settings. Understanding the factors influencing the use of this type of research is important in informing how population health monitoring systems can be planned and implemented to enhance the uptake of findings into the policy and program planning process.

Therefore, the aim of this paper is to provide a case study analysis of the utilization of the findings from an Australian series of population monitoring surveys of school-aged children’s weight and weight-related behaviors, the Schools’ Physical Activity and Nutrition Survey (SPANS). Three surveys have been conducted in 1997, 2004 and 2010 in the state of New South Wales (NSW), and provide a key source of data on NSW school children’s weight status, physical activity, sedentary behaviors and nutrition [1113]. Funded by government educational and health agencies since 1997, this monitoring program provides information on changes over time in key variables, and can potentially contribute to guiding and monitoring the progress of public health policy and programs that address children’s weight-related behaviors. However, the extent to which the surveys have actually been used for these and related purposes, has not yet been systematically investigated or documented.

The specific aims of this paper are to: i) describe if and how SPANS findings have been used at various time points to influence policy and practice; ii) explore the key facilitators and barriers to the use of the findings; and, iii) critically reflect on the methods applied to assess research use and make recommendations for future studies.

Methods

This study utilized a case study approach to explore if, and in what ways, a population health monitoring program was used to influence policy and practice, and to identify the key factors influencing its use. Case study methods are appropriate for answering ‘how’ and ‘why’ questions when the phenomenon of interest (SPANS surveys) is embedded within a real-life context (policy and practice environment) [14]. The study was approved by the University of Sydney Human Research Ethics Committee and all participants gave written informed consent to take part in the study. An overview of the study methods and key steps in the research process are given in Figure 1.
https://static-content.springer.com/image/art%3A10.1186%2F1478-4505-11-4/MediaObjects/12961_2012_Article_259_Fig1_HTML.jpg
Figure 1

Overview study methods and key steps in the research process.

Step 1 - Research scoping

The impact of SPANS was examined using the theoretical framework proposed by Banzi et al. based on a systematic review of available models [4]. The framework has five broad impact categories: i) Advancing knowledge and research related impacts (peer review articles, impact on research methods, better targeting for future research); ii) Capacity building (development of research capacity of staff, students, others); iii) Informing policies and product development (policy, guidelines, product, intervention development); iv) Health and health sector benefits (improvements in service delivery, effectiveness of services, equity of services, cost reductions, etc.); and, v) Societal and economic impacts (e.g., improvements in health status, social benefits, shift in knowledge, attitudes, behaviors, social capital, macroeconomic).

The Banzi framework was chosen as it is based on a systematic review of existing models and the set of categories provide an integrated ‘map’ of a broad range of potential areas of impact, which largely reflect the range of other commonly used models, for example, the payback framework [15]. We will briefly report on the impact of SPANS in relation to categories one and two (advancing knowledge and capacity building), and provide more detailed results relating to practice, policy and broader impacts (categories three to five), as well as reporting on the key mediators of these impacts.

Step 2 - Data collection

Chief investigator and end-user interviews

Semi-structured interviews were conducted with the Chief Investigators (CIs) for each of the three surveys (1997, 2004, 2010) and relevant end-users (EUs) including policymakers and practitioners who were in positions to make decisions regarding programs and policies related to SPANS findings on school-aged children’s nutrition, physical activity, sedentary behavior and obesity prevention in NSW. Both the CIs and EUs were invited to participate in the interviews by email, with non-responders sent a reminder email after one week and then followed-up by telephone. Interviews were conducted by an experienced research officer (RN) who had a good working knowledge of SPANS and related policy context, but was independent of the SPANS investigators. All interviews were tape recorded with participants’ permission.

The CI interviews explored their perspectives on the overall impacts of the survey, asked about specific impacts in accordance with the categories outlined in the Banzi framework, and any factors contributing to such impacts or lack thereof (List 1). The CIs were also asked to nominate up to three end-users (EUs), defined as ‘individuals who could provide a perspective on how the monitoring data had been used in policy, practice, organizational development, further research or in applications such as guidelines or teaching materials’. Attempts were made to identify EUs from a range of sectors including health, education and sport and recreation. These EUs were then approached and invited to participate in an interview exploring how the SPANS program findings had been used from their perspective.

List 1. Semi-structured interview topic guide: investigators and end-users

· Recall of research aims, key findings and implications

· Dissemination process (how, factors influencing the dissemination process)

· Interface with end-users – how research team worked with potential end-users (investigators only)

· Interface with researchers – how were end-users involved in the research project, how did they hear about the findings (end-users only)

· Overall impact – how have the findings been used

· Specific impacts – capacity building, partnerships, policy and product development, health and other sector impacts, societal and economic impacts

· Circumstances surrounding the use of the findings, or limited impact of the findings

· Evidence of impacts – documentary sources

· Nomination of end-users (investigators only)

Bibliometric and document analysis

A bibliometric analysis was undertaken in Scopus database to examine the total and mean number of citations (excluding self citations) for all peer review publications arising from SPANS for each survey year. Survey reports available in the public domain were also examined to document the key findings and recommendations arising from SPANS. CIs and EUs were also asked to provide a copy of any documentary sources which provided evidence of how SPANS findings had been used, such as policy documents, briefs, reports and curriculum materials. Additional searches of the grey literature were undertaken to identify documentary evidence of impacts identified in the interviews.

Step 3 - Data analysis

All interviews were transcribed verbatim and coded by one author (RL) using Nvivo qualitative software program (NVivo (version 9) Burlington, MA, USA). The Banzi categories were adapted and expanded for the coding framework to encompass other categories of impact arising during the interviews and to code for important contextual information. Further, additional codes were developed to capture content related to key factors influencing research use.

Detailed case reports for each survey (1997, 2004 and 2010) were compiled. These consisted of a summary of: i) key SPANS survey findings and recommendations; ii) the perspectives of CIs and EU on how the survey findings had been used and key factors influencing their use; iii) bibliometric analysis; iv) documentary evidence of impacts; and, v) notes and observations made during CI and EU interviews. The case summaries were prepared by two authors (RL and RN), who were not involved in either the implementation or administration of the SPANS surveys, in order to ensure that the ‘insider’ experiences of three of the authors who were involved in one or more surveys (LLH,LK, AB) were not unduly influential.

The three case summaries were then independently reviewed by each of the authors. There was high consistency in findings across surveys, allowing for the distillation of key themes across all three surveys, covering how and why SPANS findings had been used in policy and practice. Attention was also paid to examining any differences between surveys, the perspectives of CIs and EU, and key contextual factors.

Results

Participant characteristics

All three of the survey CIs approached agreed to participate in the interview. A total of 14 EUs were nominated and approached, and nine agreed to participate. Of the non-participant EUs, two did not respond to the invitation, two were not available during the data collection period and one had moved on from the policy area and therefore did not feel able to comment on the survey impacts. The EUs worked in a variety of sectors including education, health, and sport and recreation, and in most cases had direct responsibility for issues related to child obesity prevention (Table 1). A total of 16 interviews were conducted, with CI interviews lasting longer (mean; 64 minutes; range 53 to 70 minutes) than EU interviews (mean; 50 minutes; range 17 to 87 minutes). Not all CIs and EUs were in a position to comment on the impact of each survey. Table 2 shows CI and EU participants by sector, role and survey year.
Table 1

Participant characteristics by sector and survey

    Participant

  Participant sector and role

Survey

  

1997

2004

2010

CI 1

Academic researcher

 

CI 2

Academic researcher

 

 

CI 3

Academic researcher

  

EU 1

Health1,2,3,4

 

EU 2

Community sport and recreation2,3,4

 

EU 3

Education3,4

 

EU 4

Education4

  

EU 5

Education4

  

EU 6

Health1,2,3,4

 

 

EU 7

Health4

 

 

EU 8

Health4

  

EU 9

Health1,2,3,4

  

Total number of interviews

12

2

7

7

 

CI interviews: 3

(1 CI

(2 CIs

(1 CI

 

EU interviews: 9

1 EU)

5 EUs)

6 EUs)

1Worked in policy branch responsible for funding SPANS.

2Member of at least 1 SPANS Advisory Committee.

3Member of Government Child Obesity Prevention Implementation Committee.

4SPANS information relevant to work area.

Table 2

Key findings, implications and dissemination process for the SPANS monitoring program

Key Findings

Key Findings

Dissemination Process

Funding Source

1997 School Fitness and Physical Activity Survey

Poor fundamental movement skills (FMS) amongst school students

Need to address FMS and physical activity in schools

Main and summary report available on website

Department of Education and Training

Low levels of physical activity, particularly amongst adolescent girls

Survey should be repeated periodically with comparable measures

Summary report sent to all NSW schools

 

Few primary schools have teachers with training in sport and physical education

 

Presentations to a range of end-user groups

 
  

Peer review publications (n = 11, total citations1 = 589)

 

2004 School Physical Activity and Nutrition Survey

Almost 25% of students 5–16 years were overweight or obese, rising from 1997

Focus on FMS working, but needs to be continued

Main, short and summary report available on website

NSW Department of Health

Widening of the gap in prevalence of overweight and obesity amongst low socio-economic status and culturally and linguistically diverse groups since 1997

Need to limit small screen use to 2 hours per day

40 Regional stakeholder workshops across NSW

 

FMS and physical activity levels had improved significantly since 1997 survey but high screen time use

Reinforced the need to implement government policy that schools should offer 2 hours of planned physical activity per week

Media releases and interviews

 

Chronic disease risk factors common amongst overweigh/obese adolescents

Efforts to increase physical activity promotion in schools and community should be continued and increased

Peer review publications (n = 18, total citations1 = 131)

 

High intake of energy dense foods and drinks and low levels of vegetable consumption

Limit consumption and promotion of energy dense nutrient poor food

  
 

Implement strategies to target ‘high-risk’ groups

  

2010 School Physical Activity and Nutrition Survey

Prevalence of overweight and obesity stabilized since 2004

Continue to implement school canteen program, focus on FMS and provide minimum of 2 hours of planned physical activity per week in schools

Main, short and summary report available on website

NSW Department of Health (now Ministry of Health)

Socioeconomic and cultural disparities in prevalence of overweight/obesity remain.

Implement programs to support participation in community sport

Media releases and interviews

 

High consumption of energy dense nutrient poor foods and drinks and low levels of vegetable consumption

Advocate for national regulations to limit the marketing of unhealthy foods to children and policies to promote active transport

Peer review publications (n = 4, total citations1 = 1)

 

Decline in proportion of students meeting physical activity guidelines

Widespread dissemination of consistent messages to parents/families regarding healthy lifestyle behaviors

Presentations to stakeholder groups (with more to follow)

 

FMS showed improvement in some skills and decline in others

Importance of targeting family behaviors and early childhood sector to reduce obesity in preschoolers

  

High screen time use

   

Overweight and obesity increasing among children entering first year of school (~ 5 years old)

   

1Excluding self citations.

Research impacts

Dissemination and advancing knowledge

In general, the survey findings were considered to advance knowledge through the findings being widely reported in journal articles, and extensively disseminated through key reports, presentations to stakeholder groups, media releases and conference presentations. In total there were 32 peer review publications (1997, n = 11; 2004, n = 18; and 2010 n = 3, at the time of this analysis - July 2012) which together had been cited 720 times (Scopus, excluding self citations) with a mean of 22 citations per article (SD: 34, range: 0 to 140, H index: 12: Scopus). The key findings and recommendations from each survey, along with the dissemination process that was employed and survey funding source are presented in Table 2. While dissemination processes varied for each survey, in each case they included presentations to health and education professionals (mostly health promotion staff and teachers), as well as senior NSW policymakers in health, education, sports and community sectors.

Almost all EUs reported that they had direct involvement in the administration or funding of the survey from its inception as well as the dissemination process, including the drafting of key recommendations. Some EUs also took an active role in disseminating the findings within their sector, as discussed by this EU:

“…after each of the SPANS surveys that came out… we had these workshops… so we were provided the data and the information to our industry, but tried to do it through a meaningful way so they can apply it when they go into schools themselves.” (EU2, 2004 and 2010 surveys).

CIs and EUs discussed that the dissemination of SPANS findings was important in raising overall awareness of the issue of child obesity amongst the general community and key stakeholders such the education, health, and sport and recreation sectors, as well as highlighting areas for intervention and future research.

Capacity building impacts

The surveys had considerable impact on building research capacity with two PhDs, two masters/honors research projects and two postdoctoral positions arising from SPANS. For EU’s the survey provided professional development opportunities for the 55 school teachers who were involved in data collection, including up-skilling in the measurement of fundamental movement skills, cardiorespiratory fitness and anthropometry. Further, the survey methodology was used to inform the methods for other research studies and survey programs at both the state and national level (Additional file 1).

Policy and practice impacts

Table 3 presents the key content themes regarding ways in which the survey findings were used to inform policy and practice, as well as illustrative quotes. At a policy level, this ranged from broad agenda setting and policy debates on the importance of childhood obesity, to underpinning specific new policies such as banning soft drink sales in schools. Each survey had specific policy impacts (e.g., 1997 findings informed the development of a policy to focus on fundamental movement skills (FMS) in schools, and 2004 findings underpinned the ban of soft drink sales in schools). The EUs and CIs identified a range of policy documents which cited or applied SPANS findings (Additional file 1).
Table 3

How the SPANS program findings were used to inform policy and practice

Key Impacts

Illustrative Quotes

Policy Impacts

· Agenda and priority setting e.g., attracting funding to the issue of child obesity prevention, identifying priority groups and settings for intervention

I guess it sort of drives, to some degree, the priorities in the [government obesity] plan.. are the actions in the plan the right ones, are they working?…So it gives us a bit ofan idea of whether or not theres gaps that havent been addressed, in terms of the broader cross government strategy.. are there areas where we could pull back on, invest more, focus in another area.” (EU9, 2010 survey)

· Informed policy debates e.g., data used in briefings with health minister to inform parliamentary debates

 

· Informed policy planning e.g., identifying areas for investment, de-investment and stakeholder involvement in government obesity plan

“….theres high levels of screen time, so I guess we can kind of use it as a basis or rationale for our policy input, and also for how we approach different issues with different government agencies.” (EU9, 2010 survey)

· Directly underpinned new policy e.g., banning soft drink sales in schools

 
 

“I know that the physical activity components [1997 survey] were also used to advocate for fundamental movement skill development with the Department of Education.” (EU1, 1997 & 2004 surveys)

· Indirectly used to advocate for new policy e.g., food marketing to children

 
 

“SPANS would certainly have been one of the things that contributed to the development of that policy [nutrition in schools]. (EU5, 2010 survey)

· Used to support existing policy e.g., after school physical activity programs for children

 
 

the stats that are so terribly important, particularly when youre trying to sell something” (EU4, 2010 survey)

· Policy evaluation e.g., performance monitoring tool for government obesity plan

“And the Education Department was particularly interested to see what had happened in terms of fundamental movement skills, because theyd made a significant investment in fundamental movement skill education in primary and high schools, so they were really interested in what had happened there. And another part of it was an evaluation of the Healthy Canteen strategy that had been put in place, maybe a year or two earlier. And there was also an aspect of the study that was looking at the school environment to see again if there had been any changes following the previous study. (CI3, 2004 survey)

 

SPANS just shines a light on whats not working.” (EU2, 2004 & 2010 surveys)

Practice Impacts

School Sector

“…the initial research that was done [1997 survey]…we got very strong agreement around those fundamental movement skills, and we had some clear support materials developed for schools and the Get Skilled Get Active DVD, which is a resource that we still promote in schools.” (EU3, 2004 & 2010 surveys)

· Informed curriculum development e.g., standardized approach to teaching FMS

 

· Lead to new curriculum resources e.g., standardized teaching materials for FMS

 

· Informed Professional development for teachers e.g., measuring FMS, raising awareness of child obesity

 

· Informed and supported existing health promotion programs e.g., healthy canteens

 

Sport and Recreation Sector

“…the [sport] development officer network, they loved it [survey data] and they got it. Particularly the fundamental movement skills, because theyre out there teaching them, through sport, every daytheyve now rescheduled their format of how they go in to teach girls Rugbythey do the kicking last because they know their stronger skill is in the running and the catching. So, its really practical…” (EU2, 2004 & 2010 surveys)

· Informed sports coaching programs e.g., how FMS are taught in some sports

 

· Supported rationale for new programs or pilots e.g., healthy sports canteens

 

Cross Government / Community Sector

“…when it [2004 survey] showed that 20% of kids when they started grade 1 were already fat, it also gave emphasis to do stuff in childcarethat was really compellingwe would have never known that if that survey wasnt done.” (EU6, 2004 survey)

· Informed development and refinement of educational resources e.g., website on healthy living for parents and general community [35]

 
 

SPANS, as I said, is a moment in time that gives you the heads-up on what areas you should be focusing on in you next plan; thats what it should be doing.” (EU2, 2004 & 2010 surveys)

Health Sector

I think SPANS contributed to having treatment programs is what they areI call them treatment programs for people with alreadywithwith already with the problem.” (EU6, 2004)

· Informed program planning e.g., choice of target groups and settings for intervention and availability of treatment programs

 

At the practice level, SPANS findings were used to inform program planning across a range of sectors. In the education sector, the findings led to a standardized approach to teaching FMS in schools and underpinned the rationale for new initiatives in the community sports sector (such as healthy sports canteens). The findings provided support for some existing health promotion programs, and informed the development and refinement of educational resources both within schools and for the broader community. The impact on practices across sectors largely flowed on from policy impacts. For example, the policy focus on FMS in schools led to the development of new curriculum materials, resources and professional development for teachers (Additional file 1).

Broader health, economic and societal impacts

Most interview respondents found it difficult to comment on the broader health, economic or societal impacts of SPANS, because of the long term nature and the multiple factors contributing to such impacts. Nevertheless, one EU suggested that SPANS may have contributed to improvement in FMS over time because of the change in curriculum focus and resources provided to schools following the 1997survey. Others discussed that SPANS may have contributed indirectly to the leveling-off of childhood obesity between 2004 and 2010, as new policies and programs arose, in part, because of the survey findings.

“I think the fact that obesity has plateaued should be taken that all of the investments that have been happening…well at least since SPANS 2004…are obviously having some degree of impact” (CI 3).

Documentary checks were undertaken and documentary evidence identified relating to the majority of documents and initiatives referred to by CIs and EUs. Key documents are listed in the Additional file 1.

Factors influencing the use of SPANS findings to inform policy and practice

An analysis of key perceived facilitators and impediments to the use of the SPANS findings in influencing policy and practice revealed three main inter-related factors: i) the perceived credibility and trustworthiness of the survey findings; ii) the dissemination processes used; and, iii) a range of contextual factors (illustrative quotes about these factors are presented in Table 4). In 1997, the survey was funded by government as part of a broad child health and fitness agenda, and from 2004, as part of a government policy response to monitor child obesity. As such the survey content and findings were designed to be closely aligned to the interest of policy makers and key EU groups such as health, education, and community sport and recreation sectors. The survey was perceived by EUs interviewed to be a trustworthy source of data as it was conducted by an independent and credible research group.
Table 4

Factors influencing the use of the SPANS findings to inform policy and practice

Facilitators

Barriers

Research Quality and Content

· Survey program perceived to be of high methodological quality: longitudinal and conducted by an independent and credible research team

· Survey findings did not provide all the answers: needed to be considered alongside evidence about effective interventions

 

“…of course just monitoring is not a solution” (EU6, 2004)

· Survey program perceived to be aligned to the priorities of policy makers and practitioners with adaptations made over time to meet needs

Lack of specificity of the data: Unable to provide data at regional/sector level

 

I think SPANS cannot stand alone, it has to have an evidence summary of interventions behind it to inform good policy” (EU2, SPANS 2004 & 2010)

the Ministry/Department has worked really hardto make sure thatthe questions that the surveys asking are, I suppose, the right ones for the priority areas as wellSo I think over time the surveys have kind of adapted to have different kinds of questions in themits been purposefully closely aligned with policy priorities and government priorities and I think that helps” (EU9, 2010 survey)

 

Dissemination Process

· Use of active dissemination strategies e.g., discussion of findings at workshops between researchers and end-users.

Lack of reports tailored to specific end-user groups/sectors highlighting key implications of the findings

“… there was a major dissemination strategy amongst a set of orgs” (EU6, 2004)

 
 

“…there probably should be a report for practitionersWhat does this mean to you?”…

· A range ofknowledge transferproducts produced e.g., short reports highlighting key findings and recommendations.

What does it mean for a teacher whos 55, has never played sport before..I think if we really want to get good at this, take SPANS and write it for different audiencesSo SPANS for academia, SPANS for policy, SPANS for the general public” (EU2, 2004 & 2010 surveys)

· End-users acted asknowledge brokers’ facilitating dissemination of SPANS findings within their sector

 

“….weve done canteen newsletters and things like that and weve reported SPANS results and we reference it there” (EU5, 2010)

 

· Active engagement of the media resulting in high levels of media coverage, public debate and discussion

 

I also sit on the [policy] group for obesity, so the report was discussed on a number of occasions at that meeting as well. You see the media reports theyve generatedwe receivedraft copies of the report as well, in terms of the discussions that we had with Health” (EU5, 2010 survey)

 

Contextual Issues

· Supportive policy context for addressing child obesity with the release of the SPANS findings fitting well with some policy cycles (government obesity plan and planning of national curriculum)

Political instability and poor timing e.g., frequent changes in ministerial positions and poor fit with some policy cycles

 

Limited sector capacity and resources e.g., lack of funds to implement the findings in some sectors at certain time points

· Continuity and Partnerships between researchers and end-users from the inception of SPANS program facilitated dissemination, ownership and use of the findings

 
 

“….then theres some infrastructure issues we need to look at; there needs to be some resourcing” (EU2, 2004 & 2010 surveys)

· Mechanisms and structures in place to implement recommendations e.g., Policy relevant forums involving key end-users

“…probably prior to 2002there was a relatively limited amount of funding in this aspect of Health Promotion - …So in terms of influencing programs it would have been difficultcause weveno money to include programs” (EU1, 1997 & 2004 surveys)

· Good fit with organizational culture and ways of working e.g., value placed on having an ‘evidence base’ for practice and policy decisions amongst end-user groups

 

I think the Department of Education knew of the findings or the recommendations that were going to be made because they also had ownership of those..they had to agree on those for the draft report. So I think by that stage theyd already said: We want this recommendation here and this is how were going to respond to it.” (CI, 1997 & 2004 surveys)

 

Partnerships between EU groups and the CIs operated from the inception of the SPANS program, and more regular and specific policy forums were established as part of the NSW ‘whole of government’ approach to addressing child obesity in 2003 [16]. These policy forums provided the continuity and mechanisms for survey findings to be disseminated and considered, and for EU groups to become engaged in formulating the key policy recommendations and the wider dissemination processes. There were, however, some barriers identified to the use of the findings, including some related to the nature of the survey itself, in that it mainly identified the problems and target groups, but did not provide specific guidance on solutions, and the reports were not written for specific EU groups. A number of contextual factors were also identified as impediments to applying the findings, including ongoing organizational changes in government agencies, and limited capacity and resources to implement recommendations among some sectors at various points in time.

Discussion

This is one of the few studies to document the temporal impact of a government population health monitoring program. We found that SPANS was successful in advancing knowledge and understanding of the issue of child obesity, weight-related behaviors, building capacity of the academic researchers and some EUs, and influencing policy and practice decisions across a range of sectors. The actual policy and practice impacts were in line with key recommendations arising from the survey reports. The positive impacts of the survey findings were attributed to the perceived credibility of the survey by EU groups, the processes used to disseminate the findings and a range of contextual factors. Contextually, the survey had a supportive government policy, strong partnerships between CIs and EUs, and established mechanisms for dissemination of the findings across a range of government sectors.

This case study demonstrates the positive impact a population health monitoring program can have on a range of policy and practice decisions. In particular, the findings of this study have demonstrated that population health monitoring data can be used as an advocacy tool (to attract attention and funding to an issue), priority setting (identifying areas and target groups for intervention), support and justify existing programs/approaches, or identify the need for alternatives. While others have suggested that epidemiological research can be used in a conceptual way to improve understanding of a health problem [9, 10], this case study found that the survey findings were also used instrumentally to directly underpin new policies (such as banning soft drink sales in schools), and symbolically to justify existing policies and programs (such as introducing healthy school canteens), in addition to improving understanding of the issues associated with child obesity (conceptual use).

The instrumental and symbolic use of the survey findings in influencing policy decisions may reflect the fact that the EUs interviewed in this study were directly involved in the development and implementation of policies and programs relevant to the survey findings. In contrast, the study by de Goede and colleagues involved a broader range of EUs, most of whom were not directly involved in execution of health surveys. The use of the survey findings may also reflect the political imperative to take action to reduce child obesity, particularly in the early stage of child obesity prevention responses when there was an absence of other evidence on effective intervention approaches [17, 18]. It is important to note, however, that epidemiological evidence in itself does not provide solutions to health problems, and needs to be considered alongside the broader body of evidence about effective intervention and policy approaches, as highlighted by many of the EUs in this study.

It is clear from this study that population monitoring surveys are one of many factors influencing public health policy and practice. In line with previous research [5, 6, 1924], we found that a range of contextual factors were critical in facilitating the use of the survey findings. In particular, the 2002 NSW Child Obesity Summit [18] provided a supportive policy context which facilitated the formation of partnerships between survey researchers and EUs, and provided mechanisms and structures (such as the NSW Government Action Plan for Child Obesity Prevention 2003–2007, [16] and a Senior Officers Group with responsibility for overseeing implementation of this plan) which facilitated application of survey findings. In addition, ongoing media coverage of child obesity throughout this period reinforced the political and social relevance of these surveys [25].

The findings highlight the importance of researchers tapping into existing policy networks, structures and processes, utilizing policy windows and building partnerships and trust with policy makers and other EUs over time. This is in accordance with the findings of Innvær and colleagues who found that personal contact and timely relevance were the most commonly reported facilitators of research use in a systematic review of 24 studies of the use of evidence by health policy makers [22]. Additionally, our findings point to the importance of the dissemination process in influencing the use of findings from population health monitoring surveys, in particular, the use of active dissemination strategies, such as workshops and presentations, to encourage contact and interaction between academic researchers and EUs. The use of a range of other dissemination mechanisms such as short summary reports in lay language and media releases, as well as traditional academic publications, were all shown to be important in facilitating use of the findings by EUs. There is increasing emphasis from funding agencies on making research evidence readily available, however recent studies of public health research suggest that most dissemination activity beyond publishing academic papers appears to be undertaken in an ad hoc, unfunded fashion and that access to dissemination advice and support for researchers from funding agencies and academic institutions is lacking [26, 27]. This suggests that more emphasis should be placed on funding and supporting a range of dissemination activities on behalf of both funding bodies and academic institutions themselves.

Traditionally, tools for measuring research impacts and assessing ‘research excellence’ are based on the number and significance of academic publications (such as bibliometrics and journal impact factors). However, there is growing interest in assessing actual impact on real world policy and practice. While a number of theoretical frameworks for assessing impact have been proposed [5, 2832], the findings of our study suggest that the impact categories proposed in such frameworks, including that of Banzi et al.[4], can capture the practical ways in which survey data is utilized. However, the findings also indicate that it can be difficult to attribute impacts to a single piece of research, particularly the longer term societal, health and economic impacts, which almost always arise for a complex array of contributing and contextual factors. This highlights the need for alternative ways of conceptualizing research ‘impacts’. The use of ‘contribution mapping’ as proposed by Kok and colleagues [33] may provide an alternative way forward. This approach aims to assess ‘contributions’ of research as opposed to ‘impacts’ and focuses on understanding how research and knowledge utilization processes evolve amongst the key actors involved. Further clarity is also required by researchers and funding agencies about the intended contribution that population health monitoring surveys are expected to make to policy and practice, in order to assess the degree to which this was achieved.

This study has a number of strengths and limitations. In terms of strengths, we assessed impacts using multiple methods, including bibliometric analysis, interviews with survey researchers and EUs, and documentary checks. This allowed for triangulation of data in the form of case summaries. The documentary checks lend confidence that the perspectives of the EUs are credible. While the SPANS reports have been disseminated widely across the NSW education sector, the full range of users, and their responses and application of findings, are unknown, although likely to be highly variable. It must be acknowledged that the EU interviewees were purposefully sampled on the basis of their role as the primary target audience and having direct responsibility for child obesity prevention policy initiatives. There may also have been some degree of social desirability response bias, whereby EUs felt obliged to report positive or over-inflated impacts. We attempted to reduce social desirability response bias by having researchers not previously involved in SPANS conducting the interviews and undertaking the analysis. The recall of impacts was somewhat problematic for the 1997 survey, as this was conducted over 15 years ago and only one EU (out of three) was available to be interviewed. Hence we are unable to ascertain whether the impacts of the 1997 survey may be underestimated or overestimated in this case study. Overall, there has been a high degree of continuity in the CIs involved in SPANS and EUs in NSW child obesity prevention over the study period, which may also have increased difficulties in attributing influence to any single one of the three surveys, and lent a positive bias. We recognize that there is no single or straightforward way of determining the full range of impacts across multiple potential EUs; however this case study provides a practical illustration of how a Government population survey can be used.

Conclusions

The SPANS case study provides important new insights into how population health monitoring data can be used to influence policy and practice. This ranged from raising awareness of the issue (i.e., child obesity), identifying priority areas and target groups for interventions, underpinning new policies and supporting and justifying existing policy and/or programs. Our findings highlight the importance of survey monitoring programs being conducted by credible independent agencies, the use of a range of dissemination strategies that directly engages EUs, and the need for strong partnerships with policy makers and EUs from the inception in order to increase ownership over the findings and commitment to action. Population monitoring data, however, needs to be considered alongside other evidence about effective intervention approaches to maximize its influence on practice and policy decisions.

Notes

Abbreviations

CIs: 

Chief investigators

EUs: 

End-users

SPANS: 

Schools’ Physical Activity and Nutrition Survey.

Declarations

Acknowledgements

This work was supported by a Capacity Building Infrastructure Grant from NSW Ministry of Health. The authors would like to thank participants for their time and contribution to this work.

Authors’ Affiliations

(1)
Prevention Research Collaboration, School of Public Health, University of Sydney
(2)
School of Public Health, University of Sydney

References

  1. Anderson WP, Papadakis EM: Research to improve health practice and policy. Med J Aust. 2009, 191 (11): 646-647.PubMedGoogle Scholar
  2. Cooksey D: A review of UK health research funding. 2006, London: HMSOGoogle Scholar
  3. Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L: Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009, 6: 21-10.1186/1743-8462-6-21.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Banzi R, Moja L, Pistotti V, Facchini A, Liberati A: Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011, 9: 26-10.1186/1478-4505-9-26.View ArticlePubMedPubMed CentralGoogle Scholar
  5. Kuruvilla S, Mays N, Walt G: Describing the impact of health services and policy research. J Health Serv Res Policy. 2007, 12 (Suppl1): 23-31.View ArticleGoogle Scholar
  6. Reed RL, Kalucy EC, Jackson-Bowers E, McIntyre E: What research impacts do Australian primary health care researchers expect and achieve?. Health Res Policy Syst. 2011, 9: 40-10.1186/1478-4505-9-40.View ArticlePubMedPubMed CentralGoogle Scholar
  7. Nutbeam D, Bauman A: Evaluation in a nutshell: a practical guide to the evaluation of health promotion programs. 2006, Sydney: McGraw Hill, 1Google Scholar
  8. Rychetnik L, Bauman A, Laws R, King L, Rissel C, Nutbeam D, Colagiuri S, Caterson I: Translating research for evidence-based public health: key concepts and future directions. J Epidemiol Community Health. 2012, 66: 1187-1192. 10.1136/jech-2011-200038.View ArticlePubMedGoogle Scholar
  9. de Goede J, Putters K, van Oers HAM: Utilization of epidemiological research for the development of local public health policy in the Netherlands: a case study approach. Soc Sci Med. 2012, 74 (5): 707-714. 10.1016/j.socscimed.2011.11.014.View ArticlePubMedGoogle Scholar
  10. de Goede J, van Bon-Martens MJH, Putters K, van Oers HAM: Looking for interaction: quantitative measurement of research utilization by Dutch local health officials. Health Res Policy Syst. 2012, 10: 9-10.1186/1478-4505-10-9.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Booth ML, Macaskill P, Phongsavan P, McLellan L, Okely T: Methods of the NSW schools fitness and physical activity survey, 1997. J Sci Med Sport. 1998, 1 (2): 111-124. 10.1016/S1440-2440(98)80019-4.View ArticlePubMedGoogle Scholar
  12. Booth ML, Denney-Wilson E, Okely AD, Hardy LL: Methods of the NSW schools physical activity and nutrition survey (SPANS). J Sci Med Sport. 2005, 8 (3): 284-293. 10.1016/S1440-2440(05)80039-8.View ArticlePubMedGoogle Scholar
  13. Hardy LL, King L, Espinel P, Okely AD, Bauman A: Methods of the NSW schools physical activity and nutrition survey 2010 (SPANS 2010). J Sci Med Sport. 2011, 14 (5): 390-396. 10.1016/j.jsams.2011.03.003.View ArticlePubMedGoogle Scholar
  14. Yin R: Case Study Research: Design and Methods. 2003, London: Sage, 5Google Scholar
  15. Kalucy EC, Jackson-Bowers E, McIntyre E, Reed R: The feasibility of determining the impact of primary health care research projects using the Payback Framework. Health Res Policy Syst. 2009, 7: 11-10.1186/1478-4505-7-11.View ArticlePubMedPubMed CentralGoogle Scholar
  16. Government NSW: NSW Government Action Plan 2003–2007 - Prevention of Obesity in Children and Young People. 2003, Sydney: NSW Department of HealthGoogle Scholar
  17. King L, Turnour C, Wise M: Analysing NSW state policy for child obesity prevention: strategic policy versus practical action. Aust New Zealand Health Policy. 2007, 4: 22-10.1186/1743-8462-4-22.View ArticlePubMedPubMed CentralGoogle Scholar
  18. Nathan SA, Develin E, Grove N, Zwi AB: An Australian childhood obesity summit: the role of data and evidence in ‘public’ policy making. Aust New Zealand Health Policy. 2005, 2: 17-10.1186/1743-8462-2-17.View ArticlePubMedGoogle Scholar
  19. Belkhodja O, Amara N, Landry R, Ouimet M: The extent and organizational determinants of research utilization in Canadian health services organizations. Sci Commun. 2007, 28 (3): 377-417. 10.1177/1075547006298486.View ArticleGoogle Scholar
  20. Davis P, Howden-Chapman P: Translating research findings into health policy. Soc Sci Med. 1996, 43 (5): 865-872. 10.1016/0277-9536(96)00130-X.View ArticlePubMedGoogle Scholar
  21. de Goede J, Putters K, van der Grinten T, van Oers HAM: Knowledge in process? Exploring barriers between epidemiological research and local health policy development. Health Res Policy Syst. 2010, 8: 26-10.1186/1478-4505-8-26.View ArticlePubMedPubMed CentralGoogle Scholar
  22. Innvær S, Vist G, Trommald M, Oxman A: Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002, 7 (4): 239-244. 10.1258/135581902320432778.View ArticlePubMedGoogle Scholar
  23. Moore G, Redman S, Haines M, Todd A: What works to increase the use of research in population health policies and programmes: a review. Evidence & Policy. 2011, 7 (3): 277-305. 10.1332/174426411X579199.View ArticleGoogle Scholar
  24. Rispel LC, Doherty J: Research in support of health systems transformation in South Africa: the experience of the centre for health policy. J Public Health Policy. 2011, 32: S10-S29.View ArticlePubMedGoogle Scholar
  25. Bonfiglioli CM, Smith BJ, King LA, Chapman SF, Holding SJ: Choice and voice: obesity debates in television news. Med J Aust. 2007, 187: 442-445.PubMedGoogle Scholar
  26. Tetroe JM, Graham ID, Foy R, Robinson N, Eccles MP, Wensing M, Durieux P, Légaré F, Nielson CP, Adily A, Ward JE, Porter C, Shea B, Grimshaw JM: Health research funding agencies’ support and promotion of knowledge translation: an international study. Milbank Q. 2008, 86: 125-155. 10.1111/j.1468-0009.2007.00515.x.View ArticlePubMedPubMed CentralGoogle Scholar
  27. Wilson PM, Petticrew M, Calnan MW, Nazareth I: Does dissemination extend beyond publication: a survey of a cross section of public funded research in the UK. Implement Sci. 2010, 5: 61-10.1186/1748-5908-5-61.View ArticlePubMedPubMed CentralGoogle Scholar
  28. Landry R, Amara N, Lamari M: Climbing the ladder of research utilization: evidence from social science research. Sci Commun. 2001, 22 (4): 396-422. 10.1177/1075547001022004003.View ArticleGoogle Scholar
  29. Lavis J, Ross S, McLeod C, Gildiner A: Measuring the impact of health research. J Health Serv Res Policy. 2003, 8 (3): 165-170. 10.1258/135581903322029520.View ArticlePubMedGoogle Scholar
  30. Wooding S, Hanney S, Buxton M, Grant J: Payback arising from research funding: evaluation of the arthritis research campaign. Rheumatology. 2005, 44: 1145-1156. 10.1093/rheumatology/keh708.View ArticlePubMedGoogle Scholar
  31. Weiss AP: Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatry. 2007, 164 (2): 206-214. 10.1176/appi.ajp.164.2.206.View ArticlePubMedGoogle Scholar
  32. Australian Society for Medical Research: Exceptional returns: the value of investing in health R&D in Australia. 2008, Australia: CanberraGoogle Scholar
  33. Kok MO, Schuit AJ: Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012, 10: 21-10.1186/1478-4505-10-21.View ArticlePubMedPubMed CentralGoogle Scholar

Copyright

© Laws et al.; licensee BioMed Central Ltd. 2013

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Advertisement