What research impacts do Australian primary health care researchers expect and achieve?

Background Funding for research is under pressure to be accountable in terms of benefits and translation of research findings into practice and policy. Primary health care research has considerable potential to improve health care in a wide range of settings, but little is known about the extent to which these impacts actually occur. This study examines the impact of individual primary health care research projects on policy and practice from the perspective of Chief Investigators (CIs). Methods The project used an online survey adapted from the Buxton and Hanney Payback Framework to collect information about the impacts that CIs expected and achieved from primary health care research projects funded by Australian national competitive grants. Results and Discussion Chief Investigators (CIs) provided information about seventeen completed projects. While no CI expected their project to have an impact in every domain of the framework used in the survey, 76% achieved at least half the impacts they expected. Sixteen projects had published and/or presented their work, 10 projects included 11 doctorate awards in their research capacity domain. All CIs expected their research to lead to further research opportunities with 11 achieving this. Ten CIs achieved their expectation of providing information for policy making but only four reported their research had influenced policy making. However 11 CIs achieved their expectation of providing information for organizational decision making and eight reported their research had influenced organizational decision making. Conclusion CIs reported that nationally funded primary health care research projects made an impact on knowledge production, staff development and further research, areas within the realm of influence of the research team and within the scope of awareness of the CIs. Some also made an impact on policy and organizational decision-making, and on localized clinical practice and service delivery. CIs reported few broader economic benefits from their research. Routine use of an instrument of this type would facilitate primary health care research funders' determination of the payback for funding of research in this sector.

Results and Discussion: Chief Investigators (CIs) provided information about seventeen completed projects. While no CI expected their project to have an impact in every domain of the framework used in the survey, 76% achieved at least half the impacts they expected. Sixteen projects had published and/or presented their work, 10 projects included 11 doctorate awards in their research capacity domain. All CIs expected their research to lead to further research opportunities with 11 achieving this. Ten CIs achieved their expectation of providing information for policy making but only four reported their research had influenced policy making. However 11 CIs achieved their expectation of providing information for organizational decision making and eight reported their research had influenced organizational decision making. Conclusion: CIs reported that nationally funded primary health care research projects made an impact on knowledge production, staff development and further research, areas within the realm of influence of the research team and within the scope of awareness of the CIs. Some also made an impact on policy and organizational decision-making, and on localized clinical practice and service delivery. CIs reported few broader economic benefits from their research. Routine use of an instrument of this type would facilitate primary health care research funders' determination of the payback for funding of research in this sector.

Background
How research is developed, conducted, disseminated and assessed is changing. Knowledge translation is high on government agendas and investment in research is expected to yield returns in the form of an improved health system delivering better health outcomes. This expectation raises the level of interest in assessing research impact [1,2] which presents conceptual methodological and practical challenges to the broad and diverse field of primary health care research [3].
Foremost among the research assessment frameworks developed by different groups has been the Payback Framework [4], developed by the Health Economics Research Group from Brunel University which encompasses a diverse range of research outputs and outcomes. Through interviews with CIs of research projects and key informants, detailed narrative case studies of the impact of individual research projects can be developed. This framework has been used in case study [4] and questionnaire format [5] to review the outcomes of funding by the UK Arthritis Research Campaign (ARC), and in questionnaire format, to assess the impact of research funded by the North Thames office of the UK NHS [6] and publicly funded health service research in Hong Kong [7]. The Framework is the basis of assessment of research impact in Canada [8]. Recently the Payback Framework has been the basis of a series of tools used by RAND Europe in an internet survey to map the ARC portfolio and the impacts of this research [5]. It has also been used in health technology assessment research projects in UK (e.g. the National Health Service Health Technology Assessment Programme and the National Coordinating Centre for Health Technology Assessment) in Canada [9] and in the Netherlands [10].
None of the above studies focus specifically on primary health care research, which is notably diverse in its topics, populations, methodologies and settings [11]. Primary health care research is a small but growing field compared with other areas of medical research in Australia [12]. Since 2000 the Australian Government has specifically funded primary health care research capacity building [3], demonstrating its strong interest in primary health care research which is relevant to and informs policy and practice in Australia. This makes it important to find a way to assess the impact of primary health care research.
An initial study on the impact of Australian primary health care research was used to understand the feasibility of using the Payback Framework. Case studies of four primary health care research projects in Australia were compiled from documentary evidence and telephone interviews with CIs and key informants, leading to the conclusion that use of the Framework and associated logic model made it feasible to determine the impacts of primary health care research with some modifications [3]. This paper reports on the second stage of the study, which aimed to assess the impact of a larger sample of primary health care research projects and to explore how impact was achieved. This paper presents an overview of the expected and achieved impacts of these primary health care research projects. An analysis of pathways by which this impact was achieved will be published separately. The Flinders University Social and Behavioural Ethics Committee approved the project.

Questionnaire design
The Buxton and Hanney Payback Framework [4] consists of a logical model of the research process and definition of payback criteria. Building on the experience of the first stage of this study [3], we adapted the Buxton and Hanney Payback Framework [4] by adding a new domain of impact which assessed enhanced capacity for research transfer. This includes both improved university engagement with the community and the health care sector and enhanced pathways and relationships for research transfer to policy makers, organisational decision makers, practitioners and consumers. This domain was added as this has been a major area of emphasis by the Australian Government and has been the primary basis for explicitly funding research in the primary care sector. An extra topic was added on the specific impact of research on clinical practice. Additionally the types of outputs were broadened to include presentations, websites, and media as they are being increasingly accepted as valid alternative outputs particularly for clinical and policy impact (as opposed to traditional research impact).
Based on the understanding that the research team's mission is relevant to how and to what extent the impact should be evaluated [13], the final questionnaire included a dichotomous screening question to determine if impact in each category was expected by the research team, and a subsequent dichotomous question to indicate whether it was achieved.
As knowledge production was treated as an essential output rather than an impact, CIs were surveyed only about the number of publications or presentations achieved from the research, likewise the number of doctorates achieved through staff development.
The final draft of the questionnaire was piloted with a project officer whose project was included in trial but who did not participate in the completion of the questionnaire. This questionnaire was ultimately adapted to an on-line format. A list of questions used in the online survey is listed in Appendix 1.

The sample
The sample included all primary health care research projects funded competitively by the National Health CIs were contacted by email and provided with a link to an information sheet about the project and a link to an on-line questionnaire. A consent form was incorporated into the questionnaire. A follow up email was sent two weeks later and, where a telephone number was available; non-responders were also contacted by telephone if they did not respond to the email requests.

Analysis
Potential categories of impact were designated in five domains: research transfer; research targeting, capacity building and absorption; informing policy and product development; health and health sector benefits; and broader economic benefits. In each category a variable was computed indicating whether an impact was expected but not achieved, expected and achieved, or achieved but not expected. An impact measure of the percentage of expected impacts achieved was computed for each project.

Results
Of the 41 projects where the CI was contactable, 23 surveys were returned of which a further 6 were excluded from analysis as the project was still in progress at the time of contact with no results to date. The results of 17 projects are reported (see Figure 1). There were 12 intervention projects and 5 descriptive projects. Specific aspects of the research projects of the 17 respondents are listed in Table 1.
CIs reported producing 39 peer-reviewed publications from their projects. Two projects yielded seven publications each, while four had not yet published in journals at the time of the survey in October 2007. The projects that produced four or more publications were either large scale complex studies involving multiple partners, or clinical trials.
The large range of journals in which primary health care research is published is shown by the 39 peer reviewed publications being published in 26 separate journals. Journal Impact Factors were available for the journals in which 22 of the 39 articles were published. With the exception of two articles, most articles were published in journals with Impact Factors less than 3.0.
Researchers reported many outputs for their research in addition to peer reviewed publications although not as many as they intended. Projects resulted in conference presentations (15/17 projects), presentations to potential users (14/16), media stories (12/16) newsletter articles (12/16), reports (11/17) and project websites (10/17). Most of these outputs involved considerable effort on the part of the research team.

Impacts: expected and achieved
CIs reported on impacts they expected and which they perceived their project had achieved, in each domain and category. The broad array of impacts achieved was almost entirely in the categories that CIs expected, but the number of impacts was less than they expected. Additional File 1, Table S1 shows the expected and achieved impacts, the impact score which is the proportion of expected impacts achieved in each domain, category and project as well as the number of papers, conference presentations and doctorates from each project.
Overall, 76% (13/17 projects) of CIs considered they had achieved at least half the impacts they expected. The impact score ranged from 93% (13/14 expectations were achieved in Project 1) to 17% (2/12 expectations were achieved in Project 17). No CI expected their project to have an impact in every domain of the impact framework used in the survey. Three projects had an unexpected impact.
CIs considered they achieved all or nearly all expectations in enhanced relationships for research transfer   and was used in guideline development or systematic review (100%, 6/6 projects and 3/3 projects respectively), and staff development (94%, 15/16 projects). Further examination of the data showed that in two of the three cases the systematic review was written by the research team themselves as a side study to their projects.
All seventeen CIs expected their research to lead to further research opportunities, the only category in which they had common expectations, and 65% reported achieving this.
Three quarters of CIs expected their research to inform policy development, organizational decision making and education. The impact score was considerably higher for providing information for policy making (77%) than for influencing policy making (31%), However, CIs reported high impact scores both in providing information for (85%) and influencing organizational decision making (73%).
In terms of health and health sector benefits, CIs reported most impact in improved service delivery (70%), use in clinical practice (58%) and improved health outcomes (50%). Examination of the survey data suggests that these perceived impacts affected the health service organizations, clinicians and patients who took part in the research projects. Impact scores were low or zero in a few categoriesmore equitable service delivery (25%), cost savings (0%), intellectual property gains (0%), improvements in population health (10%), and other economic (10%) or social (0%) impacts.

Discussion
The use of an online questionnaire in this research impact study provided data on a wide range of perceived impacts of seventeen nationally funded primary health care research projects. By asking CIs to identify areas in which they expected their projects to make an impact, and then identify areas where they had achieved impact, the study has shown that all but four of the seventeen projects in the sample reported achieving more than half of the impacts they expected. The sample is not large enough to draw any conclusions about the relative proportion of impacts achieved by intervention or descriptive types of research projects.
The CIs of these projects report numerous and diverse perceived impacts, and nearly all of the potential impacts noted in the framework occurred at least once. The projects performed strongly by achieving impacts on further research and staff development, and research transfer.
Over 90% of projects generated further research opportunities and staff development, both of which are critical for the ongoing growth and development of the primary health care research sector in Australia. Developing capacity for research transfer by enhancing university engagement with user groups and enhancing relationships for research transfer is another important area of impact, indicating that researchers are aware of the importance of these processes to complement their strong dissemination efforts through papers, conferences and other media.
The proportion of projects achieving expected impacts was over 70% for providing information for policy and organizational decision making, influencing organizational decision-making, and over 50% for use in clinical practice and improved service delivery. This reflects a high level of engagement of the researchers with potential users of their research findings.
The substantial impact on education (69%, 9/16 projects achieved this) underscores the strong link between research and education at Australian Universities.
Research use depends not only on the quality of the research and the transmission process, but on the dynamics of the political context and there is no guarantee that high quality research will be used [1]. It is therefore not surprising that most impact is achieved in areas, which are under the researchers' control, such as in knowledge production, further research, or in the organizations involved with the project.
The fact that 31% of CIs achieved their expectation of influencing policy development is notable. Providing information for policy and for organizational decisionmaking, an area where CIs achieved high proportions of expected impacts (77% and 85%), is to some extent dependent on the skills and exertion of the research team, but influencing policy is usually dependent on many factors, including the dynamics of the political context [14][15][16][17]. The reported impacts on policy from these seventeen projects were substantial. Some examples were the study of the 3+ Asthma Plan in the Systematic practice-based asthma care in the Australian Setting project, which contributed to a new model of Medicare Benefits Scheme item numbers for asthma, while the Audit and Best Practice in Chronic Disease project informed the development of major Indigenous Health initiatives (Continuous Improvement projects and the Healthy for Life Program). It also influenced Northern Territory policy making as all primary care services are now expected to adopt the approaches developed in the project. The approach has also been adopted in Western Australia, New South Wales, Queensland and South Australia. The Disclosure and Attitudes to Lesbians: Outcomes in General Practice (DIALOG) project findings contributed to a Victorian Government action plan and cultural awareness training for general practitioners ( Table 1).
The results in terms of influence on organizational decision-making, and flow on effects on localised clinical practice and service delivery are consistent with an earlier Australian study of NHMRC funded projects commissioned by the National Institute of Clinical Studies (NICS) which found 24% had resulted in some kind of translation into practice, usually in the researcher's local area or health service [14]. Little impact was expected or reported on broader economic benefits, and none on cost savings or intellectual property gains.

Limitations
Limitations are the small number of projects, the Chief Investigators' knowledge of their project impacts, and risks of a bias towards positive benefits [5].
We did not expect that we would be unable to locate CIs of 18 of the 59 projects, despite extensive searching for these individuals. Due to limitations in data available from the Australian Government on funded grants, a detailed analysis of patterns of non-response could not be undertaken. The sample frame consisted of research directly funded by the Australian Government provided by competitive peer review with minimum funding of $80, 000. These results cannot be generalised to primary health care research funded through other means such as tenders, non-competitive processes, consultancies, for lesser amounts and by other organizations including service providing organizations, and professional associations [18].
In an earlier study we collected qualitative data through interviews with CIs and gathered copious information that provided more context to better understand the results. However this process was time-consuming for both the interviewee and interviewer and so unlikely to be a useful tool for routine use by funders. The use of an online questionnaire in this subsequent study took less time but yielded information which was less detailed.
Some modification of the survey may increase the accuracy of reporting. The use of dichotomous variables to assess categories of impact could more accurately be replaced by a scale depicting a range of levels of achievement (e.g. significant, some, marginal impact).
The results are likely to underestimate the impacts of these projects, as CIs do not usually know of the use made of their work beyond publication in the literature and events in their own realm of awareness. As three projects were yet to be fully completed at the time of the survey, and others were very recently completed and were still writing for publication, it is possible that further impact would be achieved over time. The collective impact of the projects is thus a work in progress and must be viewed as a snapshot and not a definitive statement of eventual impact.

Implications for primary health care research
This study demonstrates that individual primary health care research projects can have an impact in a range of areas addition to traditional production of knowledge through journal articles and reports.
An ongoing process to identify the impacts of individual research projects could raise awareness and understanding of some research impacts and how they are achieved. Researchers who could identify the impacts of their work could then use this knowledge to illustrate the relevance of their work as part of their ongoing engagement with the people in their particular sector on whose good will they often depend. However, the current Excellence in Research Australia initiative provides more incentives for academic researchers to publish in highly rated journals than to identify or achieve societal impact [19].
Proving a connection between individual research projects and impacts such as more equitable service delivery, cost savings or improvements in population health is unlikely, because of the difficulties of attribution, the need for synthesis of findings of many primary studies and the complex social and political processes through which policy change takes place. A focus on individual studies may result in underestimating the impact of primary health care research.

Conclusions
This study shows that CIs perceived their individual primary health care research projects had made numerous impacts on research transfer, knowledge production, research capacity building, informing policy and localised health and health sector benefits, the categories of the Payback framework. Such impacts are consistent with the interest of the Australian government in funding relevant primary health care research that can inform policy and practice. The impacts were in the areas expected by the Chief Investigators, though fewer than they expected.
The use of an online questionnaire was practical for collecting information retrospectively from CIs about the impact of their research, while recognising that the information they provide is limited to the impacts within their realm of awareness, and time since completion. Routine use of such a questionnaire by primary health care funders at the completion of a funded project and two years after completion of the project would allow funders to better determine value for money. This is particularly relevant as specified funding is generally justified by the critical importance of research to have an impact on the primary health care sector which in comparison to the hospital sector has had a small fraction of funding through traditional sources. It would in turn encourage investigators to keep more accurate records regarding the impacts of their research on policy and practice at the time that they become aware of these impacts and to broaden their focus beyond publications and citations.

Additional material
Additional file 1: Table S1 -Chief Investigators' perceptions of expected and achieved impact of Australian primary health care research projects.