Skip to main content

How are evidence generation partnerships between researchers and policy-makers enacted in practice? A qualitative interview study

Abstract

Background

Evidence generation partnerships between researchers and policy-makers are a potential method for producing more relevant research with greater potential to impact on policy and practice. Little is known about how such partnerships are enacted in practice, however, or how to increase their effectiveness. We aimed to determine why researchers and policy-makers choose to work together, how they work together, which partnership models are most common, and what the key (1) relationship-based and (2) practical components of successful research partnerships are.

Methods

Semi-structured qualitative interviews were conducted with 18 key informants largely based in New South Wales, Australia, who were (1) researchers experienced in working in partnership with policy in health or health-related areas or (2) policy and programme developers and health system decision-makers experienced in working in partnership with researchers. Data was analysed thematically by two researchers.

Results

Researcher-initiated and policy agency-initiated evidence generation partnerships were common. While policy-initiated partnerships were thought to be the most likely to result in impact, researcher-initiated projects were considered important in advancing the science and were favoured by researchers due to greater perceived opportunities to achieve key academic career metrics. Participants acknowledged that levels of collaboration varied widely in research/policy partnerships from minimal to co-production. Co-production was considered a worthy goal by all, conferring a range of benefits, but one that was difficult to achieve in practice. Some participants asserted that the increased time and resources required for effective co-production meant it was best suited to evaluation and implementation projects where the tacit, experiential knowledge of policy-makers provided critical nuance to underpin study design, implementation and analysis. Partnerships that were mutually considered to have produced the desired outcomes were seen to be underpinned by a range of both relationship-based (such as shared aims and goals and trust) and practical factors (such as sound governance and processes).

Conclusions

Our findings highlight the important role of policy-makers in New South Wales in ensuring the relevance of research. There is still much to understand about how to initiate and sustain successful research/policy partnerships, particularly at the highly collaborative end.

Peer Review reports

Background

Bridging the evidence–practice gap in health services has the potential to make a substantial contribution to improving health outcomes and service efficiencies [1, 2] and to reducing research waste [1]. Proposed solutions to bridging this gap originally focussed heavily on strategies to enhance the dissemination of research findings to research users [2, 3] or on building the capacity of policy-makers and clinicians to engage with and use research evidence [4]. More recently, focus has turned to the potential for evidence generation partnerships between researchers and knowledge users, such as policy-makers, community groups, clinicians and consumers, to improve the availability of relevant, timely evidence to help inform decision-making [5, 6]. There has been a proliferation of research funding initiatives internationally that require researcher/knowledge user collaboration [7] and increasing demands on researchers to demonstrate that their work has had an ‘impact’ on policy or practice [8,9,10].

A number of research approaches that centre around evidence generation partnerships have been outlined in recent years, including integrated knowledge translation [5, 11], participatory action research [12, 13] and engaged scholarship [14, 15]. Most of these approaches emphasise the co-production of knowledge [6], whereby researchers and those most likely to use or be affected by the evidence produced, work together to produce evidence. By combining the varied skills and expertise of these groups, it is hypothesised that the resultant research will have greater relevance to knowledge users and will more likely be used in decision-making [16, 17].

Much has been published about many of these approaches from a conceptual standpoint [18, 19], and there has similarly been a considerable amount of literature examining how evidence generation partnerships function [13, 20, 21]. Most of this research, however, does not explore evidence generation partnerships specifically between researchers and policy-makers [22, 23], focusing instead on partnerships with community groups [13], clinicians [24] and others. All of these partnership combinations are likely to share some common features such as a requirement to take into account the diverse perspectives and needs of all partners; however, the policy-maker/researcher combination is likely to present some unique partnership challenges and opportunities due to the vastly different work environment and pressures that these groups face [25,26,27]. Further, while rarely the case with most other common partnership combinations, evidence generation partnerships between policy agencies and researchers may be initiated and funded by the policy agency themselves or be funded through external research grants (such projects may be more likely to be researcher-initiated). These different modes of funding and initiation may have important implications for how a partnership functions [28]; however, this has rarely been explored. Thus, there remains much to be learnt about how research/policy partnerships are enacted in practice [29] and how best to bring the diffuse skills, needs and priorities of researchers together effectively [30, 31].

The current paper is the first step in a broader initiative which aims to develop a set of tools to help facilitate successful research/policy partnerships around evidence generation activities. In this paper, we explore the views and experiences of researchers and policy-makers with extensive expertise in research partnerships regarding why and how researchers and research users work together in New South Wales, Australia, and the factors which underpin successful research partnerships. In particular, we address the following research questions:

  1. 1.

    Why do researchers and policy-makers choose to work together? What are the perceived benefits?

  2. 2.

    How do they work together? Which partnership models are most common?

  3. 3.

    What are the key (1) relationship-based and (2) practical components of successful research partnerships?

Methods

Participants and setting

Data was collected between October 2017 and April 2018. We sought to interview participants from two groups; namely (1) researchers experienced in working in partnership with policy in health or health-related areas and (2) policy and programme developers and health system decision-makers (hereafter policy-makers, defined as “someone employed in a policy agency who drafts or writes health policy documents or develops health programs, or who makes or contributes significantly to policy decisions about health services, programs or resourcing” [32]) experienced in working in partnership with researchers. Participants were purposively identified by an Advisory Committee guiding this project. The role of the Advisory Committee was to provide expert advice on issues relating to the scope, implementation and dissemination of a programme of work centred on partnership research of which the current paper is a part. The Advisory Committee included an approximately equal mix of researchers and senior policy-makers, selected for their expertise in partnership research, and their extensive knowledge of researchers and policy-makers who work in partnered research. Advisory Committee members asked the permission of potential participants before passing on their contact details to the study team.

Participants were eligible to participate in the current study if they were nominated by a member of the Advisory Committee based on the criteria that they (1) were a researcher or policy-maker; (2) worked primarily in health or health-related areas (e.g. social care); (3) had extensive experience in partnering with researchers (for policy-makers) or policy-makers (for researchers) on research projects (partnership on at least five projects); and (4) were employed by a university, policy or programme agency or Local Health District. All participants provided written, informed consent to participate.

Data collection

A semi-structured approach was used to elicit participants’ opinions and experiences regarding the barriers and facilitators to successful evidence generation partnerships amongst researchers and research users in accordance with best practice guidelines (Additional file 1: Appendix 1) [33]. Interviews were conducted by AW, a researcher with extensive expertise in conducting research in partnership with policy-makers. Each interview sought information on why the interviewee engaged in partnership research and the perceived risks and benefits, how the interviewee would characterise the types of partnership models they have engaged in (and the advantages and disadvantages of each), characteristics of successful and unsuccessful partnerships (and the factors associated with each), and indicators and predictors of impact. Interviews were conducted in person or by phone depending on the preference of the interviewee and ranged from half to one hour in length. Data was audio recorded before being transcribed by a person with no personal or professional connection to the participants involved in the study.

Analysis

Data was analysed thematically. Two researchers (AW and HT) independently read all of the transcripts and coded the data to discern themes inductively. No predetermined framework was used to guide analysis. The researchers met regularly to review the draft codes and themes throughout the process until agreement was reached regarding the final version. Synthesised data and emerging themes were reviewed by the Advisory Committee who participated in sense making of emerging data.

Results

A total of 18 key informants participated in interviews, of whom 7 were primarily researchers and 11 were currently primarily policy-makers. All participating researchers were employed by universities at the Associate Professor or Professor level and had more than 15 years research experience in public health research. All of the policy-makers who participated were employed at a Manager level or above by government agencies whose work focussed on health or health-related issues (such as social care). Six of the policy-makers had PhDs and/or had previously been employed as researchers and were thus able to bring both a researcher and a policy-maker lens to their analysis of research partnerships. All participants, except for one researcher, were based in New South Wales, Australia. The themes which emerged from the interview with the participant from outside New South Wales were consistent with those which emerged more broadly.

Why do researchers and policy-makers choose to work together?

Three major themes emerged regarding why researchers choose to work with policy, namely (1) increasing the likelihood of research impact; (2) gaining access to sought after resources; and (3) obtaining funding. The reason most commonly cited by researchers for wanting to partner with policy-makers to conducting research was the belief that the resultant research would more likely have an impact on policy and thus contribute to improving health.

I do research so that I can improve health… the main way in which research impacts health is through policy... Research without any sort of policy engagement will often sit on the shelf and do nothing…” Researcher

Many researchers were also motivated to partner with policy-makers due to the access to otherwise unavailable resources this could facilitate (such as routinely collected data held by agencies or the data required to evaluate large-scale government policies or programmes). Researchers also reported seeking collaborations with policy-makers due to the funds attached to the work, either through completing a tendered project or, preferably, receiving funds to carry out a project the researchers had initiated or co-produced in collaboration with a relevant agency.

Five major themes emerged regarding what policy-makers perceived they gained by partnering with researchers, namely (1) access to additional skills; (2) links to researchers who can be called on for timely, informal advice; (3) access to the networks of their research collaborators; (4) the creation of high quality, relevant evidence; and (5) public, evidence-based support for government decisions. Policy-makers most commonly reported seeking to collaborate with researchers in order to gain access to skills and capacity over and above that already available within their agency.

…the reason you go to a researcher is that they’ve got the expertise or the resources or the time or whatever in order to be able to collect that information that you need and therefore that’s helpful to you.” Policy-maker

Having established collaborations with researchers was also said to create a situation in which policy-makers felt able to contact these researchers when they required fast expert advice on a relevant issue. Policy-makers also noted that their collaborators often assisted them in making links with researchers outside of the collaboration who possessed additional specific skills or knowledge they needed. Policy-makers were also driven to collaborate with researchers in order to create high quality evidence that was seen to have credibility to support their work.

Collaborations with researchers were also seen to be helpful when the researcher elected to support government decisions in the public domain, for example, by speaking about the evidence base underpinning policy, programme or health service delivery decisions. While research users did not report requesting assistance of this nature, they appreciated it when provided.

How do researchers and policy-makers collaborate on research projects?

Four major themes emerged in relation to researcher/policy-maker collaboration, namely (1) wide variation in extent of collaboration; (2) policy agency-initiated partnerships; (3) researcher-initiated partnerships; and (4) co-produced partnerships. Most participants agreed that “someone has to have the initial idea” for a partnership; as such, participants agreed that most partnerships could be categorised as either policy agency initiated, including projects put out for tender by agencies and commissioned work, or researcher initiated, either through a partnered funding application to an external granting body or by a direct approach to an agency to fund a particular piece of work. Once these partnerships began, there was said to be a considerable level of variation in the extent to which the researchers and agency staff involved collaborated from almost no collaboration through to high levels of collaboration throughout all stages of the research process. This sustained, high level collaboration was considered by participants to characterise co-production. Policy-makers and researchers tended to nominate research projects centred around evidence reviews and analysis of existing datasets as requiring minimal collaboration, while closer collaboration was considered important when conducting interventions and evaluations.

So, I see partnerships as a sort of continuum. At one end is that transactional type interaction, through to co-production at the other end.” Policy-maker

However, these partnership categories (policy- or researcher-initiated and extent of collaboration) were not seen to be entirely clear cut, with some researchers and policy-makers explaining that, as some policy agencies move towards embedding researchers within their teams and sometimes funding research centres, the distinction between policy-makers and researchers can sometimes be blurry.

A few policy-makers felt that highly collaborative, co-produced research partnerships were always preferable; however, most participants reported that various levels of collaboration could be effective depending on the situation.

I guess again we don’t come at it from which model works best. It’s more like what are the requirements of the project.” Policy-maker

Policy agency-initiated partnerships

A range of shared benefits (two subthemes), perceived unique risks to researchers (five subthemes), perceived unique benefits to researchers (two subthemes), and unique risks to policy-makers (five subthemes) of policy agency-initiated partnerships were identified (see Table 1 for illustrative quotes). Most researchers and policy-makers alike agreed that policy-makers are best placed to identify the critical issues around what evidence is needed to guide their decision-making. Thus, agency-initiated work was thought by both researchers and policy-makers to often be particularly well targeted for real-world impact.

Table 1 Illustrative quotes regarding policy agency-initiated partnerships

Small, short term, policy-initiated projects were rarely seen by researchers as offering opportunities to conduct ‘cutting edge research’, and reportedly often did not generate publications or turn a profit. Researchers reported willingness to engage in these and other types of agency-initiated projects anyway as they were seen as an effective way to build a relationship with a policy agency and gain a better understanding of the policy context in the hopes of developing more advantageous collaborations.

On the other hand, most researchers outlined a number of potential risks or costs that were sometimes associated with agency-initiated work. Chief among these was a perceived opportunity cost, with time spent working on agency-initiated research sometimes resulting in less traditionally valued research outputs (such as peer-reviewed publications) than time spent on researcher-initiated work. Indeed, policy-makers and researchers alike reported that, as policy-makers often needed evidence around a specific issue, within a specific timeframe and at a specific cost, research methods needed to be determined pragmatically, rather than striving for the most scientifically excellent design. In addition, the evidence produced was sometimes rendered irrelevant by changes in the political environment.

Time was again a concern amongst researchers when the amount of time spent to complete a project to an agency’s satisfaction was seen to ‘blow out’. Researchers and some policy-makers reported that this was often due to agencies not being “clear [about] what they want”, resulting in a more iterative process than was anticipated. This challenge was also sometimes seen to be related to agencies not disclosing key information to researchers, limiting their understanding of the context and what was required.

Policy-makers also reported potential risks related to engaging with researchers in agency-initiated research, including not receiving the information or evidence they required, or it being of poor quality, or delivered well after the agreed deadline. Another commonly reported barrier was researchers overpromising to win a tender but not being able to deliver on these promises. Frustrations were sometimes said to arise due to researchers ignoring the complexity of the issue being investigated.

Finally, policy-makers reported that, in policy-initiated projects, researchers often failed to adequately synthesise the reported evidence and/or provide recommendations for action. As recommendations for action were reportedly often one of the key outcomes sought by agencies in these partnerships, this was particularly disappointing.

Researcher-initiated partnerships

One shared benefit related to research-initiated partnerships was commonly identified by researchers and policy-makers. Policy-makers also identified a range of risks associated with such partnerships (four subthemes). Researchers considered that partnerships they led conferred a range of important benefits (three subthemes), but also some risks (two subthemes) (see Table 2 for illustrative quotes). Researcher-initiated partnerships were seen by policy-makers as an important source of innovation, but not necessarily of major immediate relevance to their work. Despite this, if the policy-makers agreed to partner on a researcher-initiated project, they reported that they did expect to engender benefits from this. Nonetheless, one of the key risks noted for researcher-initiated partnerships was that the evidence produced did not fill an evidence need for the agency, or that it took so long to produce that, by the time it was available, it was no longer relevant.

Table 2 Illustrative quotes about researcher-initiated partnerships

This risk was seen to be exacerbated by another commonly noted risk of this partnership type, namely that the researchers take over and offer little opportunity for policy-makers to help shape the research agenda. Relatedly, many policy-makers reported previous dissatisfactions related to researchers not sharing credit for the work the partnership produced, for example, through co-authorship or shared publicity.

Researchers cited significant benefits associated with researcher-initiated partnerships, including the opportunity to focus on their specific research interests and employ more ambitious study designs, increasing their chances of obtaining peer-reviewed grants and high impact journal articles. The chief risks researchers reported in researcher-initiated research partnerships were that the policy-maker partners did not engage (attend meetings, provide information and advice) or did not deliver on promised resources such as access to particular datasets.

Co-produced partnerships

All participants reported that co-production was a worthy goal and likely to be highly effective when done well (four subthemes). Some regarded it as the most effective partnership model for real world impact (see Table 3 for illustrative quotes). Achieving co-production was also noted to be difficult, with common challenges (two subthemes) and a range of facilitators (four subthemes) identified.

Table 3 Illustrative quotes regarding co-production

The benefits of co-production were seen to derive in part from the greater levels of engagement between all members of the team seen to be implicit in this model, driven by mutual aims and interests. This high level of engagement was in turn seen to allow the complementary skills and knowledge of all partners to be optimised. Resulting from this, a core strength of co-production was thought to be its ability to integrate tacit, experiential knowledge with traditional ‘evidence’, resulting in particularly nuanced and relevant outputs. Co-production was also seen to provide opportunities for frequent ‘relevance checks’ to ensure that the research being undertaken remains in line with ‘real world’ priorities.

Nonetheless, while co-production was the goal for many participants, some noted that, thus far, they had not been able to achieve it, with either the policy or the researcher partners dominating partnership projects in practice. Multiple reasons were given for the reported tendency for one group to dominate research partnerships, ranging from lower expectation amongst one group that the project might benefit themselves or their agency resulting in reduced engagement, to the dominating group offering little real opportunity for their partners to contribute to shaping the research agenda. The most frequently mentioned facilitators of co-production were things that allowed long-term relationships and trust to develop between researchers and policy-makers, namely stability of staff at policy agencies and policy agency-funded research centres.

What does ‘success’ in research partnerships look like?

Seven subthemes were identified in relation to the characteristics of successful partnerships. Key informants most commonly described a successful partnership as one where “all stakeholders are happy with the outcomes”. Expanding further, common perceptions of successful partnerships included all parties having a shared understanding of why they were partnering and what this would involve, the planned deliverables being delivered to a high standard and relevant, usable evidence being generated. Many participants noted that a successful partnership produces more than any partner could have alone and that it results in mutual gain. Creating evidence that impacts on policy and practice was considered an indicator of success, but as participants recognised that impact is influenced by a complex array of factors, they considered that partnerships could be considered successful even in the absence of this.

What are the components of successful research partnerships?

Participants identified a range of relationship-based (six subthemes) and practical facilitators (six subthemes) of successful partnerships as well as three cross-cutting facilitators (Table 4).

Table 4 Major themes regarding relationship-based and practical facilitators of successful partnerships

Shared aims and goals were seen as the fundamental building block of successful partnerships, and something that motivated partners to withstand the difficulties and challenges that can emerge over the course of partnerships.

So that’s my little tip there. All the energy at the start has to be about where you want to finish and why a partnership is important, because it will also help you during the times when the partnership’s tested, to remind yourself why working together was important in the first place.” Researcher

Participants were also keenly aware of the differences between researchers and policy-makers in relation to their needs and goals. For example, researchers are generally expected to demonstrate their productivity through publications and funded grants, whereas policy-makers need to provide advice or develop policies or programmes around complex issues, often with very little preparation time. Both groups emphasised the necessity of understanding and attempting to accommodate these different needs.

… it’s the relationship building that’s critical, being prepared to put the time and effort into the relationship building and having the ability to empathise with the perspective of someone who’s trying to run a health service or someone who’s having to make a policy decision…” Researcher

Practical components of successful partnerships

All of the practical components of successful partnerships were seen to be underpinned by the need to take time at the outset of the partnership (and throughout) to ensure that ‘everyone is on the same page’. Practical ways of doing this that were frequently cited included documenting and agreeing on timelines, deliverables and who is responsible for what at the outset (but allowing some flexibility for when changes arise), agreeing on governance structures and processes and a publication policy (which specifies the potential role of research users in publications). Adequate funding was seen as another core practical component of success, as was all parties delivering as promised. Ongoing engagement, expressed through attendance at meetings, participation in discussions and provision of feedback, for example, was also seen as key. Mutual benefit was cited by all participants as fundamental to partnership success.

The best partnerships are clearly win/win…Where it becomes one-sided, all the gains being on one side at the expense of the other, that’s where things fall apart because the side that’s not gaining is thinking well, what am I doing here?” Researcher

Discussion

Our findings suggest that both researcher-initiated and policy agency-initiated evidence generation partnerships were common among researchers and policy-makers in New South Wales. While policy-initiated partnerships were thought to be the most likely to result in impact [34], researcher-initiated projects were seen to play an important role in advancing the science and often provided researchers with more opportunities to achieve the outputs valued in their profession such as high impact publications. In keeping with the literature, participants acknowledged that levels of collaboration varied widely between research/policy partnerships [16, 20, 28, 35, 36].

Co-production, or collaboration across all stages of the research process, was seen by some participants as the ideal model for collaboration, increasing the relevance of the research produced and the likelihood of impact [37,38,39]. Others considered that this time and resource-intensive way of working was not always warranted [40]. While agreeing aims, goals and deliverables collaboratively was always viewed as essential [41], these informants suggested that, once this was accomplished, some types of projects, such as analyses of large datasets and evidence reviews, could be completed largely independently by researchers with no loss of quality or relevance. In contrast, co-production was seen to be particularly worthwhile in relation to evaluation and implementation projects where the tacit, experiential knowledge of policy-makers provided critical nuance to underpin study design, implementation and analysis. Of note, several policy-makers suggested that, while ideally they would like to co-produce research, in practice they had not found this to be possible. Participants suggested that trusting, long-term relationships were generally a precursor to co-production. These were seen to be facilitated by policy agencies having a stable workforce; frequent staff changes in policy agencies are frequently cited as a barrier to successful evidence generation partnerships in the literature [31, 42] and policy agency-funded research centres [43].

Key informants in the current study highlighted a clear distinction between researcher and policy agency-initiated partnerships. In keeping with the literature, policy-makers were thought by both groups to be best placed to identify what evidence is needed to guide real world decision-making [5, 44, 45]. The relevance of the ensuing research and the ability of policy agencies to utilise it meant that policy agency-initiated work was reported to be particularly likely to lead to real-world impact [22, 34]. Achieving such impact was the primary motivation reported by researchers for engaging in evidence generation partnerships [22]. Potential opportunity costs were also noted, including time taken away from producing ‘world class’ research [28, 46], challenges agreeing on final outputs [28, 29], unexpectedly iterative processes [47] and the potential for the evidence produced to remain unused due to changes in the policy landscape [29].

Many of the key frustrations expressed by policy-makers around policy agency-initiated partnerships in the current study have been less commonly reported in the literature. These include researchers overstating what is possible in order to win tenders (leading to disappointment with the final product), choosing only to focus on the elements of a topic that interest them, or oversimplifying things in order to make them ‘doable’ and failing to fully engage with the complexity of an issue. As systems thinking approaches gain momentum in research [48, 49], it will be interesting to observe whether this last challenge becomes less common. A key frustration noted by policy-makers was the failure of researchers to provide recommendations for action, one of the main outcomes reportedly sought by agencies when they initiated partnerships with researchers [28, 50, 51]. Despite the noted risks, researchers stated that even small agency-initiated projects were worthwhile as relationship-building and learning exercises [27, 52], which may later help to underpin partnerships they felt to be more favourable to themselves. Policy-makers for their part continued to initiate evidence generation partnerships in order to access additional capacity and expertise to produce high quality, credible evidence to inform their work.

Researchers in the current study placed particular value on researcher-initiated partnerships. These were considered to offer the opportunity to conduct research that was both ‘scientifically excellent’, and likely to have impact. Policy-makers, on the other hand, noted that, while sometimes providing useful evidence, these projects were less often of immediate relevance or importance to them. Unsurprisingly then, the major risk researchers reported in researcher-initiated partnerships was that policy-makers did not engage [29, 53]. This suggests that even researchers who aim to conduct research with real world impact, and who are experienced in working partnership with policy, are often not in alignment with policy agencies in their opinions as to research priorities. This finding is consistent with the large body of literature describing the widespread differences between the research and policy worlds [26, 27, 37, 54], whereby researchers most often attempt to ‘push’ their research to policy agencies rather than ascertaining when there may be a policy ‘pull’ for evidence and building at least some of their research agenda to align with this [55, 56]. One mechanism governments in New South Wales [57] and others internationally [58,59,60,61] employ to help develop long-term collaborations with researchers and increase the production of policy-relevant research is government-funded research centres. While as yet there have been few evaluations of such initiatives, it seems a potentially promising method of facilitating productive, lasting partnerships.

Many of the key barriers and facilitators reported to underpin partnership success or failure noted in the current study are consistent with those reported in the literature. Many of these appeared to speak primarily of a clear need to ensure that the different needs and drivers of all parties were understood, valued and accommodated as much as possible. Thus, participants underlined the foundational need for shared aims and goals, ensuring that all parties were benefiting from the collaboration and maintaining open communication, trust and mutual respect [16, 20, 21, 27, 42].

Amongst our key informants, practical tools were also thought to be of great value in supporting a well-functioning partnership; these included having clearly documented and agreed timelines, deliverables, workplans, publication policies and governance structures [31, 53]. While some groups appear to have well-developed methods in this area, this appears to be a partnership capacity gap for others. As noted elsewhere [29, 31], ongoing engagement, expressed through things such as attendance at meetings, participation in discussions and provision of feedback and presenting findings as they emerge, was described as critical to keeping the project on track and maintaining relationships.

A strength of the current study is the participation of both researchers and policy-makers who are highly experienced in working in partnership on evidence generation activities as we expected this group to be able to provide considerable insight into the complexities of research/policy partnerships. By exploring their views broadly, rather than in relation to a particular project, we were able to draw out their key learnings from working in partnership on evidence generation activities. We acknowledge the limitations inherent in the small sample size utilised here, the fact that most of our participants were from New South Wales and that our sample was not randomly selected. Indeed, the experiences of these highly practiced individuals may diverge from those of researchers and policy-makers who have less experience of participating in research/policy partnerships, or who work in settings where such partnerships are less common. Although many of our findings align with the existing literature, these limitations mean that our findings may not be broadly generalisable.

There is still much to understand about how to initiate and sustain successful research/policy partnerships, particularly at the highly collaborative co-production end. Our findings highlight the important role of policy-makers in ensuring the relevance of research, mostly in evaluation and implementation projects where co-production appears to be particularly valuable. The next phase of our work will seek to provide a more detailed exploration of the specific practical strategies, including governance approaches, that can be put in place to facilitate strong and effective evidence generation partnerships between researchers and policy-makers.

References

  1. Macleod M, Michie S, Roberts I, Dirnagl U, Chalmers I, John PA, Ioannidis J, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4.

    Article  Google Scholar 

  2. Dobbins M, Hanna SE, Ciliska D, Manske S, Cameron R, Mercer SL, et al. A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies. Implement Sci. 2009;4:61.

    Article  Google Scholar 

  3. Lavis JN, Wilson MG, Grimshaw JM, Haynes RB, Hanna S, Raina P, et al. Effects of an evidence service on health-system policy makers’ use of research evidence: a protocol for a randomised controlled trial. Implementation Sci. 2011;6:51.

    Article  Google Scholar 

  4. CIPHER Investigators. Supporting Policy In health with Research: an Intervention Trial (SPIRIT) – protocol for a stepped wedge trial. BMJ Open. 2014;4(7):e005293.

    Article  Google Scholar 

  5. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26:13–24.

    Article  Google Scholar 

  6. Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. 2016;11:20.

    Article  Google Scholar 

  7. McLean RKD, Graham ID, Tetroe JM, Volmink JA. Translating research into action: an international study of the role of research funders. Health Res Policy Syst. 2018;16:44.

    Article  Google Scholar 

  8. Greenhalgh T, Fahy N. Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework. BMC Med. 2015;13:232.

    Article  Google Scholar 

  9. Hanney S, Greenhalgh T, Blatch-Jones A, Glover M, Raftery J. The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews. Health Res Policy Syst. 2017;15:26.

    Article  Google Scholar 

  10. Newson R, King L, Rychetnik L, Bauman AE, Redman S, Milat AJ, et al. A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: perceptions of Australian researchers. BMJ Open. 2015;5(7):e008153.

    Article  Google Scholar 

  11. Graham ID, Kothari A, McCutcheon C, Integrated Knowledge Translation Research Network Project Leads. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implementation Sci. 2017;13:22.

    Article  Google Scholar 

  12. Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health. 2006;60(10):854.

    Article  Google Scholar 

  13. Jagosh J, Biush PL, Slalsberg J, Macaulay MC, Greenhalgh T, Wong G, et al. A realist evaluation of community participatory research: partnership synergy, trust building and ripple effects. BMC Public Health. 2015;15:725.

    Article  Google Scholar 

  14. Bowen SJ, Graham ID. From knowledge translation to engaged scholarship: promoting research relevance and utilization. Arch Phys Med Rehabil. 2013;94:S3–8.

    Article  Google Scholar 

  15. Van de Ven AH. Engaged Scholarship: A guide to Organizational and Social Research. New York: Oxford University Press; 2007.

    Google Scholar 

  16. Sibbald SL, Tetroe J, Graham ID. Research funder required research partnerships: a qualitative inquiry. Implementation Sci. 2014;9:176.

    Article  Google Scholar 

  17. Solberb LI, Glasgow KE, Unutzer J, Jaeckels N, Oftedahl G, Beck A et al. Partnership Implementation Research. Medical Care. 2010; 48(7): 576-82.

    Article  Google Scholar 

  18. Graham I, Tetroe J. Some theoretical underpinnings of knowledge translation. Acad Emerg Med. 2008;14(11):936–41.

    Article  Google Scholar 

  19. Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94(2):392–429.

    Article  Google Scholar 

  20. Corbin JH, Jones J, Barry MM. What makes intersectoral partnerships for health promotion work? A review of the international literature. Health Promot Int. 2018;33(1):4–26.

    PubMed  Google Scholar 

  21. Rycroft-Malone J, Burton C, Wilkinson J, Harvey G, McCormack B, Baker R, et al. Collective action for knowledge mobilisation: a realist evaluation of the Collaborations for Leadership in Applied Health Research and Care. Health Services and Delivery Research. Southampton: NIHR Journals Library; 2015.

    Google Scholar 

  22. Bowen S, Botting I, Graham ID, Huebner LA. Beyond “two cultures”: guidance for establishing effective researcher/health system partnerships. Int J Health Policy Manag. 2017;6(1):27–42.

    Article  Google Scholar 

  23. Voorberg WH, Bekkers VJJM, Tummers LG. A systematic review of co-creation and co-production: embarking on the social innovation journey. Public Manag Rev. 2015;17(9):1333–57.

    Article  Google Scholar 

  24. Sahs JA, Nicasio AV, Storey JE, Guarnaccia PJ, Lewis-Fernandez R. Developing research collaborations in an academic clinical setting: challenges and lessons learned. Community Ment Health J. 2017;53(6):647–60.

    Article  Google Scholar 

  25. Caplan N. The two-communities theory and knowledge utilization. Am Behav Sci. 1979;22(3):459–70.

    Article  Google Scholar 

  26. Brownson RC, Royer C, Ewing R, McBride TD. Researchers and policymakers: travelers in parallel universes. Am J Prev Med. 2006;30(2):164–72.

    Article  Google Scholar 

  27. Kothari A, MacLean L, Edwards N, Hobbs A. Indicators at the interface: managing policymaker-researcher collaboration. Knowledge Manage Res Pract. 2011;9:203–14.

    Article  Google Scholar 

  28. Martin S. Co-production of social research: strategies for engaged scholarship. Public Money Management. 2010;30:211–8.

    Article  Google Scholar 

  29. Nystrom ME, Karltun J, Keller C, Gare A. Collaborative and partnership research for improvement of health and social services: researchers experiences from 20 projects. Health Res Policy Syst. 2018;16:46.

    Article  CAS  Google Scholar 

  30. Kothari A, Wathen CN. Integrated knowledge translation: digging deeper, moving forward. J Epidemiol Community Health. 2017;71:619–23.

    Article  Google Scholar 

  31. Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implementation Sci. 2016;11:38.

    Article  Google Scholar 

  32. Haynes A, Turner T, Redman S, Milat AJ, Moore G. Developing definitions for a knowledge exchange intervention in health policy and program agencies: Reflections on process and value. Int J Soc Res Methodol. 2015;18(2):145–59.

    Article  Google Scholar 

  33. Tracy SJ. Qualitative quality: eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry. 2010;16(10):837–51.

    Article  Google Scholar 

  34. Kok MO, Gyapong JO, Wolffers I, Ofori-Adjei D, Ruitenberg J. Which health research gets used and why? An empirical analysis of 30 cases. Health Res Policy Syst. 2016;14:36.

    Article  Google Scholar 

  35. Murray L. Deliberative research for deliberative policy making: creating and recreating evidence in transport policy. Social Policy Society. 2011;10(4):459–70.

    Article  Google Scholar 

  36. Ross S, Lavis J, Rodriguez C, Woodside J, Denis J. Partnership experiences: involving decision-makers in the research process. J Health Serv Res Policy. 2003;8(Suppl 2):26–34.

    Article  Google Scholar 

  37. Campbell D, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: Practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21.

    Article  Google Scholar 

  38. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7(4):239–44.

    Article  Google Scholar 

  39. Lomas J. Improving Research Dissemination and Uptake in the Health Sector: Beyond the Sound of One Hand Clapping. In: Policy Commentary. Hamilton, ON: McMaster University Centre for Health Economics and Policy Analysis; 1997. Report No.: C97–1.

    Google Scholar 

  40. Gajda R. Utilizing collaboration theory to evaluate strategic alliances. Am J Eval. 2004;25:66–77.

    Article  Google Scholar 

  41. Gagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: what we know and do not yet know. J Epidemiol Community Health. 2017;71(2):105–6.

    Article  Google Scholar 

  42. Hofmeyer A, Scott C, Lagendyk L. Researcher-decision-maker partnerships in health services research: Practical challenges, guiding principles. BMC Health Serv Res. 2012;12:280.

    Article  Google Scholar 

  43. Thackway S, Campbell D, Loppacher T. A long-term, strategic approach to evidence generation and knowledge translation in NSW. Australia Public Health Res Pract. 2017;27(1):e2711702.

    Google Scholar 

  44. Hegger I, Marks LK, Janssen SWJ, Schuit AJ, Keijsers JFM, van Oers HAM. Research for Policy (R4P): development of a reflection tool for researchers to improve knowledge utilization. Implementation Sci. 2016;11:133.

    Article  Google Scholar 

  45. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in systematic reviews: Knowledge transfer for policy and practice. J Contin Educ Health Prof. 2008;28(2):67–72.

    Article  Google Scholar 

  46. Golden-Biddle K, Reay T, Petz S, Witt C, Casebeer A, Pablo A, et al. Toward a communicative perspective of collaborating in research: the case of the researcher-decision-maker partnership. J Health Serv Res Policy. 2003;8(Suppl 2):20–5.

    Article  Google Scholar 

  47. Johnson R, Grove A, Clarke A. It’s hard to play ball: a qualitative study of knowledge exchange and silo effects in public health. BMC Health Serv Res. 2018;18:1.

    Article  Google Scholar 

  48. Cherney A, Head B. Supporting the knowledge-to-action process: a systems-thinking approach. Evid Policy. 2011;7(4):471–88.

    Article  Google Scholar 

  49. Huang TTK, Grimm B, Hammond RA. A systems-based typological framework for understanding the sustainability, scalability, and reach of childhood obesity interventions. Children's Health Care. 2011;40(3):253–66.

    Article  Google Scholar 

  50. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2010;10(Supp 1):35–48.

    Google Scholar 

  51. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81(2):221–48 171–2.

    Article  Google Scholar 

  52. Mirzoev TN, Omar MA, Green AT, Bird PK, Lund C, Ofori-Atta A, et al. Policy partnerships-experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia. Health Res Policy Syst. 2012;10:30.

    Article  Google Scholar 

  53. Jose K, Venn A, Jarman L, Seal J, Teale B, Scott J, et al. Partnering Healthy@Work: an Australian university-government partnership facilitating policy-relevant research. Health Promot Int. 2017;32(6):964–76.

    PubMed  Google Scholar 

  54. Choi BCK, Pang T, Lin V, Puska P, Sherman G, Goddard M, et al. Can scientists and policy makers work together? J Epidemiol Community Health. 2005;59(8):632.

    Article  Google Scholar 

  55. Kerr EA, Riba M, Udow-Phillips M. Helping health service researchers and policy makers speak the same language. Health Serv Res. 2015;50(1):1–11.

    Article  Google Scholar 

  56. Oxman AD, Vandvik PO, Lavis JN, Fretheim A, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP) 2: Improving how your organisation supports the use of research evidence to inform policymaking. Health Res Policy Syst. 2009;7(Suppl 1):S2.

    Article  Google Scholar 

  57. Thackway S, Campbell D, Loppacher T. A long-term, strategic approach to evidence generation and knowledge translation in NSW, Australia. Public Health Res Practice. 2017;27(1):2711702.

    Article  Google Scholar 

  58. Soper B, Hinrichs S, Drabble S, Yaqub O, Marjanovic S, Hanney S, Nolte E. Delivering the Aims of the Collaborations for Leadership in Applied Health Research and Care: Understanding their Strategies and Contributions. Health Services and Delivery Research. Southampton: NIHR Journals Library; 2015.

    Google Scholar 

  59. Stetler CB, Mittman BS, Francis J. Overview of the VA Quality Enhancement Research Initiative (QUERI) and QUERI theme articles: QUERI Series. Implementation Sci. 2008;3:8.

    Article  Google Scholar 

  60. Heller DJ, Hoffman C, Bindman AB. Supporting the needs of state health policy makers through university partnerships. J Health Polit Policy Law. 2014;39(3):667–77.

    Article  Google Scholar 

  61. Molleman G, Fransen G. Academic collaborative centres for health promotion in the Netherlands: building bridges between research, policy and practice. Fam Pract. 2012;29(Suppl 1):157–62.

    Article  Google Scholar 

Download references

Acknowledgements

Our sincere thanks to the key informants who participated in this study and generously shared their experiences and learnings.

Funding

This work was funded through a New South Wales Health EMC Fellowship held by AW. LW is supported by a NHMRC CDF II Fellowship (APP1128348).

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due to privacy concerns but may be available from the corresponding author on reasonable request.

Author information

Authors and Affiliations

Authors

Contributions

AW led this work, including contributing to the study design, conducting and analysing the key informant interviews and drafting the paper. HT analysed the interviews and contributed to drafting the paper. All authors made substantial contributions to the design of the study and the analysis and interpretation of data and were involved in critically revising the manuscript for important intellectual content. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Anna Williamson.

Ethics declarations

Ethics approval and consent to participate

The current study was approved by the University of New South Wales human Ethics Committee (HC17843).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:

Appendix 1. Partnership research key informant interviews: policy, programme or health service deliverers. (DOCX 32 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Williamson, A., Tait, H., El Jardali, F. et al. How are evidence generation partnerships between researchers and policy-makers enacted in practice? A qualitative interview study. Health Res Policy Sys 17, 41 (2019). https://doi.org/10.1186/s12961-019-0441-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-019-0441-2

Keywords