Table 1 Comparator research impact frameworks and approaches

The Payback Framework [10]^a

This framework is one of the most widely used in research impact assessments and informs many of the newer approaches. The framework consists of a logic model of the seven stages of research from conceptualisation to impact, and five categories of impacts, called ‘paybacks’: knowledge production, research targeting and capacity-building, informing policy and product development, health and health sector benefits, and broader economic benefits. The framework incorporates various feedback loops connecting the stages.

Health Services Research Impact Framework [17]

This framework was developed in an Australian primary healthcare context and identifies four ‘broad areas of impact’: research-related impact (advancing knowledge), policy impact (informing decision-making), service impact (improving health and health systems), and societal impact (creating broad social and economic benefit). Against each broad area, the framework lists specific areas of impact, reach into different audiences, and whether impact involves ‘producer push’ dissemination or ‘producer pull’ uptake.

Canadian Academy of Health Sciences Impact Framework [9]^a

This framework adapts the Payback Framework (above) into a ‘systems approach’ to capture impacts in five main categories: advancing knowledge, research capacity-building, informing decision-making, health impacts, and broad economic and social impacts. Each main category consists of subcategories containing lists of possible indicators, which can be used to track impacts within the four ‘pillars’ of health research: basic biomedical, applied clinical, health services and systems, and population health. The framework also allows tracking of impact at individual, institutional, provincial, national or international levels.

Impact Assessment Tool [18]

This framework groups different types of impacts into ‘four levels of impact’ with subcategories: scholarly outputs (publications and citations, research funding, capacity-building, journal impact factor), translational outputs (plain language summaries and media engagement, formal knowledge exchange processes, lobbying government ministers or departments, intervention packaged for implementation), policy or practice impacts (changes to practice, changes to services, policy change, commercialisation), and long-term population outcomes (behaviour change and health outcomes, social outcomes, economic outcomes).

Alberta Innovates-Health Solutions Research to Impact Framework [22]

This framework adopts the same impact categories as the Canadian Academy of Health Sciences framework above but additionally highlights ‘inputs’ (involving stakeholder engagement and evidence to inform planning and investment strategies). The framework also incorporates performance management concepts in the form of ‘balanced scorecard’ categories (financial, enablers, internal processes and stakeholder) and logic model categories (inputs, activities, outputs, reach, short term outcomes, medium-term outcomes and long-term outcomes). These categories are mapped against the subcategories of organisational performance, research and innovation outcomes, informing decision-making, health and wellbeing, and broader economic impacts.

Realist evaluation [8]^a

Rather than representing or offering an impact framework, realist evaluation is presented in the narrative review by Greenhalgh et al. [8] as an approach that holds future promise in research impact assessment. Realist evaluation addresses the question: what works, for whom, and in what circumstances? [19] Realist evaluation models highlight the role of context in influencing outcomes.

SPIRIT Action Framework [23]^a

The purpose of this framework is to “guide action including the identification of where, how and what should be done to help agencies improve the use of research in their work”. While the framework uses a logic model structure as in many other impact frameworks, its focus is more on the receiving organisation’s need for research rather than on the research itself. The framework commences with catalysts for research use, and follows through to capacity, research engagement actions and outcome. ‘Outcome’ is divided into outcomes associated with research use, research-informed health policies and policy documents, and finally with better health systems and health outcomes. The framework also proposes hypotheses to enable the examination of whether the upstream components of the logic model result in changes further downstream.