Skip to main content

Table 2 Illustrative examples of forward and backward tracing assessments, and assessments utilising both approaches

From: Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Main assessment reason Assessment start-point Conceptual framework and methods Assessment end-point/outcomes reported Comment
FORWARD TRACING ASSESSMENTS
A: Wooding et al. 2014 [140] (Australia, Canada, UK)
Understand impacts Research projects from cardiovascular and stroke research funders Payback Framework (multiple impact categories)
29 randomly selected case studies
Data sources: researcher surveys; interviews with researchers and end-users; and external peer review
Scoring of impacts for each payback category by an expert panel
Qualitative and quantitative analysis of factors associated with impact
Sum of impacts across impact categories, impact scores, plus some specific examples reported
Analysis of impact pathways with reference to existing theories and conceptual perspectives
Factors explaining variations in impact
Forward tracing, research impact assessment where single projects were the unit of analysis
B: Kok et al. 2016 [69] (Netherlands/Ghana)
Understand impacts Research projects that were part of a Ghanaian-Dutch research programme Contribution Mapping Framework (policy and practice impacts only)
30 case studies (selected in order of funding allocation)
Data sources: research proposals, mid-terms reviews and reports; interviews with researchers and end-users
Number of ‘used’ studies
Description of how produced knowledge was used
Description of research and translation processes associated with the use of produced knowledge
Forward tracing, research impact assessment where single projects were the unit of analysis
C: Hanney et al. 2013 [55] (United Kingdom)
Accountability/advocacy
Inform research funding strategies
Grants funded by Asthma UK (project grants; professional chairs; fellowship grants; collaborative research centre) Payback Framework: (multiple impact categories)
Survey of 163 researchers; 14 purposely selected case studies
Data sources: Researcher survey and interviews; archival and document review; bibliometric analysis
Sum of impacts by impact category and some specific examples reported
Comparison of impacts reported by funding mode
Forward tracing, research impact assessment where more than one type of research grant was the unit of analysis
Analysis of multiple funding modes and comparison of outcomes
D: Hanney et al. 2006 [53] (United Kingdom)
Test methods
Understand research impacts
Body of diabetes research published in the early 1980s by one team leader of acknowledged influence No framework used – broad description of multiple types of impacts
Single case study
Data sources: bibliometric analysis; surveys and interviews with researchers; critical publication pathway analysis
Description of benefits identified
Factors associated with significant impact
Methodological issues
Forward tracing, research impact assessment where a programme of research is the unit of analysis
E: Hanney et al. 2000 [51] (United Kingdom)
Test methods and model
Understand research impacts
Research and development centres funded by a regional office of the National Health Service Payback Framework (multiple impact categories)
2 purposefully selected case studies
Data sources: document analysis; bibliometric analysis; interviews with researchers and end-users
Description of impacts identified
Methodological issues
Forward tracing, research impact assessment where research centres were the unit of analysis
Used a triangulation approach, combining analysis of selected projects with the broader longer-term contribution of the centre as a whole
F: Orians et al. 2009 [98] (United States of America)
Test methods and model
Accountability
National Institute of Environmental Health Sciences (NIEHS) Division of Extramural Research asthma-related research portfolio Logic model of pathways linking research to ultimate outcomes (multiple categories of impact)
Survey of 725 researchers; interviews with 16 end-users
Sum of impacts reported by impact category and some specific examples reported
Methodological issues
Forward tracing, research impact assessment where a portfolio of research was the unit of analysis
Combined analysis of the work of researchers who had ever received NEIHS asthma research funding (over a 30-year period) with a broader analysis of awareness and use of any research from the portfolio by end-users
G: Dobbins et al. 2004 [23] (Canada)
Understand research use
Identify factors associated with use
Systematic reviews disseminated to public health decision-makers through the Effective Public Health Practice Project (EPHPP) No framework used - policy impacts only
Survey of policy-makers who were members of technical review groups
Statistical analysis of factors associated with use
Extent of systematic review use and perceived influence on recommendations
Factors explaining variations in review use.
Forward tracing, research use assessment where a group of related projects (systematic reviews) were the unit of analysis
Systematic reviews commissioned by policy agency to address priority policy questions
BACKWARD TRACING ASSESSMENTS
H: Grant et al. 2000 [41] (United Kingdom)
Test assessment method
Understand research impacts
Clinical guidelines on disease management developed in the UK No framework used
Bibliometric analysis of publications cited in 15 guidelines
Number of papers cited and type of papers cited
Research characteristics associated with citation
Backward tracing, research impact assessment where policy documents were the unit of analysis
I: Kite et al. 2014 [68] (United States of America)
Benchmark research use Documents and oral testimony associated with legislative bills relevant to active living archived by the Minnesota State Legislature No framework used
Content analysis of policy documents
Number of documents mentioning research and other types of information Backward tracing, research use assessment where policy documents were the unit of analysis
J: Dakin et al. 2016 [17] (United Kingdom)
Understand policy decisions National Institute for Heath and Clinical Excellence (NICE) guidance documents, Health Technology Assessment (HTA) reports and appeal decision reports Content analysis of 73 NICE appraisals.
Statistical analysis estimating the impact of key coded variables on decision-making
Factors associated with decision-making including availability and quality of research Backward tracing research use assessment where policy documents were the unit of analysis
K: PausJenssen et al. 2003 [103] (Canada)
Understand policy decisions Decision-making process of the Drug Quality and Therapies Committee (DQTC) of Ontario No framework used
Single case study – committee meetings between Dec 1997 and Aug 1998
Data sources: interviews with committee members; observation of committee meetings
Qualitative analysis of factors associated with decision-making
Factors associated with decision-making including role of research Backward tracing research use assessment where a committee was the unit of analysis
L: Williams et al. 2008 [137] (United Kingdom)
Understand research use in policy decisions Technology appraisal decisions made by the NICE Technology Appraisal Committee and resource allocation decisions concerning adoption of drugs and other therapies made by four local national health service committees 5 case studies of committees: 4 local and one national organisation
Data sources: Documentary analysis; observation of committee meetings; committee member workshop discussions and interviews
Prospective data collection
Description of extent and nature of use of economic analyses in decision-making
Factors associated with the use of research evidence relating to economic analyses
Backward tracing research use assessment where committees were the unit of analysis
Compared decision-making at a national and local level
M: Shearer et al. 2014 [118] (Burkino Faso)
Understand research use in policy decisions Community integrated management of childhood illness; home management of malaria; removal of user fees for antiretroviral treatment for HIV No framework used
3 policy case studies
Data sources: Surveys with policy actors
Network analysis. Statistical analysis of probabilities of research provision and request between actors and actors use of research to inform policy
Conditions under which research is provided and requested
Factors associated with research use in policy-making
Backward tracing research use assessment where policy processes were the unit of analysis
N: Nabyonga-Orem et al. 2014 [91] (Uganda)
Understand research use in policy decisions Change in malaria drug treatment policy and its implementation in Uganda No framework used
Single case study
Data sources: interviews with policy actors and document review
Respondents rated degree of consistency between the policy decision and the available evidence
Description of the use of research and other information in the policy decision and by different actors
Type and quantity of research cited in policy documents
Factors facilitating the uptake of research
Backward tracing research use assessment where a policy process was the unit of analysis
O: Hyde et al. 2015 [61] (United States of America)
Understand research use in policy decisions Development of state-level policies to ensure that youth in foster care receive safe and appropriate psychopharmacological treatment Used an evidence framework for understanding the different types, applicability and uses of evidence to inform policy decisions
Single case study based on interviews with 72 decision-makers from 50 states
Description of research use by phase of policy development, types of research/other information used and how research was used Backward tracing research use assessment where policy processes were the unit of analysis
Compared use of global and local knowledge
P: Hutchinson et al. 2011 [60] (Malawi, Uganda, Zambia)
Understand research use in policy decisions Development of National treatment guidelines for HIV positive TB patients Overseas Development Institute RAPID Framework (analysis of process; context; evidence and links)
Policy case studies from 3 countries
Data sources: interviews with policy stakeholders; document analysis
Description of key research and policy events
Explanation of the uptake and use of research based on context, evidence and links
Backward tracing research use assessment where policy processes were the unit of analysis
Used an across country comparison to examine how context influences policy development
Q: Lavis et al. 2003 [74] (Canada)
Understand research use in policy processes Development of health service policies in 2 Canadian provinces No framework used
8 policy case studies (stratified sampling)
Data sources: policy-maker interviews; document analysis; survey of research unit-directors (identify local research that was available)
Number of policies in which citable research/other information was used, stage of policy development it was used and examples of how it was used
Ways in which policy-makers accessed research
Backward tracing research use assessment where policy processes were the unit of analysis
Stratified policy selection by policy type and location
ASSESSMENTS USING ELEMENTS OF FORWARD AND BACKWARD TRACING APPROACHES
R: Bunn et al. 2011 [11] (United Kingdom)
Understand research impact Nurse home visiting research conducted in the UK
UK policy documents relevant to home visiting
Adapted Research Impact Framework (policy impact only)
Data sources: content analysis of policy documents; citation analysis of key pieces of research; interviews with prominent researchers about the impacts of United Kingdom home visiting research
Publications cited in policy documents and type of research cited
Described examples of policy impact by levels of policy-making, type of policy and nature of policy impact
Backwards and forwards tracing elements
Analysis of policy documents compared to information from researchers and citation analysis of research outputs Described by authors as a research impact assessment
S: Morton 2015 [88] (United Kingdom)
Understand research impact Research project conducted by the Centre for Research on Families and Relationships and a voluntary organisation (ChildLine Scotland)
Development of an alcohol policy at the Scottish Government level
Research Contribution Framework (steps/process of research impact on policy)
Single case study
Data sources: policy document analysis and policy-maker interviews (policy analysis); interviews/surveys with research partners, end-users and dissemination activity participants/target audience (trace researcher activities and impacts)
Description of the activities and events that led to research impact
Description of impacts
Effect of context on research impact
Includes backwards and forwards tracing elements
Research project is the primary unit of analysis
Data from policy analysis triangulated with forward tracing elements of the study
Described by author as a research impact assessment
T: De Goede et al. 2012 [19] (Netherlands)
Understand research use in policy processes Local epidemiological research reports published as Local Health Messages
Development of local health memoranda
Framework consisting of the research and local health policy context and networks, types of research utilisation, explanations of research use
Case studies of the development of Local Health Memoranda in 3 municipalities
Data sources: interviews with researchers and key policy actors; survey of other actors; policy document analysis; meeting observation
Prospective data collection
Describe process of producing local health messages (research) and local health memorandum (policy)
Describe influence of policy-makers beliefs/characteristics on research use
Describe the interface between local epidemiologists and local policy actors to explain research use
Includes backwards and forwards tracing elements
Focus on interface between development of a specific research output and a related policy