Table 2 Illustrative examples of forward and backward tracing assessments, and assessments utilising both approaches
Main assessment reason | Assessment start-point | Conceptual framework and methods | Assessment end-point/outcomes reported | Comment |
|---|---|---|---|---|
FORWARD TRACING ASSESSMENTS | ||||
A: Wooding et al. 2014 [140] (Australia, Canada, UK) | ||||
Understand impacts | Research projects from cardiovascular and stroke research funders | Payback Framework (multiple impact categories) 29 randomly selected case studies Data sources: researcher surveys; interviews with researchers and end-users; and external peer review Scoring of impacts for each payback category by an expert panel Qualitative and quantitative analysis of factors associated with impact | Sum of impacts across impact categories, impact scores, plus some specific examples reported Analysis of impact pathways with reference to existing theories and conceptual perspectives Factors explaining variations in impact | Forward tracing, research impact assessment where single projects were the unit of analysis |
B: Kok et al. 2016 [69] (Netherlands/Ghana) | ||||
Understand impacts | Research projects that were part of a Ghanaian-Dutch research programme | Contribution Mapping Framework (policy and practice impacts only) 30 case studies (selected in order of funding allocation) Data sources: research proposals, mid-terms reviews and reports; interviews with researchers and end-users | Number of ‘used’ studies Description of how produced knowledge was used Description of research and translation processes associated with the use of produced knowledge | Forward tracing, research impact assessment where single projects were the unit of analysis |
C: Hanney et al. 2013 [55] (United Kingdom) | ||||
Accountability/advocacy Inform research funding strategies | Grants funded by Asthma UK (project grants; professional chairs; fellowship grants; collaborative research centre) | Payback Framework: (multiple impact categories) Survey of 163 researchers; 14 purposely selected case studies Data sources: Researcher survey and interviews; archival and document review; bibliometric analysis | Sum of impacts by impact category and some specific examples reported Comparison of impacts reported by funding mode | Forward tracing, research impact assessment where more than one type of research grant was the unit of analysis Analysis of multiple funding modes and comparison of outcomes |
D: Hanney et al. 2006 [53] (United Kingdom) | ||||
Test methods Understand research impacts | Body of diabetes research published in the early 1980s by one team leader of acknowledged influence | No framework used – broad description of multiple types of impacts Single case study Data sources: bibliometric analysis; surveys and interviews with researchers; critical publication pathway analysis | Description of benefits identified Factors associated with significant impact Methodological issues | Forward tracing, research impact assessment where a programme of research is the unit of analysis |
E: Hanney et al. 2000 [51] (United Kingdom) | ||||
Test methods and model Understand research impacts | Research and development centres funded by a regional office of the National Health Service | Payback Framework (multiple impact categories) 2 purposefully selected case studies Data sources: document analysis; bibliometric analysis; interviews with researchers and end-users | Description of impacts identified Methodological issues | Forward tracing, research impact assessment where research centres were the unit of analysis Used a triangulation approach, combining analysis of selected projects with the broader longer-term contribution of the centre as a whole |
F: Orians et al. 2009 [98] (United States of America) | ||||
Test methods and model Accountability | National Institute of Environmental Health Sciences (NIEHS) Division of Extramural Research asthma-related research portfolio | Logic model of pathways linking research to ultimate outcomes (multiple categories of impact) Survey of 725 researchers; interviews with 16 end-users | Sum of impacts reported by impact category and some specific examples reported Methodological issues | Forward tracing, research impact assessment where a portfolio of research was the unit of analysis Combined analysis of the work of researchers who had ever received NEIHS asthma research funding (over a 30-year period) with a broader analysis of awareness and use of any research from the portfolio by end-users |
G: Dobbins et al. 2004 [23] (Canada) | ||||
Understand research use Identify factors associated with use | Systematic reviews disseminated to public health decision-makers through the Effective Public Health Practice Project (EPHPP) | No framework used - policy impacts only Survey of policy-makers who were members of technical review groups Statistical analysis of factors associated with use | Extent of systematic review use and perceived influence on recommendations Factors explaining variations in review use. | Forward tracing, research use assessment where a group of related projects (systematic reviews) were the unit of analysis Systematic reviews commissioned by policy agency to address priority policy questions |
BACKWARD TRACING ASSESSMENTS | ||||
H: Grant et al. 2000 [41] (United Kingdom) | ||||
Test assessment method Understand research impacts | Clinical guidelines on disease management developed in the UK | No framework used Bibliometric analysis of publications cited in 15 guidelines | Number of papers cited and type of papers cited Research characteristics associated with citation | Backward tracing, research impact assessment where policy documents were the unit of analysis |
I: Kite et al. 2014 [68] (United States of America) | ||||
Benchmark research use | Documents and oral testimony associated with legislative bills relevant to active living archived by the Minnesota State Legislature | No framework used Content analysis of policy documents | Number of documents mentioning research and other types of information | Backward tracing, research use assessment where policy documents were the unit of analysis |
J: Dakin et al. 2016 [17] (United Kingdom) | ||||
Understand policy decisions | National Institute for Heath and Clinical Excellence (NICE) guidance documents, Health Technology Assessment (HTA) reports and appeal decision reports | Content analysis of 73 NICE appraisals. Statistical analysis estimating the impact of key coded variables on decision-making | Factors associated with decision-making including availability and quality of research | Backward tracing research use assessment where policy documents were the unit of analysis |
K: PausJenssen et al. 2003 [103] (Canada) | ||||
Understand policy decisions | Decision-making process of the Drug Quality and Therapies Committee (DQTC) of Ontario | No framework used Single case study – committee meetings between Dec 1997 and Aug 1998 Data sources: interviews with committee members; observation of committee meetings Qualitative analysis of factors associated with decision-making | Factors associated with decision-making including role of research | Backward tracing research use assessment where a committee was the unit of analysis |
L: Williams et al. 2008 [137] (United Kingdom) | ||||
Understand research use in policy decisions | Technology appraisal decisions made by the NICE Technology Appraisal Committee and resource allocation decisions concerning adoption of drugs and other therapies made by four local national health service committees | 5 case studies of committees: 4 local and one national organisation Data sources: Documentary analysis; observation of committee meetings; committee member workshop discussions and interviews Prospective data collection | Description of extent and nature of use of economic analyses in decision-making Factors associated with the use of research evidence relating to economic analyses | Backward tracing research use assessment where committees were the unit of analysis Compared decision-making at a national and local level |
M: Shearer et al. 2014 [118] (Burkino Faso) | ||||
Understand research use in policy decisions | Community integrated management of childhood illness; home management of malaria; removal of user fees for antiretroviral treatment for HIV | No framework used 3 policy case studies Data sources: Surveys with policy actors Network analysis. Statistical analysis of probabilities of research provision and request between actors and actors use of research to inform policy | Conditions under which research is provided and requested Factors associated with research use in policy-making | Backward tracing research use assessment where policy processes were the unit of analysis |
N: Nabyonga-Orem et al. 2014 [91] (Uganda) | ||||
Understand research use in policy decisions | Change in malaria drug treatment policy and its implementation in Uganda | No framework used Single case study Data sources: interviews with policy actors and document review Respondents rated degree of consistency between the policy decision and the available evidence | Description of the use of research and other information in the policy decision and by different actors Type and quantity of research cited in policy documents Factors facilitating the uptake of research | Backward tracing research use assessment where a policy process was the unit of analysis |
O: Hyde et al. 2015 [61] (United States of America) | ||||
Understand research use in policy decisions | Development of state-level policies to ensure that youth in foster care receive safe and appropriate psychopharmacological treatment | Used an evidence framework for understanding the different types, applicability and uses of evidence to inform policy decisions Single case study based on interviews with 72 decision-makers from 50 states | Description of research use by phase of policy development, types of research/other information used and how research was used | Backward tracing research use assessment where policy processes were the unit of analysis Compared use of global and local knowledge |
P: Hutchinson et al. 2011 [60] (Malawi, Uganda, Zambia) | ||||
Understand research use in policy decisions | Development of National treatment guidelines for HIV positive TB patients | Overseas Development Institute RAPID Framework (analysis of process; context; evidence and links) Policy case studies from 3 countries Data sources: interviews with policy stakeholders; document analysis | Description of key research and policy events Explanation of the uptake and use of research based on context, evidence and links | Backward tracing research use assessment where policy processes were the unit of analysis Used an across country comparison to examine how context influences policy development |
Q: Lavis et al. 2003 [74] (Canada) | ||||
Understand research use in policy processes | Development of health service policies in 2 Canadian provinces | No framework used 8 policy case studies (stratified sampling) Data sources: policy-maker interviews; document analysis; survey of research unit-directors (identify local research that was available) | Number of policies in which citable research/other information was used, stage of policy development it was used and examples of how it was used Ways in which policy-makers accessed research | Backward tracing research use assessment where policy processes were the unit of analysis Stratified policy selection by policy type and location |
ASSESSMENTS USING ELEMENTS OF FORWARD AND BACKWARD TRACING APPROACHES | ||||
R: Bunn et al. 2011 [11] (United Kingdom) | ||||
Understand research impact | Nurse home visiting research conducted in the UK UK policy documents relevant to home visiting | Adapted Research Impact Framework (policy impact only) Data sources: content analysis of policy documents; citation analysis of key pieces of research; interviews with prominent researchers about the impacts of United Kingdom home visiting research | Publications cited in policy documents and type of research cited Described examples of policy impact by levels of policy-making, type of policy and nature of policy impact | Backwards and forwards tracing elements Analysis of policy documents compared to information from researchers and citation analysis of research outputs Described by authors as a research impact assessment |
S: Morton 2015 [88] (United Kingdom) | ||||
Understand research impact | Research project conducted by the Centre for Research on Families and Relationships and a voluntary organisation (ChildLine Scotland) Development of an alcohol policy at the Scottish Government level | Research Contribution Framework (steps/process of research impact on policy) Single case study Data sources: policy document analysis and policy-maker interviews (policy analysis); interviews/surveys with research partners, end-users and dissemination activity participants/target audience (trace researcher activities and impacts) | Description of the activities and events that led to research impact Description of impacts Effect of context on research impact | Includes backwards and forwards tracing elements Research project is the primary unit of analysis Data from policy analysis triangulated with forward tracing elements of the study Described by author as a research impact assessment |
T: De Goede et al. 2012 [19] (Netherlands) | ||||
Understand research use in policy processes | Local epidemiological research reports published as Local Health Messages Development of local health memoranda | Framework consisting of the research and local health policy context and networks, types of research utilisation, explanations of research use Case studies of the development of Local Health Memoranda in 3 municipalities Data sources: interviews with researchers and key policy actors; survey of other actors; policy document analysis; meeting observation Prospective data collection | Describe process of producing local health messages (research) and local health memorandum (policy) Describe influence of policy-makers beliefs/characteristics on research use Describe the interface between local epidemiologists and local policy actors to explain research use | Includes backwards and forwards tracing elements Focus on interface between development of a specific research output and a related policy |