Table 2 Description of possible impact categories and relative indicators (adapted from Canadian Academy of Health Science [20])
Impact category | Proposed Indicators | Data collection methodology | Level of application | Theoretical Models quoting this category | Advantages | Disadvantages |
|---|---|---|---|---|---|---|
Advancing knowledge | • Activity (number of peer-reviewed publications absolute or relative - e.g. to the department publications); • Quality (impact factor, relative citation impact, high impact publications, download numbers); • Outreach (co-author analysis, field analysis of citations); • Context and structure (relative activity index); • Other possible indicators (expanded relative citation impact, relative download rate). | Bibliometric and citational analysis; desk analysis. | Basic, clinical, applied and social research. Assessment of individual researchers, teams, institution, projects, funding agencies. | Payback, research impact, research utilization ladder, Lavis decision-making impact model, societal impact, RAE | -wide range of applications -close to research itself -objective -attributed relatively straightforwardly -accessibility and feasibility -limited cost | -surrogate indicators; -not always relevant; -not comparable across different disciplines; -robust only if based on a sufficient set of publications (not fully appropriate to individual researcher evaluation). |
Capacity building | • Staff (number of PhD, Master, researchers, member of staff); • Funding (external sources of funding); • Infrastructure (grant for infrastructure and coordination activities); • Other possible indicators (receptor and absorptive capacity) | Desk analysis, database and interviews | Basic, clinical, applied and social research Assessment of teams, institution, projects, funding agencies. Not recommended at individual level. | Payback, research impact, research utilization ladder, Lavis decision-making model societal impact, RAE | -wide range of applications -quite close to research itself -attributed relatively straightforwardly (partially) -accessibility and feasibility -limited cost | --surrogate indicators; -quite subjective; -attribution issues; -self referenced; -non mutually exclusive (double counting) |
Informing policies and product development | • Health care (guidelines and policy documents citations - e.g. regional plans, educational material, panel representatives) • Research (references used as background for successful funding proposals, consulting and support activity, curricula citations) • Industrial (patents and industrial collaboration, clustering) • Citizens (informative packages, dissemination activities) • Media (journals, radio, tv, web) | Desk analysis, database and interviews | Clinical, applied, social research Assessment of individual researchers, teams, institution, projects, funding agencies. | Payback, research impact, research utilization ladder, Lavis decision-making model, societal impact | -Optimal for projects funded ad hoc to inform decision making -robustness -relevance -feasibility -limited cost | -limited spectrum of application -time-lag between input (research) and output (result) --quite subjective -self referenced |
Health and health sector benefits | • Health (Epidemiologic data, incidence, prevalence, mortality QALYs1, PROMs2) • Health determinants (risk factors, educational and social level of cohesion, pollution) • System (patient satisfaction, waiting lists, compliance and adherence to clinical guidelines, hospitalization, length of inpatient stay, adverse effects/complications) | Desk analysis, database and interviews Case studies, audit | Clinical and applied research Evaluation of teams, institutions, projects, funding agencies | Payback, research impact, research utilization ladder | -robustness -relevance | -many confounders -feasibility and cost of data collection; -time-lag between input (research) and output (result) -possible underestimation of real impact -attribution issue |
Economic and social benefits | • Economic rent (salaries, employments) • Licensing returns • Product sales revenues • Spin-off companies • Health benefit (QALY and PROM per health care dollar) • Well-being (happiness, level of social isolation) • Social benefits (socio-economic social) | Desk analysis, database and interviews Case studies, audit Econometrics | Clinical and applied research Evaluation of teams, institutions, projects, funding agencies | Payback, societal impact, cost-benefit. | -robustness -relevance | -many confounders -feasibility and cost of data collection; -time-lag between input (research) and output (result) -possible underestimation of real impact -attribution issue -use of models and assumptions |