Table 1 A qualitative description of the most widespread frameworks for the evaluation of research impact

1. Payback

Organizes together in a sequential and systematic way the different aspects on the impact of research projects from dissemination to potential benefits for health care

i) knowledge production;

ii) research targeting and capacity building;

iii) informing policies and product development;

iv) health benefit;

v) broader economic benefits.

i) journal articles, conference presentations, books, research reports, other disseminative material;

ii) new research lines or know how, career promotions, PhD, masters;

iii) guidelines and documents addressing policies citations, membership of decision panels;

iv) health outcomes, QALY, savings for health care systems

v) benefits in occupation and economic development, productivity

Research programs financed by the NHS UK [12, 38, 40, 44]

2. Research Impact

Evaluates the influence of research results and of the potential concurrent/competing factors (cultural context, policy content, decision process) in policy making

i) research related impact

ii) policy impact

iii) service impact

iv) societal impact

i) knowledge and methodology advancements, networking, leadership

ii) establishment of collaborations and networks

iii) evidence-based practice; cost-containment, quality of care;

iv) knowledge, attitudes and behaviour, health literacy, social capital, equity, macroeconomy

London School of Hygiene and Tropical Medicine researchers evaluation [45]

3. Research utilization ladder

Evaluates the ways in which research progresses towards its application by practitioners and policy makers

i) transmission (of research results to practitioners and policy makers)

ii) cognition (reading and understanding)

iii) reference (quoting of research results in reports, studies, actions)

iv) effort (to adopt research results)

v) influence (on choices and decisions)

vi) application

-

[46]

4. Lavis decision-making impact model

Evaluates the impact on decision making of any individual or organisation, considering the target audience of research and the resources available for the assessment

i) policy makers are the ones seeking research (user-pull)

ii) researchers actively disseminating results (producer-pull)

iii) researchers and policy-makers are both involved actively (exchange measures)

Process measures (if limited resources are available)

Intermediate outcome measures (by performing surveys)

Outcome measures (by performing cases studies)

Canada

[11]

5. Weiss Logic Model

Analyzes the ratio between input (resources), process (activity) and results of research (products)

i) initial benefits

ii) intermediate benefits

iii) long-term benefits

Output: publications

i) awareness of medical research results in policy making

ii) any changes in practice

iii) any changes in well-being and health

[47]

6. HTA Organisation assessment framework

Effectiveness is measured by the ability to impact on decision making

i) productivity

ii) capacity to attract and maintain resources and to mobilize external support

iii) culture and values maintenance (independence in action, transparency of the process, accountability to stakeholders)

i) volume and productivity of outputs, quality, comprehensiveness, and accessibility

ii) measure of visibility and credibility

Quebec Council on Health Care Technology, Canada [48]

7. Societal Impact framework

Research is considered as the valuation of the communication of research groups with relevant surroundings (industry, general public, scientific community, public and policy institutions)

i) knowledge products

ii) knowledge exchange and esteem

iii) knowledge use

iv) attractiveness

i) publications, patents, products

ii) presentations, consultancies, and public lectures

iii) citations, product use

iv) further funding

Royal Netherlands Academy of Arts and Sciences 2002 [49, 50]

8. Balanced scorecard

Measures performance and drives organizational strategy by incorporating organisational aspects together with financial performance

i) financial;

ii) customers;

iii) business process;

iv) learning and growth.

-

[51]

Ontario University Health Network [52]

9. Research Assessment Exercise (RAE)

To produce quality profiles for each submission of research activity made by UK academic institutions

Three quality profiles are defined (panel decides the weight given to each profile):

i) research output (minimum 50%);

ii) research context (minimum 5%);

iii) other indicators (minimum 5%).

i) RAE1: staff information (volume and type of contracts, external collaborators), analysis of funding for research fellows;

ii) RAE2: research output (publications, patents, reports, database, software, etc);

iii) RAE3: research scholarships

iv) RAE4: attractiveness for external funding

v) RAE5a: further information on groups of research (networking, development of a research culture, etc)

RAE 2008 [53]

10. Cost-benefit Analysis

Research impact evaluated in monetary terms

i) savings for health care systems (direct costs)

ii) savings for the community on the whole (indirect costs)

i) QALY

ii) profits

NIH, USA

[54]

National Institute of Health 1993 [55]