Table 2 The 174 method-based indicators to evaluate organisational knowledge brokers
From: Indicators to evaluate organisational knowledge brokers: a scoping review
1. General climate | |
Output indicators | |
1.1 Number of activities identified 1.2 Availability of synthesised and packaged evidence 1.3 The organisation has the skills, structures, processes and a culture to promote and use research findings in decision-makinga 1.4 Feedback on context/culture 1.5 Facilitators, barriers, lessons learnt | |
Outcome indicators | |
1.6 Increased demand or value of KT products or knowledge from policy-makers 1.7 Number of times evidence is mentioned in policy/parliamentary discussions 1.8 Increased awareness of importance of EIP initiatives 1.9 Changes in government allocated funding | |
2. Production of research | |
Output indicators | |
2.1. Number of peer-reviewed journal articles 2.2. Citations per article 2.3. Citation of research results by other researchers 2.4. Journal impact factor 2.5. Number of projects per research approach 2.6. Funds invested per project 2.7. Project duration 2.8. Number of projects liaising with users 2.9. Number of projects that led to subsequent research 2.10. Researcher feedback on project alignment with priorities 2.11. Mean score of scientific accuracy | 2.12. Mean score of readability 2.13. Mean score of usability 2.14. Mean score of ease of access 2.15. Applicability of research for decision making 2.16. Developed priority report (i.e. research agenda, list of priorities, country assessment) 2.17. Revision with stakeholders 2.18. Feedback on support and/or awareness 2.19. Feedback on priority development 2.20. Feedback on priorities |
Outcome indicators | |
2.21. Changes in policies or programmes consistent with evidence produced 2.22. Policy-makers, stakeholders and researchers report that relevant and understandable health research evidence is more readily available and cite this research evidence in media | |
3. KT activities: push efforts | |
Output indicators | |
3.1. Number of downloads 3.2. Number of page visits (total and unique) 3.3. Number of countries visiting the website 3.4. Number of page views per visit 3.5. Number of requests for materials 3.6. Extent of media exposure 3.7. Referrals made to distributed materials 3.8. Number of materials distributed 3.9. Transmitted to relevant stakeholder (discussed at policy dialogues, dissemination workshops) 3.10. Disseminated materials are read and understood 3.11. Efforts have been made to adopt the disseminated knowledge 3.12. Platform survey responses 3.13. Usage analytics of promotional products 3.14. Research is presented to decision-makers in a useful waya 3.15. Multiple formats of written and/or other forms of presentation (e.g. newsletter, website summary, interim report, oral presentation) 3.16. Presentation formats include layman’s terms and recommendations 3.17. Where appropriate, presentation formats are concise (e.g. less than two pages) | 3.18. Users contacted researchers to discuss results 3.19. Relevant documents disseminated in hardcopy 3.20. Website or online evidence database is established 3.21. Number of dissemination workshops 3.22. Percentage of grantees presenting at conferences 3.23. Percentage of grantees submitting work for publication 3.24. Percentage of grantees with published research at time of review 3.25. Feedback from grantees on competence and opportunities for dissemination 3.26. Number/amount of grant (applications) 3.27. Number of researcher internships 3.28. Number of trainees publishing research 3.29. Feedback on improved quality of research results 3.30. Percentage of research applications headed by a national 3.31. Increased interest by young nationals in research 3.32. Establishment of a PhD programme 3.33. Number of projects supported |
Outcome indicators | |
3.34. Number of project findings used/expected to be used in policy 3.35. Number of projects leading to/expecting to change behaviour 3.36. Increase in inquiries and applications 3.37. Phasing out of external funding | |
4. KT activities: pull efforts | |
Output indicators | |
4.1. Seeking, Engaging with, and Evaluating Research (SEER)a 4.2. Organisational Research Access, Culture and Leadership (ORACLe)a | |
Outcome indicators | |
4.3. Staff Assessment of enGagement with Evidence (SAGE)a | |
5. KT activities: exchange efforts | |
Output indicators | |
5.1. Grants for collaboration 5.2. Research projects are produced with policy-makers 5.3. Disciplinary backgrounds of contributing authors 5.4. Invitations to publish special issues 5.5. Partners views on using research results 5.6. Negotiation occurs during the research process 5.7. Negotiated items are clearly understood by all 5.8. Deciding on objectives together 5.9. Built mutual trust 5.10. Communication tools established 5.11. Sharing of information and responsibility 5.12. Transparency 5.13. Share profits equally 5.14. Build on achievements 5.15. Communication is clear 5.16. Communication is relevant 5.17. Communication is timely 5.18. Communication is respectful 5.19. Density and centrality 5.20. Connectedness of networks 5.21. Partners mention each other 5.22. Partners are flexible about meeting partner’s changing needs and revising research plans and timelines 5.23. Partners understand the limits of each other’s flexibility 5.24. Partners understand research findings, their limits and their implications for Ministry work 5.25. Conflict is dealt with openly, informally and promptly | 5.26. Trust has increased between partners 5.27. Comfort has increased between partners 5.28. Openness has increased between partners 5.29. Partners begin speaking a common language regarding research 5.30. Partners facilitate removal of barriers for each other’s work 5.31. Linkage with partner enhances partner linkage with community/other stakeholders 5.32. There is joint commitment to the research project 5.33. There is an increase in joint activity around the project 5.34. Clear leadership of partnerships 5.35. Team mentality 5.36. Early engagement of members 5.37. Number of members 5.38. Number/percentage of members present at activities 5.39. Level of engagement 5.40. Number/percentage of partners active 5.41. Member affiliation and profession 5.42. Joint meetings occur at most stages of research 5.43. Joint meetings occur to discuss research dissemination and utilisation plans 5.44 Feedback on linkage and exchange mechanisms 5.45. Number of partners involved in KT activities 5.46. Stakeholders involved |
Outcome indicators | |
5.47. Partners are perceived as experts in the research/policy area and are referred to as such to others 5.48. Value of network 5.49. Feedback on awareness and perceptions of network 5.50. Partnerships are built and sustained | |
6. KT activities: integrated efforts | |
Output indicators | |
6.1. Number of KTPs viewing their work as a long-term initiative 6.2. Number of KTPs engaging in priority-setting with stakeholders 6.3. Number of KTPs building capacity for priority-setting 6.4. Number of KTPs producing/in process of KT products (by type, e.g. evidence briefs, clearinghouses, rapid response services, deliberative dialogues, systematic reviews) 6.5. Number of KTPs that built capacity for KT (evidence briefs; deliberative dialogues; accessing, assessing and using research evidence) 6.6. Number of KTPs training research users in KT (systematic reviews, evidence briefs, deliberative dialogues) 6.7. Number of organisations using the products 6.8. Functional website or clearinghouse providing KT resources 6.9. Amount of resources utilised in knowledge brokering activities (e.g. cost, time, materials) 6.10. Number of KT materials 6.11. Products (e.g. website, policy dialogues, evidence briefs) aligned with and address priorities | 6.12. Topic of KT materials 6.13. Number of KT materials translated/available in different languages 6.14. Policy dialogues about high-priority policy issues take place regularly 6.15. Scoring of quality dimensions (mean, standard deviation) 6.16. KT activities regarded as beneficial for bringing together stakeholders and facilitating the development of partnerships |
Outcome indicators | |
6.17. Uptake and/or influence of evidence (reports, policy briefs, recommendations, other) in decision-making 6.18. Financial and organisational support to the KTP | |
7. Evaluation | |
Output indicators | |
7.1. Number of KTPs evaluating KT product(s) quality 7.2. Perception of sustainability (no outcome indicators identified) | |
8. Capacity-building | |
Output indicators | |
8.1. Number of activities 8.2. Type of activity 8.3. Number of people invited 8.4. Number of people attended 8.5. Number of people trained 8.6. Reasons for participation non completion 8.7. Participant occupation 8.8. Participant affiliation 8.9. Participant education level 8.10. Participant gender 8.11. Participant age 8.12. Participant’s number of years in current position 8.13. Participant’s level of policy influence 8.14. Country of participants 8.15. Participant’s experience with evidence-informed policy-making 8.16. Training workshops for policy-makers and researchers are designed and implemented regularly 8.17. Mean programme ratings and feedback 8.18. Intent to return 8.19. Survey response rate 8.20. Percentage increase in pre/post scores of skill abilities (e.g. access evidence, synthesise evidence, policy dialogues, evidence briefs, collaboration etc.) and value of knowledge use | 8.21. Comments in the media reflect capacity changes 8.22. Ability to acquire research 8.23. Increased research capacity 8.24. Change in research/policy-maker relationship 8.25. Comments in the media reflect relationship changes 8.26. Perceived EIP skills/changes in skills (acquire, assess, adapt, apply)a 8.27. Number of participants reporting benefits 8.28. Awareness of key government documents 8.29. Perceived change in skills and confidence to interact with experts 8.30. Perceived impact on current position and/or future career advancementa 8.31. Contribution to decision-making by partners and policy-makers |
Outcome indicators | |
8.32. Feedback on behavioural changes 8.33. Number/percentage of trainees reporting intent to use skills gained | |