Table 2 ORACLe Interview Questions and Marking Guide

1. Does your organisation have documented processes for how policies should be developed?

Domain 1: Documented processes to develop policy that encourage or mandate the use of research

There are standard, written guidance that describe how policies should be developed and these are organisation-specific.

There are documented processes for some aspects of policy development but not all, not at a very high level, and with little detail.

There are no documented processes.

2. Do these processes encourage or require staff to use research in policy development?

Domain 1: Documented processes to develop policy that encourage or mandate the use of research

The requirement to use research must be explicitly and unequivocally noted in agency documentation (either as a requirement or encouraged), and must include how to use research as well as a requirement that it should be used.

Research use is implied but not explicitly encouraged or required in relevant documentation or if the documentation does not include how research should be used.

N/A if no above.

No if there are documented processes but they do not refer to the use of research. Or, if there is a ‘culture’ or assumption of research use but it is not mandated or encouraged in relevant documentation.

It is implied, if it refers to “supporting evidence” in general, rather than research evidence specifically.

3. Are programs available for leaders to improve their confidence or expertise in use of research in policy-making?

Domain 2: Tools and programs to assist leaders of the organisation to actively support the use of research in policy and program development

Must specifically target leaders (rather than programs for all staff, including leaders).

Programs for everyone but includes leaders.

There are no programs that leaders would attend or that are specifically for leaders.

(Leaders mean any level of executive or management, or anyone else with a formal or informal leadership role.)

These programs should be offered regularly, that is, at least once a year.

OR Has one-off or occasional programs for leaders only.

4. Do the position descriptions or performance management systems for senior policy makers in your organisation cover expertise in use of research in policy-making?

Domain 2: Tools and programs to assist leaders of the organisation to actively support the use of research in policy and program development

The expertise in use of research must be explicit and in most senior policy-makers’ PDs or similar. (policy makers not senior staff generally)

The position descriptions of senior policy makers – might refer to expertise that implies using research but is not explicit.

There is no reference to the use of research in position descriptions of senior policy makers.

OR the expertise in use of research, although explicit, is present in only a few of the senior policymakers’ PDs

5. In the last 6 months, have leaders of your organisation referred to research in their internal communication (e.g. newsletters, bulletins, updates, tweets, etc.)?

Domain 2: Tools and programs to assist leaders of the organisation to actively support the use of research in policy and program development

This should happen at least once a month and have happened within the last 6 months.

Referral to research in internal communications is irregular and infrequent. It happens less than once a month.

There are no relevant internal communications or if there are, leaders either do not refer to research in them or have not done so in the last 6 months.

OR the newsletters or communications are at least monthly, but only refer to research inconsistently (less than once a month) OR SPORADICALLY

6. Does your organisation provide access to training for staff in how to access research, appraise and apply research for policy development/implementation/evaluation?

Domain 3: Availability of programs to provide staff with training in using evidence from research in policy and in maintaining these skills

The training should be research skills specific not just referred to in the course of other training. The organisation needs to provide training internally, or allow staff to attend external training. Access to programs is actively offered to most staff, not just on an on-request basis.

Yes training is available if people say they need it – but it is not generally offered or supported, and not on an ongoing basis. OR

There is no training provided internally, and there is no support for staff to attend courses externally.

Yes but staff may not be aware of it.

There is training on how to access, appraise, and/or apply research in general, not specifically for the purpose of policymaking

Should be regular, that is a least once a year, and available to everyone.

7. Is participation in training on how to access research, appraise and apply research for policy development, implementation, or evaluation considered in staff performance management

Domain 3: Availability of programs to provide staff with training in using evidence from research in policy and in maintaining these skills

Staff performance management must explicitly mention training in research use or evaluation for most relevant staff.

Performance management covers only one or two of these areas, for example, applying research is not included, or evaluation is not included.

Participation in training is not considered in performance management.

OR

This is only considered relevant to the performance management of a very small group of staff e.g., people whose entire job is in evaluation but not regular policy makers.

OR it is implied in the performance management, but not explicitly stated

OR it is considered as an issue, only if it has come up as an issue to have these research skills.

8. In the last 6 months, has relevant research (papers, reports, syntheses or summary bulletins) been disseminated within your organisation?

Domain 4: Availability of supports and tools to help staff access and apply research findings

This should happen frequently, that is at least several times a month, and must have happened in the last 6 months.

This happens less than twice a month.

Relevant research has not been disseminated in the last 6 months or not at all.

It does not matter who sends these around, i.e. colleagues on an ad hoc basis or a more systematic approach.

9. Does your organisation have resources that provide guidance on how to access, appraise and apply research?

Domain 4: Availability of supports and tools to help staff access and apply research findings

The organisation must have documentary resources (handbooks, guidelines, online learning modules, etc.) on all three and readily available to staff.

There are limited resources or they do not cover all three aspects of research use.

There are no documentary resources.

10. Does your organisation have staff with recognised expertise in accessing, appraising and applying research to policy development/implementation/evaluation?

Domain 4: Availability of supports and tools to help staff access and apply research findings

This expertise needs to be accessible by most staff, high level and tied to a particular role rather than a person serendipitously having these skills.

This expertise is not tied to a role. Some people may have these skills but it is serendipitous and/or other staff are not generally able to access their expertise.

No – no one is available.

11. Does your organisation have research resources such as

Domain 4: Availability of supports and tools to help staff access and apply research findings

i. Topic specific journals – yes, access to all or most relevant journals is available. – This access must be provided by the organisation and not from a university login

ii. For example, Medline, Embase, PsycInfo, etc. as relevant

iii. A library that provides access to a range of resources, not just a shared filing system, and research is quickly and easily available through an electronic/online database (similar to how a university library works)

iv. Yes to Endnote or something similar (including access if they ask for it)

(This refers to the organisation providing access, not to access through other means e.g., staff member’s university affiliation).

i. Yes some journal access but cannot access many of the journals needed.

ii. Yes access to some databases or a database but several key databases are not available

iii. Yes but it takes a long time to get access to full text articles, or can’t get full text, or yes but doesn’t stock many of the key books required

iv. No midpoint here as you would not require more than one

i. No journal subscriptions

ii. No subscriptions to databases

iii. No access to a library or electronic library

iv. No licenses for reference management software

i. Subscriptions to research journals? (e.g. …)

ii. Subscriptions to databases of research publications?

iii. A library or an electronic library?

iv. Licenses for reference management software (e.g. Endnote)

Note: these questions refer to resources provided by the organisation, and does NOT include resources owned by individual employees (such as university logins, their own licenses to EndNote). If this is the case for any of these, score 1.

12. Does your organisation have established methods for commissioning reviews of existing research?

Domain 4: Availability of supports and tools to help staff access and apply research findings

Yes there is a standard written process which staff are expected to use when commissioning research.

Yes but the methods are verbal/ad hoc/situation by situation.

There are no methods for commissioning reviews OR the organisation does not commission reviews of existing research.

If the Sax E-check is used, or any other organisation’s or institution’s rapid review process, then that would be scored in this category

13. Does your organisation have systems for managing knowledge from research? (e.g. systems for retrieving, collating, storing and translating external and internal research)

Domain 4: Availability of supports and tools to help staff access and apply research findings

There are shared filing systems, databases, etc. that are easily searchable and accessible by most relevant staff.

This is kept in one place and accessible but not indexed or easily searchable. The organisation relies on corporate memory to know what research has been done and where it is.

There is no central storage place and no process for managing knowledge from research.

Needs to be well-organized and structured; not simply a big folder or drive where the whole range of files (including non-research related documents)

Centralised system but disorganized, or not completely developed yet. TRIM is one example, unless they have highly organised it.

14. In the last 6 months, has your organisation undertaken internal research to support policy development/implementation/evaluation?

Domain 5: Presence of systems/methods to generate new research evidence to inform the organisation’s work

Must have been in the last 6 months and undertaken by staff of the organisation. It includes at least one large or in-depth piece of internal research, or several smaller pieces of internal research.

One small pieces of basic internal research.

No never or not in the last 6 months.

(For example focus groups, satisfaction surveys), but not evidence check.)

This question does NOT include whether the organisation has undertaken evaluations of their policies. This is captured in Qs 16-18

15. In the last 6 months, has your organisation commissioned external research to support policy development/implementation/evaluation?

Domain 5: Presence of systems/methods to generate new research evidence to inform the organisation’s work

Research undertaken by another organisation (potentially in partnership with the organisation Must have been in the last 6 months and more than once.

In the last 6 months but only once

No never or not in the last 6 months.

External research to inform policy development Or to inform the implementation or evaluation of a policy or program. This question is not about whether the organisation evaluates their policies.

16. Does your organisation encourage or require that evaluation be built into policy development and program planning?

Domain 6: Clear methods to allow adequate, evidence-informed evaluations of the organisations’ policies and programs

There is an explicitly documented organisational requirement that evaluation be built into every policy/program.

Yes this is expected but not required, or is not required of all programs

Evaluations do not occur or occur occasionally but there is no organisational requirement to conduct them.

(Questions 16–18 include situations where evaluations are commissioned externally).

OR is just about to be rolled out (and so is happening partially, over some programs and policies)

17. Does your organisation have documented processes for how policies should be evaluated?

Domain 6: Clear methods to allow adequate, evidence-informed evaluations of the organisations’ policies and programs

The processes must explain in detail how the policies should be evaluated.

Yes there are documented processes which are very general.

No documented processes

OR documented processes are developed on a case by case basis or following initial preparations

18. Do these processes encourage or require staff to use research in policy evaluation OR are these evaluation processes and methods based on research?

Domain 6: Clear methods to allow adequate, evidence-informed evaluations of the organisations’ policies and programs

The requirement to use research must be explicit and unequivocal.

The processes refer to research but do not encourage or require that research be used.

There are documented processes but there is no requirement to use research or there are no documented processes regarding evaluation.

Or the evaluation is conducted by an expert who we assume has been influenced by research in their approach (more indirect use of research)

NOTE: this does NOT refer to collecting data as part of the evaluation. This is about whether the evaluation approach used by the organisation is based on research, or requires staff to use research to guide the evaluation

(This might include researching evaluation methods as well as scene setting). “Research” does not include data collection whereas question 14 does.

Either the guidelines are research-evidence based, or the guidelines instruct the individual to search for research evidence to support their evaluation approach.

NOTE: If answered NO to question 17, then score 1 for this question, even if the evaluation is performed by an expert.

19. In the last 6 months, has your organisation been represented at any research forums or conferences?

Domain 7: Mechanisms that help strengthen staff relationships with researchers

Attendance at such events was common and by a range of staff.

Only a certain level of policy maker attends, or only attends as invited speakers, or only attends rarely.

No not in the last 6 months or not at all

20. Does your organisation have formal, contractual relationships with external research organisations?

Domain 7: Mechanisms that help strengthen staff relationships with researchers

Any formally documented relationship counts. Short term relationships are fine, if these are active at the time of the interview. There need to be several such relationships and a sense that these (or others) were likely to continue, and that having such relationships was important to their ongoing work.

Just one currently.

No, this does not happen or there are none currently.

21. Does your organisation have informal, collaborative relationships with external research organisations?

Domain 7: Mechanisms that help strengthen staff relationships with researchers

Any un-formalised relationship (including on a staff to staff basis) counts here.

Just one currently.

No, this does not happen or there are none currently.

22. Do members of your organisation have joint or adjunct appointments in research organisations?

Domain 7: Mechanisms that help strengthen staff relationships with researchers

Usual examples would be adjunct appointments in universities. A high rating would mean that several staff had such positions. This also includes where staff work part time at the agency in question and are also employed by a research organisation.

Just one currently.

No, this does not happen or there are none currently.

23. In the last 6 months, have researchers participated in policy advisory committees (or similar) in your organisation?

Domain 7: Mechanisms that help strengthen staff relationships with researchers

The involvement for researchers in these types of roles is frequent, that is, it happens more than once in 6 months, and is systematic (not serendipitous).

Only once in the last 6 months.

No this does not happen or has not happened in the last 6 months