The development of ORACLe: a measure of an organisation’s capacity to engage in evidence-informed health policy

Abstract

Background

Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations’ research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations’ capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership).

Method

We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe.

Results

The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4).

Conclusion

We utilised qualitative and quantitative methods to develop a system to assess and score organisations’ capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.

Background

There are a number of factors that bear influence on the formulation of health policy, including political pressures, stakeholder interests, media influence, resource availability, previous policies, and evidence from research [1-4]. However, there have been increased calls worldwide to strengthen the use of research evidence in the development, evaluation and implementation of policies [5-7]. Research is purported to provide reliable and valid evidence with which to inform decisions and formulate effective solutions in response to problems, and numerous studies have demonstrated associations between evidence-informed policies and improvements in health and spending [8]. Despite this, evidence suggests that, currently, many opportunities to use research to inform policymaking are missed [9-18].

There is growing recognition that an organisation’s technical capacity, climate and culture collectively affect employees’ work performance, adoption of and adherence to innovative and best work practices, and their expectations, attitudes, commitment, and value towards their work [19-23]. Indeed, evidence indicates that policymakers’ use of research can be improved if their organisations (1) have a receptive attitude and culture towards research use and (2) invest in resources that support research use capacity among staff (e.g. training programs, availability of research expertise, access to research databases, and other tools [24-28]). In light of this evidence, it is essential that valid and reliable measures of the capacity and culture of organisations to support the use of research in policymaking are developed. Such measures are necessary to enable agencies to understand whether there are opportunities for increasing their use of evidence, where these opportunities for improvement exist within the agency, what particular strategies could be implemented to improve the capacity of staff to better engage with and use research evidence in policymaking, and whether or not these capacity building strategies have been effective [29,30].

A small number of measures of organisational research use capacity are available, although they possess some key limitations. For example, (1) they are not specifically directed at policy agencies and evidence-informed policy [31]; (2) they measure barriers to research use rather organisational capacity to support research use [31]; (3) they lack reference to an explicit conceptual framework [31,32], thus failing to elucidate the influence of the wider policymaking context on organisational capacity and the potential impact of such capacity on policymakers’ engagement with and use of research [33,34]; (4) some aspects of organisational capacity are not addressed, such as the availability of tools to support leadership for research use [23] or mechanisms to commission and generate research to inform policy [28]; (5) the relative importance of different organisational tools and supports are not taken into account in these measures’ scoring systems, which has important implications for the practical utility of these measures for policy organisations; and finally (6) these measures are typically completed by staff members rather than the organisations’ leaders or executives. Consequently, what they measure are staff perceptions rather than the objective availability of tools within the organisation to support research use.

Given the likely critical importance of organisational capacity, and the dearth of suitable objective measures, we aimed to develop a comprehensive, valid, objective, and theory-based measure of organisational capacity to engage with and use research in policy development, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership). ORACLe arose from a need to develop a comprehensive suite of measures to evaluate the impact of a multifaceted intervention program designed to increase organisations’ use of research in policy, entitled SPIRIT (Supporting Policy In health with Research: an Intervention Trial) [35]. Specifically, measures were developed to evaluate the impact of the SPIRIT intervention on organisations’ capacity to support research use (i.e. ORACLe), policymakers’ self-reported capacity, engagement with, and use of research (i.e. SEER (Seeking, Engaging, and Evaluating Research) [35]), and the use of research in the development of discrete policy documents (i.e. SAGE [35]).

As a measure of organisational capacity, ORACLe is grounded in the SPIRIT Action Framework [34], which postulates that the extent to which an organisation and staff have the capacity to engage with research directly mediates whether staff will effectively engage with and use research to inform policymaking. Because of the link between organisational capacity and staff capacity to engage with and use research to inform policymaking, ORACLe is designed to assess multiple aspects of organisational capacity, including the systems, supports, and tools organisations have in place to enable research use, and the value placed on research by the organisation. ORACLe is administered as a structured interview with organisation leaders as they are in the best position to know the extent to which supports are present within their organisations. It is comprised of 23 questions inviting respondents to describe whether, and to what degree, a range of supports are in place within their organisations to facilitate research use. Responses to each question are later scored on a three-point scale. Typically, only one leader from each organisation is required to complete ORACLe.

Unlike other measures of organisational capacity, ORACLe has a scoring system that calculates total scores by assigning a different weight to each organisational capacity domain based on its relative importance. The scoring system calculates context-appropriate total scores, and can inform organisations as to areas where they could enhance their research use capacity.

In this paper, we aim to (1) describe how the ORACLe interview was developed; (2) describe how the ORACLe interview is scored; and (3) outline how a discrete choice experiment (DCE) was used to generate the system to score ORACLe. DCEs have been applied in the area of health economics to understand patients’ preferences for different healthcare services [36-43], and can be used to determine not only what products or objects individuals prefer, but also the product/object attributes that drive these preferences and their relative importance [41,43].

Methods

Ethics

Ethics approval was granted by the University of Western Sydney Human Research Ethics Committee HREC Approval H9413 and H9431. Written consent was obtained from all potential respondents prior to their participation in the study.

Development of the ORACLe interview

We aimed to develop a measure that would determine the capacity of an organisation to support and use evidence in developing policy through a number of key domains, linked to the SPIRIT action framework. A combination of strategies was used to generate the items within these domains for the ORACLe interview and each is described in detail below.

Review of the literature

We conducted a search for articles in the area of knowledge translation, particularly those focusing on organisational barriers and facilitators to research use on SCOPUS using search terms such as ‘Research’ or ‘Evidence’, combined with ‘Health Policy’. Abstracts were examined to determine if articles were relevant, and referred to organisational barriers or facilitators to some extent. Each relevant paper was read thoroughly by two of the authors (TT and SM) to identify a wide range of concrete examples of possible tools and supports within organisations that can enhance the capacity of staff to effectively engage with research. The analysis included examples and items used in existing instruments (e.g. [23,32]). These tools and supports were then categorised into a smaller number of key domains of organisational research use capacity reflected in the SPIRIT action framework.

Semi-structured interviews with policymakers

A purposive expert sample of (n = 9) senior Australian health policymakers were consulted to determine whether the organisational capacity domains identified in the literature were practical and applicable to real-world policy settings. Each expert was emailed a list of the organisational capacity domains identified in the literature review prior to undertaking a semi-structured interview (these domains are listed in Table 1). In the interview, experts responded to an open-ended question asking them to identify organisational strategies that facilitate the use of research in policy and program decision making. They were then asked specific questions about the relevance, applicability, and importance of each capacity domain identified in the literature search. The interviews were recorded, transcribed, and analysed to help refine the existing list of domains and identify additional tools and supports. Further details of the method and results of the interviews are described elsewhere [44].

Table 1 Key domains and associated examples of organisational capacity and systems to support research use in policy development identified from review of literature

Iterative interactions with policymakers and pilot testing

Following the literature analysis and interviews with senior policymakers, an initial set of interview questions for ORACLe was compiled. Feedback was sought from a number of senior Australian policymakers and researchers in health to determine whether the questions were appropriately worded, relevant and clear. Interview questions were then modified on the basis of this feedback. To further establish the face-validity, comprehensibility and applicability of the interview items, six pilot ORACLe interviews were undertaken in three Australian health policy agencies followed by pilot testing of the SPIRIT intervention and the associated measurement tools (including ORACLe) in one policy agency. A number of minor changes to the ORACLe interview were proposed following pilot testing, including the removal of redundant questions, modification of item wording, and inclusion of additional items to address other research resources (e.g. knowledge management systems).

Development of the ORACLe scoring system using discrete choice experiments (DCEs)

After generating the interview items, quantitative techniques were utilised to devise a scoring system for ORACLe that appropriately weighted scores on the seven domains to produce a total score. It did not make theoretical or practical sense to assign each ORACLe domain an equal score because certain domains are likely to be more critical than others at enhancing the capacity of staff to use research in policymaking. To obtain appropriate numeric weights for each of the seven domains, we elicited the opinions of experts in health research and policy (n = 24) through the use of a DCE [43,45] (described in detail in elsewhere [46]). Prior to recruiting the expert sample, the DCE was pilot-tested to ensure it was optimal in terms of completion time, comprehensibility and appropriateness. Experts were then recruited by contacting corresponding authors of articles identified in the literature review described above. Snowballing was then used to expand this pool of participants by asking the identified experts to nominate others with similar knowledge and expertise. We sought the input of experts because they are highly knowledgeable about the policy context as well as the organisational barriers and facilitators to research use. Consequently, they can provide objective and context-sensitive judgments regarding the relative importance of different organisational capacity domains.

In a typical DCE, respondents do not rate individual domains (which are the attributes), but instead select between pairs of attribute combinations called profiles [41]. In our study, respondents were shown two profiles of organisations with different domain combinations and selected which organisation made the most effective use of research evidence for policy decisions. This is an ecologically valid approach because different organisations often possess multiple domains, but in different combinations (e.g. one organisation may have excellent tools to support leaders, and tools to support training in research use, but have few systems to facilitate staff capacity to access research) and they should be evaluated as such.

Following regression analyses of respondents’ choices, a utility value was calculated for each attribute (or more specifically, the particular level of each attribute), indicating the effect of that attribute on respondents’ preferences. By applying the discrete choice approach in our study, we were able to establish experts’ preferences regarding which capacity domains represented more important organisational tools and supports to facilitate research use, and obtain utilities for each capacity domain in order to calculate total scores.

Choice of attributes (domains) and levels, experimental design

The attributes that were used to create each profile were domains 1–7 listed in Table 1. From this point onwards, we will refer to the attributes as domains. Each domain consisted of three levels: (1) it was present to a large degree within the organisation; (2) it was present to some or a limited extent; (3) it was not present (Table 2). A series of hypothetical profiles were generated using an Orthogonal Main Effects Plan (OMEP) [43]. This method generates a series of orthogonal and balanced profiles, which allows the estimation of utility values for each domain level (i.e. main effects), but makes no provisions for the estimation of interactions [41]. Each profile represented an organisation that contained a combination of levels from all seven domains (Fig. 1). The OMEP was used to generate three sets or versions of eight pairs of profiles. In the first version, each domain consisted of only levels (1) or (2). In the second version, each domain consisted of levels (1) and (3), whereas in the third set, each domain consisted of levels (2) or (3). This produced three sets of eight profile pairs, generating 24 pairs of profiles. Participants were randomly assigned to one version of eight profile pairs.

Table 2 ORACLe Interview Questions and Marking Guide

Fig. 1

Example of two organisational profiles in a choice pair. Respondents are required to select which organisation makes the best use of research in policy decisions

The OMEP was also used to generate a common set of eight profile pairs. In this set, the domains contrasted level (1) (i.e. the attribute was not present) with level (3) (i.e. it was present to a large extent). Altogether, each respondent was asked to evaluate 16 pairs of profiles: eight profile pairs came from one of the three versions described above, and the other eight profiles pairs came from the common set. For the DCE itself, participants were exposed to each profile pair through an online survey. The profile pair consisted of two organisations that contained different combinations of levels of each of the seven domains. They were required to select which organisation in the pair made more effective use of research evidence in policy decisions (see Fig. 1 for an example).

Model calculation

After respondents made their choices, a conditional logit model was estimated from the combination of the common and master designs. A quadratic, as opposed to a linear relationship between domain levels and choice was used in the model calculation as it better represented the data. Consequently, each domain has a regression coefficient for its linear and quadratic component. These regression coefficients were used to calculate importance values to determine which domains had the strongest impact on respondents’ choices.

Results

The ORACLe interview

Eight domains of organisational capacity were identified from the literature review: (1) documented processes to develop policy that encourage or mandate the use of research; (2) tools and programs to assist leaders of the organisation to actively support the use of research in policy and program development; (3) availability of programs to provide staff with training in using evidence from research in policy and in maintaining these skills; (4) availability of support and tools to help staff access and apply research findings; (5) presence of systems/methods to generate new research evidence to inform the organisation’s work; (6) clear methods to ensure adequate evidence-informed evaluations of the organisations’ policies and programs; (7) mechanisms that help strengthen staff relationships with researchers; and (8) systems that analyse the ways that research can inform policies and programs. Table 1 lists each domain and concrete examples of each domain identified from the literature search.

The semi-structured interviews with senior policymakers verified the applicability and appropriateness of domains 1–7. However, there was a lack of consensus regarding the definition of domain 8 (systems in place to analyse the ways research can better inform policies and programs), with only two interviewees being able to provide concrete examples of this domain. Consequently, a preliminary ORACLe interview schedule was developed incorporating domains 1–7 only, and this was further refined through iterative discussions with senior policymakers and pilot testing. The final ORACLe interview schedule contains 23 questions which address capacity domains 1–7 as displayed in Table 1. Table 2 displays the full ORACLe interview schedule and the organisational capacity domain addressed by each question.

The ORACLe scoring system

The DCE data was analysed and the estimated conditional logit model is displayed in Table 3 and Fig. 2. Results revealed that each domain was significantly (and positively) related to respondents’ choices. In other words, respondents preferred organisations that possessed greater amounts of each domain.

Table 3 Conditional logit model estimated from choices of expert respondents

Fig. 2

Conditional logit model estimated from experts’ choices in the DCE. This model is used as the basis for calculating total ORACLe scores

Using this model, importance values were calculated to determine which domains had the strongest impact on respondents’ choices (Table 4). Based on the results, all domains significantly contributed to respondents’ choices, although domain 3 (tools/programs to provide staff with training in using research evidence) had the largest impact, followed by domains 2 (tools to assist leaders actively support research use within the organisation) and 4 (availability of support and tools to help staff access and apply research findings).

Table 4 Relative importance values of each ORACLe domain

The conditional logit model provides the basis for calculating total ORACLe scores. The model weight each capacity domain differently, based on the experts’ opinions regarding which domains are most important to strengthening organisations’ capacity to use research in policy. The steps involved in calculating ORACLe total scores are described below.

Using the interview and model to score ORACLe

ORACLe should be scored by an objective coder who was not directly involved in interviewing respondents. Preferably, coders should have experience in psychometric measurement or rating observable behaviour, but not necessarily be experts in knowledge translation or implementation science. To score ORACLe, objective coders examine the responses given to each of the 23 questions administered in interviews with organisational leaders (using interview transcripts and audio recordings) and assign a score indicating the extent to which each support is present within the organisation using the following three-point scale: (1) The tool/support is present to a large extent (score of 3), (2) The tool/support is present to some or a limited extent (score of 2), or (3) the tool/support is not present at all (score of 1). Scoring of each item is guided by a marking guide, which provides detailed descriptions of each of the abovementioned scoring categories (Table 2). The full interview and marking guidelines for each question is displayed in Table 2. Experience to date suggests that the ORACLe scores assigned by independent raters using the scoring guide exhibit a high level of inter-rater agreement [46].

After scoring each individual question, scores for questions within the same domain are averaged to produce a domain score, one for each of the seven domains (Fig. 3). These seven domain scores are then mean-centred (i.e. subtract 2 from each domain score) and substituted into the equation in Fig. 2 to compute the total ORACLe score which ranges from 0 to 10. This represents the score assigned to that particular agency, where higher scores are indicative of greater tools and support within the organisation to support research use in policy. These steps are outlined in Additional file 1, which provides SPSS syntax to calculate total ORACLe scores from raw data entered into an the data frame provided in Additional file 2.

Fig. 3

Domain score formulae. These are the formulae to calculate scores for each of the seven ORACLe domains (i.e. domain scores)

Discussion

This paper describes the development of a comprehensive, theory-based, structured interview measure of an organisation’s capacity to engage with and use research in policymaking, named ORACLe, and the establishment of a scoring system for the measure. A multifaceted approach was used to generate an interview schedule that encompassed a vast range of tools and systems that organisations may have in place to enhance staff capacity to engage with and use research. Further, a DCE was used to generate a scoring system for ORACLe that assigns appropriate weights to each ORACLe domain in the calculation of total scores, based on expert opinion regarding the most important strategies to support research use capacity within organisations. Our research thus represents the first attempt to empirically quantify the relative importance of different organisational structures and supports, and use this information to generate a valid system to score organisations’ research use capacity.

From the DCE results, domain 3 – tools/programs to provide staff with training in using research evidence – yielded the largest importance value. Therefore, our expert sample regarded the provision of staff training in accessing and applying research to policy as the most important component of an organisations’ research use capacity. This result parallels those in other sectors, including Meijers et al.’s systematic review [47], where research use among nurses was strongly associated with the degree of multifaceted access to research resources and support, and the provision of training in research use by the organisation.

Domain 2 – tools to assist leaders actively support research use within the organisation – had the next largest importance value. Thus, the expert sample placed a great deal of importance in organisations providing programs to support research use leadership. Helmsley-Brown [28] emphasised that leaders were critical to establishing an evidence-based organisational culture and direction that encouraged and motivated reflection, criticism of existing practices, free expression of experiences, and use of research. Gold [27] echoed this perspective, stating that leadership was important in allaying employees’ concerns about using research and promoting a mutual understanding between policymakers and researchers. In support of these views and our findings, El-Jardali et al. [48] found that strong organisational leadership was a key factor in promoting evidence-informed policy initiatives (e.g. programs to improve research use capacity among decision makers); establishing collaborative partnerships between researchers, users, and funders; increasing awareness of the value of research use in policy; and bringing about greater use of research in policymaking. One currently available leadership program is EXTRA (Executive Training for Research Application), which aims to increase the capacity of health service executives to access, promote, and use research, as well as to increase their organisation’s receptivity to research use [49]. Evidence indicates that the EXTRA program leads to self-reported improvements in research literacy (e.g. accessing and conducting research) and skills in promoting use of research evidence within the organisation and modest changes in staff receptivity to research.

Domain 4 – the availability of supports and tools to help staff access and apply research also emerged as a relatively important domain of organisational research use capacity. This finding coincides with numerous studies emphasising the importance of providing staff with resources to assist research access and use. For example, Canadian policymakers and research specialists stated that the most important element impacting upon research use was the organisational climate, and in particular, the availability of infrastructure and specialist staff to help policymakers use and apply research evidence [19]. Similarly, Evans et al. [26] found in their interviews with policymakers, that the availability of central guidance, technical and academic support, supportive resources, and standardized frameworks to conducting research and evaluation, were positively related to policymakers’ capacity to use research and conduct rigorous evaluations of policies.

In contrast to these results, the lowest importance value emerged for domain 5 – presence of methods to generate new research to inform the organisation’s work. Therefore, relative to the domains described above, our experts did not regard the presence of processes to conduct or commission data analyses, research studies or evaluation as particularly important to building an organisations’ capacity to engage with and apply research to policy. This is most likely due to the overwhelming emphasis on increasing organisations’ capacity to engage with existing research findings as opposed to producing new research to inform policy and that, in general, policy agencies are not in the business of doing research [19,50].

The expert sample also regarded domain 7 – mechanisms that help strengthen staff relationships with researchers – as less important in contributing to overall capacity of the organisation to use research in policy, relative to the other domains. This is unexpected given the preponderance of existing evidence emphasising the importance of organisations establishing ongoing partnerships with researchers and developing mechanisms to allow such contact to be established [16,18,51,52]. In light of their expertise in both policy and research, the expert sample may have felt that if policymakers had developed research skills themselves through training programs and professional development opportunities (i.e. domain 2), there might be less of a need to consult researchers. Furthermore, the experts may have felt that policymakers would be more likely to gain a balanced perspective on policy issues by evaluating the research themselves, versus consulting researchers who may have fixed opinions on those issues [53,54]. These explanations, however, are only conjecture and further qualitative research is required to understand why the experts prioritised particular domains over others.

ORACLe has been developed to overcome many of the limitations of previous measures described in the introduction [20,32,47]. Firstly, it specifically addresses organisational tools and capacity to use research in policymaking, as opposed to measuring organisational culture/capacity very generally. Secondly, ORACLe is a theory-based measure, grounded in the SPIRIT Action Framework, and therefore emphasises that organisational research capacity and culture is critical to promoting staff capacity to engage with and use research, and enabling the development of evidence-informed policies. Third, it is a comprehensive measure, assessing seven domains of organisational capacity and culture which expands upon previous organisational capacity measures (e.g. [32]). Fourth, because ORACLe is completed by organisational leaders, it directly assesses the availability of tools and supports to encourage research use, as opposed to measuring staff perceptions of these supports and tools [23,32,47]. Leaders are the most reliable respondents in this context since they would have been responsible for putting in place these research use tools and supports within their respective organisations. This is a key advantage of ORACLe over previous measures of organisational research use capacity. Agencies, however, often have multiple divisions led by different senior staff. Consequently, a single executive might not be aware of all the tools and systems in place across the agency. One approach, as utilised by Kothari et al. [32], would be to gather a small representative group (four to six) of senior staff from each organisation to complete ORACLe to ensure a more valid and objective assessment of the agency’s systems and culture. This kind of inter-rater reliability should ideally be incorporated into future testing and use of the measure.

Fifth, one of the major strengths of ORACLe is its empirically derived scoring system. By conducting a DCE with an expert sample, we were able to generate a model to calculate total scores on ORACLe. In this calculation, the model assigns different weights to each of the seven domains based on experts’ opinions regarding which domains are most important to strengthening organisations’ capacity to use research in policy. As a result, it allows users to calculate total scores on ORACLe that are context-sensitive, appropriate, and in line with experts’ preferences. This represents a major advance on previous tools which were unable to weigh different aspects of organisational research use capacity based on their importance.

Another key advantage of the ORACLe scoring system is that it allows organisations to identify their specific capacity development needs, and the relative importance of these capacities. Organisations can use their scores on each domain to guide decisions about which tools and supports to invest in to improve their research use capacity. For example, if an organisation scores low on domain 3, its leaders may choose to invest in a range of training programs to help staff better engage with and use research in policy. Furthermore, given that domain 3 yielded the highest importance value, investments into improving this capacity may be more likely to yield considerable improvements in the organisations’ overall research use capacity.

In terms of limitations, our sample size was relatively small compared to other DCE studies [55]. Hence, it is possible that the model underlying our scoring system is specific to our particular sample of experts, and that a different model would emerge in another, larger sample. A review of choice studies conducted in the health field revealed that the average sample was 259, and ranged from 13 to 1258 [56]. Sample sizes should be at least 150 if the target population is very large, although this would not be the case for the population of knowledge translation experts [55]. Orme [55] recommended that, when conducting pilot work or developing hypotheses about a particular target group, 30–60 participants may be sufficient. Our current sample approaches this cut-off.

Despite the relatively small sample size, we are confident in our findings for three main reasons. Firstly, our sample consisted of a diverse group of experienced knowledge translation experts. Secondly, supplementary analyses revealed that there were no statistically significant differences between experts in their choice preferences (see [46] for details). Thirdly, we have no reason to believe that the DCE survey was overly complex for respondents and invalidly elicited their preferences, since the number of domains was kept low (i.e. seven), and all domains and levels were clearly presented, concretely defined, and operationally distinct [41].

ORACLe aims to measure a range of domains in order to inform agencies of opportunities to improve policymakers’ use of research. It is important to note, however, that organisational change is highly complex, and that top-down initiatives alone are unlikely to bring about changes in staff capacity and attitudes towards research use [57]. Recognition of the wider organisational system is essential. Many studies emphasise the importance of social processes, including the role of networks and key change agents (e.g. champions and opinion leaders), in facilitating the uptake of innovations [58,59]. Further, the concept of ‘good practice’ and how different types of knowledge are valued are often shaped by collective responses to local circumstances which may demand local (rather than standardised) solutions [58]. These social and contextual influences are not explicitly addressed in ORACLe (as they are less amenable to quantitative measurement and intervention), but are nonetheless critical considerations when designing and implementing initiatives for building organisational capacity in using research.

ORACLe has yet to be validated as a measure of organisational capacity and culture to support research use. Validity testing may involve investigating its factorial structure, and verifying whether items load onto the seven organisational domains obtained in Table 1 as well as a common organisational capacity factor. However, given that ORACLe is completed by agency executives, it would take time to recruit a sufficiently large sample to undertake factor analysis. We considered the possibility of inviting non-senior staff (i.e. policymakers) to do ORACLe as a strategy to increase sample size. It is likely that staff would have different perceptions regarding their organisation’s systems and tools, relative to executives. For example, staff might not be aware of all the tools available within their organisation. However, ORACLe was designed to be an objective (and thus, valid) measure of whether a range of systems and tools exist within the organisation to support research use, as opposed to staff perceptions (or awareness) of these systems and tools. Hence, ORACLe targets informants who would have first-hand knowledge of this information, which, for the most part, would be the executive staff. The need for a knowledgeable informant necessarily places limitations on sample size, and means that ORACLe is a tool more suited to formative than summative purposes. We have developed a companion tool, SEER [35], which includes scales measuring staff perceptions/awareness of organisational systems and support. Measurement of perceptions is important, since a lack of awareness of organisational tools and systems among staff is likely to negatively impact upon their use of research in policy [19]. We plan to examine convergence of responses on ORACLe and SEER as part of further tests of the validity of both measures (see below).

A practical approach to validity testing would be to test whether ORACLe scores are predictive of relevant outcomes such as policymakers’ skills and values towards research use, and their use of research in policymaking. Presumably, if organisations score well on ORACLe, then policymakers should hold more positive attitudes to research use, exhibit greater engagement with research (e.g. actively searching for and appraising research), participate in more active collaborations with researchers, and show greater use of research in policymaking. SEER [35] is one such measure that evaluates policymakers’ attitudes and skills regarding research use, as well as their engagement with, and use of research in their work. Another measure we have developed, SAGE [60-62], measures the extent to which research was engaged with and used in discrete policy or program documents. We are currently testing the convergent validity between ORACLe and these two measures. Further validity testing might involve examining whether ORACLe is predictive of more distal outcomes such as financial expenditures and health outcomes.

Conclusions

In this paper, we have described the qualitative and quantitative development of a system to measure and score organisations’ capacity to engage with and apply research to policymaking. The qualitative development ensured that the measure was thorough, face-valid, and captured all the main elements of organisational capacity. The quantitative development produced a scoring system that not only assigns context-appropriate total scores, but can inform organisations about what capacities and tools require further development and investment, and the relative importance of these capacities. We hope that the development of this measure will trigger initiatives to improve organisations’ tools and support, increase research capacity among staff, and drive the ongoing development of evidence-informed health policies.