- Open Access
- Open Peer Review
The development of ORACLe: a measure of an organisation’s capacity to engage in evidence-informed health policy
Health Research Policy and Systemsvolume 14, Article number: 4 (2016)
Evidence-informed policymaking is more likely if organisations have cultures that promote research use and invest in resources that facilitate staff engagement with research. Measures of organisations’ research use culture and capacity are needed to assess current capacity, identify opportunities for improvement, and examine the impact of capacity-building interventions. The aim of the current study was to develop a comprehensive system to measure and score organisations’ capacity to engage with and use research in policymaking, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership).
We used a multifaceted approach to develop ORACLe. Firstly, we reviewed the available literature to identify key domains of organisational tools and systems that may facilitate research use by staff. We interviewed senior health policymakers to verify the relevance and applicability of these domains. This information was used to generate an interview schedule that focused on seven key domains of organisational capacity. The interview was pilot-tested within four Australian policy agencies. A discrete choice experiment (DCE) was then undertaken using an expert sample to establish the relative importance of these domains. This data was used to produce a scoring system for ORACLe.
The ORACLe interview was developed, comprised of 23 questions addressing seven domains of organisational capacity and tools that support research use, including (1) documented processes for policymaking; (2) leadership training; (3) staff training; (4) research resources (e.g. database access); and systems to (5) generate new research, (6) undertake evaluations, and (7) strengthen relationships with researchers. From the DCE data, a conditional logit model was estimated to calculate total scores that took into account the relative importance of the seven domains. The model indicated that our expert sample placed the greatest importance on domains (2), (3) and (4).
We utilised qualitative and quantitative methods to develop a system to assess and score organisations’ capacity to engage with and apply research to policy. Our measure assesses a broad range of capacity domains and identifies the relative importance of these capacities. ORACLe data can be used by organisations keen to increase their use of evidence to identify areas for further development.
There are a number of factors that bear influence on the formulation of health policy, including political pressures, stakeholder interests, media influence, resource availability, previous policies, and evidence from research [1-4]. However, there have been increased calls worldwide to strengthen the use of research evidence in the development, evaluation and implementation of policies [5-7]. Research is purported to provide reliable and valid evidence with which to inform decisions and formulate effective solutions in response to problems, and numerous studies have demonstrated associations between evidence-informed policies and improvements in health and spending . Despite this, evidence suggests that, currently, many opportunities to use research to inform policymaking are missed [9-18].
There is growing recognition that an organisation’s technical capacity, climate and culture collectively affect employees’ work performance, adoption of and adherence to innovative and best work practices, and their expectations, attitudes, commitment, and value towards their work [19-23]. Indeed, evidence indicates that policymakers’ use of research can be improved if their organisations (1) have a receptive attitude and culture towards research use and (2) invest in resources that support research use capacity among staff (e.g. training programs, availability of research expertise, access to research databases, and other tools [24-28]). In light of this evidence, it is essential that valid and reliable measures of the capacity and culture of organisations to support the use of research in policymaking are developed. Such measures are necessary to enable agencies to understand whether there are opportunities for increasing their use of evidence, where these opportunities for improvement exist within the agency, what particular strategies could be implemented to improve the capacity of staff to better engage with and use research evidence in policymaking, and whether or not these capacity building strategies have been effective [29,30].
A small number of measures of organisational research use capacity are available, although they possess some key limitations. For example, (1) they are not specifically directed at policy agencies and evidence-informed policy ; (2) they measure barriers to research use rather organisational capacity to support research use ; (3) they lack reference to an explicit conceptual framework [31,32], thus failing to elucidate the influence of the wider policymaking context on organisational capacity and the potential impact of such capacity on policymakers’ engagement with and use of research [33,34]; (4) some aspects of organisational capacity are not addressed, such as the availability of tools to support leadership for research use  or mechanisms to commission and generate research to inform policy ; (5) the relative importance of different organisational tools and supports are not taken into account in these measures’ scoring systems, which has important implications for the practical utility of these measures for policy organisations; and finally (6) these measures are typically completed by staff members rather than the organisations’ leaders or executives. Consequently, what they measure are staff perceptions rather than the objective availability of tools within the organisation to support research use.
Given the likely critical importance of organisational capacity, and the dearth of suitable objective measures, we aimed to develop a comprehensive, valid, objective, and theory-based measure of organisational capacity to engage with and use research in policy development, which we entitled ORACLe (Organisational Research Access, Culture, and Leadership). ORACLe arose from a need to develop a comprehensive suite of measures to evaluate the impact of a multifaceted intervention program designed to increase organisations’ use of research in policy, entitled SPIRIT (Supporting Policy In health with Research: an Intervention Trial) . Specifically, measures were developed to evaluate the impact of the SPIRIT intervention on organisations’ capacity to support research use (i.e. ORACLe), policymakers’ self-reported capacity, engagement with, and use of research (i.e. SEER (Seeking, Engaging, and Evaluating Research) ), and the use of research in the development of discrete policy documents (i.e. SAGE ).
As a measure of organisational capacity, ORACLe is grounded in the SPIRIT Action Framework , which postulates that the extent to which an organisation and staff have the capacity to engage with research directly mediates whether staff will effectively engage with and use research to inform policymaking. Because of the link between organisational capacity and staff capacity to engage with and use research to inform policymaking, ORACLe is designed to assess multiple aspects of organisational capacity, including the systems, supports, and tools organisations have in place to enable research use, and the value placed on research by the organisation. ORACLe is administered as a structured interview with organisation leaders as they are in the best position to know the extent to which supports are present within their organisations. It is comprised of 23 questions inviting respondents to describe whether, and to what degree, a range of supports are in place within their organisations to facilitate research use. Responses to each question are later scored on a three-point scale. Typically, only one leader from each organisation is required to complete ORACLe.
Unlike other measures of organisational capacity, ORACLe has a scoring system that calculates total scores by assigning a different weight to each organisational capacity domain based on its relative importance. The scoring system calculates context-appropriate total scores, and can inform organisations as to areas where they could enhance their research use capacity.
In this paper, we aim to (1) describe how the ORACLe interview was developed; (2) describe how the ORACLe interview is scored; and (3) outline how a discrete choice experiment (DCE) was used to generate the system to score ORACLe. DCEs have been applied in the area of health economics to understand patients’ preferences for different healthcare services [36-43], and can be used to determine not only what products or objects individuals prefer, but also the product/object attributes that drive these preferences and their relative importance [41,43].
Ethics approval was granted by the University of Western Sydney Human Research Ethics Committee HREC Approval H9413 and H9431. Written consent was obtained from all potential respondents prior to their participation in the study.
Development of the ORACLe interview
We aimed to develop a measure that would determine the capacity of an organisation to support and use evidence in developing policy through a number of key domains, linked to the SPIRIT action framework. A combination of strategies was used to generate the items within these domains for the ORACLe interview and each is described in detail below.
Review of the literature
We conducted a search for articles in the area of knowledge translation, particularly those focusing on organisational barriers and facilitators to research use on SCOPUS using search terms such as ‘Research’ or ‘Evidence’, combined with ‘Health Policy’. Abstracts were examined to determine if articles were relevant, and referred to organisational barriers or facilitators to some extent. Each relevant paper was read thoroughly by two of the authors (TT and SM) to identify a wide range of concrete examples of possible tools and supports within organisations that can enhance the capacity of staff to effectively engage with research. The analysis included examples and items used in existing instruments (e.g. [23,32]). These tools and supports were then categorised into a smaller number of key domains of organisational research use capacity reflected in the SPIRIT action framework.
Semi-structured interviews with policymakers
A purposive expert sample of (n = 9) senior Australian health policymakers were consulted to determine whether the organisational capacity domains identified in the literature were practical and applicable to real-world policy settings. Each expert was emailed a list of the organisational capacity domains identified in the literature review prior to undertaking a semi-structured interview (these domains are listed in Table 1). In the interview, experts responded to an open-ended question asking them to identify organisational strategies that facilitate the use of research in policy and program decision making. They were then asked specific questions about the relevance, applicability, and importance of each capacity domain identified in the literature search. The interviews were recorded, transcribed, and analysed to help refine the existing list of domains and identify additional tools and supports. Further details of the method and results of the interviews are described elsewhere .
Iterative interactions with policymakers and pilot testing
Following the literature analysis and interviews with senior policymakers, an initial set of interview questions for ORACLe was compiled. Feedback was sought from a number of senior Australian policymakers and researchers in health to determine whether the questions were appropriately worded, relevant and clear. Interview questions were then modified on the basis of this feedback. To further establish the face-validity, comprehensibility and applicability of the interview items, six pilot ORACLe interviews were undertaken in three Australian health policy agencies followed by pilot testing of the SPIRIT intervention and the associated measurement tools (including ORACLe) in one policy agency. A number of minor changes to the ORACLe interview were proposed following pilot testing, including the removal of redundant questions, modification of item wording, and inclusion of additional items to address other research resources (e.g. knowledge management systems).
Development of the ORACLe scoring system using discrete choice experiments (DCEs)
After generating the interview items, quantitative techniques were utilised to devise a scoring system for ORACLe that appropriately weighted scores on the seven domains to produce a total score. It did not make theoretical or practical sense to assign each ORACLe domain an equal score because certain domains are likely to be more critical than others at enhancing the capacity of staff to use research in policymaking. To obtain appropriate numeric weights for each of the seven domains, we elicited the opinions of experts in health research and policy (n = 24) through the use of a DCE [43,45] (described in detail in elsewhere ). Prior to recruiting the expert sample, the DCE was pilot-tested to ensure it was optimal in terms of completion time, comprehensibility and appropriateness. Experts were then recruited by contacting corresponding authors of articles identified in the literature review described above. Snowballing was then used to expand this pool of participants by asking the identified experts to nominate others with similar knowledge and expertise. We sought the input of experts because they are highly knowledgeable about the policy context as well as the organisational barriers and facilitators to research use. Consequently, they can provide objective and context-sensitive judgments regarding the relative importance of different organisational capacity domains.
In a typical DCE, respondents do not rate individual domains (which are the attributes), but instead select between pairs of attribute combinations called profiles . In our study, respondents were shown two profiles of organisations with different domain combinations and selected which organisation made the most effective use of research evidence for policy decisions. This is an ecologically valid approach because different organisations often possess multiple domains, but in different combinations (e.g. one organisation may have excellent tools to support leaders, and tools to support training in research use, but have few systems to facilitate staff capacity to access research) and they should be evaluated as such.
Following regression analyses of respondents’ choices, a utility value was calculated for each attribute (or more specifically, the particular level of each attribute), indicating the effect of that attribute on respondents’ preferences. By applying the discrete choice approach in our study, we were able to establish experts’ preferences regarding which capacity domains represented more important organisational tools and supports to facilitate research use, and obtain utilities for each capacity domain in order to calculate total scores.
Choice of attributes (domains) and levels, experimental design
The attributes that were used to create each profile were domains 1–7 listed in Table 1. From this point onwards, we will refer to the attributes as domains. Each domain consisted of three levels: (1) it was present to a large degree within the organisation; (2) it was present to some or a limited extent; (3) it was not present (Table 2). A series of hypothetical profiles were generated using an Orthogonal Main Effects Plan (OMEP) . This method generates a series of orthogonal and balanced profiles, which allows the estimation of utility values for each domain level (i.e. main effects), but makes no provisions for the estimation of interactions . Each profile represented an organisation that contained a combination of levels from all seven domains (Fig. 1). The OMEP was used to generate three sets or versions of eight pairs of profiles. In the first version, each domain consisted of only levels (1) or (2). In the second version, each domain consisted of levels (1) and (3), whereas in the third set, each domain consisted of levels (2) or (3). This produced three sets of eight profile pairs, generating 24 pairs of profiles. Participants were randomly assigned to one version of eight profile pairs.
The OMEP was also used to generate a common set of eight profile pairs. In this set, the domains contrasted level (1) (i.e. the attribute was not present) with level (3) (i.e. it was present to a large extent). Altogether, each respondent was asked to evaluate 16 pairs of profiles: eight profile pairs came from one of the three versions described above, and the other eight profiles pairs came from the common set. For the DCE itself, participants were exposed to each profile pair through an online survey. The profile pair consisted of two organisations that contained different combinations of levels of each of the seven domains. They were required to select which organisation in the pair made more effective use of research evidence in policy decisions (see Fig. 1 for an example).
After respondents made their choices, a conditional logit model was estimated from the combination of the common and master designs. A quadratic, as opposed to a linear relationship between domain levels and choice was used in the model calculation as it better represented the data. Consequently, each domain has a regression coefficient for its linear and quadratic component. These regression coefficients were used to calculate importance values to determine which domains had the strongest impact on respondents’ choices.
The ORACLe interview
Eight domains of organisational capacity were identified from the literature review: (1) documented processes to develop policy that encourage or mandate the use of research; (2) tools and programs to assist leaders of the organisation to actively support the use of research in policy and program development; (3) availability of programs to provide staff with training in using evidence from research in policy and in maintaining these skills; (4) availability of support and tools to help staff access and apply research findings; (5) presence of systems/methods to generate new research evidence to inform the organisation’s work; (6) clear methods to ensure adequate evidence-informed evaluations of the organisations’ policies and programs; (7) mechanisms that help strengthen staff relationships with researchers; and (8) systems that analyse the ways that research can inform policies and programs. Table 1 lists each domain and concrete examples of each domain identified from the literature search.
The semi-structured interviews with senior policymakers verified the applicability and appropriateness of domains 1–7. However, there was a lack of consensus regarding the definition of domain 8 (systems in place to analyse the ways research can better inform policies and programs), with only two interviewees being able to provide concrete examples of this domain. Consequently, a preliminary ORACLe interview schedule was developed incorporating domains 1–7 only, and this was further refined through iterative discussions with senior policymakers and pilot testing. The final ORACLe interview schedule contains 23 questions which address capacity domains 1–7 as displayed in Table 1. Table 2 displays the full ORACLe interview schedule and the organisational capacity domain addressed by each question.
The ORACLe scoring system
The DCE data was analysed and the estimated conditional logit model is displayed in Table 3 and Fig. 2. Results revealed that each domain was significantly (and positively) related to respondents’ choices. In other words, respondents preferred organisations that possessed greater amounts of each domain.
Using this model, importance values were calculated to determine which domains had the strongest impact on respondents’ choices (Table 4). Based on the results, all domains significantly contributed to respondents’ choices, although domain 3 (tools/programs to provide staff with training in using research evidence) had the largest impact, followed by domains 2 (tools to assist leaders actively support research use within the organisation) and 4 (availability of support and tools to help staff access and apply research findings).
The conditional logit model provides the basis for calculating total ORACLe scores. The model weight each capacity domain differently, based on the experts’ opinions regarding which domains are most important to strengthening organisations’ capacity to use research in policy. The steps involved in calculating ORACLe total scores are described below.
Using the interview and model to score ORACLe
ORACLe should be scored by an objective coder who was not directly involved in interviewing respondents. Preferably, coders should have experience in psychometric measurement or rating observable behaviour, but not necessarily be experts in knowledge translation or implementation science. To score ORACLe, objective coders examine the responses given to each of the 23 questions administered in interviews with organisational leaders (using interview transcripts and audio recordings) and assign a score indicating the extent to which each support is present within the organisation using the following three-point scale: (1) The tool/support is present to a large extent (score of 3), (2) The tool/support is present to some or a limited extent (score of 2), or (3) the tool/support is not present at all (score of 1). Scoring of each item is guided by a marking guide, which provides detailed descriptions of each of the abovementioned scoring categories (Table 2). The full interview and marking guidelines for each question is displayed in Table 2. Experience to date suggests that the ORACLe scores assigned by independent raters using the scoring guide exhibit a high level of inter-rater agreement .
After scoring each individual question, scores for questions within the same domain are averaged to produce a domain score, one for each of the seven domains (Fig. 3). These seven domain scores are then mean-centred (i.e. subtract 2 from each domain score) and substituted into the equation in Fig. 2 to compute the total ORACLe score which ranges from 0 to 10. This represents the score assigned to that particular agency, where higher scores are indicative of greater tools and support within the organisation to support research use in policy. These steps are outlined in Additional file 1, which provides SPSS syntax to calculate total ORACLe scores from raw data entered into an the data frame provided in Additional file 2.
This paper describes the development of a comprehensive, theory-based, structured interview measure of an organisation’s capacity to engage with and use research in policymaking, named ORACLe, and the establishment of a scoring system for the measure. A multifaceted approach was used to generate an interview schedule that encompassed a vast range of tools and systems that organisations may have in place to enhance staff capacity to engage with and use research. Further, a DCE was used to generate a scoring system for ORACLe that assigns appropriate weights to each ORACLe domain in the calculation of total scores, based on expert opinion regarding the most important strategies to support research use capacity within organisations. Our research thus represents the first attempt to empirically quantify the relative importance of different organisational structures and supports, and use this information to generate a valid system to score organisations’ research use capacity.
From the DCE results, domain 3 – tools/programs to provide staff with training in using research evidence – yielded the largest importance value. Therefore, our expert sample regarded the provision of staff training in accessing and applying research to policy as the most important component of an organisations’ research use capacity. This result parallels those in other sectors, including Meijers et al.’s systematic review , where research use among nurses was strongly associated with the degree of multifaceted access to research resources and support, and the provision of training in research use by the organisation.
Domain 2 – tools to assist leaders actively support research use within the organisation – had the next largest importance value. Thus, the expert sample placed a great deal of importance in organisations providing programs to support research use leadership. Helmsley-Brown  emphasised that leaders were critical to establishing an evidence-based organisational culture and direction that encouraged and motivated reflection, criticism of existing practices, free expression of experiences, and use of research. Gold  echoed this perspective, stating that leadership was important in allaying employees’ concerns about using research and promoting a mutual understanding between policymakers and researchers. In support of these views and our findings, El-Jardali et al.  found that strong organisational leadership was a key factor in promoting evidence-informed policy initiatives (e.g. programs to improve research use capacity among decision makers); establishing collaborative partnerships between researchers, users, and funders; increasing awareness of the value of research use in policy; and bringing about greater use of research in policymaking. One currently available leadership program is EXTRA (Executive Training for Research Application), which aims to increase the capacity of health service executives to access, promote, and use research, as well as to increase their organisation’s receptivity to research use . Evidence indicates that the EXTRA program leads to self-reported improvements in research literacy (e.g. accessing and conducting research) and skills in promoting use of research evidence within the organisation and modest changes in staff receptivity to research.
Domain 4 – the availability of supports and tools to help staff access and apply research also emerged as a relatively important domain of organisational research use capacity. This finding coincides with numerous studies emphasising the importance of providing staff with resources to assist research access and use. For example, Canadian policymakers and research specialists stated that the most important element impacting upon research use was the organisational climate, and in particular, the availability of infrastructure and specialist staff to help policymakers use and apply research evidence . Similarly, Evans et al.  found in their interviews with policymakers, that the availability of central guidance, technical and academic support, supportive resources, and standardized frameworks to conducting research and evaluation, were positively related to policymakers’ capacity to use research and conduct rigorous evaluations of policies.
In contrast to these results, the lowest importance value emerged for domain 5 – presence of methods to generate new research to inform the organisation’s work. Therefore, relative to the domains described above, our experts did not regard the presence of processes to conduct or commission data analyses, research studies or evaluation as particularly important to building an organisations’ capacity to engage with and apply research to policy. This is most likely due to the overwhelming emphasis on increasing organisations’ capacity to engage with existing research findings as opposed to producing new research to inform policy and that, in general, policy agencies are not in the business of doing research [19,50].
The expert sample also regarded domain 7 – mechanisms that help strengthen staff relationships with researchers – as less important in contributing to overall capacity of the organisation to use research in policy, relative to the other domains. This is unexpected given the preponderance of existing evidence emphasising the importance of organisations establishing ongoing partnerships with researchers and developing mechanisms to allow such contact to be established [16,18,51,52]. In light of their expertise in both policy and research, the expert sample may have felt that if policymakers had developed research skills themselves through training programs and professional development opportunities (i.e. domain 2), there might be less of a need to consult researchers. Furthermore, the experts may have felt that policymakers would be more likely to gain a balanced perspective on policy issues by evaluating the research themselves, versus consulting researchers who may have fixed opinions on those issues [53,54]. These explanations, however, are only conjecture and further qualitative research is required to understand why the experts prioritised particular domains over others.
ORACLe has been developed to overcome many of the limitations of previous measures described in the introduction [20,32,47]. Firstly, it specifically addresses organisational tools and capacity to use research in policymaking, as opposed to measuring organisational culture/capacity very generally. Secondly, ORACLe is a theory-based measure, grounded in the SPIRIT Action Framework, and therefore emphasises that organisational research capacity and culture is critical to promoting staff capacity to engage with and use research, and enabling the development of evidence-informed policies. Third, it is a comprehensive measure, assessing seven domains of organisational capacity and culture which expands upon previous organisational capacity measures (e.g. ). Fourth, because ORACLe is completed by organisational leaders, it directly assesses the availability of tools and supports to encourage research use, as opposed to measuring staff perceptions of these supports and tools [23,32,47]. Leaders are the most reliable respondents in this context since they would have been responsible for putting in place these research use tools and supports within their respective organisations. This is a key advantage of ORACLe over previous measures of organisational research use capacity. Agencies, however, often have multiple divisions led by different senior staff. Consequently, a single executive might not be aware of all the tools and systems in place across the agency. One approach, as utilised by Kothari et al. , would be to gather a small representative group (four to six) of senior staff from each organisation to complete ORACLe to ensure a more valid and objective assessment of the agency’s systems and culture. This kind of inter-rater reliability should ideally be incorporated into future testing and use of the measure.
Fifth, one of the major strengths of ORACLe is its empirically derived scoring system. By conducting a DCE with an expert sample, we were able to generate a model to calculate total scores on ORACLe. In this calculation, the model assigns different weights to each of the seven domains based on experts’ opinions regarding which domains are most important to strengthening organisations’ capacity to use research in policy. As a result, it allows users to calculate total scores on ORACLe that are context-sensitive, appropriate, and in line with experts’ preferences. This represents a major advance on previous tools which were unable to weigh different aspects of organisational research use capacity based on their importance.
Another key advantage of the ORACLe scoring system is that it allows organisations to identify their specific capacity development needs, and the relative importance of these capacities. Organisations can use their scores on each domain to guide decisions about which tools and supports to invest in to improve their research use capacity. For example, if an organisation scores low on domain 3, its leaders may choose to invest in a range of training programs to help staff better engage with and use research in policy. Furthermore, given that domain 3 yielded the highest importance value, investments into improving this capacity may be more likely to yield considerable improvements in the organisations’ overall research use capacity.
In terms of limitations, our sample size was relatively small compared to other DCE studies . Hence, it is possible that the model underlying our scoring system is specific to our particular sample of experts, and that a different model would emerge in another, larger sample. A review of choice studies conducted in the health field revealed that the average sample was 259, and ranged from 13 to 1258 . Sample sizes should be at least 150 if the target population is very large, although this would not be the case for the population of knowledge translation experts . Orme  recommended that, when conducting pilot work or developing hypotheses about a particular target group, 30–60 participants may be sufficient. Our current sample approaches this cut-off.
Despite the relatively small sample size, we are confident in our findings for three main reasons. Firstly, our sample consisted of a diverse group of experienced knowledge translation experts. Secondly, supplementary analyses revealed that there were no statistically significant differences between experts in their choice preferences (see  for details). Thirdly, we have no reason to believe that the DCE survey was overly complex for respondents and invalidly elicited their preferences, since the number of domains was kept low (i.e. seven), and all domains and levels were clearly presented, concretely defined, and operationally distinct .
ORACLe aims to measure a range of domains in order to inform agencies of opportunities to improve policymakers’ use of research. It is important to note, however, that organisational change is highly complex, and that top-down initiatives alone are unlikely to bring about changes in staff capacity and attitudes towards research use . Recognition of the wider organisational system is essential. Many studies emphasise the importance of social processes, including the role of networks and key change agents (e.g. champions and opinion leaders), in facilitating the uptake of innovations [58,59]. Further, the concept of ‘good practice’ and how different types of knowledge are valued are often shaped by collective responses to local circumstances which may demand local (rather than standardised) solutions . These social and contextual influences are not explicitly addressed in ORACLe (as they are less amenable to quantitative measurement and intervention), but are nonetheless critical considerations when designing and implementing initiatives for building organisational capacity in using research.
ORACLe has yet to be validated as a measure of organisational capacity and culture to support research use. Validity testing may involve investigating its factorial structure, and verifying whether items load onto the seven organisational domains obtained in Table 1 as well as a common organisational capacity factor. However, given that ORACLe is completed by agency executives, it would take time to recruit a sufficiently large sample to undertake factor analysis. We considered the possibility of inviting non-senior staff (i.e. policymakers) to do ORACLe as a strategy to increase sample size. It is likely that staff would have different perceptions regarding their organisation’s systems and tools, relative to executives. For example, staff might not be aware of all the tools available within their organisation. However, ORACLe was designed to be an objective (and thus, valid) measure of whether a range of systems and tools exist within the organisation to support research use, as opposed to staff perceptions (or awareness) of these systems and tools. Hence, ORACLe targets informants who would have first-hand knowledge of this information, which, for the most part, would be the executive staff. The need for a knowledgeable informant necessarily places limitations on sample size, and means that ORACLe is a tool more suited to formative than summative purposes. We have developed a companion tool, SEER , which includes scales measuring staff perceptions/awareness of organisational systems and support. Measurement of perceptions is important, since a lack of awareness of organisational tools and systems among staff is likely to negatively impact upon their use of research in policy . We plan to examine convergence of responses on ORACLe and SEER as part of further tests of the validity of both measures (see below).
A practical approach to validity testing would be to test whether ORACLe scores are predictive of relevant outcomes such as policymakers’ skills and values towards research use, and their use of research in policymaking. Presumably, if organisations score well on ORACLe, then policymakers should hold more positive attitudes to research use, exhibit greater engagement with research (e.g. actively searching for and appraising research), participate in more active collaborations with researchers, and show greater use of research in policymaking. SEER  is one such measure that evaluates policymakers’ attitudes and skills regarding research use, as well as their engagement with, and use of research in their work. Another measure we have developed, SAGE [60-62], measures the extent to which research was engaged with and used in discrete policy or program documents. We are currently testing the convergent validity between ORACLe and these two measures. Further validity testing might involve examining whether ORACLe is predictive of more distal outcomes such as financial expenditures and health outcomes.
In this paper, we have described the qualitative and quantitative development of a system to measure and score organisations’ capacity to engage with and apply research to policymaking. The qualitative development ensured that the measure was thorough, face-valid, and captured all the main elements of organisational capacity. The quantitative development produced a scoring system that not only assigns context-appropriate total scores, but can inform organisations about what capacities and tools require further development and investment, and the relative importance of these capacities. We hope that the development of this measure will trigger initiatives to improve organisations’ tools and support, increase research capacity among staff, and drive the ongoing development of evidence-informed health policies.
Black N. Evidence based policy: proceed with care. Brit Med J. 2001;323(7307):275–9.
Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99(9):1576–83.
Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003;8(2):94–9.
Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. PLoS One. 2013;8(10):e77404.
Lavis JN, Lomas J, Hamid M, Sewenkambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006;84(8):620–8.
Bosch-Capblanch X, Lavis JN, Lewin S, Atun R, Røttingen J-A, Dröschel D, et al. Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development. PLoS Med. 2012;9(3):e1001185.
Hanney SR, Gonzalez-Block MA. Yes, research can inform health policy; but can we bridge the 'Do-Knowing It's Been Done' gap? Health Res Policy Syst. 2011;9:23.
Lemay MA, Sa C. The use of academic research in public health policy and practice. Res Evaluat. 2014;23:79–88.
Brownson RC, Fielding JE, Maylan CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175–201.
Ham C, Hunter DJ, Robinson R. Evidence based policymaking: research must inform health policy as well as medical care. Brit Med J. 1995;310:71–2.
Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011;6(7):e21704.
Weiss C. Knowledge creep and decision accretion. Sci Commun. 1980;1(3):381–404.
Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004;26(1):75–106.
Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21.
Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5:41.
El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012;7:2.
Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009;20:70–5.
Trostle J, Bronfman M, Langer A. How do researchers influence decision makers? Case studies of Mexican policies. Health Policy Plan. 1999;14(2):103–14.
Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bedard PO. Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implement Sci. 2011;6:60.
Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, et al. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Adm Policy Ment Health. 2008;35(1–2):98–113.
Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S, et al. Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Adm Policy Ment Health. 2008;35(1–2):124–33.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
Oxman AD, Vandvik PO, Lavis JN, Fretheim A, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP) 2: Improving how your organisation supports the use of research evidence to inform policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S2.
Dobbins M, DeCorby K, Twiddy RN. A knowledge transfer strategy for public health decision makers. Worldviews Evid Based Nurs. 2004;1(2):120–8.
Ettelt S, Mays N. Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011;16 Suppl 2:48–60.
Evans BA, Snooks H, Howson H, Davies M. How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study. Implement Sci. 2013;8:17.
Gold M. Pathways to the use of health services research in policy. Health Res Educ Trust. 2009;44(4):1111–36.
Helmsey-Brown J. Facilitating research utilisation: a cross-sector review of research evidence. Int J Publ Sect Manage. 2004;17(6):534–52.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):2.
Squires JE, Estabrooks CA, O’Rourke HM, Gustavsson P, Newburn-Cook CV, Wallin L. A systematic review of the psychometric properties of self-report research utilization measures used in healthcare. Implement Sci. 2011;6(1):83.
McCaughan D, Thompson C, Cullum N, Sheldon TA, Thompson DR. Acute care nurses' perceptions of barriers to using research information in clinical decision-making. Issues Innov Nursing Prac. 2002;39(1):46–60.
Kothari A, Edwards N, Hamel N, Judd M. Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implement Sci. 2009;4:46.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.
Redman S, Turner T, Davies H, Haynes A, Williamson A, Milat A, et al. The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136–137:147–55.
The CIPHER Investigators. Supporting Policy In health with Research: an Intervention Trial (SPIRIT)-protocol for a stepped wedge trial. BMJ Open. 2014;4(7):e005293.
Ryan M, McIntosh E, Shackley P. Methodological issues in the application of conjoint analysis in health care. Health Econ. 1998;7:373–8.
Ryan M. Using conjoint analysis to take account of patient preferences and go beyond health outcomes: an application to in vitro fertilisation. Soc Sci Med. 1999;48:535–46.
Ryan M. A role for conjoint analysis in technology assessment in health care? Int J Technol Assess Health Care. 1999;15(3):443–57.
Ryan M, Hughes J. Using conjoint analysis to assess preferences for miscarriage management. Health Econ. 1997;6:216–73.
San Miguel F, Ryan M, McIntosh E. Applying conjoint analysis in economic evaluations: an application to menorrhagia. Appl Econ. 2010;32:823–33.
Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Multivariate Data Analysis. 6th ed. New Delhi: Pearson; 2006.
Carson RT, Louviere J. A common nomenclature for stated preference elicitation approaches. Environ Resource Econ. 2011;49:539–59.
Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: a user’s guide. Pharmacoeconomics. 2008;26(8):661–77.
Huckel Schneider C, Campbell D, Milat A, Haynes A, Quinn E. What are the key organisational capabilities that facilitate research use in public health policy? Public Health Res Pract. 2014;25(1).
Louviere J, Flynn TN, Carson RT. Discrete choice experiments are not conjoint analysis. J Choice Modelling. 2010;3(3):57–72.
Louviere J, Cong K, Flynn T, Turner T, Williamson A. Using discrete choice experiments to model experts’ decisions about effective organisational use of medical research evidence in medical policy decisions. Med Decis Making. 2016. In press.
Meijers JM, Maaike AP, Cummings GG, Wallin L, Estabrooks CA, Halfens RYG. Assessing the relationships between contextual factors and research utilization in nursing: systematic literature review. J Adv Nurs. 2006;55(5):622–35.
El-Jardali F, Lavis JN, Moat KA, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014;12:2.
Denis JL, Lomas J, Stipich N. Creating receptor capacity for research in the health system: the Executive Training for Research Application (EXTRA) program in Canada. J Health Serv Res Policy. 2008;13 Suppl 1:1–7.
Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10 Suppl 1:S1:35–48.
Mirzoev TN, Omar MA, Green AT, Bird PK, Lund C, Ofori-Atta A, et al. Research-policy partnerships - experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia. Health Res Policy Syst. 2012;10:30.
Ross S, Lavis J, Rodriguez C, Woodside J, Denis J. Partnership experiences: involving decision-makers in the research process. J Health Serv Res Policy. 2003;8 Suppl 2:26–34.
Haynes A, Brennan S, Carter S, O’Connor D, Huckel Schneider C, Turner T, et al. Protocol for the process evaluation of a complex intervention designed to increase the use of research in health policy and program organisations (the SPIRIT study). Implement Sci. 2014;9:113.
Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S14.
Orme D. Getting Started with Conjoint Analysis: Strategies for Product Design and Pricing Research. 2nd ed. Madison, Wisconsin: Research Publishers LLC; 2010.
Marshall D, Bridges JF, Hauber AB, Cameron R, Donnalley L, Fyie K, et al. Conjoint analysis applications in health – how are studies being designed and reported? Patient. 2010;3(4):249–56.
Thomas K, Allen S. The learning organisation: a meta-analysis of themes in literature. Learn Organ. 2006;13(2):123–39.
Ferlie E, Dopson SUE, Fitzgerald L, Locock L. Renewing policy to support evidence-based health care. Public Adm. 2009;87(4):837–52.
Greenhalgh T, Robert G, Bate P, Kyriakidou O, Macfarlane F, Peacock R. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.
Makkar SR, Brennan S, Williamson A, Turner T, Redman S, Green S. The development of SAGE: a tool to evaluate how policymakers’ engage with and use research in health policymaking. Res Evaluat. 2016. In press.
Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system to score research engagement actions by health decision makers. Health Res Policy Syst. 2015;13:22.
Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system of scoring policymakers’ use of research in policy and program development. Health Res Policy Syst. 2015;13:35.
Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88(4):444–83.
Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, et al. National policymakers speak out: are researchers giving them what they need? Health Policy Plan. 2011;26:73–82.
Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006;10:163–80.
Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J, Knowledge Transfer Study Group. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81(2):221–48. 171–2.
Buse K, Mays N, Walt G. Making health policy. Glasgow: Open University Press; 2005.
Koon ADRKD, Tran NT, Ghaffar A. Embedding health policy and systems research into decision-making processes in low- and middle-income countries. Health Res Policy Syst. 2013;11:30.
Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence for public health policy on inequalities: 1: The reality according to policymakers. J Epidemiol Community Health. 2004;58(10):811–6.
Albert M, Fretheim A, Maiga D. Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali’s essential medicines. Health Res Policy Syst. 2007;5:2.
Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A. Information transfer: what do decision makers want and need from researchers? Implement Sci. 2007;2:20.
Burris H, Parkhurst J, Adu-Sarkodie Y, Mayaud P. Getting research into policy - Herpes simplex virus type-2 (HSC-2) treatment and HIV infection: international guidelines formulation and the case of Ghana. Health Res Policy Syst. 2011;9 Suppl 1:S5.
Anderson M, Cosby J, Swan B, Moore H, Broekhoven M. The use of research in local health service agencies. Soc Sci Med. 1999;49(8):1007–19.
Danseco E, Sundar P, Kasprzak S, Witteveen T, Woltman H, Manion I. Are we there yet? Evaluation and the knowledge translation journey. J Can Acad Child Adolesc Psychiatry. 2009;18(1):7–15.
Beyer JM, Trice HM. The utilization process: a conceptual framework and synthesis of empirical findings. Adm Sci Q. 1982;27(4):591–622.
Gerhardus A, Kielmann K, Sanou A. The use of research for decision making in the health sector: the case of ‘shared care’ in Burkina Faso. In: Lessons in Research to Action and Policy. Geneva: COHRED, Document, The Council on Health Research for Development; 2000. p. 19–27.
Ginsburg LR, Lewis S, Zackheim L, Casebeer A. Revisiting interaction in knowledge interaction. Implement Sci. 2007;2:34.
Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7(4):239–44.
Jacobson N, Butterill D, Goering P. Consulting as a strategy for knowledge transfer. Milbank Q. 2005;83(2):299–321.
Jansen MW, De Leeuw E, Hoejimakers M, De Vries NK. Working at the nexus between public health policy, practice and research. Dynamics of knowledge sharing in the Netherlands. Health Res Policy Syst. 2012;10:33.
Kothari A, Birch S, Charles C. ‘Interaction’ and research utilisation in health policies and programs: does it work? Health Policy. 2005;71(1):117–25.
Lavis J. Research, public policymaking, and knowledge-translation processes: Canadian efforts to build bridges. J Contin Educ Heal Prof. 2006;26:37–45.
Lomas J. Connecting Research and Policy. ISUMA Canadian J Policy Res. 2000;Spring:140–4.
Mbonye AK, Magnussen P. Translating health research evidence into policy and practice in Uganda. Malar J. 2013;12:274.
Milat A, Laws R, King L, Newson R, Rychetnik L, Rissel C, et al. Policy and practice impacts of applied research: a case study analysis of the New South Wales Health promotion Demonstration Research Grants Scheme 2000–2006. Health Res Policy Syst. 2013;11:5.
Patera N, Wild C. Linking public health research with policy and practice in three European countries. J Public Health. 2013;21(5):473–9.
Rich R. Utilization of social science related information by federal policy-makers. Inquiry. 1975;12(3):239–45.
Whitehead M, Petticrew M, Graham H, Macintyre S, Bambra C, Egan M. Evidence for public health policy on inequalities 2: assembling the evidence jigsaw. J Epidemiol Community Health. 2004;58:817–21.
McGrath PJ, Lingley-Pottie P, Emberly DJ, Thurston C, McLean C. Integrated knowledge translation in mental health: family help as an example. J Can Acad Child Adolesc Psychiatry. 2009;18(1):30–7.
Campbell D, Donald B, Moore G, Frew D. Evidence check: knowledge brokering to commission research reviews for policy. Evidence Policy. 2011;7(1):97–107.
Lomas J. The in-between world of knowledge brokering. Br Med J. 2007;334(7585):129–32.
We wish to thank all the leaders of health organisations that agreed to complete ORACLe and contribute to its development and ongoing refinement. We acknowledge all the senior policymakers who provided valuable contributions to refining and enhancing the practical utility of the measure. We would finally like to thank the chief investigators of CIPHER, who conceptualized the development of ORACLe and other innovations designed to increase the use of research in policymaking. CIPHER is a Centre for Research Excellence and is a joint project of the Sax Institute; Australasian Cochrane Centre, Monash University; the University of Newcastle; The University of New South Wales; The University of Technology, Sydney; the Research Unit for Research Utilization, University of St Andrews and University of Edinburgh; and the University of Western Sydney.
The development of this measure was funded as part of the Centre for Informing Policy in Health with Evidence from Research (CIPHER), an Australian National Health and Medical Research Council Centre for Research Excellence (APP1001436), and administered by the University of Western Sydney. The Sax Institute receives a grant from the NSW Ministry of Health. The Australasian Cochrane Centre is funded by the Australian Government through the National Health and Medical Research Council (NHMRC). AW holds an NHMRC Post-Doctoral Fellowship (ID: 510 391).
SM led the production and drafting of the manuscript, and contributed to reliability testing of the measure. AW contributed to the theoretical direction of the manuscript and analyses, undertook reliability testing, and provided ongoing input into the production of the manuscript. TT was the lead developer of the ORACLe interview schedule and marking guide. JL undertook the statistical analyses that led to the production of the ORACLe scoring system. SR conceptualized the measure and the overarching theoretical approach of the study and manuscript. SB, AH, and SG contributed to the development and piloting of the ORACLe interview and marking guide and provided theoretical and structural guidance to the production of the manuscript. All named authors contributed substantially to and approved the final manuscript.
SPSS Instructions and syntax. These are the instructions and SPSS syntax to calculate domain scores and total scores for ORACLe. (DOCX 29 kb)
Data frame. This file provides the data frame in .csv format, to enter raw data following scoring of the ORACLe interview. (CSV 162 bytes)
About this article
- Discrete choice experiments
- Health policy
- Knowledge translation
- Research use