Skip to main content

Evaluation of a training program for medicines-oriented policymakers to use a database of systematic reviews

The Original Article was published on 29 July 2016

Abstract

Background

Suboptimal prescribing and medications use is a problem for health systems globally. Systematic reviews are a comprehensive resource that can help guide evidence-informed decision-making and implementation of interventions addressing such issues; however, a barrier to the use of systematic reviews is their inaccessibility (due to both dispersion across journals and inaccessibility of content). Publicly available databases, such as Rx for Change, provide quick access to summaries of appraised systematic reviews of professional and consumer-oriented interventions to improve prescribing behaviour and appropriate medication use, and may help maximise the use of evidence to inform decisions. The present study aims to evaluate a training program to improve attitudes towards, confidence in skills, intentions to use, and use of systematic review evidence contained within Rx for Change.

Methods

Guided by the Knowledge to Action framework, a training program with content customised to local provider and consumer contexts was developed with knowledge user input. The training program consisted of a 6 minute information video, a 1 hour workshop with hands-on, interactive and didactic components, and two post-training reminders. Forty-nine people from five medicines-focused organisations in Canada and Australia attended one of six workshops. Participants were surveyed immediately pre and post and 3 months after training to evaluate their attitudes towards, confidence in skills, intentions to use, and use of Rx for Change, and attitudes towards and confidence in skills for using evidence for decision-making. Analyses for differences for each of the outcomes at three time points (pre, post and 3 months after training) was performed using a random effects model.

Results

Immediately post-training, there were higher respondent attitudes towards Rx for Change (mean increase = 0.54 out of 5, 95% CI, 0.18–0.83, P < 0.005); intention to use Rx for Change (0.53, 95% CI, 0.21–0.86, P < 0.005); confidence in skills for using Rx for Change (2.08, 95% CI, 1.74–2.42, P < 0.005); and confidence in skills for using evidence in policy decision-making (0.50, 95% CI, 0.22–0.77, P < .005) compared to pre-training. Confidence in skills for using both Rx for Change and evidence were maintained 3 months after training (both P < 0.005).

Conclusions

Participants of this training program reported sustained improvements in their confidence in skills for using evidence in policy decision-making. This may have important implications for uptake of systematic review evidence promoting improved prescribing and medication use.

Peer Review reports

Background

Health systems globally are focused on addressing inappropriate prescribing by healthcare professionals (providers) and inappropriate use of medications by patients (consumers) to maximise patient outcomes and minimise healthcare costs [1, 2]. Sub-optimal medication use and prescribing practices exist across most high-income countries [1, 3], with medication errors, mistakes and laboratory error in Australia and Canada being reported for approximately 27% and 30% of patients, respectively [1]. Poor prescribing practices and medication errors are associated with negative health outcomes, including increased morbidity [4] and preventable hospital admissions [5], increased rates of antibiotic resistant organisms due to overprescribing of antibiotics [6] and high rates of preventable adverse drug events [7]. Suboptimal medication use by patients is also problematic, with adherence to medicines rates of approximately 50% [2]. Problems affecting consumers’ use of medicines include high rates of errors by prescribers and patients, preventable adverse effects, and difficulties in communication and transitional care [8].

Systematic reviews comprehensively synthesise available evidence about potential policies and interventions to improve medication use but knowledge users find it difficult to access relevant systematic reviews and make sense of research evidence [9]. Knowledge users also face significant challenges to using systematic review evidence. Some of the most frequently identified barriers by knowledge users include constraints on time and resources, and resistance towards change [10]. Knowledge users report that increasing the use of research evidence for policy decision-making requires more accessible and efficient systems as well as approaches to ensuring accountability for evidence use [10]. There are also barriers related to the context in which policy decision-making occurs, including varying pressure from multiple interest groups, limited time for decision-making and the use of research evidence being one of many types of information required for policy decision-making [9]. Further barriers to evidence-informed decision-making include lack of senior-level leadership and support for using evidence, politicised (as opposed to evidence) decision-making and constant shifting of attention across changing priorities making a focus on evidence impossible [11].

In response to these challenges, a range of innovative knowledge translation resources have been developed to facilitate the use of evidence to inform decision-making, including simplified frameworks developed with contribution from the policy sector to better understand how any given approach to increasing the use of evidence can be applied and understood [12], websites to facilitate research engagement by policymakers [13], and databases to facilitate access to evidence [14, 15]. The Cochrane Collaboration has developed training courses for systematic review use specific to public health [16] and overviews of systematic reviews have been published to facilitate summaries of evidence [8, 17].

One such freely available online database is Rx for Change, an evidence-based resource of appraised and summarised evidence on the effectiveness of interventions targeting healthcare professionals and patients to improve prescribing and medication use [14] (http://www.rxforchange.ca). Rx for Change is a collaboration between researchers at the Cochrane Effective Practice and Organisation of Care (EPOC) Group and the Cochrane Consumers and Communication Review Group (CC&CRG) primarily funded by the Canadian Agency for Drugs and Technologies in Health [14]. Between 2007 and April 2014, the database was periodically updated to ensure that this evidence remains relevant and timely. To populate the database, potentially relevant systematic reviews of interventions targeting professionals (e.g. audit and feedback) and consumers (e.g. acquiring skills and competencies) were identified by sensitive searches of major databases including Medline, The Cochrane Library, and EMBASE. Full text copies of potentially relevant reviews were retrieved and assessed for relevance [8]. Each review was quality-assessed using AMSTAR, an instrument validated for assessing quality of systematic reviews [1820]. High and medium quality reviews (AMSTAR score > 4) were data-extracted and summarised using a structured process to highlight key characteristics and evidence relevant to optimal use of medicines decision-making. To facilitate the ease of finding and using information, the Rx for Change database is organised according to intervention type (e.g. education, reminders, etc.) and category (e.g. professional, consumer, organisational, financial, regulatory) using taxonomies developed by EPOC and CC&CRG [21]. Currently, Rx for Change contains more than 300 summarised reviews that synthesise findings from thousands of individual studies, spanning all conditions and diseases, and evaluating interventions targeting healthcare provider or consumer behaviours to improve medication prescribing or use.

The primary objective of Rx for Change is to maximise the awareness and use of appraised systematic reviews of professional and consumer-oriented interventions to improve prescribing behaviour and appropriate medication use. There has been a steady trend of increased site visits and time spent on the site since its inception and the database has received positive feedback from healthcare providers, researchers and policymakers [14]. However, despite efforts to increase the use of Rx for Change (e.g. an on-line tutorial), anecdotal evidence suggests the need for further training to maximise use of the database for evidence-informed decisions.

Objectives

The aim of this project was to develop and evaluate a training program for Rx for Change in collaboration with relevant knowledge users. The development of the training was guided by the Knowledge to Action framework [22]. This framework outlines the necessary steps to bridge the gap between knowledge and the application of that knowledge in healthcare settings. The framework steps emphasise collaboration with knowledge users and includes adapting knowledge to the local context, understanding local barriers and tailoring interventions as needed. The comprehensive findings of the development stage of the training intervention (Phase 1), which involved interviewing 16 knowledge users from a variety of settings to identify the needs of the organisations and to guide the content, delivery and tailoring of the workshop, are presented elsewhere [23]. The focus of the current study (Phase 2) was to evaluate the resultant training program in terms of its impact on subsequent attitudes towards, confidence in skills, intentions to use, and use of Rx for Change, as well as attitudes towards and confidence in skills for using evidence in general.

Methods

This study used a pre–post design with a 3-month follow-up. Five knowledge user organisations responsible for the development of programs to improve prescribing and medication use contributed to this project, three from Canada (provider focused) and two from Australia (consumer focused). All five of these organisations expressed an interest in improving their use of Rx for Change and a need for additional training. The three Canadian sites focused on evidence for organisational, professional and regulatory interventions, which are consistent with the primary interest (and expertise) of EPOC. Consistent with the primary interests (and expertise) of CC&CRG, the knowledge user organisations from the two Australian sites focus primarily on evidence for consumer-oriented interventions. The organisations were a mix of national (n = 2), provincial (n = 3), and government (n = 4) or non-government (n = 1) sites. All programs provided support and information that was needs driven, with the Canadian sites focusing on the ultimate goal to influence local provider behaviour, and the Australian sites focusing on consumer behaviour.

Stage 1

Training development

Key informants (n = 16) from each knowledge user organisation participated in interviews to shape the content and format for the training programs so that they were (1) tailored to their needs and (2) designed to address provider or consumer issues and barriers across the two countries. The semi-structured interviews included questions on their current use of evidence for decision-making, barriers and facilitators to using evidence as well as the Rx for Change database, and their needs and preferences for training. The interviews were transcribed and analysed using a directed content analysis [24]. The main themes from the interviews identified the need for the workshops to cover specific knowledge (e.g. definition of a systematic review), skills for both navigating the database as well as how to incorporate evidence into decision-making, activities to promote the routine use of Rx for Change, and the need to address a variety of preferences for how the training should be delivered (tailoring). There was also an interest in including practical and locally important examples on incorporating evidence within their workplace context. In order to respond to this need and the desire to learn how to use evidence for policy decision-making, we developed a workshop that incorporated content from an existing framework for using evidence in policy decision-making – the SUPPORT Tool process [25]. The SUPPORT Tool process includes the following steps: identifying the problem and/or question at hand, What do I want to achieve?; identifying the available options to address the problem; identifying the available resources; decision-making; and implementation of most suitable options. The developed training program consisted of (1) a 6 minute information video to be viewed prior to the workshop (the video provided basic background information about the database and systematic reviews); 2) a 60 minute face-to-face workshop with didactic, hands-on and interactive components (these components included a 30 minute skills demonstration and practice with the database that took participants through the SUPPORT Tool process using a local and current prescribing-based issue in the organisation); and (3) post workshop reminders at 1 and 3 months following the workshop to remind participants about Rx for Change and encourage its use. Following the development of the training, a final feasibility check for the proposed training was completed with a manager from each of the five participating organisations. A more comprehensive presentation of the findings of the interviews and details of intervention design that resulted from this initial stage of the project is presented elsewhere [23].

Stage 2

Training program implementation and evaluation

Participants were recruited from each of the knowledge user organisations via email invitations distributed to one key contact within each of the organisations and forwarded to potentially interested staff. The training was delivered to the five knowledge user organisations (training was provided twice to one organisation with a significant geographic range). Participants were surveyed immediately pre and post and 3 months after training. The surveys included questions on self-reported use of Rx for Change in the 3 months prior to the workshop and in the 3 months following the workshop (i.e. how many times in the past three months had they used Rx for Change), attitudes towards using evidence and Rx for Change for policy decision-making (i.e. their rating on a scale from 1–5 as to the appropriateness, importance and usefulness of both evidence and Rx for Change), confidence in skills (i.e. rating on a scale of 1–5 for level of confidence in six relevant skills for using evidence and 11 relevant skills for using Rx for Change), and intention to use Rx for Change (i.e. global rating of intention on a scale from 1–5). Surveys were developed for the study and were piloted in terms of relevancy, clarity and completeness.

Given the large number of variables in the survey (i.e. 24), the problems associated with multiple testing [26], and the fact that our survey contained many similarly phrased questions for attitudes and skills, we planned a principal component analysis of groupings of questions in order to determine if the data supported the use of a mean score for each of these groupings. The resulting principal component analysis indicated that the groupings of questions were highly correlated – all corrected item-total correlations were above the recommended threshold of r > 0.30 [27]. We therefore, used these groupings, and the associated mean scores, for the analysis under the four labels of (1) attitudes towards Rx for Change, (2) confidence in skills for using Rx for Change, (3) attitudes towards using evidence in decision-making, and (4) confidence in skills for using evidence for decision-making. Table 1 provides a detailed summary of this analysis. Combined with intention (analysed as a single variable), this gave us five outcomes to test.

Table 1 Results of the principal component analysis

Data analyses

The survey responses were analysed descriptively and with frequencies and counts. Differences before and after training as well as between provider and consumer oriented organisations were analysed using a random effects model, with organisational unit used as a random effect and robust variance estimates used for each of the five outcome measures, each assessed on a scale from 1 to 5 time period (pre, post and 3 month follow-up) was the exposure variable, with the pre-training time point being used as the reference point. As participant responses were anonymised at the point of collection, repeated measures analyses could not be performed. To formally test if the effect of the intervention may vary between countries or organisations, interaction terms were fitted between time points and these clusters. A Bonferroni correction for multiple analyses (10 comparisons, two each for five outcome variables) was used, resulting in a P value of < 0.005 being considered as significant. All statistical analyses were undertaken using Stata (v13.1, College Station, Tx).

Results

A total of 49 participants attended the workshops and submitted the pre-workshop survey (one with incomplete responses). For the post-training survey and 3-month follow-up, we had 47 and 28 participants submit the survey, respectively (two and one with incomplete responses, respectively). Figure 1 provides a detailed summary of countries, organisations and surveys completed for all time points. The participant sample was comprised of 79% females (38 female, 10 male) and 88% full-time workers (42 full-time, 6 part-time). Of the participants attending the workshops, 60% (29/48) had previously heard about the Rx for Change database.

Fig. 1
figure 1

Number of participant responses for pre, post and follow-up surveys at each site

Self-reported use of Rx for Change

Prior to the training, 78% (38/49) of participants indicated that they never used Rx for Change in the preceding 3 months. When asked to consider their use of Rx for Change in the 3 months since the training, only 39% (11/28) indicated that they never used Rx for Change. The number of people indicating that they used Rx for Change in the previous 3 months at least 2–3 times per month increased from 8% (4/49) during the 3 months prior to the training to 32% (9/28) during the 3 months after the training. Table 2 shows the full summary of self-reported use of Rx for Change.

Table 2 Self-reported use of Rx for Change in 3 months pre and post the workshop

Statistical analysis for group differences

Differences in pre to post and pre to 3 month follow-up

Results indicated that confidence in skills to use Rx for Change and confidence in skills to use evidence were statistically significantly greater than pre training at both the post and 3 month follow-up time points (all P < 0.005). Attitudes towards using evidence for decision-making were not significantly different from pre training at post training (P = 0.635) or at 3 months follow-up (P = 0.662). Both attitudes towards, and intentions to use, Rx for Change increased at post training (both P < 0.005), but not at follow-up (P = 0.445 and P = 0.908, respectively). Table 3 provides a summary of the analysis for group differences. There was no evidence that either the organisational unit or country modified these associations (all P values for interactions > 0.01), although there was limited power to detect such effects.

Table 3 Statistical Analysis for Group Differences

Discussion

We developed user-informed training to increase the use of an online database of systematic reviews (Rx for Change) that policymakers could use to inform the design of programs to improve medication use. We tested this training for effects on attitudes towards, confidence in skills, intentions to use, and use of Rx for Change as well as attitudes towards and confidence in skills for using evidence for decision-making. We measured our outcomes pre and post the training workshop as well as at 3 months post training. Pre–post training effects were found for attitudes towards, and confidence in skills for using Rx for Change, as well as confidence in skills for using evidence for decision-making. More importantly, effects on confidence in skills for using Rx for Change and for using evidence for decision-making were maintained at 3 months post training. Therefore, the results indicate that this workshop, developed with input from knowledge users and using an evidence-based process of using evidence for policy decision-making as a guide, has potential to improve immediate and long-term confidence in skills in using this particular database, and in using research evidence in general.

We combined several evidence-based approaches in the design of our training. We used a workshop that incorporated interactive components [28], content tailored to the needs and preferences of the knowledge users [29], reminders [30], and the integration of knowledge user input [22]. Systematic review evidence indicates that each of these approaches has been shown to have an effect on behaviour, albeit modest. Our training appears to have resulted in increased confidence in skills and increased use of Rx for Change, although it did not have an effect on intention, thus reducing the likelihood of sustaining the increased use of Rx for Change [31]. The use of a 1-hour workshop is a common approach to education and behaviour change [32] and the leads in our knowledge user organisations had a preference for this style of training as it fits with demands placed on staff in these organisations. While more understanding is needed as to dosage and the types of educational activities for maximising effects, it continues to be worthwhile to test the effects of short educational training interventions and ensure follow-up measures to optimise our understanding of the maintenance of any gains made.

There were no significant differences observed in attitudes towards using evidence for decision-making at any time point. Given the high pre-training means for this variable (4.74 on a 5 point scale), there was either little room for improvement (suggesting a ceiling effect) or our scale was not able to detect differences. Attitudes towards using evidence tend to be positive, even if behaviour is not consistent with this attitude. Programs of this nature should probably not solely focus their efforts on changing attitudes towards using evidence but rather behaviours around the use of evidence. We did not observe any differences in intentions to use Rx for Change at the 3 month follow-up. The reasons for this are unclear, but could include perceptions of the value and/or functionality of the database itself or broader issues related to the use of evidence in general. Although we did incorporate the needs and preferences of the knowledge users into the training, we focused on this local input, perhaps at the expense of considering other barriers to change such as organisational culture [33]. While a 1 hour workshop can only cover a limited amount of content, future efforts to address knowledge user needs as well as other barriers identified in the literature is warranted.

Forty percent of the sample indicated that they had not used Rx for Change since the training, yet there were fewer participants who indicated that they did not use Rx for Change in the 3 months after training than in the 3 months prior to training. We were unable to explore if participants had a need to use Rx for Change since the training, making it difficult to fully interpret our Rx for Change use data. However, the results suggest that with no change in their intentions to use Rx for Change, a decrease in their attitudes towards it, and confidence in skills potentially declining, it is likely that use rates will revert to pre-training levels. This is often the case in knowledge translation interventions [34], and methods for promoting knowledge translation intervention sustainability are needed.

Confidence in skills for using evidence improved after the workshop, with the improvement being maintained 3 months later. This is an important result, particularly for a 1 hour workshop. Similar to confidence in skills for using evidence, the workshop improved perceptions of confidence in skills for using Rx for Change, and this was maintained at 3 months. However, there is a trend towards declining confidence in skills. This might signal that additional skill development activities might be required to support the ongoing use of a database like Rx for Change.

While consumer-based and provider-based organisations from different countries were used in the study, the staff within these organisations undertook similar duties – outreach to others to improve use of evidence. The ‘others’ were different but the people we targeted with the intervention were not that different. If, for example, physicians had been the targets of the education, differences would have been expected. This finding might improve the generalisability of our findings to a broad array of policy-based groups. However, caution in generalising is warranted given our context of only two countries and both being of high income.

We guided our research using the Knowledge to Action framework [22], which gave us a structure that underpinned the development, pilot and evaluation of this multi-faceted training program to improve use of Rx for Change [23]. Another strength of this study is that the research builds on international collaborations between Canada and Australia by involving five knowledge user groups who have expressed the interest in Rx for Change training, three from Canada and two from Australia. By involving targeted user audiences we can better understand the needs of individuals or groups concerned with the identification and/or implementation of interventions used to change provider (health practitioner/professional) and consumer (patient) behaviours as it specifically relates to medicines prescribing or practice. Limitations of this study include a relatively small number of sites and participants. In addition, the 3 month follow-up data only included 28 of the 49 participants who attended the workshop, resulting in an even smaller sample at the follow-up, thus increasing the potential for bias and reducing power to observe effects. While we did include a 3 month follow-up, the timing of this follow-up was short, and no further follow-up took place. The most significant limitations were the design of the study (a pre–post measurement with no control group and no ability to measure changes in responses over time within individuals) and the use of a survey that was designed specifically for the study. While we did pilot test the survey for relevancy, clarity and completeness, we did not undertake additional measures of reliability and validity. Our conclusions are therefore tentative and additional experimental research is needed.

Conclusion

The evaluation component aids understanding and future activities to optimise use of Rx for Change. Participants of this tailored training program reported sustained improvements in confidence in skills for using evidence in policy decision-making. This may have important implications for uptake of systematic review evidence to promote improved prescribing and medication use.

Abbreviations

CC&CRG:

Cochrane Consumers and Communication Review Group

EPOC:

Effective Practice and Organisation of Care

References

  1. Schoen C, Osborn R, Huynh PT, Doty M, Zapert K, Peugh J, Davis K. Taking the pulse of healthcare systems: experiences of patients with health problems in six countries. Health Aff. 2005;10:1377.

    Google Scholar 

  2. Haynes RB, Ackloo E, Sahota N, McDonald HP, Yao X. Interventions for enhancing medication adherence. Cochrane Database Syst Rev. 2008;2:CD000011.

    PubMed  Google Scholar 

  3. Schoen C, Osborn R, How SK, Doty MM, Peugh J. In chronic condition: experiences of patients with complex health care needs, in eight countries, 2008. Health Aff. 2009;28(1):w1–16.

    Article  Google Scholar 

  4. Avery AJ, Sheikh A, Hurwitz B, Smeaton L, Chen Y-F, Howard R, Cantrill J, Royal S. Safer medicines management in primary care. Br J Gen Pract. 2002;52:S17–21.

    PubMed  PubMed Central  Google Scholar 

  5. Winterstein AG, Sauer BC, Hepler CD, Poole C. Preventable drug-related hospital admissions. Ann Pharmacother. 2002;36(7–8):1238–48.

    Article  PubMed  Google Scholar 

  6. Arnold SR, Straus SE, Arnold S. Interventions to improve antibiotic prescribing practices in ambulatory care. Cochrane Database Syst Rev. 2005;4:CD003539.

    PubMed  Google Scholar 

  7. Gandhi TK, Weingart SN, Borus J, Seger AC, Peterson J, Burdick E, Seger DL, Shu K, Federico F, Leape LL. Adverse drug events in ambulatory care. N Engl J Med. 2003;348(16):1556–64.

    Article  PubMed  Google Scholar 

  8. Ryan R, Santesso N, Hill S, Lowe D, Kaufman C, Grimshaw J. Consumer‐oriented interventions for evidence‐based prescribing and medicines use: an overview of systematic reviews. Cochrane Database Syst Rev. 2011;5:CD007768.

    PubMed  Google Scholar 

  9. Lavis JN. Research, public policymaking, and knowledge‐translation processes: Canadian efforts to build bridges. J Contin Educ Health Prof. 2006;26(1):37–45.

    Article  PubMed  Google Scholar 

  10. Ellen ME, Léon G, Bouchard G, Ouimet M, Grimshaw JM, Lavis JN. Barriers, facilitators and views about next steps to implementing supports for evidence-informed decision-making in health systems: a qualitative study. Implement Sci. 2014;9(1):179.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Bowen S, Erickson T, Martens PJ, Crockett S. More than “using research”: the real challenges in promoting evidence-informed decision-making. Healthcare Policy. 2009;4(3):87.

    PubMed  PubMed Central  Google Scholar 

  12. Colquhoun H, Leeman J, Michie S, Lokker C, Bragge P, Hempel S, McKibbon KA, Peters GJ, Stevens KR, Wilson MG, et al. Towards a common terminology: a simplified framework of interventions to promote and integrate evidence into health practices, systems, and policies. Implement Sci. 2014;9:51.

    PubMed  PubMed Central  Google Scholar 

  13. Makkar SR, Gilham F, Williamson A, Bisset K. Usage of an online tool to help policymakers better engage with research: web CIPHER. Implement Sci. 2015;10:56.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Weir MC, Ryan R, Mayhew A, Worswick J, Santesso N, Lowe D, Leslie B, Stevens A, Hill S, Grimshaw JM. The Rx for Change database: a first-in-class tool for optimal prescribing and medicines use. Implement Sci. 2010;5:89.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kirkpatrick CM, Roughead EE, Monteith GR, Tett SE. Consumer involvement in Quality Use of Medicines (QUM) projects–lessons from Australia. BMC Health Serv Res. 2005;5:75.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Pettman TL, Armstrong R, Jones K, Waters E, Doyle J. Cochrane update: building capacity in evidence-informed decision-making to improve public health. J Public Health. 2013;35(4):624–7.

    Article  Google Scholar 

  17. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001;39(8 Suppl 2):II2–45.

    CAS  PubMed  Google Scholar 

  18. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007;2(12):e1350.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

    Article  PubMed  Google Scholar 

  21. Lowe D, Ryan R, Santesso N, Hill S. Development of a taxonomy of interventions to organise the evidence on consumers’ medicines use. Patient Educ Couns. 2011;85(2):e101–7.

    Article  PubMed  Google Scholar 

  22. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26(1):13–24.

    Article  Google Scholar 

  23. Colquhoun H, Helis E, Lowe D, Belanger D, Hill S, Mayhew A, Taylor M, Grimshaw J. Development of training for medicines-oriented policymakers to apply evidence. Health Res Policy Syst. 2016;14:57.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    Article  PubMed  Google Scholar 

  25. Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009;7 Suppl 1:S14.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Dunn OJ. Multiple comparisons among means. J Am Stat Assoc. 1961;56(293):52–64.

    Article  Google Scholar 

  27. Nunnally J, Bernstein I. Psychometric Theory. 3rd ed. New York: MacGraw-Hill; 1994.

    Google Scholar 

  28. Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, Davis D, Odgaard-Jensen J, Oxman AD. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009;2(2):CD003030.

    PubMed  Google Scholar 

  29. Cheater F, Baker R, Gillies C, Hearnshaw H, Flottorp S, Robertson N, Shaw EJ, Oxman AD. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2005;3:CD005470.

    PubMed  Google Scholar 

  30. Arditi C, Rège‐Walther M, Wyatt JC, Durieux P, Burnand B. Computer‐generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2012;12:CD001175.

    PubMed  Google Scholar 

  31. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179–211.

    Article  Google Scholar 

  32. Scott SD, Albrecht L, O’Leary K, Ball G, Hartling L, Hofmeyer A, Jones CA, Klassen TP, Kovacs Burns K, Newton AS. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:70.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Dobbins M, Cockerill R, Barnsley J, Ciliska D. Factors of the innovation, organization, environment, and individual that predict the influence five systematic reviews had on public health decisions. Int J Technol Assess Health Care. 2001;17(4):467–78.

    CAS  PubMed  Google Scholar 

  34. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Katrina Sullivan for her support during the development of the grant associated with this project. We would also like to thank Julia Worswick for her expertise related to the Rx for Change database.

Funding

This study was funded by a Canadian Institutes of Health Research Knowledge to Action grant awarded Oct 2011 (FRN# 122163).

Availability of data and materials

The datasets generated and/or analysed during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

HC, AM, DB, SH, and JMG contributed to the conception and design of the study. In addition, HC and JMG contributed to the acquisition, analysis and interpretation of data. EH and DL contributed to the management of the study, the delivery of the intervention, as well as the acquisition, analysis and interpretation of data. DB, SH, AM and MT contributed to the interpretation of data. HC drafted the manuscript. All authors contributed edits to, read and approved the final version of the manuscript.

Competing interests

Two of the authors (BE, DB) worked at the Canadian Agency for Drugs and Technology in Health (CADTH) at the time of this study. One author (EH) is presently working at CADTH but was not at the time of the study. CADTH was the primary funder of Rx for Change, an online publicly available database of systematic reviews. Other contributing partners to Rx for Change included the Canadian Institutes of Health Research, KT Canada, Ottawa Hospital Research Institute, the Cochrane Effective Practice and Organisation of Care group, and the Cochrane Consumers and Communications Review Group. This study develops an intervention for policymakers to use this database of systematic reviews and was funded by the Canadian Institutes of Research. JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Ethics approval was sought and obtained through Ottawa Hospital Research Ethics Board’s and La Trobe University Human Ethics Committee (#20130010-01H).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Heather L. Colquhoun.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Colquhoun, H.L., Lowe, D., Helis, E. et al. Evaluation of a training program for medicines-oriented policymakers to use a database of systematic reviews. Health Res Policy Sys 14, 70 (2016). https://doi.org/10.1186/s12961-016-0140-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-016-0140-1

Keywords