- Open Access
- Open Peer Review
RE-AIMing conferences: evaluating the adoption, implementation and maintenance of the Rick Hansen Institute’s Praxis 2016
Health Research Policy and Systems volume 17, Article number: 39 (2019)
In April 2016, the Rick Hansen Institute (RHI) hosted an innovative, 2-day conference called Praxis 2016. RHI aimed to bring together a diverse group of stakeholders to develop solutions for overcoming the challenges of translating spinal cord injury (SCI) research into practice. To understand the impact of Praxis, RHI funded an independent team to evaluate Praxis at the individual and setting level using the RE-AIM framework. Individual-level findings are published elsewhere. The aim of this evaluation is to report on the impact of Praxis at the setting level in terms of its adoption, implementation and maintenance.
Data sources included interviews with attendees (n = 13) and organisers (n = 9), a fidelity assessment conducted at the conference, and observation notes provided by the evaluation team.
Main findings indicated that the Praxis model was adopted by organisers and attendees, implemented by RHI as intended, and has the potential for long-term maintenance. Lessons learned highlighted the importance of including SCI community members throughout the entire process from development to post-conference activities as well as in the research process, the value of facilitation and fostering interactive problem solving, and the need to identify leadership and funds to foster long-term efforts.
Developing and implementing a solutions-focused conference that brings together a diverse group of SCI stakeholders was challenging and rewarding for attendees and organisers. Other domains could learn from, adopt and build on the Praxis 2016 approach to address research-to-practice gaps.
It is estimated that traumatic spinal cord injury (SCI) costs the Canadian government approximately $2.7 billion per year, with an estimated average lifetime cost per individual of $1.5 million for incomplete paraplegia to $3.0 million for complete tetraplegia [1, 2]. While funding is provided to conduct research to enhance the lives of people with SCI, few research findings are translated into practice . Indeed, the 17- to 20-year gap between bench-to-bedside and bedside-to-real world application is so difficult to overcome, that they are often referred to as the “valleys of death” [3, 4].
The Rick Hansen Institute (RHI) is a Canadian-based, not-for-profit organisation that is committed to accelerating knowledge translation (KT) of discoveries and best practices into improved outcomes for people with SCI . In April 2016, RHI developed and hosted an innovative conference called Praxis 2016 (https://rickhanseninstitute.org/work/praxis/model/88-rhi-praxis-model-no-comment-allowed/448-praxis-2016-conference). Praxis had three primary goals, namely to (1) develop a shared understanding and synergy concerning KT and desired outcomes; (2) foster the sharing of real-world experience between attendees, and (3) identify actions to overcome the barriers and challenges to achieve KT. In terms of format, RHI aimed to change the mould of traditional scientific meetings. To foster interdisciplinary thinking, RHI invited a diverse group of adults living with SCI, researchers, clinicians, funding organisations, regulators, policy-makers and industry partners to attend the conference. Rather than following a didactic format, the conference included presentations on speakers’ translational lessons learned, interactive panel discussions, and working table discussions . All conference topics were first presented by three to four expert speakers per topic area. Rather than following a traditional conference style, speakers were asked to outline their translational lessons learned. Speakers were then asked to engage in a panel discussion about their lessons learned, which was moderated by the conference facilitators. Finally, a working table session was held in which attendees worked in assigned groups of 10 to develop solutions to the barriers outlined by the speakers. To guide future KT efforts in SCI research, strategies, opportunities, tactics and recommendations developed during Praxis were to be synthesised into a conference report  and action plan .
To learn from, improve and inform future KT initiatives, RHI funded an independent evaluation team to examine the impact of Praxis. The evaluation was guided by the RE-AIM evaluation framework, which captures five factors associated with measuring impact – Reach, Effectiveness, Adoption, Implementation and Maintenance . Reach and effectiveness were conceptualised as individual-level factors, adoption and implementation as setting-level factors, and maintenance as both an individual and setting level factor. This paper focuses on setting-level factors (i.e. adoption, implementation and maintenance). Adoption refers to the proportion and representativeness of settings and/or groups that are willing to adopt the initiative. Implementation examines the extent to which the initiative was delivered as intended. Maintenance refers to the extent to which the initiative becomes part of an organisation or groups’ long-term practices or policies [8, 9]. By examining these setting-level factors, other teams can learn about the successes and challenges of developing and implementing large-scale, event-based KT initiatives.
A separate paper reported on the individual-level impact of Praxis on attendees in terms of reach, effectiveness and maintenance. Overall, findings indicated that Praxis reached a wide range of attendees, but further representation from the SCI community was needed. Praxis improved attendees’ knowledge about KT barriers and solutions and increased the number of groups attendees believed they needed to work with to translate research. Given that setting-level factors can help to explain and translate individual-level successes and failures [9, 10], this paper reported on the impact of Praxis at the setting level (i.e. the RHI and Programme Advisory Committee (PAC)) in relation to adoption, implementation and maintenance. Specifically, we examined how speakers and organisers adopted the Praxis 2016 model, the extent to which Praxis 2016 was implemented as intended, and how Praxis could have a long-term impact in the SCI field.
Praxis 2016 consisted of a 3-day conference held in Vancouver, BC (April 25–27, 2016). A detailed description of the conference is provided in Additional file 1.
Setting-level data were obtained from five different sources, as follows: (1) semi-structured interviews with attendees; (2) semi-structured interviews with organisers, referring to RHI staff and the PAC; (3) fidelity assessment; (4) the evaluation team’s notes; and (5) observations of teamwork at working table sessions. Cost-effectiveness data were not available. Given this project is a programme evaluation, ethics approval was not required; however, informed consent was obtained from all interview participants and raw data were only accessed by members of the evaluation team. Procedures associated with data collection and data analyses for each source are outlined in Additional file 1.
Findings from all five data sources were used to evaluate Praxis 2016 in terms of the RE-AIM domains of adoption, implementation and maintenance. Data sources associated with each dimension are outlined in Table 1.
All interview guides are provided in Additional file 2. Interviews were audio-recorded, transcribed verbatim and subsequently analysed using both deductive and inductive approaches. First, one evaluator (KB) familiarised herself with the data by reading the transcription while listening to the recording. KB then deductively analysed the data into higher-order themes of adoption, implementation and maintenance (i.e. the ‘AIM’ in RE-AIM). Specifically, KB identified blocks of text from the interview transcripts that were relevant to the adoption, implementation or maintenance of Praxis 2016. Once the 22 transcripts were initially organised deductively into higher-order themes, KB then inductively analysed the data to search for lower-order themes within each of the higher-order themes. For this part of the analysis, we followed Braun and Clarke’s six-phase process for thematic analysis [11, 12]. Lower-order themes were coded through a collaborative and iterative process involving three evaluators (KB, HG, SS). Specifically, the definitions and names (labels) of the lower-order themes were discussed over the course of seven meetings (approximately 13 h). To finalise the analysis, each higher- and lower-order theme was given a description to ensure that each was distinct from the rest of the themes in the analysis.
Seven evaluators used a fidelity assessment tool based on the conference agenda to evaluate implementation. All session tasks and goals were marked as either complete or incomplete by all seven evaluators. If > 50% of the evaluators marked the task or goal as complete, it was considered completed. A percentage completion score (i.e. the percentage of tasks completed) was also calculated for each session on completion of all goals.
All evaluation comments collected from the fidelity assessment tool were transcribed verbatim in an Excel sheet. Only comments related to implementation were thematically analysed. Consistent with Braun and Clarke’s [11, 12] recommendations, one researcher (KB) first assigned codes to each comment. She then reviewed the codes to identify key themes and subsequently definitions for each theme. Finally, the researcher worked with the lead evaluators (HG and SS) to refine themes.
Working table teamwork scores were assessed using a modified version of the Jefferson Teamwork Observation Guide  (JTOG) (Additional file 3). JTOG scores were calculated by creating a sum of all JTOG competencies for each table (maximum score: 56). To allow for descriptive comparisons, mean, mode and ranges were calculated for the conference overall as well as for each session.
Thirteen attendees (53% women, mean age 48 years, SD 15 years) were interviewed following the Praxis 2016 conference. A diverse group of attendees indicated they wanted to be interviewed, including two individuals with SCI, two KT specialists, three researchers, one industry partner, three members of the SCI community, one policy-maker and one clinician. All interviews were completed within 3-months post-Praxis (mean duration 40 min, SD 7 min). Key themes, associated definitions and illustrative quotes are outlined in Table 2.
Regarding implementation, attendees expressed challenges related to understanding the unconventional goals of the conference before arrival. However, goals related to creating collaborative networks and developing solutions to translational challenges became clearer at the conference. Attendees expressed mixed opinions regarding the working tables and presentations. Some attendees found the assigned seating limiting; whereas, others felt that the tables were beneficial and expanded their networks. Some attendees also felt that the speakers’ presentations should have been shorter, allowing for more time for working table activities.
Regarding setting-level maintenance, after the conference attendees felt that Praxis 2016 would help to highlight challenges and solutions for bridging the ‘valleys of death’ long term; however, attendees were sceptical as to whether solutions would be implemented without strong leadership and input by diverse groups, including members of the SCI community. Themes related to adoption were not derived.
Five RHI staff and four members of the PAC participated in a telephone interview (66% women, mean age 52 years, SD 9.7 years). Five interviews were conducted by HG and four by SS (mean duration 52 min ± 11 min). Key themes, definitions, and illustrative quotes are outlined in Table 3.
In terms of adoption, organisers became involved with Praxis 2016 as it aligned with their personal and/or professional interests. While organisers adopted the Praxis 2016 conference, they perceived some hesitancy on behalf of stakeholders to adopt the non-traditional and interactive format. Further, the organisers said they found it was difficult to convince stakeholders of the value of the conference’s innovative format.
Regarding implementation, high-level leadership within RHI and professional conference facilitators were brought in to facilitate the successful development implementation of the conference. Almost all organisers indicated that strong leadership and the facilitators were essential to the planning and delivery of Praxis 2016. Organisers aimed to ensure adults living with SCI were represented at every table and RHI hosted an event before the conference to help unite and empower adults with SCI to share their opinions at the conference. However, the non-traditional format did create challenges while planning Praxis 2016. The goals and format of the conference developed iteratively, which resulted in delays in the development of the conference programme and the selection of speakers. These delays were particularly challenging in terms of personnel and resources as many individuals had to work long hours to host Praxis 2016. RHI staff indicated that they felt burnt out after the conference. Organisers indicated that staff burnout and lack of a clear post-Praxis 2016 plan likely led to significant delays in the delivery of the Praxis report and action plan. Overall, organisers indicated that the conference exceeded their expectations and organisers were pleased with the non-traditional and collaborative format of the conference. Organisers were particularly pleased with the representation of adults with SCI at Praxis 2016.
In terms of setting-level maintenance, PAC and staff members felt that Praxis 2016 had become a movement in-and-of-itself that encouraged SCI research to focus on translation, and they highlighted the importance of building on the momentum established at Praxis. Members highlighted the need for continued leadership and face-to-face meetings to develop and build on short- and long-term goals identified through Praxis 2016. Of particular note, the involvement of adults with SCI in creating and implementing solutions to overcome the KT gaps was seen as an important and lasting lesson from Praxis. Members felt it was important to have efforts to continue to support and foster the engagement initiatives specific to adults with SCI such as the recent North American Spinal Cord Injury Consumer Consortium (http://www.themiamiproject.org/nascic-2017/).
In sessions 1–3, the majority of session goals were completed (minimum 50%, maximum 86%). In session 4, the presentation and panel discussion goals were met, but the working table was canceled due to time restraints. While the facilitator consistently achieved the outlined goals, there was variability in speakers’ ability to achieve the outlined tasks and goals. Additional file 4 outlines the percentage of session goals that were completed as intended.
Evaluation team notes
In total, 313 implementation-based comments were extracted from the evaluation team’s notes. Key themes and definitions derived from evaluator notes as well as illustrative quotes are outlined in Table 4. Themes related to presenters and panel qualities, audience engagement and facilitation, and accessibility and engagement of adults with SCI. The majority of evaluator comments related to speakers and panels. While evaluators observed that speakers and panels had challenges adhering to the pre-defined speaker notes or goals and used jargon, presenters were generally able to foster audience engagement and present content clearly. Success stories and personal experiences were not always shared; however, evaluators noted that these stories seemed to add depth, motivation and hope for future solutions. Evaluators noted that the facilitators appeared to foster the organisation and flow of the conference and audience participation. The facilitators encouraged the audience to interact and discuss topics even when differing opinions arose, they provided clear summaries and were flexible when challenges, such as time restraints, arose. However, the audience did seem distracted at points and our reach data suggests attendance at the conference dwindled on day 2 (Table 4). Finally, evaluators noted concerns related to accessibility and the engagement and inclusion of adults with SCI. While care was taken to ensure the conference was accessible to people with SCI, the evaluators felt that adults with SCI still faced barriers to participation in all conference activities and that there was a lack of representation of adults with SCI on panels after the opening session.
Working table observation
JTOG scores were high across the working tables (mean 46.7, SD 4.07; minimum 42, maximum 54). These data potentially indicate that the composition of tables fostered teamwork among attendees.
To our knowledge, this evaluation was the first to examine the setting-level impact of a large-scale KT conference such as Praxis 2016 using the RE-AIM framework. At the setting level, both speakers and organisers adopted the Praxis model and the conference was implemented as intended. However, there were delays in post-Praxis activities and efforts were taken to ensure that Praxis 2016 impacts the SCI field long term. Broadly, our findings can be used by other groups to improve future efforts to develop large-scale KT conference solutions for translating research into practice. Key findings and recommendations in terms of adoption, implementation and maintenance are unpacked below.
RHI staff were able to establish a PAC and recruit speakers to attend and adopt the alternative conference format of Praxis 2016. However, this alternative format had benefits but also presented challenges. The hesitancy of some stakeholders and speakers to adopt the new format may explain the poor adherence to some of the implementation metrics. For example, some speakers used traditional methods of presentations in terms of length, content and style, which did not align with implementation goals of Praxis. Hesitancy to adopt alternative formats and innovations is not uncommon, as according to Rogers’ Diffusion of Innovations Theory , it takes time for a social system to adopt new innovations. Given this was the inaugural Praxis conference, it is likely that speakers who adhered to the implementation goals were either closely connected to RHI or vested in developing solutions to ensuring SCI research was used in practice (i.e. innovators or early adopters). If the results and movement of Praxis 2016 continue to be shared positively among SCI stakeholders, it is hoped that future iterations of this conference will see higher rates of adoption among SCI stakeholders and speakers . Based on these findings, it is recommended that future KT initiatives that aim to adopt non-traditional methods work closely with stakeholders to create ownership over the methods and select individuals with established relationships with the organisation developing the initiative.
Regarding the implementation, Praxis 2016 was overall delivered as intended, with only minor deviations such as the delay in some deliverables (e.g. Action Plan) and on-site planning change (e.g. working table session 4 canceled). The impact achieved by Praxis 2016 did require extensive effort and resources from RHI staff members. Across all data sources, several key aspects related to Praxis 2016’s implementation were highlighted, including the interactive format, the value of facilitation and the value of the voice of adults with SCI. Both attendees and organisers as well as the evaluation team commented on the value of the interactive format and working tables as a strength of the Praxis 2016 format. Indeed, research has indicated that bringing together diverse opinions and fostering interdisciplinary teamwork can be valuable for KT [14, 15]. Likewise, hiring trained and professional facilitators to encourage discussion and interaction was seen as an important and valuable asset to Praxis 2016. In particular, it is recommended that organisers of future events do not underestimate the value of facilitation and bringing together diverse groups in face-to-face interactions.
An ongoing and important implementation theme was the importance of engaging the SCI community both within the Praxis 2016 initiative as well as in research and implementation more broadly. While Praxis 2016 aimed to ensure accessibility, and include people with SCI, our data indicated that further efforts are needed to foster and champion the voices of people with SCI within the translational process and at events such as Praxis. This finding aligned with calls within the SCI community for there to be “no research about us, without us”  and research funders’ emphasis on utilising and enhancing participatory and integrated KT approaches [14, 17,18,19]. Given organisers’ enthusiastic support for initiatives that engage adults with SCI, such as the recent North American Spinal Cord Injury Consumer Consortium (http://www.themiamiproject.org/nascic-2017/), it is anticipated that future events will have greater ownership and involvement of the SCI community throughout all phases of the event (i.e. planning, implementation and evaluation). Given these findings, it is recommended that organisers of future large-scale KT initiatives aim to partner with people with lived experience early and often throughout the planning, implementation and evaluation of their initiatives.
Findings related to the domain of maintenance begin to highlight future directions for RHI and initiatives that aim to close the gap between research and practice. While PAC and RHI staff members viewed Praxis 2016 as a movement that needs to be fostered, attendees were less clear about the long-term impact of Praxis. In particular, attendees were unsure as to whether solutions to the KT gaps between research and practice could be feasibly implemented. As the Praxis 2016 conference points out, closing the gap between research and practice is challenging. Our data suggested that ensuring the long-term impact of Praxis 2016 will require continued and persistent leadership from organisations such as RHI, as well as continued engagement from a diverse group of stakeholders, which includes people with SCI. Given this finding, it is recommended that future large-scale and event-based KT initiatives plan for, communicate about, and invest funds and personnel in post-event activities that support long-term impact.
Our evaluation methods have several strengths and limitations. In terms of strengths, the RE-AIM framework guided our evaluation, which allowed us to examine the impact of Praxis 2016 based on several perspectives and criteria. Our evaluation highlighted the value of using both qualitative and quantitative data to examine the RE-AIM domains and showed the value of triangulation for investigating the impact of complex initiatives such as Praxis. That being said, there are some limitations to our approach. In particular, fidelity checks and evaluator comments were completed in real-time rather than using recordings. Audio- and video-recordings are considered the gold standard for assessing fidelity . However, the organisers did not want the tables or sessions recorded. Our team had to be embedded within the working tables and conference to complete the evaluation. While efforts were made to declare biases and maintain impartial, our findings were limited by the perceptions of the individual evaluators. Finally, our evaluation was commissioned 1 month prior to the Praxis conference. Integration of the evaluation team early in the development of Praxis 2016 may have led to additional insights.
Overall, our findings indicated that RHI achieved its goals in terms of implementing Praxis 2016. Praxis 2016 was adopted by staff and stakeholders, it was broadly implemented as intended, and data indicate that Praxis 2016 has the potential to have long-term impacts in the field of SCI research and practice. This evaluation provided several lessons learned that can help to foster efforts by RHI and other groups to develop future solution-focused conferences.
Jefferson Teamwork Observation Guide
Programme Advisory Committee
reach, evaluation, adoption, implementation, maintenance
Rick Hansen Institute
spinal cord injury
Krueger H, Noonan V, Tenaman LM, Joshi P, Rivers CS. The economic burden of traumatic spinal cord injury in Canada. Chronic Dis Inj Can. 2013;33(3):113–22.
Rick Hansen Institute. Rick Hansen Spinal Cord Injury Registry - A look at traumatic spinal cord injury in Canada in 2015. 2017. https://rickhanseninstitute.org/work/our-projects-initiatives/rhscir. Accessed 21 Mar 2019.
Barrable B, Thorogood N, Noonan V, Tomkinson J, Joshi P, Stephenson K, et al. Model for bridging the translational ‘valleys of death’ in spinal cord injury research. J Healthc Leadersh. 2014;6:15–27.
Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20.
Sweet SN, Michalovic E, Baxter K, Baron J, Gainforth HL. RE-AIMing conferences: a reach, effectiveness, and maintenance evaluation of the Rick Hansen Institute’s Praxis 2016. Translational Behav Med; 2019. https://doi.org/10.1093/tbm/ibz002.
Rick Hansen Institute. Praxis 2016 Bridging the Gaps from SCI Research to Improved Outcomes: Conference Report. 2016. https://rickhanseninstitute.org/work/praxis/model/88-rhi-praxis-model-no-comment-allowed/448-praxis-2016-conference. Accessed 21 Mar 2019.
Rick Hansen Institute. Praxis Action Plan 2017. https://rickhanseninstitute.org/work/praxis/action-plan. Accessed 21 Mar 2019.
Glasgow R, Vogt T, Boles S. Evaluating the Public Health Impact of Health Promotion Intervention: The RE-AIM Framework. Am J Public Health. 1999;89(9):1322.
Gaglio B, Shoup J, Glasgow R. The RE-AIM framework: a systematic review of use over time. Am J Public Health. 2013;103(6):e38–46. https://doi.org/10.2105/ajph.2013.
Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. 2007;28:413–33.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
Clarke V, Braun V. Teaching thematic analysis: overcoming challenges and developing strategies for effective learning. Psychologist. 2013;26(2):120–3.
Rogers EM. Diffusion of Innovations (5th ed.). New York: Free Press; 2003.
Canadian Institutes of Health Research. Guide to Knowledge Translation Planning at CIHR: Integrated and End-of-grant Approaches. 2012. http://www.cihr-irsc.gc.ca/e/45321.html. Accessed 21 Mar 2019.
Gainforth HL, Latimer-Cheung AE, Athanasopoulos P, Martin Ginis KA. Examining the effectiveness of a knowledge mobilization initiative for disseminating the physical activity guidelines for people with spinal cord injury. Disabil Health J. 2013;6(3):260–5.
Woodill G. Independent Living and Participation in Research: A Critical Analysis. Toronto: Centre for Independent Living in Toronto (CILT); 2006.
Graham ID, Tetroe J. Some theoretical underpinnings of knowledge translation. Acad Emerg Med. 2007;14(11):936–41. https://doi.org/10.1197/j.aem.2007.07.004.
Kothari A, Armstrong R. Community-based knowledge translation: unexplored opportunities. Implement Sci. 2011;6:59.
Noonan V, Wolfe D, Thorogood N, Park S, Hsieh J, Eng J. Knowledge translation and implementation in spinal cord injury: a systematic review. Spinal Cord. 2014;52(8):578–87.
Borrelli B. The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. J Public Health Dent. 2011;71(s1):S52–63.
We would like to acknowledge Rhyann McKay and Jena Shank for their help with data collection during the Praxis 2016 meeting. Emily Giroux and Brianna Tsui are also acknowledged for their assistance with transcription. Rick Hansen Institute funded this project. The evaluation was independent and the Rick Hansen Institute did not influence the findings reported in this paper.
The evaluation was funded by the Rick Hansen Institute (G2016–25).
Availability of data and materials
Example quotes are provided in supplementary files. A full list of de-identified quotes and observations are available upon request. Further data is not available due to privacy limitations.
Ethics approval and consent to participate
Given this project is a programme evaluation, approval from an ethics board was not required. Prior to the interviews, we informed attendees and organisers of the purpose of the evaluation, that their responses remained confidential, provided a contact name of the evaluation team, and asked for consent prior to data collection. We kept in regular contact with the UBC Okanagan Behavioural Research Ethics Board to confirm our processes fell under the definition of evaluation.
Consent for publication
Not applicable to this manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.