Table 1 Summary of implementation measures and success thresholds for ECHO-ONMH

Acceptability: how agreeable, palatable or satisfactory the innovation is to its stakeholders

Participants’ satisfaction with ECHO

Mean score for full cycle of weekly session satisfaction survey statement “Overall, I am satisfied with the session” on a 5-point Likert scale

Achieves a mean score of ≥ 4/5 each cycle, indicating ECHO-ONMH is acceptable (satisfactory) by participants’ self-report

All four cycles of ECHO-ONMH (100%) met the threshold, demonstrating high levels of acceptability among participants

Adoption: uptake of a practice or innovation by an individual or organization, including both intent to try and action itself

Utilization of ECHO by participants (intent to adopt and act of adopting)

Intent to adopt: number of PCPs that registered for ECHO-ONMH each cycle

Act of adopting: number of participants who attend ≥ 1 ECHO-ONMH sessions within a given cycle, and average number of participants in attendance per session (i.e. action per session) within a given cycle

Achieves ≥ 25 PCPs registered (intent to try), as well as achieves ≥ 20 participants attending ≥ 1 session(s) and an average of ≥ 6 participants per session (action) each cycle

All four ECHO-ONMH cycles (100%) met both thresholds, thereby considered successful in the adoption outcome

Appropriateness: perceived fit, compatibility and relevance of an innovation to an individual’s or organization’s problem, challenge and/or setting

Relevance of ECHO curriculum/whether sessions meet participant learning needs

Mean score for weekly session satisfaction survey statement “this session addressed my learning need” on a 5-point Likert scale

Achieves a mean score of ≥ 4/5 each cycle, indicating ECHO-ONMH is appropriate (meeting learning needs), as identified by participants’ self-report

All four cycles of ECHO-ONMH (100%) met the threshold and are considered successful in the appropriateness outcome

Cost: pricing of the intervention and its implementation. Is it less expensive than other options?

Estimate of cost to participate in ECHO compared to in-person CME conference

Cost comparison (in CAD) for the ECHO model compared to an in-person conference in Toronto, comparing:

Per-person cost to participate (estimated at 72 CME hours), total cost per CME hour and total cost for all individuals (using total number of participants across 4 ECHO-ONMH cycles)

Costs to participate in ECHO-ONMH for all three cost comparisons is ≤ costs to participate in an in-person conference

The cost savings for an individual to participate in a cycle of ECHO-ONMH is about $25 per CME hour, $1833 per annual cycle, and $747 864 by model/programme per year. No success threshold, but this does constitute a significant cost savings for both the individual and the public healthcare funder

Feasibility: considers whether an innovation is practical for a provider and/or organization, shaping whether it can be implemented

How practical is ECHO for participants to participate

Average number of sessions participants attend within a given cycle

Further exploratory analysis will look at attendance rates by profession, practice types and practice location

The average number of sessions participants attend in each cycle is ≥ the global average from ECHO Institute for each cycle (6 sessions) [4]

Threshold is not established for stratified attendance, as this analysis is exploratory in nature

All four ECHO-ONMH cycles (100%) met the threshold and are considered successful for the feasibility outcome

Fidelity: actual implementation compared to that which was prescribed by a particular protocol or model to determine adherence, quality and integrity

Fidelity to the ECHO model as identified by the ECHO Institute

Adherence to four global ECHO principles and the presence of an “all teach/all learn” environment within a random sample of five videorecorded sessions per ECHO-ONMH cycle:

Best Practices

Case-Based Learning

Technology

Assess Outcome

All Teach/All Learn

100% of the fidelity criteria (adherence to principles and presence of all teach/all learn environment) are met across video samples selected for each cycle

All four ECHO-ONMH cycles (100%) met the threshold, having exceptionally high rates of fidelity to the Project ECHO model (adhered to 100% of criteria)

Penetration: integration or spread of a particular service, practice or innovation to its potential settings and subsystems

Reach across all targeted regions in Ontario

Proportion of Ontario’s 14 regional area LHINs reached (as represented by participants) in each cycle. LHINs are artificial regions that support the funding, planning and delivery of care, as the basis for the regional areas

Achieves registration from 100% of the LHINs (i.e. 14 out of 14 regions) in each cycle

None of the four cycles of ECHO-ONMH (0%) met the threshold and achieved successful penetration

Sustainability: innovation is maintained or established as an ongoing, institutionalized offering

Sustained adoption across cycles (required in order to meet funding agreements and sustain funding)

Number of years minimum adoption was sustained (≥ 25 PCPs registered, a roster of ≥ 20 participants, and a minimum average of 6 participants per session)

Meets the adoption threshold for each of the four cycles (2015–2019)

All four cycles of ECHO-ONMH (100%) met the threshold. ECHO-ONMH can be seen as a sustainable programme