Table 3 Sample stakeholder quotes for each research direction

Examine how organizations/networks use CCO’s PM tools and processes, and determine how to encourage and facilitate productive use of the PM system and its data

“It would be good to figure out is there is a relationship between the performance of a region, and how they are using our system” (P04, CCO, cancer, manager)

“It’s obvious to me that some regions apply it more effectively than others. There are some regions where their performance changes and there are ones which don’t. And there is something that they’re doing with what CCO gives them. Or maybe they’re not and they have some other things that they’re doing. Either way we should be learning from what they’re doing and seeing how CCO could be better supporting that, for them to make changes” (P07, CCO, cancer, manager)

“I think you’re going to get very different results for the 14 regions…it’s probably going to be a mishmash of things” (P17, CCO, cancer, administrative staff)

“I think how they use the tools is just a symptom of other things…I think there’s a reason there are differences in the way the regions are using the tools, and it gets more into the local environment” (P18, CCO, cancer, administrative staff)

“I think it’s a bit top-down and not stepping back enough. If there is an opportunity as a province to improve, you wouldn’t keep your head down and stay where you’re at. So, I’m not sure how much this is going to help us to move things forward” (G19, network, cancer and renal)

Describe and compare PM tools and processes across high-performing health systems, including CCO

“I feel like we should be learning from what other people do” (P07, CCO, cancer, manager)

“I think we should reach out to those who performance manage well…Go there, learn the nuts and bolts of how they do it, bring that back, and assimilate the best practices that they use” (P57, CCO, cancer, clinician)

“We think we’re doing a great job and we feel a sense of real ownership of how CCO measures performance. I would certainly love to know how that looks against other models” (G13, network, cancer)

“See what the best are doing and steal shamelessly—Yes! It is hard for me to advise on the feasibility of implementing what is learned, but benchmarking against other jurisdictions is vitally important” (P16, patient)

“Do I ultimately believe we’ll get something revolutionary out of this? No. Their set-up is not the same as ours…Most of our tools are homemade tools which have evolved over the years. So, will their tools really fit here, into this environment? I don’t know. I would put it towards the bottom” (P46, CCO, cancer, manager)

“The problem is defining what’s high-performing because there was a recent presentation that the UK has the highest performing health system out there, and yet they’ve got the worst cancer survival rates in the Western world” (G15, network, cancer)

“It’s interesting, but I don’t know that it’s practical and applicable. Would it be of much help? Can you translate it to our context?” (G21, network, cancer and renal)

Explore how environmental and organizational factors influence compliance with CCO’s performance requirements and continuous improvement

(e.g., competing accountability requirements, organizational culture, leadership style, CCO-network relationships)

“I think it would go a long way to helping educate CCO on what those factors are, so that we better understand the regions, and why they respond the way that they do. I think it will enable us to do a better job” (P18, CCO, cancer, administrative staff)

“It’s important because we are ranked against one another even though we may have very different resources, demographics, and barriers that influence performance, so how that can be taken into consideration or supported from a provincial level? I think it’s very important to consider that when we’re looking at performance and outcomes. One size doesn’t fit all” (G3, network, cancer)

“They will listen to you after the fact, if you’re not where you need to be, based on your local factors. But, I do think that needs to be taken into consideration up front, at the beginning of the process, rather than at the end” (G9, network, renal)

“This one has applicability beyond the cancer programmes. The findings could be a very helpful tool to hospitals in Ontario” (G11, network, cancer and renal)

“I think it’s not useful, unless you have the capacity and ability to change the environmental and organizational factors” (P42, CCO, cancer, manager)

“How will we be able to apply the results from this? I really do worry this will end up being too abstract and too far into the weeds of local issues…when you talk about leadership style, for example, what would you do with that information at a provincial level?” (G10, network, cancer and renal)

“We sometimes get bogged down in conversations about how our unique characteristics might be a reason why we’re not performing and I think we need to avoid those types of conversations…the indicators are meant to have broader applicability…this is what we want to do everywhere for all patients no matter what door you’re walking through” (G21, network, cancer and renal)

Examine which of CCO’s PM tools and processes are most effective in motivating performance improvement

“I think this is really important, in part because I think we need more guidance on how to incent the change that we want” (P2, CCO, cancer, manager)

“How we’re using them is not as important as why we’re using them. And you can get at the why by looking at what motivates a response and what does not” (G11, network, cancer and renal)

“I really like this one. I mean at the end of the day what are we trying to do? We’re trying to change behaviour either that of the system or of individuals to improve, right? So, what methods instigate a change in behaviour or practice?” (G20, network, cancer and renal)

“My largest interest is in this one as it relates to getting information down to the care providers and empowering a large roster of staff to buy-in or abide. That, in fact, might be my only interest” (G14, network, renal, clinician)

“Metrics only work if the entity being measured feels compelled to engage and align to the metric” (P16, patient)

“Some programmes really do seem to respond to these performance management initiatives that we have and others don’t, and, boy, would I ever like to know why because then you could make them more effective and you can reach your goals more easily” (G18, network, renal, clinician)

“I think that they work in combination with each other, so I am not sure…maybe it’s not which ones motivate improvement, but when do they, together as a toolbox, motivate improvement and when do they not and why?” (P07, CCO, cancer, manager)

“I feel like if you ask that question you’re going to get very different answers from one programme to another” (P3, CCO, cancer, manager)

“Some are going to be better for some centres and worse for others, and it’s just basically looking at what we’re doing now” (G12, network, renal)

“I think your question is ambiguous because you’re looking at effectiveness in motivating improvement. That’s a very qualitative question. I would just say, ‘most effective in achieving improvement’” (P47, CCO, cancer, clinician)

Identify unintended negative consequences stimulated by CCO’s PM system, and determine how important they are and how they can be mitigated

“If there is something damaging in our processes, the longer we maintain those damaging processes the worse it is. So, with a bit of a risk lens on that, it comes up in my ranking” (P28, CCO, cancer, manager)

“I love this one. I’ve seen really interesting examples of unintended consequences. We need to plug in some mitigation factors” (P14, CCO, cancer, manager)

“I don’t know that we every really talk about the unintended negative consequences because we’re so focused on the positive outcome we want to see” (P63, CCO, renal, manager)

“I think this is potentially an important question to look into as well. There is always a concern about focusing on one metric too much and therefore ignoring everything else and detracting from quality in other areas” (G20, network, cancer and renal)

“I thought that was important because it’s your risk management process. You need to know what the downside is, so it has to be relatively high up in the ranking. Because you can’t go into all this and not be aware” (G17, external, LHINs)

“I think it’s important, but I’m not sure how impactful it might be. Even if we know that it exists, is there something we can do about it, without causing other unintended consequences?” (P18, CCO, cancer, administrative staff)

“I am hesitant because it has a more negative spin on things. And, for me, it’s just trying to focus on what we can do better” (P07, CCO, cancer, manager)

“I don’t think it will move us forward in terms of performance management, beyond doing some tweaks around some pain points” (P71, CCO, cancer, manager)

“I think that unintended negative consequences don’t need research. There’s going to be somebody screaming really early on in the process. It doesn’t require research to summarize that” (G1, network, cancer)

“It’s kind of a negative spin and I think it will come out in other ways, so I think that that one would be less of a priority to me” (P40, external, policy-maker)