Table 2 Themes and exemplar quotes

Perceived strengths of the measures

 Measures cover the major aspects of being a productive researcher/comprehensive

Using these ten criteria as an assessor, I think I can get a pretty good feel for what that researcher is doing and what their successes have been and what their challenges have been (012 JS mid clinical)

 Measures are relevant and applicable to different research disciplines

I think one of the strengths is the breadth of and the various different angles that are being used to assess the strengths of a research program that takes into consideration or acknowledges the fact that there is a wide range of research and research approaches being conducted whether it is at UHN or at another organization. (017 BM)

 Moving away from impact factor allows researchers to personalize reporting, resulting in fair assessment

What I perceive as the strengths of the measures is the fair assessment of research advances and of individuals involved in research. (05 AC late biomedical)

 Measures are similar to those in current use

These are all the same measures that we at < institute > use for our evaluations. It’s just structured in a nice way (01 AC late biomedical)

 Easy to understand and use; response options are limited to key achievements, which minimizes reporting burden

It’s fairly clear and reasonably concise. Everybody that’s filling out this kind of form is busy…Its nice to have something that’s a straightforward template. (014 AC late biomedical)

 Measures were generated using a rigorous scientific process

And I was thinking that a lot of it is very science-based and makes sense for UHN. (02 AC early health services)

 Referring to specific measures:

Challenges to productivity

Where it talks about describing challenges based in the last 5-years, I think is a good way of accounting for things that might have changed, pregnancy or pandemics or changes in position or those kinds of things. (06 AC mid biomedical)

Collaboration

I don’t think we explicitly ask them about their collaborative work. We infer this information from these group grants. The question here is a little cleaner in a sense that it gets them to describe it and to explain how they’re advancing knowledge or having a larger impact through their collaborative work. It’s a little bit more explicit than what we did before. (018 BM)

Research contributions

I like the list of key research outputs. It gives you things like databases and computational and informatics tools and public domain resources that are so important these days. Researchers might not always be published in a traditional peer-review paper. I really like that you have it explicitly stated those kinds of outputs are something that can be highlighted (01 AC late biomedical)

Perceived limitations or gaps in the measures

 Measures do not reflect non-research activities such as teaching or service

There’s a lot more that goes into what we do, and I understand that this is supposed to be focused on research, but many of us are also teachers, many of us are mentors and this sort of moves into the realm of not acknowledging those contributions which are fundamental to the operation of UHN Research that really aren’t captured by this. And I think it takes an awfully narrow scope of what’s involved in our roles here. (013 S early biomedical)

 Output options listed are not comprehensive:

Medical education research

Adding curriculum onto that would capture the health professions education element more appropriately (02 AC early health services)

Intangible outputs of community-based research collaboration

We are going to be engaging in co-creation of some kind of a written project, possibly a paper, if that’s what suits their community. Fitting in some of the indigenous requirements for research like ownership of data and that their research belongs to the community and goes back to the community, and doesn’t necessarily turn into a publication (02 AC early health services)

 Measures do not include the effort of failed attempts to capture research funding

There should be some attempt to glean some kind of learning from failed attempts as well, especially on large grants, on what did they learn out of the entire process of applying for a big grant and not getting it. Maybe those kinds of things could be collated and made available to other people who are going on to apply for such grants in the future. (010 BM)

 Measures may not be relevant across research disciplines

I think the disciplines are so different that to use this collection of measures equally across disciplines is extremely difficult (06 AC mid biomedical)

 Five-year time frame is too short and may not reflect impact and quality

One of the deficiencies is that it focuses very much on the last 5 years and sometimes it’s important to put things into context over a longer research career (016 ID late biomedical)

Potential barriers to reporting the measures (researchers)

 Time consuming or requires effort

It’s gonna be a lot of work for people to do this on an annual basis, because as I understand it, the administrative staff goes in, copy from their CV…they assist with pulling these publications into the annual activity report or it’s automatically pulled up by the research analytics group…This one’s gonna take a little bit more work to describe how their research is leading to a certain outcome (018 BM)

 Some people are better at describing their research than others but this does not reflect the quality and impact of their research

People differ in their ability to describe the impact of their work but that does not necessarily reflect the impact of the work, it reflects their ability to describe it (09 ID mid biomedical)

 Potential barriers to assessing measures (evaluators)

 Inertia or reluctance to implement or adopt the measures

Acceptance of the measures, because any kind of change is difficult. So this is very new way of assessing research and I think it requires a lot of shift in that we deeply hold on to what we’ve been doing, the norms of assessment, and I think the shift to these kinds of measures is the biggest challenge instead of the measures themselves or gaps in the measures (010 BM)

 Increased workload to assess merits of publications

There’s an expectation that evaluators will actually read the paper and make an assessment of the value of that research from the research article itself, not just from the title of the journal (01 AC late biomedical)

 Unclear how to weight different measures to generate or distinguish an overall evaluation

You can’t just count the number of publications and so on. You have to take into account whether sitting on an international panel of experts to determine a set of guidelines or core outcome measures or something is equivalent to one publication or two publications and so on. (011 SS late clinical)

 Without impact factor, may be difficult to assess research in disciplines that are new or unfamiliar, leading to biased or inaccurate assessments

If it’s someone from outside of their field, how are they gonna know whether that’s a significant contribution or not…And I think that that’s the place where bias could be an issue (02 AC early health services)

Strategies needed to address barriers and support adoption of the measures

 Allow research institutes to decide if to apply select measures

I think if a research institute, something didn’t kind of resonate with them or wasn’t as important; I think perhaps they could choose not to use one of those measures (04 BM)

 Researchers and evaluators will comply if measures are formally endorsed as the standard and become normalized

First, it comes from leadership. Leadership has to implement this over any formalized protocol or strategy that’s currently used for evaluation of scientists (07 AC mid biomedical)

 Hold an official launch and communicate in various ways to raise awareness on why this is important

We do town halls and open forum, that kind of a thing to better socialize it and make it an open discussion and gather people’s comments and their apprehensions and appreciation of the entire process (010 BM)

Ensure that researchers have admin support or automate the reporting process

Is there smart text to link our CV’s in some UHN format and then you can link what they’re looking for in the DORA document to your CV to make it even more efficient. (012 JS mid clinical)

Inform researchers about the metrics so that they can track them

Knowing these in advance, so if I was preparing my career, knowing these in advance, these can all be tracked. (06 AC mid biomedical)

Train researchers or appointment committee members on how to report or assess the measures

Maybe some brief training modules for people who are serving on these various committees could be helpful. (07 AC mid biomedical)

Provide guidance to evaluators on how to assess and interpret measures

We also really need guidance on what is considered a good job and not a good job when we no longer have those numbers to attach ourselves to (017 BM)

Evaluators should be from same discipline as researcher under review

There should be at least one reviewer who assesses that from within that specialty or area of research. (03 AC mid health services and population)

Employ multiple evaluators

If you have more than one person in the assessing these indicators, then hopefully you would have convergence at some point on what the outcome is. (011 SS late clinical)

Evaluators must contextualize measures to both discipline and career stage

You have to put in the context of the person you’re reviewing (08 BM)