Using the Nine Common Themes of Good Practice checklist as a tool for evaluating the research priority setting process of a provincial research and program evaluation program

Background Given the context-specific nature of health research prioritization and the obligation to effectively allocate resources to initiatives that will achieve the greatest impact, evaluation of priority setting processes can refine and strengthen such exercises and their outcomes. However, guidance is needed on evaluation tools that can be applied to research priority setting. This paper describes the adaption and application of a conceptual framework to evaluate a research priority setting exercise operating within the public health sector in Ontario, Canada. Methods The Nine Common Themes of Good Practice checklist, described by Viergever et al. (Health Res Policy Syst 8:36, 2010) was used as the conceptual framework to evaluate the research priority setting process developed for the Locally Driven Collaborative Projects (LDCP) program in Ontario, Canada. Multiple data sources were used to inform the evaluation, including a review of selected priority setting approaches, surveys with priority setting participants, document review, and consultation with the program advisory committee. Results The evaluation assisted in identifying improvements to six elements of the LDCP priority setting process. The modifications were aimed at improving inclusiveness, information gathering practices, planning for project implementation, and evaluation. In addition, the findings identified that the timing of priority setting activities and level of control over the process were key factors that influenced the ability to effectively implement changes. Conclusions The findings demonstrate the novel adaptation and application of the ‘Nine Common Themes of Good Practice checklist’ as a tool for evaluating a research priority setting exercise. The tool can guide the development of evaluation questions and enables the assessment of key constructs related to the design and delivery of a research priority setting process. Electronic supplementary material The online version of this article (doi:10.1186/s12961-016-0092-5) contains supplementary material, which is available to authorized users.


Priority Setting Process
Participants were asked to register for one of seven different subject areas (Table 1) that were selected through the Research and Evaluation Topic Survey that was administered to public health units in January 2012.
Each subject area group had representation from a range of public health units and was provided with the list of topics generated through the Research and Evaluation Topic Survey within their subject area. Over the course of the day, subject area teams, led by a facilitator, worked towards the prioritization of a single research or evaluation question that will form the basis of their collaborative project.
At different stages, participants were asked to consider each of the following three priority setting criteria as they discussed and selected their top choices (see Figure 1): Interest: Alignment with the priorities and direction of health units and the public health system.
Impact: Ability to generate knowledge and evidence, to support health units to meet the Ontario Public Health Standards and influence change in the public health system.
Balance: Address the priorities of health units from different regions and of various sizes with the goal of meeting the demands of the majority and the needs of the minority.
Content experts from a range of organizations -including Public Health Ontario, the Alcohol Policy Network, the Ontario Neurotrauma Foundation, SMARTRISK, and the University of Toronto -acted as a resource to the LDCP collaborative teams by helping to identify existing initiatives and providing methodological support. Although content experts provided invaluable input into the discussions, decisions surrounding the prioritization of specific research questions remained in the hands of the health units.

Workshop Outcomes
Below is a brief overview of the outcomes of Workshop 1. Each table includes the top five research or evaluation questions that were developed through a group consensus and voting process, and highlights in blue the final question that will form the basis of each LDCP subject area's ongoing collaboration. N/A refers to questions that were not among the LDCP collaborative team's top 3 choices and, consequently, were not ranked. For a list of participants in each subject area group, please see Appendix C.

TOP 5 RESEARCH OR EVALUATION QUESTIONS: VACCINE PREVENTABLE DISEASES -IMMUNIZATION FINAL RANK
What parent/school-based strategies are effective to increase immunization coverage rates in school-aged children? 1 What is the feasibility of connecting existing data (OHIP billing, FHT, IRIS, other) to a provincial immunization strategy?

2
What are primary factors that influence administration of school-based vaccines? (E.g. system barriers, inadequate HR, failure to obtain consents, age at which immunization is given, perceived risk of disease vs. risk of immunization).

3
What strategies targeted at parents/guardians/students are effective to increase uptake of HPV vaccines?

N/A
What factors influence the public's acceptance of various vaccines? (E.g. HPV, influenza) N/A

TOP 5 RESEARCH OR EVALUATION QUESTIONS: FALLS ACROSS THE LIFESPAN FINAL RANK
What are the sustainable and effective strategies for engaging community stakeholders that serve families with children 0-4 years to implement key components of Alberta's Million Message program when adopted and implemented in Ontario communities? 1 What are the best practices to prevent sport and recreational injury among children 10-19 in Ontario?

2
How might falls prevention efforts be translated through the lifespan to maximize impact at every age?

3
What are the perceived barriers and effective strategies for delivering consistent and repetitive messaging among stakeholders that service families with children ages 0-4 years?

N/A
What are the best practices for preventing future adult falls to inform the development of a strategy for Ontario? N/A

BREASTFEEDING
1 The final research question selected to move forward was an amalgamation of the two questions below it in the table.

TOP 5 RESEARCH OR EVALUATION QUESTIONS: BUILT ENVIRONMENT FINAL RANK
What are the best practices, tools and suitable alternatives in planning policies and procedures for rural and small towns for improving health outcomes within the built environment? 1 What are the barriers to implementing set-backs to protect sensitive populations from environmental exposures?

2
What are the economic indicators and health benefits that demonstrate the value of active transportation and transportation policy?

3
What are the barriers and facilitations to using an economic argument for active transportation to influence policy at the municipal and provincial level? N/A To assess use of BORN information system for population health surveillance in following PH priorities: 1. Alcohol use, illicit drug use during pregnancy. 2. FASD. 3.

Maternal weight gain
2 What is the current rate of pregnant women having a discussion with HCP re: importance of physical activity and healthy weights in pre-conception and pregnancy? (What strategies can HU implement to increase rate)

3
To evaluate the effectiveness of the use of social media tools to increase reach and awareness related to prenatal health N/A What are the determinants/SDOH factors resulting in high rates of teen pregnancy in rural and or northern Ontario N/A

Next Steps
The second of three workshops, Building Collaborative Teams, Beginning Proposals, will be held on April 5 th , 2012 and April 13 th , 2012. Public Health Ontario is busy planning a range of activities that will assist LDCP collaborative teams to further scope their research or evaluation questions, develop SMART objectives and begin discussing team member roles and responsibilities. Throughout the day, LDCP teams will also learn more about library services as well as other resources and supports that are available from PHO.
The objectives of Workshop 2 are to: 1. Work in collaborative teams to develop proposals including research objectives and methods 2. Introduce participants to tools and resources available to support proposal development 3. Complete an action plan, identifying roles and responsibilities of team members Similar to Workshop 1, context experts will be available to provide methodological support and insight into current projects and initiatives that are happening provincially and nationally. Representatives from community organizations and associations that are interested in partnering or collaborating on the LDCP projects will also be in attendance.
In order to support LDCP 2012 teams as they work on their proposals, Public Health Ontario will host Workshop 3 on May 31 st , 2012. Workshop 3 will provide LDCP collaborative teams with an opportunity to come together to develop their proposals. The focus of Workshop 3 will be on assisting teams with specific aspects of their project planning, such as developing timelines and Knowledge Exchange plans, and budgeting. LDCP proposals are due on July 13, 2012. More information about 2012 LDCP important dates can be found on PHO`s website.

Appendix B. Workshop Evaluation
At the end of the workshop, participants were asked to complete a short survey that explored the extent to which they felt that Workshop 1 met its stated objectives. Fifty-nine participants completed the questionnaire, a response rate of 64%.

WORKSHOP SUCCESSES
Participants noted that there were a range of excellent components to Workshop 1. In particular, participants commented on the collaborative nature of the day and the "opportunity to articulate common areas of interest [and] potential areas for effort." Several participants also noted that the facilitators were "great" and valued the "guidance by PHO." These comments were reflected in the average responses to the five structured questions included in the evaluation form. Figure 2 displays the percentage of participants that agreed or disagreed with each of the five structured statements in the questionnaire based on a scale where 1 indicated strongly disagree and 5 indicated strongly agree.

RECOMMENDATIONS
While there were many positive comments about Workshop 1, participants also provided several recommendations for future workshops. Participants would like to have "more nutritious food" and to ensure that the day's activities remain on time. In terms of the priority setting process, some participants wanted "more information on how to formulate research questions" and the opportunity to discuss and debrief after a voting process had occurred. A GOOD FIRST STEP TO BUIDLING  EFFECTIVE PARTNERSHIPS AND COLLABORATIONS WITH  OTHERS WHO ARE INTERESTED IN RESEARCH AND  EVALUATION IN PUBLIC HEALTH   I FELT THAT THE PROCESS USED TO SELECT PRIORITIES HELPED TO  BUILD CONSENSUS   I HAD OPPORTUNITIES TO EXPRESS MY OPINIONS AND IDEAS