Table 4 An example of an approach to the formative evaluation of a policy briefs series
• The McMaster Health Forum surveys those to whom it sends a policy brief, with the long term goal of identifying which design features work best for particular types of issues, and in which particular health system contexts. Participation is voluntary, confidentiality assured, and anonymity safeguarded |
|---|
• Twelve features of the policy briefs series are the focus of questions in the formative evaluation survey: |
   • Describes the context of the issue being addressed |
   • Describes different features of the problem, including (where possible) how it affects particular groups |
   • Describes three options for addressing the problem |
   • Describes key implementation considerations |
   • Employs systematic and transparent methods to identify, select, and assess synthesised research evidence |
   • Takes quality considerations into account when discussing the research evidence |
   • Takes local applicability considerations into account when discussing the research evidence |
   • Takes equity considerations into account when discussing the research evidence |
   • Does not conclude with particular recommendations |
   • Employs a graded-entry format (i.e. a list of key messages and a full report) |
   • Includes a reference list for those who want to read more about a particular systematic review or research study, and |
   • Is subject to a review by at least one policymaker, at least one stakeholder, and at least one researcher. This process is termed a merit review to distinguish it from a standard peer review which would typically only involve researchers in the review process |
• For each design feature, the survey asks: |
   • How useful did they find this approach (on a scale from 1 = Worthless to 7 = Useful)? |
   • Are there any additional comments or suggestions for improvement? |
• The survey also asks: |
   • How well did the policy brief achieve its purpose, namely to present the available research evidence on a high-priority issue in order to inform a policy dialogue where research evidence would be just one input to the discussion (on a scale from 1 = Failed to 7 = Achieved)? |
   • What features of the policy brief should be retained in future? |
   • What features of the policy brief should be changed in future? |
   • What key stakeholders can do better or differently to address the high-priority issue and what they personally can do better or differently? |
   • Their role and background (so that the McMaster Health Forum can determine if different groups have different views and experiences related to policy briefs) |
• The Evidence-Informed Policy Networks (EVIPNet) operating in Africa, Asia and the Americas plan to use a similar approach in the formative evaluation of their policy briefs |