Skip to main content

Development of a STandard reporting guideline for Evidence briefs for Policy (STEP): context and study protocol

Abstract

Background

Evidence briefs for policy (EBP) draw on best-available data and research evidence (e.g., systematic reviews) to help clarify policy problems, frame options for addressing them, and identify implementation considerations for policymakers in a given context. An increasing number of governments, non-governmental organizations and research groups have been developing EBP on a wide variety of topics. However, the reporting characteristics of EBP vary across organizations due to a lack of internationally accepted standard reporting guidelines. This project aims to develop a STandard reporting guideline of Evidence briefs for Policy (STEP), which will encompass a reporting checklist and a STEP statement and a user manual.

Methods

We will refer to and adapt the methods recommended by the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network. The key actions include: (1) developing a protocol; (2) establishing an international multidisciplinary STEP working group (consisting of a Coordination Team and a Delphi Panel); (3) generating an initial draft of the potential items for the STEP reporting checklist through a comprehensive review of EBP-related literature and documents; (4) conducting a modified Delphi process to select and refine the reporting checklist; (5) using the STEP to evaluate published policy briefs in different countries; (6) finalizing the checklist; (7) developing the STEP statement and the user manual (8) translating the STEP into different languages; and (9) testing the reliability through real world use.

Discussion

Our protocol describes the development process for STEP. It will directly address what and how information should be reported in EBP and contribute to improving their quality. The decision-makers, researchers, journal editors, evaluators, and other stakeholders who support evidence-informed policymaking through the use of mechanisms like EBP will benefit from the STEP.

Registration We registered the protocol on the EQUATOR network. (https://www.equator-network.org/library/reporting-guidelines-under-development/#84)

Peer Review reports

Background

The need for evidence-informed policies and decisions to achieve the Sustainable Development Goals (SDGs) and other global-development targets is increasingly and widely recognized among policymakers, managers, providers, the public, researchers, and other stakeholders [1,2,3,4]. As an approach to ensuring policies and decisions are informed by the best available data and research evidence (e.g., systematic reviews), the concept of “evidence-informed policymaking” (EIP) includes helping clarify policy issues, frame options for addressing them, and identify implementation considerations [2, 5]. Evidence briefs for policy (EBP, sometimes referred to as “policy briefs” or “evidence briefs”) are increasingly used to support EIP in both health and non-health sectors and they typically have the following characteristics: (1) starting with a priority policy issue based on policymakers’ perspectives; (2) clearly presenting the nature and magnitude of the policy problem; (3) formulating possible policy options based on evidence about benefits, harms, costs, stakeholder views and experiences, and other elements; (4) combing global research evidence (prioritizing the best-available evidence syntheses related to the issue) and local evidence in a comprehensive and systematic way; (5) providing potential facilitators of and barriers to the implementation of selected options, and strategies to leverage facilitators and address barriers [6, 7]. Moreover, these documents are developed as inputs into deliberative dialogues in which participants use the pre-circulated evidence brief as a ‘jumping off’ point for deliberations, while bringing their views and experiences to bear on the issue in support of EIP.

Nowadays, more and more health and non-health EBPs are being prepared to address different issues by international organizations or academic groups, such as the Evidence-Informed Policy Network (EVIPNet). Moreover, related theories, methodologies, and practical handbooks about the development and usage of EBP are continuously evolving as experience grows.

However, the reporting format, characteristics, and lengths of EBP varies greatly across the organizations preparing them [8, 9]. Adam et al. analyzed 109 documents developed for policymakers and found little consistency in both the names used to label and describe “policy briefs” and in their contents. There were many variations in the feature and characteristics, including political/health system contexts, problems, options, implementation considerations, cost implications, methods, the quality of the evidence [8]. Most would not meet the above definition of an EBP. Zhang et al. analyzed 129 health-related EBPs published from 2016 to 2020 and also found there are differences in reporting details among different organizations, and observed many opportunities to improve their transparency and reliability [10]. Although some guidance on writing and reporting EBP are available [6, 8, 9], a comprehensive and widely accepted reporting guideline for EBP is needed to resolve some of the challenges described above.

Reporting guidelines are an important method to promote scientific research and developing guidelines for research-derived products such as EBP would similarly promote more systematic and transparent approaches to the development and reporting in these increasingly common documents. A clear, accurate, and complete EBP can not only help decision-makers and other stakeholders correctly understand the contents of the brief, but also help users make more evidence-informed decisions. Moreover, as one popular evidence-packaging document that addresses policy issues and provides policy options for policymakers, it is necessary to develop a widely accepted reporting guideline for EBP. The STandard reporting of Evidence briefs for Policy (STEP) project aims to develop a reporting guideline which will encompass a reporting checklist, a STEP statement & a user manual.

Methods

We will refer to and adapt the methods recommended by the EQUATOR network [11]. Table 1 shows the detailed process with the proposed timeline.

Table 1 Adapted recommended-stages and project timeline

Stage 1 preparation

Action 1. Identify the need for developing the STEP checklist

To identify the need for developing the STEP checklist, we searched the peer-reviewed literature and related documents on EBP and found there is no internationally accepted standard reporting checklist for EBP in health and non-health sectors. Moreover, our several interactions with different groups of decision-makers reaffirmed the need to develop a STEP checklist through a systematic and transparent process.

Action 2. Obtain funding

Our project is funded by the major project of the National Social Science Foundation of China: “Research on the Theoretical System, International Experience and Chinese Path of Evidence-based Social Science” (Project No. 19ZDA142). The funder will not interfere in the design of the study, in any aspect of data collection, analysis or interpretation, or in the writing and publication of the output.

Action 3. Draft and register the protocol

To enhance the transparency and quality of the STEP development process, we drafted a research protocol and registered it on the EQUATOR network (https://www.equator-network.org/library/reporting-guidelines-under-development/#84). The protocol will be published in the peer-reviewed journal and the development of STEP will be initiated after publication of the protocol.

Action 4. Identify the participants and establish the working group

The STEP working group consists of two subgroups: the Coordination Team (CT) and the Delphi panel (DP). We will collect both direct and indirect disclosures for all participants at the start of the STEP project and before publication.

  1. 1.

    Coordination Team (CT)

The members of the CT represent skills and experience covering project coordination, Delphi technique, development of EBP, reporting guidelines, guideline development, and implementation science. The roles of the CT are to coordinate the project; recruit the DP; draft the protocol and potential items; organize the Delphi process; collect and analyze data from the Delphi process; and draft the final checklist, statement, and explanatory document.

  1. 2.

    Delphi Panel (DP)

Considering the diversity of methodological expertise, gender equality and wide geographic representation, we will invite 10–20 individuals to the DP group, based on a review of the main authors in the field, as well as the recommendations from our working group. We will include experts with technical expertise in EBP development and implementation; EIP; health and non-health sectors; policy research; practice guideline development and implementation; knowledge translation; and reporting guidelines. To reflect the views and perspective of EBP target users and population, representatives of policymakers and other decision-makers (e.g., target users), researchers and the public (e.g., target population) will be also invited into this group. The roles of the DP group are to contribute to the protocol and development of checklist; and provide the comments on the final STEP checklist.

Stage 2 formulation

Action 5. Generate a list of items for consideration in the Delphi process

We will conduct a literature search to extract potential reporting items. The search strategy, including the search sources and terms, will be developed with the help of an information scientist. We will mainly search six electronic databases: Web of Science, PubMed, EBSCO, PIE, Health Systems Evidence, and Social Systems Evidence. Also, we will additionally search related websites: 3IE International Initiative for Impact Evaluation, European Observatory on Health Systems and Policies, Evidence for Action (E4A), Evidence-Informed Policy Network (EVIPNet), Health Affairs, Health Systems Global (HSG), World Health Organization (WHO), and United Nations (UN). Moreover, we will search Google and Google Scholar for first 200 records and check the references of included studies and papers recommended by experts. The search strategy for PubMed is presented in Table 2.

Table 2 Search strategy for PubMed

We will include the publications with the following characteristics: (1) publicly available, which clearly mentions one of EBP characteristics in both health and social science sectors between 2017 and 2021; and (2) publicly available research literature and/or documents related to EBP methodology, including development, reporting, use or implementation of EBP. We will exclude conference abstracts, editorials, commentaries, letters, opinions, correspondence, and news. Two reviewers will independently conduct the screening for literature and documents and the discrepancies will be resolved through discussions and consulting a third reviewer (if necessary).

We will develop a standardized form to extract the characteristics of each included document, including title, publication year, journal or development organization, and country focus (if applicable). We will extract and code the reporting characteristics (such as how to report problems, options, implementation considerations, methodology) from included documents, and conduct a thematic analysis based on the coding. Two reviewers will independently conduct the data extraction and coding for documents and the discrepancies will be solved through discussions and consulting a third reviewer (if necessary). During the process, iterative discussion and feedback among the research team members will be undertaken. Through the virtual meeting among research team, a list of potential items for reporting EBP will be developed and refined.

Action 6. Conduct the modified Delphi process

To achieve consensus on reporting items in the final STEP checklist, considering the possible attrition and participant fatigue [12], we will undertake three rounds of Delphi surveys, although there is a possibility that we may still be unable to reach consensus on some items during the third round. To reduce the response bias, we will accept a round as valid if there is a 70% response rate. Also, we will use several methods to enhance the response rate, including: setting up the expected time (usually within 10 days) for participants to return the questionnaire; sending a reminder letter to non-responders after 1 week; re-sending the questionnaire after 2 weeks. Our modified Delphi process is described in Table 3 [13,14,15].

Table 3 Organizing the Delphi process

The questionnaire will be piloted among the CT. The content and format of the questionnaire will be adjusted through discussion. We will send the initially prepared items in Microsoft Word format to score each item of the initial STEP checklist and collect suggestions and modifications. We will use a 7-point scale (1 = strongly disagree, 2 = disagree, 3 = disagree slightly, 4 = neither disagree or agree, 5 = agree slightly, 6 = agree and 7 = strongly agree) to rate each potential item [14,15,16].

Participants will be provided with the opportunity to suggest refinements and modifications to each of the candidate items and to suggest additional items not addressed in the list. The responses for each round will be analyzed using Microsoft Excel. For the scored quantitative data, we calculate the median score. The definition of agreement and consensus are presented in Table 4. Based on the response for each round of Delphi: (1) items with ‘agreement’ will be included in the final version and be removed from subsequent rounds; (2) items with ‘disagreement’ will be deleted; (3) items which are rated as ‘ambivalent’ or where there is no consensus will be modified according to the comments from the expert panel and the discussion of the research team, then will be included in the next Delphi round. Additional items nominated by the panel will be reviewed by the research team and reworked to align with the style and format of the other candidate items [13, 15]. According to the Delphi results, we will draft a checklist of included reporting items for EBP.

Table 4 Definition of consensus

Action 7. Use the STEP to evaluate published policy briefs

After the second or third round of Delphi surveys is completed, we will generate a draft version of the STEP checklist. To validate the utility, applicability, and reliability of the STEP checklist, we plan to use the STEP checklist to evaluate a sample of policy briefs published in different languages (such as Chinese, Spanish, Korean, Portuguese, Arabic) in the last three years. Through the evaluation of a sample of EBP, we hope to identify current deficiencies and weaknesses in the tool, record them and use them to inform the final updates and improvements to the STEP checklist. CT will collect and analyze results, and discuss them with DP in the next phase (e.g., Action 8).

Action 8. Approve the final checklist

Due to the COVID-19 pandemic, we will organize a virtual (rather than in-person) conference including representatives of the CT and DP groups to present and discuss the results of the Delphi process and evaluation results, then will draft the final checklist. We will send it to the representatives of the CT and DP for their review to ensure accuracy.

Action 9. Develop the STEP statement and user manual

The CT will develop STEP statement and user manual, including rationale for developing the checklist, development process, and the checklist. A purposive sample of participants will be recruited to for pilot testing. We will not limit participants according to their experience and exposure to EBP development and/or use, which will help us better estimate of the usability of the guideline among its expected users. We will seek feedback by pilot testing, which will include contacting individuals and groups involved in developing EBP, asking them to use the checklist to inform the preparation of their EBP, followed by the collection of feedback (by online survey) from them about any further revisions that need to be made to the guideline. We will publish the STEP in a peer-reviewed journal.

Stage 3 implementation

Action 10. Translate the STEP into other languages

We will welcome and collaborate with other EBP developers who want to translate this guideline into other languages.

Action 11. Test the reliability through real world use and update the guideline

We will test the reliability of the checklist through continuous collection, comparison and analysis of user data using a feedback mechanism by emails. Moreover, we plan to update the STEP every 3–5 years.

Discussion and conclusion

The main output of our project is a STandard reporting guideline of Evidence briefs for Policy (STEP), which will encompass a checklist of reporting items for EBP in health and non-health sectors, a STEP statement, and a user manual. The STEP will directly address what and how information should be reported in EBP. We also hope to reach a consensus on the term of EBP in this project. Moreover, policy makers, decision-makers, researchers, users, journal editors, and evaluators will benefit from the STEP statement.

The main strength of our research is the systematic, transparent and rigorous methods that will be applied to identify EBPs not only in the health sector, but also in the social sciences. The multi-layered approach to developing a checklist through systematic literature review followed by refining the list through a Delphi process will enhance the transparency and credibility of our findings. STEP working group members have years of experience in developing and evaluating EBP and publishing them (or studies evaluating them) in academic journals. However, there some challenges may exist: first, EBP that are not in the public domain will not be included for obvious reasons; second, we do not plan to conduct a quality assessment for all included EBP because there are no appraisal tools widely accepted.

We will collaborate with some organizations, institutions, and groups who are preparing and/or using EBP to implement STEP, and continuously seek feedback from developers, users, and stakeholders. We will disseminate the STEP by publishing into a peer-reviewed journal, presenting to relevant stakeholders and translating into different languages. We will update the checklist based on the continuous feedback and new relevant research evidence.

Availability of data and materials

Not applicable.

Abbreviations

3IE:

International Initiative for Impact Evaluation

CT:

Coordination team

DP:

Delphi panel

E4A:

Evidence for action

EBP:

Evidence briefs for policy

EIP:

Evidence-informed policymaking

EQUATOR:

Enhancing the QUAlity and Transparency Of health Research

EVIPNet:

Evidence-Informed Policy Network

HSG:

Health systems global

PEERSS:

Partnership for evidence and equity in responsive social system

SDGs:

Sustainable development goals

STEP:

STandard reporting guideline of Evidence briefs for Policy

UN:

United Nations

WHO:

World Health Organization

References

  1. Chalmers I. If evidence-informed policy works in practice, does it matter if it doesn’t work in theory? Evid Policy. 2005;1(2):227–42. https://doi.org/10.1332/1744264053730806.

    Article  Google Scholar 

  2. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: what is evidence-informed policymaking? Health Res Policy Sys. 2009;7:S1. https://doi.org/10.1186/1478-4505-7-S1-S1.

    Article  Google Scholar 

  3. Task Force on Health Systems Research. Informed choices for attaining the Millennium Development Goals: towards an international cooperative agenda for health-systems research. The Lancet. 2004;364(9438):997–1003. https://doi.org/10.1016/S0140-6736(04)17026-8.

    Article  Google Scholar 

  4. United Nations. Transforming Our World: The 2030 Agenda for Sustainable Development. https://www.un.org/ga/search/view_doc.asp?symbol=A/RES/70/1&Lang=E. Accessed 13 Dec 2021.

  5. Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Res Policy Sys. 2009;7:I1. https://doi.org/10.1186/1478-4505-7-S1-I1.

    Article  Google Scholar 

  6. World Health Organization. Evidence briefs for policy: using the integrated knowledge translation approach: guiding manual. World Health Organization. Regional Office for Europe; 2020.

  7. Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 13: Preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 2009;7(1):1–9. https://doi.org/10.1186/1478-4505-7-S1-S13.

    Article  Google Scholar 

  8. Adam T, Moat KA, Ghaffar A, Lavis JN. Towards a better understanding of the nomenclature used in information-packaging efforts to support evidence-informed policymaking in low-and middle-income countries. Implement Sci. 2014;9(1):1. https://doi.org/10.1186/1748-5908-9-67.

    Article  Google Scholar 

  9. SURE Collaboration. SURE Guides for preparing and using evidence-based policy briefs. Organizing and running policy dialogues. Kampala. 2011.

  10. Zhang J. Reporting standard research on evidence-based health policy brief. 2021. https://kns.cnki.net/kns8/defaultresult/index. Accessed 13 Dec 2021.

  11. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2): e1000217. https://doi.org/10.1371/journal.pmed.1000217.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12. https://doi.org/10.1111/j.1365-2648.2006.03716.x.

    Article  PubMed  Google Scholar 

  13. Ako-Arrey DE, Brouwers MC, Lavis JN, Giacomini MK. Health system guidance appraisal—concept evaluation and usability testing. Implement Sci. 2015;11(1):1–3. https://doi.org/10.1186/s13012-015-0365-3.

    Article  Google Scholar 

  14. Linstone HA, Turoff M, editors. The Delphi method. Reading: Addison-Wesley; 1975.

    Google Scholar 

  15. Wang X, Zhou Q, Chen Y, Yao L, Wang Q, Wang M, Yang K, Norris SL. Protocol of reporting items for public versions of guidelines: the Reporting Tool for Practice Guidelines in Health Care—public versions of guidelines. BMJ Open. 2019;9(3): e023147. https://doi.org/10.1136/bmjopen-2018-023147.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Chen Y, Yang K, Marušić A, Qaseem A, Meerpohl JJ, Flottorp S, Akl EA, Schünemann HJ, Chan ES, Falck-Ytter Y, Ahmed F. A reporting tool for practice guidelines in health care: the RIGHT statement. Ann Intern Med. 2017;166(2):128–32. https://doi.org/10.7326/M16-1565.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This project is funded by the major project of the National Social Science Foundation of China: “Research on the Theoretical System, International Experience and Chinese Path of Evidence-based Social Science” (Project No. 19ZDA142). The funder will not interference in the design of the study, any aspect of data collection, analysis or interpretation, or in the writing and publication of the output.

Author information

Authors and Affiliations

Authors

Contributions

YL conceptualized the manuscript. XY and QW drafted the protocol manuscript. KM, CM, CV, DP, YA, FE, RF, JH, MK, LB, ML, and JL reviewed and edited the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Yaolong Chen.

Ethics declarations

Ethics approval and consent to participate

The protocol was registered on the websites of EQUATOR network (https://www.equator-network.org/library/reporting-guidelines-under-development/#84). Participants will be in written informed about the purpose and methodology of the study to decide whether they will be willing to participate voluntarily and anonymously. Signed consent forms will be delivered to the researchers prior to the completion of the questionnaires. We received the ethics approval from the Ethics Committee of the School of Public Health, Lanzhou University, China (IRB21111901).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, X., Wang, Q., Moat, K. et al. Development of a STandard reporting guideline for Evidence briefs for Policy (STEP): context and study protocol. Health Res Policy Sys 20, 82 (2022). https://doi.org/10.1186/s12961-022-00884-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-022-00884-5

Keywords