Skip to main content

Evaluating the impact of the global evidence, local adaptation (GELA) project for enhancing evidence-informed guideline recommendations for newborn and young child health in three African countries: a mixed-methods protocol

Abstract

Background

Poverty-related diseases (PRD) remain amongst the leading causes of death in children under-5 years in sub-Saharan Africa (SSA). Clinical practice guidelines (CPGs) based on the best available evidence are key to strengthening health systems and helping to enhance equitable health access for children under five. However, the CPG development process is complex and resource-intensive, with substantial scope for improving the process in SSA, which is the goal of the Global Evidence, Local Adaptation (GELA) project. The impact of research on PRD will be maximized through enhancing researchers and decision makers’ capacity to use global research to develop locally relevant CPGs in the field of newborn and child health. The project will be implemented in three SSA countries, Malawi, South Africa and Nigeria, over a 3-year period. This research protocol is for the monitoring and evaluation work package of the project. The aim of this work package is to monitor the various GELA project activities and evaluate the influence these may have on evidence-informed decision-making and guideline adaptation capacities and processes. The specific project activities we will monitor include (1) our ongoing engagement with local stakeholders, (2) their capacity needs and development, (3) their understanding and use of evidence from reviews of qualitative research and, (4) their overall views and experiences of the project.

Methods

We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change. A series of interconnected qualitative and quantitative data collections methods will be used, including knowledge translation tracking sheets and case studies, capacity assessment online surveys, user testing and in-depth interviews, and non-participant observations of project activities. Participants will comprise of project staff, members of the CPG panels and steering committees in Malawi, South Africa and Nigeria, as well as other local stakeholders in these three African countries.

Discussion

Ongoing monitoring and evaluation will help ensure the relationship between researchers and stakeholders is supported from the project start. This can facilitate achievement of common goals and enable researchers in South Africa, Malawi and Nigeria to make adjustments to project activities to maximize stakeholder engagement and research utilization. Ethical approval has been provided by South African Medical Research Council Human Research Ethics Committee (EC015-7/2022); The College of Medicine Research and Ethics Committee, Malawi (P.07/22/3687); National Health Research Ethics Committee of Nigeria (01/01/2007).

Peer Review reports

Background

Sub-Saharan Africa (SSA) has the highest under-five mortality rate in the world [1]. Although the global under-five mortality rate declined from 76 to 38 per 1000 live births between 2000 and 2019, more than half of the deaths in children and youth in 2019 were among children under 5 years, approximately 5.2 million deaths [1]. Poverty-related diseases including pneumonia, diarrhoea and malaria remain amongst the leading causes of death in children under-5 years [2].Thus, despite progress in the health of young children globally, most countries in SSA fall below the average gains and do not meet maternal and child health targets set by the United Nations Sustainable Development Goal 3 to ‘ensure healthy lives and promote wellbeing’ (1). As of December 2021, under-five mortality rates were reported as 113.8, 38.6 and 32.2 per 1000 live births for Nigeria, Malawi and South Africa, respectively [3]. Factors accounting for regional disparities in child mortality rates include poverty, socioeconomic inequities, poor health systems and poor nutrition, with coronavirus disease 2019 (COVID-19) adding substantially to the burden [4].

Addressing healthcare issues such as these requires an evidence-informed approach, where intervention design and implementation are based on the best available evidence, to ensure that scarce resources are used effectively and efficiently, avoid harm, maximize good and improve healthcare delivery and outcomes [5,6,7]. Evidence-informed practices have been growing in SSA [6], and evidence ecosystems are becoming stronger. The evidence ecosystem reflects the formal and informal linkages and interactions between different actors (and their capacities and resources) involved in the production, translation and use of evidence [6, 8, 9]. Guidance that can be developed through this ecosystem includes evidence-based health technology assessments (HTA) and clinical practice guidelines (CPGs). CPGs include recommendations that are actionable statements that are informed by systematic reviews of evidence, and an assessment of the benefits and harms of alternative care options and are intended to optimize patient care [10]. They can help bridge the gap between research evidence and practice and are recognized as important quality-improvement tools that aim to standardize care, inform funding decisions and improve access to care, among others.

CPG method advancements, challenges and research gaps

Over the past decade, internationally and in SSA, there has been a rapid growth of CPGs developed for a range of conditions [11]. In particular, rapid evidence syntheses and guideline development methods has advanced in response to urgent evidence needs, especially during COVID [12, 13]. For example, WHO has developed guidelines for all key infectious conditions that cause most deaths. This development has been accompanied by a growing volume of research evidence around CPGs, including the processes for their rapid development, adaptation, contextualization, implementation and evaluation, and further spurred on by COVID. For example, global knowledge leaders, such as the WHO and the GRADE Working Group, have set standards for CPG development, outlining the steps of what is known as ‘de novo’ (from scratch) CPG development [14]. Another global group, the Guidelines International Network (G-I-N), is a network dedicated to leading, strengthening and supporting collaboration in CPG development, adaptation and implementation. They have published minimum standards and the G-I-N McMaster guideline checklist, which contains a comprehensive list of topics and items outlining the practical steps to consider for developing CPGs [15].

As CPG standards have evolved, however, so has the complexity of development and adaptation. In the context of poorer settings, such as sub-Saharan Africa (SSA), CPG development is prohibitively human and finance resource intensive. It requires scarce skills, even in the growing evidence-based healthcare (EBHC) community, and financial investments by government where resources are often directed to healthcare services, rather than policymaking processes. Against this backdrop, several studies have found that CPGs in the region often perform poorly on reporting on their rigour of development and editorial independence [16,17,18]. Other, more resource-efficient methods for guideline development in SSA are, therefore, essential and urgently needed. Moreover, investment in the overall management of the process is needed, including convening the guideline group and moving stepwise through a rigorous process.

Approaches for and challenges of guideline adaptation

There is also increased international recognition of the value of taking guidelines developed in one country and applying them to other countries. This can avoid duplication of effort and research waste in de novo guideline development, when useful guidelines may exist elsewhere [12, 19]. Against this backdrop, several adaptation methods are emerging for contextualization of recommendations to country needs (e.g. ADAPTE, adolopment and SNAP-it, amongst others) [19,20,21]. For example, WHO is developing strategies for adapting and implementing their CPGs at country level. One example is the WHO Antenatal Care Recommendations Adaptation Toolkit lead by the Department of Sexual and Reproductive Health and Research [22]. Their approach is pragmatic and transparent. Another approach is so-called ‘adolopment’, a GRADE method, in which the original guideline evidence is used, either adopted or adapted, considering contextual evidence such as costs and feasibility and local values [20]. Adolopment involves convening a guideline panel, reviewing available evidence and local contextual evidence and weighing up the panel’s judgements to make recommendations that are fit for purpose [20].

Despite these advances in CPG adaptation methods, many countries and professional associations in sub-Saharan Africa still use expert opinion-based approaches or proceed to prepare their own systematic reviews and guidelines, ultimately perpetuating resource wastage and duplication of efforts [23]. Moreover, when countries do adapt and contextualize other countries’ guidelines, there is frequently a lack of transparency and reporting on changes, without clarity on why or by whom. This in turn casts doubts on the recommendation’s credibility. For example, guidelines for child health in sub-Saharan Africa are usually derived from the WHO and UNICEF. However, adaptation of such guidelines and recommendations to national contexts is not well described [24]. Transparency in guideline adaptation is critical for creating trustworthy, context-sensitive recommendations. What guideline adaptation methods work best and how these can be transparently implemented in the context of lower resource settings, remain key research questions. Therefore, despite the emergence of several guideline adaptation approaches, we need to explore and understand how best to adapt recommendations from one context to another [25].

Qualitative evidence to inform guideline panels decisions

Another major advancement within guideline research has been growing recognition of the potential contribution of qualitative research evidence [26, 27]. Traditionally, guidelines have been informed by systematic reviews of the effectiveness of specific interventions [14]. Such reviews provide robust evidence about which interventions ‘work’. However, there is appreciation that evidence regarding the potential effectiveness of an intervention is not sufficient for making recommendations or decisions. Policymakers also need to consider other issues, including how different stakeholders’ value different outcomes, the intervention’s acceptability to those affected by it and the feasibility of implementing the intervention [28,29,30]. Evidence from qualitative research is particularly well suited to exploring factors that influence an intervention’s acceptability and feasibility [31, 32]. The use of qualitative research to inform recommendations by guidelines has become easier in recent years as systematic reviews of qualitative studies have become more common, and the methods for these reviews are now well developed [33]. The first WHO guideline to systematically incorporate reviews of qualitative studies was published in 2012 in the field of task-shifting for maternal and child health [31]. The inclusion of this qualitative evidence helped shape the panel’s recommendations [32], and this approach is now included in the WHO Handbook for Guideline Development and has been applied in many other WHO CPGs [34, 35].

However, a key challenge in using findings from systematic reviews of qualitative evidence is communicating often complex findings to users such as guideline panel members to facilitate effective knowledge translation. While there is now considerable research on communicating findings from reviews of intervention effectiveness [36], there is limited experience on the usefulness of different options for packaging and presenting findings from systematic reviews of qualitative evidence to CPG panels. To make best use of this evidence, we need presentation formats that are accessible to users who may be unfamiliar with qualitative methods, are concise and simple while retaining sufficient detail to inform decisions and clearly present ‘confidence in the evidence from systematic reviews of qualitative evidence’ (GRADE-CERQual) assessments of how much confidence users should place in each finding [37]. In addition, we need to understand how qualitative evidence included in global guidelines, such as those produced by WHO, is interpreted and used in country-level guideline adaptation processes.

Communicating clinical practice guidelines to end-users

A final key guideline method advancement has been around the development of multi-layered and digitally structured communication formats for end users [38, 39]. Guidelines are not an end in themselves. Recommendations may lack impact if not adequately communicated and disseminated to those who need to implement them, namely healthcare providers, managers and the public. Indeed, in a South African study of primary care guideline national policymakers, subnational health managers and healthcare providers agreed that dissemination is a particular gap [40]. While guidelines typically are produced as static documents (e.g. PDF formats), information technology is needed to enhance dissemination. The MAGIC authoring and publication Platform (MAGICapp/) was developed for this purpose (https://magicevidence.org/magicapp/). MAGICapp is a web-based tool that enables evidence synthesizers and guideline organizations to create, publish and dynamically update trustworthy and digitally structured evidence summaries, guidelines and decision aids in user-friendly formats on all devices. Such digital multi-layered formats allow different users to rapidly find recommendations, while having the supporting evidence for them one click away [41]. MAGICapp, used by WHO, NICE and professional societies across the world, holds potential to enhance the impact of evidence-informed guideline recommendations in practice, in an enhanced evidence ecosystem [9]. However, the usability of the MAGICapp in sub-Saharan Africa, based on local user preferences for different communication formats, are key research questions.

Against this backdrop, the Global Evidence, Local Adaptation (GELA) project will maximize the impact of research on poverty-related diseases through enhancing researchers and decision makers’ capacity to use global research to develop locally relevant guidelines for newborn and child health in Malawi, Nigeria and South Africa. These guidelines will build on and add value to the large-scale programme of child health guideline development from agencies such as the WHO, to support adaptation and implementation led by national ministries in collaboration with WHO Afro regional office.

Brief overview of the GELA project aim, objectives and approach

The overarching aim of GELA is to bridge the gap between current processes and global advances in evidence-informed decision-making and guideline development, adaptation and dissemination by building skills and sharing resources in ways that can be sustained beyond the project period. The project has seven linked and related work packages (WPs) to support delivery of the planned project deliverables. Table 1 provides a brief summary of the activities of each WP. This protocol outlines our approach for the monitoring and overall evaluation of the project activities and impact (WP 6).

Table 1 Overview of GELA project work packages

The project will be implemented in three SSA countries: Malawi, South Africa and Nigeria over a 3-year period. The project adopts a multi-faceted multidisciplinary research and capacity strengthening programme using primary and secondary research, guideline adaptation methodology and digital platforms to support authoring delivery and dynamic adaptation. These processes will offer bespoke capacity strengthening opportunities for policy makers, researchers and civil society. Throughout the project, we plan for innovations in the tools we use, accompanied by comprehensive evaluation of all aspects of the research, research uptake into policy and capacity strengthening.

This current proposal is for WP6: monitoring and evaluation

Ongoing monitoring and evaluation of project processes and activities will help facilitate ongoing engagement between researchers and stakeholders throughout the research project. This will in turn help ensure that the project is centred on a common goal, with clear understandings of the different research activities and potential impact. This can also promote research uptake and enable researchers to make adjustments to project activities, maximizing stakeholder engagement and research utilization.

M&E aims & objectives

The overarching aim of the monitoring and evaluation work package is to monitor and evaluate the various GELA project activities and processes, including whether, how and why activities took place or if goals were met.

The specific monitoring and evaluation objectives are to:

  1. 1.

    Monitor ongoing engagement with local stakeholders across work packages and explore what worked and didn’t and why;

  2. 2.

    Assess the capacity development needs of guideline panels and steering group committees and explore their views and experiences of the project’s capacity development activities;

  3. 3.

    Explore guideline panelists’ experiences with reading and using evidence from reviews of qualitative research, including their preferences regarding how qualitative review findings are summarized and presented;

  4. 4.

    Evaluate guideline panelists’, steering group committees’ and project team members’ overall views and experiences of the project, including the what works or not, to influence evidence-informed decision-making and guideline adaptation processes

Methods

Overall approach

We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change (Table 2). The theoretical underpinning for the GELA project across all work packages is related to the three-layered behaviour change wheel comprising opportunity, capability and motivation [42]. The design, delivery and implementation of multi-stakeholder integrated activities based on identified priority areas and needs is expected to lead to guideline related improved capacity, practice and policy within each country’s health system. Certain objectives also have specific underpinning theoretical frameworks, in addition to the overarching project Theory of Change, which are explained under the respective objectives below. A series of interconnected qualitative and quantitative data collections methods will be used to address each objective.

Table 2 Overarching GELA project Theory of Change

In what follows, we describe each objective and the methods we will use to achieve it, separately. However, in many cases the qualitative data collection cuts across objectives, with the same interviews and observations being used to explore multiple issues simultaneously (e.g. knowledge translation, capacity, overall views and experiences of the project, etc.). The relationship between the different objectives and associated methods are depicted in Tables 3 and 4. Table 3 outlines the stakeholder groups included in the monitoring and evaluation work package, including their composition and for which objectives they are targeted. Table 4 provides the timeline for the different data collection methods and how they relate to each across the objectives.

1. Objective 1: monitor ongoing engagement with local stakeholders across work packages and explore what worked and did not work and why

Table 3 Monitoring and evaluation stakeholder matrix
Table 4 Timeline and relationship between different data collection activities

Overall approach for this objective

This objective will be guided by an integrated knowledge translation (IKT) approach. IKT focuses on the important role of stakeholder engagement in enhancing evidence-informed decision-making [43]. As part of work package 4 (‘dissemination and communication’), knowledge translation (KT) champions have been identified in each of the three countries and will work together to develop and implement country-level KT strategies. This will include defining KT objectives, identifying and mapping relevant stakeholders, prioritizing those we will actively engage and developing a strategy for engaging each priority stakeholder. We will monitor these engagements through the development and implementation of a tracking sheet, qualitative case studies and semi-structured interviews.

Participants

Participants will comprise of knowledge translation (KT) champions and relevant country-level stakeholders. KT champions are GELA project staff who have dedicated time to work on the communication, dissemination and engagement aspects at a country-level. At least one KT champion has been identified for each of Malawi, Nigeria and South Africa.

Relevant country-level stakeholders will be identified as part of the KT strategy development (WP4) and will comprise any health decision-makers, e.g. health practitioners, community groups, health system managers, policy-makers, researchers and media.

Tracking sheet and qualitative case studies

A tracking sheet will be used to capture information for each stakeholder related to the purpose, message, medium or forum, messenger, timing and resources for engagement. KT champions in each country will be responsible for tracking these details on a continuous basis, and the tracking sheet will be monitored bi-monthly at a meeting with KT champions from the three country teams. This will help us monitor whether and how engagement activities are taking place, as well as the strategies for implementation. The tracking sheets will consist of different in-country stakeholders (e.g. government officers, health professional associations, researchers, media, etc.), and there may be several goals for engaging each individual stakeholder. The engagement strategy will be reviewed and updated as priority stakeholders change over the research stages and project period. As such, the sample size will be determined iteratively.

We will analyse information with descriptive statistics. For example, we will group and count by categories: number and type of stakeholders, type of engagement activities, type of KT products produced, type of forum or medium used for dissemination, frequency and duration of engagement, follow-ups, intensive engagement period and resources required for engagement.

We will also develop case stories (or impact stories) describing engagement activities and processes between project staff and relevant stakeholders. The case studies will help us monitor successful engagement, disseminate best practice scenarios and draw out lessons for future engagements. We will identify case stories through the tracking sheet and at bi-monthly meetings with the KT co-ordinator, where KT champions will be asked to share success stories or learning moments. KT champions will not know which ‘case’ will be selected for the case study in advance. The information will be collected by the KT co-ordinator, who is not involved in any of the country strategy implementation. The information collected from the KT champions (and messenger, if the messenger is not the KT champion) will be via a standard case story template, including aim of engagement, what the engagement was, experiences from both sides (quotes to be included in stories), success of engagement, lessons learnt and any future engagement plans. The number of cases will be determined iteratively. The intention is to develop one case story from each country annually, showcasing different cases, e.g. type of KT goal, type of stakeholder, type of KT medium/forum, etc.

Semi-structured interviews

At project close (month 30), we will conduct semi-structured interviews to explore if, why and how project KT goals were met and what planned stakeholder engagements worked (and did not work) and why. The interviews will be conducted with KT champions, other messengers (e.g. communication officers), country leads and selected stakeholders. At least two people from each county (KT champion and messenger and/or stakeholder) will be interviewed, and so there will be six to eight interviews in total. Participants will be selected purposively for information-rich cases that can help yield insights and in-depth understanding of the nature and success (or not) of our stakeholder engagements [44].

These interviews will form part of the interviews conducted with project team members more broadly as part of objective 4, the methods of which are therefore described in more detail below.

2. Objective 2: assess the capacity development needs of guideline panels and steering group committees and explore their views and experiences of the project’s capacity development activities.

Overarching theoretical lens

We will draw on the Kirkpatrick model [45] as the underpinning theoretical framework for this objective. This model evaluates training effectiveness across four levels: (1) reaction, (2) learning, (3) behaviour and (4) results. The ‘reaction level’ assesses the degree of satisfaction of participants with the training event. The ‘learning level’ examines learning among participants both before and after the training event to determine any change in knowledge [46, 47]. The ‘behaviour level’ assesses whether the training event has provided any favourable change in behaviour among participants. The final ‘results level’ assesses the use of knowledge gained through the training event within the workplace [46, 47].

To assess the potential difference that project capacity development activities make, the outcomes of interest will be those related to training in evidence-based healthcare (EBHC). An overview of systematic reviews by Young and colleagues identified that EBHC training often aims to ‘improve critical appraisal skills and integration of results into decisions, and improved knowledge, skills, attitudes and behaviour among practising health professionals’ [48, 49].

Overall approach for this objective

We will employ mixed methods to achieve this objective, including three rounds of online surveys (at baseline, mid-line and at the project close) as well as semi-structured interviews (at project close) and non-participant observations of meetings (various). The first online survey at baseline will assess the capacity needs of the guideline panels and steering group committees in South Africa, Malawi and Nigeria, and the two subsequent online surveys will assess the potential difference project capacity development activities make on these groups across all the four levels of the Kirkpatrick model, i.e. reaction, learning, behaviour and results. The capacity needs and progress of these groups will also be explored qualitatively through semi-structured interviews and observations of meetings.

Details of the project capacity development activities that will be implemented as part of work package 5 (‘capacity strengthening and sharing’) of the GELA project are outlined in Table 1 (above). All members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria will be invited and encouraged to attend all project capacity development activities. ‘On the job’ capacity building will also take place during the various meetings convened with these groups, as they are supported to identify priority topics, to appraise and discuss the evidence used to inform the recommendations and to formulate the final recommendations.

Participants

Participants will comprise members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria. Table 3 (above) provides details of the composition of the guideline panels and steering group committees.

Online surveys

Procedures and data collection tools

At baseline (at approximately 6 months before engagement in any project training activities), at mid-line (month 18) and at the project close (month 30), all members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria will be invited, via email, to participate in a survey. In each of the three countries the guideline development group and steering group committees will include approximately 20 and 10 members, respectively; we will therefore aim to have 90 participants in total complete the survey. The email invitation to all three survey rounds will inform participants about the nature of the study and direct them to an online survey. The landing page of the survey will provide information about the purpose of the research project and what is being requested from the participants, with a consent statement at the end which the participant will be required to agree to before being able to continue with the survey. Data will only be collected from participants who consent to freely participate in the study. The survey will be carried out using a secure online survey platform (such as Microsoft Forms) where all cookies and IP address collectors will be disabled to protect the confidentiality of the participants and to avoid tracking of the participant activities online. Unique identifiers (last six numbers of their ID) will be used to track participants responses over time and link data from baseline to project close.

The baseline survey will be a short (10–25 min) form that will ask participants about their capacity needs and knowledge/skills in evidence-based healthcare (EBHC) and decision-making. The survey will capture demographic variables of participants at baseline, mid-term and at the end of the project. It will assess the training needs of participants at baseline, participants’ satisfaction at the end of each training activity, the knowledge and skills at baseline, mid-term and at the end of the project. Participants’ behaviour will also be assessed using open-ended questions and vignettes. The surveys will focus on all four levels (i.e. reaction, learning, behaviour and results) of the Kirkpatrick model.

Data management and analysis

All data collected on the secure online survey platform will be coded, cleaned and entered into STATA. Data collected for the baseline survey will be analysed using descriptive statistics to determine the frequency of the various training needs and qualitative data gathered using the open-ended questions will be analysed thematically using manual coding (or if available and dataset is large), and NVivo or a similar tool will be used to identify the recurring themes which emerge in the data collected about the key training needs of participants.

Data collected for the surveys conducted at midpoint and at project close will be analysed using descriptive statistics to determine if there has been a change in the learning, knowledge gained and behaviours over time, as well as the extent of the potential application of evidence-based practice, while the data collected using the open-ended questions will be analysed using thematic analysis outlining how project capacity development activities informed particular outcomes and results in the participant’s workplace. To determine change in skills (and trends over time such as confidence improvement or decay), the descriptive statistics will be supplemented by appropriate inferential statistics for repeated measures (paired data) such as McNemar or paired t-tests, reporting change in percentages as mean differences (such as self-reported confidence) with 95% confidence intervals or/and frequencies. Descriptive trends over time will also be presented graphically using line graphs or other visual aids as appropriate. However, these will be interpreted with caution as the primary analysis is descriptive. Statistical significance will be set at a p value of 0.05.

Semi-structured interviews

Procedures and data collection tools

At project close (month 30), we will conduct semi-structured interviews with a sample of members from the guideline panels and steering group committees in South Africa, Malawi and Nigeria. Sampling will be purposive, with the aim of understanding the broad range of needs, experiences and perspectives and ensuring that the sample reflects a range of socio-demographic characteristics and stakeholder categories. We will begin with a sample size of 10–15 participants in each country; however, sampling will continue if we have not reached saturation of the data through the initial sample size [44].

Participants will be contacted, either by telephone or via email, and invited to participate in an interview. Interviews will be conducted face-to-face or electronically (e.g. using Microsoft Teams) at a date and time chosen by participants. Face-to-face interviews will take place at a location convenient to participants, which is conducive to a confidential exchange. The interviews will last between 45 and 60 min and will be conducted by researchers trained in qualitative research methodologies and interviewing techniques. The interviews will be guided by a semi-structured topic guide and will include questions informed by the four levels (i.e. reaction, learning, behaviour and results) of the Kirkpatrick model. Specifically, the questions will explore participants’ views and experiences regarding their capacity development needs and expectations of the project; whether and why these expectations were met (or not), the project capacity development activities, what they learned (or not) from these activities and what impact participants believe they have had (or may have) on their practices.

Verbal and written information about the study will be provided to all participants taking part in interviews. Written informed consent will be obtained from all participants before proceeding with the interview. With the permission of participants, all interviews will be digitally recorded.

Non-participant observations

We will conduct non-participant observations of guideline panel and steering group committee meetings. Observational methods can provide useful data on what people do, how they interact with each other and how they engage with particular artefacts in situ (rather than their accounts of these) [50]. The steering group committees in each country will meet approximately twice over the project duration (with the option for additional meetings): an initial meeting for project orientation (month 2/3) and again to identify priority topics and guideline gaps (month 6). Guideline panels in each country will meet approximately three times over the project duration (with the option for additional meetings): an initial meeting for project orientation and outcome prioritization (month 6/7), another potential meeting if necessary to finalize outcome prioritization and a final meeting to draft recommendations for the guideline (months 17–20). Meetings for both groups will be held virtually or in person, informed by preferences of the committee.

With the exception of the initial steering group committee (month 2/3), at least one researcher will be present to observe guideline panel and steering group committee meetings. The observer will aim to identify any capacity-related needs, expectations, gaps, strengths, achievements and challenges and the contexts in which these occur. He or she will also pay particular attention to group dynamics and the interactions between members and different stakeholder groups, and the potential impact of these on capacity-related issues. Observations will be informed by Lofland’s [51] criteria for organizing analytical observations (acts, activities, meanings, participation, relationships and settings). The observer will take detailed observational notes. With consent of the attendees, all meetings will also be digitally recorded. The recordings will be used to identify further issues not identified and to deepen or clarify issues noted, through the real-time observations of verbal engagements.

Data management and analysis: semi-structured interviews and observations

Interview and meeting recordings will be transcribed verbatim, and all personal identifying information will be removed from transcripts. The anonymized transcripts, together with observational notes, will be downloaded into Nvivo, a software programme that aids with the management and analysis of qualitative data. Analysis of the qualitative data will proceed in several rounds. First, as with all qualitative data analysis, an ongoing process of iterative analysis of the data will be conducted throughout the data collection period. Second, we will use a thematic analysis approach, using the phases described by Braun and Clarke [52], to identify key themes pertaining to participants’ capacity development needs and expectations and whether, how and why project capacity development activities met (or not) these needs and expectations. Finally, findings from the surveys (as described above) will also be integrated with the findings from the thematic analysis using a ‘narrative synthesis’ approach, a technique recommended by the Cochrane Collaboration as a way of synthesizing diverse forms of qualitative and quantitative evidence in mixed methods studies [53, 54]. This approach will allow for both robust triangulation, and a more comprehensive interpretation of the difference project capacity development activities may have made on the guideline panels and steering group committees.

3. Objective 3: explore guideline panelists’ experiences with reading and using evidence from reviews of qualitative research, including their preferences regarding how qualitative review findings are summarized and presented.

Overall approach for this objective

Objective 3 of the monitoring and evaluation stakeholder matrix work package explores how guideline panels view and experience evidence from the review(s) of qualitative research, including how it is summarized and presented. Here, we will employ a user testing approach, drawing on the methods and guidance of the SURE user test package 2022 developed by Cochrane Norway (https://www.cochrane.no/our-user-test-package) and which has been used to test various evidence-related products [55,56,57,58]. User testing involves observing people as they engage with a particular product and listening to them ‘think-aloud’. The goal is to gain an understanding of users’ views and experiences, the problems they face and to obtain suggestions for how a product may be improved [55,56,57,58].

We will begin by identifying or preparing relevant reviews of qualitative research. We will then develop review summary formats and explore guideline panel members’ views and experiences of these formats. We will revise the formats in multiple iterative cycles.

  1. 1.

    Identifying or preparing relevant reviews of qualitative research

As part of WP2 of the project (‘evidence synthesis’), we will identify relevant review(s) of qualitative research, including reviews exploring how people affected by the interventions of interest value different outcomes, the acceptability and feasibility of the intervention and potential equity, gender and human rights implications of the intervention. These reviews need to be assessed as sufficiently recent and of a sufficient quality. They also need to have applied GRADE-CERQual assessments to the review findings. Where necessary, we will update existing reviews or prepare reviews ourselves.

  1. 2.

    Developing the review summaries

In WP3 of the project (‘decision-making’) the evidence from these reviews will be provided to guideline panels as part of the evidence-to-decision (‘EtD’) frameworks that will inform the recommendations they develop (see Table 1 for further details about project work packages 2 and 3). Our next step will therefore be to prepare summaries of the reviews in a format that can easily be included in the EtD frameworks.

Each summary needs to present review findings that are relevant to specific parts of the EtD framework (typically the ‘values’, ‘acceptability’, ‘feasibility’ and ‘equity’ components). It also needs to include information about our confidence in these findings. Finally, the summary needs to indicate where this evidence comes from and to allow guideline panels to move from the summary to more detailed information about the evidence.

Most of this information is found in the review’s Summary of Qualitative Findings tables. However, these tables are usually too large for EtD frameworks and are not tailored to each framework component. We will, therefore, start by creating new summaries, using a format that we have previously used in EtD frameworks [59,60,61] but that we have not user tested. As opposed to the Summary of Qualitative Findings tables, where each finding and our confidence in the finding, is presented individually in separate rows, this format involves pulling the findings and confidence assessments together in short, narrative paragraphs.

  1. 3.

    User testing the summary format

For our first set of user tests, we will observe guideline panels participating in the CPG panel simulation workshops. For our second round of user tests, we will observe how the guideline panels experience and interact with this qualitative evidence during the real guideline processes. Third, we will then test a potentially refined format with a selection of guideline panel members using a semi-structured interview guide. Finally, at the end of the project, we will conduct semi-structured interviews with a selection of guideline panel members to explore their broader views and experiences of interpreting and using evidence from reviews of qualitative studies in their deliberation processes. Figure 1 provides a visual depiction of this iterative process.

Fig. 1
figure 1

Iterative approach for user testing evidence from reviews of qualitative research

Overarching theoretical lens

We will draw on the adapted version of Peter Morville’s original honeycomb model of user experience [62] as the underpinning theoretical framework for this objective [63] (Fig. 1). This adapted version extends and revises the meaning of the facets of user experience depicted in the original model. It includes eight facets: accessibility, findability, usefulness, usability, understandability, credibility, desirability and affiliation. Accessibility involves whether there are physical barriers to gaining access; findability is about whether the person can locate the product or the content that they are looking for; usefulness is about whether the product has practical value for the person; usability comprises how easy and satisfying the product is to use; understandability is about whether the person comprehends correctly both what kind of product it is and the content of the product (and includes both user's subjective perception of her own understanding and an objective measure of actual/correct understanding); credibility comprises whether the product/content is experienced as trustworthy; desirability is about whether the product is something the person wants and has a positive emotional response to it; affiliation involves whether the person identifies with the product, on a personal or a social level, or whether it is alienating and experienced as being not designed for ‘someone like me’. The adapted model also adds to the original model a dimension of user experience over time, capturing the chronological and contingent nature of the different facets.

Participants

Participants will comprise members of the guideline panels in South Africa, Malawi and Nigeria. Table 3 (above) provides details of the composition of the guideline panels.

Non-participant observations: guideline panel simulation workshops and guideline panel meetings

We will conduct non-participant observations of the CPG panel simulation workshops and the subsequent guideline panel meetings for developing the recommendations. The CPG panel simulation workshops will run a simulation of a real guideline process and give guideline panels an opportunity to understand how the guideline process works before they participate in real panel meetings. The guideline panels in all three countries will be invited and encouraged to attend these workshops, which will form part of the project capacity development activities of WP5 (Table 1).

With the participants’ consent, both the simulation workshops and meetings will be digitally recorded and at least two observers will observe and take notes. The observations will focus on how guideline panel members refer to and interact with the summaries of qualitative evidence. Drawing on a user testing approach (https://www.cochrane.no/our-user-test-package), we will also look specifically for both problems and facilitators in the way the qualitative evidence is formatted, including ‘show-stoppers’ (the problem is so serious that it hindered participants from correct understanding or from moving forward), ‘big problems/frustrations’ (participants were confused or found something difficult but managed to figure it out or find a way around the problem eventually), ‘minor issues/cosmetic things’ (small irritations, frustrations and small problems that do not have serious consequences, as well as likes/dislikes), ‘positive/negative feedback’, ‘specific suggestions’, ‘preferences’ and any other ‘notable observations’, e.g. feelings of ‘uncertainty’.

Structured user testing interviews

Procedures and data collection tools

Based on the insights gained from the non-participant observations (above), we may make changes or refinements to our original summary format (Fig. 1). Once the guideline panel meetings have concluded (approximately by month 20), we will then conduct structured user testing interviews to test the potentially refined summary format. These interviews will be conducted with a sample of members from the guideline panels in South Africa, Malawi and Nigeria. Sampling will be purposive, with the aim of understanding the broad range of experiences and perspectives and ensuring the sample reflects a range of socio-demographic characteristics and stakeholder categories. As recommended (https://www.cochrane.no/our-user-test-package), we will begin with a sample size of six to eight participants in each country; however, sampling will continue until saturation is achieved [44].

Participants will be contacted, either telephonically or via email, and invited to participate in an interview. Interviews will be conducted face-to-face or electronically (e.g. using Skype or Teams) at a date and time chosen by participants. Face-to-face interviews will take place at a location convenient to participants, which is conducive to a confidential exchange. In line with the SURE user test package 2022 guidance, the interviews will last approximately 60 min (https://www.cochrane.no/our-user-test-package). They will be facilitated by a test leader, who will accompanied by at least one observer who will take notes. Both the test leader and observer(s) will be trained in user testing interviewing methodology and techniques. Verbal and written information about the study will be provided to all participants taking part in interviews. Written informed consent will be obtained from all participants before proceeding with the interview. With the permission of participants, all interviews will be video recorded.

For these interviews we will show panel members the latest version of the format, explore immediate first impressions, and then opinions about different elements of the summary. We may also show panel members different formats where we think this may be helpful. We will use a structured interview guide which draws heavily on other interview guides that been developed to user test evidence-related products [55,56,57,58]. It will include questions related the participant’s background; their immediate first impressions of the summary format(s); in-depth walk-through of the summary format(s), with prompts to think aloud what they are looking at, thinking, doing and feeling; and suggestions for improving the way the summary is formatted and for improving the user testing itself. We may ask follow-up questions to specific issues we observed in the simulation workshops and guideline panel meetings and/or create scenarios that resemble issues we observed in the workshops/meetings. This will be decided upon based on the findings that emerge from these workshops/meetings. The guide will be finalized once the relevant qualitative evidence (from WP2) has been produced and we have gained insights from the workshops and meetings.

As with the non-participant observations of meetings and workshops, throughout the interview, the observers will make notes about the participant’s experience as heard, observed and understood. Drawing on a user testing approach, they will look specifically for both problems and facilitators, specific suggestions, preferences and any other notable observations (as described above under ‘non-participant observations’).

Semi-structured interviews

Procedures and data collection tools

At project close (month 30), we will also conduct semi-structured interviews with a sample of members from the guideline panels in South Africa, Malawi and Nigeria. These will be the same interviews with guideline panel members as described in objective 2. In addition to exploring participants’ capacity development needs, expectations and achievements, the semi-structured topic guide will also explore their views and experiences of (and specific capacity in) interpreting and using evidence from reviews of qualitative studies in guideline processes. More specifically, questions will investigate participants’ familiarity/experience with qualitative evidence; their perceptions of different types of evidence, what constitutes qualitative evidence and the role of qualitative evidence in guideline processes; and their experiences of using the qualitative evidence in their deliberations as part of the project, including what influenced its use and whether they found it useful. Details pertaining to sampling, data collection procedures and collection tools are described in objective 2.

Data management and analysis

All interview and meeting recordings will be transcribed verbatim, and all personal identifying information will be removed from transcripts. The anonymized transcripts, together with observational notes (from the workshops, meetings and interviews), will be downloaded into a software programme that aids with the management and analysis of qualitative data. Analysis of the data will be guided by the user testing analysis methods described in the SURE user test package 2022 (https://www.cochrane.no/our-user-test-package). The analysis will proceed in several, iterative rounds to develop and revise the summary format and to inform the focus of subsequent data collection. After each user test, we will review our notes, first separately and then together. In line with the SURE user test package 2022 guidance, we will look primarily for barriers and facilitators related to correct interpretation of the summary’s contents, ease of use and favourable reception, drawing on the facets of the revised honeycomb model of user experience (Fig. 2). We will trace findings back to specific elements or characteristics of the summaries that appeared to facilitate or hinder problems. Before the next set of user tests, we will discuss possible changes that could address any identified barriers and make changes to the summary format.

Fig. 2
figure 2

Adapted version of Peter Morville’s honeycomb model of user experience

4. Objective 4: evaluate guideline panelists’, steering group committees’ and project team members’ overall views and experiences of the project, including what works or not, to influence evidence-informed decision-making and guideline adaptation processes.

Overall approach for this objective

This objective explores overall views and experiences of the project, with a focus on guideline panelists, steering group committees and project team members. Specifically, it seeks to gain an understanding of these three stakeholder groups’ more general views and experiences of the project activities they were involved with and whether, why and how these activities may influence (or not) evidence-informed decision-making and guideline adaptation processes. This will be achieved through semi-structured interviews.

Participants

Participants will comprise members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria, as well as members of the project team (as described in Table 3 above).

Semi-structured interviews

Procedures and data collection tools

At project close (month 30), we will conduct semi-structured interviews with a sample of members from the guideline panels and steering group committees in South Africa, Malawi and Nigeria. These will be the same interviews and participants as described in objective 2. In addition to exploring issues around capacity development and qualitative evidence, the interviews will also investigate participants’ views and experiences of the various project activities they were involved with, and whether, why and how these activities may influence (or not) evidence-informed decision-making and guideline adaptation processes. Details pertaining to sampling, data collection procedures and collection tools are described in objective 2.

At project close (month 30), we will also conduct semi-structured interviews with members of the project team (see Table 3 for details of project team composition). We will begin by interviewing all project management team members, WP leads and KT champions. Additional participants will be determined iteratively (depending on what emerges from initial interviews) and purposively, with the aim of understanding the broad range of experiences and perspectives and ensuring the sample reflects the various groups which make up the project team. Interviews will be conducted face-to-face or electronically (e.g. using Skype or Teams) at a date and time chosen by the interviewee. The interviews will last between 45 and 60 min and will be guided by a semi-structured topic guide. The questions will explore participants’ views and experiences of the respective work packages in which they were involved, including what the primary goals of the work package were; if, why and how these goals were met; and what worked and what did not work and why.

Data management and analysis: semi-structured interviews and observations

The same qualitative data analysis procedures and methods will be used as described in objective 2. For this objective, the thematic analysis will identify key themes pertaining to views and experiences of project activities, including what worked (or not) and why, whether, why and how the project may (or not) influence evidence-informed decision-making and guideline development, adaptation and dissemination processes in South Africa, Malawi and Nigeria and potential barriers and facilitators to the sustainability of this influence.

Discussion

Evidence-based guideline development is a multi-stakeholder, multi-perspective, complex set of tasks. There is limited, if any, research that has followed these steps from the perspectives of policymakers or researchers from start to end. The GELA project protocol sets out to monitor and evaluate various key steps in the process, using in-depth qualitative methods alongside appropriate surveys not only to inform the project as it progresses but also to understand the overall impact of all steps on development of transparent and contextually-rich guideline recommendations. Following WHO’s guideline steps, the tasks range from scoping stakeholder-informed priority topics to conducting relevant data gathering and evidence synthesis, followed by guideline panel meetings to reach consensus decisions and finally to produce recommendations that can be useful to end-users and improve health and care outcomes. The GELA project is undertaking a 3-year project to conduct these tasks in the context of newborn and child health priorities. We are doing this in collaboration with national ministries of health, academics, non-governmental partners and civil society groups in Malawi, Nigeria and South Africa. Overall, we aim build capacity across all collaborators for evidence-informed guideline development, while producing fit for context guideline recommendations, in accessible formats that benefit children, caregivers and health care providers.

As such, this is a practical research project, in that the products should directly impact care decisions at the national level but with the added benefit of being able to learn about what works or does not work for collaborative guideline development in country. We will also be applying emergent guideline adaptation methods to explore reducing duplication of expensive guideline development efforts in our lower resource settings. Our project addresses newborn and child health, keeping this most vulnerable population in our focus, hoping that producing sound evidence-based recommendations has the potential to impact care.

Through some of our formative work, we have completed a landscape analysis identifying and describing all available newborn and child health guidelines in each of the partner countries. In all countries there were similar findings, (1) there is no easy access to guidelines for end-users, thus locating a guideline requires effort and screening through multiple sources; (2) considering national priority conditions in this age group, there were often gaps in available current guidelines for managing children; and (3) when we appraised the guidelines using the global standard, AGREE II tool, we found that the reporting of guideline methods were poor, leaving it uncertain whether the recommendations were credible or whether any influences or interests had determined the direction of a recommendation. Finally, we expected to find many adapted guidelines, based on WHO or UNICEF or similar guidance available globally; however, very few of the identified guidelines stated clearly whether they had been adapted from other sources and, if so, which recommendations were adopted and which adapted.

Given progress globally in methods for guideline development, the continued poor reporting on guideline methods at the country level speak to a breakdown in skills-sharing globally, for example, WHO produces guidelines that are recognized as rigorous and follow good practice and reporting, but the same standards are not supported in country. Overall, GELA aims to address these key gaps in national guideline approaches for adaptation, but we need to recognize that this will be a long term process and that we need to learn from each other about what works and what may not serve us. Therefore, this protocol outlines our approach for monitoring several aspects of the project in our efforts to move closer to trustworthy and credible guidelines that all can use and trust for countries like ours.

Availability of data and materials

Not applicable.

Abbreviations

(PRD):

Poverty-related diseases

SSA:

Sub-Saharan Africa

CPGs:

Clinical practice guidelines

EIDM:

Evidence-informed decision-making

EBHC:

Evidence-based healthcare

GELA:

Global Evidence Local Adaptation

KT:

Knowledge translation

References

  1. Levels and trends in child mortality: report 2020. Estimates developed by UNICEF. ISBN: 978-92-806-5147-8. https://www.who.int/publications/m/item/levels-and-trends-in-child-mortality-report-2020. United Nations Children’s Fund. Accessed Aug 2024.

  2. Liu L, Oza S, Hogan D, et al. Global, regional, and national causes of under-5 mortality in 2000–15: an updated systematic analysis with implications for the sustainable development goals. Lancet. 2016;388(10063):3027–35.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Under-five mortality. 2021 https://data.unicef.org/topic/child-survival/under-five-mortality/. United Nations International Children’s Fund. Accessed Aug 2024.

  4. Stewart R, El-Harakeh A, Cherian SA. Evidence synthesis communities in low-income and middle-income countries and the COVID-19 response. Lancet. 2020;396(10262):1539–41.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Mijumbi RM, Oxman AD, Panisset U, Sewankambo NK. Feasibility of a rapid response mechanism to meet policymakers’ urgent needs for research evidence about health systems in a low income country: a case study. Implementation Sci. 2014;9:114.

    Article  Google Scholar 

  6. Stewart R, Dayal H, Langer L, van Rooyen C. The evidence ecosystem in South Africa: growing resilience and institutionalisation of evidence use. Palgrave Commun. 2019;5(1):90.

    Article  Google Scholar 

  7. Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Promotion of evidence-informed health policymaking in Nigeria: bridging the gap between researchers and policymakers. Glob Public Health. 2012;7(7):750–65.

    Article  PubMed  Google Scholar 

  8. Young T, Garner P, Clarke M, Volmink J. Series: clinical epidemiology in South Africa Paper 1: evidence-based health care and policy in Africa.: past, present, and future. J Clin Epidemiol. 2017;83:24–30.

    Article  PubMed  Google Scholar 

  9. Vandvik PO, Brandt L. Future of evidence ecosystem series: evidence ecosystems and learning health systems: why bother? J Clin Epidemiol. 2020;123:166–70.

    Article  PubMed  Google Scholar 

  10. Graham R, Mancher M, Wolman D, Greenfield S, Steinberg E. Clinical practice guidelines we can trust. Washington, DC: The National Academy Press; 2011.

    Book  Google Scholar 

  11. Kredo T, Bernhardsson S, Machingaidze S, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8.

    Article  PubMed  PubMed Central  Google Scholar 

  12. McCaul M, Tovey D, Young T, et al. Resources supporting trustworthy, rapid and equitable evidence synthesis and guideline development: results from the COVID-19 evidence network to support decision-making (COVID-END). J Clin Epidemiol. 2022;151:88–95.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Tricco AC, Garritty CM, Boulos L, et al. Rapid review methods more challenging during COVID-19: commentary with a focus on 8 knowledge synthesis steps. J Clin Epidemiol. 2020;126:177–83.

    Article  PubMed  PubMed Central  Google Scholar 

  14. World Health Organization. WHO handbook for guideline development. Geneva: World Health Organization; 2011.

    Google Scholar 

  15. Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–42.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Kredo T, Gerritsen A, van Heerden J, Conway S, Siegfried N. Clinical practice guidelines within the Southern African development community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases. Health Res Policy Syst. 2012;10:1.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Malherbe P, Smit P, Sharma K, McCaul M. Guidance we can trust? The status and quality of prehospital clinical guidance in sub-Saharan Africa: a scoping review. African J Emerge Med. 2021;11(1):79–86.

    Article  Google Scholar 

  18. McCaul M, Clarke M, Bruijns SR, et al. Global emergency care clinical practice guidelines: a landscape analysis. African J Emerge Med. 2018;8(4):158–63.

    Article  Google Scholar 

  19. Dizon JM, Grimmer K, Louw Q, Kredo T, Young T, Machingaidze S. South African guidelines excellence (SAGE): adopt, adapt, or contextualise? South African Med J. 2016;106(12):1177–8.

    Article  CAS  Google Scholar 

  20. Schünemann HJ, Wiercioch W, Brozek J, et al. GRADE evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.

    Article  PubMed  Google Scholar 

  21. ADAPTE Collaboration. 2009 The ADAPTE process: Resource toolkit for guideline adaptation, version 2. http://www.g-i-n.net/. Accessed Aug 2024.

  22. Barreix M, Lawrie TA, Kidula N, et al. Development of the WHO antenatal care recommendations adaptation toolkit: a standardised approach for countries. Health Res Policy Syst. 2020;18(1):70.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Opiyo N, Shepperd S, Musila N, English M, Fretheim A. The, “child health evidence week” and grade grid may aid transparency in the deliberative process of guideline development. J Clin Epidemiol. 2012;65(9):962–9.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implemen Sci. 2018;13(1):72.

    Article  Google Scholar 

  26. Langlois EV, Tunçalp Ö, Norris SL, Askew I, Ghaffar A. Qualitative evidence to improve guidelines and health decision-making. Bull World Health Organ. 2018. https://doi.org/10.2471/BLT.17.206540.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Carmona C, Baxter S, Carroll C. Systematic review of the methodological literature for integrating qualitative evidence syntheses into health guideline development. Res Synth Methods. 2021;12(4):491–505.

    Article  PubMed  Google Scholar 

  28. Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Res Alloc. 2012;10(1):9.

    Article  Google Scholar 

  29. Verkerk K, Van Veenendaal H, Severens JL, Hendriks EJ, Burgers JS. Considered judgement in evidence-based guideline development. Int J Qual Health Care. 2006;18(5):365–9.

    Article  PubMed  Google Scholar 

  30. Alonso-Coello P, Schunemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016. https://doi.org/10.1136/bmj.i2016.

    Article  PubMed  Google Scholar 

  31. Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process. Implement Sci. 2016;11:98.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Lewin S, Glenton C. Are we entering a new era for qualitative research? using qualitative evidence to support guidance and guideline development by the World Health Organization. Int J Equity Health. 2018;17(1):126.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Chapter 21: Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook. Accessed Aug 2024.

  34. Glenton C, Lewin S, Norris S. Using evidence from qualitative research to develop WHO guidelines (Chapter 15) World Health Organization Handbook for Guideline Development. 2nd ed. WHO: Geneva; 2016.

    Google Scholar 

  35. World Health Organisation. WHO recommendations: intrapartum care for a positive childbirth experience. Geneva: WHO; 2018.

    Google Scholar 

  36. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620–6.

    Article  PubMed  Google Scholar 

  37. Lewin S, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table. Implement Sci. 2018;13(Suppl 1):10.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Brandt L, Vandvik PO, Alonso-Coello P, et al. Multilayered and digitally structured presentation formats of trustworthy recommendations: a combined survey and randomised trial. BMJ Open. 2017;7(2): e011569.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Kredo T, Cooper S, Abrams AL, et al. ’Building on shaky ground’-challenges to and solutions for primary care guideline implementation in four provinces in South Africa: a qualitative study. BMJ Open. 2020;10(5): e031468.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9.

    Article  PubMed  Google Scholar 

  42. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Jessani NS, Rohwer A, Schmidt B-M, Delobelle P. Integrated knowledge translation to advance noncommunicable disease policy and practice in South Africa: application of the exploration, preparation, implementation, and sustainment (EPIS) framework. Health Res Policy Syst. 2021;19(1):82.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229–45.

    Article  PubMed  Google Scholar 

  45. Roos M, Kadmon M, Kirschfink M, et al. Developing medical educators–a mixed method evaluation of a teaching education program. Med Educ Online. 2014;19:23868.

    Article  PubMed  Google Scholar 

  46. Bijani M, Rostami K, Momennasab M, Yektatalab S. Evaluating the effectiveness of a continuing education program for prevention of occupational exposure to needle stick injuries in nursing staff based on Kirkpatrick’s model. J Natl Med Assoc. 2018;110(5):459–63.

    PubMed  Google Scholar 

  47. Carlfjord S, Roback K, Nilsen P. Five years’ experience of an annual course on implementation science: an evaluation among course participants. Implement Sci. 2017;12(1):101.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Young T, Rohwer A, Volmink J, Clarke M. What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLoS ONE. 2014;9(1): e86706.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Young T, Dizon J, Kredo T, et al. Enhancing capacity for clinical practice guidelines in South Africa. Pan Afr Med J. 2020;36:18.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Green G, Thorogood N. Qualitative methods for health research. London: Sage; 2004.

    Google Scholar 

  51. Lofland J. Analyzing social settings. Belmont, CA: Wadsworth Publishing Company; 1971.

    Google Scholar 

  52. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101.

    Article  Google Scholar 

  53. Popay J, Roberts H, Sowden A. Guidance on the conduct of narrative synthesis in systematic reviews a product from the ESRC methods programme. I Health Research: Lancaster; 2006.

    Google Scholar 

  54. Pope C, Mays N, Popay J. How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers? Healthc Manage Forum. 2006;19(1):27–31.

    Article  PubMed  Google Scholar 

  55. Glenton C, Santesso N, Rosenbaum S, et al. Presenting the results of Cochrane systematic reviews to a consumer audience: a qualitative study. Med Dec Making. 2010;30(5):566–77.

    Article  Google Scholar 

  56. Rosenbaum SE, Glenton C, Cracknell J. User experiences of evidence-based online resources for health professionals: user testing of The Cochrane library. BMC Med Inform Decis Mak. 2008;8:34.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607–19.

    Article  PubMed  Google Scholar 

  58. Rosenbaum SE, Glenton C, Wiysonge CS, et al. Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ. 2011;89(1):54–61.

    Article  PubMed  Google Scholar 

  59. Downe S, Finlayson KW, Lawrie TA, et al. Qualitative evidence synthesis (QES) for guidelines: Paper 1 - using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):76.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Lewin S, Glenton C, Lawrie TA, et al. Qualitative evidence synthesis (QES) for guidelines: Paper 2 - using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Res Policy Syst. 2019;17(1):75.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Glenton C, Lewin S, Lawrie TA et al. Qualitative Evidence Synthesis (QES) for Guidelines: Paper 3 – Using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Res Policy Sys. 2019;17:74. https://doi.org/10.1186/s12961-019-0450-1

    Article  PubMed  PubMed Central  Google Scholar 

  62. Morville P. 2004 User Experience Design [honeycomb model]. http://www.semanticstudios.com/publications/semantics/000029.php.

  63. Rosenbaum S. 2010 Improving the user experience of evidence: A design approach to evidence-informed health care. PhD thesis. Oslo, Norway: The Oslo School of Architecture and Design.

Download references

Acknowledgements

We gratefully acknowledge the representatives from the National Ministries of Health in Nigeria, Malawi and South Africa for their support and partnership. We would also like to thank the appointed Steering Committees who have been providing input for the research project and guiding the prioritization of topics. We would also like to thank Joy Oliver and Michelle Galloway for their contribution an support of the project.

Funding

The GELA project is funded by EDCTP2 programme supported by the European Union (grant number RIA2020S-3303-GELA). The funding will cover all the activities for this Monitoring and Evaluation work package, including costs for personnel and publication of papers.

Author information

Authors and Affiliations

Authors

Contributions

T.K., S.C., T.Y., S.L., C.G. and P.O.V. conceptualized the protocol idea and S.C. drafted the protocol with input from TK, D.M., A.R., B.M., M.M., I.I., C.G., T.Y., S.L. and P.O.V.; all authors approved the final version for submission for publication.

Corresponding author

Correspondence to Tamara Kredo.

Ethics declarations

Ethics approval and consent to participate

Ethics approval has been obtained in each partner country (South Africa, Malawi and Nigeria) from the respective Health Research Ethics Committees or Institutional Review Boards. Information about the project will be provided to, and consent obtained from, all participants completing the online surveys and interviews and all participants taking part in the meetings. The consent forms will make explicit the voluntary nature of participation, that there will be no negative consequences if they decide not to participate and in the case of the interviews and meetings observations will ask explicitly for permission for the interview or meeting to be recorded. The online surveys will ask participants to provide the last six numbers of their ID as a unique identifier to track their capacity development needs and progress throughout the project. To help protect their confidentiality, the information they provide will be private, deidentified and no names will be used. In addition, all cookies and IP address collectors will be disabled to ensure confidentiality. All interview and meeting recordings on the digital recorders will be destroyed following safe storage and transcription, and any identifying information will be redacted from all transcripts. All study data, including recordings, will be stored electronically using password-controlled software only accessible to key project members and project analysts. Reports of study findings will not identify individual participants. We do not anticipate any specific harms or serious risks to participants. However, there is a risk of breaches of confidentiality for participants who take part in guideline panel and steering group committee project meetings. At the start of all meetings, participants will be introduced to each other. The member names of these groups will not be anonymous as they will play an ongoing role in the GELA project. At the start of each meeting, we will discuss the importance of maintaining confidentiality by everyone. As part of guideline development processes, all guideline members will need to declare conflicts of interests and sign a confidentiality agreement. We will explain, however, that while the researchers undertake to maintain confidentiality, we cannot guarantee that other meeting participants will, and there is, thus, a risk of breaches of confidentiality. We will ensure participants are aware of this risk. Participants may also feel anxiety or distress expressing negative views about project activities. Where there is this potential and where participants identify concerns, we will reassure participants of the steps that will be taken to ensure confidentiality.

Consent for publication

Not applicable.

Competing interests

All authors declared no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kredo, T., Effa, E., Mbeye, N. et al. Evaluating the impact of the global evidence, local adaptation (GELA) project for enhancing evidence-informed guideline recommendations for newborn and young child health in three African countries: a mixed-methods protocol. Health Res Policy Sys 22, 114 (2024). https://doi.org/10.1186/s12961-024-01189-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-024-01189-5

Keywords