Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Can a quality improvement project impact maternal and child health outcomes at scale in northern Ghana?

  • Kavita Singh1, 2Email author,
  • Paul Brodish2,
  • Ilene Speizer1, 2,
  • Pierre Barker3,
  • Issac Amenga-Etego4,
  • Ireneous Dasoberi4,
  • Ernest Kanyoke4,
  • Eric A. Boadu4,
  • Elma Yabang4 and
  • Sodzi Sodzi-Tettey5
Health Research Policy and Systems201614:45

https://doi.org/10.1186/s12961-016-0115-2

Received: 15 December 2015

Accepted: 23 May 2016

Published: 16 June 2016

Abstract

Background

Quality improvement (QI) interventions are becoming more common in low- and middle-income countries, yet few studies have presented impact evaluations of these approaches. In this paper, we present an impact evaluation of a scale-up phase of ‘Project Fives Alive!’, a QI intervention in Ghana that aims to improve maternal and child health outcomes. ‘Project Fives Alive!’ employed a QI methodology to recognize barriers to care-seeking and care provision at the facility level and then to identify, test and implement simple and low-cost local solutions that address the barriers.

Methods

A quasi-experimental design, multivariable interrupted time series analysis, with data coming from 744 health facilities and controlling for potential confounding factors, was used to study the effect of the project. The key independent variables were the change categories (interventions implemented) and implementation phase – Wave 2a (early phase) versus Wave 2b (later phase). The outcomes studied were early antenatal care (ANC), skilled delivery, facility-level under-five mortality and attendance of underweight infants at child welfare clinics. We stratified the analysis by facility type, namely health posts, health centres and hospitals.

Results

Several of the specific change categories were significantly associated with improved outcomes. For example, three of five change categories (early ANC, four or more ANC visits and skilled delivery/immediate postnatal care (PNC)) for health posts and two of five change categories (health education and triage) for hospitals were associated with increased skilled delivery. These change categories were associated with increases in skilled delivery varying from 28% to 58%. PNC changes for health posts and health centres were associated with greater attendance of underweight infants at child welfare clinics. The triage change category was associated with increased early antenatal care in hospitals. Intensity, the number of change categories tested, was associated with increased skilled delivery in health centres and reduced under-five mortality in hospitals.

Conclusions

Using an innovative evaluation technique we determined that ‘Project Fives Alive!’ demonstrated impact at scale for the outcomes studied. The QI approach used by this project should be considered by other low- and middle-income countries in their efforts to improve maternal and child health.

Keywords

Child HealthGhanaImpact evaluationMaternal healthScale-upQuality improvement

Background

Quality improvement (QI) approaches are increasingly being used in low- and middle-income countries in efforts to improve service delivery and health outcomes. Much of the literature on QI approaches in these settings focuses on documentation of implementation and process evaluation [1,2]. A few studies have described scale-up processes for QI interventions in Ecuador [3,4], India [5] and South Africa [6,7]. All of these studies provide valuable information to guide countries and projects. However, documenting the impact of such approaches, both during pilot stages and at scale, is also important. This paper presents an impact evaluation of the scale-up phase of ‘Project Fives Alive!’, a national QI intervention in Ghana that aimed to improve maternal and child health outcomes. The project was implemented by the National Catholic Health Service and Institute for Healthcare Improvement in collaboration with the Ghana Health Service. The project design has been described previously in detail [8], and a prior evaluation study documented the impact of the pilot phase of the project [9].

The objective of ‘Project Fives Alive!’ was to assist and accelerate Ghana’s efforts to achieve Millennium Development Goals (MDGs) 4 (reducing under-five mortality) and 5 (reducing maternal mortality). Though Ghana did not meet the targets for MDG-4 and MDG-5, large improvements were made. In 2013, Ghana reported having a maternal mortality ratio of 380 maternal deaths per 100,000 live births, a large decline from the estimate of 760 maternal deaths per 100,000 live births in 1990 [10]. In 2015, the under-five mortality rate was estimated at 62 under-five deaths per 1000 live births compared to 122/1000 in 1990 [11].

‘Project Fives Alive!’ began in July 2008 with an innovation and testing phase, Wave 1, which included 27 health facilities in Northern Ghana. These facilities were purposively selected to reflect a mix of government facilities and faith-based facilities, which are affiliated with a religious institution. Wave 1 provided an opportunity for the implementation team to develop a package of locally identified and tested change ideas (interventions) focused on improving care seeking and care giving for mothers and children. Following Wave 1, the project rapidly introduced the locally developed interventions (changes) through a subsequent scale-up phase, Wave 2, to all government and faith-based facilities in Northern Ghana. Wave 2 included over 800 health facilities and covered the time period of September 2009 to March 2013.

‘Project Fives Alive!’ used the Model for Improvement and its underlying QI approach of identifying gaps in performance and the process failures that led to those gaps. The next step is identifying and testing simple low cost change ideas (or interventions) that can be employed to address those failures [12]. The process of generating, testing and sharing those ideas is accelerated through the Institute for Healthcare Improvement’s Collaborative Breakthrough Series Model that brought multiple sub-district or facility QI teams together repeatedly to share knowledge for improving performance. These teams formed Improvement Collaborative Networks at the district level [13] within specific geographic locations. During Wave 1, QI teams were formed at the level of the facility, while in Wave 2 teams were formed at the sub-district level such that all health centres and health posts within a sub-district contributed team members to form one team. Due to the higher volume and higher acuity of patients in hospitals settings, each hospital in Wave 2 formed its own team. Both Waves used the same basic Breakthrough Series approach; the QI teams attended four Learning Sessions (structured workshops led by project staff) where they learned QI methods and had a chance to share progress and ideas with other QI teams. During the 4–6 months between the Learning Sessions, both Waves 1 and 2 included Activity Periods when QI teams conducted Plan-Do-Study-Act cycles, the primary mechanism for testing and implementing changes. These cycles involved small tests of changes followed by rapid evaluations and adaptions.

Though the basic approach was the same, there were some key differences in implementation between Waves 1 and 2. Due to the small scale of Wave 1 (n = 27) and focus on innovation and development of change packages, the project team spent considerable time coaching each QI team during the Activity Periods. However, due to the large numbers of facilities in Wave 2 and only a small increase in project staff, it was not feasible to continue with this programmatically intense strategy. Thus, district health staff undertook intensive training and performed more of the coaching activities under the close supervision of project staff.

The main objective of this paper is to determine whether ‘Project Fives Alive!’ influenced maternal and child health outcomes at scale. A secondary objective is to present a methodology of using facility-based routine health data for a large-scale impact evaluation.

Methods

We employed a quasi-experimental design with a multivariable interrupted time series analysis controlling for potential confounding factors, to understand the impact of the intervention. Outcome data for this analysis are derived from data measured and reported by the facilities, while independent variables come from facility and program records. A specific programmatic decision was made to use routinely reported data rather than institute a parallel project data collection system, since the intervention was designed to be sustainable and scalable. To support this decision, a major effort was undertaken to improve the timeliness, completeness and accuracy of the data being submitted to and reported by the Ghana District Health Information Management System (DHIMS), a system whereby facilities complete monthly reports of key indicators and these are compiled at the district and national levels.

A total of 744 facilities were included in our analysis. Some newer facilities could not be included because of a lack of pre-intervention data. Since the intervention has the potential to differentially impact health outcomes by facility type and also due to different degrees of missing data, the analyses are stratified by facility type: health posts (or first level facility), health centres, and hospitals. Ethics review approval was obtained by the Ghana Health Service and the University of North Carolina at Chapel Hill.

Outcome Data

This evaluation used data from January 2009 to March 2013. In the initial phase of Wave 2, facilities used paper-based forms to report on key outcome indicators in a system called DHIMS 1. These forms were compiled and entered at the district level and then electronically sent onwards to the national level. In January 2012, Ghana shifted to a complete electronic system, DHIMS 2, whereby facilities entered the data and submitted the forms directly to the national level.

The four outcome variables in this assessment were chosen based on relevance to the project, and three were also included in the Wave 1 impact evaluation. Each outcome variable studied and the exact metric we used to define the variable are described in Table 1. The maternal health variables are early antenatal care (ANC) and skilled delivery coverage. We were able to study coverage for skilled delivery because health facilities record births that occur both at home and in facilities. The child health outcomes included the percent of child welfare clinic (CWC) attendees who are underweight and facility-level under-five mortality (for hospitals only). Underweight is defined as low weight for age in comparison to WHO reference standards. Our definition encompasses both moderate (less than two standard deviations below the median of the reference) and severe underweight (less than three standard deviations below the median of the reference standard) [14]. We could not study neonatal and infant mortality because of data quality concerns stemming from changes in reporting from DHIMS 1 to DHIMS 2. Fewer facilities reported these mortality outcomes in the DHIMS 2.
Table 1

Outcome variables and their definition/metric

Outcome

Definition/Metric

Early antenatal care (ANC)

% of ANC registrants in the first trimester at the time of registration

Numerator: number of ANC registrants in first trimester at registration

Denominator: number of ANC registrants

Skilled delivery coverage

% of total deliveries that are attended by skilled personnel

Numerator: number of total deliveries that are attended by skilled health personnel

Denominator: number of total deliveries (both skilled and unskilled) from facility and non-facility settings

Underweight infants at child welfare clinics (CWC)

% of 1- to 11-month-old CWC attendees < 60% weight for age

Numerator: number of 1- to 11-month-old CWC attendees who are moderately or severely underweight

Denominator: total number of 1- to 11-month-old CWC attendees with weight checked

Under-five mortality

Facility-level mortality among children less than 5 years of age

Numerator: number of deaths among children aged 0–59 months in hospitals

Denominator: total number of hospital admissions of children aged 0–59 months

Key independent variables

One of the two key independent variables was the interventions or change ideas implemented at a facility. For health centres and health posts, the change ideas were grouped into five categories – early pregnancy identification, four or more ANC visits, skilled delivery/immediate postnatal care (PNC), PNC on day 1 or 2, and PNC on day 6 or 7. Examples of these change interventions included community stakeholder meetings and registration of pregnant women by community volunteers for the early ANC; ANC defaulter tracing and visit time reduction for the four or more ANC visits; use of partographs and immediate checks of mother and newborn for the skilled delivery/immediate PNC; and home visits for both PNC interventions.

Hospitals had a separate set of change categories more suited to their patient loads and the presence of higher level staff. Hospital change categories were health education, targeting/engaging primary providers, training, triage and task shifting/nurse empowerment. The hospital changes were more broadly targeted than the health centre and health post changes since hospitals cover all types of services. Hospital changes were expected to improve maternal and child health by shortening visit times, prioritizing sick mothers and children, and improving communication between providers and pregnant women and mothers.

The other key independent variable was the implementation phase in which these improvement activities occurred – Wave 2a or Wave 2b. The earlier phase, Wave 2a, included the majority of facilities, whereas Wave 2b included the later set of facilities to engage in implementation. We study this variable to understand whether all facilities benefit equally or whether facilities that initiate implementation earlier, benefit more.

Control variables

The facility-level control variables included in this analysis were the type of health facility (hospital, health centre or health post) and affiliation of the health facility (government or faith based). A dummy variable was also included to represent the project officer assigned to work with a particular QI team. We also included as control variables profession of the QI team leader and number of QI team members. Since health insurance, particularly Ghana’s National Health Insurance Scheme, may be a potential confounding factor, a monthly time varying health insurance control variable (which was measured as the percent of outpatients who had insurance) was included.

Analysis

Descriptive analysis

In our descriptive analysis, we present a comparison of the pre-intervention, transition phase and post-intervention means of the outcome variables. The pre-intervention phase was defined by the project implementation team as the period of time before Learning Session 1, when QI teams were still learning the methodology of testing changes. The transition phase was the time period from Learning Session 1 to the end of Learning Session 2, which was considered a time when teams had just completed the training needed to fully implement the QI approach. The post-intervention or full saturation phase began at the very end of Activity Period 2 and was considered the cut-off point when the QI teams were expected to have the skills and knowledge to fully implement change ideas.

Missing outcome data

The unit of observation for the outcome data was facility-months. Each facility had several months of data. The number of facility months varied by outcome because not all facilities reported on all outcomes, and some facilities were new and were not in existence during the early time points or did not report on a particular outcome at exactly time 1. In addition, missing data, which was defined as an outcome not reported in a particular month once a facility has initiated reporting, was also responsible for some of the differences. Furthermore, with Ghana’s change in facility-level reporting of outcomes in January 2012, some facilities no longer provided denominators needed for the skilled delivery outcome. For these facilities, we ended their observation interval in December 2011 for skilled delivery to avoid considering the data points from January 2012 onwards as missing. Figure 1 presents the amount of missing data both by facility and outcome. Hospitals and health centres had relatively low amounts of missing data for the maternal health outcomes, whereas health posts had slightly more missing for these outcomes. Missing data for attendance of underweight infants at CWCs varied from 31% for health centres to 41% for hospitals. Hospitals had 43% missing data for under-five mortality.
Fig. 1

Percent of facilities with missing and non-missing data for health outcomes

Multivariable time series analysis

To study the impact of ‘Project Fives Alive!’ on the outcomes, we employed a multivariable interrupted time series regression analysis. This type of analysis answers the question of whether an intervention is associated with a change in the underlying trend for the outcome of interest after controlling for key variables [15]. The methodology of using repeated or monthly observations from the same facilities both pre- and post-intervention offers a strong evaluation design [16]. Data came from the period of January 2009 to March 2013, and the first set of facilities did not reach full implementation until July 2010. It was thus possible to establish underlying trends using the pre-intervention and transition phase data. In this analysis each facility served as its own control with the pre-intervention trend compared to the post-intervention trend.

In our model, there are two key parameters that were of interest – the immediate impact of the change category and the longer term impact or change category trend. Adding the coefficients from these two parameters yields the overall effect of the change category. Our model also included a quadratic term to account for a potential non-linear trend. A detailed description of the regression model and equation are presented by Singh et al. [9]. For each outcome variable, several multivariable regression models were run with relevant change categories included in separate models. Not all change categories were expected to have an effect on all four outcomes. For example, the PNC changes would not be expected to have an effect on skilled delivery, and thus regressions with these change categories are not presented for skilled delivery. A separate set of regressions to study the effect of program intensity, defined as the monthly number of change categories tested, was also run. These models also controlled for the independent variables presented earlier.

Due to the amount of missing data and the presence of both serial autocorrelation and clustering, we used generalized estimating equations (GEE) to run the regression analyses. GEE uses all data that is available and assumes data is missing completely at random, which is a plausible assumption for these monthly facility-level data. Autocorrelation and clustering violate the ordinary least squares assumption of uncorrelated error terms, biasing the standard errors when using standard linear regression. GEE is an extension of the quasi-likelihood approach used in generalized linear models and is often applied to modelling longitudinal data [1719].

In addition, a sensitivity analysis was conducted as a check against our results from the main GEE analysis. In these analyses, single imputation was used to either (1) impute all missing values or (2) impute missing values only for facilities with less than 25% of their observations missing. The imputation was conducted by taking the average value of the nearest non-missing preceding and succeeding values. Because the results of our sensitivity analyses generally corroborated results from the main model, only results for the main model are presented.

The number of observations for each regression model varies slightly by facility type due to the varying amounts of outcome data available. In addition, not all independent variables were available for each facility. Comparisons of mean outcomes for facilities that have all control variables and those that do not were made and found not to be significantly different.

Results

Descriptive presentation of the independent variables

All independent variables are presented in Tables 2 and 3. The majority of facilities were health centres (45.5%) and health posts (50.4%), while only 4% were hospitals. Ninety-two percent of the facilities were government affiliated and 8% were faith based. A total of seven project officers were part of the Wave 2 program team, and the average percent of insured patients at a facility was 78%. The PNC change intervention activities were the most common changes implemented in health centres and health posts, while triage was the most common change category in hospitals. Eighty-seven percent of facilities were part of Wave 2a, and 13% were part of Wave 2b.
Table 2

Descriptive statistics for control variables and independent variable phase

 

n

% or mean

Control variables

  

 Type of facility

  

  Hospital

30

4.1

  Health centre

334

45.5

  Health post

370

50.4

 Affiliation of facility

  

  Government

676

92.1

  Faith-based

58

7.9

 Number of facilities for each Project Officer

  

  Project Officer 1

69

10.6

  Project Officer 2

110

16.9

  Project Officer 3

80

12.3

  Project Officer 4

78

12.0

  Project Officer 5

63

9.7

  Project Officer 6

125

19.2

  Project Officer 7

125

19.2

 Profession of QI Team Leader

  

  Midwife

267

51.3

  Nurse

119

22.9

  Other

134

25.8

 Mean number of QI Team Members

NA

9.9

 Outpatients with health insurance

NA

78.1

Independent variable

  

 Phase

  

  Wave 2a

638

87.1

  Wave 2b

96

12.9

NA not applicable, QI quality improvement

Table 3

Descriptive statistics on change category variables, by facility type, n (%)

Change category

Hospitals

Health centres

Health posts

Health education

16 (53.3)

  

Targeting primary providers

2 (6.7)

  

Triage

25 (83.3)

  

Training

7 (23.3)

  

Task shifting/Nurse empowerment

12 (40.0)

  

Early pregnancy identification

 

186 (55.7)

160 (43.2)

Four ANC visits

 

153 (45.8)

154 (41.6)

Skilled delivery/immediate PNC

 

211 (63.2)

210 (56.8)

PNC day 1 or 2

 

238 (71.3)

224 (60.5)

PNC day 6 or 7

 

235 (70.4)

223 (60.3)

Total

30 (100)

335 (100)

370 (100)

ANC antenatal care, PNC postnatal care

Descriptive analysis

The comparison of means for the pre-intervention, transition and post-intervention phases is presented in Table 4. Overall, there are improvements in the maternal health outcomes over time. At the aggregate level, there is an increase in early ANC from 37% to 42% to 48% from the pre-intervention phase to the transition phase to the post-intervention phase, respectively. Overall, skilled delivery is at 42% in the pre-intervention phase, increases to 47% in the transition phase and then further increases to 51% in the post-intervention phase. In terms of the child health outcomes, there was an overall increase in the percent of underweight infants attending CWCs, from 2% in the pre-intervention phase to 8% in the post-intervention phase. Under-five mortality decreases from 36/1000 in the pre-intervention phase to 29/1000 in the post-intervention phase.
Table 4

Means of the outcome variables by implementation period and facility type

 

Pre-intervention

Transition phase

Post-intervention

Overall

Outcomes

Facility-Months

Mean

Range

Facility-Months

Mean

Range

Facility-Months

Mean

Range

Facility-Months

Mean

Range

Early antenatal care (%)

8832

37

0–100

2077

42

0–100

11,671

48

0–100

22,580

43

0–100

 Hospitals

580

40

2–100

117

40

3–94

496

44

3–100

1193

41

2–100

 Health centres

5227

36

0–100

1150

41

0–100

6080

45

0–100

12,457

41

0–100

 Health posts

3025

38

0–100

810

43

0–100

5095

51

0–100

8930

46

1–100

Skilled delivery (%)

8156

42

0–100

1923

47

0–100

9573

51

0–100

19,652

47

0–100

 Hospitals

562

86

0–100

119

89

0–100

416

90

0–100

1097

88

0–100

 Health centres

4930

49

0–100

1109

55

0–100

5296

65

0–100

11,335

57

0–100

 Health posts

2664

19

0–100

695

27

0–100

3861

28

0–100

7220

25

0–100

Underweight in infants (%)

7482

2

0–100

1942

2

0–67

7685

8

0–100

17,109

5

0–100

 Hospitals

213

2

0–27

50

2

0–18

182

6

0–51

445

4

0–51

 Health centres

4368

2

0–100

1030

2

0–52

3968

7

0–70

9366

4

0–100

 Health posts

2901

2

0–63

862

2

0–67

3535

9

0–100

7298

5

0–100

Under-five mortality rate (Hospitals)

269

36 per 1000

0–100 per 1000

49

25 per 1000

0–23 per 1000

253

29 per 1000

0–39 per 1000

571

32 per 1000

0–100 per 1000

Time series analysis

Health posts

Three of the categories of change interventions – early ANC (β = 0.3540, P < 0.01), four or more ANC visits (β = 0.2882, P < 0.05) and skilled delivery/immediate PNC (β = 0.2822, P < 0.01) – were significantly and positively associated with the skilled delivery outcome (Table 5). Facilities that tested these changes saw 28–35% higher rates of skilled delivery than facilities that did not test such changes. The corresponding trend variables had small positive associations with the skilled delivery outcome, suggesting that the initial positive effect continued over time, but this trend effect was significant only for the early ANC and skilled delivery/immediate PNC changes.
Table 5

Results of generalized estimating equation regressions of health outcomes on change categories at health posts

 

Early ANC

Skilled delivery

Underweight infants

β (95% CI)

β (95% CI)

β (95% CI)

Wave 2b vs. 2a

−0.0035 (−0.0580 to 0.0510)

0.0080 (−0.0443 to 0.0602)

−0.0163 (−0.1294 to 0.0968)

−0.0281 (−0.1381 to 0.0819)

−0.0268 (−0.1359 to 0.0823)

−0.0049 (−0.0271 to 0.0173)

−0.0044 (−0.0267 to 0.0179)

Early ANC change

0.1316 (−0.0320 to 0.2951)

 

0.3540 (0.1097 to 0.5983)**

    

Early ANC trend

0.0001 (−0.0000 to 0.0003)

 

0.0003 (0.0000 to 0.0006)*

    

Four or more ANC visits change

 

0.0836 (−0.0769 to 0.2441)

 

0.2882 (0.0470 to 0.5293)*

   

Four or more ANC visits trend

 

0.0002 (−0.0000 to 0.0003)

 

0.0002 (−0.0000 to 0.0005)

   

Skilled delivery change

    

0.2822 (0.0671 to 0.4973)*

  

Skilled delivery trend

    

0.0003 (0.0000 to 0.0005)*

  

PNC on day 1 or 2 change

     

0.0667 (0.0209 to 0.1124)**

 

PNC on day 1 or 2 trend

     

0.0000 (−0.0000 to 0.0001)

 

PNC on day 6 or 7 change

      

0.0578 (0.0141 to 0.1015)**

PNC on day 6 or 7 trend

      

0.0000 (−0.0000 to 0.0001)

Constant

0.2305 (0.1765 to 0.2845)***

0.2233 (0.1693 to 0.2774)***

0.1465 (0.0517 to 0.2413)**

0.1458 (0.0513 to 0.2403)**

0.1348 (0.0394 to 0.2301)**

0.0400 (0.0203 to 0.0597)***

0.0387 (0.0188 to 0.0585)***

n

4902

4902

3923

3923

3923

4154

4154

All models control for project officer, government vs. catholic facility, insurance status, profession of QI team leader, and number of QI team members

ANC antenatal care, PNC postnatal care

* P < 0.05; ** P < 0.01; *** P < 0.001

In terms of findings for child health, both PNC change categories were significantly associated with a greater percentage of underweight infants among CWC attendees. The β2 coefficient was 0.0667 at P < 0.01 for the PNC day 1 or 2 change category, and the β2 coefficient was 0.0578 at P < 0.01 for the PNC day 6 or 7 change category. The corresponding trend variables were not significant.

There was one significant finding for the measure of intensity in the health post analysis, namely the monthly number of change categories tested (Table 6). Intensity was significantly associated with a lower percentage of underweight infants among all CWC attendees (β = –0.0092; P < 0.001).
Table 6

Results of generalized estimating equation regressions of health outcomes on intensity for health posts and health centres

 

Early ANC

Skilled delivery

Underweight infants

Under-five mortality

β (95% CI)

β (95% CI)

β (95% CI)

β (95% CI)

Health posts

    

 Wave 2b vs. 2a

0.0038 (−0.0486 to 0.0563)

−0.0150 (−0.1248 to 0.0948)

−0.0185 (−0.0443 to 0.0074)

NA

 Intensity

−0.0048 (−0.0106 to 0.0010)

0.0066 (−0.0035 to 0.0167)

−0.0092*** (−0.0115 to −0.0068)

NA

 Constant

0.2356*** (0.1892 to 0.2821)

0.1942*** (0.1103 to 0.2780)

−0.0320** (−0.0515 to −0.0124)

NA

 N

4902

3923

4154

NA

Health centres

    

 Wave 2b vs. 2a

−0.0047 (−0.0522 to 0.0428)

−0.0047 (−0.1196 to 0.1102)

−0.0244* (−0.0435 to −0.0053)

 

 Intensity

0.0017 (−0.0026 to 0.0061)

0.0089* (0.0008 to 0.0169)

−0.0088*** (−0.0105 to −0.0071)

 

 Constant

0.2671*** (0.2317 to 0.3025)

0.3055*** (0.2298 to 0.3812)

–0.0032 (–0.0175 to 0.0110)

 

 N

6888

6259

5272

 

All models control for project officer, government vs. catholic facility, insurance status, profession of QI team leader, and number of QI team members

ANC antenatal care

* P < 0.05; ** P < 0.01; *** P < 0.001

Health centres

None of the specific change categories were associated with the early ANC and skilled delivery outcomes (Table 7). Both PNC change categories, however, were associated with an increased percent of underweight infants among attendees at CWCs (β = 0.0701, P < 0.001 and β = 0.0452, P < 0.001, respectively). The trend variables were significant and positive, indicating that the initial increased effect was maintained. Wave 2b was significantly and negatively associated with attendance of underweight infants at CWCs in the models with the PNC change categories.
Table 7

Results of generalized estimating equation regressions of health outcomes on change categories at health centres

 

Early ANC

Skilled delivery

Underweight Infants

β (95% CI)

β (95% CI)

β (95% CI)

Wave 2b vs. 2a

0.0016 (–0.0466 to 0.0497)

–0.0068 (–0.0542 to 0.0407)

0.0079 (–0.1083 to 0.1241)

–0.0180 (–0.1334 to 0.0975)

–0.0112 (–0.1255 to 0.1032)

–0.0195 (–0.0364 to –0.0026)*

0.0196 (–0.0366 to –0.0026)*

Early ANC change

–0.0519 (–0.1456 to 0.0418)

 

0.0045 (–0.1593 to 0.1683)

    

Early ANC trend

–0.0001 (–0.0002 to 0.0000)

 

0.0000 (–0.0002 to 0.0002)

    

Four or more ANC visits change

 

–0.0676 (–0.1468 to 0.0116)

 

0.0136 (–0.1318 to 0.1590)

   

Four or more ANC visits trend

 

–0.0001 (–0.0002 to 0.0000)

 

0.0000 (–0.0002 to 0.0002)

   

Skilled delivery change

    

–0.0317 (–0.1685 to 0.1052)

  

Skilled delivery trend

    

0.0000 (–0.0002 to 0.0002)

  

PNC on day 1 or 2 change

     

0.0701 (0.0436 to 0.0967)***

 

PNC on day 1 or 2 trend

     

0.0001 (0.0000 to 0.0001)***

 

PNC on day 6 or 7 change

      

0.0452 (0.0216 to 0.0688)***

PNC on day 6 or 7 trend

      

0.0000 (0.0000 to 0.0001)**

Constant

0.2526 (0.2133 to 0.2919)***

0.2538 (0.2142 to 0.2935)***

0.2753 (0.1928 to 0.3579)***

0.2688 (0.1834 to 0.3501)***

0.2724 (0.1885 to 0.3563)***

0.0425 (0.0281 to 0.0568)***

0.0425 (0.0278 to 0.0571)***

N

6888

6888

6259

6259

6259

5272

5272

All models control for project officer, government vs. catholic facility, insurance status, profession of QI team leader, and number of QI team members

ANC antenatal care, PNC postnatal care

* P < 0.05; ** P < 0.01; *** P < 0.001

There were two significant associations between the measure of intensity and the outcome variables (Table 6). A greater number of change categories tested was significantly associated with increased skilled delivery (β = 0.0089, P < 0.05) and a smaller percentage of underweight infants among child wellness attendees (β = –0.0088, P < 0.001). There was a significant association in the intensity models between Wave 2b and decreased underweight infants at CWCs.

Hospitals

There were several significant associations between the change categories in the regressions for hospitals (Tables 8 and 9). Facilities that tested a health education change had 58% higher rates of skilled delivery compared to facilities not testing this change category (β = 0.5753, P < 0.05). The trend variable was slightly positive and significant, indicating that the increase in skilled delivery was continued although at a lower level than the initial increase. The triage change category was associated with a 42% increase in early ANC (β = 0.4236, P < 0.05) and a 50% increase in skilled delivery (β = 0.4989, P < 0.05), and the trend variable for the latter indicated a slight but significant increase over time (β = 0.0004, P < 0.05). Across all outcomes, there were no significant associations with the implementation phase variable in the hospital settings.
Table 8

Results of generalized estimating equation regressions of maternal health outcomes on change categories at hospitals

 

Early ANC

Skilled delivery

β (95% CI)

β (95% CI)

Wave 2b vs. 2a

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

Health education change

0.3535 (–0.0397 to 0.7468)

    

0.5753 (0.0743 to 1.0763)*

    

Health education trend

0.0002 (–0.0001 to 0.0005)

    

0.0005 (0.0001 to 0.0008)*

    

Targeting primary providers change

 

0.0000 (0.0000 to 0.0000)

    

–3.3742 (–26.4808 to 19.7323)

   

Targeting primary providers trend

 

0.0000 (0.0000 to 0.0000)

    

–0.0032 (–0.0225 to 0.0161)

   

Triage change

  

0.4236 (0.0440 to 0.8031)*

    

0.4989 (0.0159 to 0.9819)*

  

Triage trend

  

0.0002 (–0.0001 to 0.0005)

    

0.0004 (0.0001 to 0.0008)*

  

Training change

   

0.5092 (–0.0439 to 1.0624)

    

–0.0142 (–0.6329 to 0.6045)

 

Training trend

   

0.0004 (–0.0001 to 0.0008)

    

0.0000 (–0.0004 to 0.0005)

 

Task shifting/nurse empowerment change

    

0.2949 (–0.1446 to 0.7345)

    

0.2800 (–0.2507 to 0.8107)

Task shifting/nurse empowerment trend

    

0.0001 (–0.0003 to 0.0004)

    

0.0003 (–0.0001 to 0.0007)

Constant

0.1668 (0.0480 to 0.2857)**

0.1649 (0.0467 to 0.2832)**

0.1650 (0.0477 to 0.2822)**

0.1717 (0.0538 to 0.2896)**

0.1667 (0.0509 to 0.2824)**

1.0583 (0.9345 to 1.1821)***

1.0816 (0.9583 to 1.2048)***

1.0669 (0.9441 to 1.1897)***

1.0744 (0.9514 to 1.1975)***

1.1076 (0.9837 to 1.2316)***

n

570

570

570

570

570

574

574

574

574

574

All models control for project officer, government vs. catholic facility, insurance status, profession of QI team leader, and number of QI team members

ANC antenatal care

* P < 0.05; ** P < 0.01; *** P < 0.001

Table 9

Results of generalized estimating equation regressions of child health outcomes on change categories at hospitals

 

Underweight infants

Under-five mortality

β (95% CI)

β (95% CI)

Wave 2b vs. 2a

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

Health education change

0.0241 (–0.2843 to 0.3324)

    

0.0528 (–0.1509 to 0.2565)

    

Health education trend

0.0001 (–0.0002 to 0.0003)

    

0.0001 (–0.0001 to 0.0002)

    

Targeting primary providers change

 

0.0000 (0.0000 to 0.0000)

    

0.8415 (–0.3567 to 2.0396)

   

Targeting primary providers trend

 

0.0000 (0.0000 to 0.0000)

    

0.0006 (–0.0002 to 0.0013)

   

Triage change

  

0.0241 (–0.2843 to 0.3324)

    

0.0798 (–0.1135 to 0.2731)

  

Triage trend

  

0.0001 (–0.0002 to 0.0003)

    

0.0000 (–0.0001 to 0.0002)

  

Training change

   

0.2223 (–0.1437 to 0.5883)

    

0.1525 (–0.3240 to 0.6289)

 

Training trend

   

0.0002 (–0.0001 to 0.0005)

    

0.0001 (–0.0002 to 0.0005)

 

Task shifting/nurse empowerment change

    

0.2558 (–0.1129 to 0.6246)

    

–0.0539 (–0.2073 to 0.3151)

Task shifting/nurse empowerment trend

    

0.0002 (–0.0001 to 0.0005)

    

0.0000 (–0.0002 to 0.0002)

Constant

0.0412 (–0.1164 to 0.1987)**

0.0406 (–0.1177 to 0.1989)

0.0412 (–0.1164 to 0.1987)

0.0099 (–0.1492 to 0.1691)

0.0220 (–0.1367 to 0.1808)

0.0687 (0.0232 to 0.1142)**

0.0687 (0.0296 to 0.1079)***

0.0542 (0.0092 to 0.0093)*

0.0761 (0.0308 to 0.1215)***

0.0678 (0.0197 to 0.1159)**

n

226

226

226

226

226

284

284

284

284

284

All models control for project officer, government vs. catholic facility, insurance status, profession of QI team leader, and number of QI team members

* P < 0.05; ** P < 0.01; *** P < 0.001

Greater intensity was significantly associated with two of the outcomes in hospitals (Table 10). Intensity was associated with a 0.9% decrease in underweight infants attending CWCs (β = –0.0093, P < 0.01) and a 0.4% decrease in under five-mortality (β = –0.0038, P < 0.05). Once again there were no significant associations for the implementation phase variable.
Table 10

Results of generalized estimating equation regression of health outcomes on intensity for hospitals

 

Early ANC

Skilled delivery

Underweight infants

Under-five mortality

 

β (95% CI)

β (95% CI)

β (95% CI)

β (95% CI)

Wave2b vs. 2a

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

0.0000 (0.0000 to 0.0000)

Intensity

0.0073 (–0.0044 to 0.0189)

–0.0005 (–0.0137 to 0.0127)

–0.0093** (–0.0201 to 0.0014)

–0.0038* (–0.0090 to 0.0013)

Constant

0.1773** (0.0601 to 0.2945)

1.0985*** (0.9772 to 1.2197)

–0.0815 (–0.2571 to 0.0942)

0.0527* (0.0052 to 0.1002)

N

570

574

226

284

ANC antenatal care

Discussion

As more low- and middle-income countries implement QI projects to improve health outcomes, there is a need to evaluate the approaches both during pilot and scale-up phases. Evaluations of pilot phases can help demonstrate the evidence needed to justify scale-up and/or can provide valuable information to inform implementation modifications for the scale-up phase [20]. Due to the magnitude of scale-up phases and the difficulty of finding control or comparison groups, innovative evaluation approaches are needed that can take advantage of existing monitoring data [21,22]. In this paper, we present an innovative evaluation of the scale-up phase of ‘Project Fives Alive!’ using data from Ghana’s routine health information system supplemented by facility characteristics and program records.

Findings from the evaluation indicated a positive effect of ‘Project Fives Alive!’ on key maternal and child health outcomes. There was evidence of some sustained program effect on underweight infants attending CWCs and skilled delivery as was seen in Wave 1 [9]. All the maternal health-focused change categories were associated with an increase in skilled delivery for health posts, and the health education and triage change categories were associated with the early ANC and skilled delivery outcomes for hospitals. Greater intensity was associated with increased skilled delivery for health centres.

There were positive effects of the PNC change categories in getting more underweight infants into care in health posts and health centres; however, greater intensity was also negatively associated with the percentage of underweight infants at CWCs across the facility types. These differing findings need a nuanced explanation. It could be that the PNC change categories initially increased care-seeking of caregivers of underweight infants. Over time, as these facilities implemented more changes and more fully incorporated the QI approach into their daily work, there could have been overall improvements in the health and nutrition of children in the catchment area, leading to a lower percentage of children in facilities who were underweight. These findings on underweight children are important given that under-nutrition is estimated to be an underlying factor in 45% of under-five deaths [23].

In Wave 1, there were no significant associations between the change categories or intensity with mortality. Perhaps due to the longer time period of Wave 2 compared to Wave 1 (51 months versus 21 months) we see evidence of impact on mortality for Wave 2. In hospitals, greater intensity was associated with slightly decreased under-five mortality. As health providers engaged more fully in the QI approach over time, they may have been able to improve the quality of services provided and/or increase early care-seeking such that mortality declined.

There were few significant differences by phase of implementation, indicating that all facilities benefited from the intervention. In ‘Project Fives Alive!’ the first set of Wave 2a reached full saturation in March 2010 compared to March 2011 for the first set of Wave 2b facilities. This is an important finding given that scale-up strategies of many projects need to have a phased approach to attain broad reach.

There are several limitations to this analysis, including our inability to study population-level mortality. We only had data on facility deaths and not deaths that occurred in communities. In Ghana, as in many low- and middle-income countries, many under-five deaths occur at home or in non-facility environments. An additional data challenge is that we could only study skilled delivery until December 2011 for a large number of facilities because of the change in reporting. Finally, it is difficult to find comparison groups for the evaluation of a scale-up phase of a project, and our analysis lacked such groups. We were able to use each facility as its own control in an interrupted time series analysis with additional control for potential confounding factors, including program and facility characteristics. We also controlled for National Health Insurance Scheme registration, which also has a strong focus on maternal and child health. The use of repeated monthly outcome data from each facility, both pre- and post-intervention, offers a strong evaluation design [16]. We cannot, however, completely rule out the possibility that other ongoing maternal and child health initiatives could have also influenced the results.

Conclusion

Findings from the scale-up phase of ‘Project Fives Alive!’ indicate program effects on the key maternal and child health outcomes studied, including reduced under-five mortality. The QI approach of identifying barriers to care and care-seeking with local, simple and inexpensive solutions has demonstrated impact at scale and should be considered a feasible approach for improving maternal and child health outcomes in other low- and middle-income settings. We also demonstrate the feasibility of using existing outcome data in a multivariable time series analysis to evaluate the scale-up phase of an intervention.

Abbreviations

ANC, Antenatal care; DHIMS, District Health Information Management System; GEE, Generalized estimating equations; MDG, Millennium Development Goal; PNC, Postnatal care; QI, Quality improvement

Declarations

Acknowledgements

This study was funded by the Bill & Melinda Gates Foundation. The donor was not involved in this paper. We would like to thank Sudhanshu Handa and Chirayath Suchindran for statistical advice, and Rick O’Hara for programming advice. We also acknowledge the Carolina Population Center and its NIH Center grant (P2C HD050924) for general support.

Authors’ contributions

KS led the overall writing and designed the analysis. PB led the data management and analysis. IAE, ID, EK, EAB and EY led the data collection and assisted with data management and interpretation of findings. IS, PB and SST provided extensive feedback and suggestions on all earlier drafts of the papers and provided guidance on the analysis and interpretation of findings. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Department of Maternal and Child Health, Gillings School of Global Public Health, University of North Carolina at Chapel Hill
(2)
Carolina Population Center, University of North Carolina at Chapel Hill
(3)
The Institute for Healthcare Improvement (IHI)
(4)
The National Catholic Health Service
(5)
The Institute for Healthcare Improvement (IHI)

References

  1. Leatherman S, Ferris TG, Berwick D, Omaswa F, Crisp N. The role of quality improvement in strengthening health systems in developing countries. In J Qual Health Care. 2010;22:237–43.View ArticleGoogle Scholar
  2. Heiby J. The use of modern quality improvement approaches to strengthen African health systems: a 5-year agenda. Int J Qual Health Care. 2014;26:117–23.View ArticlePubMedGoogle Scholar
  3. Hermida J, Robalino ME, Vaca L, Ayabaca P, Romero P, Vieira L. Scaling up and institutionalizing continuous quality improvement in the free maternity and child care program in Ecuador, LACHSR Report Number 65. Published for the U.S. Agency for International Development (USAID) by the Quality Assurance Project. http://www.urc-chs.com/sites/default/files/ScalingupandInstitutionalizingCQIinEcuadorLACHSRNo65.pdf.
  4. Hermida J, Salas B, Sloan NL. Sustainable scale-up of active management of the third stage of labor for prevention of postpartum hemorrhage in Ecuador. Int J Gyn Obs. 2012;117:278–82.View ArticleGoogle Scholar
  5. Anchalia MM, D’Ambruoso L. Seeking solutions: scaling-up audit as a quality improvement tool for infection control in Gujarat, India. Int J Qual Health Care. 2011;23:464–70.View ArticlePubMedGoogle Scholar
  6. Webster PD, Sibanyoni M, Malekutu D, Mate KS, Venter WD, Barker PM, Moleko M. Using quality improvement to accelerate highly active antiretroviral treatment coverage in South Africa. BMJ Qual Saf. 2012;21:315–24.Google Scholar
  7. Mate KS, Ngubane G, Barker PM. A quality improvement model for the rapid scale-up of a program to prevent mother-to-child HIV transmission in South Africa. Int J Qual Health Care. 2013;25:373–80.View ArticlePubMedGoogle Scholar
  8. Singh K, Speizer I, Handa S, Boadu R, Atinbire S, Barker P, Twum-Danso N. Impact evaluation of a quality improvement intervention on maternal and child health outcomes in Northern Ghana: early assessment of a national scale-up project. Int J Qual Health Care. 2013;25:477–87.Google Scholar
  9. World Health Organization. Trends in maternal mortality 1990–2013: estimates by WHO, UNICEF, UNFPA, The World Bank and the United Nations Population Division. Geneva: World Health Organization; 2014.Google Scholar
  10. United Nations Children’s Fund. Levels & trends in child mortality report 2015: estimates developed by the UN Inter-agency Group for Child Mortality Estimation. New York: United Nations Children’s Fund; 2015.Google Scholar
  11. Langley GJ. The improvement guide: a practical approach to enhancing organizational performance. 2nd ed. San Francisco: Jossey-Bass; 2009.Google Scholar
  12. Institute for Healthcare Improvement. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Performance. Cambridge: Institute for Healthcare Improvement; 2003.Google Scholar
  13. Twum-Danso NA, Akanlu GB, Osafo E, Sodzi-Tettey S, Boadu RO, Atinbire S, Adondiwo A, Amenga-Etego I, Ashagbley F, Boadu EA, Dasoberi I, Kanyoke E, Yabang E, Essegbey IT, Adjei GA, Buckle GB, Awoonor-Williams JK, Nang-Beifubah A, Twumasi A, McCannon CJ, Barker PM. A nationwide quality improvement project to accelerate Ghana’s progress towards Millennium Development Goal 4: design and implementation progress. Int J Qual Health Care. 2012;24:601–11.Google Scholar
  14. WHO Multicentre Growth Reference Study Group. WHO Child Growth Standards: Length/height-for-age, weight-for-age, weight-for-length, weight-for-height and body mass index-for-age: Methods and development. Geneva: World Health Organization; 2006.Google Scholar
  15. Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin; 1963.Google Scholar
  16. Biglan A, Ary D, Wagenaar AC. The value of interrupted time series experiments for community intervention research. Prev Sci. 2000;1:31–49.View ArticlePubMedPubMed CentralGoogle Scholar
  17. Hanley JA, Negassa A, Edwardes MD de B, Forrester JE. Statistical analysis of correlated data using generalized estimating equations: An orientation. Am J Epidemiol. 2003;157:364–75.View ArticlePubMedGoogle Scholar
  18. Sheu C-F. Regression analysis of correlated binary outcomes. Behav Res Methods Instrum Comput. 2000;32:269–73.View ArticlePubMedGoogle Scholar
  19. Ballinger GA. Using generalized estimating equations for longitudinal data analysis. Organ Res Meth. 2004;7:127–50.View ArticleGoogle Scholar
  20. Dufulo E. Scaling up and evaluation. In: Bourguignon F, Pleskovic B, editors. Accelerating Development. Washington, DC and Oxford, UK: World Bank and Oxford University Press; 2004. p. 342–67.Google Scholar
  21. Bryce J, Victora CG, Boerma T, Peters DH, Black RE. Evaluating the scale-up for maternal and child survival: a common framework. Int Health. 2011;3:139–46.View ArticlePubMedGoogle Scholar
  22. Victora CG, Black RE, Boerma JT, Byrce J. Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness and evaluation. Lancet. 2011;377:85–95.View ArticlePubMedGoogle Scholar
  23. Black RE, Victora CG, Walker SP, Bhutta ZA, Christian P, de Onis M, Ezzati M, Grantham-McGregor S, Katz J, Martorell R, Uauy R, the Maternal and Child Nutrition Study Group. Maternal and child undernutrition and overweight in low-income and middle-income countries. Lancet. 2013;382:427–51.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2016

Advertisement