Skip to main content

Repurposing NGO data for better research outcomes: a scoping review of the use and secondary analysis of NGO data in health policy and systems research

Abstract

Background

Non-governmental organisations (NGOs) collect and generate vast amounts of potentially rich data, most of which are not used for research purposes. Secondary analysis of NGO data (their use and analysis in a study for which they were not originally collected) presents an important but largely unrealised opportunity to provide new research insights in critical areas, including the evaluation of health policy and programmes.

Methods

A scoping review of the published literature was performed to identify the extent to which secondary analysis of NGO data has been used in health policy and systems research (HPSR). A tiered analytical approach provided a comprehensive overview and descriptive analyses of the studies that (1) used data produced or collected by or about NGOs; (2) performed secondary analysis of the NGO data (beyond the use of an NGO report as a supporting reference); and (3) analysed NGO-collected clinical data.

Results

Of the 156 studies that performed secondary analysis of NGO-produced or collected data, 64% (n = 100) used NGO-produced reports (mostly to a limited extent, as a contextual reference or to critique NGO activities) and 8% (n = 13) analysed NGO-collected clinical data. Of these studies, 55% (n = 86) investigated service delivery research topics and 48% (n = 51) were undertaken in developing countries and 17% (n = 27) in both developing and developed countries. NGOs were authors or co-authors of 26% of the studies. NGO-collected clinical data enabled HPSR within marginalised groups (e.g. migrants, people in conflict-affected areas), albeit with some limitations such as inconsistent and missing data.

Conclusion

We found evidence that NGO-collected and produced data are most commonly perceived as a source of supporting evidence for HPSR and not as primary source data. However, these data can facilitate research in under-researched marginalised groups and in contexts that are hard to reach by academics such as conflict-affected areas. NGO–academic collaboration could help address issues of NGO data quality to facilitate their more widespread use in research. The use of NGO data use could enable relevant and timely research in the areas of programme evaluation and health policy and advocacy to improve health and reduce health inequalities, especially in marginalised groups and developing countries.

Peer Review reports

Background

The lower estimate of the number of non-governmental organisations (NGOs; non-profit groups formed voluntarily) in the world is 1 million, but there may be as many as 10.3 million (based on the number of registrations of .org and .ngo domain names) [1, 2]. An estimated 35,000 are large, established NGOs and many operate in the health sector; in the year 2000, there were over 2000 international health NGOs and this number is expected to have increased since [3, 4]. These NGOs deliver, monitor and advocate for health services and equitable healthcare at the community, national and international levels [4, 5]. To a lesser extent, they are engaged in performing and disseminating research [1, 6, 7].

Health policy and systems research (HPSR) is a multidisciplinary field of research conducted to inform and influence policies and systems to improve health outcomes for all [8, 9]. Within the context of HPSR, NGOs necessarily produce data on the services and programmes they deliver, collect data on the (often marginalised and hard to reach by researchers) recipient populations and the health conditions being treated [10, 11]. These data can be patient records for populations who do not access national healthcare, administrative data on the number of drugs dispensed or days that patients spent in NGO-run healthcare facilities, data on community responses to health crises (e.g. in the aftermath of extreme weather events), or reviews of health policy [12]. Although NGOs are only rarely collecting data for the purposes of research, the records and data held by them are a vast (and largely untapped) source of potentially rich data [6].

NGO-collected data are especially valuable for research in developing countries, on populations under-served by the national health system, and where there may be a data gap due to inadequate national data collection and monitoring infrastructure [6, 13, 14]. The analysis of NGO data presents an opportunity for researchers to conduct relevant, timely and relatively cheap secondary research that has the potential to improve health outcomes [6, 15,16,17]. However, there is a dual problem of these data being ignored by researchers and not made available by NGOs for secondary analysis [6, 18, 19]. Although some examples of NGO–academic collaboration and open access repositories for NGO data exist [11, 20, 21], at other times, researchers will have to approach NGOs to request access to data about them or collected by them [22, 23].

HPSR is led largely by questions from ‘the field’ rather than being theory driven but conceptual frameworks are used to describe and analyse the health systems studied [24]. For example, the WHO framework of the six health system building blocks required to improve health and health equity uses the categories of (1) service delivery; (2) healthcare workforce; (3) information; (4) medical products, vaccines and technologies; (5) financing; and (6) leadership and governance [25]. The framework is widely used in HPSR, particularly in developing country contexts, as it helps locate, describe and classify health system constraints, where investment is needed, and how change can be monitored [24, 26,27,28,29,30].

Although the WHO framework has received some criticism (e.g. a lack of inter-connectedness between the blocks and a failure to place healthcare recipients at the centre of the health system), it remains widely used in HPSR as it is founded on the human right of the highest standard of physical and mental health and reinforces improved implementation (universal access to efficient health services) as the research outcome [25, 29, 31]. NGO data, which are, by their nature, collected in ‘the field’, therefore have considerable potential to inform and improve research into the questions asked in HPSR [7, 32]. We used the WHO building blocks framework to assess the appropriateness of studies for our scoping review as it would enable a broad overview of the different areas of HPSR in which NGO data are used and we expected most of the studies in our review to use data from developing countries [33].

Secondary analysis is the analysis of qualitative or quantitative data not produced or collected for the study in which they are later used [34, 35]. The secondary analysis of data collected, generated or about health NGOs can provide valuable insights into healthcare practice, highlight discontinuity between policy and practice, demonstrate inequitable access to healthcare, and show changes over time [18, 36]. For example, data from the patient records of NGO-run health facilities can be compared with those of government-run facilities [37]. These comparisons can highlight differences in the health and services available to populations with different sociodemographic, health or other characteristics [37, 38]. However, secondary analysis of NGO data is used infrequently in academic research in general and in HPSR in particular [10, 15, 39]. To our knowledge, there have been no previous studies on what HPSR has been conducted through secondary analysis of NGO data with a view to making recommendations to prevent the ongoing underuse of these valuable sources of information.

Health NGOs act in the public arena to improve the health and represent the health-related interests of specific groups of people or of society as a whole. Their most common undertaking is health service delivery. They are frequently commissioned by public authorities to provide services or they identify and meet the service needs of a particular and often marginalised population group (we use the term ‘marginalised groups’ to include vulnerable and hard-to-reach population groups), whilst the public health system provides more generic services [40]. In many countries in the global south, whilst universal health coverage may be the stated aim of health systems [41], in locations or disease areas with low national health system coverage, private and NGO providers frequently step in. For example, in Malawi, the government provides 62% of health services, with 37% being provided by the NGO Christian Health Association of Malawi [42, 43]. In areas affected by conflict or natural disasters, NGOs often operate as an emergency health system until there is sufficient stability for public services to be reinstated or NGO services are scaled up through NGO–public/private collaboration [44]. NGO data can therefore sometimes be the only data available in some settings or for certain population groups [44, 45].

NGOs can also perform an essential monitoring function – assessing the scale of healthcare needs and identifying the disease and related healthcare priorities and issues in accessing health services [26]. This can be especially important for settings where the public healthcare system has collapsed or provision is reduced (often affecting the most marginalised communities), whether in everyday settings in much of the global south or during disasters or crises.

As NGOs work with the patient groups, they are also increasingly involved in advocacy to influence health policy and education to ensure the capacity and sensitivity of health workforces and systems to address the needs of the community [4, 46, 47]. For example, NGOs perform independent reviews of services or reports on humanitarian crises. As such, NGOs have an opinion-forming role, sometimes formalised in the guise of think tanks, in addition to the roles of service delivery and monitoring. These varied operations illustrate NGOs’ diverse potential engagement with HPSR as the end-user (the implementer) and/or funder, data source, author or collaborator. As such, they are a key stakeholder in HPSR – they can inform the HPSR research agenda, benefit from HPSR research, and disseminate HPSR findings to the study populations and other non-academic stakeholders [48, 49].

Despite the obvious potential mutual benefits and shared aim of improving health outcomes, challenges to the use of NGO data by academics and NGO–academic collaboration remain and are well documented [7, 11, 15, 50]. Concerns include time and funding for collaboration, lack of methodological rigour and poor data quality [18, 51, 52]. For example, the timeframes of academics and NGOs can differ as the NGOs often prioritise developing trust and collaboration with the patient group whilst academics may be more concerned with data collection over a short time period [53]. Another example is the pragmatic data collection that NGOs tend to employ, rather than being concept or research-question driven. This can result in data that are inconsistent or partial, with data sometimes collected in differing samples (e.g. locations), over different timeframes and not always available in easily accessible formats [36, 54]. This causes practical difficulties of aggregation and uncertainties in measures and interpretation [6]. Academic concerns about secondary analysis of qualitative data, such as interviews or focus groups more generally, and not just qualitative data collected by NGOs include the inability to verify the source and key characteristics, difficulties assessing and addressing any bias in the data collection process, e.g. via additional data collection, and the collection of data to fit their research questions [55]. Despite these challenges, the volume, access and often depth as well as the immediacy of data generated by NGOs represent significant untapped potential for secondary analysis, especially for population groups where there is no other source of data [48].

Methods

A scoping review of existing published literature was conducted following the framework initially outlined by Arksey and O’Malley [56]. Scoping reviews aim to examine the extent, range and nature of research activity by ‘charting’ the key concepts underlying the research area and the main sources and types of evidence available. They are valuable for gaining a rapid understanding of areas that are complex and/or have not been reviewed comprehensively before.

We used the five core stages of the methodological framework, namely (1) identify the research question; (2) identify relevant studies; (3) select the studies; (4) chart the data; and (5) collate, summarise and report the results. It was outside the scope of this review to perform the optional ‘sixth stage’ consultation exercise to validate the findings of the review. As a scoping review, breadth rather than quality of the studies was prioritised. A quality appraisal was not performed [56].

Stage 1: identifying the research question

The primary objective of this review was to summarise and critically appraise the extent to which NGO data have been used in HPSR in relation to the types of NGO data used, the ways in which they have been used and areas of HPSR to which they have been applied, and to identify opportunities for greater use in future research via secondary analysis. We seek to show how these data are being used in the HPSR context and highlight their potential for health system development, particularly in developing countries [57].

Stage 2: identifying relevant studies

We performed a systematic search of papers published between January 2010 and February 2019 in the databases Web of Science, Scopus, Medline (OVID), and Health Management Information Consortium (HMIC). A post-hoc search was performed in Embase (Ovid) (Table 1).

Table 1 The literature search strategies used in the different databases

Searching interdisciplinary databases (Web of Science, Scopus) as well as those with a health focus (Medline, HMIC) ensured a comprehensive and inclusive approach as relevant papers were expected in journals for the social sciences, particularly global development, as well as health research. HMIC covers the areas of health service policy, management and administration, and public health. It contains information from DH Data (produced by the United Kingdom Department of Health) and the King’s Fund Information and Library Service database but is not limited to United Kingdom-only research.

A broad search strategy was used to identify records with an NGO key term (including community organisation/group, charity, civil society organisation) and the terms health system, delivery or governance in the title, abstract or key terms. The key terms used were informed by the authors’ knowledge of NGOs, development and health research in both high- and low-income countries. To increase the sensitivity of the searches, the key terms were mapped to subject headings, where possible. The date range was restricted to manage the large number of records retrieved by the strategy and to meet the research objective for the contemporary HPSR arena.

Following the screening process (Stage 3 below), a further post-hoc search was performed for the same period in the database Embase (Ovid), which includes some medical and related journals not indexed in Medline (Ovid). This aimed to increase the number of studies available for the review in response to concerns that the initial strategy may have missed some studies, particularly on health policy, or which used the key term ‘medicine’ rather than ‘health’.

Stage 3: study selection

Titles and abstracts, followed by the full text of potentially included articles, were screened according to the inclusion criteria by at least two reviewers (SM, AM, MB; only SM reviewed the citations identified via the post-hoc Embase search). The article inclusion criteria were (1) research that examines organisations, people and actions whose primary intent is to promote, restore and maintain health and health equity via appraisal of at least one of the six WHO health system building blocks; (2) data collected, produced by or about one or more NGOs have been used to investigate the research question; and (3) published in English in a peer-reviewed publication. The use of NGO data was determined by searching the main text and reference lists for references to NGO data and assessing how it had been used in the study/article.

The classification of NGOs can be problematic and there is considerable debate surrounding the taxonomy of NGOs. However, there is broad agreement that NGOs can be defined – and are for this paper – through the following shared structural and organisational features: (1) private or non-state; (2) self-governing; (3) formalised; and (4) not-for-profit organisations [58]. Multilevel (a mix of NGO and state/regional government agencies) and humanitarian organisations, such as the UN, WHO, and International Committee of the Red Cross, are exempt from this definition as their legal status and roles are distinct from that of NGOs. Accordingly, data collected and produced by these organisations were excluded from our review.

Stage 4: charting the data

Information from the included studies was ‘charted’ by the lead author (SM). This is the term used by Arksey and O’Malley to describe the process of identifying, recording and organising key items of information from each study according to key issues and themes. To enable comparisons between many studies with diverse study designs and contexts, the following information was recorded for each study: NGO role in the publication – author/co-author/other contribution; geographical context – research setting (country/region); developed/developing country; HPSR area – the goal and applicable WHO building blocks; study details –design (e.g. literature review, case study, evaluation); study about NGO activities (e.g. NGO programme evaluation) (Yes/No); primary data collection in addition to secondary analysis (Yes/No); NGO data – which named NGOs are referenced; data type (e.g. an unpublished report, administrative information about the NGO or NGO-collected data such as patient records); data use (e.g. providing context, case study or quantitative analysis); and health category – according to the Health Research Classification System (HRCS) [59].

Following the initial charting exercise, an additional data extraction exercise was performed that focused on one of the data categories – studies with secondary analysis of NGO-collected clinical data. To enable a more in-depth appraisal of the strengths and limitations of the NGO data and its secondary analysis in HPSR, data on the study outcomes, data strengths and limitations were extracted.

Stage 5: collating, summarising and reporting the results

Given the expected number of included studies with minimal secondary analysis of NGO data, a pragmatic approach to presenting the data was taken. Summaries of different depths are provided to give an overview of the limitations and opportunities of secondary analysis of NGO data in published HPSR and more broadly to highlight the assumption of its primary use (in the form of unpublished NGO reports) as a supporting reference and not as a potential source of data for more in-depth analysis, as follows:

  1. 1)

    a brief descriptive summary of the corpus of included studies – as a scoping review, this serves to show the expected scale of the underuse of NGO data in HPSR by summarising the extent to which studies in this field use NGO data solely or mostly as a contextual reference;

  2. 2)

    a more detailed description of the studies which analysed the NGO data to some extent (excluding studies that only used NGO data in the form of a report as a contextual or corroboratory reference) – this stage seeks to showcase the diverse types of NGO data that have been used for HPSR, where these data originate and how they are being used; and

  3. 3)

    a qualitative analysis of the studies that performed analyses on NGO-collected clinical data – this stage enables a more in-depth investigation of the opportunities for the secondary analysis of a specific type of often rich data collected by health NGOs, when and who is using these data.

During the study selection and data extraction stages (3–5 above), the lead author (SM) made notes reflecting on trends observed in the use of NGO data in HPSR and the difficulties identifying NGO data and their use (e.g. unclear attribution of data to NGOs). We provide a brief summary of the opportunities and limitations of secondary analysis of NGO data that emerged as themes in these notes. A discussion of the implications of the results, gaps and opportunities follows.

Results

The search produced 8979 records, of which 238 studies (2.7%) used NGO data to investigate an HPSR topic (Figs. 1, 2, 3 and 4) and were included in the review. Of these, 156 (66%) performed some secondary analysis of NGO data (Figs. 5, 6 and 7); 13 included secondary analyses of NGO-collected clinical data (6% of all studies; 8% of the studies performing secondary analysis) (Table 2).

Fig. 1
figure1

Flow diagram of the process of study selection

Fig. 2
figure2

Geographical context of the included studies by region. n = 238 studies; regions not mutually exclusive

Fig. 3
figure3

The types of NGO data used in the included studies. n = 238 studies; types of data not mutually exclusive. The ‘other’ category includes press releases, clinical guidelines and workshop proceedings

Fig. 4
figure4

The methods of data use in the included studies. n = 238 studies; methods of data use not mutually exclusive. The ‘other’ category includes using NGO data (e.g. reports) as guidance for programme development or to provide a definition

Fig. 5
figure5

Research setting by UN World Economic Situation and Prospects Categorisation. n = 156 studies

Fig. 6
figure6

Classification by research area using the WHO health system building blocks framework [25]. n = 156 studies; research areas not mutually exclusive

Fig. 7
figure7

Categorisation of the studies by clinical area using the Health Research Classification System. n = 156 studies; clinical areas not mutually exclusive – 10 studies (6%) investigated two or more categories

Table 2 Summary of the characteristics and data used in the studies with NGO clinical data (n=13)

Overview of all included studies

This overview describes all the studies that included NGO data in research on a health policy and systems topic (n = 238). Most of the studies were conducted in single country contexts (70%; n = 166). The remainder investigated HPSR topics across multiple countries (either within the same or different geographical regions, e.g. West Africa (14%; n = 34), or from a global perspective (16%; n = 37)) (Fig. 2). Unpublished reports produced by NGOs were the most common source of NGO data and were used in 76% of the studies (n = 182; Fig. 3).

The NGO administrative data analysed in 22% (n = 29) of the studies consisted of databases of NGOs, service information (what service was provided, to whom and where), service evaluation (programme coverage rates, outcome indicator prevalence), and NGO financial accounts (e.g. [73]). Other types of data analysed (14% (n = 27) of the studies) included project plans, operational guidelines and data, e.g. a framework for health needs assessment (n = 7) [74,75,76,77,78,79,80], press releases and news stories (n = 5) [81,82,83,84,85], definitions produced by NGOs (used as operational constructs in the studies) (n = 4) [86,87,88,89,90], clinical guidelines and guidance on clinical guideline development (n = 4) [74, 80, 91, 92], educational resources and information for clinicians (n = 3) [93,94,95], unspecified contributions of background information by NGO members (n = 3) [96,97,98], contracts (n = 3) [76, 77, 99], an interview transcript (n = 1) [82], and workshop proceedings (n = 1) [100]. The clinical data analysed in 5% of the studies were collected by NGOs via drug distribution and treatment monitoring systems and health monitoring information systems (patient records).

Half the studies (50%; n = 120) exclusively performed secondary analyses of available NGO data. The remainder performed additional data collection exercises for the purposes of the study (n = 118). Many of the studies were either literature reviews or had an initial review component (34%; n = 81). As in the wider set of studies, there was variation in the use of NGO data in these review elements. Only a minority (35%; n = 28/81) found and included grey literature (i.e. unpublished reports) in the analysis (e.g. [26, 88, 101]) – the majority excluded unpublished NGO-produced reports. Others included means of identifying relevant grey literature in their search strategies but did not find or exclude NGO reports at the screening stage [102,103,104].

More commonly, NGO-produced reports were used only as a supporting (corroboratory) reference or to provide a contextual detail, such as a statistic about the study population. For example, in a document analysis investigating armed conflict in Pakistan and the role of NGOs in restoring health services, a report by (the NGO) International Crises Group was used to support the statement, “the destruction of health centers and killing and kidnaping of doctors by the terrorists have made it more complicated for the locals to access basic health facilities” [105]. Neither this, nor any other NGO reports were included in the analysis. This is an example of corroboratory reference use. In a study on the contributions of aid organisations and international NGOs to health in Nepal, an NGO report was used to state that Nepal is ranked 146th out of 178 countries on the Corruption Perception Index [106]. No NGO resources were included in the literature review component of the study. This is an example of contextual reference use.

Of the studies, 35% (n = 82) had more than one different use of NGO data (e.g. as a supporting reference in the introduction and NGO-collected patient data in the main analysis) (Fig. 4). As well as a range of different types of NGO data used (qualitative and quantitative), the extent of the secondary analysis of NGO data varied from cursory to in-depth.

Overview of included studies with secondary analysis of NGO data

The following summary is only for those studies that performed secondary analysis of NGO data, excluding studies that only used NGO data as a corroboratory or contextual reference (n = 82 omitted (36% of the 238 studies included in the review); n = 156 included). Almost half of the studies (n = 156) investigated HPSR topics in developing countries (Fig. 5) [107].

Using the WHO health system building blocks framework, most of the studies (55%; n = 86/156) had the goal of improving health via research on service delivery (Fig. 6) [4]. For example, via evaluation of the role of the NGO in health system delivery or the efficacy of scaling-up an NGO-delivered service from the regional to national level (e.g. [76, 108, 109]). The majority focused on one building block (81%; n = 127/156); four examined all six [26, 70, 110, 111].

There were studies relating to 15 of the 21 HRCS categories. Most of the studies covered topics of generic health relevance (57%; n = 89), followed by infection (e.g. HIV, tuberculosis, sexually transmitted infection; 18%, n = 28), and reproductive health and childbirth (10%; n = 16) (Fig. 7). Eight were on mental health topics (only two were conducted in developing countries – West Africa and Lebanon [89, 112]).

NGOs had no stated involvement in the publication or funding of 67% of the studies (n = 105). NGOs were the sole author of 6% (n = 9) or co-author of 20% (n = 31) of the studies and funded either the study or the researcher in 7% (n = 11). Most of the studies with NGO authors or co-authors included secondary analysis of at least one source of their own data (78%; n = 31/40). NGO funding of the studies authored by NGOs can be assumed and is likely (but was not reported) for some of those co-authored.

Summary of the findings for studies that used NGO-collected clinical data

For a more detailed investigation of the use of NGO data in HPSR, this summary presents the findings of the studies that performed secondary analysis of clinical data collected by NGOs (n = 13).

NGO clinical data are data collected by NGOs about the health of people using their services (e.g. patient records) and not about their own activities (e.g. NGO accounts and performance monitoring systems). These data, sometimes collected over a period of many years and often in populations without access to a national health system, result in unique and longitudinal datasets, which can be used in a range of exploratory and comparative studies [65, 113], for example, to examine how the health and healthcare use of marginalised population groups is different from national patterns, how they change (in health, budgets and service provision) over time and seasonal variations [23, 60, 114,115,116,117].

Of the 13 studies in this review that performed secondary analysis of NGO-collected clinical data, 69% (n = 9) were studies in developing countries. As in the preceding overview of the studies performing secondary analysis, most of the studies using clinical data investigated service delivery-related research questions (69%; n = 9). Of the HRCS categories, most were of generic relevance (39%; n = 5), followed by infection (23%; n = 3) and reproductive health and childbirth (15%; n = 2). The characteristics and use of NGO data in these studies are summarised in Table 2.

Emergent themes

Notes reflecting on the studies and NGO data identified were made by the lead author during the screening process and analysis of the 238 studies and were discussed with the co-authors. To conclude the results section, emergent themes from these notes and discussions are described with indicative references. The themes largely arise from the disjunct between our understanding of the significant role of NGOs in health policy and systems and the perceived underuse of these data, given the sheer amount being collected and generated by health NGOs but not used in research.

Indeed, several studies included in our review were either aimed at ascertaining just how great the contribution of NGOs was in specific fields, (e.g. surgical practice, cancer care and lesbian gay bisexual and transgender health services [118,119,120]) or the scope for greater collaboration between NGOs and others (e.g. business, multilateral and other NGOs) [45, 81, 121, 122]. This research derives from the knowledge that there are large numbers of health NGOs worldwide but there is limited knowledge of the extent of their activities (outside the organisation) [105, 123,124,125] and barriers to partnerships [81, 122, 126].

When grey literature was included in reviews or as background information, we observed a tendency for authors to look to large, usually international and sometimes national, NGOs and multilevel organisations for information in the form of reports, rather than looking for information from small but potentially highly relevant regional (or national) organisations (e.g. [88, 127,128,129]). None of the studies that report searching NGO websites for relevant documents list the NGOs or search strategy used (e.g. [102, 130]). Furthermore, we noted that even studies wholly or partially about NGOs and their health-related activities sometimes did not include (or attribute) any NGO-produced or collected data (e.g. [131,132,133,134,135]). For example, one study exclusively reviewed grey literature on the mental health and psychosocial response to the 2015 earthquake in Nepal, which they obtained through online information-sharing platforms and response coordinators. Although this paper mentions the work of NGOs, and we can assume that some of the 168 documents included in the review were produced by NGOs, there is no attribution of these resources (therefore, this review was excluded from our study) [136]. Developments in these areas could both provide opportunities to improve health in the communities where NGOs operate and facilitate HPSR via data sharing and influencing data collection.

NGO data use

NGOs that are embedded in a community or act in the context of an emergency or crisis and provide a valued service, are likely to be trusted and have access to key stakeholders, enabling exploratory research on sensitive or contentious issues [66, 117, 137, 138]. The collection of data ‘in the field’ enables evaluation of the efficacy of interventions and services in the real world and differing clinical settings, adding to the data from clinical trials and to support service scale-up [61, 139,140,141]. These data can also be used to show the extent to which health systems and other development targets are being met (e.g. by mapping changes in health outcomes against development goals), by highlighting deficits and increasing pressure for these goals to be achieved [26, 142, 143]. However, only two papers used NGO data for performance monitoring in this way [26, 144] and none referenced Sustainable Development Goal (SDG) 3, which outlines targets to ensure healthy lives and promote wellbeing at all ages [145].

Although there were examples of the secondary analysis of NGO data, the number of studies doing so was relatively small given the amount of data inevitably collected and/or generated by NGOs. The effective use of NGO-collected and produced data in published HPSR shows that NGO data can be accessed and used by researchers to answer HPSR questions but is largely not [146]. NGOs with a research agenda, who might be more aware than academic institutions of the data collected by other NGOs or have established partnerships with other NGOs (e.g. joint service delivery or members of the same NGO network organisations), also appear reticent to use other NGOs’ data. For example, of the studies with NGO authors or co-authors (indicating a research agenda/interest), only four (15.4%; n = 26) either performed a secondary analysis of data collected by another NGO or referenced literature by other NGOs [86, 108, 147, 148]. One of the included studies found that NGOs may be less likely than the academic and public sector to draw on the expertise of others, including other NGOs, in the production of Health Impact Assessments [149]. The same could be true for other areas of health research.

Identification and limitations of NGO data

It was not always easy to identify which NGO’s data were being used and where the data had been acquired. In the studies, it was common practice to name an NGO and state their aims, scope, etc. but not to link to their websites, thereby not attributing the (most likely) source of the data (e.g. [132]). It was often not clear how much of the data were provided by NGO-delivered services, particularly when documents or case studies were analysed (e.g. [68, 96, 150, 151]). These practices result in the under-acknowledgment of NGO data in HPSR studies. An example of appropriate, yet limited, attribution was demonstrated by Cancedda et al. [110], who referenced the ‘Our Work’ section of the website for the NGO used as a case study (and co-author) in the research article. However, it remained unclear whether the co-author from the NGO (Partners in Health) was the primary source of the data used in the evaluation or whether NGO-produced (internal and/or external) documents and the website were the main source.

Limitations of NGO data were reported in some of the 238 included studies. In some instances, NGOs may be commissioned or tendered to provide a service within a country’s national health system. The synthesis of data across NGO and health system services was hampered by using different record-keeping systems that did not all record the same data or in the same way or data recording was incomplete for some services [77, 115, 152, 153]. This issue may be even greater where services are rolled out to new communities without consistent record-keeping and electronic data monitoring systems [64, 115].

Data collected by NGOs in challenging geographical areas or amid humanitarian crises were largely designed for practice and not for research [63, 67, 147]. There may be limited data collection and incomplete or inaccurate data [65, 72, 116], or researchers may not be fully aware what data have been collected and may be available. Elements of rigorous research are often neglected as an NGO’s first consideration is to treat and support the population in need by acquiring only essential information [38]. Adequate descriptions of the context, intervention and/or strategy, control groups and randomisation for intervention studies and generalisability were largely absent in the studies [71, 116, 154].

However, we note that claims about data limitations attributable to NGOs are not valid in all contexts. For example, health surveillance information collected by NGOs and other healthcare providers in the Central African Republic is inconsistent as disease screening programmes have been erratic in some regions due to security issues [155]. In a study on undocumented migrants visiting health clinics in Denmark, the generalisability of the findings was limited by the lack of data for the wider undocumented migrant population [72]. In both instances, the data limitations were outside the control of the NGO that provided the data for the studies.

Knowledge of these data limitations or concerns about the quality of the research using them are possible explanations for the limited use of NGO data in HPSR.

Discussion

This review is the first, to our knowledge, to use a systematic method to provide a comprehensive examination of how data collected and produced by NGOs are being used in HPSR and the extent to which secondary analyses of these data are being performed. We found evidence of the analysis of NGO data in HPSR in 66% of the studies included in the review (n = 156). NGO-produced reports were the most common form of data used (in 64% of the studies) but with limited analysis of these data (e.g. their use to critique the NGO’s activities or provide a brief example). Only 8% of the studies performed detailed analyses that used clinical data collected by NGOs. When the scale and diversity of NGO practice (and therefore data collection) and the potential value of NGO data to research are considered, our results indicate limited use of secondary analysis of NGO data. The use is limited both in the quantity of studies and the depth of analysis.

For a majority of HPSR topics, relevant data are being collected by NGOs and could be used to answer, or contribute to answering, research questions of relevance to both NGOs and academics either as a primary or supplementary data source (i.e. action research). The opportunities for NGO data use lie far beyond the use of unpublished reports as supporting references. Our study highlights the frequency of this minimal use of NGO data as well as some innovative uses of NGO-produced data, for example, NGO administrative data to assess the scale (and spending) of NGO operations to learn more about the contribution of NGOs to world health [77, 108, 156, 157]. We highlight the value of NGO-collected data for research in hard-to-reach populations, including undocumented migrants, people experiencing domestic violence and in conflict-affected areas (e.g. [105, 117, 158, 159]). Therefore, while some researchers are accessing and performing secondary analysis on NGO data, it is our view that real and perceived barriers to NGO–academic collaboration and NGO data access result in the entrenched underuse of NGO data in HPSR. This view is shared by others working in development research [7, 11, 15,16,17, 32, 48, 50, 57, 160].

As expected, NGO data use in HPSR, beyond inclusion as a corroboratory or contextual reference (n = 156 studies), occurred in studies with NGO authors/co-authors (26%), in developing countries (48% of studies), with the aim of evaluating a service delivered by NGOs (55% of studies), and in clinical areas of generic health relevance (57%). Our review was not able to identify why researchers do not perform secondary analysis of NGO data, although some of the challenges that they face when using NGO data were highlighted in Table 2, namely incomplete, inconsistent or aggregated data and lack of control/comparison groups drawn from the same population. We can, however, identify opportunities based on examples of the successful use of NGO data and suggest how its underuse might be mitigated to encourage more routine use of this largely untapped but highly valuable resource. The following discussion draws on the studies included in our review and the wider literature on NGO data and NGO–academic collaboration.

Opportunities for NGO data use

The population groups for which secondary analysis of NGO-collected clinical data (n = 13) were performed were all marginalised groups with restricted access to healthcare services, such as rural communities, people affected by conflict, and violence or drug addiction. It is highly likely that the clinical data collected by NGOs, whilst providing a valued healthcare service, were not collected for research purposes but were the best, and possibly only, source of data for these groups [6, 13]. Therefore, using NGO data (and collaboration with NGOs to collect data) is an opportunity to raise awareness of health issues in groups who are often overlooked or hard to reach by academic researchers [38, 161]. This may be the case especially in developing countries with challenging political and/or cultural contexts and where there may be stigma associated with certain health conditions such as sexually transmitted diseases [11, 66, 117]. NGOs may also be in a better position, in terms of trust, to obtain these sorts of data [113].

Greater use of NGO data could have a particularly important role to play in increasing awareness of health conditions, health needs and health service use for marginalised groups and reducing the inequalities experienced by these groups. For example, the use of NGO data could provide new insights into disparities in the health of marginalised groups compared with the general population, which could inform the development of policy and potential interventions, as well as being used more extensively in programme and facility evaluation and advocacy [38, 109, 162].

Some of the studies we analysed used longitudinal NGO data. These are extremely valuable for monitoring changes in health over time and are important in the context of determinants of health, including changing social, economic and environmental conditions [163]. Longitudinal data are especially valuable when environmental and political changes are occurring at an unprecedented rate such as in humanitarian crises. Conducting long-term studies has substantial cost implications that all organisations face. Accessing available longitudinal data sets produced by NGOs can facilitate the research of interest whilst limiting the costs for research institutions [6]. In other cases, NGOs work with a community for a relatively short period of time, ceasing activity when donor funding ends. It is important that the benefits of these data are realised, not least because demonstration of impact can support requests for further funding [54, 160, 164].

Longitudinal data are also important for conditions that develop over time or may be rooted in childhood or mental health conditions [141, 163]. However, only eight of the studies that performed secondary analysis on NGO data (n = 156) addressed mental health topics and only two in developing countries. This is perhaps not surprising as, despite mental health accounting for 27% of all years lived with disability worldwide, mental health has received far less interest in research and practice in developing than developed countries [165, 166]. The evidence of an absence of available (including NGO) data in these contexts can also help build the case for funding for, for example, the scaling up of NGO service delivery and research activities (including to collect better quality data) in these (developed and developing country) locations [45, 167, 168]. The assumption (and sometimes reality) of poor-quality data is a common academic explanation for not attempting secondary analysis of NGO data [8]. However, it is not true of all NGO data [148]; for example, the NGO, Reproductive Health Uganda, provides training on data collection, storage and reporting to ensure minimum data standards across their network of 17 health clinics [69]. Entering into collaborations with NGOs working in the field of interest can benefit both partners in their shared aim of improving health outcomes [11, 17, 167]. NGO data and NGO–academic collaboration can be particularly valuable in action research, especially within the contexts of refining approaches to achieve the SDGs and developing research methods to collect high quality data in challenging settings [48, 49, 52]. HPSR is increasingly using the SDGs as a framework for agenda-setting [8]. NGO data can be used for measuring progress against SDG targets, not least because health intersects with many other areas of development [36, 145, 169].

Collaboration could also help the HPSR based on NGO data to be disseminated faster, especially in disaster or conflict areas. For example, most of the operational research on the Ebola outbreak from March 2014–December 2017 was published after the WHO had initially declared the outbreak over in January 2016 rather than during the outbreak [170]. NGO–academic collaboration could possibly have enabled analysis and dissemination of the data from these contexts to the Ebola research community and NGOs operating in the field sooner, more rapidly advancing understanding of the disease and policies to respond to outbreaks [170].

The benefits of collaboration for academics include accessing NGO data that provides them with an opportunity to influence data collection tools and methods to improve data quality and relevance for their purposes [11]. For example, academics can work with NGOs to help ensure that data are collected in a way that means they are consistent over time and can be used for temporal analysis [63, 67]. Academics need to recognise the potential of the data whilst accepting the inevitable limitations of data collection by non-academics in challenging settings, with changing social and environmental landscapes, aiming to collect only essential information [64, 66, 71, 113].

The lack of experience of many NGOs in data handling and management can also limit the further use of their data, but this is another area where there can be positive sharing of best practice leading to improved capacity over time [13, 15, 32]. In addition, for researchers, secondary analysis provides the benefit of being able to assess data quality prior to performing the analysis [148]. The process of appraising NGO data has time and cost implications, but much can be learnt by academics, such as what additional data collection activities are required and how their research questions can be adapted based on the available data [34].

Through collaboration, an NGO develops its research capacity, the ability to evaluate its activities and can influence its partners’ research agenda. The ability to produce evidence of local health needs and deficits in service provision can also strengthen NGOs’ advocacy for health policy and funding reforms from governments and donors [6, 57]. Developing research (and importantly, evaluation) capacity has been shown to facilitate the sustainability and scaling-up of NGO activities [167, 171, 172]. However, in the year 2000, only 23% of 37,000 international NGOs were performing research activities (no more up-to-date data were available). Explanations for limited or no collaboration with academic institutions include suspicion of the academics’ agendas (including competing interest and power inequalities in decision-making about the ethics, purpose, application and dissemination of the research [7, 32, 173]); power and global north–south inequalities [174,175,176]; and doubts about the value of communicating with academic audiences [167, 171, 172]. Issues of competing interests can arise between what data collection are considered necessary by the NGO (e.g. to treat or monitor health in the population) versus by the academics (i.e. to produce high quality, publishable research), the ownership of this data and linkage between new and existing data collected [32, 173, 177, 178].

We advocate for greater NGO–academic collaboration. The sharing of data could work both ways as, through collaboration, NGOs that perform research could benefit by accessing other datasets such as those collected by academic or public sector institutions [179]. NGO–NGO, NGO–business and NGO–public sector collaborations can also help develop research capacity (and programme evaluation), thus facilitating the collection and use of NGO data in HPSR [121, 172, 180, 181].

Developing and implementing data standards and protocols to be adopted by (or together with) NGOs could be a way to enhance the wider use of the data they collect. Secondary data analysis requires the NGOs to grant permission and re-issue data for another purpose than originally intended. We are aware of the increasingly stringent ethical requirements constraining research organisations [182, 183]. The absence of ethical standards informing the collection and management of data by NGOs (or equivalence of this governance to the standards used by research institutions) may prohibit the use of NGO data by academics [184]. The need for standardised data protection and for inter- and extra-NGO sharing procedures is a current debate for humanitarian and development agencies [185, 186]. Given the potential importance of NGO data for raising attention of marginalised groups and calls for data interoperability (joined up data) to achieve development goals [187], differences in methodological protocols and data standards can prevent the conditions of marginalised groups being brought to wider attention [50]; this could maintain inequalities or even exacerbate them.

During the review, we found many instances of the under- or ambiguous acknowledgement of NGO data in various forms. Some studies stated that the data were provided by an NGO or that a number of NGOs had been involved in the data collection but did not provide their names. Examples include referring to “the NGO forum of Cambodia” (comprised of several unnamed NGOs) [188], stating that “six of the nine data providers in the study were NGOs” but giving no further details [77], and acknowledging contributions by NGOs but not stating whether they provided data [97]. Elsewhere, NGO activities were used in case studies or given as exemplars (e.g. [96, 189]), sometimes using information from their websites (e.g. [190]). Oftentimes, these data were either not attributed in the references (thus the study becomes the de facto data source) or the reference was for another source where the data had been reported (i.e. not the original source of the data) [189]. Two studies refer to data on funding received by NGOs but, as they do not reference the source(s) of this NGO administrative data, the study becomes the source [106, 126]. Further, when an example of a specific NGO activity is used in a WHO publication and this publication is referenced, WHO becomes the data source and not the NGO.

By not attributing the NGO directly, inequalities of knowledge and power between NGOs and research organisations, multilevel or network organisations such as WHO, the UN, World Bank and are upheld [53]. This failure to attribute research to NGOs was also seen when the Global Health Watch Report 4 [191], which has NGO and NGO network co-authors (e.g. Health Action International and the People’s Health Movement), was cited [23]. However, it is worth noting that collaboration between NGOs and multilevel organisations does enable NGO research capacity and can raise the profile of their activities. For example, Kilic et al. [192] refer to documents on the healthcare system monitoring activities of the Turkish Diabetes Foundation, which were jointly published by the Turkish Ministry of Health, WHO Europe and the International Diabetes Federation of Europe. The multinational and multilevel organisations may have assisted the Turkish Diabetes Foundation with publication if they did not have the capacity or resources to publish these documents independently. Their support with dissemination could also greatly increase the reach of these documents. Whatever the reason for this and similar NGO–multilevel organisation collaborations, we would stress the need for greater transparency in data use, clarity in relation to source attribution, and appropriate and full acknowledgment of NGO data and contributions [32].

Our analysis has focused on published research literature, but NGO research is more prevalent in grey literature, as demonstrated by the number of studies using unpublished NGO literature as contextual or corroboratory references. The relevant grey literature is harder to locate, leading researchers to use the pragmatic strategy of reference list searching and looking for relevant documents on the websites of organisations that they know to work in the field of interest [79, 193,194,195,196]. This practice perpetuates imbalances in the visibility of research by large NGOs and multilevel organisations compared with smaller, less well-known NGOs [197]. There are search tools and guidelines for searching grey literature, but the academic preference for using peer-reviewed published literature in formal reviews remains [198]. If grey literature was more routinely included in literature reviews in HPSR, researchers would find more NGO-produced (unpublished) reports that may contain valuable data for inclusion in literature syntheses. The inclusion of these references could be particularly beneficial in areas with little published research (e.g. in marginalised groups) or areas dominated by published research from high-income countries (e.g. mental health). Given the more common use of reports produced by multilevel or international NGOs, systematic searches of grey literature and use of the unpublished data identified could also help raise awareness of research activities by smaller NGOs. This increased exposure could also help them attract funding and academic collaborators to grow their research capacity [15].

We recognise that a review of this nature inevitably has limitations. We performed a literature scoping exercise across a range of interdisciplinary and health-specific databases, favouring a broad search strategy in a few key databases rather than a more focused strategy in every potentially relevant database. We may also have missed some relevant studies due to the under-acknowledgement of NGO contributions, the challenge of identifying whether named organisations fitted the NGO definition and the inclusion of only studies published in English. All of this points to the importance of further research in this field to examine in more depth the value of different types of NGO data identified here but not investigated in detail. Greater rigour in data sharing agreements and more systematic access to the data collected by NGOs is also important. Additionally, the comparison of practices of NGO data use in other areas of development research could help researchers begin to mitigate the issues of NGO data use in HPSR, e.g. by adopting best practices and NGO–academic collaboration standards used elsewhere [48].

We gave a broad overview of how NGO-collected and produced data have been used and the extent of the secondary analysis of NGO data in HPSR, with a more in-depth look at the use of NGO-collected clinical data. It was not possible to provide a comprehensive analysis of how each different type of data identified were used, although we identified some examples of innovative uses of NGO-produced data such as NGO accounts. The public availability of electronic data produced by and about NGOs (including news stories, financial information and court proceedings) is a potentially rich seam for secondary analysis by researchers willing to use less traditional data sources.

Conclusions

In this review, we have given an overview and specific examples of how, in HPSR, NGO-collected and produced data are being used and in what contexts secondary analysis of NGO data is being performed. There were frequent examples of the use of secondary analysis of NGO data in service delivery evaluations, especially in developing countries and when the NGO is the author or co-author of the study. To a lesser extent, we found examples of the use of NGO-collected clinical data and NGO administrative and other types of data published by researchers without any (known) connections to NGOs.

We have argued that given the scale of health NGO operations worldwide, NGO data constitute a vast and valuable source of data for HPSR. Yet, the value of these data is under-realised, and the data underused and under-acknowledged in HPSR. By drawing on the studies included in the review and wider literatures on NGO data and NGO–academic collaboration, we have offered suggestions for routes to the greater use of secondary analysis of NGO data in HPSR. These include the routine inclusion of grey literature in literature reviews and greater NGO-academic collaboration that is informed by clear and agreed standards for research protocols, ethics and data management. With its broad scope, this review offers an entry point for further discussion of how secondary analysis of NGO data can be used more extensively in HPSR and other areas of research driven by development goals.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Abbreviations

NGO:

Non-governmental organisations

HMIC:

Health Management Information Consortium

HPSR:

Health policy and systems research

HRCS:

Health Research Classification System

SDGs:

Sustainable Development Goals

References

  1. 1.

    Lewis D, Kanji N. Non-government organizations and development. London: Routledge; 2009.

    Google Scholar 

  2. 2.

    Public Interest Registry. About PIR. 2015. https://thenew.org/org-people/about-pir/. Accessed 14 Feb 2020.

    Google Scholar 

  3. 3.

    Sarkar A. NGOs, the New Lexicon of Health Care. New Delhi: Concept Publishing Company; 2005.

    Google Scholar 

  4. 4.

    Ascher W, Brewer GD, Cheema GS, Heffron JM. Evolving roles of NGOs in developing countries. The evolution of development thinking. New York: Palgrave Macmillan; 2016.

    Google Scholar 

  5. 5.

    Banks N, Hulme D. The role of NGOs and civil society in development and poverty reduction. Brooks World Poverty Institute Working Paper 171. Manchester: BWPI; 2012.

    Google Scholar 

  6. 6.

    Delisle H, Hatcher Roberts J, Munro M, Jones L, Gyorkos TW. The role of NGOs in global health research for development. Health Res Policy Syst. 2005;3:3.

    PubMed  PubMed Central  Article  Google Scholar 

  7. 7.

    Sellers S. Promoting NGO–academic partnerships for population, health, and environment learning. Dev Pract. 2017;27(2):208–19.

    Article  Google Scholar 

  8. 8.

    Alliance for Health Policy and Systems Research. World report on health policy and systems research. Geneva: World Health Organization; 2017.

    Google Scholar 

  9. 9.

    Sheikh K, Gilson L, Agyepong IA, Hanson K, Ssengooba F, Bennett S. Building the field of health policy and systems research: framing the questions. PLoS Med. 2011;8(8):e1001073.

    PubMed  PubMed Central  Article  Google Scholar 

  10. 10.

    de Las CL, Gyatend T, Pritchard D. The power of data: Is the charity sector ready to plug in? London: New Philanthropy Capital; 2013.

    Google Scholar 

  11. 11.

    Aniekwe CC, Hayman R, Mdee A, Akuni J, Lall P, Stevens D. Working paper. Academic-NGO collaboration in international development research: a reflection on the issues. Hove: Development Studies Association; 2012.

    Google Scholar 

  12. 12.

    Piotrowicz M, Cianciara D. The role of non-governmental organizations in the social and the health system. Przegl Epidemiol. 2013;67(1):69–74.

    PubMed  Google Scholar 

  13. 13.

    Esty DC, Ivanova MH. Global environmental governance: options & opportunities. New Haven: Yale School of Forestry & Environmental Studies; 2002.

    Google Scholar 

  14. 14.

    Third Sector Research Centre. Understanding the UK third sector. The Work of the Third Sector Research Centre 2008–2013. Birmingham: University of Birmingham; 2014.

    Google Scholar 

  15. 15.

    Gooding K. The role of NGOs’ service delivery experience in developing relevant research agendas: experience and challenges among NGOs in Malawi. Health Res Policy Syst. 2017;15:38.

    PubMed  PubMed Central  Article  Google Scholar 

  16. 16.

    Brotchie J. Deepening interactions between academia and the third sector - the alliance for useful evidence. London: Alliance for Useful Evidence; 2016. https://www.alliance4usefulevidence.org/deepening-interactions-between-academia-and-the-third-sector/. Accessed 14 Feb 2020.

    Google Scholar 

  17. 17.

    Citrin D, Mehanni S, Acharya B, et al. Power, potential, and pitfalls in global health academic partnerships: review and reflections on an approach in Nepal. Glob Health Action. 2017;10(1):1367161.

    PubMed  PubMed Central  Article  Google Scholar 

  18. 18.

    Green D. Why is it so hard for academics and NGOs to work together. From poverty to power. London: Oxfam; 2016. https://oxfamblogs.org/fp2p/why-is-it-so-hard-for-academics-and-ngos-to-work-together/. Accessed 15 July 2019.

    Google Scholar 

  19. 19.

    Rogers K. Are NGOs doing enough to share data? Washington, DC: Devex; 2016. https://www.devex.com/news/sponsored/are-ngos-doing-enough-to-share-data-88455. Accessed 14 Feb 2020.

    Google Scholar 

  20. 20.

    Berkeley Library. Library guides: non governmental organizations (NGOs). Berkeley: Data Berkeley, UC Berkeley; 2020. https://guides.lib.berkeley.edu/c.php?g=496970&p=3401927. Accessed 14 Feb 2020.

    Google Scholar 

  21. 21.

    National Council for Voluntary Organisations. UK Civil Society Almanac 2019. London: NCVO; 2019. https://almanac.fc.production.ncvocloud.net/. Accessed 20 Mar 2020.

    Google Scholar 

  22. 22.

    Bahuguna P, Mukhopadhyay I, Chauhan AS, Rana SK, Selvaraj S, Prinja S. Sub-national health accounts: experience from Punjab State in India. PLoS One. 2018;13(12):e0208298.

    PubMed  PubMed Central  Article  Google Scholar 

  23. 23.

    Baum F, Freeman T, Sanders D, Labonte R, Lawless A, Javanparast S. Comprehensive primary health care under neo-liberalism in Australia. Soc Sci Med. 2016;168:43–52.

    PubMed  Article  Google Scholar 

  24. 24.

    Shakarishvili G, Atun R, Berman P, Hsiao W, Burgess C, Lansang MA. Converging health systems frameworks: towards a concepts-to-actions roadmap for health systems strengthening in low and middle income countries. Glob Health Gov. 2010;3(2). https://blogs.shu.edu/ghg/2010/04/01/converging-health-systems-frameworks/. Accessed 4 June 2020.

  25. 25.

    Alliance for Health Policy and Systems Research. Systems thinking for health systems strengthening. Geneva: WHO; 2009.

    Google Scholar 

  26. 26.

    Qirbi N, Ismail SA. Health system functionality in a low-income country in the midst of conflict: the case of Yemen. Health Policy Plan. 2017;32(6):911–22.

    PubMed  Article  Google Scholar 

  27. 27.

    Manyazewal T. Using the World Health Organization health system building blocks through survey of healthcare professionals to determine the performance of public healthcare facilities. Arch Public Health. 2017;75:50.

    PubMed  PubMed Central  Article  Google Scholar 

  28. 28.

    World Health Organization. Strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: WHO; 2007.

    Google Scholar 

  29. 29.

    Backman G, Hunt P, Khosla R, et al. Health systems and the right to health: an assessment of 194 countries. Lancet. 2008;372(9655):2047–85.

    PubMed  Article  Google Scholar 

  30. 30.

    Bonds MH, Garchitorena A, Cordier L, et al. Advancing a science for sustaining health: establishing a model health district in Madagascar. BioRxiv. 2017;141549. https://doi.org/10.1101/141549.

  31. 31.

    Peters DH. Health policy and systems research: the future of the field. Health Res Policy Syst. 2018;16(1):84.

    PubMed  PubMed Central  Article  Google Scholar 

  32. 32.

    Olivier C, Hunt MR, Ridde V. NGO–researcher partnerships in global health research: benefits, challenges, and approaches that promote success. Dev Pract. 2016;26(4):444–55.

    Article  Google Scholar 

  33. 33.

    van Olmen J, Marchal B, Van Damme W, Kegels G, Hill PS. Health systems frameworks in their political context: framing divergent agendas. BMC Public Health. 2012;12:774.

    PubMed  PubMed Central  Article  Google Scholar 

  34. 34.

    Johnston MP. Secondary data analysis: a method of which the time has come. Qual Quant Methods Lib. 2014;3(3):619–26.

    Google Scholar 

  35. 35.

    Irwin S. Qualitative secondary data analysis: ethics, epistemology and context. Prog Dev Stud. 2013;13(4):295–306.

    Article  Google Scholar 

  36. 36.

    Lansang MA, Tollman S, Kasonde J, Butler P. Health research for development: the continuing challenge. A discussion paper. Bangkok: International Conference on Health Research for Development; 2000.

    Google Scholar 

  37. 37.

    Mercer A, Khan MH, Daulatuzzaman M, Reid J. Effectiveness of an NGO primary health care programme in rural Bangladesh: evidence from the management information system. Health Policy Plan. 2004;19(4):187–98.

    PubMed  Article  Google Scholar 

  38. 38.

    Flanagan SM, Hancock B. ‘Reaching the hard to reach’ - lessons learned from the VCS (voluntary and community Sector) A qualitative study. BMC Health Serv Res. 2010;10:92. https://doi.org/10.1186/1472-6963-10-92.

    Article  PubMed  PubMed Central  Google Scholar 

  39. 39.

    Cole AP, Friedlander DF, Trinh QD. Secondary data sources for health services research in urologic oncology. Urol Oncol. 2018;36(4):165–73.

    PubMed  Article  Google Scholar 

  40. 40.

    Gilson L, Sen PD, Mohammed S, Mujinja P. The potential of health sector non-governmental organizations: policy options. Health Policy Plan. 1994;9(1):14–24.

    CAS  PubMed  Article  Google Scholar 

  41. 41.

    World Health Organization and the International Bank for Reconstruction and Development/The World Bank. Tracking universal health coverage: 2017 global monitoring report. Geneva: World Health Organization and the International Bank for Reconstruction and Development/The World Bank; 2017.

    Google Scholar 

  42. 42.

    Kalinga OJM. Historical dictionary of Malawi. Lanham: Rowman & Littlefield; 2012.

    Google Scholar 

  43. 43.

    Christian Health Association of Malawi. History of the Christian Health Association of Malawi. Lilongwe: Christian Health Association of Malawi; 2019.

    Google Scholar 

  44. 44.

    Alliance for Health Policy and Systems Research. Neglected health systems research: health policy and systems research in conflict-affected fragile states. Geneva: World Health Organization; 2008.

    Google Scholar 

  45. 45.

    Davis SL, Goedel WC, Emerson J, Guven BS. Punitive laws, key population size estimates, and Global AIDS Response Progress Reports: an ecological study of 154 countries. J Int AIDS Soc. 2017;20(1):21386.

    PubMed  PubMed Central  Article  Google Scholar 

  46. 46.

    Knabe A, McCarthy M. Civil society organisations and public health research-evidence from eight European union new member states. Cent Eur J Public Health. 2012;20(4):287–93.

    PubMed  Article  Google Scholar 

  47. 47.

    Young WB, Ryu H. Secondary data for policy studies: benefits and challenges. Policy Polit Nurs Pract. 2000;1(4):302–7.

    Article  Google Scholar 

  48. 48.

    Roper L. Achieving successful academic-practitioner research collaborations. Dev Pract. 2010;12(3–4):338–45.

    Google Scholar 

  49. 49.

    Sanders D, Labonte R, Baum F, Chopra M. Making research matter: a civil society perspective on health research. Bull World Health Organ. 2004;82(10):757–63.

    PubMed  PubMed Central  Google Scholar 

  50. 50.

    Green D. The NGO-academia interface: the impact initiative. London: The impact initiative; 2017. https://www.theimpactinitiative.net/impact-lab/collection/ngo-academia-interface.

    Google Scholar 

  51. 51.

    Moran N. Successfully connecting NGO practice and health research. Medicus Mundi Schweiz Bull. 2015;135(12).

  52. 52.

    Zachariah R, Ford N, Maher D, et al. Is operational research delivering the goods? The journey to success in low-income countries. Lancet Infect Dis. 2012;12(5):415–21.

    PubMed  Article  Google Scholar 

  53. 53.

    Newman K, Bharadwaj S, Fransman J. Rethinking research impact through principles for fair and equitable partnerships. IDS Bull. 2019;50:1.

    Article  Google Scholar 

  54. 54.

    Monaghan A, Lycett N. Big data and humanitarian supply networks: can big data give voice to the voiceless? San Jose: IEEE Global Humanitarian Technology Conference; 2013.

    Google Scholar 

  55. 55.

    Heaton J. Secondary analysis of qualitative data: an overview. Hist Soc Res/Historische Sozialforschung. 2008;33(3(125)):33–45.

    Google Scholar 

  56. 56.

    Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2007;8(1):19–32.

    Article  Google Scholar 

  57. 57.

    Shroff ZC, Javadi D, Gilson L, Kang R, Ghaffar A. Institutional capacity to generate and use evidence in LMICs: current state and opportunities for HPSR. Health Res Policy Syst. 2017;15(1):94.

    PubMed  PubMed Central  Article  Google Scholar 

  58. 58.

    Gordenker L, Weiss TG. Pluralising global governance: analytical approaches and dimensions. Third World Q. 1995;16(3):357–87.

    Article  Google Scholar 

  59. 59.

    Health Research Classification System Online. Health categories. London: UKCRC; 2004. https://hrcsonline.net/health-categories/.

    Google Scholar 

  60. 60.

    Bini S, Cerri C, Rigamonti AE, Bertazzi PA, Fiorini G, Cella SG. Pharmacoepidemiological data from drug dispensing charities as a measure of health patterns in a population not assisted by the Italian National Health Service. J Public Health Res. 2016;5(2):623.

    PubMed  PubMed Central  Article  Google Scholar 

  61. 61.

    Carlson LC, Hatcher KW, Tomberg L, Kabetu C, Ayala R, van der Burg R. Inequitable access to timely cleft palate surgery in low- and middle-income countries. World J Surg. 2016;40(5):1047–52.

    PubMed  Article  Google Scholar 

  62. 62.

    Cunningham C, Wensley R, Blacker D, Bache J, Stonier C. Occupational therapy to facilitate physical activity and enhance quality of life for individuals with complex neurodisability. Br J Occupat Ther. 2012;75:106–10.

    Article  Google Scholar 

  63. 63.

    Deboutte D, O'Dempsey T, Mann G, Faragher B. Cost-effectiveness of caesarean sections in a post-conflict environment: a case study of Bunia, Democratic Republic of the Congo. Disasters. 2013;37:S105–20.

    PubMed  Article  Google Scholar 

  64. 64.

    Gurung A, Narayanan P, Prabhakar P, et al. Large-scale STI services in Avahan improve utilization and treatment seeking behaviour amongst high-risk groups in India: an analysis of clinical records from six states. BMC Public Health. 2011;11(Suppl. 6):S10.

    PubMed  PubMed Central  Article  Google Scholar 

  65. 65.

    Jacobs B, Thome JM, Overtoom R, Sam SO, Indermuhle L, Price N. From public to private and back again: sustaining a high service-delivery level during transition of management authority: a Cambodia case study. Health Policy Plan. 2010;25(3):197–208.

    PubMed  Article  Google Scholar 

  66. 66.

    Kohli A, Makambo MT, Ramazani P, et al. A Congolese community-based health program for survivors of sexual violence. Confl Health. 2012;6(1):6.

    PubMed  PubMed Central  Article  Google Scholar 

  67. 67.

    Lindgren TG, Deutsch K, Schell E, et al. Using mobile clinics to deliver HIV testing and other basic health services in rural Malawi. Rural Remote Health. 2011;11(2):1682.

    CAS  PubMed  Google Scholar 

  68. 68.

    Marsden J, Eastwood B, Jones H, et al. Risk adjustment of heroin treatment outcomes for comparative performance assessment in England. Addiction. 2012;107(12):2161–72.

    PubMed  Article  Google Scholar 

  69. 69.

    Odwe G, Gray K, Kyarimpa A, Obare F, Nagendi G. Introduction of subcutaneous depot Medroxyprogesterone Acetate (DMPA-SC) injectable contraception at facility and community levels: pilot results from 4 Districts of Uganda. Glob Health Sci Pract. 2018;6(4):711–22.

    PubMed  PubMed Central  Article  Google Scholar 

  70. 70.

    Poenaru D. Getting the job done: analysis of the impact and effectiveness of the SmileTrain program in alleviating the global burden of cleft disease. World J Surg. 2013;37(7):1562–70.

    CAS  PubMed  Article  Google Scholar 

  71. 71.

    Ruckstuhl L, Lengeler C, Moyen JM, Garro H, Allan R. Malaria case management by community health workers in the Central African Republic from 2009–2014: overcoming challenges of access and instability due to conflict. Malar J. 2017;16(1):388.

    PubMed  PubMed Central  Article  Google Scholar 

  72. 72.

    Wendland A, Ehmsen BK, Lenskjold V, Astrup BS, Mohr M, Williams CJ, Cowan SA. Undocumented migrant women in Denmark have inadequate access to pregnancy screening and have a higher prevalence Hepatitis B virus infection compared to documented migrants in Denmark: a prevalence study. BMC Public Health. 2016;16:426.

    PubMed  PubMed Central  Article  Google Scholar 

  73. 73.

    Nikpay SS, Ayanian JZ. Hospital charity care-effects of new community-benefit requirements. N Engl J Med. 2015;373(18):1687–90.

    PubMed  Article  Google Scholar 

  74. 74.

    Kempers J, Ketting E, Chandra-Mouli V, Raudsepp T. The success factors of scaling-up Estonian sexual and reproductive health youth clinic network - from a grassroots initiative to a national programme 1991–2013. Reprod Health. 2015;12:2.

    PubMed  PubMed Central  Article  Google Scholar 

  75. 75.

    Zaidi S, Mayhew SH, Palmer N. Bureaucrats as purchasers of health services: limitations of the public sector for contracting. Public Adm Dev. 2011;31:135–48.

    Article  Google Scholar 

  76. 76.

    Kelly RP, Burke J, Waddell S, Lachance L. Increasing opportunities for health in a southeast Michigan community through local policy change. Health Promot Pract. 2018;20(1):116–27.

    PubMed  Article  Google Scholar 

  77. 77.

    Taylor CB, Stevenson M, Jan S, Liu B, Tall G, Middleton PM, Fitzharris M, Myburgh J. An investigation into the cost, coverage and activities of helicopter emergency medical services in the state of New South Wales, Australia. Injury. 2011;42(10):1088–94.

    PubMed  Article  Google Scholar 

  78. 78.

    George AS, Mohan D, Gupta J, LeFevre AE, Balakrishnan S, Ved S, Khanna R. Can community action improve equity for maternal health and how does it do so? Research findings from Gujarat, India. Int J Equity Health. 2018;17:125.

    PubMed  PubMed Central  Article  Google Scholar 

  79. 79.

    Patel P, Roberts B, Conteh L, Guy S, Lee-Jones L. A review of global mechanisms for tracking official development assistance for health in countries affected by armed conflict. Health Policy. 2011;100(2–3):116–24.

    PubMed  Article  Google Scholar 

  80. 80.

    Tabash T, Abuqamar M. Assessment of the status of private and non-governmental pharmaceuticals supply warehouses in Gaza Strip, Palestine. Asian J Pharmaceut Clin Res. 2018;11(2):300–7.

    Article  Google Scholar 

  81. 81.

    Murdie A. Scrambling for contact: the determinants of inter-NGO cooperation in non-Western countries. Rev Int Organ. 2014;9(3):309–31.

    Article  Google Scholar 

  82. 82.

    Evensen JV, Stokke K. United against HIV/AIDS? Politics of local governance in HIV/AIDS treatment in Lusikisiki, South Africa. J South Afr Stud. 2010;36(1):151–67.

    Article  Google Scholar 

  83. 83.

    Dunn JT, Lesyna K, Zaret A. The role of human rights litigation in improving access to reproductive health care and achieving reductions in maternal mortality. BMC Pregnancy Childbirth. 2017;17:367.

    PubMed  PubMed Central  Article  Google Scholar 

  84. 84.

    Libman K, Freudenberg N, Sanders D, Puoane T, Tsolekile L. The role of urban food policy in preventing diet-related non-communicable diseases in Cape Town and New York. Public Health. 2015;129(4):327–35.

    CAS  PubMed  Article  Google Scholar 

  85. 85.

    Song P. The Ice Bucket Challenge: the public sector should get ready to promptly promote the sustained development of a system of medical care for and research into rare diseases. Intractable Rare Dis Res. 2014;3(3):94–6.

    PubMed  PubMed Central  Article  Google Scholar 

  86. 86.

    Cantril C, Haylock PJ. Patient navigation in the oncology care setting. Semin Oncol Nurs. 2013;29(2):76–90.

    PubMed  Article  Google Scholar 

  87. 87.

    Aseyo RE, Mumma J, Scott K, Nelima D, Davis E, Baker KK, Cumming O, Dreibelbis R. Realities and experiences of community health volunteers as agents for behaviour change: evidence from an informal urban settlement in Kisumu, Kenya. Hum Resour Health. 2018;16(1):53.

    PubMed  PubMed Central  Article  Google Scholar 

  88. 88.

    Chandra-Mouli V, Lenz C, Adebayo E, Lang Lundgren I, Gomez Garbero L, Chatteriee S. A systematic review of the use of adolescent mystery clients in assessing the adolescent friendliness of health services in high, middle, and low-income countries. Glob Health Action. 2018;11(1):1536412.

    PubMed  PubMed Central  Article  Google Scholar 

  89. 89.

    Yassin N, Taha AA, Ghantous Z, Atoui MM, Forgione F. Evaluating a mental health program for Palestinian refugees in Lebanon. J Immigr Minor Health. 2018;20(2):388–98.

    PubMed  Article  Google Scholar 

  90. 90.

    Piotrowicz M, Cianciara D. Associations and foundations in the field of health care and their role in the health system of Poland. Przegl Epidemiol. 2013;67(1):63–8.

    PubMed  Google Scholar 

  91. 91.

    Guinness L. What can transaction costs tell us about governance in the delivery of large scale HIV prevention programmes in southern India? Soc Sci Med. 2011;72(12):1939–47.

    PubMed  PubMed Central  Article  Google Scholar 

  92. 92.

    Goldman J, Meuser J, Lawrie L, Rogers J, Reeves S. Interprofessional primary care protocols: a strategy to promote an evidence-based approach to teamwork and the delivery of care. J Interprof Care. 2010;24(6):653–65.

    PubMed  Article  Google Scholar 

  93. 93.

    Shipman KE, Doyle A, Arden H, Jones T, Gittoes NJ. Development of fracture liaison services: what have we learned? Injury. 2017;48:S4–9.

    PubMed  Article  Google Scholar 

  94. 94.

    Hafeez S, Singhera M, Huddart R. Exploration of the treatment challenges in men with intellectual difficulties and testicular cancer as seen in Down syndrome: single centre experience. BMC Med. 2015;13:152.

    PubMed  PubMed Central  Article  Google Scholar 

  95. 95.

    Fehling M, Nelson BD, Ahn R, Eckardt M, Tiernan M, Purcell G, El-Bashir A, Burke TF. Development of a community-based maternal, newborn and child emergency training package in South Sudan. Public Health. 2013;127(9):797–805.

    CAS  PubMed  Article  Google Scholar 

  96. 96.

    Mercer MA, Thompson SM, de Araujo RM. The role of international NGOs in health systems strengthening: the case of Timor-Leste. Int J Health Serv. 2014;44(2):323–35.

    PubMed  Article  Google Scholar 

  97. 97.

    Chowdhury AMR, Bhuiya A, Chowdhury ME, Rasheed S, Hussain Z, Chen LC. The Bangladesh paradox: exceptional health achievement despite economic poverty. Lancet. 2019;382(9906):1734–45.

    Article  Google Scholar 

  98. 98.

    Rosenquist R, Golichenko O, Roosen T, Ravenscroft J. A critical player: the role of civil society in achieving universal health coverage. Glob Health Gov. 2013;2 http://blogs.shu.edu/wp-content/blogs.dir/109/files/2014/02/GHGJ_62_100-105_ROSENQUIST_ET_AL.pdf. Accessed 15 Oct 2019.

  99. 99.

    Lavoie J, Varcoe C, Wathen CN, Ford-Gilboe M, Browne AJ. Sentinels of inequity: examining policy requirements for equity-oriented primary healthcare. BMC Health Serv Res. 2018;18:705.

    PubMed  PubMed Central  Article  Google Scholar 

  100. 100.

    Colombini M, Ali SH, Watts C, Mayhew SH. One stop crisis centres: a policy analysis of the Malaysian response to intimate partner violence. Health Res Policy Syst. 2011;9:25.

    PubMed  PubMed Central  Article  Google Scholar 

  101. 101.

    Zabdyr-Jamróz M. Voluntary (NGO) Sector’s involvement in health promotion for older population in Europe. Epidemiol Biostatist Public Health. 2017;14(2):e12420–2.

    Google Scholar 

  102. 102.

    Ahmed SM, Nordeng H, Sundby J, Aragaw YA, de Boer HJ. The use of medicinal plants by pregnant women in Africa: a systematic review. J Ethnopharmacol. 2018;224:297–313.

    PubMed  Article  Google Scholar 

  103. 103.

    Bradley EH, Curry LA, Taylor LA, et al. A model for scale up of family health innovations in low-income and middle-income settings: a mixed methods study. BMJ Open. 2012;2(4):e000987.

    PubMed  PubMed Central  Article  Google Scholar 

  104. 104.

    Sasaki H, Bouesseau MC, Marston J, Mori R. A scoping review of palliative care for children in low- and middle-income countries. BMC Palliative Care. 2017;16:60.

    PubMed  PubMed Central  Article  Google Scholar 

  105. 105.

    Khan MA, Xiaoying J, Kanwal N. Armed conflict in the federally administered tribal areas of Pakistan and the role of NGOs in restoring health services. Soc Work Public Health. 2016;31(4):215–30.

    PubMed  Article  Google Scholar 

  106. 106.

    Giri A, Khatiwada P, Shrestha B, Chettri RK. Perceptions of government knowledge and control over contributions of aid organizations and INGOs to health in Nepal: a qualitative study. Glob Health. 2013;9:1.

    Article  Google Scholar 

  107. 107.

    Development Policy and Analysis Division (DPAD) UNS. World economic situation and prospects 2014: country classification. New York: United Nations Secretariat; 2014.

    Google Scholar 

  108. 108.

    Cancedda C, Davis SM, Dierberg KL, et al. Strengthening health systems while responding to a health crisis: lessons learned by a nongovernmental organization during the Ebola virus disease epidemic in Sierra Leone. J Infect Dis. 2016;214(Suppl. 3):S153–63.

    PubMed  PubMed Central  Article  Google Scholar 

  109. 109.

    Tynan A, Vallely A, Kelly A, et al. Vasectomy as a proxy: extrapolating health system lessons to male circumcision as an HIV prevention strategy in Papua New Guinea. BMC Health Serv Res. 2012;12:299.

    PubMed  PubMed Central  Article  Google Scholar 

  110. 110.

    Sodhi C, Singh P. Health service system in transition. Int J Health Govern. 2016;21(4):204–21.

    Article  Google Scholar 

  111. 111.

    Abdulmalik J, Fadahunsi W, Kola L, Nwefoh E, Minas H, Eaton J, Gureje O. The Mental Health Leadership and Advocacy Program (mhLAP): a pioneering response to the neglect of mental health in Anglophone West Africa. Int J Ment Health Syst. 2014;8:5.

    PubMed  PubMed Central  Article  Google Scholar 

  112. 112.

    Maleka EN. Monitoring and evaluation of sport-based HIV/AIDS awareness programmes: Strengthening outcome indicators. SAHARA J. 2017;14(1):1–21.

    PubMed  Article  Google Scholar 

  113. 113.

    Holden RJ, McDougald Scott AM, Hoonakker PL, Hundt AS, Carayon P. Data collection challenges in community settings: insights from two field studies of patients with chronic disease. Qual Life Res. 2015;24(5):1043–55.

    PubMed  Article  Google Scholar 

  114. 114.

    Puett C, Guerrero S. Barriers to access for severe acute malnutrition treatment services in Pakistan and Ethiopia: a comparative qualitative analysis. Public Health Nutr. 2015;18(10):1873–82.

    PubMed  Article  Google Scholar 

  115. 115.

    Steinhardt LC, Aman I, Pakzad I, Kumar B, Singh LP, Peters DH. Removing user fees for basic health services: a pilot study and national roll-out in Afghanistan. Health Policy Plan. 2011;26:ii92–ii103.

    PubMed  PubMed Central  Article  Google Scholar 

  116. 116.

    Martiniuk AL, Adunuri N, Negin J, Tracey P, Fontecha C, Caldwell P. Primary care provision by volunteer medical brigades in Honduras: a health record review of more than 2,500 patients over three years. Int J Health Serv. 2012;42(4):739–53.

    PubMed  Article  Google Scholar 

  117. 117.

    Verma R, Shekhar A, Khobragade S, et al. Scale-up and coverage of Avahan: a large-scale HIV-prevention programme among female sex workers and men who have sex with men in four Indian states. Sex Transm Infect. 2010;86:i76–82.

    PubMed  PubMed Central  Article  Google Scholar 

  118. 118.

    Ng-Kamstra JS, Riesel JN, Arya S, Weston B, Kreutzer T, Meara JG, Shrime MG. Surgical non-governmental organizations: global surgery's unknown nonprofit sector. World J Surg. 2016;40(8):1823–41.

    PubMed  Article  Google Scholar 

  119. 119.

    Xu T. Chinese Anti-Cancer Association as a non-governmental organization undertakes systematic cancer prevention work in China. Chin J Cancer Res. 2015;27:423–7.

    CAS  PubMed  PubMed Central  Google Scholar 

  120. 120.

    Martos AJ, Wilson PA, Meyer IH. Lesbian, gay, bisexual, and transgender (LGBT) health services in the United States: origins, evolution, and contemporary landscape. PLoS One. 2017;12(7):e0180544.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  121. 121.

    Kolk A, Lenfant F. Business–NGO collaboration in a conflict setting: partnership activities in the Democratic Republic of Congo. Business Soc. 2012;51(3):478–511.

    Article  Google Scholar 

  122. 122.

    Shah NK. Corporate philanthropy and conflicts of interest in public health: ExxonMobil, Equatorial Guinea, and malaria. J Public Health Policy. 2013;34(1):121–36.

    PubMed  Article  Google Scholar 

  123. 123.

    Ekirapa A, Mgomella GS, Kyobutungi C. Civil society organizations: capacity to address the needs of the urban poor in Nairobi. J Public Health Policy. 2012;33(4):404–22.

    PubMed  Article  Google Scholar 

  124. 124.

    Ingrosso L, Schmidt T, Sherally J, et al. A desk review on institutional and non-institutional organizations active in the field of migrant's health in the WHO European Region. Ann Ist Super Sanita. 2015;51(4):313–20.

    PubMed  Google Scholar 

  125. 125.

    Chu K, Rosseel P, Trelles M, Gielis P. Surgeons without borders: a brief history of surgery at Medecins Sans Frontieres. World J Surg. 2010;34(3):411–4.

    PubMed  Article  Google Scholar 

  126. 126.

    Humphries D, Gomez L, Hartwig K. Sustainability of NGO capacity building in southern Africa: successes and opportunities. Int J Health Plann Manage. 2011;26(2):e85–101.

    PubMed  Article  Google Scholar 

  127. 127.

    Bobrovitz N, Heneghan C, Onakpoya I, et al. Medications that reduce emergency hospital admissions: an overview of systematic reviews and prioritisation of treatments. BMC Med. 2018;16(1):115.

    PubMed  PubMed Central  Article  Google Scholar 

  128. 128.

    Asikhia OA, Mohangi K. A case study of school support and the psychological, emotional and behavioural consequences of HIV and AIDS on adolescents. SAHARA J. 2015;12:123–33.

    PubMed  Article  Google Scholar 

  129. 129.

    Berglas NF, Battistelli MF, Nicholson WK, Sobota M, Urman RD, Roberts SCM. The effect of facility characteristics on patient safety, patient experience, and service availability for procedures in non-hospital-affiliated outpatient settings: a systematic review. PLoS One. 2018;13(1):e0190975.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  130. 130.

    Biswas T, Islam SMS, Islam A. Prevention of hypertension in Bangladesh: a review. J Med. 2016;17(1):30–5.

    Article  Google Scholar 

  131. 131.

    Ibrahim U, Wan-Puteh SE. An overview of civil society organizations’ roles in health project sustainability in Bauchi State, Nigeria. Pan Afr Med J. 2018;30:150.

    PubMed  PubMed Central  Article  Google Scholar 

  132. 132.

    Scott K, George AS, Harvey SA, Mondal S, Patel G, Vr R, Sheikh K. Government helper and citizen advocate? A case study of the multiple roles and pressures facing a nongovernmental organization contracted by government to strengthen community health in northern India. Int J Health Plann Manage. 2018;33(2):391–404.

    PubMed  Article  Google Scholar 

  133. 133.

    Bach-Mortensen AM, Lange BCL, Montgomery P. Barriers and facilitators to implementing evidence-based interventions among third sector organisations: a systematic review. Implement Sci. 2018;13:103.

    PubMed  PubMed Central  Article  Google Scholar 

  134. 134.

    Bardach AE, Elorriaga N, Alcaraz AO, Rubinstein AL, Tavella JM. Community-based cardiovascular health promotion in Argentina. A systematic review. Health Promot Int. 2018;33(4):695–712.

    PubMed  Google Scholar 

  135. 135.

    Chen JS. Education as networking: rethinking the success of the harm reduction policy of Taiwan. Health. 2015;19(3):280–93.

    PubMed  Article  Google Scholar 

  136. 136.

    Chase LE, Marahatta K, Sidgel K, et al. Building back better? Taking stock of the post-earthquake mental health and psychosocial response in Nepal. Int J Ment Health Syst. 2018;12:44.

    PubMed  PubMed Central  Article  Google Scholar 

  137. 137.

    Castro R, Ribeiro-Alves M, Correa RG, et al. The men who have sex with men HIV care cascade in Rio de Janeiro, Brazil. PLoS One. 2016;11(6):e0157309.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  138. 138.

    Atkin K, Berghs M, Dyson S. ‘Who’s the guy in the room?’ Involving fathers in antenatal care screening for sickle cell disorders. Soc Sci Med. 2015;128:212–9.

    PubMed  Article  Google Scholar 

  139. 139.

    Mkoka DA, Kiwara A, Goicolea I, Hurtig AK. Governing the implementation of emergency obstetric care: experiences of rural district health managers, Tanzania. BMC Health Serv Res. 2014;14:333.

    PubMed  PubMed Central  Article  Google Scholar 

  140. 140.

    Huaynoca S, Chandra-Mouli V, Yaqub N Jr, Denno DM. Scaling up comprehensive sexuality education in Nigeria: from national policy to nationwide application. Sex Educ. 2014;14(2):191–209.

    Article  Google Scholar 

  141. 141.

    Furey A, Rourke J, Larsen H. Building the capacity to manage orthopaedic trauma after a catastrophe in a low-income country. J Orthop Trauma. 2015;29:S14–6.

    PubMed  Article  Google Scholar 

  142. 142.

    Sparrow A, Almilaji K, Tajaldin B, Teodoro N, Langton P. Cholera in the time of war: implications of weak surveillance in Syria for the WHO’s preparedness—a comparison of two monitoring systems. BMJ Global Health. 2016;1:e000029.

    PubMed  PubMed Central  Article  Google Scholar 

  143. 143.

    Sandoval C, Cáceres CF. Influence of health rights discourses and community organizing on equitable access to health: the case of HIV, tuberculosis and cancer in Peru. Global Health. 2013;9:23.

    PubMed  PubMed Central  Article  Google Scholar 

  144. 144.

    Basu S, Andrews J, Kishore S, Panjabi R, Stuckler D. Comparative performance of private and public healthcare systems in low- and middle-income countries: a systematic review. PLoS Med. 2012;9(6):e1001244.

    PubMed  PubMed Central  Article  Google Scholar 

  145. 145.

    United Nations. Sustainable development goals. Sustainable development knowledge platform. New York: United Nations Department of Public Information; 2019 https://sustainabledevelopment.un.org/sdgs. Accessed 18 July 2019.

  146. 146.

    Carter JG, Sherbon BJ, Viney IS. United Kingdom health research analyses and the benefits of shared data. Health Res Policy Syst. 2016;14:48.

    PubMed  PubMed Central  Article  Google Scholar 

  147. 147.

    Todrys KW, Amon JJ, Malembeka G, Clayton M. Imprisoned and imperiled: access to HIV and TB prevention and treatment, and denial of human rights, in Zambian prisons. J Int AIDS Soc. 2011;14:8.

    PubMed  PubMed Central  Article  Google Scholar 

  148. 148.

    Berti PR, Mildon A, Siekmans K, Main B, Macdonald C. An adequacy evaluation of a 10-year, four-country nutrition and health programme. Int J Epidemiol. 2010;39(2):613–29.

    PubMed  PubMed Central  Article  Google Scholar 

  149. 149.

    Gaber J. Seeing the community’s perspective through multiple emic and etic vistas. Health Promot Int. 2017;32(6):1025–33.

    PubMed  Google Scholar 

  150. 150.

    Penas Defago MA, Moran Faundes JM. Conservative litigation against sexual and reproductive health policies in Argentina. Reprod Health Matters. 2014;22(44):82–90.

    PubMed  Article  Google Scholar 

  151. 151.

    Berman J, Mitambo C, Matanje-Mwagomba B, et al. Building a knowledge translation platform in Malawi to support evidence-informed health policy. Health Res Policy Syst. 2015;13:73.

    PubMed  PubMed Central  Article  Google Scholar 

  152. 152.

    Round J, Jones L, Morris S. Estimating the cost of caring for people with cancer at the end of life: a modelling study. Palliative Med. 2015;29(10):899–907.

    Article  Google Scholar 

  153. 153.

    Wang EA, Hong CS, Samuels L, Shavit S, Sanders R, Kushel M. Transitions clinic: creating a community-based model of health care for recently released California prisoners. Public Health Rep. 2010;125(2):171–7.

    PubMed  PubMed Central  Article  Google Scholar 

  154. 154.

    Dunn J, Herron L, Adams C, Chambers S. Engaging NGOs in national cancer-control efforts. Lancet Oncol. 2013;14(11):1044–6.

    PubMed  Article  Google Scholar 

  155. 155.

    Franco JR, Cecchi G, Priotto G, Paone M, Diarra A, Grout L, Simarro PP, Zhao W, Argaw D. Monitoring the elimination of human African trypanosomiasis: Update to 2016. PLoS Negl Trop Dis. 2018;12(12):e0006890.

    PubMed  PubMed Central  Article  Google Scholar 

  156. 156.

    Floyd K, Fitzpatrick C, Pantoja A, Raviglione M. Domestic and donor financing for tuberculosis care and control in low-income and middle-income countries: an analysis of trends, 2002–11, and requirements to meet 2015 targets. Lancet Global Health. 2013;1(2):PE105–E15.

    Article  Google Scholar 

  157. 157.

    Alam K, Ahmed S. Cost recovery of NGO primary health care facilities: a case study in Bangladesh. Cost Eff Resour Alloc. 2010;8:12.

    PubMed  PubMed Central  Article  Google Scholar 

  158. 158.

    Hall MA. Access to care provided by better safety net systems for the uninsured: measuring and conceptualizing adequacy. Med Care Res Rev. 2011;68(4):441–61.

    PubMed  Article  Google Scholar 

  159. 159.

    Fiorini G, Cerri C, Bini S, Rigamonti AE, Perlini S, Marazzi N, Sartorio A, Cella SG. The burden of chronic noncommunicable diseases in undocumented migrants: a 1-year survey of drugs dispensation by a non-governmental organization in Italy. Public Health. 2016;141:26–31.

    CAS  PubMed  Article  Google Scholar 

  160. 160.

    Gomes D, Stavropoulou C. The impact generated by publicly and charity-funded research in the United Kingdom: a systematic literature review. Health Res Policy Syst. 2019;17:22.

    PubMed  PubMed Central  Article  Google Scholar 

  161. 161.

    Bonevski B, Randell M, Paul C, Chapman K, Twyman L, Bryant J, Brozek I, Hughes C. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol. 2014;14:42.

    PubMed  PubMed Central  Article  Google Scholar 

  162. 162.

    Biswas T, Haider MM, Das Gupta R, Uddin J. Assessing the readiness of health facilities for diabetes and cardiovascular services in Bangladesh: a cross-sectional survey. BMJ Open. 2018;8(10):e022817.

    PubMed  PubMed Central  Article  Google Scholar 

  163. 163.

    Webb P, Bain C, Page A. Essential epidemiology: an introduction for students and health professionals. 3rd ed. Cambridge: Cambridge University Press; 2016.

    Google Scholar 

  164. 164.

    Kelly CM, LaRose J, Scharff DP. A method for building evaluation competency among community-based organizations. Health Promot Pract. 2014;15(3):431–7.

    PubMed  Article  Google Scholar 

  165. 165.

    Sweetland AC, Oquendo MA, Sidat M, et al. Closing the mental health gap in low-income settings by building research capacity: perspectives from Mozambique. Ann Glob Health. 2014;80(2):126–33.

    PubMed  PubMed Central  Article  Google Scholar 

  166. 166.

    Patel V. Mental health in low- and middle-income countries. Br Med Bull. 2007;81–82:81–96.

    PubMed  Article  Google Scholar 

  167. 167.

    Reeve J, Cooper L, Harrington S, Rosbottom P, Watkins J. Developing, delivering and evaluating primary mental health care: the co-production of a new complex intervention. BMC Health Serv Res. 2016;16:470.

    PubMed  PubMed Central  Article  Google Scholar 

  168. 168.

    Rathod S, Pinninti N, Irfan M, Gorczynski P, Rathod P, Gega L, Naeem F. Mental health service provision in low- and middle-income countries. Health Serv Insights. 2017;10:1178632917694350.

    PubMed  PubMed Central  Google Scholar 

  169. 169.

    Kelley PW. Global health: governance and policy development. Infect Dis Clin North Am. 2011;25(2):435–53.

    PubMed  PubMed Central  Article  Google Scholar 

  170. 170.

    Hurtado C, Meyer D, Snydera M, Nuzzoa JB. Evaluating the frequency of operational research conducted during the 2014–2016 West Africa Ebola epidemic. Int J Infect Dis. 2018;77:29–33.

    PubMed  Article  Google Scholar 

  171. 171.

    Clark N, Hall A, Sulaiman R, Naik G. Research as capacity building: the case of an NGO facilitated post-harvest innovation system for the Himalayan hills. World Dev. 2003;31(11):1845–63.

    Article  Google Scholar 

  172. 172.

    Parisi J. Capacity building in nonprofit organisations in the development aid sector: explanatory research of capacity building in Indonesia in 2008 and an investigation into the diffusion of capacity building techniques between sectors. Lismore: Southern Cross University; 2009.

    Google Scholar 

  173. 173.

    Cornish H, Fransman J, Newman K, on behalf of the Rethinking Research Partnerships collaboration. Rethinking research partnerships: discussion guide and toolkit. London: Christian Aid, Economic & Social Research Council, The Open University; 2017.

    Google Scholar 

  174. 174.

    United Nations Development Programme. Human development report 2002: deepening democracy in a fragmented world. New York: United Nations; 2002.

    Google Scholar 

  175. 175.

    Nurse K, Wight D. Development assistance and research capacity strengthening: the commissioning of health research in East Africa. J East Afr Stud. 2011;5(2):233–51.

    Article  Google Scholar 

  176. 176.

    Council on Health Research for Development (COHRED) Global Forum for Health Research. Beyond aid. Research and innovation as key drivers for health, equity and development. Geneva: COHRED; 2012.

    Google Scholar 

  177. 177.

    World Health Organization. Minimum Data Set (MDS). Geneva: WHO; 2014. https://www.who.int/healthinfo/survey/ageingmds/en/. Accessed 1 Apr 2020.

    Google Scholar 

  178. 178.

    Kanyegere PBN. The ‘NGO-ization’ of research: what are the risks? From poverty to power. London: Oxfam; 2020. Accessed 1 Apr 2020.

    Google Scholar 

  179. 179.

    Office of Behavioural and Social Sciences Research, Department of Health and Human Services, National Institutes of Health. Ethical challenges of research. Bethesda: National Institutes of Health; 2011. https://obssr.od.nih.gov/wp-content/uploads/2016/05/Ethical-Challenges.pdf. Accessed 19 July 2019.

    Google Scholar 

  180. 180.

    Haggerty KD. Ethics creep: governing social science research in the name of ethics. Qual Sociol. 2004;27(4):391–414.

    Article  Google Scholar 

  181. 181.

    Perkins NI. Making space for science in NGO practice: SciDevNet; 2013. http://www.scidev.net/index.cfm?originalUrl=/global/editorials/making-space-for-science-in-ngo-practice.html. Accessed 20 Feb 2020.

  182. 182.

    Mazet JAK, Uhart MM, Keyyu JD. Stakeholders in One Health. Rev Sci Tech. 2014;33(2):443–52.

    CAS  PubMed  Article  Google Scholar 

  183. 183.

    Hobday M. Cancer charities need access to patient data too. BMJ. 2013;347:f5864.

    PubMed  Article  Google Scholar 

  184. 184.

    Morales LG, Orrell T. Data interoperability: a practitioner’s guide to joining up data in the development sector. Collaborative on SDG Data Interoperability; 2018. http://www.data4sdgs.org/resources/interoperability-practitioners-guide-joining-data-development-sector. Accessed 4 June 2020.

  185. 185.

    World Vision International. Humanitarian agencies to donors: “help us ensure the best data protection for beneficiaries”. Monrovia: World Vision; 2017. https://www.wvi.org/pressrelease/humanitarian-agencies-donors-%E2%80%9Chelp-us-ensure-best-data-protection-beneficiaries%E2%80%9D-0. Accessed 19 July 2019.

    Google Scholar 

  186. 186.

    Olafsson G. Dealing with disasters. Seattle: Blogger.com; 2013. Accessed 19 July 2019.

    Google Scholar 

  187. 187.

    Collaborative on SDG data interoperability, Global Partnership for Sustainable Development Data. Using data to join up development efforts. New York: United Nations; 2018.

    Google Scholar 

  188. 188.

    Ui S, Heng L, Yatsuya H, Kawaguichi L, Akashi H, Aoyana A. Strengthening community participation at health centers in rural Cambodia: role of local non-governmental organizations (NGOs). Crit Public Health. 2010;20(1):97–115.

    Article  Google Scholar 

  189. 189.

    Bennett S, Corluka A, Doherty J, et al. Influencing policy change: the experience of health think tanks in low- and middle-income countries. Health Policy Plan. 2012;27(3):194–203.

    PubMed  Article  Google Scholar 

  190. 190.

    Mprah WK, Anafi P, Sekyere FO. Does disability matter? Disability in sexual and reproductive health policies and research in Ghana. Int Q Community Health Educ. 2014;35(1):21–35.

    PubMed  Article  Google Scholar 

  191. 191.

    Global Health Watch. Global Health Watch 4. London: ZedBooks; 2014.

    Google Scholar 

  192. 192.

    Kilic B, Kalaca S, Unal B, Phillimore P, Zaman S. Health policy analysis for prevention and control of cardiovascular diseases and diabetes mellitus in Turkey. Int J Public Health. 2015;60:S47–53.

    PubMed  Article  Google Scholar 

  193. 193.

    Iliffe S, Davies SL, Gordon AL, et al. Provision of NHS generalist and specialist services to care homes in England: review of surveys. Prim Health Care Res Dev. 2016;17(2):122–37.

    PubMed  Article  Google Scholar 

  194. 194.

    Bemelmans M, Baert S, Negussie E, et al. Sustaining the future of HIV counselling to reach 90–90-90: a regional country analysis. J Int AIDS Soc. 2016;19(1):20751.

    PubMed  PubMed Central  Article  Google Scholar 

  195. 195.

    Kraak VI, Harrigan PB, Lawrence M, Harrison PJ, Jackson MA, Swinburn B. Balancing the benefits and risks of public-private partnerships to address the global double burden of malnutrition. Public Health Nutr. 2012;15(3):503–17.

    PubMed  Article  Google Scholar 

  196. 196.

    El Arifeen S, Christou A, Reichenbach L, Osman FA, Azad K, Islam KS, Ahmed F, Perry HB, Peters DH. Community-based approaches and partnerships: innovations in health-service delivery in Bangladesh. Lancet. 2013;382(9909):P2012–26.

    Article  Google Scholar 

  197. 197.

    Okma KG, Kay A, Hockenberry S, Liu J, Watkins S. The changing role of health-oriented international organizations and nongovernmental organizations. Int J Health Plann Manage. 2016;31(4):488–510.

    PubMed  Article  Google Scholar 

  198. 198.

    University College London Great Ormond Street Institute of Child Health. Search tools for grey literature. London: UCL; 2019. https://www.ucl.ac.uk/child-health/support-services/library/resources-z/search-tools-grey-literature. Accessed 14 Oct 2019.

    Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This research was funded as part of a Global Challenges Research Fund Institutional Impact Acceleration Award from the Economic and Social Research Council to the University of York (ES/S501499/1), held at the Interdisciplinary Global Development Centre.

Author information

Affiliations

Authors

Contributions

JG, HA and PW conceived the study. AM and MB developed and performed the literature search and performed the initial screening stage. SM performed the subsequent screening stages and extracted, analysed, and interpreted the data retrieved from the included studies. SM drafted the manuscript. All other authors contributed to the editing of the manuscript. All authors read and approved the final manuscript.

Authors’ information

Not applicable.

Corresponding author

Correspondence to Sarah C. Masefield.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Masefield, S.C., Megaw, A., Barlow, M. et al. Repurposing NGO data for better research outcomes: a scoping review of the use and secondary analysis of NGO data in health policy and systems research. Health Res Policy Sys 18, 63 (2020). https://doi.org/10.1186/s12961-020-00577-x

Download citation

Keywords

  • Non-government organisations
  • Health policy and systems research
  • Developing countries
  • Secondary data analysis
  • Marginalised groups
  • Sustainable development goals