A decade of investments in monitoring the HIV epidemic: how far have we come? A descriptive analysis

Background The 2001 Declaration of Commitment (DoC) adopted by the General Assembly Special Session on HIV/AIDS (UNGASS) included a call to monitor national responses to the HIV epidemic. Since the DoC, efforts and investments have been made globally to strengthen countries’ HIV monitoring and evaluation (M&E) capacity. This analysis aims to quantify HIV M&E investments, commitments, capacity, and performance during the last decade in order to assess the success and challenges of national and global HIV M&E systems. Methods M&E spending and performance was assessed using data from UNGASS country progress reports. The National Composite Policy Index (NCPI) was used to measure government commitment, government engagement, partner/civil society engagement, and data generation, as well as to generate a composite HIV M&E System Capacity Index (MESCI) score. Analyses were restricted to low and middle income countries (LMICs) who submitted NCPI reports in 2006, 2008, and 2010 (n =78). Results Government commitment to HIV M&E increased considerably between 2006 and 2008 but decreased between 2008 and 2010. The percentage of total AIDS spending allocated to HIV M&E increased from 1.1% to 1.4%, between 2007 and 2010, in high-burden LMICs. Partner/civil society engagement and data generation capacity improved between 2006 and 2010 in the high-burden countries. The HIV MESCI increased from 2006 to 2008 in high-burden countries (78% to 94%), as well as in other LMICs (70% to 77%), and remained relatively stable in 2010 (91% in high-burden countries, 79% in other LMICs). Among high-burden countries, M&E system performance increased from 52% in 2006 to 89% in 2010. Conclusions The last decade has seen increased commitments and spending on HIV M&E, as well as improved M&E capacity and more available data on the HIV epidemic in both high-burden and other LMICs. However, challenges remain in the global M&E of the AIDS epidemic as we approach the 2015 Millennium Development Goal targets. Electronic supplementary material The online version of this article (doi:10.1186/1478-4505-12-62) contains supplementary material, which is available to authorized users.


Background
Since the 2001 United Nations General Assembly Special Session (UNGASS) Declaration of Commitment on HIV/ AIDS [1], countries and the international community have channeled unprecedented political and financial support to the fight against AIDS. Investments increased eight-fold during the 10 years that followed the Declaration to over $16.8 billion in 2011 and $18.9 billion in 2012 [2]. There has been a parallel increase in focus on accountability to ensure that this funding is spent responsibly and to enable programmatic improvement, policy change, and a greater understanding of the evolving nature of the AIDS epidemic. This is dependent on high quality, representative data, which is in large part gathered from national monitoring and evaluation (M&E) efforts. These data, and the related data systems, are essential to enable countries to have greater ownership of their response by allowing them to plan for, collect, analyze, disseminate, and use HIV-related data to inform the national response [3]. Strengthening national M&E system capacity is therefore a critical component of combating the AIDS epidemic. Country-owned HIV M&E systems without duplication are supported by the "Three Ones" principles [4], which sought to create a nationally unified data system to support the information needs of a dramatically escalating HIV epidemic.
Creating, maintaining, and improving national health information systems requires sustained investment of political, human, and financial resources [5]. General guidance states that 5% to 10% of the national AIDS budget should be used for HIV M&E activities [6]. There has been substantial investment in HIV M&E systems in the last 10 years [7], as well as several efforts to define the basic components of a functional HIV M&E system [3,8]. However, to our knowledge, there has not been a quantitative assessment of M&E investments and national HIV M&E system capacity on a global scale.
The goal of this paper is to quantify HIV M&E investments, commitments, capacity, and performance in the decade following the Declaration of Commitment to inform the current global discussion on expanding access and use of development data, and ensure increased support for accountability systems in the context of the post-2015 agenda.

Data source
Data on M&E spending, systems, and performance were derived from UNGASS country reports submitted in 2006 [9], 2008 [10], and 2010 [11]. Country Progress Reports, including a core set of indicators, have been submitted to the Joint United Nations Programme on HIV/ AIDS (UNAIDS) every second year since 2004 as part of the UNGASS reporting system [11]. Information on government commitment, government engagement, partner/civil society engagement, data generation, and data dissemination/utilization at the national level was taken from the National Composite Policy Index (NCPI) questionnaire [11], which is an integral part of the UNGASS process and the most routinely collected data globally on M&E system capacity. The NCPI is a two-part questionnaire, the first section of which solicits government feedback on the HIV/AIDS strategic plan, political support, prevention, treatment, care and support, and M&E. The second section is administered to civil society organizations and bilateral and multilateral agencies, and assesses human rights, civil society involvement, prevention and treatment, and care and support. Country response rates over the 2006, 2008, and 2010 reporting cycles ranged from 50% to 90% (Table 1).

Sample
These analyses were restricted to low-and middle-income countries (LMICs) who submitted NCPI questionnaires in each of the 2006, 2008, and 2010 reporting cycles (n =78) in order to assess trends over time. The 2004 NCPI reporting was excluded as it did not include a M&E section. expenditures, HIV M&E system capacity, and HIV M&E system performance.

Government commitments
Government commitment was measured using three questions in the NCPI, outlined in Additional file 1. The scores for each question were averaged to create a domain score for each country, as explained further below and detailed in Additional file 1. There were two questions that could not be included in the domain score as they were not asked in all three reporting cycles, but are displayed in Table 2.

HIV M&E expenditures
HIV M&E expenditures were drawn from UNGASS Indicator 1 which measures levels of HIV/AIDS spending and is collected through a standard reporting form [11]. This national funding matrix is a two-dimensional matrix showing how funds are spent at the national level and where those funds are sourced. This is accomplished by relating financing sources (funding entities that disburse money to agents) with AIDS spending categories (goods, services, and activities delivered as part of the HIV response). Domestic spending can be classified in the National Funding Matrix under 89 specific spending categories. The spending categories follow a functional classification described in the National AIDS Spending classifications and definitions [12,13]. We analyzed the spending trends and funding sources of M&E expenditures which are categorized under AIDS spending category 04.03 "Monitoring and evaluation" [12]. In addition, analyses included a broader sub-set of spending categories which we considered important components of an adequate M&E LMICs with reported data for at least two of these four years was conducted. Spending data prior to 2007 was not available. Comparable data was used to reveal the trend of the share of M&E and M&E + out of total spending on HIV/AIDS. When necessary, the quantity of data points was broadened by conducting a linear interpolation between two known points in the dataset.
HIV M&E system capacity HIV M&E system capacity was calculated as a composite of three domains: government engagement, partner/civil society engagement, and data generation. All NCPI questions related to M&E from both government and civil society sections administered as part of the 2006, 2008, and 2010 UNGASS reporting cycle were reviewed. Questions that remained consistent over the three cycles and were relevant to HIV M&E system capacity were grouped into three domains: government engagement, partner and civil society engagement, and data generation. Percentage scores for each question were calculated using the scoring rubric outlined in Additional file 2.

Index construction and analysis
The three domains, (i) government engagement, (ii) partner and civil society engagement, and (iii) data generation, were also aggregated to form an index score, the HIV M&E System Capacity Index (MESCI) (see Additional file 2 for individual components and scoring calculations). While information on data utilization was collected through the NCPI, only one question was asked in all three reporting cycles, and is therefore not included in domain or index calculations.
To calculate the MESCI, an arithmetic mean was generated for each of the three domains in the MESCI using the scoring methodology outlined in Additional file 2. Means included only questions that were asked in 2006, 2008, and 2010 NCPI cycles. The three domain scores were then averaged to generate the overall HIV M&E System Capacity Index, which ranges from 0% to 100%, for each country and each time-point. MESCI component, domain, and overall scores, including 95% confidence intervals, are presented for each round of UNGASS reporting, stratified by high-burden countries vs. other LMICs. Countries that did not answer a particular question were given a missing value for that question, rather than imputing a response.

HIV M&E performance
UNGASS indicator reporting from 2006 to 2010 was used to represent HIV M&E system performance. Reported UNGASS indicators were analyzed to assess levels of country reporting for three data streams, namely indicators measured from nationally representative, populationbased surveys (UNGASS indicator numbers 7, 13 15, 16, 17), behavioral surveillance for key populations at higher risk of HIV, i.e., men who have sex with men, sex workers, and people who inject drugs (UNGASS indicator numbers 8,14,18,19,20,21,23), and service delivery metrics were derived from program data (UNGASS indicator numbers 4,5,6,24) (Additional file 3) [9][10][11]. Countries reporting any of the indicators in each data stream were counted as having reported in that particular data stream. The scores for each of the three streams were averaged to create an HIV M&E performance score. We also examined the number of countries able to report program data on antiretroviral therapy by sex; this metric was not available in 2008 and is therefore not included in the performance score calculations.
All analyses were conducted using SAS v. 9.2.

Data generation
Overall, high-burden countries had very high scores in the data generation domain, with improvements from 2006 (81%) to 2008 (97%) that were maintained into 2010 (Table 4). Other LMICs did not report as many improvements over time, and in fact showed decreases between 2008 and 2010 in the number of countries reporting that their national HIV M&E plan included i) guidelines on tools for data collection (88% to 85%), ii) a strategy for assessing data quality (71% to 59%), iii) a data dissemination and use strategy (84% to 79%) and iv) the presence of a functional national and subnational HIS (91% to 85% and 80% to 78%, respectively).

Data utilization
While no domain score was calculated for data utilization, the percentage of countries producing annual HIV M&E reports increased steadily from 2006 to 2010 across all LMICs (Table 4).    Indicates the number of countries with a response for that question. 2 Excluded from calculation of domain score, as question was not asked in all three reporting cycles. 3 No domain score calculated, as only one question was asked in all three survey rounds.

HIV M&E system capacity index
Overall, the index improved between 2006 and 2008, though this improvement was greater in high-burden LMICs (a significant shift from 78% to 94%) than in other LMICs (70% to 77%) ( Table 4). High-burden LMICs saw a slight decrease between 2008 and 2010. Among high burden LMICs, the trajectories of government commitment and HIV M&E system capacity map closely onto one another, while in other LMICs, government commitment decreases from 2008-2010 as HIV M&E system capacity increases (Figure 1).

HIV M&E system performance
M&E system performance generally shows improvements over time among all LMICs regardless of HIV burden (Table 5). For high-burden countries, the overall progress summary score significantly increased from 52% in 2006 to 89% in 2010 and from 37% to 89% for other LMICs. In 2006, only half of the high-burden countries and just one in five of the other LMICs could report on indicators related to the population based indicators. By 2010, this had significantly increased to 95% of high-burden and 84% of other LMICs. The same trend is also seen for indicators related to key populations, with increases from 23% to 73% for highburden countries and from 50% to 82% for other LMICs, as well as for the service delivery indicators (from 82% to 100% for high-burden and 41% to 100% for other LMICs). Among high burden countries, both MESCI and HIV M&E performance increase from 2006-2008, and level or slightly decrease by 2010 (Figure 2). In contrast, among other LMICs, HIV M&E performance increases much faster than the MESCI between 2006-2008 ( Figure 2).

Discussion
The rapid expansion of available resources to fight the global AIDS epidemic since the new millennium has brought with it commensurate expectations for accountability at subnational, national, and international levels. Our analysis shows that government commitment to HIV M&E, particularly among high-burden LMICS, has increased considerably since 2006, as has the percentage of AIDS spending allocated to HIV M&E. As government commitments and spending have increased, so have government and partner engagement, data use, and data generation. Over this same time period, HIV M&E system performance has increased by over 70% in highburden LMICs, and by 140% in other LMICs.
Reported HIV M&E expenditures were low across the board when judged against the 5% to 10% of total AIDS spending as a recommended benchmark [6], never being higher than 1.4% in high-burden countries. While it is likely that there were additional sources of funding for HIV M&E systems beyond those reported in UNGASS reporting, it seems unlikely that they would reach recommended levels. However, the suggested level may also need to be reconsidered at this time as evidenced by the improved performance of the M&E systems at current funding levels.
Many countries still lack the systems to actually track spending on M&E, a barrier that is not confined to the AIDS response. Recognizing this challenge, overall support to national statistical services from international agencies and donor countries has grown considerably, from $US1.0 billion in 2006 to $US2.3 billion in 2010-2012 [14].
The relationship seen between the MESCI score and HIV M&E systems performance ( Figure 2) indicates that among high-burden LMICs, countries with high capacity   should be better able to produce HIV M&E data to meet national and international needs. This relationship does not seem to hold true for other LMICs, where system performance is higher than system capacity. One potential explanation is that countries with lower burdens of HIV are able to monitor their epidemic with a lower system capacity than that required for high-burden countries.
The mandate for national and international HIV M&E provided in the 2011 Political Declaration on HIV/AIDS [15], to further strengthen and support HIV data systems while working to integrate them with broader national health systems, is a significant shift from the previous strategy of often parallel systems. There has also been a call to use HIV resources more strategically, to encourage greater efficiencies and higher performance [16]. The decrease seen in the number of high-burden countries reporting a National HIV M&E Committee or Working Group that meets regularly may be of concern, as that is one of the main bodies that will be able to guide the integration agenda that has been set for HIV M&E. However, it may also be the evolution of M&E systems from the early and intensive development stages to the current and more established systems maintenance stage. Folding the resources and experience from the HIV response in with those of other health sectors offers the opportunity for synergies that could be used to leverage the maximum benefit out of all systems and resources. Neither the NCPI M&E section nor the MESCI as they are currently structured are able to measure integration between HIV M&E systems and other national M&E systems, but this is an important modification that should be considered in future years.
Since the beginning of the AIDS epidemic, civil society has played a key role in the response in countries all over the world. The engagement in the monitoring of the AIDS epidemic has also been shown to be important [17]. Civil society engagement in the development of M&E plans was reported to be high already in 2008 (around 80% of countries) and increased to more than 90% of countries in 2010. However, the perceived extent of civil society inclusion was lower, at only 60% in highburden countries and 50% in other LMICs, leaving room for considerable improvements.
This study has several key limitations. Data on government commitments and system capacity is derived from the NCPI, which is self-reported data representing both national government representative's opinions of the status and performance of their HIV M&E systems and civil society/bilateral/multilateral opinion on perceived inclusion in HIV M&E activities. The nature of response to some of these questions is therefore dependent to some extent on the respondent and their personal measures of performance. Calculations of domain scores and the MESCI were limited to questions that appeared in all three reporting cycles analyzedas NCPI reporting continues, the contents of each domain may be enhanced by a greater breadth of questions asked over time. Indicator performance and patterns of reporting over time was used as a proxy for M&E system performance in countries. Although there are limitations in this approach, consistent increases in reported data, from varying sources and data collection systems (program data, population-based surveys, behavioral surveillance), indicate that there has been progress in strengthening national M&E systems in recent years.
The most important reason for conducting M&E is to provide the data needed for guiding policy and program implementation. A functional M&E system collates and presents the data in a way that facilitates data use at all levels, including the general public and beneficiaries of HIV services [3]. Countries rated their data utilization for the development/revising of national AIDS strategies, resource allocation and program improvement in 2010 above average in most countries, though there is room for improvement. Accurate and regular data utilization is critical to ensuring epidemiologically appropriate and evidence-based resource allocation; gaps in this process may be in part responsible for the variable resource allocation strategies currently seen across countries [18].
Through the UNGASS reporting mechanism, countries were asked to submit nationally representative data. However, for an effective response it is not sufficient to collect data at the national level but data is also needed at the subnational level, e.g., to identify geographic areas where localized HIV epidemics or specific populations most affected by the epidemic are not being reached by services. In recent years, more focus has been put on these efforts [19].
The gathering of reliable data on the AIDS epidemic and the response to it is of increasing importance for donors such as the Global Fund to Fight AIDS, Tuberculosis, and Malaria, for whom performance-based funding is a core principle [20]. M&E is a cornerstone of this approach and allows donors to make funding decisions against proven measurable results. More recently, the Global Fund to Fight AIDS, Tuberculosis, and Malaria's New Funding Model has put greater emphasis on impact measurement and allocative efficiency [21]. Further examples include the Investment Framework at a national level [16] and the PEPFAR reporting requirements at a donor level [22].

Conclusions
This analysis demonstrates increases in all domains of HIV M&E systems assessed between 2006 and 2010, as well as in HIV M&E funding, though the latter remained well below the recommended benchmarks. In this same