The findings below are structured according to (1) conceptualisation; (2) uptake or implementation; and (3) further policy development, once a policy has been implemented. We then examine what respondents told us about the mechanisms of learning, which operate at international, regional and national levels. This follows the themes that emerged inductively from the interviews conducted and reflects not just findings on the specific tracer policies but also respondent’s wider comments on learning and evidence use. Finally, we present cross-cutting themes in relation to facilitators and barriers to learning, which are grouped into factors focussed on the demand for and supply of evidence.
Conceptualisation
All of the reforms either started from or were accompanied by a local recognition of a problem. In relation to the origin of the policies, looking across the eight contexts, five broad models emerged, ranging from least to most home-grown, as follows:
-
1.
In the case of the initial phase of the Integrated Management of Child Illness programme in Nepal, the country was adopting a specific international package, which was more or less standard practice across most countries.
-
2.
In three cases (the Sector-Wide Approach in Bangladesh, health financing reforms in Georgia, and health financing reforms in Burkina Faso), the broad idea behind the policies was initially promoted by major international agencies, but was more actively adopted in the sense of being seen to meet a local need and fit with local contexts.
-
3.
In three cases (Health Equity Funds and contracting in Cambodia, CBHI and performance-based financing in Rwanda, and community clinics in Bangladesh), the policies emerged from a partnership of development partners and government, with ideas being introduced from other contexts but being incubated and developed in substantive ways in-country. Later iterations of Nepal’s Integrated Management of Child Illness followed this path too, through the shift to community-based delivery and the introduction of the package of newborn care.
-
4.
In one case (Role Delineation Policy in Solomon Islands), the idea was developed locally as a means of achieving more equitable, but affordable, health services after a period of ethnic tension. The approach drew on some regional inspiration and technical support from bilateral and multilateral partners.
-
5.
Finally, in the Ethiopian HEP, there was no significant external input, though the policy was influenced in cross-sectoral learning internally from agricultural extension workers in Ethiopia within one state, and later scaled up.
These points illustrate how countries adopted international ideas, but the case studies that were undertaken uncovered many situations in which evidence was not sought, or was altogether ignored. Non-adoption of international ideas and the rejection of advice from other countries had varying consequences. In Ethiopia, the international consensus was antagonistic to community health workers in the late 1990s, when the HEP programme was being developed in Tigray. The government continued to support it, however, as it seemed one of the few feasible ways to reach a dispersed rural population in a context of limited resources and infrastructure. The decision is widely seen to have paid off. Similarly, Cambodia has resisted adopting a clear purchaser-provider split for Special Operating Agencies, despite some international encouragement to do so. Nepal has resisted a number of WHO-recommended adjustments to clinical guidelines, on the basis that they are not in line with wider health system strategy or capacity. Georgia pursued hospital privatisation in the face of cautionary international advice and the legacy of that has been much more mixed.
Uptake
It is clear from the case studies that the drivers of uptake, or moving ahead with implementation of a policy, are rooted firmly in the local political economy. In the case of Ethiopia, for example, the drivers were historical as well as ideological (the government having recently been engaged in grassroots mobilisation during civil war), combined with political imperatives (the need to deliver basic services to a large, poor population as a new regime) and pragmatism (other options were not feasible with the resources available). Ideological influences, industry lobbying and the powerful role of international agencies (such as the World Bank, during the period of reforms in transitional economies in the late 1990s) are also documented in Georgia, for instance.
Published, peer-reviewed evidence was rarely mentioned as the impetus or main source of information for policy development in the case studies. It was most likely to be consulted for review of clinical protocols, as this is an area in which local contextualisation is regarded as less critical. The influence of published studies is also seen to occur through their dissemination from international agencies such as in the influence of international researchers on healthcare in Burkina Faso and the research on sector-wide approaches that was incorporated by proposals from donor partners in Bangladesh. That said, local evidence being published in an international peer-reviewed journal was said to give it credibility and feed a sense of pride, with both increasing the likelihood of it being acted on.
Robust evidence may be lacking for a policy (like community clinics, in the case of Bangladesh), but if the concept fits well into the socio-political context and enjoys political patronage, then reforms will still be undertaken. The cases of Cambodia and Georgia, where senior politicians made executive policy decisions that were not exactly aligned with the evidence presented, also highlight how governments can set the parameters for when they will or will not over-ride evidence, and how the choice and application of evidence is often ‘purpose-driven’ and predefined by political agendas. In Cambodia, early evidence suggested that contracting services out (to non-government organisations) achieved positive results. The government has been concerned about the sustainability of this option, and adopted a contracting in approach instead. This is an example of some policy options being beyond consideration, even if the evidence may have appeared to be in their favour. This is in contrast to some evidence-informed modifications that have been made by the same government to the operationalisation of Health Equity Funds (though here again, political constraints apply).
Drivers of policy development (once adopted)
The case studies suggest that internal learning is the key to successful policy development over time. Further, capacities, skills and culture that support good examples in this respect are likely to be linked to the ability to filter experiences from other contexts intelligently.
The case studies illustrated the effective use of annual reviews to assess and improve policy performance (for example, in Rwanda and Ethiopia), adjustment of policies based on local evidence (in Cambodia, Nepal and Rwanda), using national and international routine data sources for monitoring (for example, in Georgia, which used regional comparators for benchmarking), use of evidence from operational research (in Cambodia), and technical assistance to identify the existing – and possible future – cost structures and affordability of interventions (Solomon Islands). Countries like Rwanda, Nepal, Cambodia and Ethiopia were also effective at sharing lessons across sites internally.
By contrast, the role of policy evaluation was much more contested. In some settings, like Bangladesh and Ethiopia, there was resistance to formally evaluating high-priority national programmes, while in others, like Nepal, there were reported tussles over the ownership of the evaluation process. While some countries (e.g. Cambodia) used evaluations actively as a means of lesson-learning and mid-course corrections, many of the apparently successful policies were never formally evaluated, reflecting the higher stakes and more politicised nature of evaluative processes, compared to continuous learning through observation of a policy’s outcomes over time.
Mechanisms for learning
A wide range of mechanisms that had supported learning processes within and across countries were mentioned by key informants. These are outlined in rough order of frequency, starting with the international ones.
International study tours were the most commonly mentioned mechanism for international learning, used across all eight sites, typically early on in the policy development process and including a variety of constituencies (technical, parliamentary, etc.). These are typically facilitated by development partners and were seen as important, although suggestions for improving their effectiveness (such as better follow-up) were also made.
Country decision-makers and technical staff also use direct relationships with development partner staff to gain advice on topics of interest at all policy stages. Development partners facilitate access to and share ideas and evidence in all settings. Some organisations are widely influential, for example, WHO. Others are seen as offering specific expertise (for example, the World Bank on health financing or International Labour Organization for social protection), though bilateral and multilateral funding agencies are also seen as having their own agendas. Personal relationships with development partner staff are highly important, especially when their presence in-country is long-term, or the country has a small population.
Attending international meetings on specific topics of relevance was also highlighted as influential in five settings (Georgia, Nepal, Solomon Islands, Rwanda and Burkina Faso), particularly regional meetings that focused on a specific, shared problem.
Technical assistance programmes were perceived to be of particular importance in learning about reforms in other countries and in supporting implementation in Bangladesh, Cambodia, Georgia, Solomon Islands, Rwanda and Burkina Faso.
Many countries shared ideas and evidence internally and with external stakeholders such as development partners through routine health system governance structures, such as coordination and technical working groups (highlighted in Cambodia, Georgia, Nepal, Rwanda and Burkina Faso). In some instances, countries systematically established groups to review international published evidence to refine specific health packages (Nepal and Ethiopia).
Capacity-building through formal training or on the job experience also played a role, with countries tending to initially train abroad but gradually develop local capacity and institutions (for example, in Rwanda and Cambodia), also in order to better retain trained staff.
Regional networks also played a role, though these were less frequently mentioned. In the Solomon Islands, regional professional networks may have facilitated idea transfer, including through contractors working across countries, and regional training networks were highlighted as significant. In (former-)francophone African countries (Rwanda and Burkina Faso), influential individual consultants working across countries and community of practice networks were cited as having contributed to the spread of ideas, including through their reports. Burkina Faso was the only context where civil society – in the form of advocacy groups, working with international partners – was cited as having influenced policy uptake.
Within countries, pilot projects supported by international non-governmental organisations played an important role in developing some of the policies (in Rwanda, Cambodia, Nepal and Burkina Faso). Some countries also used domestic study tours and meetings to exchange learning across regions within their country (e.g. Ethiopia).
It was also encouraging that some countries have started to focus on how to share lessons from their own experiences and becoming ‘centres of excellence’ in particular areas, such as Rwanda, which has set up institutions to share lessons on performance-based financing (amongst others), and Ethiopia, which has established an international institute for training and research on rural primary healthcare.
Facilitators
Facilitators of learning were grouped into those which predominantly affect the demand for evidence, those which are more linked to evidence supply, and finally some which are related to the evidence topic and its presentation.
In relation to demand, having a performance-oriented organisational culture within government was mentioned as a key factor in three settings (Ethiopia, Solomon Islands and Rwanda). Linked to this is proactive identification of evidence needs by the country (highlighted in Bangladesh, Ethiopia, Nepal and Solomon Islands). Growing government financing, confidence and leadership in setting parameters within which evidence is used was highlighted in Cambodia, where a process of growing government leadership was accompanied by a transition in the demand for evidence originating within international organisations to originating within government. This demand can often be focused on internal learning, however, more than seeking evidence from other contexts.
Factors tending to increase confidence in suppliers of evidence included that the latter have in-country staff with embedded knowledge of the health system (highlighted in Bangladesh and Ethiopia). In some cases, authority derives from international agency authority (e.g. for the WHO package), as well as from donor funding and endorsement (Nepal).
In terms of credible evidence supply, this can be facilitated by the development of networks of international and local researchers, producing strong evidence on local policies and building capacity for local analysis (Cambodia). Similarly, consulting groups which maintain deep local roots in the local context while also connecting to international evidence can be effective evidence suppliers (Georgia).
Regional factors were again less prominent but, within West Africa, shared regional identities may play a role, facilitating learning across countries (Burkina Faso), while Nepal has consistently looked to India and Bangladesh for their experiences of community-based care. Shared languages also play a role, for example, francophone African policy, teaching and consulting networks were cited as influential in Rwanda.
The content of the reforms also matters. If reforms are technical and do not imply large structural changes, they will be easier to adopt (Nepal). In terms of the topic and its presentation, evidence is considered by decision-makers when it is politically relevant, accessible and locally applicable (Georgia). It needs to be adapted to the local cultural and geographic context. It is also important that it is presented at the right time in the budget or policy cycle and is communicated in the most acceptable way (for example, oral presentations were highlighted as sometimes preferable in Solomon Islands).
Barriers
In relation to demand for or use of evidence, cited barriers are grouped into those relating to incentives and those relating to capacity, while on the evidence supply side, capacity and resource factors dominated. Some specific gap areas were also mentioned.
Despite good leadership at the top, lack of accountability for results and weaknesses in supervision at middle management level and below were both barriers to acquiring and implementing learning from others (Solomon Islands). Politicised priorities and institutional constraints to be able to put evidence into effective use were highlighted as barriers in Bangladesh, while fragmentation in the sector and unclear roles was another constraint for operationalisation of policies (Nepal). Civil society was not reported to have played a strong role in the policy cycle in most places (only in Burkina Faso was its influence noted). The lack of an evaluation culture was mentioned in Bangladesh and Solomon Islands, and the issue of decisions being made outside the sector was also raised in the latter. The role of vested interests was highlighted in the Georgia case study, while in others, donor funding was noted to skew priorities. All of these undermine the role and utility of evidence.
Sharing and accessing information can also face barriers. A controlling approach to evidence release was highlighted in Rwanda and Ethiopia and, in some contexts, access to information was even more limited at local (sub-national) levels (e.g. Burkina Faso). Others highlighted the per diem-orientation in relation to participation in meetings, where lesson-learning is further weakened if there is a lack of dialogue and feedback from meetings (Solomon Islands). Sharing of information and evidence is largely personal and unstructured in some settings, rather than being institutional (Burkina Faso). In some places, simple factors like lack of connectivity and ICT skills remain a barrier (Nepal).
Lack of capacity to use evidence well was also mentioned (in Burkina Faso), leading to lack of adaptation of policies from the surrounding region, while in other places (Solomon Islands) participants did not perceive the relevance of evidence from other countries, even evidence from close neighbours (Fiji and Papua New Guinea), which share some similarities but have differences in governance and financing.
On the supply side, a number of countries noted weak in-country capacity to generate evidence (Georgia, Solomon Islands), including the lack of a national institute to perform close-to-policy work; indeed, the Solomon Islands had just one person specifically responsible for research in the Health Ministry, which is not atypical in low-income settings (some have nobody with this role). Having a smaller territory and being geographically isolated may be factors here. Researchers are often unable to be independent because of funding constraints (e.g. Burkina Faso), leading to ad hoc and poor-quality research. Limited national resources to support evidence generation locally were highlighted, especially for health systems research (Ethiopia, Georgia). In some cases, the withdrawal of international support aggravated these challenges (Georgia).
In relation to international agency advice, it is also worth highlighting that pressures and ideas coming from international actors are not always supported by international consensus; indeed, in many cases, international players provide conflicting advice (Georgia), even over technical decisions like on best procedures for Integrated Management of Maternal, Neonatal and Childhood Illness in Nepal. Advice can also be biased by donors’ ‘pet projects’ (Burkina Faso). This is manageable if governments have clear priorities; however, capacity to set clear priorities is itself commonly a barrier in these settings.
Some noted that, while there is relatively good access to policy documents and general guidance online, it is harder to find operational information on how to implement specific reforms (Ethiopia, Rwanda). Furthermore, it was noted by several respondents that the substantial amount of online information may be useful, yet it is difficult to access and time-consuming to sift through. There is therefore a demand for a brokerage function that would identify high quality, implementable information from other studies and reports. Some also felt that there was a lack of access to practical information, such as regional drug prices or trends in non-communicable diseases (Solomon Islands), while language barriers and limited access to journals remain challenges in some areas such as in Burkina Faso.