- Open Access
- Open Peer Review
Fostering access to and use of contextualised knowledge to support health policy-making: lessons from the Policy Information Platform in Nigeria
Health Research Policy and Systems volume 17, Article number: 38 (2019)
Contextualising evidence to inform policy-making is increasingly recognised as key to developing and implementing effective health policies. Creating a one-stop shop for evidence is an approach that can facilitate timely access to the best evidence to inform policy decisions. We report outcomes after implementation of the Policy Information Platform (PIP), a pilot one-stop evidence repository in Nigeria designed to alleviate barriers to accessing policy-relevant knowledge.
This cross-sectional study involved five phases, namely (1) consultation with Nigerian policy-makers to identify priority policy issues, areas of health policy information needs, and challenges and capacity constraints in accessing evidence for policy-making; (2) a stakeholder engagement workshop to formally launch the PIP; (3) extraction of data and other information from scientific articles, policy briefs, evaluation reports, grey literature and health policy documents relevant to policy-making in Nigeria (identified by Google and PubMed searches and by examination of websites of relevant Nigerian government ministries, agencies and parastatals), for use in developing the PIP website; (4) promotion of the PIP in national and state health policy meetings; and (5) evaluation of the PIP using a stakeholder survey questionnaire distributed via email and critical appraisal of the grey literature included in the PIP using the authority, accuracy, coverage, objectivity, date and significance (AACODS) checklist.
Priority policy areas identified by policy-makers were disease control and prevention, population health issues and health administration. Challenges identified by policy-makers were a lack of adequate capacity to access policy-relevant evidence and transform the evidence into policy. Policy-makers suggested using systematic reviews, policy briefs and rapid response mechanisms and involving policy-makers in research as ways of increasing evidence uptake for policy. A total of 126 policy-relevant, peer-reviewed scientific articles, 85 health policy documents and 201 policy-relevant grey literature documents were selected for inclusion in the PIP. Of the 195 individuals contacted via email to evaluate the PIP, 31 (15.9%) provided a response. Respondents noted that the PIP facilitated access to information based on local evidence and context-sensitive data. Barriers identified included lack of knowledge about the PIP and limited capacity of end-users to use the data compiled in the platform.
An easily accessible one-stop shop of policy-relevant evidence can considerably improve policy-makers’ access to evidence for use in policy-making and practice.
Contextualising evidence to inform policy-making is increasingly recognised as key to developing and implementing effective health policies. Several previous studies have indicated that research evidence can enhance the processes of health policy development and implementation [1,2,3,4,5]. Despite the importance of science in health policy-making, a considerable gap remains between research evidence and the formulation and implementation of health policies, particularly in low- and middle-income countries (LMICs) . Several studies have attempted to understand the suboptimal use of research evidence in policy-making, with findings suggesting the following key factors impeding evidence uptake: (1) lack of available research; (2) poor dissemination of research findings; (3) limited access to research, for various reasons including cost; (4) lack of clarity, relevance and reliability of research findings; and (5) unsuitability of the format of research output [5, 7, 8].
WHO has underlined that policy-makers need access to the right information at the right time to support evidence-informed decision-making . Timely, suitably packaged and policy-relevant research can foster the use of evidence in policy processes in LMICs [10,11,12,13]. Additionally, the importance of developing concise materials and tools to communicate various types of information to policy-makers and those supporting them is well recognised . Evidence exists for the value of information-packaging efforts designed to support action based on messages arising from research and other policy-relevant information [11,12,13]. Additional evidence suggests that effective communication relies on various factors, including readily understandable research, presentation of timely data in visually compelling formats, use of illustrative anecdotes where appropriate, creation of clear key messages about the meaning of the data, suggestions of ways to use research findings for answering important policy questions, involvement of policy-makers in the planning and execution of health research, as well as involvement of researchers in the planning and execution of health programmes, prompts of relevant articles or evidence briefs targeted to appropriate policy-makers, and establishment of relationships of trust and credibility with policy-makers [14,15,16,17].
Furthermore, the complexity of decision-making requires inputs from a broad evidence base beyond scientific research, including knowledge generated from local evidence and good practices, as well as tacit knowledge . For instance, Pang  noted that epidemiological studies, qualitative research, experience, know-how, consensus and local knowledge should all be taken into account in health policy-making. Although the vast majority of research evidence is published in peer-reviewed journals, much contextualised and policy-relevant knowledge is confined to the grey literature and is not widely shared. The grey literature comprises a wide range of material, including policy documents, statistical publications, newsletters, fact sheets, working papers, technical reports, conference proceedings, dissertations and multimedia content . There is increasing interest in the pivotal role played by the grey literature in the evidence-to-policy process . For example, in May 2014, the Pisa Declaration on Policy Development for Grey Literature Resources was signed . Its signatories called for increased recognition of the grey literature’s role and value by governments, academics and other stakeholders, particularly in terms of its importance for open access to research, open science, innovation, evidence-based policy and knowledge transfer . Despite this interest, knowledge from the grey literature is not optimised by decision-makers, researchers and other stakeholders in the development of policies and programmes . In 2012, the WHO Strategy on Health Policy and Systems Research suggested synthesising and consolidating relevant research evidence and other knowledge via country-specific national repositories of evaluations, best practices and grey literature to enable greater access to existing knowledge that could improve decision-making . One of the approaches for achieving this synthesis and consolidation is the development of ‘one-stop shops’ of high-quality research evidence and policy-relevant knowledge products.
A one-stop shop for research evidence can allow health system policy-makers, stakeholders and researchers to efficiently find and use the best available research evidence in the limited time they have available to make, inform or advocate for a decision . Several one-stop shops already in existence were designed to provide information related to clinical programmes and services, prescription drugs, and public health programmes and services [23,24,25,26,27,28]. In recent years, global one-stop shops have been developed to address issues related to health policy and health systems, with a focus on decision-makers, including the Health Systems Evidence repository, the Health Technology Assessment Database, EVIPNet (the Evidence to Policy Network), the Virtual Health Library and the PDQ-Evidence repository [23, 28,29,30,31]. However, to date, few one-stop shops have been established in LMICs to address their needs for health policy and health systems evidence. More specifically, there has been no health policy repository in Nigeria that could be regarded as a one-stop shop for various types of high-quality evidence (e.g. peer-reviewed research publications, expert opinion, policy documents and grey literature) specifically relevant to policy-makers’ needs for decision-making in that country.
To address this gap, the WHO Alliance for Health Policy and Systems Research and the African Institute for Health Policy & Health Systems of Ebonyi State University established a pilot repository called the Policy Information Platform (PIP) , to alleviate barriers to accessing the existing policy-relevant knowledge. The platform was designed to include both indexed publications and the grey literature, as well as to ensure that relevant and reliable information is available in a user-friendly format for policy and management decisions in the country. The aim of the current study was to assess the implementation of the PIP in Nigeria, with a view to documenting its functioning and policy influence, as well as understanding decision-makers’ perceptions of and satisfaction with this resource.
This cross-sectional study involved five phases in developing the PIP, as follows: (1) consultation with policy-makers and identification of priority policy issues; (2) a stakeholder engagement workshop, with formal launch of the PIP; (3) extraction of data from policy-relevant publications and development of the PIP website; (4) promotion of the PIP website; and (5) evaluation of the PIP.
Consultation with policy-makers and identification of priority policy issues
The PIP was planned to represent a decision-making resource and an actionable repository of knowledge, with its content designed to address key priorities identified at the national level in Nigeria. To determine the priority health policy issues to be showcased within the platform, we engaged with key policy-makers representing various areas of the health sector in Nigeria (Tables 1 and 2). These policy-makers were interviewed during face-to-face discussions or by telephone. We asked them to identify key priorities in the health policy-making process within both the government and the health sector in Nigeria, for which policy-relevant information was needed.
Stakeholder engagement workshop and formal launch of the PIP
A 1-day stakeholder engagement event was convened in September 2015 at Abakaliki, Nigeria, during which the PIP was formally launched. The purposes of the meeting were to bring together policy-makers, researchers and other stakeholders in the health sector policy-making process (including health practitioners, civil society organisations and media practitioners), to formally present the PIP Nigeria, including the website, and to elicit insights on the implementation and effectiveness of the platform to support health policy-making. A structured questionnaire (Additional file 1) was administered to participants to assess (1) their health policy-relevant information needs; (2) the challenges and capacity constraints they experienced in accessing evidence; (3) the ways in which they utilised evidence in policy-making; and (4) their suggestions of ways and formats in which policy-relevant information could be made easily available and accessible through the PIP.
Extraction of data and other information from policy-relevant publications and development of the PIP website
The potential content of the PIP was classified into five main categories, namely (1) scientific articles, (2) policy briefs, (3) evaluation reports, (4) grey literature and (5) health policy documents. A description of the process for extracting data and other information from these publication types is provided below.
Scientific publications reporting research done in Nigeria related to policy-making in the field of maternal, newborn and child health (MNCH) were sought. The emphasis on MNCH was a recommendation arising from the stakeholder engagement event. A PubMed search of the MEDLINE database was performed in August 2015, and studies published in English that investigated MNCH in Nigeria in relation to health policy were identified. To be included as scientific articles, these publications had to be original studies and had to contain policy recommendations. Eligible studies were retrieved and indexed in the PIP, with links to the full articles in PubMed.
For the purpose of the PIP, a policy brief was defined as a policy document that clarifies a health policy problem, renders the evidence for addressing the problem concise and understandable, explains why the evidence is important, describes evidence-informed policy options that would be suitable actions for policy-makers to take, and provides key implementation considerations [33,34,35]. A Google search was performed in August 2015 with the keywords ‘policy brief’, ‘health’ and ‘Nigeria’, yielding a total of 313 entries, of which 46 policy briefs were selected. Each selected document fulfilled the following inclusion criteria: (1) must be a policy brief that meets the definition given above, (2) must focus on Nigeria, (3) must focus on the health of the population, and (4) must highlight recommendations relevant to health policy-making.
For the purpose of the PIP, an evaluation report was defined as a document that reports a systematic assessment of a health activity, project, programme, policy or institutional performance and that provides evidence-based information relevant to the decision-making processes, to allow an understanding of achievements or the lack thereof . To identify evaluation reports, another Google search was performed in August 2015, with keywords such as ‘evaluation reports’, ‘health policy’ and ‘Nigeria’; this search yielded 363 entries, of which 23 were selected. The selected documents fulfilled the following criteria: (1) must be an evaluation report meeting the definition given above, (2) must focus on Nigeria, (3) must focus on the health of the population, and (4) must highlight recommendations relevant to health policy-making.
For the purposes of the PIP, the grey literature was defined as documents produced by all levels of government, academics, business and industry, in print or electronic formats, but not controlled by commercial publishers [19, 21]. The websites of more than 30 health-related organisations were searched for policy-relevant grey literature in August 2015. The websites were hosted by organisations such as the Nigerian health-related government ministries, departments and agencies, non-governmental organisations (NGOs), civil society organisations, and health and professional associations. The materials obtained included policy documents, statistical publications, newsletters, bulletins, fact sheets, working papers, technical reports, conference proceedings, dissertations and multimedia content. The criteria used for the selection were as follows: (1) must be a grey literature document that fulfils the definition provided above, (2) must focus on Nigeria, (3) must focus on health of the population, and (4) must be relevant to health policy-making.
Critical appraisal of grey literature
Each document from the grey literature was critically appraised using a modified version of the Authority, Accuracy, Coverage, Objectivity, Date and Significance (AACODS) checklist  (Additional file 2). This tool considers the following criteria: (1) Authority: Is the document from a reputable organisation or individual author from a reputable organisation? (2) Accuracy: What are the aims of the document? Has it been peer-reviewed or edited? Is the basis for the document clear? Is the document well structured? (3) Coverage: Are any limits clearly stated? (4) Objectivity: Does the work seem to be balanced in presentation? (5) Date: Is the date of the document given? (6) Significance: In the researcher’s estimation, will the document be of interest? The AACODS checklist has been modified by the National Institute for Clinical and Health Excellence (NICE, United Kingdom) and included among that organisation’s checklists for evidence evaluation . We further refined the tool to include grading and a cut-off threshold of three points for inclusion or exclusion from the PIP. Notably, in this study, the modified AACODS checklist was used as a decision-making tool for inclusion or exclusion of grey documentation, but not to provide a scientific quality appraisal per se (e.g. low, moderate or high quality). Of 393 documents assessed, 201 (51.1%) met the threshold for inclusion.
Health policy documents
We searched for relevant policy documents at the websites of all Nigerian federal ministries that deal, directly or indirectly, with health, including Health and Social Services, Women Affairs and Social Development, Water Resources and Rural Development, Science and Technology, Finance and Economic Development, Environment, Education and Youth Development, and Agriculture and Natural Resources. We also searched the websites of all agencies and parastatals under the Ministry of Health, including the National Health Insurance Scheme and the National Primary Health Care Development Agency. A total of 85 health policy documents were identified and included in the PIP.
Promotion of PIP website in national and state health policy meetings
A presentation about the PIP was made at one national and two state health policy meetings. The national meeting was a stakeholder engagement event on MNCH organised by the West African Health Organization and held in October 2015 in Abuja; 92 participants were in attendance. The two state meetings took place at Ebonyi State University in Ebonyi State, Nigeria, with 32 participants in the November 2015 event and 35 in the April 2016 meeting. Attendees at all three meetings included policy-makers, researchers and other stakeholders. The engagement process involved interaction between policy-makers and researchers on issues related to the research-to-policy interface and the establishment of formal mechanisms for continuous partnership. The PIP was introduced to the participants as a platform that would continue to provide policy-relevant information; contact information was provided so that users could request additional relevant information to address future policy-making needs.
Evaluation of the PIP using stakeholder survey questionnaire
To assess the impact of the PIP among stakeholders who had been informed about its existence, we conducted an evaluation survey via email 6 months after the PIP was established. For this survey, 195 individuals were contacted via email. These contacts included individuals who had participated in the national and state meetings where presentations about the PIP had been made. The following questions were posed in the email survey:
How has the PIP Nigeria benefited or served you in your work as a policy-maker or in research as a researcher?
What are the drivers or motivation in your demand for information, and engagement with the PIP Nigeria website?
How can we make sure that the resource is known by policy-makers and researchers and used to inform the different policy steps?
What do you think are the facilitators and impediments to your use of the platform?
How can we improve the PIP Nigeria?
The written responses to the evaluation survey were analysed according to Giorgi’s phenomenological approach [39, 40]. Qualitative data analysis was conducted by assessing narratives and textual information, identifying all comments that appeared significant, abstracting units of meaning, categorising and summarising the abstractions, and returning to the extracted text to ensure a good fit.
Outcome of consultation with policy-makers and identification of priority policy issues
The key priority areas (n = 17) identified by the policy-makers during the initial consultation were categorised as disease control and prevention, population health issues and health administration (Table 3). There was much emphasis on the need to prioritise policy issues regarding vulnerable populations, including pregnant women and children under 5 years of age, as well as issues relating to adolescent, reproductive and geriatric health.
Outcome of stakeholder engagement workshop
A total of 52 participants attended the stakeholder engagement workshop and formal launch of PIP, and the profile of this group is presented in Table 4. Most participants (57.4%) were older than 45 years of age, were from the Local Government Service Commission (44%), were heads of department (39.2%) and possessed a bachelor degree (64%). Table 5 summarises the responses of participants regarding (1) areas of health policy information needs, (2) challenges and capacity constraints in accessing evidence for policy-making, (3) how evidence is used in policy-making and (4) the formats in which policy-relevant information could be made available to policy-makers (n = 32). Key policy information needs identified by participants pertained to adolescent health, reproductive health, health policy advocacy, programme planning and implementation, and health management information systems. Among their major capacity constraints, participants identified a lack of adequate capacity to transform evidence into policy and a lack of skill in accessing evidence and policy-relevant information. They suggested using systematic reviews, policy briefs and rapid response mechanisms, and involving policy-makers in research as ways of increasing evidence uptake for policy (Table 5).
Outcome of data and information extraction from policy-relevant publications for the PIP
The extraction of data and other information from policy-relevant publications for development of the PIP was guided by the priority policy issues identified via consultation and interaction with policy-makers. Figure 1 summarises the outcome of this extraction process. A total of 126 policy-relevant peer-reviewed scientific articles, 85 health policy documents, and 201 policy-relevant grey literature documents were selected and included in the PIP.
Outcome of PIP evaluation survey
Of the 195 individuals contacted via email with the survey questionnaire, a total of 31 provided responses, which represented a 15.9% response rate. The respondents consisted of policy-makers (22.6%), researchers (35.5%), health practitioners (35.5%) and NGO executives (6.5%). Themes that emerged from analysis of respondents’ comments are highlighted below.
Access and efficiency
Stakeholders highlighted the drivers of and motivation for demand for information and engagement with the PIP, including access to information and efficiency in identifying context-sensitive data.
“What we look out for in engagement with PIP Nigeria website includes informational materials on health systems and support, health policy, health financing, advocacy and mobilisation for evidence-based and inclusive health policy and systems, links for engagement with wider networks for sustainable and resource saving systems.” (NGO Executive)
“The website helps to connect me to articles relevant for research, other materials and books not easy to come by. These articles and books are eye opening and very informative.” (Researcher)
End-users underlined the usefulness of the wide array of local evidence readily available for policy and systems decisions, and its applicability in the Nigerian health system setting.
“The PIP Nigeria indeed served [as] a-one-stop-shop when seeking for policy documents on health policy. Its organisation made searching easy. The collections of local researches made discussion in relation to our Nigerian context robust.” (Policy-maker)
Networking and collaboration
Stakeholders identified that a collaborative space such as the PIP can foster exchanges and inter-linkages related to evidence-informed decision-making. Researchers highlighted the importance of such platforms in catalysing dialogue with policy-makers vis-à-vis implementation research needs.
“The Policy Information Platform PIP Nigeria has served me as a researcher in Nigeria by providing information on critical areas and issues that require research intervention and networking for optimum results. As a Research Team Leader, the PIP has also been of benefit to me and my research team by providing the platform for collaboration between researchers and policy-makers to ensure that research outputs are made readily available to policy implementers.” (Researcher)
Strengthening capacities of policy-makers to use research
Policy-makers mentioned that engaging with the PIP supported their capacities to access and use research evidence in decision-making, with a particular focus on policy analysis.
“Strengthened my capacity to apply health research in making policy decisions.” (Policy-maker)
“The PIP exposed me to health policies … it provided relevant resources for policy analysis … it provided scientific and evidence-based policies.” (Policy-maker)
Stakeholders identified the practicality and ease of navigation throughout the PIP.
“My motivations for demand are versatility, quality, scope of studies as well as ease of use.” (Researcher)
Facilitators and barriers
Respondents identified factors stimulating and impeding their use of the PIP in the context of the Nigerian health system. Facilitators included the open access feature and ongoing, iterative development and updating of content, as well as outreach features of the platform, such as prompts to users about specific evidence showcased by the PIP. Barriers included lack of knowledge about the platform and its content and limited capacities of end-users in accessing and using data included in the platform.
Regarding the facilitators, one researcher noted that “the website is created and maintained by University free for our use, thereby making information on global issues and updates available”.
Concerning the impediments, one policy-maker stated that ignorance of the existence and availability of the platform limited access to the information that it contains.
Supporting demand and sustainability
Various stakeholders identified the need to advocate for the evidence and updates provided by the platform, and to make sure that the PIP is known by policy-makers and used to inform the different policy steps.
“This would be possible by periodically engaging the stakeholders in the health sector where this platform would be introduced and its practical use made in the formulation of policy. Such programmes should be sustained.” (Researcher)
“The resources can be made known to people by advertising it in journals, newspapers, referring friends to the site, encouraging users to cite it in their publications, and presenting it in government/stakeholder meetings in different ministries not only the health ministry.” (NGO executive)
“Visibility for the resource can be assured by any or a combination of the following options, creating a link with the websites of affiliated institutions and agencies, advertise via online newsletters and blogs, writing feature articles via mass media, in-house magazines and newsletters of Ebonyi State University and affiliated organisations, etc.” (NGO executive)
“Do more stakeholder engagement/sensitization for them to have the knowledge of PIP, the importance and the use of the website.” (Policy-maker)
The PIP as a ‘one-stop shop’ innovation
This study represents the first attempt in Nigeria to promote a mechanism (one-stop shop) for fostering access to and use of relevant knowledge in decision-making. The PIP was designed with consideration of policy-makers as the target key users. This one-stop shop is the only platform indexing Nigerian health policy documents, policy briefs, evaluation reports and grey literature that are available online from the Ministry of Health and other related ministries, NGOs, civil society organisations and development partners. Our study suggests that the PIP platform is perceived by policy-makers and stakeholders as a useful tool to inform various steps in the policy-making process. The acceptability and reported utility of the PIP platform in Nigeria corroborates experiences in high-income health system settings, where one-stop shops have been used to ensure that policy-makers have timely access to research evidence when pressing issues emerge, which increases the prospects that research will be used in decision-making [23, 41].
The outcome of our evaluation of the PIP showed strong acceptability and feasibility of the one-stop shop platform as a resource to inform policy and systems decisions in Nigeria, including but not limited to policy analysis. Our study also identified the PIP as an asset that might strengthen the interest of policy-makers in using evidence for policy change. Decision-makers in the present study expressed their willingness to use research, which corroborates previous experiences showing that decision-makers are likely to use research evidence for policy-making when the evidence is available and accessible in a timely and user-friendly fashion [2, 4,5,6]. In this regard, the PIP explicitly aims to address the main barriers to integrating research into policy and practice, including high costs of access, poor clarity, relevance and reliability of research findings, and complex format of research outputs [7, 8]. Our study further reinforces previous recommendations to improve reporting formats to enhance the intelligibility of policy-relevant research outputs [2, 6, 14]. It is pertinent to state that the PIP may not be able to entirely support the translation of evidence into policy and enhance the capacity to access evidence. We are, however, confident that the availability and accessibility of the evidence in this platform are stepping stones towards enhancing any effort by policy-makers in evidence uptake and improvement of their capacities to access evidence.
Lessons from policy-makers’ engagement
This study included active engagement of policy-makers in the process of developing the PIP platform. Interest in exchanges and engagement between researchers and policy-makers is gaining momentum worldwide, and such engagement seems better suited for the complex nature of policy-making processes than static ‘generalisable’ research (i.e. research initiated by the researchers without any interaction with policy-makers and not targeting the needs of policy-makers) [42, 43]. Interactions with policy-makers enabled the identification of key areas of policy priorities, as well as policy information needs, capacity constraints and formats in which policy-relevant information could be made available and easily accessible. The information that we obtained from the policy-makers aided in development of the platform, making it highly suitable to policy-makers’ needs and thereby facilitating the concept of ownership.
Previous evidence has suggested that, because of inadequate engagement of policy-makers, health policy needs neither drive nor determine the research priority-setting process, thereby resulting in lack of ownership of the health research agenda by policy-makers . It is important for both researchers and decision-makers to recognise the value of coming together in what is in fact a symbiotic relationship in which decision-makers generate feedback from the front lines, while researchers provide expertise in research methods needed for trustworthy studies . This suggests that engagement between policy-makers and researchers may ensure that knowledge generated is better aligned with the health needs of society [44, 45]. Interactions during the stakeholder engagement workshop likely led to differential use of the material in the platform, which might have increased participants’ willingness to use research or their access to the research.
Some vital lessons can be drawn from policy-makers’ involvement in the development of the PIP. One of these lessons is that policy-makers are knowledgeable about Nigerian health policy priority areas and are willing to offer helpful information to researchers. Previous studies in Nigeria have suggested that the engagement of policy-makers is crucial to the use of evidence in the planning and execution stages of health policies [17, 46]; the PIP experience corroborates this knowledge. The topics identified by policy-makers were consistent with the key health policy priority areas outlined in the most recent Nigerian National Strategic Health Development Plan (NSHDP) . The NSHDP has the following eight strategic priority areas: leadership and governance for health, health service delivery, human resources for health, financing for health, national health management information system, partnerships for health, community participation and ownership, and research for health . By increasing access to and usability of policy-relevant documentation, the PIP can serve as a tool to foster evidence-informed policy-making, directed towards the implementation of strategic health programmes in Nigeria, under the aegis of the NSHDP. Health policy documents, policy briefs and policy-relevant peer-reviewed articles covering these priority areas were among the publications included in the PIP.
Value of grey literature
A major innovation of the PIP repository was the inclusion of policy-relevant grey documentation, which accounted for the largest number of publications indexed in the platform. Policy-makers specifically requested the inclusion of policy-relevant information presented in easily available and accessible formats. Specific requests pertained to publications in the form of expert information, pilot studies, case studies, national surveys, monthly and annual reports, bulletins, executive summaries and policy advocacy documents. Policy-makers thus seemed to have an understanding of and confidence in the value of grey literature as a source of vital policy-making resources and tools.
According to Lawrence et al. , peer-reviewed journals might be considered “the most credible source of evidence”, but the reality is that policy-relevant evidence is found in many kinds of resources circulating in the public sphere, most of which fall into the category of grey literature. The importance of grey literature therefore lies in its ability to communicate complex information in simple terms and to disseminate results more quickly . Previous experience has shown that, in policy settings, information in the grey literature may be given greater emphasis than information from peer-reviewed journals because the language is more accessible; furthermore, more rapid and more flexible delivery of information can facilitate knowledge uptake in situations where decisions are based on competing factors (e.g. the pressures of political processes) [50,51,52,53,54].
It is pertinent to state that concern about the reliability of grey literature has generated some reservations. The lack of editorial control may also raise questions about the authenticity and reliability of documents in the grey literature . In addition, Adams et al.  noted that grey literature is not bound by the publishing conventions that characterise peer-reviewed literature, and that it comes in a variety of forms, posing challenges for data management, extraction and synthesis. To address these limitations, the PIP in Nigeria employed a modified AACODS checklist to critically appraise the identified grey literature and to ensure that the platform included only documents that respect standards of scientific quality and integrity . The modified AACODS checklist was used to include or exclude grey documentation from the platform, and there is a need to further study and validate the modified AACODS checklist before it can be used for traditional appraisal and grading of scientific quality. We also need additional empirical evidence through pilot testing and application of the modified and traditional AACODS approaches to study face validity and implementation determinants (e.g. appropriateness and feasibility).
This study had several limitations. One major limitation was our inability to critically appraise the scientific studies that were identified with PubMed and included in the PIP. Our intention is to continue to improve the quality of studies included in the PIP; to achieve this goal, we will conduct critical appraisal of scientific publications that are considered for future inclusion. Another limitation was our use of a threshold score from the AACODS checklist to determine the inclusion or exclusion of each document from the grey literature. In an earlier report, Jüni et al.  highlighted the problems associated with using summary scores, noting that the incorporation of quality scores as weights lack statistical or empirical justification, as previously pointed out by Detsky et al. . Our intention for the future is to individually assess the relevant methodological aspects of each document to be included in the PIP, as recommended by Jüni et al. . The rather low response rate to the email survey (15.9%) is yet another major limitation. Many decision-makers in Nigeria do not use email frequently, and email-based distribution is therefore the survey strategy that generates the least number of responses among policy-makers .
The need for constant improvement and sustenance of the PIP Nigeria cannot be overstated. Therefore, we will continue to update the contents of the PIP on a monthly basis. We will also continue to expand the scope of the PIP by identifying other priority areas of interest to the policy-makers. Although the PIP focused on evidence specifically produced in Nigeria, global evidence may well be relevant to Nigerian policy-makers, as may be research from neighbouring countries. As part of our efforts to improve the reliability of the PIP, we intend to link the platform to other sources of relevant evidence for policy-makers. Potential improvements to the platform include ongoing and iterative stakeholder engagement, additional dissemination and advertising of the platform through social media and scientific conferences, and integration into the platform of a ‘frequently asked questions’ section and further user-friendly navigation guidance.
An easily accessible one-stop shop of policy-relevant evidence can considerably improve policy-makers’ willingness to use evidence in policy-making and practice. The comments of some of the policy-makers suggest that they are willing to work with researchers to improve the evidence-informed policy-making process when appropriate opportunities are made available to them. It is therefore important for researchers to initiate processes to encourage the engagement of policy-makers. The PIP is one such initiative that can facilitate interaction between policy-makers and researchers; this innovation merits further consideration.
Authority, accuracy, coverage, objectivity, date and significance
low- and middle-income countries
maternal, newborn and child health
National Strategic Health Development Plan
Policy Information Platform
Mitton C, Adair CE, McKenzie E, Patten SB, Waye Perry B. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85(4):729–68.
Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21.
Dobrow MJ, Goel V, Upshur REG. Evidence-based health policy: context and utilisation. Soc Sci Med. 2004;58(1):207–17.
Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.
Innvar S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44.
Lavis JN, Becerra Posada F, Haines A, Osei E. Use of research to inform public policymaking. Lancet. 2004;364:1615–21.
Lavis JN, Davies HTO, Oxman AD, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10((Suppl 1)):35–48.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
World Health Organization. Global Report for Research on Infectious Diseases of Poverty. 2012. http://www.who.int/tdr/publications/global_report/en/. Accessed 28 Feb 2018.
Adam T, Moat KA, Ghaffar A, Lavis JN. Towards a better understanding of the nomenclature used in information-packaging efforts to support evidence-informed policymaking in low- and middle-income countries. Implement Sci. 2014;9:67.
Lavis JN. How can we support the use of systematic reviews in policymaking? PLoS Med. 2009;6(11):e1000141.
Rosenbaum SE, Glenton C, Wiysonge CS, Abalos E, Mignini L, Young T, et al. Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ. 2011;89(1):54–61.
Lavis JN, Lomas J, Harnid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006;84:620–8.
Feldman PH, Nadash P, Gursen M. Improving communication between researchers and policy makers in long-term care or “researchers are from Mars; policy makers are from Venus”. Policy Brief (Cent Home Care Policy Res). 2001;5:1–6.
Langlois EV, Tunçalp Ö, Norris SL, Askew I, Ghaffar A. Qualitative evidence to improve guidelines and health decision-making. Bull World Health Organ. 2018;96(2):79.
Lomas J. Using “linkage and exchange” to move research into policy at a Canadian foundation. Health Aff. 2000;19:236–40.
Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Promotion of evidence-informed health policymaking in Nigeria: bridging the gap between researchers and policymakers. Glob Public Health. 2012;7(7):750–65.
Pang T. Evidence to action in the developing world: what evidence is needed? Bull World Health Organ. 2007;85:247.
Nineteenth International Conference on Grey Literature. Public Awareness and Access to Grey Literature. Rome: National Research Council of Italy, CNR; 2017. http://greyguide.isti.cnr.it/attachments/category/30/GL19_Conference_Proceedings.pdf. Accessed 28 Feb 2018.
Korjonen H, Ford J. Grey literature in public health: valuable evidence? UK Health Forum 2013. https://wachslibrary.health.wa.gov.au/ld.php?content_id=32308111. Accessed 28 Feb 2018.
Pisa Declaration on Policy Development for Grey Literature Resources. http://www.greynet.org/images/Pisa_Declaration,_May_2014.pdf. Accessed 28 Feb 2018.
World Health Organization. Changing mindsets: strategy on health policy and systems research. Geneva: WHO; 2012. http://www.who.int/alliance-hpsr/alliancehpsr_changingmindsets_strategyhpsr.pdf. Accessed 28 Feb 2018.
Lavis JN, Wilson MG, Moat KA, Hammill AC, Boyko JA, Grimshaw JM, et al. Developing and refining the methods for a ‘one-stop shop’ for research evidence about health systems. Health Res Policy Syst. 2015;13:10.
Cochrane Collaboration. The Cochrane Library. 2018. http://www.cochranelibrary.com/. Accessed 28 Feb 2018.
Canadian Agency for Drugs and Technologies in Health. CADTH Evidence Driven. 2018. https://www.cadth.ca/. Accessed 28 Feb 2018.
Weir M, Ryan R, Mayhew A, Worswick J, Santesso N, Lowe D, et al. The Rxfor Change database: a first-in-class tool for optimal prescribing and medicines use. Implement Sci. 2010;5:89.
Health-Evidence.ca. Health Evidence. 2018. https://www.healthevidence.org/. Accessed 28 Feb 2018.
Evidence Alerts. 2018. https://plus.mcmaster.ca/EvidenceAlerts/. Accessed 28 Feb 2018.
Polisena J, Lavis JN, Juzwishin D, McLean-Veysey P, Graham ID, et al. Supporting the use of health technology assessments by decision-makers. Healthc Policy. 2015;10(4):10–5.
Moat KA, Lavis JN. Supporting the use of research evidence in the Americas through an online “one-stop shop”: the EVIPNet VHL. Cad Saude Publica. 2014;30(12):2697–701.
PDQ-Evidence. 2018. http://www.pdq-evidence.org/. Accessed 28 Feb 2018.
Policy Information Platform Nigeria. 2018. www.pipnigeria.org. Accessed 28 Feb 2018.
Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 13: preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 2009;7(suppl 1):S13.
Brownson RC, Dodson EA, Stamatakis KA, Casey CM, Elliott MB, Luke DA, et al. Communicating evidence-based information on cancer prevention to state-level policy makers. J Natl Cancer Inst. 2011;103(4):306–16.
International Development Research Centre, Swiss Agency for Development and Cooperation. Chapter 8: The Two-Pager: Writing a Policy Brief. In: The RM Knowledge Translation Toolkit: A Resource for Researchers. Ottawa, IDRC and SADC; 2008.
World Health Organization. WHO Reform, WHO Evaluation Policy. Executive Board EB130/5 Add. 8, 130th session 22. Provisional Agenda Item 5. 2011. http://apps.who.int/gb/ebwha/pdf_files/EB130/B130_5Add8-en.pdf?ua=1. Accessed 6 Nov 2018.
Tyndall J. The AACODS Checklist is Designed to Enable Evaluation and Critical Appraisal of Grey Literature: Flinders University; 2010. http://dspace.flinders.edu.au/jspui/bitstream/2328/3326/4/AACODS_Checklist.pdf. Accessed 28 Feb 2018.
National Institute for Health and Care Excellence (NICE). Interim Methods Guide for Developing Service Guidance. 2014. https://www.nice.org.uk/process/pmg8/chapter/appendix-2-checklists. Accessed 28 Feb 2018.
Giorgi A. Sketch of a Psychological Phenomenological Method. In: Giorgi A, editor. Phenomenology and Psychological Research: Essays. Pittsburgh: Duquesne University Press; 1985. p. 8–22.
Albert MA, Fretheim A, Maïga D. Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali’s essential medicines. Health Res Policy Syst. 2007;5:2.
Lavis JN, Davies HTO, Oxman AD, Denis J-L, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10(Suppl 1):35–48.
Langlois EV, Becerril Montekio V, Young T, Song K, Alcalde-Rabanal J, Tran N. Enhancing evidence informed policymaking in complex health systems: lessons from multi-site collaborative approaches. Health Res Policy Syst. 2016;14:20.
Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.
Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide: Alliance for Health Policy and Systems Research, World Health Organization; 2013. http://www.who.int/alliance-hpsr/alliancehpsr_irpguide.pdf. Accessed 28 Feb 2018.
World Health Organization. Implementation research for the control of infectious diseases of poverty: strengthening the evidence base for the access and delivery of new and improved tools, strategies and interventions: World Health Organization on behalf of the Special Programme for Research and Training in Tropical Diseases; 2011. http://apps.who.int/iris/handle/10665/75216. Accessed 28 Feb 2018.
Uneke CJ, Aulakh BK, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Bridging the divide between research and policy in Nigeria: the role of health policy advisory committee. J Public Health Policy. 2012;33(4):423–9.
Federal Ministry of Health, Nigeria. National Strategic Health Development Plan 2010–2015. Abuja: MoH; 2010. https://www.healthresearchweb.org/files/NSHDP.pdf. Accessed 28 Feb 2018.
Lawrence A, Houghton J, Thomas J, Weldon P. Where is the Evidence: Realising the Value of Grey Literature for Public Policy and Practice. Melbourne: Swinburne Institute for Social Research; 2014. https://researchbank.swinburne.edu.au/file/8546f00c-f178-48e6-bf86-cd04d0907bef/1/PDF%20%28Published%20version%29.pdf. Accessed 28 Feb 2018.
National Library of Medicine. Health Services Research and Health Policy Grey Literature Project: Summary Report. 2006. https://www.nlm.nih.gov/nichsr/greylitreport_06.html. Accessed 28 Feb 2018.
MacDonald BH, Ross JD, Soomai SS, Wells PG. How information in grey literature informs policy and decision-making: a perspective on the need to understand the processes. Grey J. 2015;11(1):7–16.
Bremer S, Glavovic B. Mobilizing knowledge for coastal governance: re-framing the science-policy interface for integrated coastal management. Coastal Manage. 2013;41:39–56.
Pielke RA. The Honest Broker: Making Sense of Science in Policy and Politics. New York: Cambridge University Press; 2007.
Shanley P, López C. Out of the loop: why research rarely reaches policy makers and the public and what can be done. Biotropica. 2009;41:535–44.
Nove A, Cometto G, Campbell J. Assessing the health workforce implications of health policy and programming: how a review of grey literature informed the development of a new impact assessment tool. Hum Resour Health. 2017;15(1):79.
Mathews B. Gray literature: resources for locating unpublished research. C&RL News. 2004;65:3.
Adams RJ, Smart P, Huff AS. Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies. Int J Manage Rev. 2017;19:432–54.
Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11):1054–60.
Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbé KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992;45:255–65.
Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Enhancing health policymakers’ capacity to use information and communication technology in Nigeria. J Health Inform Dev Countr. 2011;5(2):228–46.
The authors wish to thank all of the policy-makers from the Federal Ministry of Health (Abuja) and the Ebonyi State Ministry of Health (Abakaliki) for their participation in the study. The authors also gratefully acknowledge the participation in this study of senior researchers from Ebonyi State University (Abakaliki) and other stakeholders in the Nigerian health sector.
This investigation received financial support from the Alliance for Health Policy and Systems Research, WHO (AHPSR/WHO) (Reference 2015/529432–0).
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request.
Ethics approval and consent to participate
Ethics approval for this study was provided by Ebonyi State University Research Ethics Committee (UREC). The study strictly adhered to the UREC guideline, which stipulates voluntary participation, obtaining of informed consent, maintenance of anonymity of participants and confidential treatment of all participant-related information. Informed consent was obtained from all participants who took part in the study during the workshop and via email.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.