SUPPORT Tools for evidence-informed health Policymaking (STP) 12: Finding and using research evidence about resource use and costs
© Oxman et al; licensee BioMed Central Ltd. 2009
Published: 16 December 2009
This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.
In this article, we address considerations about resource use and costs. The consequences of a policy or programme option for resource use differ from other impacts (both in terms of benefits and harms) in several ways. However, considerations of the consequences of options for resource use are similar to considerations related to other impacts in that policymakers and their staff need to identify important impacts on resource use, acquire and appraise the best available evidence regarding those impacts, and ensure that appropriate monetary values have been applied. We suggest four questions that can be considered when assessing resource use and the cost consequences of an option. These are: 1. What are the most important impacts on resource use? 2. What evidence is there for important impacts on resource use? 3. How confident is it possible to be in the evidence for impacts on resource use? 4. Have the impacts on resource use been valued appropriately in terms of their true costs?
This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The series is intended to help such people ensure that their decisions are well-informed by the best available research evidence. The SUPPORT tools and the ways in which they can be used are described in more detail in the Introduction to this series . A glossary for the entire series is attached to each article (see Additional File 1). Links to Spanish, Portuguese, French and Chinese translations of this series can be found on the SUPPORT website . Feedback about how to improve the tools in this series is welcome and should be sent to: STP@nokc.nohttp://www.support-collaboration.org.
You work in the Ministry of Health and the Minister of Health has asked you to brief her on the costs of options being considered as part of a healthcare reform programme.
In this article, we present four questions that policymakers and those who support them can ask when assessing the costs of a policy or programme option. Such questions could be applied, for instance, in the scenario outlined above. Our focus is on finding and using evidence related to resource use and the costs of a policy or programme option, rather than on cost-effectiveness analysis or other types of economic analysis.
Policymakers want to ensure that policies represent good value for money, as do those affected by them. To do this it is essential to consider the costs of options as well as their health and other impacts. Option costs differ from other impacts in a number of key ways :
Healthcare costs are typically shared. For most impacts other than costs, it is usually clear who will be advantaged and who will be disadvantaged, though this may not be the case for all outcomes. An entire community will benefit from a vaccination programme because of the herd effect (the reduced transmission of the disease once most community members are vaccinated). Similarly, in the case of the widespread use of antibiotics to treat individual infections, downstream adverse consequences of drug resistance may occur for the wider community. These are exceptions for health outcomes. On the other hand, healthcare costs are typically shared by the government, private insurers, employers and patients. And within a society, how costs are shared may differ still further depending on patient age (e.g. whether they are under or over 65) or circumstance (e.g. whether the patient is receiving social assistance).
Unit costs tend to vary widely across jurisdictions. For instance, the cost per unit of drugs is largely unrelated to the actual costs of production but is instead more closely related to marketing decisions and national policies. Thus, for example, most medicines under patent cost substantially more in the United States than in Canada . Further, costs may vary widely even within jurisdictions. Hospitals or health maintenance organisations may be able to negotiate special arrangements with pharmaceutical companies for substantially lower prices than those available to patients or other providers. Unit costs also vary over time due to inflation, but may vary over time due to factors relating to demand, too (e.g. when a drug is indicated for use in an increased range of clinical applications), and supply (e.g. when a drug comes off patent).
Resource use is likely to vary across jurisdictions. In addition to unit costs, the amount of resources used may vary. This is due to a range of factors, including professional practices (e.g. the extent to which a diagnostic test is requested by clinicians for a particular health problem), service settings (e.g. the balance between primary and secondary care), levels of patient adherence, and reimbursement policies.
Resource implications vary widely across jurisdictions. Even when resource use remains constant, resource implications may vary widely across jurisdictions. A year's supply of a very expensive drug may pay one nurse's salary in the United States, six nurses' salaries in Eastern Europe, and 30 nurses' salaries in Africa. What one can buy with the resources saved if one foregoes the purchase of a drug, vaccine or procedure - and the health benefits achievable with those expenditures - may thus differ significantly .
Stakeholders have different perspectives regarding the budgetary envelope in which they are considering resource implications. Individual patients may only be interested in their out-of-pocket costs or may have varying views about risk sharing or who should bear the costs of healthcare. Hospital or district managers who are operating within fixed budgets may consider the cost of an option relative to other possible uses for the same money. Or they may examine the opportunities available to them to shift resources from one use to another. Similarly, a Minister of Health may be interested primarily in healthcare costs and the healthcare budget. Other policymakers, such as those in a Ministry of Finance, may apply a broader perspective and consider the overall government budget, including non-healthcare expenditures and tax increases or reductions.
Conflicting interests related to costs are common. For example, the economic interests of health professionals or industry executives (who typically want to earn as much as possible) may often be in conflict with the interests of society or governments (which typically want to get as much as they can for as little as possible).
Questions to consider
What are the most important impacts on resource use?
What evidence is there for important impacts on resource use?
How confident is it possible to be in the evidence for impacts on resource use?
Have the impacts on resource use been valued appropriately in terms of their true costs?
1. What are the most important impacts on resource use?
Example: Identifying potentially important resource consequences for a national programme of outreach visits to improve prescribing for hypertension
Systematic reviews have found that educational outreach visits (i.e. personal visits to healthcare professionals in their own settings by trained outreach visitors) have relatively consistent and small, but potentially worthwhile, effects on prescribing . In a randomised trial in Norway, these visits were found to increase the use of thiazides, in adherence with clinical practice guidelines, from 11% to 17% among patients with newly diagnosed hypertension . To determine whether this improvement was worthwhile (in relation to the cost of a national outreach programme), the following uses of resources were considered :
• Development of software (used to audit medical records and provide feedback to physicians)
• Training outreach visitors (pharmacists)
• Printed materials
• Travel for the pharmacists doing the outreach visits
• Pharmacists' time
• Administrative time (e.g. making appointments for the outreach visits)
• Physicians' time (for the outreach visits)
• Technical support
• Drug expenditure
• Patient visits
• Laboratory tests
Examples of potentially important resource consequences*
1. Changes in use of healthcare resources
• Policy or programme delivery
- Human resources/time
- Consumable supplies
- Land, buildings, equipment
• Additional (or fewer) hospitalisations, outpatient visits or home visits
• Additional (or less) use of laboratory tests or examinations
• Paid transportation (e.g. emergency transportation)
2. Changes in use of non-healthcare resources
• Transportation to healthcare facilities
• Special diets
• Social services (e.g. housing, home assistance, occupational training)
• Home adaptation
• Crime (such as theft, fraud, violence, police investigation, court costs), for example, in relation to options targeted at drug or alcohol abuse
3. Changes in use of patient and informal caregiver time
• Outpatient visits
• Hospital admissions
• Time of family or other informal caregivers
4. Changes in productivity
• We suggest that changes in productivity and the intrinsic value of changes in health status should be captured in terms of the value or importance attached to health outcomes and should not be included as resource consequences
When considering which impacts on resource use are important it is essential to consider both the resources used to implement the option (i.e. resource inputs such as drugs, equipment and care) and subsequent resource use arising from the impacts of the option on health or other outcomes (e.g. increases or decreases in healthcare utilisation due to the impacts of the option). Incentives to patients to improve adherence to tuberculosis treatment, for example, require substantial resource inputs. These may be offset by subsequent savings if there is a reduction in failed treatment and less spread of the disease (and therefore less subsequent resource use for retreatment and the treatment of others who become infected).
Changes in the productivity of patients may also be important. People with AIDS, for instance, may place a high value on being able to work and earn money, but the process of measuring and valuing actual changes in productivity is controversial . Like others [2, 8, 10], we suggest that such changes in productivity should be considered as components of the intrinsic value of changes in health status, and should not be included as resource consequences.
On the other hand, some outcomes such as hospitalisations or days in hospital can be considered as important in their own right and also as a component of resource use.
When deciding which resource consequences are potentially important it is necessary to specify the viewpoint from which recommendations are made. One option is to adopt a societal perspective: this is a broad viewpoint that includes all important healthcare and non-healthcare resources . This option has the advantage of ensuring that the issue of who pays does not determine whether resource use is included.
Policymakers may sometimes have a remit to make decisions about the use of resources within a healthcare system. In such instances, costs or savings outside of the healthcare system would not be included. This exclusion would not preclude a consideration of the impacts of an option on issues such as social services or crime, in addition to health outcomes. But any costs or savings associated with those impacts would not be relevant to the healthcare budget unless there was a transfer of funds (e.g. from criminal justice to health).
It is also necessary to specify the time horizon for a policy decision (i.e. the period of time for which resource use, as well as health outcomes and other impacts, will be considered).
2. What evidence is there for important impacts on resource use?
Example: Finding evidence for resource consequences. The following data sources were used to estimate the differences in resource use between a programme of outreach visits (targeted at all general practitioners in Norway) and no programme (the status quo) . The programme is described further in Table 1.
Development of software
Invoices, estimates of time spent
Training of outreach visitors
Estimate of time spent; invoices
Record of travel days, estimate of travel distances
Record of number of visits and days spent on visits
Records and estimates of time expenditure
Record of length of outreach visit and number of physicians present
Records of invoices
Systematic reviews, randomised trials and observational studies may provide evidence of the impacts of options on resource use. Such evidence can be published in, or separately from, clinical studies or impact evaluations. The use of resources in specific settings can be retrieved from national or local databases, such as prescription databases for drug use, and hospital databases for information related to hospitalisations .
Evidence of resource use may also come from sources other than those used to obtain evidence of health benefits. This may be the case because:
Trials or impact evaluations (and systematic reviews of these) do not fully report resource use
Trials and impact evaluations may not fully reflect the circumstances - and thus the resource use - in the setting where a policy decision must be made, and
The relevant resource use may extend beyond the duration of the trial or impact evaluation
Evidence of resource use should be in natural units, such as visits, hospitalisations or the number of doses of ACT. There are two reasons for this. Firstly, when only total costs are reported (i.e. the number of units of a resource multiplied by the unit cost of the resource), resource use cannot be separated from unit costs, which might vary considerably between settings and over time. Secondly, without information about resource use it is difficult to make judgements about the validity and the applicability of the evidence.
Unfortunately, studies sometimes report costs but do not report the underlying levels of resource use. This was apparent in an economic evaluation of magnesium sulphate for pre-eclampsia which reported the total cost, but not the resource use for magnesium sulphate, or the resources for administering magnesium sulphate and other hospital resources . Differences in costs could be due to differences in underlying levels of resource use, differences in unit costs, or both.
Often it is not possible to find evidence for components of resource use that are important for policy decisions. A guideline panel convened by WHO to develop recommendations for the prevention of postpartum haemorrhage, for instance, found very limited evidence of resource use for oral misoprostol compared to intramuscular oxytocin . The panel considered hospitalisation, personnel time, and drugs to be potentially important resource consequences but did not find any evidence for the first two types of resources. The resource consequences of these two options for preventing postpartum haemorrhage were therefore very uncertain.
3. How confident is it possible to be in the evidence for impacts on resource use?
The quality of evidence for resource use must be assessed for each important resource consequence (see Table 4). This is because the quality of evidence may be better for some consequences (e.g. drug use) than for other consequences (e.g. personnel time). The criteria for assessing the quality of evidence for resource use are largely the same as those for health outcomes [2, 7, 8, 13]. These include: assessing the study design and other study limitations (i.e. the risk of bias), the precision of the estimate, the consistency of the results, the directness of the evidence (see below), and the risk of publication bias. Factors that often lower the quality of resource evidence (i.e. those that result in less confidence in estimates of resource consequences) include:
The unavailability of data due to resource use not having been measured or reported, or reported only as cost estimates (in other words, without the data upon which those estimates were based)
Weak (observational) study designs
Indirectness due to uncertainty about the transferability of resource evidence from one setting to another, and
Indirectness due to inadequate follow-up periods. This makes it necessary to extrapolate beyond the length of available studies in order to estimate resource consequences
Example: Assessing the quality of evidence for resource consequences. The quality of the evidence for the estimates of difference in resource use between a programme of outreach visits (targeted at all general practitioners in Norway) and no programme (the status quo) varied. (See also Tables 1 and 3.)
Development of software
Training of outreach visitors
Moderate to low quality†
Moderate to low quality†
Laboratory test (potassium)
Moderate to low quality†
Typically, when estimating the cost-effectiveness of a policy or programme, many assumptions must be made. Economic models that are used to estimate cost-effectiveness are valuable given that they can help to make such assumptions explicit. They also allow for sensitivity analyses that test how robust estimates of cost-effectiveness are in relation to those assumptions. It should be noted however, that the various checklists used to assess the quality of economic analyses in the healthcare literature are not constructed to assess the quality of the evidence upon which the analyses were based . Rather, these checklists tend to focus on the quality of the reporting.
Moreover, although published cost-effectiveness analyses can be helpful, particularly for developing a model, they are often of limited value to policymakers when they are not from a policymaker's own setting. The assumptions made and the unit costs that were used may not be transferable from the setting where the analysis was done to one where a decision must be made. Also, as with any research, cost-effectiveness analyses can be flawed. Without knowledge of the complete model it is difficult to make informed judgements about either the quality of the evidence or its applicability [2, 8, 15, 16].
4. Have the impacts on resource use been valued appropriately in terms of their true costs?
Example: Attaching monetary values to resource consequences. The following data sources were used to estimate the monetary value of differences in resource use between a programme of outreach visits (targeted at all general practitioners in Norway) and no programme (the status quo) . (See also Tables 1, 3 and 4.)
Data sources for monetary values
Development of software
Invoices, salary payments
Training of outreach visitors
Salary payments, standard estimates for overheads, office rental figures
Standard tariff for interdisciplinary meetings
"Felleskatalogen 2003" (a Norwegian list of drugs and prices)
Standard tariff for consultation
Laboratory test (potassium)
Cost calculations based on reliable databases or data sources in the same jurisdiction are the most reliable sources of data for unit costs . Monetary valuations of resource use should be made with data that are specific to the context where a policy decision must be made using transparent and locally relevant unit costs. If this is not possible, purchasing power parity (PPP), exchange rates and inflation factors could be used to assist interpretation of monetary valuations from other settings or times . In a study estimating the cost of cervical cancer screening in five developing countries , for example, unit cost data were derived from more than one year. Country-specific deflators were therefore used to adjust all costs to the same price year. Further, to aid cross-country comparability, PPP exchange rates were used to convert costs expressed in local currency units to dollars. Both were measured according to the relevant values in the price year 2000.
Discounting is used in economic evaluations to adjust for social or individual preferences over the timing of costs and health benefits. This means that less weight is given to costs or benefits occurring further in the future than those expected imminently. Recommended discount rates differ between countries and are often varied in sensitivity analyses.
When costs are presented, these should be reported using the appropriate discount rate for the context where the policy decision must be made. Data used to calculate the discounted costs - including quantities of all resource items, unit costs, and the discount rate - should be transparent so that it is possible to assess the validity and applicability or appropriateness of each component.
Evidence of resource use and costs is also needed to inform judgements about equity . In addition to considering the overall costs (and cost-effectiveness) of policies and programmes, policymakers need to consider who will bear particular costs and the impact that this will have on inequities.
In terms of both efficiency and equity it is important to ensure that all potentially important resource consequences are identified. It is also essential that the best available evidence is used, and that important uncertainties about resource (and other) consequences are acknowledged and addressed [23, 24].
Useful documents and further reading
Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Vist GE, Schunemann HJ, and the GRADE Working Group. Incorporating considerations of resource use. BMJ 2008; 336:1170-3.
Brunetti M, Oxman AD, Pregno S, Lord J, Shemilt I, Vale L, et al. GRADE guidelines: 10. Special challenges - resource use. J Clin Epidemiol. In press.
Links to websites
Campbell & Cochrane Economics Methods Group. http://www.c-cemg.org - The Campbell & Cochrane Economic Methods Group is an international network of individuals with an interest and expertise in approaches to evidence synthesis that combine economics and systematic review methods.
GRADE Working Group. http://www.gradeworkinggroup.org/index.htm - The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group has developed a system for grading the quality of evidence and the strength of healthcare recommendations. The system includes an approach to the grading of resource use evidence and the incorporation of evidence into recommendations.
International Health Economics Association. http://www.healtheconomics.org - The International Health Economics Association was formed to increase communication among health economists, foster a higher standard of debate in the application of economics to health and healthcare systems, and assist young researchers at the start of their careers.
Office of Health Economics, United Kingdom. http://www.ohe.org/page/index.cfm - The Office of Health Economics provides independent research, advisory and consultancy services on policy implications and economic issues within the pharmaceutical, healthcare and biotechnology sectors.
CCEMG - EPPI-Centre Cost Converter. http://eppi.ioe.ac.uk/costconversion/default.aspx - a simple web-based tool that can be used to adjust an estimate of cost expressed in one currency and price year, to a target currency and/or price year.
NHS Economic Evaluation Database (NHS EED). http://www.crd.york.ac.uk/crdweb/Home.aspx?DB=NHS%20EED%26SessionID=%26SearchID=%26E=0%26D=0%26H=0%26SearchFor= - NHS EED contains 24,000 abstracts of health economics papers including over 7,000 quality-assessed economic evaluations. The database aims to assist decision makers by systematically identifying and describing economic evaluations, appraising their quality, and highlighting their relative strengths and weaknesses.
Please see the Introduction to this series for acknowledgements of funders and contributors. In addition, we would like to acknowledge Ian Shemilt and Luke Vale for helpful comments on an earlier version of this article.
This article has been published as part of Health Research Policy and Systems Volume 7 Supplement 1, 2009: SUPPORT Tools for evidence-informed health Policymaking (STP). The full contents of the supplement are available online at http://www.health-policy-systems.com/content/7/S1.
- Lavis JN, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). Introduction. Health Res Policy Syst. 2009, 7 (Suppl 1): I1-10.1186/1478-4505-7-S1-I1.PubMed CentralView ArticlePubMedGoogle Scholar
- Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Liberati A, Vist GE, Schunemann HJ, GRADE Working Group: Incorporating considerations of resources use into grading recommendations. BMJ. 2008, 336: 1170-3. 10.1136/bmj.39504.506319.80.PubMed CentralView ArticlePubMedGoogle Scholar
- Quon BS, Firszt R, Eisenberg MJ: A comparison of brand-name drug prices between Canadian-based Internet pharmacies and major U.S. drug chain pharmacies. Ann Intern Med. 2005, 143: 397-403.View ArticlePubMedGoogle Scholar
- Russell LB: Opportunity costs in modern medicine. Health Aff (Millwood). 1992, 11: 162-9. 10.1377/hlthaff.11.2.162.View ArticleGoogle Scholar
- Lavis JN, Oxman AD, Grimshaw J, Johansen M, Boyko JA, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 7. Finding systematic reviews. Health Res Policy Syst. 2009, 7 (Suppl 1): S7-10.1186/1478-4505-7-S1-S7.PubMed CentralView ArticlePubMedGoogle Scholar
- Lewin S, Oxman AD, Lavis JN, Fretheim A, García Martí S, Munabi-Babigumira S: SUPPORT Tools for evidence-informed health Policymaking (STP). 11. Finding and using research evidence about local conditions. Health Res Policy Syst. 2009, 7 (Suppl 1): S11-10.1186/1478-4505-7-S1-S11.PubMed CentralView ArticlePubMedGoogle Scholar
- Oxman AD, Fretheim A, Lavis JN, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 12. Finding and using research evidence about resource use and costs. Health Res Policy Syst. 2009, 7 (Suppl 1): S12-10.1186/1478-4505-7-S1-S12.PubMed CentralView ArticlePubMedGoogle Scholar
- Brunetti M, Oxman A, Pregno S, Lord J, Shemilt I, Vale L, Sisk J, Ruiz F, Hill S, Guyatt GH: GRADE guidelines: 10. Special challenges - resource use. J Clin Epidemiol.Google Scholar
- Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL: Methods for the economic evaluation of health care programmes. 2005, Oxford: Oxford University PressGoogle Scholar
- Luce BR, Manning WG, Siegel JE: Estimating costs in cost-effectiveness analysis. Cost-Effectiveness in Health and Medicine. Edited by: Gold MR, Siegel JE, Russell LB, Weinstein MC. 1996, New York: Oxford University Press, 176-85.Google Scholar
- Lewin S, Oxman AD, Lavis JN, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 8. Deciding how much confidence to place in a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S8-10.1186/1478-4505-7-S1-S8.PubMed CentralView ArticlePubMedGoogle Scholar
- Simon J, Gray A, Duley L: Cost-effectiveness of prophylactic magnesium sulphate for 9996 women with pre-eclampsia from 33 countries: economic evaluation of the Magpie Trial. BJOG. 2006, 113: 144-51. 10.1111/j.1471-0528.2005.00785.x.View ArticlePubMedGoogle Scholar
- Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ: What is "quality of evidence" and why is it important to clinicians?. BMJ. 2008, 336: 995-8. 10.1136/bmj.39490.551019.BE.PubMed CentralView ArticlePubMedGoogle Scholar
- Evers S, Goossens M, de Vet H, van Tulder M, Ament A: Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. Int J Technol Assess Health Care. 2005, 21: 240-5.PubMedGoogle Scholar
- Friedberg M, Saffran B, Stinson TJ, Nelson W, Bennett CL: Evaluation of conflict of interest in economic analyses of new drugs used in oncology. JAMA. 1999, 282: 1453-7. 10.1001/jama.282.15.1453.View ArticlePubMedGoogle Scholar
- Hill SR, Mitchell AS, Henry DA: Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme. JAMA. 2000, 283: 2116-21. 10.1001/jama.283.16.2116.View ArticlePubMedGoogle Scholar
- Palmer S, Raftery J: Economic Notes: opportunity cost. BMJ. 1999, 318: 1551-2.PubMed CentralView ArticlePubMedGoogle Scholar
- Cooper N, Coyle D, Abrams K, Mugford M, Sutton A: Use of evidence in decision models: an appraisal of health technology assessments in the UK since 1997. J Health Serv Res Policy. 2005, 10: 245-50. 10.1258/135581905774414187.View ArticlePubMedGoogle Scholar
- Shemilt I, Mugford M, Byford S, Drummond MF, Eisenstein E, Knapp M, Mallender J, McDaid D, Vale L, Walker D: Chapter 15: Incorporating economics evidence. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1 (updated September 2008). Edited by: Higgins JPT, Green S. 2008, The Cochrane CollaborationGoogle Scholar
- Goldhaber-Fiebert JD, Goldie SJ: Estimating the cost of cervical cancer screening in five developing countries. Cost Eff Resour Alloc. 2006, 4: 13-10.1186/1478-7547-4-13.PubMed CentralView ArticlePubMedGoogle Scholar
- Oxman AD, Lavis JN, Fretheim A, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 16. Using research evidence in balancing the pros and cons of policies. Health Res Policy Syst. 2009, 7 (Suppl 1): S16-10.1186/1478-4505-7-S1-S16.PubMed CentralView ArticlePubMedGoogle Scholar
- Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 10. Taking equity into consideration when assessing the findings of a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S10-10.1186/1478-4505-7-S1-S10.PubMed CentralView ArticlePubMedGoogle Scholar
- Oxman AD, Lavis JN, Fretheim A, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 17. Dealing with insufficient research evidence. Health Res Policy Syst. 2009, 7 (Suppl 1): S17-10.1186/1478-4505-7-S1-S17.PubMed CentralView ArticlePubMedGoogle Scholar
- Fretheim A, Oxman AD, Lavis JN, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 18. Planning monitoring and evaluation of policies. Health Res Policy Syst. 2009, 7 (Suppl 1): S18-10.1186/1478-4505-7-S1-S18.PubMed CentralView ArticlePubMedGoogle Scholar
- O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard-Jensen J, Kristoffersen DT, Forsetlund L, Bainbridge D, Freemantle N, Davis DA: Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007, 4: CD000409-PubMedGoogle Scholar
- Fretheim A, Oxman AD, Treweek S, Bjorndal A: Rational Prescribing in Primary Care (RaPP-trial). A randomised trial of a tailored intervention to improve prescribing of antihypertensive and cholesterol-lowering drugs in general practice. BMC Health Serv Res. 2003, 3: 5-10.1186/1472-6963-3-5.PubMed CentralView ArticlePubMedGoogle Scholar
- Fretheim A, Aaserud M, Oxman AD: Rational prescribing in primary care (RaPP): economic evaluation of an intervention to improve professional practice. PLoS Med. 2006, 3: e216-10.1371/journal.pmed.0030216.PubMed CentralView ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.