Study protocol: a mixed-methods study of women’s healthcare in the safety net after Affordable Care Act implementation – EVERYWOMAN
Health Research Policy and Systems volume 17, Article number: 58 (2019)
Evidence-based reproductive care reduces morbidity and mortality for women and their children, decreases health disparities and saves money. Community health centres (CHCs) are a key point of access to reproductive and primary care services for women who are publicly insured, uninsured or unable to pay for care. Women of reproductive age (15–44 years) comprise just of a quarter (26%) of the total CHC patient population, with higher than average proportions of women of colour, women with lower income and educational status and social challenges (e.g. housing). Such factors are associated with poorer reproductive health outcomes across contraceptive, preventive and pregnancy-related services. The Affordable Care Act (ACA) prioritised reproductive health as an essential component of women’s preventive services to counter these barriers and increase women’s access to care. In 2012, the United States Supreme Court ruled ACA implementation through Medicaid expansion as optional, creating a natural experiment to measure the ACA’s impact on women’s reproductive care delivery and health outcomes.
This paper describes a 5-year, mixed-methods study comparing women’s contraceptive, preventive, prenatal and postpartum care before and after ACA implementation and between Medicaid expansion and non-expansion states. Quantitative assessment will leverage electronic health record data from the ADVANCE Clinical Research Network, a network of over 130 CHCs in 24 states, to describe care and identify patient, practice and state-level factors associated with provision of recommended evidence-based care. Qualitative assessment will include patient, provider and practice level interviews to understand perceptions and utilisation of reproductive healthcare in CHC settings.
To our knowledge, this will be the first study using patient level electronic health record data from multiple states to assess the impact of ACA implementation in conjunction with other practice and policy level factors such as Title X funding or 1115 Medicaid waivers. Findings will be relevant to policy and practice, informing efforts to enhance the provision of timely, evidence-based reproductive care, improve health outcomes and reduce disparities among women. Patient, provider and practice-level interviews will serve to contextualise our findings and develop subsequent studies and interventions to support women’s healthcare provision in CHC settings.
Contributions to the literature
Use of electronic health record and community-level data from the ADVANCE Clinical Research Network will enable broader, more inclusive assessment of policy impacts on women’s healthcare at the patient, provider and practice levels.
Patients in ADVANCE are evenly divided between 24 Medicaid expansion and non-expansion states; our study will provide the most comprehensive and in-depth assessment of state and local policy impacts on women’s healthcare in community health centre settings to date.
Patient, provider and community health centre leader interviews will provide a practical context to our quantitative findings and inform subsequent studies and interventions.
Evidence-based reproductive healthcare decreases morbidity and mortality for women and their children, reduces disparities, and has lasting social and economic benefits for women, their families and their communities [1,2,3,4]. However, many women do not receive needed care, leading the Institute of Medicine to identify reproductive healthcare, including the full spectrum of contraceptive, prenatal, postpartum, and preventive education, counselling and care, as a “critical gap” in women’s preventive services . Women who are poor and from racial and ethnic minorities are the least likely patient population to receive needed reproductive healthcare services . Cost and lack of health insurance are known barriers that contribute to this gap . Reproductive healthcare can result in significant out-of-pocket expenditures for women [8, 9], and even modest co-payments may deter patients from receiving needed preventive services such as mammograms and cervical cancer screening [10, 11]. Gaps in health insurance also put women at risk of not receiving needed preventive services; low-income women are disproportionately vulnerable to gaps in coverage, with approximately 40% reporting that they were uninsured at the end of 2013 . However, when cost barriers are removed, women are more likely to receive needed preventive care . For example, studies suggest that health insurance coverage leads to better reproductive service utilisation  and that women enrolled in Medicaid use preventive care at rates on par with women that have higher incomes and private insurance . Moreover, for women with health insurance, lower out-of-pocket costs are associated with increased utilisation of reproductive services such as intrauterine device placement [15, 16].
Community health centres (CHCs), including Federally Qualified Health Centers (FQHCs) and similar organisations providing primary care services regardless of patients’ ability to pay, are the largest system of primary care for the United States’ low-income, uninsured and publicly insured patients as well as an important source of reproductive healthcare provision. Prior to the expansion of Medicaid under the Affordable Care Act (ACA), CHCs served one in five low-income reproductive age women in the United States . Of the 20 million patients who received care in CHCs in 2011, 25% were women of reproductive age . To adequately serve this population, it is imperative that CHCs provide a broad scope of reproductive healthcare services, including immunisation against HPV, screening for sexually transmitted infections, screening for cervical cancer, preconception and family planning services, and prenatal and postpartum care .
Historically, the federal government has relied on targeted programmes and initiatives to provide reproductive healthcare to low income women. Such efforts include extending Medicaid coverage during pregnancy and the postpartum period to millions of women not otherwise eligible for public insurance, and the Title X National Family Planning Program, which funds contraception and other sexual and reproductive health services for low-income women and men. In 2013, roughly one-quarter of CHCs reported receiving Title X funding  and previous work has found that Title X is the single biggest predictor of the availability of comprehensive reproductive health services in CHC settings [21, 22].
The ACA has the potential to greatly enhance the provision of women’s reproductive healthcare, especially among low-income and vulnerable populations. The ACA prioritised reproductive healthcare as an essential component of women’s preventive services  and expanded insurance coverage to millions of low-income Americans by providing states the opportunity to expand Medicaid coverage to adults with incomes at or below 138% of the Federal Poverty Level. In 2012, the United States Supreme Court ruled that states were not legally required to implement ACA Medicaid expansions, creating a unique natural experiment . As of January 2016, 31 states and the District of Columbia had implemented Medicaid expansion and 19 had not , thus enabling comparison of expansion and non-expansion states. Previous studies suggested that Medicaid expansion is associated with improved healthcare access, increased CHC utilisation and improved health outcomes [25,26,27,28,29,30,31,32,33,34,35,36,37,38,39], but relatively fewer studies have examined the impact on reproductive healthcare [14, 19, 40, 41] and none across CHC populations in multiple states. This protocol paper describes an observational study to assess the impact of the ACA and other individual-, clinic- and state-level factors on the provision and utilisation of reproductive care in low-income and vulnerable populations served by our nation’s safety net CHCs.
Reproductive Care in the Safety Net: Women’s Health after Affordable Care Act Implementation (EVERYWOMAN) is a mixed-methods study to assess differences in reproductive healthcare among women receiving care in CHCs in Medicaid expansion versus non-expansion states before and after ACA implementation. We will augment quantitative analyses leveraging electronic health record (EHR) data at clinic-, visit- and patient-levels with qualitative assessments of patient and provider perceptions of reproductive healthcare and the impact of the ACA and other contextual factors on the full spectrum of women’s healthcare provision in CHC settings. Our specific aims are as follows:
Aim 1: Describe provision of women’s reproductive healthcare before and after implementation of the ACA and between Medicaid expansion and non-expansion states. Hypothesis: Provision of women’s reproductive healthcare will increase post-ACA; increases will be greater in Medicaid expansion vs. non-expansion states.
Aim 2: Identify individual, clinic and state-level factors associated with provision of women’s reproductive healthcare. Hypothesis: Receipt of reproductive healthcare will be associated with (1) individual factors (e.g. age, parity, health insurance, socioeconomic status, frequency of health visits, comorbidities; (2) clinic factors (e.g. gynaecologic procedures, prenatal and obstetric care, receipt of Title X funding); (3) state-level factors (e.g. reproductive health policies, Medicaid expansion, income disparities, health education policies).
Aim 3: Understand provider and patient perceptions of the provision and utilisation of reproductive healthcare in low-income and vulnerable populations in expansion and non-expansion states. Hypothesis: Patterns of reproductive healthcare utilisation and delivery are shaped by patient and provider perceptions and experiences.
We will conduct this study within the ADVANCE (Accelerating Data Value across a National Community Health Center Network) Clinical Research Network, representing over 4.5 million patients from 132 CHCs across 24 states (Fig. 1) . The ADVANCE collaborative is led by OCHIN, a non-profit health information and innovation company, in partnership with the Health Choice Network and Fenway Health. ADVANCE comprises patient EHR data, external clinical information (e.g. labs, imaging), demographic data specific to FQHC (e.g. income and federal poverty level), and geocoded community-level social determinants of health data (e.g. housing, education, income, employment) from publicly available sources such as the American Community Survey and United States Census, linked to individual-level clinical data using information on patient address [42, 43]. The patients in ADVANCE are in majority female (56%), with over half (53%) earning less than 100% of the Federal Poverty Level; 38% have their care covered through Medicaid and 27% are uninsured . ADVANCE is one of the nation’s largest data sources for research to improve equitable health outcomes, policy and primary care delivery in underserved and vulnerable populations .
Qualitative components will be conducted within OCHIN’s Practice-based Research Network (PBRN) . The OCHIN PBRN comprises 91 member organisations using the OCHIN-hosted Epic EHR and contributing data to ADVANCE. The OCHIN PBRN provides research-ready infrastructure and relationships to engage CHCs in qualitative work.
This study uses a mixed-methods design based on a conceptual framework for evaluating how individual and contextual variables influence receipt of needed care in low-income populations . Extending on the Aday–Anderson Behavioral Health Model , which suggests that people’s use of healthcare services are determined by individual predisposing, enabling/hindering and need factors, Davidson et al.  presented a framework that includes additional contextual, or community-level, variables to represent the social, economic, political and environmental/geographic factors that influence utilisation of healthcare in low-income populations. As shown in Fig. 2 below, examples of contextual or community-level factors include the demographic and social composition of the community (e.g. median household income, percentage uninsured, percentage unemployed), public policy support (e.g. Medicaid eligibility levels, Title X funding) and availability of healthcare services (e.g. physician density, FQHC density, availability of reproductive services). At the individual level, predisposing factors include social and demographic characteristics (e.g. age, race/ethnicity, education, language, marital status) associated with decreased access to medical care; enabling/hindering factors are financial or structural factors associated with under-utilisation of healthcare (e.g. insurance status, household income); and need factors are conceptualised as an individual’s underlying health status and their frequency of engagement with the healthcare system. In our application of this framework (Fig. 2), we highlight the individual and community-level factors that may affect reproductive healthcare utilisation, conceptualised as receipt of contraceptive, preventive, prenatal and postpartum services. This schematic facilitates development of hierarchical models to assess multi-level predictors of health service delivery.
Quantitative methods and analysis (Aims 1 and 2)
We will utilise patient-level clinical data from ADVANCE member clinics to describe provision of women’s reproductive services before and after ACA implementation and between expansion and non-expansion states (Aim 1) and to identify individual, clinic and state level factors associated with reproductive service provision over time (Aim 2). Our quantitative sample includes all eligible patients in the ADVANCE dataset (approximately n = 650,000). Patient eligibility is defined as biologically female, age 9–44 years old, and an established patient (as defined by at least two visits within the study period 2011–2016, with at least one visit pre- and one visit post-ACA implementation) at an ADVANCE member site.
Dependent and independent variables
Dependent variables (Table 1) represent recommended reproductive healthcare for women 9–44 years old. Dependent variables are constructed following established quality metric guidance where possible. Independent variables (Table 2) were selected and organised according to our conceptual framework and consensus guidelines [49, 52, 53]. Included variables can be feasibly measured using ADVANCE data (OCHIN has created methods to adapt preventive quality metrics for use with the EHR) [48, 54,55,56,57,58] and are relevant to CHC populations. The ADVANCE research data warehouse includes community-level variables from publicly available sources (e.g. American Community Survey), which are geocoded and linked to patient records using information on patient address . Additional variables will be added over the course of study analysis based on review of state level reproductive health policies.
We will use descriptive statistics and data visualisation (e.g. histograms, scatter plots) to summarise baseline measures across expansion groups. Our primary methodological approach will utilise difference-in-differences (DID) methodology. DID analysis will capture the relative change in patient outcomes attributable to Medicaid expansion using non-expansion states as a reference in comparing outcomes over up to 36 months pre- and up to 36 months post-ACA implementation (Aim 1). Analysis will be extended to include individual, clinic and state-level factors that may be associated with care provision (Aim 2). This will be done through testing of three-way interactions between individual-, clinic- and state-level factors, time, and Medicaid expansion status.
Our DID approach will apply General Linear Mixed Modelling  to produce flexible regression models to accommodate different sources of correlation (serial, intra-clinic and intra-state), categorical and continuous covariates, and fixed and time-dependent covariates. The distribution of the outcomes of interest will be examined before selecting an analysis model; specific models will be refined through an iterative process, guided by the hypotheses, conceptual model and preliminary analyses. If we observe significant patient or clinic differences in states that did or did not expand Medicaid, we will use propensity score weighting methods to reduce the observed bias, help minimise external threats to the validity of the results and adjust for imbalances between expansion and non-expansion groups. Patient and clinic characteristics that remain unbalanced after propensity score weighting will be included in as covariates in General Linear Mixed Modelling to control for residual confounding.
Qualitative methods and analysis
We will use a purposive qualitative sample to include six healthcare organisations in six states, three Medicaid expansion and three non-expansion. In the expansion and non-expansion subsets we will aim to recruit a CHC mix to include prenatal care provision, rural service area and Title X funding. We will attempt to recruit 4–6 patients, 3–5 care team members, and 1–2 health system leaders (e.g. Medical Director, Chief Operating Officer) from each participating CHC. We will conduct a total of approximately 70 interviews, 24–36 with patients and 24–42 across a mix of care providers, clinic staff and CHC leaders.
We will coordinate with each site to identify and coordinate practice-level interviews with providers, staff and CHC leaders such as Practice Managers, Medical Directors or executive leaders. The study team will work closely with each site team to recruit patients from their CHC who are interested in participating in an interview. Each site will receive an impact fee of $1000 in recognition of their time coordinating and supporting qualitative activities. Interviews will be conducted by telephone and will last 45–60 min. Providers and staff will receive $25 each and patients will receive $50 each in appreciation of their time. Semi-structured interview guides are based on our conceptual framework and tailored to each respondent group. Guides will undergo iterative development over the course of the interviews. Interviews will cover perspectives and experiences with routine reproductive and sexual healthcare, including where services are accessed/delivered, perceived barriers and facilitators to care, and perceived impact of the ACA and state-level policy on care access and delivery.
Interviews will be transcribed and entered into NVivo for management of our analysis. We will follow Crabtree and Miller’s five-phase strategy . Phase 1: Describing – the qualitative team will meet regularly to debrief after interviews and provide data immersion. They will provide routine, iterative summaries (crystallisation) of emerging themes for discussion with the study team. Phase 2: Organising – Themes identified in Phase 1 will be organised into a coding template. Phase 3: Connecting – In a second cycle of immersion and crystallisation, the team will review coded data to compare and connect between respondents, sites and themes. Phase 4: Corroborating, legitimising – We will seek additional data sources to confirm, refute, connect or clarify findings. Additional interpretation and review will be solicited from patient and provider representatives and other ADVANCE study teams evaluating ACA impact outcomes in CHC settings. Phase 5: Representing – Final analysis and interpretation will be reported back to target audiences and through broader study dissemination.
We will integrate qualitative and quantitative data after analysis and interpretation consistent with a mixed-method triangulation design quantitative data validation model . This approach is consistent with current guidance on process evaluation of complex interventions that advocates complementing quantitative analyses with in-depth data from purposively selected samples .
Limitations of EHR data
EHR data have inherent limitations, but as noted above, outperform claims and self-reported data. EHR data are not originally developed for research, but our research team has conducted multiple validation studies [55, 64,65,66,67], built many EHR research datasets, and successfully conducted numerous research studies using these datasets in the past [48, 55,56,57, 68,69,70,71].
We anticipate missing data, either from services documented inaccessibly in the EHR (likely random) or from patients who went outside the ADVANCE network to receive services (perhaps not random). Our analyses can accommodate missing data resulting from patient attrition. We will explicitly model missingness by including related variables in the analysis as covariates . If non-trivial levels of missing data are observed, we will use multiple imputation to include these patients in analyses . As with any observational study, unobserved changes may occur over time, making it difficult to isolate the impact of Medicaid expansion. Our approach is strengthened by our plan to perform propensity score methods in situations where there are differences between expansion and non-expansion states, and the use of a DID approach to minimise these biases.
Limitations of qualitative data
Qualitative findings will be limited to a small number of clinic and state contexts due to sample size. Qualitative findings will also be limited in depth given our pragmatic, mixed-methods approach and consideration of the full spectrum of women’s reproductive care rather than a single set of outcomes (e.g. contraceptive care). We will aim to balance these limitations through our qualitative analysis by looking both within and across cases grouped by respondent roles, clinics and Medicaid expansion status and triangulating with quantitative findings [62, 74].
This protocol is based on a proposal reviewed and funded by the Agency for Healthcare Research and Quality in 2017. At the time of submission, the research team is preparing for our first quantitative analyses and recruitment for qualitative components.
The ACA has the potential to greatly enhance the provision of women’s reproductive healthcare, especially among low-income and vulnerable populations
Historically, the federal government has relied on targeted programmes and initiatives to provide reproductive healthcare to low-income women. Since the 1980s, Medicaid has provided special coverage during pregnancy and the postpartum period to millions of women not otherwise eligible for public insurance. Most states set eligibility levels for pregnant women at or near 200% of poverty . Recognising the importance of reproductive healthcare prior to pregnancy, since the mid-1990s, approximately half of the states in the United States have sought and received permission from the federal government to implement Medicaid 1115 waivers to expand coverage for family planning services to individuals not otherwise eligible for Medicaid. The Title X National Family Planning Program has funded nearly 4200 family planning centres that provide high-quality and cost-effective services for low-income women and men. In 2013, roughly one-quarter of CHCs reported receiving Title X funding to provide additional resources necessary to develop the staffing and supplies needed to provide more robust family planning services . Although nearly all CHCs report providing some level of reproductive healthcare, Title X funding is the single biggest predictor of the availability of comprehensive reproductive health services [21, 22], including greater access to a broad range of contraceptive services  such as on-site long-acting reversible contraception methods [77, 78].
Within the boundaries of this federal legislation, states have considerable latitude in how they implement the ACA, including whether to expand Medicaid, determining which preventive services are covered and whether to charge co-payments for those services. As a result, there is likely to be a great deal of variability by state with respect to the number of new women covered under the ACA and differences in both the scope and accessibility of preventive services. Perhaps most important, as a result of the United States Supreme Court ruling that states are not legally required to implement Medicaid expansions , as of January 2016, 31 states and the District of Columbia had implemented expansions and 19 states had not . Estimates from January 2015 show significant increases in Medicaid in enrolment in expansion versus non-expansion states (26% vs. 8%) . Many women directly affected by states’ decisions to expand or not expand Medicaid are seen at CHCs, thus we hypothesise that provision of reproductive healthcare services in CHCs located in expansion states will increase more post-ACA relative to their counterparts in non-expansion states.
This study is uniquely positioned to examine receipt of reproductive health services, an essential component of women’s preventive services, at the patient level across the safety net system
Research to date has focused on the commercially insured population, clinic-level Title X data, Medicaid claims, or data from state family planning programmes. Our study allows us to see across these fragmented payors and to include the uninsured population and those who move in and out of insurance.
Moreover, alongside ACA implementation, complex state policies, social and political environments, and clinic factors will continue to shape the availability and utilisation of reproductive health services in CHCs [21, 77, 78, 80, 81]. Within these contexts, individual and community-level preferences and behaviours and differences in knowledge and attitudes about pregnancy and healthcare also contribute to patterns of reproductive healthcare utilisation and disparities in outcomes [6, 82, 83]. Ultimately, the ACA will have the biggest impact on reproductive healthcare if women not only have access to the benefits, but are also aware of the benefits and act upon them. Experts have heralded the call for research, practice and policy efforts to understand women’s preferences regarding their reproductive healthcare, how and whether CHCs are responding to their needs, and their perceptions and knowledge of the ACA, especially among low-income populations who are particularly vulnerable to disparities in access and outcome . A recent Journal of the American Medical Association editorial stated, “…transforming the system for women’s health is much needed; however, it is only one essential component necessary to realize change in women’s health and birth outcomes. [Reproductive healthcare] will continue to be a complex interplay of human behavior, social mores, technology, and science. The demand for research to explore these nuanced relationships and better understand the needs of women and families will only increase with the removal of traditional barriers to health care access” . Acknowledging the importance of these less tangible factors, in this study we augment our quantitative analysis with qualitative assessments of patient and provider perceptions of reproductive healthcare and the impact of the ACA and other contextual factors on the provision of reproductive healthcare services in CHC settings.
Findings will be relevant to policy and practice, informing efforts to enhance the provision of timely, evidence-based reproductive care, improve health outcomes, and reduce disparities among low-income and vulnerable populations of women. Patient, provider and practice level interviews will serve to contextualise our findings and develop subsequent studies and interventions to support women’s healthcare provision in CHC settings.
Affordable Care Act
Accelerating Data Value across a National Community Health Center Network
community health centre
Electronic health record
Reproductive Care in the Safety Net: Women’s Health after Affordable Care Act Implementation
Federally Qualified Health Center
Practice-based research network
Maciosek MV, Coffield AB, Flottemesch TJ, Edwards NM, Solberg LI. Greater use of preventive services in U.S. health care could save lives at little or no cost. Health Affairs. 2010;29(9):1656–60.
Conde-Agudelo A, Rosas-Bermudez A, Kafury-Goeta AC. Birth spacing and risk of adverse perinatal outcomes: a meta-analysis. JAMA. 2006;295(15):1809–23.
Singh S, Darroch JE, Ashford LS. Adding it Up: The Costs and Benefits of Investing in Sexual and Reproductive Health 2014. Washington, DC: Guttmacher Institute; 2014.
American College of Obstetricians and Gynecologists’ Committee on Health Care for Underserved Women. Committee Opinion No. 654: Reproductive life planning to reduce unintended pregnancy. Obstet Gynecol. 2016;127(2):e66–9.
Rosenstock L, Berg AO, Brindis CD, Diaz A, Nelson HD, Ness RB. Clinical Preventive Services for Women: Closing the Gaps. Washington, DC: Institute of Medicine; 2011.
Dehlendorf C, Rodriguez MI, Levy K, Borrero S, Steinauer J. Disparities in family planning. Am J Obstet Gynecol. 2010;202(3):214–20.
Kaiser. Women’s Health Insurance Coverage. The Henry J. Oakland: Kaiser Family Foundation; 2016.
Bertakis KD, Azari R, Helms LJ, Callahan EJ, Robbins JA. Gender differences in the utilization of health care services. J Fam Pract. 2000;49(2):147–52.
Kjerulff KH, Frick KD, Rhoades JA, Hollenbeak CS. The cost of being a woman: a national study of health care utilization and expenditures for female-specific conditions. Womens Health Issues. 2007;17(1):13–21.
Solanki G, Schauffler HH, Miller LS. The direct and indirect effects of cost-sharing on the use of preventive services. Health Serv Res. 2000;34(6):1331–50.
Trivedi AN, Rakowski W, Ayanian JZ. Effect of cost sharing on screening mammography in Medicare health plans. N Engl J Med. 2008;358(4):375–83.
Salganicoff A, Ranji U, Beamesderfer A, Kurani N. Women and Health Care in the Early Years of the Affordable Care Act: Key Findings from the 2013 Kaiser Women's Health Survey. Oakland: Kaiser Family Foundation; 2014.
Haley S, Jared F. Increasing receipt of women's preventive services. J Womens Health. 2015;24(11):875–81.
Arora P, Desai K. Impact of Affordable Care Act coverage expansion on women's reproductive preventive services in the United States. Prev Med. 2016;89:224–9.
Evans ML, Breeze JL, Paulus JK, Meadows A. Revolving loan fund: a novel approach to increasing access to long-acting reversible contraception methods in community health centers. J Public Health Manage Pract. 2017;23(6):684–9.
Heisel E, Kolenic GE, Moniz MM, Kobernik EK, Minadeo L, Kamdar NS, et al. Intrauterine device insertion before and after mandated health care coverage: the importance of baseline costs. Obstet Gynecol. 2018;131(5):843–9.
Shin P, Sharac J. Role of Community Health Centers in Providing Services to Low-income Women. Washington, DC: George Washington University, School of Public Health and Health Services, Department of Health Policy; 2012.
Wood S, Goetz-Goldberg D, Beeson T, Bruen BK, Johnson K, Mead H, Shin P, Lewis J. Health Centers and Family Planning: Results of a Nationwide Study. Washington, DC: The George Washington University; 2013.
Lanese BG, Oglesby WH. The affordable care act and family planning services: the effect of optional Medicaid expansion on safety net programs. Contracept Reprod Med. 2016;1(1):17.
Wood S, Goetz-Goldberg D, Beeson T, Bruen B, Johnson K, Mead H, et al. Health Centers and Family Planning: A Nationwide Survey. Washington, DC: George Washington University Department of Health Policy; 2013.
Frost JJ, Gold RB, Frohwirth LF, Blades N. Variation in Service Delivery Practices Among Clinics Providing Publicly Funded Family Planning Services in 2010. 2012. https://www.guttmacher.org/report/variation-service-delivery-practices-among-clinics-providing-publicly-funded-family-planning. Accessed 11 Apr 2019.
Hall KS, Fendrick AM, Zochowski M, Dalton VK. Women's health and the Affordable Care Act: high hopes versus harsh realities? Am J Public Health. 2014;104(8):e10–e3.
States, S.C.o.t.U. National Federation of Independent Businesses et al. v. Sebelius, Secretary of Health and Human Services et al. 2011; http://www.supremecourt.gov/opinions/11pdf/11-393c3a2.pdf. Accessed 23 Apr 2019.
Status of State Action on the Medicaid Expansion Decision. Henry J Kaiser Family Foundation; 2018 [updated 9/11/2018]. https://www.kff.org/health-reform/state-indicator/state-activity-around-expanding-medicaid-under-the-affordable-care-act/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Location%22,%22sort%22:%22asc%22%7D. Accessed 23 Oct 2018
Piper JM, Ray WA, Griffin MR. Effects of Medicaid eligibility expansion on prenatal care and pregnancy outcome in Tennessee. JAMA. 1990;264(17):2219–23.
Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH, Newhouse JP, et al. The Oregon experiment--effects of Medicaid on clinical outcomes. N Engl J Med. 2013;368(18):1713–22.
Sommers BD, Arntson E, Kenney GM, Epstein AM. Lessons from early Medicaid expansions under health reform: interviews with Medicaid officials. Medicare Medicaid Res Rev. 2013;3(4):1010–8.
Shane DM, Ayyagari P. Will health care reform reduce disparities in insurance coverage?: Evidence from the dependent coverage mandate. Med Care. 2014;52(6):528–34.
Alegria M, Lin J, Chen CN, Duan N, Cook B, Meng XL. The impact of insurance coverage in diminishing racial and ethnic disparities in behavioral health services. Health Serv Res. 2012;47(3 Pt 2):1322–44.
Langellier BA, Guernsey de Zapien J, Rosales C, Ingram M, Carvajal SC. State Medicaid expansion, community interventions, and health care disparities in a United States-Mexico border community. Am J Public Health. 2014;104(8):e94–e100.
Fiscella K, Franks P, Gold MR, Clancy CM. Inequality in quality: addressing socioeconomic, racial, and ethnic disparities in health care. JAMA. 2000;283(19):2579–84.
Burns ME, Dague L, DeLeire T, Dorsch M, Friedsam D, Leininger LJ, et al. The effects of expanding public insurance to rural low-income childless adults. Health Serv Res. 2014;49(Suppl 2):2173–87.
DeVoe JE, Hoopes M, Nelson CA, Cohen DJ, Sumic A, Hall J, et al. Electronic health record tools to assist with children’s insurance coverage: a mixed methods study. BMC Health Serv Res. 2018;18(1):354.
Huguet N, Hoopes MJ, Angier H, Marino M, Holderness H, DeVoe JE. Medicaid expansion produces long-term impact on insurance coverage rates in community health centers. J Prim Care Community Health. 2017;8(4):206–12.
Pourat N, Bonilla AG, Young ME, Rodriguez MA, Wallace SP. There and back again: how the repeal of ACA can impact community health centers and the populations they serve. Fam Commun Health. 2018;41(2):83–94.
Cole MB, Wright B, Wilson IB, Galárraga O, Trivedi AN. Medicaid expansion and community health centers: care quality and service use increased for rural patients. Health Aff. 2018;37(6):900–7.
VanGarde A, Yoon J, Luck J, Mendez-Luck CA. Racial/ethnic variation in the impact of the affordable care act on insurance coverage and access among young adults. Am J Public Health. 2018;108(4):544–9.
Decker E, Ahrens KI, Fowler C, Carter M, Gavin L, Moskosky S. Trends in Health Insurance Coverage of Title X Family Planning Program Clients, 2005–2015. J Womens Health. 2018;27(5):684–90.
Johnston EM, Strahan AE, Joski P, Dunlop AL, Adams EK. Impacts of the Affordable Care Act's Medicaid expansion on women of reproductive age: differences by parental status and state policies. Womens Health Issues. 2018;28(2):122–9.
Smith AJB, Fader AN. Effects of the Affordable Care Act on young women with gynecologic cancers. Obstet Gynecol. 2018;131(6):966–76.
Lipton BJ, Decker SL. ACA provisions associated with increase in percentage of young adult women initiating and completing the HPV vaccine. Health Affairs. 2015;34(5):757–64.
ADVANCE Collaborative. Portland, Oregon. 2018. http://advancecollaborative.org/. Accessed 21 Nov 2018.
A Short Guide to the ADVANCE Research Data Warehouse. 2018. http://advancecollaborative.org/wp-content/uploads/2018/11/Short-Guide-to-the-ADVANCE-RDW-v2018-10-26.pdf. Accessed 20 Nov 2018.
OCHIN. Portland, Oregon. 2018. http://www.ochin.org. Accessed 11 Oct 2018.
Davidson PL, Andersen RM, Wyn R, Brown ER. A framework for evaluating safety-net and other community-level factors on access for low-income populations. Inquiry. 2004;41(1):21–38.
Aday LA, Andersen R, Anderson OW. Social surveys and health policy implications for national health insurance. Public Health Rep. 1977;92(6):508–17.
Performance Measures. Washington, DC: Office of Population Affairs; 2018. https://www.hhs.gov/opa/performance-measures/index.html.
Gold R, DeVoe J, Shah A, Chauvie S. Insurance continuity and receipt of diabetes preventive care in a network of federally qualified health centers. Med Care. 2009;47(4):431–9.
Status of State Action on the Medicaid Expansion Decision. Oakland: Kaiser Family Foundation; 2016
American Community Survey (ACS). United States Census Bureau; 2018. https://www.census.gov/programs-surveys/acs.html.
Butler DC, Petterson S, Phillips RL, Bazemore AW. Measures of social deprivation that predict health care access and need within a rational area of primary care service delivery. Health Serv Res. 2013;48(2 Pt 1):539–59.
National Cancer Institute. Cancer Stat Facts: Cervical Cancer. Surveillance, Epidemiology, and End Results Program. 2015.
Guttmacher Institute. State Policy Updates: Major Developments in Sexual & Reproductive Health. 2016. https://www.guttmacher.org/state-policy.
Gold R, Angier H, Mangione-Smith R, Gallia C, McIntire PJ, Cowburn S, et al. Feasibility of evaluating the CHIPRA care quality measures in electronic health record data. Pediatrics. 2012;130(1):139–49.
Angier H, Gold R, Gallia C, Casciato A, Tillotson CJ, Marino M, et al. Variation in outcomes of quality measurement by data source. Pediatrics. 2014;133(6):e1676–82.
Bailey SR, O'Malley JP, Gold R, Heintzman J, Marino M, DeVoe JE. Receipt of diabetes preventive services differs by insurance status at visit. Am J Prev Med. 2015;48(2):229–33.
Cowburn S, Carlson MJ, Lapidus JA, DeVoe JE. The association between insurance status and cervical cancer screening in community health centers: exploring the potential of electronic health records for population-level surveillance, 2008-2010. Prev Chronic Dis. 2013;10:E173.
DeVoe JE, Gold R, Cottrell E, Bauer V, Brickman A, Puro J, et al. The ADVANCE network: accelerating data value across a national community health center network. J Am Med Inform Assoc. 2014;21(4):591–5.
Bazemore AW, Cottrell EK, Gold R, Hughes LS, Phillips RL, Angier H, et al. “Community vital signs”: incorporating geocoded social determinants into electronic records to promote patient and population health. J Am Med Inform Assoc. 2016;23(2):407–12.
Dean CB, Nielsen JD. Generalized linear mixed models: a review and some extensions. Lifetime Data Anal. 2007;13(4):497–512.
Crabtree BF, Miller WL. Doing qualitative research. 2nd ed. Thousand Oaks: Sage Publications; 1999. p. xvii. 406
Creswell JW, Plano Clark VL. Designing and Conducting Mixed-Methods Research. 3rd ed. Los Angeles: SAGE Publishing; 2018.
Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.
Devoe JE, Gold R, McIntire P, Puro J, Chauvie S, Gallia CA. Electronic health records vs Medicaid claims: completeness of diabetes preventive care data in community health centers. Ann Fam Med. 2011;9(4):351–8.
Heintzman J, Bailey SR, Hoopes MJ, Le T, Gold R, O'Malley JP, et al. Agreement of Medicaid claims and electronic health records for assessing preventive care quality among adults. J Am Med Inform Assoc. 2014;21(4):720–4.
Heintzman J, Marino M, Hoopes M, Bailey SR, Gold R, O'Malley J, et al. Supporting health insurance expansion: do electronic health records have valid insurance verification and enrollment data? J Am Med Inform Assoc. 2015;22(4):909–13.
Marino M, Angier H, Valenzuela S, Hoopes M, Killerby M, Blackburn B, et al. Medicaid coverage accuracy in electronic health records. Prev Med Rep. 2018;11:297–304.
Angier H, Gold R, Crawford C, P O’Malley J, J Tillotson C, Marino M, et al. Linkage methods for connecting children with parents in electronic health record and state public health insurance data. Matern Child Health J. 2014;18(9):2025–33.
Angier H, Hoopes M, Gold R, Bailey SR, Cottrell EK, Heintzman J, et al. An early look at rates of uninsured safety net clinic visits after the Affordable Care Act. Ann Fam Med. 2015;13(1):10–6.
Angier H, Likumahuwa S, Finnegan S, Vakarcs T, Nelson C, Bazemore A, et al. Using geographic information systems (GIS) to identify communities in need of health insurance outreach: An OCHIN practice-based research network (PBRN) report. J Am Board Fam Med. 2014;27(6):804–10.
Bailey SR, O'Malley JP, Gold R, Heintzman J, Likumahuwa S, DeVoe JE. Diabetes care quality is highly correlated with patient panel characteristics. J Am Board Fam Med. 2013;26(6):669–79.
Little TD, Schnabel KU, Jr B. Modeling Longitudinal and Multilevel Data: Practical Issues, Applied Approaches, and Specific Examples. Mahwah: Lawrence Erlbaum Associates; 2000. p. 297. vii
Little RJA, Rubin DB. Statistical Analysis with Missing Data. 2nd ed. Hoboken: Wiley; 2002. p. 381. xv
Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117.
Heberlein M, Brooks T, Guyer J, Artiga S, Stephens J. Holding Steady, Looking Ahead: Annual Findings OF A 50-State Survey of Eligibility Rules, Enrollment and Renewal Procedures, AND Cost Sharing Practices in Medicaid and CHIP, 2010-2011. Kaiser Commission on Medicaid and the Uninsured. Oakland: Kaiser Family Foundation; 2011.
Thiel de Bocanegra H, Rostovtseva D, Menz M, Karl J, Darney P. The 2007 Family PACT Medical Record Review: Assessing the Quality of Services. Sacramento: Bixby Center for Global Reproductive Health; 2009. http://bixbycenter.ucsf.edu/publications/files/MRR%20Full%20Report.pdf
Beeson T, Wood S, Bruen B, Goldberg DG, Mead H, Rosenbaum S. Accessibility of long-acting reversible contraceptives (LARCs) in Federally Qualified Health Centers (FQHCs). Contraception. 2014;89(2):91–6.
Bornstein M, Carter M, Zapata L, Gavin L, Moskosky S. Access to long-acting reversible contraception among US publicly funded health centers. Contraception. 2018;97(5):405–10.
Artiga S, Rudowitz R, Gates A, Snyder L. Recent Trends in Medicaid and CHIP Enrollment as of January 2015: Early Findings from the CMS Performance Indicator Project. Oakland: Kaiser Family Foundation; 2015.
Fernandez LM, Becker JA. Women’s Select Health Issues in Underserved Populations. Prim Care. 2017;44(1):47–55.
Pazol K, Robbins CL, Black LI, Ahrens KA, Daniels K, Chandra A, et al. Receipt of selected preventive health services for women and men of reproductive age – United States, 2011–2013. MMWR Surveill Summ. 2017;66(SS-20):1–31.
Oglesby WH. Perceptions of and preferences for federally-funded family planning clinics. Reprod Health. 2014;11:50.
Janiak E, Clark J, Bartz D, Langer A, Gottlieb B. Barriers and pathways to providing long-acting reversible contraceptives in Massachusetts community health centers: a qualitative exploration. Perspect Sex Reprod Health. 2018;50(3):111–8.
Saleeby E, Brindis CD. Women, reproductive health, and health reform. JAMA. 2011;306(11):1256–7.
The authors deeply appreciate the contributions of Andrew Brickman, Dana Crosby, MaryAnn McBurnie, Jee Oakley, Jennifer Potter and Teresa Schmidt.
EVERYWOMAN is funded by the Agency for Healthcare Research and Quality award 1R01HS025155–01. EVERYWOMAN is conducted with the ADVANCE (Accelerating Data Value across a National Community Health Center Network) Clinical Research Network, a member of the People-Centered Research Foundation Consortium, an initiative of the Patient Centered Outcomes Research Institute (PCORI). The ADVANCE network is led by OCHIN in partnership with the Health Choice Network, Fenway Health, Oregon Health and Science University, and the Robert Graham Center/HealthLandscape. ADVANCE is funded through People-Centered Research Foundation contract number 1237.
Availability of data and materials
The datasets generated and analysed in this study are not publicly available because they are from a privately hosted electronic health record system but are available through the corresponding author on reasonable request.
Ethics approval and consent to participate
This study has been reviewed and approved by the Western Institutional Review Board.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cottrell, E., Darney, B.G., Marino, M. et al. Study protocol: a mixed-methods study of women’s healthcare in the safety net after Affordable Care Act implementation – EVERYWOMAN. Health Res Policy Sys 17, 58 (2019). https://doi.org/10.1186/s12961-019-0445-y