This article has Open Peer Review reports available.
Integrating and evaluating sex and gender in health research
© The Author(s). 2016
Received: 24 August 2016
Accepted: 22 September 2016
Published: 10 October 2016
Both sex (biological factors) and gender (socio-cultural factors) shape health. To produce the best possible health research evidence, it is essential to integrate sex and gender considerations throughout the research process. Despite growing recognition of the importance of these factors, progress towards sex and gender integration as standard practice has been both slow and uneven in health research. In this commentary, we examine the challenges of integrating sex and gender from the research perspective, as well as strategies that can be used by researchers, funders and journal editors to address these challenges. Barriers to the integration of sex and gender in health research include problems with inconsistent terminology, difficulties in applying the concepts of sex and gender, failure to recognise the impact of sex and gender, and challenges with data collection and datasets. We analyse these barriers as strategic points of intervention for improving the integration of sex and gender at all stages of the research process. To assess the relative success of these strategies in any given study, researchers, funders and journal editors would benefit from a tool to evaluate the quality of sex and gender integration in order to establish benchmarks in research excellence. These assessment tools are needed now amidst growing institutional recognition that both sex and gender are necessary elements for advancing the quality and utility of health research evidence.
Sex and gender in health research: a persisting data gap
To inform the development of effective and equitable healthcare policies, programs and delivery models, health researchers have long been called upon to examine the impact of sex and gender on health outcomes, health-related behaviours and service provision [1–4]. A number of international funding mechanisms have developed policies requiring sex and gender to be integrated in research proposals, including WHO, The European Commission Directorate-General for Research and Innovation, the Canadian Institutes of Health Research (CIHR), and the United States National Institutes of Health . The European Association of Scientific Editors has formed a Gender Policy Committee to improve sex and gender reporting practices across all scientific fields . Some peer-reviewed journals have also developed sex and gender reporting policies . Building on this momentum, a global call to action has been issued for gender to be included in the research impact assessment undertaken by research funders, institutions and evaluators in order to inform more equitable health policy and practice .
Despite this widespread and growing recognition that including sex and gender is integral to better evidence, the need for better integration has been acknowledged across a range of health and biomedical fields [9–18]. A recent study of research designs in funding applications to CIHR found that uptake of sex and gender is uneven in health research and varies by discipline: sex was most often incorporated in clinical research, gender most often in population health research, and both sex and gender were least often incorporated in biomedical studies . Additionally, reporting on sex and gender in health research publications has yet to be adopted as standard practice [10, 19]. A review of sex and gender in medical literature found that, while incorporation of these factors in study design and evaluation has increased over time, the total number of publications remains small and underrepresentation of sex and gender differences is widespread . Similarly, a review of sex and gender in randomised controlled trials in high-impact international journals found that only 6% of the reviewed papers that included both men and women examined interactions between sex or gender and other variables of interest in the study . Clearly, challenges persist.
As one of 12 research centres comprising the Ontario Strategy for Patient-Oriented Research Support for People and Patient-Oriented Research and Trials Unit, Women’s Xchange works with health research teams to improve the quality of sex and gender integration throughout all stages of their research projects. In this commentary, we expand upon recent observations that a tendency towards ‘path dependency’ has hindered progress in considering sex and gender in health research systems , offering some insights into the difficulties of changing the status quo in health knowledge production. Drawing on our experiences in consulting on health research projects, we identify and examine the persisting challenges with sex and gender integration from the researcher perspective. We discuss some of the strategies that have been proposed by sex and gender scholars to address these challenges, concluding with a call to evaluate how well these variables have been integrated in health research.
The challenges in (and strategies for) sex and gender integration
From the researcher perspective, we have identified four interrelated challenges to incorporating sex and gender in health research, which can be summarised as follows: the use of inconsistent terminology, difficulties in applying the concepts, a failure to recognise the impact of sex and gender on research design and outcomes, and challenges with data collection and datasets. Each will be explored below, along with potential strategies for moving forward.
Canadian Institutes of Health Research (CIHR) definitions of sex and gender 
The biological and physiological characteristics that distinguish males from femalesa
The socially constructed roles, expectations, relationships, behaviours, relative power, and other traits that societies ascribe to women, men and people of diverse gender identitiesb
While these definitions may appear to categorise sex and gender as mutually exclusive concepts, in reality, sex and gender are interrelated and intersect. The body is at once both biological and social , and thus physical health is simultaneously influenced by both sex (the biological) and gender (the social). For example, both biological processes as well as factors shaped by gender (including clothing, physical activity and occupation) are implicated in bone health . Similarly, differences in both sex (e.g. gene expression, protein patterns) and gender (e.g. patients’ health seeking behaviour, physicians’ disease management strategies) are factors in the manifestation and outcomes of heart disease . Agreement on definitions of sex and gender that can be used across the fields of health research – as well as accurate use of each term – is an important first step in working through these complexities .
Applying the concepts
Applying the concepts of sex and gender represents another challenge for health researchers. Examples of how sex and gender can influence the spectrum of health and disease include gene expression, hormone levels and enzymatic processes (biological: sex) and differences in societal roles, societal perceptions and factors affecting healthcare access (social: gender) . Capturing these variables requires asking different research questions, designing and taking up different kinds of research, and identifying and observing different phenomena. The meaningful integration of sex and gender requires attending to these factors from the outset. Thinking about sex and gender will inform the formation of and approach to research questions and the literature review, shaping the orientation of the research through to analysis, interpretation, reporting and recommendations . Proactively considering sex and gender from the beginning thus helps to avoid the costly, inefficient and insufficient revisiting of research to ‘add in’ sex and gender post-hoc .
Part of the challenge in applying the concepts of sex and gender is with determining the explanatory limitations of these concepts. For instance, disaggregating data by sex may illuminate differences between males and females but will not allow exploration of the gender-based inequities in social and economic power that additionally shape health . Furthermore, finding that there are sex differences in a study outcome does not necessarily tell us anything about the biological mechanisms behind this difference. To address this challenge, it is important to ensure that mechanisms be directly tested wherever possible . For example, attributing occupational injuries to assumed sex differences without investigating the more precise explanatory power of differences in body size can impact the design and implementation of safety measures . Misidentifying the influential mechanism thus has consequences not only for how sex and gender differences are explained, but also for developing preventative strategies.
To further challenge the application of sex and gender in research, intersectionality has emerged as an important, additional consideration. Intersectionality describes the recognition that sex and gender intersect simultaneously with other factors that shape health outcomes, including age, race and ability, among others, all from within specific historical, geographic and cultural contexts [31, 32]. In this broader and more nuanced view, sex and gender are embedded within and inseparable from a multitude of factors that influence health and wellbeing . More sophisticated analyses will require health researchers to consider the ways that sex and gender interact with these additional variables in order to develop more effective and equitable policy and practice.
Recognising the impact
As illustrated by the uneven uptake of sex and gender considerations across the health disciplines , challenges also remain with acknowledging and understanding the impact that these concepts can have on the evidence produced by health research. This is particularly the case in basic experimental biomedical research where sex and gender considerations may be less readily perceived as relevant in working with non-human study populations such as cells or laboratory animals . There is also a tendency for sex and gender to be treated as an ‘extra’ or optional consideration and thus not necessarily incorporated as a standard research practice  – or only incorporated in a tokenistic way in order to satisfy funding application requirements . In order to shift perspectives and practice in a meaningful way, one strategy is to emphasise that sex and gender integration leads to better health through novel research and medical discovery. Failure to consider these concepts is to risk significant sources of error, endangering patient safety . Sex and gender are also crucial for developing personalised medical therapies , a particularly salient contribution amidst growing emphasis on patient-centred care. This requires attending not only to sex/gender differences, but also sex/gender-specific conditions .
Data collection and datasets
Finally, health researchers may face challenges with obtaining the relevant data for analysing by sex and gender. This is particularly the case for health research that relies on large administrative datasets. While administrative data may often (though not always) contain basic sex disaggregated data, indicators pertinent to gender analysis (e.g. income, household composition, caregiving responsibilities) are rarely collected . Researchers cannot be expected to take up a more nuanced examination of sex and gender when data collection tools do not adequately incorporate these concepts. An important strategy for moving forward will be to consider how datasets can be expanded to collect information beyond clinical indicators with limited explanatory power, as well as alternative methods and mixed-methods analyses to obtain different kinds of data. This approach is particularly salient given that attending to the influence of gender requires asking multiple and multifaceted questions, as there is no one standard way to measure gender .
Next steps: evaluating sex and gender integration
Web-based resources for sex and gender inclusion in health research
The Gender Awakening Tool 
Provides a checklist to determine whether relevant sex and gender considerations have been included in life sciences research
Gendered Innovations 
Provides methods for integrating sex and gender into each stage of the research process, from design to knowledge translation
Sex and Gender in Systematic Reviews: Planning Tool 
Provides a step-by-step guide for asking questions about sex and gender when planning a systematic review
Toolkit Gender in EU Funded Research 
Provides guidance in conducting gender-sensitive scientific research
There is additionally a need to carefully consider the quality of sex and gender integration in health research and whether the uptake of these concepts is occurring in a meaningful way (when it happens at all). While existing guidelines and checklists are useful starting points for thinking through the inclusion and impact of sex and gender, to date, we are unaware of any accompanying evaluation system to assess the relative success of this integration throughout a given project. In other words, there is currently a lack of standards by which to assess whether and to what degree researchers are rising to the challenges of sex and gender integration. One notable attempt to reconcile this gap is offered by Tomas et al. , who have developed a questionnaire that allows reviewers to distinguish between three levels in assessing the strength of sex/gender integration in a project. However, this tool is limited to yes/no answers, namely whether sex/gender has been considered at a given stage in the project or not, which does not allow either reviewers or researchers to evaluate the relative quality of sex/gender integration throughout the project.
We propose that a scaled evaluation strategy will help to address calls to incorporate attention to sex and gender in the peer review decision-making process , giving reviewers a standardised protocol for assessing how sex and gender impact the rigor of a study. A scaled evaluation will also help to clarify what constitutes the ‘best practices’ for integrating sex and gender, addressing recent calls for researchers to be connected with such resources to foster routine uptake of sex and gender considerations . To address these needs, and as a part of our mandate to promote sex and gender as ‘the key to better research’, our team at Women’s Xchange is presently engaged in developing and validating an assessment scale for the integration of sex and gender, applicable to both researchers and reviewers.
Conclusions: sex and gender for stronger evidence and improved health
There is a clear need to strengthen our collective efforts to promote the integration of sex and gender as standard practice in health research. Examining the influence of sex and gender on health is an essential strategy to promote innovative, useful health research. While a number of resources have been developed towards this purpose, researchers and reviewers require more comprehensive strategies for both integration and evaluation if we are to advance the uptake of sex and gender in health research in support of better evidence. This objective has far-reaching implications for shaping decision-making at the policy and planning level. Consider, for example, the recent directive publically issued to the Minister of the Status of Women by Canada’s new federal government shortly after being elected in October 2015, calling for attention to gender in decision-making. Advancing the integration of sex and gender in health research will provide decision-makers with the necessary evidence “to ensure government policy, legislation, and regulations are sensitive to the different impacts that decisions can have on men and women” .
The authors would like to acknowledge the support of the Ontario Ministry of Health and Long-Term Care and the Ontario Strategy for Patient Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit.
Women’s Xchange is funded by the Ontario Ministry of Health and Long-Term Care and by the Ontario Strategy for Patient Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit.
SD wrote the initial draft of the manuscript with substantial input from RM. SD, RM, SL and PR all reviewed and provided critical feedback for revising and finalising the manuscript. All authors read and approved the final manuscript.
SD is a sociologist specialising in gender, health and long-term care. RM is a psychologist with expertise in public health and medical education. SL is a project manager with expertise in knowledge translation. PR is a health services researcher, geriatrician and RTO/ERO Chair in Geriatric Medicine, with a particular focus on older women.
The authors of this paper are the team behind Women’s Xchange, a women’s health research knowledge translation and exchange centre based at Women’s College Hospital in Toronto, Canada. A mandate of Women’s Xchange is to enhance the integration of sex and gender in health research.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Bierman AS. Sex matters: gender disparities in quality and outcomes of care. CMAJ. 2007;177(12):1520–1.View ArticlePubMedPubMed CentralGoogle Scholar
- Coen S, Banister E. What a difference sex and gender make: a gender, sex and health research casebook. Ottawa: Canadian Institutes of Health Research; 2012.Google Scholar
- Doyal L. Sex, gender, and health: the need for a new approach. BMJ. 2001;323(7320):1061–3.View ArticlePubMedPubMed CentralGoogle Scholar
- Greaves L. Why put gender and sex into health research? In: Oliffe JL, Greaves L, editors. Designing and Conducting Gender, Sex and Health Research. Thousand Oaks: Sage Publications Inc.; 2011. p. 3–14.Google Scholar
- Gendered Innovations. Sex and gender analysis policies of major granting agencies. 2015. https://genderedinnovations.stanford.edu/sex-and-gender-analysis-policies-major-granting-agencies.html. Accessed 5 Aug 2016.Google Scholar
- European Association of Scientific Editors. Gender Policy Committee. 2014. http://www.ease.org.uk/about-us/gender-policy-committee/. Accessed 5 Aug 2016.Google Scholar
- Gendered Innovations. Sex and gender analysis policies of peer-reviewed journals. 2015. https://genderedinnovations.stanford.edu/sex-and-gender-analysis-policies-peer-reviewed-journals.html. Accessed 5 Aug 2016.Google Scholar
- Ovseiko PV, Greenhalgh T, Adam P, Grant J, Hinrichs-Krapels S, Graham KE, Valentine PA, Sued O, Boukhris OF, Al Olaqi NM, Al Rahbi IS, Dowd AM, Bice S, Heiden TL, Fischer MD, Dopson S, Norton R, Pollitt A, Wooding S, Balling GV, Jakobsen U, Kuhlmann E, Klinge I, Pololi LH, Jagsi R, Smith HL, Etzkowitz H, Neilsen MW, Carrion C, Solans-Domènech M, Vizcaino E, Naing L, Cheok QHN, Eckelmann B, Simuyemba MC, Msiska T, Declich G, Edmunds LD, Kiparoglou V, Buchan AMJ, Williamson C, Lord GM, Channon KM, Surender R, Buchan AM. A global call for action to include gender in research impact assessment. Health Res Policy Syst. 2016;14:50.View ArticlePubMedPubMed CentralGoogle Scholar
- Johnson J, Sharman Z, Vissandjeé B, Stewart DE. Does a change in health research funding policy related to the integration of sex and gender have an impact? PLoS One. 2014;9(6):e99900.View ArticlePubMedPubMed CentralGoogle Scholar
- Johnson J, Beaudet A. Sex and gender reporting in health research: why Canada should be a leader. Can J Public Health. 2013;104(1):e80–1.Google Scholar
- Mazure CM, Jones DP. Twenty years and still counting: including women as participants and studying sex and gender in biomedical research. BMC Womens Health. 2015;15:94.View ArticlePubMedPubMed CentralGoogle Scholar
- Nieuwenhoven L, Klinge I. Scientific excellence in applying sex- and gender-sensitive methods in biomedical and health research. J Women’s Health. 2010;19(2):313–21.View ArticleGoogle Scholar
- Nowatzki N, Grant KR. Sex is not enough: the need for gender-based analysis in health research. Health Care Women Int. 2011;32(4):263–77.View ArticlePubMedGoogle Scholar
- Oertelt-Prigione S. Sex and gender in medical literature. In: Oertelt-Prigione S, Regitz-Zagrosek V, editors. Sex and Gender Aspects in Clinical Medicine. London: Springer; 2012. p. 9–15.View ArticleGoogle Scholar
- Ritz SA, Antle DM, Côté J, Deroy K, Fraleigh N, Messing K, Parent L, St-Pierre J, Vaillancourt C, Mergler D. First steps for integrating sex and gender considerations into basic experimental biomedical research. FASEB J. 2014;28(1):4–13.View ArticlePubMedGoogle Scholar
- Sharman Z, Johnson J. Towards the inclusion of gender and sex in health research and funding: an institutional perspective. Soc Sci Med. 2012;74(11):1812–6.View ArticlePubMedGoogle Scholar
- Schiebinger L, Stefanick ML. Gender matters in biological research and medical practice. J Am Coll Cardiol. 2016;67(2):136–8.View ArticlePubMedGoogle Scholar
- Greyson DL, Becu AR, Morgan SG. Sex, drugs and gender roles: mapping the use of sex and gender based analysis in pharmaceutical policy research. Int J Equity Health. 2010;9:26.View ArticlePubMedPubMed CentralGoogle Scholar
- Gahagan J, Gray K, Whynacht A. Sex and gender matter in health research: addressing health inequities in health research reporting. Int J Equity Health. 2015;14:12.View ArticlePubMedPubMed CentralGoogle Scholar
- Oertelt-Prigione S, Parol R, Krohn S, Preißner R, Regitz-Zagrosek V. Analysis of sex and gender-specific research reveals a common increase in publications and marked differences between disciplines. BMC Med. 2010;8:70.View ArticlePubMedPubMed CentralGoogle Scholar
- Philips SP, Hamberg K. Doubly blind: a systematic review of gender in randomised controlled trials. Glob Health Action. 2016;9:29597.View ArticleGoogle Scholar
- King BM. Point: a call for proper usage of “gender” and “sex” in biomedical publications. Am J Physiol Regul Integr Comp Physiol. 2010;298(6):R1700–1.View ArticlePubMedGoogle Scholar
- Ristvedt SL. The evolution of gender. JAMA Psychiat. 2014;71(1):13–4.View ArticleGoogle Scholar
- Canadian Institutes of Health Research. Gender, sex and health research guide: a tool for CIHR applicants. 2010. http://www.cihr-irsc.gc.ca/e/32019.html. Accessed 5 Aug 2016.Google Scholar
- Fausto-Sterling A. The bare bones of sex: part 1 – sex and gender. Signs. 2005;30(2):1491–527.View ArticleGoogle Scholar
- Regitz-Zagrosek V. Sex and gender differences in cardiovascular disease. In: Oertelt-Prigione S, Regitz-Zagrosek V, editors. Sex and gender aspects in clinical medicine. London: Springer; 2012. p. 17–44.View ArticleGoogle Scholar
- Christianson M, Alex L, Wiklund AF, Hammarström A, Lundman B. Sex and gender traps and springboards: a focus group study among gender researchers in medicine and health sciences. Health Care Women Int. 2012;33(8):739–55.View ArticlePubMedGoogle Scholar
- Schiebinger L, Klinge I. Gendered innovation in health and medicine. Gender. 2015;7(2):29–50.View ArticleGoogle Scholar
- Springer KW, Stellman JM, Jordan-Young RM. Beyond a catalogue of differences: a theoretical frame and good practice guidelines for researching sex/gender in human health. Soc Sci Med. 2012;74(11):1817–24.View ArticlePubMedGoogle Scholar
- Messing K, Stellman JM. Sex, gender and women’s occupational health: the importance of considering mechanism. Environ Res. 2006;101(2):149–62.View ArticlePubMedGoogle Scholar
- Hankivsky O. Women’s health, men’s health, and gender and health: implications of intersectionality. Soc Sci Med. 2012;74(11):1712–20.View ArticlePubMedGoogle Scholar
- Hammarström A, Johansson K, Annandale E, Ahlgren C, Aléx L, Christianson M, Elwér S, Eriksson C, Fjellman-Wiklund A, Gilenstam K, Gustafsson PE, Harryson L, Lehti A, Stenberg G, Verdonk P. Central gender theoretical concepts in health research: the state of the art. J Epidemiol Community Health. 2014;68(2):185–90.View ArticlePubMedGoogle Scholar
- Sims ST, Stefanick ML, Kronenberg F, Sachedina NA, Schiebinger L. Gendered innovations: a new approach for nursing science. Biol Res Nurs. 2010;12(2):156–61.View ArticlePubMedGoogle Scholar
- Holdcroft A, Snidvongs S, Berkley KJ. Incorporating gender and sex dimensions in medical research. Interdiscipl Sci Rev. 2011;36(2):180–92.View ArticleGoogle Scholar
- Miller VM, Walter AR, Faubion SS. Sex differences research, precision medicine, and the future of women’s health. J Women's Health. 2015;24(12):969–71.View ArticleGoogle Scholar
- Johnson J, Greaves L, Repta R. Better science with sex and gender: facilitating the use of a sex and gender-based analysis in health research. Int J Equity Health. 2009;8:14.View ArticlePubMedPubMed CentralGoogle Scholar
- Doull M, Welch V, Puil L, Runnels V, Coen SE, Shea B, O’Neill J, Borkoff C, Tudiver S, Boscoe M. Development and evaluation of ‘briefing notes’ as a novel knowledge translation tool to aid the implementation of sex/gender analysis in systematic reviews: a pilot study. PLoS One. 2014;9(11):e110786.View ArticlePubMedPubMed CentralGoogle Scholar
- McGregor AJ, Madsen TE, Clyne B. Foundations for a novel emergency medicine subspecialty: sex, gender, and women's health. Acad Emerg Med. 2014;21(12):1469–77.View ArticlePubMedGoogle Scholar
- Nieuwenhoven L, Bertens M, Klinge I. Gender awakening tool, bibliography: sex and gender in research. Gender Basic. 2007. http://www.genderbasic.nl/downloads/pdf/WISER%20fest%20booklet%20extra%20pages.pdf. Accessed 17 Aug 2016.
- Gendered Innovations. Health and medicine checklist. 2015. https://genderedinnovations.stanford.edu/methods/health_med_checklist.html. Accessed 17 Aug 2016.Google Scholar
- Doull M, Runnels V, Tudiver S, Boscoe M. Sex and gender in systematic reviews: planning tool. The Campbell and Cochrane Equity Methods Group. 2011. http://methods.cochrane.org/equity/sites/methods.cochrane.org.equity/files/uploads/SRTool_PlanningVersionSHORTFINAL.pdf. Accessed 17 Aug 2016.Google Scholar
- European Commission. Toolkit for gender in research: checklist for gender in research. European Commission. 2011. http://bookshop.europa.eu/en/toolkit-gender-in-eufunded-research-pbKINA24840/. Accessed 17 Aug 2016.
- Tomás C, Yago T, Eguiluz M, Samitier ML, Oliveros T, Palacios G. A tool to assess sex-gender when selecting health research projects. Aten Primaria. 2015;47(4):220–7.View ArticlePubMedGoogle Scholar
- Office of the Prime Minister of Canada. Open letter to the Minister of the Status of Women. 2015. http://pm.gc.ca/eng/minister-status-women-mandate-letter. Accessed 28 Jul 2016.Google Scholar
- Blackless M, Charuvastra A, Derryck A, Fausto-Sterling A, Lauzanne K, Lee E. How sexually dimorphic are we? Review and synthesis. Am J Hum Biol. 2000;12(2):151–66.View ArticlePubMedGoogle Scholar
- Fausto-Sterling A. Sexing the Body: Gender Politics and the Construction of Sexuality. New York: Basic Books; 2000.Google Scholar
- Bauer GR, Hammond R, Travers R, Kaay M, Hohenadel KM, Boyce M. “I don’t think this is theoretical; this is our lives”: how erasure impacts health care for transgender people. J Assoc Nurses AIDS Care. 2009;20(5):348–61.View ArticlePubMedGoogle Scholar