Consider a validation study reporting 90% overall accuracy. The subgroup analysis in the supplementary material reports 92% accuracy for White patients and 78% for Black patients. This 14 percentage point difference is the substantive finding, and it raises the question of whether the tool should be deployed at all. Such patterns are not hypothetical; they have been documented repeatedly across health systems, in which an algorithm performs well for the majority population and substantially less well for others. The cause is rarely intentional. It reflects bias encoded in healthcare data, which remains undetected unless subgroup analysis is performed.

Why Bias in Medical AI Differs from Bias in Other Domains

Bias in a recommendation system or a credit model is consequential, but bias in a clinical model differs in kind, because the cost is borne in patient safety. When an algorithm systematically under-serves a group, the resulting harm includes delayed diagnosis, missed referral for definitive treatment, and worse outcomes than would have obtained for a patient with an identical presentation from another group. Moreover, the effect may be self-reinforcing: patients who experience such disparities may disengage from care, which reduces the data available for training subsequent models on those populations and widens the performance gap with each iteration. The difficulty of addressing bias in healthcare derives from the nature of the data and of the decisions those data record.

Healthcare Data Encode Historical Inequity

A medical record is not a neutral representation of biology. It encodes documented disparities arising from structural racism, socioeconomic inequity, access barriers, and discrimination, along several axes. With respect to access, underserved patients have fewer documented encounters, fewer diagnostic tests, and sparser records than patients with greater resources. With respect to diagnosis, an identical complaint may be described and coded differently by demographic group; chest pain in a White man may be investigated as possible cardiac ischemia, whereas the same symptom in a woman or a Black patient is more frequently attributed to anxiety. With respect to treatment, different groups receive different interventions for identical conditions, and these differences are incorporated into the outcome labels on which a model is trained. The model does not represent any of this as bias. It identifies the statistical structure of the data and optimizes to reproduce it, including the components of that structure that reflect bias rather than pathophysiology. An algorithm, whether classical or deep, learns correlations and has no representation of the processes that produced them.10

Mathematical Objectivity Does Not Confer Freedom from Bias

It is frequently assumed that an algorithm, being mathematical, is objective and therefore free of the biases that affect human judgment. This assumption does not hold. Every model trained on healthcare data learns from decisions made by clinicians operating within a system that exhibits documented disparities. Those decisions are distributed throughout the dataset: which patients are referred to a specialist, a decision shaped by insurance status, geographic proximity, implicit bias, and the patient's capacity to advocate; which symptoms are documented rather than dismissed, shaped by available time and by the degree to which the patient's report is credited; which tests are ordered; and how pain is assessed and treated, a domain with well-documented racial and sex-based disparities. Consider a model predicting which patients require cardiology consultation. It learns from historical referral patterns. If those patterns under-referred women and minority patients presenting with chest pain, as the literature indicates, the model reproduces that pattern, because the biased referrals constitute its ground-truth labels. By its own objective function the model performs correctly, in that it predicts the historical pattern accurately. However, the labels encode bias, and optimizing against them perpetuates rather than corrects the inequity.

Points of Entry Across the Development Pipeline

Bias does not enter at a single point. It accumulates across the development lifecycle, from data collection and curation through training and validation to post-deployment monitoring. Mitigation requires localization, and the principal entry points have been characterized across data handling, model development, and evaluation.7-9

1. Dataset Composition

Most medical AI is trained on data from academic medical centers and large, well-resourced urban health systems, populations that differ systematically from the broader patient population. Such datasets tend to underrepresent rural patients distant from tertiary care, patients with limited English proficiency whose records may be incomplete, and minority groups whose representation at academic centers does not correspond to their share of the general population. This composition bias, introduced at data collection, is among the most basic sources of inequity, because models generalize poorly to groups sparsely represented in training.7 Patterns learned from the majority population may not transfer to minority patients with different clinical features or presentations. For example, a diabetic retinopathy screening model trained predominantly on fundus photographs from White patients may underperform on Black and Hispanic patients, plausibly because differences in fundus pigmentation affect image contrast and because acquisition parameters vary across settings.

2. Label Bias

Training labels are frequently proxies for disease rather than measurements of disease. Billing codes capture what was recognized, documented, and coded for reimbursement rather than the full extent of a patient's pathology. Laboratory thresholds for normal values rest on reference ranges that may have been established in non-representative populations. Defining a label by procedure utilization, for example defining a requirement for advanced imaging by which patients underwent MRI, conflates clinical need with access, physician practice pattern, insurance authorization, and the patient's capacity to attend the appointment. The best-documented instance is the algorithm examined by Obermeyer and colleagues, which predicted health needs from healthcare expenditure on the assumption that patients with greater illness burden incur greater cost.1 However, Black patients receive less care and therefore incur lower costs at equivalent severity, and the model consequently assigned them lower predicted need than equally ill White patients. The estimator was not misspecified; the label was. Optimizing accuracy against a biased label reproduces the bias. This indicates that labels should be validated against clinical ground truth rather than against administrative or utilization proxies.7

3. Proxy Variables

Some algorithms use race or ethnicity as an explicit input, justified by claimed biological differences that are frequently unsupported or outdated. However, models that deliberately exclude race may reconstruct racial bias through proxy variables, that is, features sufficiently correlated with race that the model effectively recovers the category. In United States healthcare data the principal proxies are ZIP code, which reflects historical redlining and continuing residential segregation; insurance type, since Medicaid enrollment is strongly patterned by race; primary language, a proxy for immigration status and ethnicity; and admitting hospital, since geographic segregation produces differing racial composition across hospitals within a single metropolitan area. This phenomenon is termed redundant encoding: when race is removed from the feature set, the model reconstructs it from correlated variables and continues to produce predictions that vary systematically by race.

Race Correction in Estimated Glomerular Filtration Rate

For several decades, standard equations for estimating glomerular filtration rate (eGFR) incorporated a race coefficient that adjusted creatinine-based estimates upward for Black patients, on the assumption that Black individuals have greater muscle mass and therefore higher baseline creatinine. The effect was to overestimate kidney function in Black patients, which may have delayed recognition of progressive chronic kidney disease, nephrology referral, transplant waitlisting, and initiation of dialysis. In 2021, following accumulating evidence of harm, the major nephrology organizations recommended removal of the race coefficient, acknowledging that the stated biological rationale conflated race, a social construct, with ancestry and genetics. It is likely that the biased estimates delayed appropriate care for a substantial number of Black patients over the period of use. This case demonstrates that an adjustment presented as physiological may nevertheless cause harm when it rests on a categorical racial assumption standing in for complex genetic, environmental, and social factors.

Clinical Examples

Race-Based Coefficients in Nephrology and Related Fields

Beyond eGFR, race has been incorporated into algorithms for:

  • Kidney stone risk: lower predicted risk assigned to Black patients despite comparable disease prevalence
  • Preeclampsia screening: race-specific thresholds for proteinuria
  • Urinary tract infection diagnosis: race-adjusted clinical prediction rules

The common feature is the treatment of race as a biological variable when it functions as a social variable capturing disparities in care, environment, and exposure to discrimination.

Pulse Oximetry and Compounded Measurement Error

Pulse oximeters are less accurate in patients with darker skin pigmentation, overestimating oxygen saturation by approximately 2–3% on average. Consider a model trained on pulse oximetry data to predict:

  • Requirement for supplemental oxygen
  • Requirement for intensive care admission
  • Eligibility for extracorporeal membrane oxygenation

Such a model learns from systematically biased measurements in Black patients. The expected consequence is underestimation of disease severity and under-triage.

This is not hypothetical. During the COVID-19 pandemic, pulse oximeters were shown to overestimate oxygen saturation in patients with darker skin, and occult hypoxemia was detected less frequently in Black patients. Any triage tool constructed on those measurements would be expected to inherit and propagate this systematic error.

Under-Triage of Minority Patients

Triage algorithms predict which patients require higher-acuity care. Where such models are trained on historical patterns in which:

  • Minority patients were less likely to be admitted to intensive care, whether owing to bias or to structural barriers
  • Pain was under-recognized and under-treated in Black patients
  • Cardiac symptoms in women were more likely to be attributed to non-cardiac causes

the model reproduces those patterns, identifying fewer minority and female patients as high-risk even when they are clinically equivalent to White male patients.

Why Bias Is Difficult to Detect Clinically

Bias Is Not Observable at the Point of Care

At the point of care, the clinician observes:

  • The prediction for the individual patient
  • The recommendation for the individual patient

The clinician does not observe:

  • How predictions vary across demographic groups
  • False negative rates stratified by race
  • Whether the model underperforms in the local population relative to the validation cohort

Bias is a population-level property. It cannot be identified reliably from individual cases, however experienced the observer.

Limited Transparency in Commercial Systems

Most commercially procured AI tools do not disclose:

  • Demographic composition of the training data
  • Performance metrics stratified by subgroup
  • Disparate impact assessments
  • Bias mitigation methods applied

Users are consequently expected to accept a general assertion of validity without the information required to assess fairness independently.

Definitions of Fairness and Their Incompatibility

Fairness admits of multiple formal definitions that are mathematically incompatible, and the choice of performance metric used to evaluate fairness determines what constitutes acceptable performance.9

Group Fairness (Demographic Parity)

Definition: the algorithm predicts positive outcomes at the same rate across groups.

Example: if 10% of White patients are identified as high-risk, then 10% of Black patients should also be identified as high-risk.

Limitation: where disease prevalence differs across groups, whether owing to social determinants or otherwise, enforcing equal rates causes the algorithm to under-predict in the higher-prevalence group.

Individual Fairness

Definition: similar patients receive similar predictions irrespective of group membership.

Example: two patients with identical vital signs, laboratory values, and history receive the same risk score regardless of race or sex.

Limitation: the similarity metric is itself a modeling choice. Where that metric does not account for structural inequities such as access to care, neighborhood conditions, and environmental exposure, patients who are similar on measured features may not be comparable.

Equalized Odds

Definition: the model achieves equal sensitivity and specificity across groups.

Example: among patients with the condition, the model identifies 85% regardless of race; among patients without the condition, it correctly excludes 90% regardless of race.

Limitation: achieving equalized odds frequently requires group-specific decision thresholds, which raises separate ethical and legal questions.

The Impossibility Result

These fairness criteria cannot in general be satisfied simultaneously. Except under restrictive conditions, such as equal base rates across groups or perfect prediction, optimizing for one criterion necessarily degrades another.

It follows that no algorithm is unbiased in a formal sense. There are only different tradeoffs, made explicitly or by default.

The operative question is therefore not whether a model is fair, but which fairness criterion is appropriate to the clinical use case, and which group bears the cost of that choice.

Tradeoffs Between Aggregate Accuracy and Equity

Optimizing for fairness may reduce aggregate accuracy. Consider a sepsis prediction model with 90% accuracy overall and 82% accuracy in Hispanic patients, attributable to sparse representation in training data. Three options are available:

  • Option A: deploy the model as constructed, accepting 90% aggregate accuracy with disparate subgroup performance.
  • Option B: retrain to equalize performance across groups, potentially yielding approximately 87% accuracy in all groups.
  • Option C: defer deployment and collect additional representative data.

No option is correct on technical grounds alone. The choice depends on which of the following objectives is prioritized:

  • Maximizing aggregate benefit, which favors Option A
  • Equalizing performance across groups even at the cost of aggregate accuracy, which favors Option B
  • Deferring deployment until both objectives can be met, which favors Option C, with the acknowledged cost that patients who would have benefited receive no tool in the interim

Why Excluding Race Is Insufficient

The intuitive response to race-based bias is to exclude race from the feature set. However, race is correlated with a large number of variables routinely present in healthcare data, including:

  • ZIP code
  • Admitting hospital
  • Insurance type
  • Comorbidity burden, which reflects access as well as biology
  • Laboratory values, which reflect differences in baseline health status

When race is removed, the model reconstructs it from these correlates. This is termed redundant encoding: group membership remains inferable from other variables. Model selection, architecture, and training procedure all influence the degree to which such patterns are encoded.8

Illustrative Case: Prediction of Appointment Non-Attendance

Consider a health system that develops a model to predict which patients will not attend scheduled appointments, in order to inform overbooking. Race is not included as a feature. The features used include:

  • Prior missed appointments, which correlate with transportation barriers and inflexible work schedules, both of which are patterned by race
  • Insurance type, since Medicaid enrollment is strongly patterned by race as a consequence of historical policy
  • Neighborhood of residence, a proxy for residential segregation

The expected result is that Black and Hispanic patients are identified as high-risk for non-attendance at elevated rates. Where such predictions inform scheduling policy, the affected patients may receive less flexible scheduling and more restrictive policies, with consequent effects on the therapeutic relationship. The model does not use race as an input, yet its outputs are patterned by race.

Practical Guidance

Questions for Vendors

Prior to institutional adoption of an AI tool, the following should be requested:

  1. What was the demographic composition of the training data?
    Inability to provide this information should be regarded as disqualifying.7
  2. What is model performance across racial and ethnic subgroups?
    Aggregate accuracy is uninformative where performance is 95% in White patients and 75% in Black patients. Stratified metrics, comprising sensitivity, specificity, positive predictive value, and negative predictive value by demographic group, should be required.9
  3. Were disparate impact assessments performed?
    Specifically, was the model evaluated for systematic over- or under-prediction in defined groups?
  4. Does the model use race as an input variable?
    If so, what is the stated justification, and does it reflect a biological mechanism or a social determinant?2
  5. How is missing data handled?
    Minority patients frequently have sparser electronic health record data, and the imputation strategy may itself introduce differential error.7
  6. Was external validation performed on diverse populations?
    Internal validation on data from a single institution is insufficient. Models should be evaluated on external datasets representing different demographic composition and practice settings.9
  7. What is the plan for ongoing bias monitoring?
    Performance should be tracked by subgroup following deployment, with defined thresholds for intervention when disparities are detected.

Indicators of Elevated Risk

The following features are associated with an elevated probability that an algorithm exhibits disparate performance:

  • The validation study does not report subgroup performance
  • Training data derive from a single health system or geographic region
  • The model uses cost or utilization as a proxy for illness severity
  • The vendor cannot describe which features drive predictions
  • No plan exists for monitoring fairness following deployment

Where a vendor characterizes fairness concerns as not applicable to their model, that response should itself be treated as informative.

The Role of the Clinician

The clinician constitutes the final point at which an algorithmic recommendation is translated into a clinical action. Even where a model exhibits disparate performance, harm may be mitigated by:

  • Questioning predictions that conflict with clinical judgment, particularly for patients from underrepresented groups
  • Documenting overrides, which generates the data required for subsequent bias detection
  • Requiring fairness assessment before procurement through institutional governance
  • Reporting observed disparities where outcomes in AI-assisted care appear worse in specific groups

An algorithm has no intent. The system in which it operates nevertheless produces patterned outcomes, and clinicians are a component of that system.

Conclusion

Bias in medical AI is not primarily a technical problem admitting a technical solution. It is a question of healthcare equity, and it requires transparency, measurement, and accountability rather than a modified objective function. The problem is distributed across the pipeline, including data provenance, label quality, deployment context, and ongoing monitoring, and addressing it therefore requires clinicians, data scientists, ethicists, patients, and administrators rather than engineers alone.

Complete elimination of bias is not achievable, since it is encoded in the historical data from which models learn. However, the impossibility of perfect fairness does not justify inaction. Responsible practice requires that bias be named, that disparities be measured across subgroups, and that evidence-based mitigation be applied to reduce them as far as is feasible. In operational terms, this means requiring vendors to disclose training data demographics; requiring pre-deployment evaluation on populations corresponding to the intended deployment setting rather than an aggregate accuracy figure alone; monitoring stratified performance after implementation; being prepared to withdraw a tool whose disparate impact persists despite mitigation; and treating equity as a design criterion rather than as a terminal review step.

Returning to the model with 92% accuracy in White patients and 78% in Black patients: the appropriate institutional response is to defer adoption until the disparity is reduced, irrespective of the aggregate figure. Deployment under those conditions purchases accuracy for the majority group at the direct expense of the minority group. The 14 percentage point difference is not an abstraction; it corresponds to missed diagnoses, delayed treatment, and worse care in identifiable patients whom the health system is obligated to serve equitably.

Key Takeaways

  • Bias reflects training data not developer intent: Machine learning algorithms learn statistical patterns from historical healthcare data that encodes centuries of systemic inequities including access barriers, diagnostic bias in symptom interpretation and documentation, and treatment disparities across demographic groups. These learned patterns perpetuate rather than correct existing inequities.
  • Outcome labels frequently represent biased proxies: Ground truth labels used to train supervised learning models often reflect who received care rather than who needed care, with healthcare utilization and costs serving as particularly problematic proxies that systematically underestimate disease severity in underserved populations experiencing care disparities.
  • Excluding race as an explicit variable is insufficient: Algorithms can reconstruct racial categories through redundant encoding using proxy variables highly correlated with race including residential ZIP code, insurance type, primary language, and hospital of admission. Simply removing race from the feature set does not eliminate racial bias in predictions.
  • Multiple fairness definitions are mathematically incompatible: Group fairness (demographic parity), individual fairness (equal treatment of similar cases), and equalized odds (equal accuracy across groups) represent distinct fairness frameworks that cannot be simultaneously optimized. Algorithm designers must make explicit choices about which fairness criterion to prioritize, and these choices have distributional consequences.
  • Critical vendor accountability questions: Healthcare institutions evaluating AI systems should demand detailed information regarding training data demographic composition, algorithm performance metrics stratified by demographic subgroups, disparate impact assessments, biological versus social justifications for including race or proxies, approaches to handling missing data that may be differential across groups, external validation on diverse populations, and plans for ongoing fairness monitoring post-deployment.
  • Clinicians serve as critical gatekeepers: Individual physicians can mitigate algorithmic bias through critical evaluation of predictions that conflict with clinical judgment particularly for patients from underrepresented groups, systematic documentation of algorithm overrides to create data enabling bias detection, institutional advocacy for fairness assessments before procurement decisions, and reporting of observed disparate outcomes in AI-assisted care.

References & Further Reading

  1. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations. Science. 2019;366(6464):447-453. doi:10.1126/science.aax2342
  2. Vyas DA, Eisenstein LG, Jones DS. Hidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms. N Engl J Med. 2020;383(9):874-882. doi:10.1056/NEJMms2004740
  3. Sjoding MW, Dickson RP, Iwashyna TJ, Gay SE, Valley TS. Racial Bias in Pulse Oximetry Measurement. N Engl J Med. 2020;383(25):2477-2478. doi:10.1056/NEJMc2029240
  4. FDA. Artificial Intelligence and Machine Learning in Software as a Medical Device. Updated 2023. FDA.gov
  5. World Health Organization. Ethics and Governance of Artificial Intelligence for Health. WHO Guidance. 2021. WHO.int
  6. Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data. JAMA Intern Med. 2018;178(11):1544-1547.
  7. Rouzrokh P, Wyles CC, Philbrick KA, Ramazanian T, Weston AD, Cai JC, Taunton MJ, Kremers WK, Lewallen DG, Erickson BJ. A Deep Learning Tool for Automated Radiographic Measurement of Acetabular Component Inclination and Version After Total Hip Arthroplasty. Part 1: Mitigating Bias in Machine Learning—Data Handling. J Arthroplasty. 2022;37(6S):S406-S413. doi:10.1016/j.arth.2022.02.092
  8. Zhang Y, Wyles CC, Makhni MC, Maradit Kremers H, Sellon JL, Erickson BJ. Part 2: Mitigating Bias in Machine Learning—Model Development. J Arthroplasty. 2022;37(6S):S414-S420. doi:10.1016/j.arth.2022.02.085
  9. Faghani S, Khosravi B, Moassefi M, Rouzrokh P, Erickson BJ. Part 3: Mitigating Bias in Machine Learning—Performance Metrics, Healthcare Applications, and Fairness in Machine Learning. J Arthroplasty. 2022;37(6S):S421-S428. doi:10.1016/j.arth.2022.02.087
  10. Erickson BJ, Kitamura F. Artificial Intelligence in Radiology: a Primer for Radiologists. Radiol Clin North Am. 2021;59(6):991-1003. doi:10.1016/j.rcl.2021.07.004