Radiology AI products are increasingly described as "foundation models," a term that implies a broad, general-purpose base upon which many applications may be constructed. Examination of the technical documentation generally does not support that implication. A representative medical "foundation model" is trained on chest CT alone, optimized for lung nodule detection, using 50,000 examinations from three academic medical centers. Such a model may be clinically useful, but it is a narrow, specialized system, and the foundation designation overstates its scope.

In the general machine learning literature, the term foundation model denotes a large model, such as GPT-4 or BERT, trained on extensive heterogeneous data comprising billions of text tokens or millions of varied images, which generalizes to tasks for which it received no explicit training. Medical imaging has adopted the term for a substantially narrower construct: a model pre-trained on a curated set of medical images by self-supervised learning and subsequently fine-tuned for a specific task within a closely related domain. These models have genuine value, principally in reducing the quantity of labeled data required to develop a clinical tool. However, nearly all are constrained to a single modality, a single body region, and a single class of patient population, and their generalization depends substantially on the composition of the pre-training data.

Definition and Scope in the Medical Context

In computer vision and natural language processing, the foundational property derives from the breadth of the training data: millions of images spanning most visual categories, or billions of text tokens from books, web content, and scientific literature across many languages. That breadth is what supports generalization. A vision model trained on ImageNet recognizes thousands of object categories, and a language model trained on web-scale text performs question answering, code generation, translation, and summarization without task-specific training. The designation is earned by the transfer of learned representations across genuinely distinct domains.

Medical foundation models are characterized by specificity rather than breadth, and three constraints are typical. First, they are restricted to a single modality, whether CT, MRI, radiography, or ultrasound, and rarely to combinations. Second, they are restricted to a single anatomical region, such as brain, chest, or abdomen, rather than to whole-body imaging. Third, they are pre-trained without diagnostic labels, using self-supervised objectives such as prediction of masked image regions, reconstruction of corrupted images, or contrastive discrimination between similar and dissimilar examinations. A brain MRI foundation model, for example, may be pre-trained by predicting masked patches or by clustering anatomically similar images, without exposure to any diagnosis. It may subsequently be fine-tuned on a small labeled set to detect multiple sclerosis lesions, classify brain tumors, or identify acute infarction, in each case within brain MRI. It cannot interpret a chest radiograph, an abdominal CT, or a cranial ultrasound, and still less an electrocardiogram or a pathology slide. In this context, foundation denotes a specialized starting point whose learned features can be adapted to related tasks within one narrow domain using substantially less labeled data than would be required to train from initialization.1,2

Why Medical Foundation Models Are Narrow

This narrowness is not arbitrary. It follows from technical properties of medical imaging data and from the cost of training across all of it jointly.

Modality Fragmentation

Natural photographs share a common structure comprising three color channels, similar edge and texture statistics, and pixel values with consistent meaning across images. Medical modalities do not share such a structure. Each rests on different physics and encodes a different quantity, which limits transfer between them. CT produces grayscale images in Hounsfield units, in which the value is a quantitative measure of tissue attenuation. MRI intensities, by contrast, are not absolute, and the same structure has a substantially different appearance across T1-weighted, T2-weighted, FLAIR, and diffusion-weighted sequences. Ultrasound is acquired as video, is frame-selected, and is operator-dependent. PET captures low-resolution metabolic activity and requires co-registration with anatomic imaging for interpretation. Plain radiography is a projection through three-dimensional anatomy. A model trained on CT constructs internal representations organized around attenuation patterns, and those representations do not transfer to MRI or ultrasound. The relationship is therefore not analogous to transfer between visually similar object categories; it more closely resembles transfer between photographs and audio spectrograms, that is, between data types requiring different feature extractors.

Anatomical Specialization

Even within a single modality, anatomical regions differ in normal appearance, in the pathology encountered, and in clinical context, and these differences also limit generalization. A chest CT model learns representations of ribs, lung parenchyma, mediastinum, and vasculature, none of which appear in brain imaging, where the relevant structures are gray and white matter, ventricles, and specific anatomic regions. An abdominal MRI model learns liver, kidney, and bowel, which are not applicable to spine imaging, in which the relevant structures are vertebral bodies, intervertebral discs, neural foramina, and spinal cord. Because MRI appearance depends on the pulse sequence, the same organ has a different appearance across protocols, and the model must therefore learn sequence-specific representations in addition to anatomic ones. Training a single model across all regions and modalities would require assembling a dataset spanning all such combinations and expending compute several orders of magnitude greater than that required for the modality- and anatomy-specific models currently in use.3

Self-Supervised Pre-Training and Task-Specific Fine-Tuning

Unlike earlier approaches to medical AI, which required labeled data from the outset, foundation models are pre-trained by self-supervised learning, in which the training signal is derived from unlabeled images through prediction of masked regions, reconstruction of corrupted images, or discrimination between similar and dissimilar examinations. The motivation is practical: annotation of medical images is time-consuming, requires domain expertise, and must be performed consistently for the resulting labels to be informative.

However, construction of a clinically useful tool still requires labeled data for fine-tuning, comprising diagnostic annotations, pathology outcomes, or treatment responses. The foundation model supplies learned image representations, and fine-tuning establishes the clinical meaning of those representations. Although self-supervised pre-training is task-agnostic, the modality and anatomical constraints described above remain in effect.4

The Principal Benefit: Data Efficiency

Despite their narrow scope, medical foundation models confer a substantive advantage. That advantage is not breadth but data efficiency.

Training from Random Initialization

Before pre-trained medical models were available, the standard procedure was:

  1. Initialize a neural network randomly
  2. Collect and annotate thousands of MRI examinations with lesion annotations
  3. Train the model from initialization, requiring substantial computational resources
  4. Assume the dataset is sufficiently large for informative features to be learned

Limitation: this procedure requires a large labeled dataset, frequently on the order of tens of thousands of examples, to achieve acceptable performance. It is therefore not feasible for rare diseases or for single-institution datasets.

Transfer Learning

The subsequent approach was to initialize from a model pre-trained on ImageNet, a large collection of natural photographs. Although these are not medical images, they share low-level properties with medical images, including edges and elementary textures. The procedure became:

  1. Initialize from a model that has learned general representations of photographic structure
  2. Apply transfer learning by adapting those weights to medical images, in a manner analogous to current use of foundation models

Foundation models extend this approach. They are constructed deliberately to perform no task in isolation but to be adapted to one. The corresponding procedure is:

  1. Initialize from a model that has learned general representations of brain MRI structure
  2. Fine-tune on a substantially smaller dataset of the target condition, on the order of hundreds rather than thousands of cases
  3. Obtain comparable or superior performance with less data and less compute

Mechanism: the foundation model has already learned low-level features, including edges, textures, and intensity patterns, and mid-level features, including anatomical structures and tissue boundaries, through self-supervised pre-training. Fine-tuning establishes the mapping from those representations to a specific clinical task.

This is transfer learning, and it is the principal reason medical foundation models are constructed. By reducing the labeled data requirement for clinical AI development, they make model development feasible for rare diseases, underserved populations, and institution-specific applications.2,5

Reported Data Efficiency

Published comparisons indicate that use of pre-trained medical foundation models can reduce the labeled data requirement by a factor of approximately 5 to 10 relative to training from random initialization. For rare conditions with limited available cases, this difference may determine whether model development is feasible at all.

For example, a foundation model pre-trained on 100,000 chest radiographs has been reported to support fine-tuning for detection of pediatric pneumonia using as few as 500 labeled pediatric cases, whereas training from initialization may require 5,000 to 10,000 cases to reach comparable accuracy.5

Capabilities Not Conferred by Pre-Training

Although foundation models improve data efficiency, they do not address several other requirements of clinical AI:

  • Explainability: self-supervised pre-training learns abstract representations rather than clinical concepts. A model may detect disease accurately without producing an explanation in clinically interpretable terms. Separate methods, such as attention maps or saliency techniques, are required to generate explanations, and these may not correspond to radiologist reasoning.
  • Localization: foundation models learn image-level features and do not necessarily encode spatial localization. Where a task requires precise lesion boundaries or anatomic localization, additional fine-tuning strategies or task-specific architectures are typically required, as transfer learning does not confer pixel-level precision.
  • Clinical context: models trained on images alone have no representation of patient history, laboratory values, or clinical presentation. Multi-modal foundation models combining imaging with electronic health record data are under development but remain uncommon.

Foundation models are therefore useful initializations rather than complete solutions, and their limitations warrant the same attention as their advantages.

Demographic Composition of Pre-Training Data

A question frequently addressed only in supplementary materials, or omitted, concerns the demographic composition of the pre-training data.

Fine-tuning a foundation model on a local dataset does not begin from an uninformed state. It begins from a model that has already learned patterns from its pre-training data. Where those data are demographically skewed, the fine-tuned model may inherit the resulting bias even when the fine-tuning data are balanced.

Propagation of Bias Through Transfer Learning

Foundation models encode the demographic distribution on which they were trained. Where the pre-training dataset is:

  • 90% White patients, the learned representations are optimized for that population
  • Drawn predominantly from academic medical centers, the learned patterns reflect tertiary care populations
  • Imbalanced with respect to sex, age, or socioeconomic status, minority groups are correspondingly underrepresented in the learned representation

When such a model is fine-tuned on a small dataset, which is the principal use case for transfer learning, the pre-training priors are likely to dominate. A fine-tuning set of 200 examples cannot be expected to override the representations learned from millions of pre-training examples.6,7

Expected consequence: performance is lower in populations underrepresented in the pre-training set, even where those populations are represented in the fine-tuning data.

Example: Skin Lesion Classification

Dermatology models, including foundation models for skin lesion classification, have been shown to perform substantially worse on darker skin tones. This is attributable to the composition of the pre-training datasets, which are frequently derived from public dermatology atlases in which lighter skin tones predominate.

Even where such a model is fine-tuned on a dataset balanced across skin tones, the pre-trained representation has already been optimized for lighter skin, including the edge detection, texture analysis, and color distribution characteristics that are most informative for that population.8

Fine-tuning may improve performance in underrepresented groups but is unlikely to eliminate the effect of the pre-training prior.

Analogy with Pulse Oximetry

This mechanism is conceptually comparable to the pulse oximetry problem discussed in our article on algorithmic bias. Pulse oximeters were calibrated predominantly on individuals with lighter skin, producing systematic error in patients with darker skin. Recalibration is possible, but the original calibration establishes a baseline bias that is difficult to remove entirely.

Foundation models behave analogously, with the pre-training dataset functioning as the calibration. Where that dataset is demographically skewed, downstream applications inherit the skew unless substantial corrective effort is applied, and such correction requires labeled data from underrepresented groups, which reduces the data efficiency that motivated the approach.

Questions to Ask Before Adoption

The following questions are recommended for institutions considering a foundation model, whether for research or for development of clinical tools.

1. What is the scope of the model?

The foundation designation should not be taken to imply breadth. The following should be specified:

  • Imaging modality: CT, MRI, radiography, ultrasound, or PET
  • Anatomical region: brain, chest, abdomen, or extremities
  • Pre-training objectives: segmentation, classification, or detection

Inability to define the model's scope should be regarded as a limitation of the product documentation. A well-characterized model states its domain of applicability and its limitations explicitly.

2. What are the demographics of the training data?

The following should be requested:

  • Racial and ethnic composition. A model trained on a population that is 90% White may underperform in minority populations.
  • Age distribution. Models trained predominantly on adults may not generalize to pediatric or geriatric patients.
  • Sex distribution. Imbalance may produce disparate performance by sex.
  • Contributing institutions. Academic medical centers serve different populations from community hospitals and international sites.

A response asserting that the model is unbiased, or that demographics were not recorded, is not adequate. Every dataset has a demographic distribution, and failure to disclose it precludes assessment of generalization.7

3. Has performance been validated on diverse populations?

Diversity in pre-training data is necessary but not sufficient; validation across populations is also required. The following should be requested:

  • Whether the model has been evaluated on external datasets with different demographic composition
  • Whether performance metrics, including sensitivity, specificity, and AUC, are reported stratified by demographic subgroup
  • The magnitude of the performance difference between the best- and worst-performing subgroups

An aggregate AUC of 0.95 is uninformative where the value is 0.97 in White patients and 0.88 in Black patients. Stratified metrics should be required.9

4. What quantity of fine-tuning data is required?

Data efficiency is the principal claim made for foundation models, and it should be quantified. The following should be requested:

  • The minimum recommended fine-tuning dataset size for the intended task
  • A learning curve describing how performance scales with fine-tuning dataset size
  • The risk of overfitting or bias amplification when fine-tuning on a small dataset

Guidance on data requirements should be specific rather than qualitative, and the associated tradeoffs should be stated.

Directions for Development

Notwithstanding current limitations, work is in progress toward foundation models that are genuinely broad, that is, multi-modal, multi-organ, and multi-task.

Multi-Modal Foundation Models

Subsequent generations of medical AI are expected to integrate across modalities:

  • Imaging with electronic health record data: combining radiology, pathology, and clinical variables
  • Multi-scale imaging: linking organ-level imaging by CT or MRI with cellular-level histopathology
  • Temporal integration: learning from longitudinal data describing change over time

Such models will require large curated datasets, but they may constitute foundations in the sense in which the term is used outside medicine.10

Self-Supervised Learning at Scale

Rather than depending on expert annotation, subsequent foundation models are expected to learn from unlabeled data using self-supervised objectives:

  • Contrastive learning: discriminating similar from dissimilar images without labels
  • Masked prediction: predicting withheld portions of an image or record
  • Temporal prediction: predicting subsequent examinations from prior ones

This approach permits training on large volumes of unlabeled examinations, which may increase dataset diversity and reduce demographic skew, conditional on the unlabeled data themselves being diverse.11

Federated Learning

One reason medical foundation models are demographically skewed is that data are concentrated in a small number of large institutions, which frequently serve relatively homogeneous populations.

Federated learning permits model training across multiple institutions without transfer of patient data. Participating institutions train local models on their own data and share only model updates. This permits pre-training on more diverse populations while preserving data locality.12

If adopted broadly, federated approaches could yield foundation models pre-trained on more representative datasets, incorporating community hospitals, international sites, and populations not represented in academic-only datasets.

Conclusion

Medical foundation models are not foundational in the sense in which the term is used elsewhere in machine learning. They are specialized, domain-specific systems pre-trained on a single modality, a single anatomical region, and a restricted class of tasks.

Within that domain, however, they are effective. They permit clinical AI development with substantially less labeled data than earlier approaches required, which makes development feasible for rare diseases, small institutions, and applications with limited case volume.

Their value is nevertheless conditional on the composition of the pre-training data. Where that dataset is demographically skewed, downstream applications may inherit the skew even when fine-tuning data are balanced, because the fine-tuning set is small relative to the pre-training set by design.

Prior to adoption, the scope of the model, the demographic composition of its pre-training data, its stratified performance across subgroups, and the quantity of fine-tuning data required should each be established explicitly.

A model whose pre-training distribution is unknown cannot be evaluated for generalization to a specific deployment population. Transparency regarding that distribution, together with stratified validation, is therefore a precondition for responsible use rather than an optional disclosure.

Key Takeaways

  • The term foundation is not used as it is elsewhere in machine learning: most medical foundation models are narrow, comprising one modality and one anatomical region. They use self-supervised pre-training rather than task-specific labels, but remain specialized to their domain.
  • Data efficiency is the principal benefit: transfer learning from a pre-trained model has been reported to reduce labeled data requirements by a factor of approximately 5 to 10, which makes clinical AI development feasible for rare diseases and small datasets.
  • Explainability is not conferred by pre-training: self-supervised learning produces abstract representations rather than clinically interpretable features. Localization and explanation require additional methods beyond transfer learning.
  • Pre-training demographics determine downstream performance: models pre-trained on skewed populations propagate that skew to downstream tasks even where fine-tuning data are balanced, because the fine-tuning set is small relative to the pre-training set by design.
  • Validation should be stratified: performance metrics should be reported by demographic subgroup, since an aggregate AUC conceals disparities between populations.
  • Questions to resolve before adoption: the scope of the model, the demographic composition of its pre-training data, its performance in the intended deployment population, and whether it provides the localization and explanation the application requires.

References & Further Reading

  1. Moor M, Banerjee O, Abad ZSH, et al. Foundation Models for Generalist Medical Artificial Intelligence. Nature. 2023;616(7956):259-265. doi:10.1038/s41586-023-05881-4
  2. Krishnan R, Rajpurkar P, Topol EJ. Self-supervised Learning in Medicine and Healthcare. Nat Biomed Eng. 2022;6(12):1346-1352. doi:10.1038/s41551-022-00914-1
  3. Willemink MJ, Koszek WA, Hardell C, et al. Preparing Medical Imaging Data for Machine Learning. Radiology. 2020;295(1):4-15. doi:10.1148/radiol.2020192224
  4. Tajbakhsh N, Shin JY, Gurudu SR, et al. Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Trans Med Imaging. 2016;35(5):1299-1312. doi:10.1109/TMI.2016.2535302
  5. Raghu M, Zhang C, Kleinberg J, Bengio S. Transfusion: Understanding Transfer Learning for Medical Imaging. Advances in Neural Information Processing Systems. 2019;32.
  6. Rouzrokh P, Wyles CC, Philbrick KA, Ramazanian T, Weston AD, Cai JC, Taunton MJ, Kremers WK, Lewallen DG, Erickson BJ. A Deep Learning Tool for Automated Radiographic Measurement of Acetabular Component Inclination and Version After Total Hip Arthroplasty. Part 1: Mitigating Bias in Machine Learning—Data Handling. J Arthroplasty. 2022;37(6S):S406-S413. doi:10.1016/j.arth.2022.02.092
  7. Zhang Y, Wyles CC, Makhni MC, Maradit Kremers H, Sellon JL, Erickson BJ. Part 2: Mitigating Bias in Machine Learning—Model Development. J Arthroplasty. 2022;37(6S):S414-S420. doi:10.1016/j.arth.2022.02.085
  8. Daneshjou R, Vodrahalli K, Novoa RA, et al. Disparities in Dermatology AI Performance on a Diverse, Curated Clinical Image Set. Sci Adv. 2022;8(32):eabq6147. doi:10.1126/sciadv.abq6147
  9. Faghani S, Khosravi B, Moassefi M, Rouzrokh P, Erickson BJ. Part 3: Mitigating Bias in Machine Learning—Performance Metrics, Healthcare Applications, and Fairness in Machine Learning. J Arthroplasty. 2022;37(6S):S421-S428. doi:10.1016/j.arth.2022.02.087
  10. Acosta JN, Falcone GJ, Rajpurkar P, Topol EJ. Multimodal Biomedical AI. Nat Med. 2022;28(9):1773-1784. doi:10.1038/s41591-022-01981-2
  11. Azizi S, Mustafa B, Ryan F, et al. Big Self-Supervised Models Advance Medical Image Classification. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021:3478-3488.
  12. Rieke N, Hancox J, Li W, et al. The Future of Digital Health with Federated Learning. NPJ Digit Med. 2020;3:119. doi:10.1038/s41746-020-00323-1