Multimodal Applications › Medical Imaging

Medical imaging is both the highest-stakes and the most technically demanding application domain for multimodal AI. The stakes are high because errors can directly cause patient harm, whether through missed diagnoses that delay treatment or false positive findings that lead to unnecessary procedures. The technical demands are high because medical images have unusual properties: they are often grayscale or pseudocolored rather than RGB, they encode three-dimensional anatomy in two-dimensional projections, and the diagnostically relevant features may be subtle variations in pixel intensity that a standard ImageNet-pretrained model would never have learned to detect.

The mismatch between natural image statistics and medical image statistics is not a minor technical detail. It is a basic challenge that shapes how medical AI systems must be built and validated. A vision encoder pre-trained on ImageNet has learned to detect edges and textures alongside object boundaries at the scale of everyday objects: a dog, a chair, a car. A chest X-ray contains none of these. What it contains are subtle variations in radiodensity that correspond to lung tissue, vascular structures, and possible pathologies, all projected onto a flat 2D plane from a three-dimensional anatomy. Learning to read these signals requires either extensive domain-specific pre-training or careful fine-tuning with curated medical data.

Radiology report generation is the task of automatically creating a structured medical report from a chest X-ray or CT scan. A radiologist writes such reports by systematically examining the image for findings (observable abnormalities), then formulating an impression (clinical interpretation) that synthesizes the findings into a diagnosis and recommendation.

The model must solve several sub-problems simultaneously:

• Finding detection: Which regions of the image contain abnormalities?
• Finding description: How should each abnormality be characterized in clinical language?
• Severity assessment: Is this finding incidental or clinically significant?
• Report structure: How should findings be organized into sections (Findings, Impression, Recommendation)?

Early approaches treated this as image captioning and applied standard CNN-LSTM architectures. These models generated fluent text but often hallucinated findings not present in the image or missed real findings that were subtle. Hallucination is especially dangerous in medicine: a model that generates "No evidence of pneumothorax" when one is visible creates real patient risk. This is qualitatively different from hallucination in a general-purpose chatbot, where the consequences of a confident but wrong statement are usually minor. In radiology, the consequences can be life-threatening.

Modern approaches address hallucination through several strategies. Grounded report generation requires the model to attend to specific image regions for each finding it reports, which makes it harder to confabulate content that is not visually supported. The model must, in effect, show its work: for each clinical term it generates, there must be an image region it can point to. Factual consistency verification adds a secondary model that reads the generated report and the image and flags claims that cannot be grounded in the visual evidence. Retrieval-augmented approaches retrieve similar cases from a training database and use their associated reports as conditioning context, encouraging the model to stay close to verified clinical language rather than generating novel combinations that may be clinically incoherent.

The mathematical formulation of grounded generation introduces an attention mask conditioned on image regions. For a token $y_t$ in the generated report, the attention weight over image patch $i$ is:

$$\alpha_{t,i} = \frac{\exp(e_{t,i})}{\sum_{j} \exp(e_{t,j})}$$

where $e_{t,i} = \mathbf{q}_t^\top \mathbf{k}_i$ is the dot product between the query vector derived from the current decoding state and the key vector for patch $i$.

where:

• $\mathbf{q}_t \in \mathbb{R}^d$: query vector computed from the decoder's hidden state at step $t$
• $\mathbf{k}_i \in \mathbb{R}^d$: key vector for image patch $i$, encoded by the vision encoder
• $\alpha_{t,i}$: the attention weight, representing how much the model "looks at" patch $i$ when generating token $t$

This mechanism produces an implicit saliency map for each generated token: we can visualize which image regions the model attended to when writing "pneumothorax" or "consolidation," and verify that these align with radiologically plausible locations. When they do not, that misalignment is evidence of hallucination, and automated grounding verification can use this signal to flag suspect claims before they reach a clinician.

A key advance in medical vision-language models has been the development of radiology-specific pre-training strategies that use the enormous amount of paired image-report data available in hospital systems. CheXpert, MIMIC-CXR, and similar datasets contain hundreds of thousands of chest X-rays with associated radiology reports, covering a wide range of pathologies and patient populations. These datasets, when combined with contrastive pre-training objectives similar to CLIP but adapted for the medical domain, produce vision encoders that are far more sensitive to clinically relevant image features than general-purpose encoders.

The BioViL-T model, for example, applies a temporal contrastive objective that trains the model to align changes in a patient's X-rays over time with changes in the associated radiology reports. This temporal grounding teaches the model concepts like "worsening infiltrate" or "resolving effusion" that require comparing two images, not just reading one in isolation. Such temporal reasoning is needed for real clinical workflows, where radiologists routinely compare the current study to prior studies to detect interval changes.

Pathology slides present a fundamentally different challenge from radiology. A whole-slide image (WSI) of a tissue biopsy may be 100,000 by 100,000 pixels at full resolution, far too large to process as a single input to a transformer. Pathologists examine slides at multiple magnifications: low magnification to identify tissue architecture and high magnification to examine individual cells.

Multiple-instance learning (MIL) is the dominant paradigm for WSI analysis. The slide is divided into thousands of small patches (typically 256x256 pixels at a fixed magnification). Each patch is independently encoded by a vision encoder pre-trained on pathology data, such as the CONCH or UNI models trained on millions of pathology images from diverse tissue types and staining protocols. The patch embeddings are then aggregated into a slide-level representation using attention pooling:

$$\mathbf{z}_{\text{slide}} = \sum_{i=1}^{N} a_i \mathbf{h}_i$$

where:

• $N$: total number of patches from the slide
• $\mathbf{h}_i \in \mathbb{R}^d$: the encoded representation of patch $i$
• $a_i$: the attention weight for patch $i$, computed by a small neural network that assigns higher weight to diagnostically relevant patches
• $\mathbf{z}_{\text{slide}} \in \mathbb{R}^d$: the aggregated slide-level representation used for downstream classification or generation

The attention weights $a_i$ give interpretability, which is especially important in pathology where clinicians need to understand which tissue regions informed a model's conclusion. The patches with the highest weights are the ones the model found most informative, and visualizing these regions allows pathologists to verify whether the model's reasoning aligns with known pathological features such as tumor cells, mitotic figures, stromal invasion, or necrosis.

A necessary challenge in pathology is the variability introduced by staining. Hematoxylin and eosin (H&E) staining, which is the standard in most pathology labs, produces images where cell nuclei appear purple and cytoplasm appears pink. But the exact shade varies with staining protocol, tissue thickness, and scanner settings, and these variations can shift feature distributions in ways that cause models trained at one institution to fail at another. Stain normalization preprocessing and stain-augmented training are common mitigations, but achieving reliable cross-institution generalization remains an open problem.

A key advantage of multimodal AI in medicine is the ability to integrate imaging with structured clinical data. A patient's chest X-ray alone is not the whole picture: their age, smoking history, prior diagnoses, and current lab values all modulate how imaging findings should be interpreted. A new opacity in a patient with a history of lung cancer has a very different clinical significance than the same opacity in a healthy 25-year-old. A consolidation in a patient with fever and a high white blood cell count points toward pneumonia; the same consolidation in an afebrile patient with weight loss points toward malignancy.

Models that fuse imaging with electronic health record (EHR) data encode the clinical notes using a clinical language model such as BioBERT or ClinicalBERT, which have been pre-trained on large corpora of medical text to understand clinical abbreviations, drug names, and diagnostic terminology. The imaging modality is encoded using a radiology-specific vision encoder. The two representations are then combined using cross-attention before the final classification or generation head.

The challenge is handling the heterogeneity of clinical data. Notes may be long, reaching thousands of tokens for complex patients with multi-year histories. They contain abbreviations that vary by institution and clinician (SOB for shortness of breath, CP for chest pain, RLL for right lower lobe). They reference prior visits not in the current context window, requiring either longer context handling or efficient retrieval of relevant historical content. They mix structured and unstructured content, alternating between narrative prose and structured lists of medications and allergies alongside problem lists.

Despite these challenges, fusion models consistently outperform image-only models on downstream clinical tasks. This shows that clinical context gives signal that cannot be inferred from images alone. The performance gain is especially pronounced for tasks where clinical context is directly diagnostic, such as distinguishing malignant from benign pulmonary nodules in patients with known cancer history.