Disparate privacy risks from medical AI

Main

Medical artificial intelligence (AI) has immense potential to improve health outcomes, particularly in regions in which specialized medical expertise is scarce1. At the same time, AI also poses new challenges and risks, including security vulnerabilities that arise when models are deployed. Untrusted users with access to an AI model may, by merely observing its predictions, steal its parameters8,9 or perform privacy attacks2,3,4,5,6,7, which can extract sensitive details about the data used for model training.

Privacy attacks against an AI model can enable detailed inferences about the individuals who contributed to its training data. For example, a membership inference attack (MIA)2 attempts to determine whether the data of a specific patient were included in the training dataset of a model. The extent to which this constitutes a privacy violation is nuanced and depends on factors such as the underlying training population and the deployment context of the model. Although inferring membership for a model trained on a general population may be benign, doing so for a model trained on a narrow, disease- or centre-specific cohort acts as a direct proxy for sensitive medical information. For example, a successful MIA against the model in ref. 10, which predicts anti-cancer immunotherapy efficacy from routine blood test data, reveals that an individual has cancer.

The accelerating deployment of medical AI models trained on sensitive patient data11 calls for rigorous privacy risk assessments. However, previous studies primarily quantified the success rate of MIAs, in aggregate, across all records in a training dataset. This implicitly averages risk across records, thereby obscuring important information on record- and patient-level attack success. Consequently, the risk that an individual faces by contributing their personal data (often multiple records) to an AI training dataset is poorly understood. Given that medical data are a key target for cybercriminals12,13, and pseudonymization alone is increasingly recognized as insufficient to prevent the re-identification of individuals in large, high-dimensional datasets14,15,16, there is a need to improve our understanding of the threat that AI privacy attacks pose to individual patients.

Here we show that deploying medical AI models without protective measures can pose substantial privacy risks to individual data-contributing patients. These risks are particularly acute when membership in a training population itself reveals sensitive medical information. Our privacy audit of AI models trained to perform standard diagnostic (supervised classification) tasks quantifies state-of-the-art MIA success3,4 at the resolution of individual data contributors. Using seven large datasets comprising real-world clinical data, including various types of medical images, electrocardiograms and electronic health records, we demonstrate that the success of a MIA is unequally distributed among data-contributing patients. We show that this disparity exists at two levels: (1) the individual patient level, at which some patients experience near-perfect attack success, whereas others remain essentially unaffected; and (2) the group level, at which patient groups underrepresented in a training dataset are often overrepresented among records most vulnerable to MIAs.

Together, our results indicate that privacy attacks against AI models may be much more effective at compromising the privacy of individual data contributors than previously thought. This suggests that current AI privacy risk reporting practices may underestimate individual-level risk and thus motivates the integration of mathematically verifiable risk mitigation strategies such as differential privacy (DP) into medical AI model development workflows.

Attacking AI by simple hypothesis tests

A popular deployment strategy for AI models gives users access to a model through a prediction interface, which, for a given input (for example, the chest radiograph of a patient), returns a corresponding prediction (for example, a 78% chance of pneumonia). This black-box access to a model can be exploited by an untrusted user to conduct a MIA that shows the membership status of a target record, that is, whether the target record was a member of the training dataset of a model or not (Fig. 1a). To infer membership status, MIAs typically make use of the fact that AI models are often slightly more confident about their predictions on training than on non-training data.

Likelihood-ratio MIAs (LR-MIAs)3,4, the current state-of-the-art in MIAs, frame membership inference as a simple vs. simple hypothesis testing problem on the prediction confidence provided by the target model. In essence, LR-MIAs compare the likelihood of the predicted confidence of the target model for the target record under the null (the target record was not a member) and the alternative hypothesis (the target record was a member). Here, the parameters of the distributions under the two hypotheses are specified by parametric fitting of sample confidence values obtained from reference models. Reference models are models assumed to be trained by the attacker and are ideally, but not necessarily, of similar architecture as the target model and trained on data similar to the training dataset of the target model.

Note that objectively larger threats are posed by privacy attacks with stronger assumptions on a potential attacker, such as access to model parameters17, access to parameter updates during model training18 or, furthermore, the ability to modify the model architecture19,20. However, we do not consider them in this study as their strong assumptions are not realistic for careful, practical deployment scenarios. By contrast, the type of attack we consider here requires querying the target model only once (to obtain a prediction for the target record) and may thus be executed by any attacker posing as a real user of an AI system. Notably, as the attacks we study are executed against fully trained models, data-governance-preserving techniques such as federated/swarm-learning21 provide no protection.

From aggregate to patient-level risk

MIA performance is evaluated through a receiver operating characteristic (ROC) analysis22 on numerous repetitions of the MIA game scenario, in which an untrusted user is challenged to guess the membership status of a given record (Fig. 1a). In practice, owing to the computational cost of training AI models, attack success is typically evaluated using a single target model. More specifically, a target model is trained on a random subset of the training dataset, and subsequently, an ROC analysis is performed on the aggregated membership predictions for all records in the dataset (Fig. 1b). Although practical, this approach has a key shortcoming: it provides no indication of the performance of the attack for individual records or patients.

To address this issue, we propose a simple technique for estimating record-, and by extension, patient-level vulnerability to LR-MIAs (Fig. 1c). In brief, using a large set of target models (N = 200) trained on random patient subsets, we estimate, for each training record, sampling distributions of the confidence of the target model under the null and alternative hypotheses in LR-MIAs. In other words, we estimate empirical distributions of confidence values as provided by target models, partitioned into models trained and not trained on the target record. Because these distributions are assumed to take Gaussian form in LR-MIAs3,4, record-level attack success, as measured by the area under the ROC curve (AUC), can be calculated in closed form (Methods). A high AUC score, close to the maximum value of 1.0, suggests high privacy risk: a MIA for this record could achieve high sensitivity with little to no false positives. Notably, the record-level MIA AUC also offers a probabilistic interpretation22: the record-level MIA AUC is the probability that a confidence score from a target model trained on the target record is larger than a score from a target model not trained on the target record.

Correctly determining the membership status for one of the records contributed by an individual patient reveals the membership status of the patient. Thus, we compute patient-level scores by taking the maximum across all record-level scores for a given patient. The raw record-level scores and the average patient-level scores can be found in Extended Data Fig. 1.

Notably, our technique for measuring record-level attack success reduces to estimating the bi-normal AUC from sample statistics and thus has desirable statistical properties. Its standard error at the record level can be computed in closed form23 (Methods). As expected, using a total of N = 200 target models evenly split between null and alternative hypotheses for each record, the standard error of the record-level MIA AUC is small across all records in the investigated datasets (Extended Data Fig. 2).

Attacking open-source models

Recent advances in attack design have made LR-MIAs much more practical. To illustrate the practical feasibility of conducting MIAs, we demonstrate attacks against two chest radiograph models from the TorchXrayVision24 library. We used the Robust Membership Inference Attack4 (RMIA), an improved LR-MIA that requires only one or two reference models, compared with more than 100 for the Likelihood Ratio Attack3 (LiRA). RMIA achieves this efficiency gain by effectively using reference data (data similar to the target record) alongside the target record to query the target model. Crucially, the attack does not require knowledge of the membership status of the reference data.

We simulated a realistic attack setting in which an attacker lacks access to the training dataset of the target model to train reference models and is further constrained by computational resources. Specifically, we used only a single pre-trained PadChest25 model as a reference model to perform attacks against the CheXpert26 and MIMIC-CXR27 models of the library. In this setting, also known as an offline attack, an attacker incurs no computational cost in training the reference model. Instead, they simply need to obtain predictions from the reference model for both the target record and the reference data. This can be done efficiently on commodity hardware without a graphics processing unit (GPU). To conduct the attack, we queried the target model once to collect confidence values for all target records. Using this collection, we then computed RMIA test statistics for each target record by randomly selecting, independent of membership status, N = 500 confidence values from the other targets in this collection as reference data. This strategy would effectively conceal the additional reference data queries to the target model in a real attack.

We evaluated attack success on a combined dataset of records from CheXpert and MIMIC-CXR (N = 25, 000 each), which were, respectively, labelled as members and non-members for the CheXpert model (v.v. for the MIMIC-CXR model). In this setting, RMIA achieved substantial aggregate success with respective AUC scores of 0.61 and 0.65 (Fig. 2a). Note that owing to the distribution shift between members and non-members, results from this evaluation setting are not directly comparable to the standard evaluation protocol in which members and non-members are sampled at random from the training dataset. Notably, however, such a distribution shift is expected in a real attack, and this setting is thus of high interest.

Near-perfect success for some patients

After demonstrating realistic attacks against two open-source models, we next investigated how effectively MIAs can compromise the privacy of individual patients. To this end, we measured patient-level MIA success across a diverse range of medical datasets using, for each, a large set of target models. Notably, we used state-of-the-art model training techniques (for example, data augmentation, weight decay and learning rate schedules) and furthermore, took explicit countermeasures to prevent overfitting, which is known to exacerbate privacy risks2,28. As a result, the investigated target models, despite being trained on roughly half of the available data each, provide high diagnostic performance within a few percentage points of published baselines (Methods).

Across all investigated datasets and models, we identified a small subset of patients who are highly vulnerable to LR-MIAs. This is indicated by empirical survival functions (eSF) of patient-level MIA AUC scores, which, for a given score, show the proportion of patients with this score or higher (Fig. 2b). By contrast, ROC curves of aggregate attack success and their corresponding AUC scores do not deviate substantially from the random-guessing baseline, thus incorrectly indicating a low attack vulnerability (Fig. 2c and Extended Data Fig. 1c–e). This suggests that average-case metrics of attack success, as used in the standard evaluation protocol, are unsuitable measures of privacy risk. They do not accurately reflect that some records or patients may be highly vulnerable, whereas the vast majority are not.

For the two non-imaging datasets, MIMIC-IV-ED29 (electronic health records) and PTB-XL30 (electrocardiograms), we simulated attack settings in which an attacker only has partial access to the target record (Extended Data Fig. 3). Although MIA success generally decreases under partial data access, a subset of patients retain high AUC scores, even in settings in which the attacker has access to only basic clinical information—such as a patients’ age, sex, chief complaints and vital signs (MIMIC-IV-ED), or only the lead I signal from a 12-lead electrocardiogram (PTB-XL).

We verified how resolvable the discovered vulnerabilities are by training models with different levels of record-level (ε, δ)-DP protection (Fig. 2d,e). As expected, we find that patient-level MIA risk decreases with stronger levels of privacy protection (smaller ε values). Moreover, in most scenarios, we observe no violation of the record-level DP guarantee (indicated by the square brackets in the panel legend), although many patients contributed multiple records. Violations are observed only for a subset of patients under strong privacy protection (ε = 1), in which some patients have MIA AUC scores exceeding the upper bound on the MIA AUC implied by the record-level DP guarantee. This behaviour is expected and could be mitigated by implementing patient-level DP accounting.

Larger models, greater risks

Many of the recent AI success stories have been driven not by methodological advances but by scaling up model and dataset sizes31. In light of this scaling trend, we next investigated the impact of model capacity on MIA success. For Fitzpatrick 17k32 and CheXpert, we trained models with increasing capacity, including wide residual networks33 (WRN-28-2 and WRN-40-4) and vision transformers34 (ViT-B/16 and ViT-L/16). Where computationally feasible, vision transformers were trained on images of different sizes: 64 × 64 and 128 × 128 pixels; this is indicated by a trailing number behind the model name (for example, ViT-B/16-64 and ViT-B/16-128).

We find that MIA success (both at the aggregate and patient levels) increases with model capacity. We observe that the relative share of patients highly vulnerable to MIAs increases greatly for larger models, often by an order of magnitude (Fig. 2f,g). For the dermatology dataset (Fitzpatrick 17k), increasing model capacity yields large gains in diagnostic performance with a pronounced increase between WRN-40-4 and ViT-B/16-128, which was pre-trained on a large dataset of more than 14 million natural images. However, simultaneously, the number of patients with near-perfect attack success (AUC score of 0.95 or higher) increases substantially: 0 (WRN-28-2), 1 out of 10,000 (WRN-40-4), 1 out of 1,000 (ViT-B/16-64) and 1 out of 10 (ViT-B/16-128). We observe a similar trend in the much larger dataset CheXpert, although attack success is generally lower. Notably, for CheXpert, vision transformer models do not achieve diagnostic performance competitive with WRN-based models. This is probably because of the diminished utility of natural-image pre-training for medical greyscale images35.

Attack success varies by subgroup

Motivated by recent findings36,37, which revealed that the diagnostic performance of AI models can differ across patient subgroups, we investigated whether differences in privacy risk exist between subgroups. To this end, we focused our analysis on the most vulnerable records (99th MIA AUC percentile) and compared how frequently a subgroup appears in this extreme-risk tail compared with the overall dataset. We did not consider differences in aggregate attack success, as we previously identified this metric as an unsuitable measure of privacy risk.

We find that extreme MIA risk is unequally distributed across patient subgroups when stratifying by disease status, self-reported race, sex, imaging protocol or health insurance. More precisely, for most comparisons, we observe significant differences in subgroup composition between the most vulnerable records and the overall dataset (Fig. 3 and Extended Data Fig. 4). For example, in MIMIC-IV-ED, records from Black patients, patients with Medicaid insurance or patients diagnosed with cancer were observed more frequently than expected among the most vulnerable records (+31%, +126%, and +18% relative change to the overall dataset, respectively). Raw data on the composition of the extreme MIA risk tails as well as the overall datasets are provided in Supplementary Tables 3–16. To find factors that could explain the observed differences, we performed a post hoc test analysis and computed Pearson residuals for all subgroup comparisons (Fig. 3 and Extended Data Fig. 4).

We primarily observe large, positive Pearson residuals for underrepresented groups in the datasets, suggesting that relative group size influences MIA risk. Consider, for example, EMBED38, a mammography dataset comprising mostly negative findings, that is, unremarkable mammograms of healthy breasts with no indication of a tumour. Models for this dataset are trained to predict breast density, and thus never have direct access to tumour findings. Despite this, benign tumour findings (BI-RADS-2) and tumour findings suspicious of malignancy (BI-RADS-4) account for a disproportionately large share of the most vulnerable records (+60% and +1,179% relative change to the overall dataset, respectively). Similarly, otherwise relatively uncommon images of almost entirely fatty (BI-RADS-A) or extremely dense (BI-RADS-D) breasts also occur disproportionately frequently (+90% and +755%, respectively).

To further investigate the relationship between group size and MIA risk, we conducted a meta-analysis of all computed Pearson residuals (Extended Data Fig. 5). Confirming previous observations, we find that large positive Pearson residuals occur mostly for small groups (those that contribute less than 20% of the records of a dataset). Moreover, we observe a weak to moderate negative correlation between group size and Pearson residuals. This suggests that the observed differences in MIA risk may, at least in part, be driven by group-size differences in the training data.

Discussion

We present data from the first patient-level privacy audit of medical AI models. Our findings confirm early observations of MIA risk heterogeneity39,40,41,42 and, at the same time, substantially advance previous AI privacy auditing efforts along three key dimensions. First, our work marks a shift towards patient-level risk assessment, which is crucial for real-world clinical datasets, in which individuals often contribute multiple, similar records. Second, we demonstrate that aggregate success rates, as used in the standard evaluation protocol and previous subgroup analyses41,42, underestimate true privacy risks. Third, we confirm that MIA vulnerabilities previously observed on low-dimensional benchmark datasets2,3,4,39,40,41,42 are present, and arguably more critical, in large representative clinical datasets. Below, we briefly discuss our findings and their implications.

The fact that MIAs can achieve near-perfect success rates for individual patients is not adequately captured by the standard evaluation protocol, which measures attack success in aggregate across records. This remains true even when evaluating aggregate attack success at very low false-positive rates (for example, 10−4), which is the current standard practice. Thus, reporting standards for AI privacy audits need to change. Audits should report the success of privacy attacks at the level of individual data contributors or, if the necessary patient- or person-level identifiers are unavailable, at the record level.

We observed that the number of patients highly vulnerable to MIAs increases drastically for larger models. Although the magnitude of this change in patient-level risk was previously unknown, other works have also reported greater attack success against larger, more performant models3,5,7,43. This observation that privacy risks grow with model size and predictive performance is explained by theoretical research44,45, which postulates that, for long-tailed data distributions, fitting atypical records from the tail is necessary to achieve optimal performance on unseen data at test time. Our results provide further empirical support for this theory and, together, suggest that a trade-off between patient privacy and model performance is inevitable, particularly for rare diseases. Generally, as we found that the number of patients highly vulnerable to MIAs increases by orders of magnitude with larger models, we recommend carefully evaluating the need for the performance improvements they offer.

We found substantial differences in the frequency with which patients from different subgroups experience extreme MIA risk. The fact that some of these groups (for example, self-reported race subgroups in chest radiographs) are not readily distinguishable by human experts raises concerns that MIA risk differences, which probably exist beyond the stratification variables we investigated, may pass unnoticed in practice. We found that the observed risk differences are driven, at least in part, by group-size differences in the training data. Groups of patients that are underrepresented in a model training dataset are often overrepresented among the records most susceptible to MIAs. By contrast, the opposite often holds for majority groups. This finding—that a disproportionately large share of the AI privacy risk burden rests on underrepresented groups—complements the existing literature on health inequalities, which has reported worse health outcomes and life expectancy for marginalized and minority groups46. Our findings suggest that current trends in medical AI development and deployment could exacerbate these health inequalities. Previous research has shown that the diagnostic performance of AI models, which typically increases with the amount of suitable training data, can be significantly lower for underrepresented (minority) groups36,37,47. Thus, there is a possibility of a vicious cycle in which minority groups place decreasing levels of trust in AI model performance and security, leading to a decreased willingness to contribute to model training datasets.

MIAs facilitate data extraction attacks against generative AI models5,6,7. Thus, our findings have potentially far-reaching implications for generative AI privacy risk assessments. Extraction attacks allow for high-fidelity reconstruction of full individual records from the training dataset of a model and have been demonstrated for large language models5, diffusion-based image generation models6 and recently, aligned, production-level large language models7. Although our study focused on discriminative (diagnostic) AI models, the type of attack we studied is generally applicable and can be used against generative models with little to no modification. We thus see the exploration of our proposed methodology for estimating record- and patient-level MIA success against generative models as an interesting direction for future research. Given the substantial computational resources this would require, exploring scalable approximation techniques is another valuable avenue to investigate.

Unlocking the full potential of medical AI will require training models on vast medical datasets; this depends on gaining and upholding the trust of data-contributing patients. To this end, mathematically verifiable approaches to risk mitigation, such as DP48, are emerging as the most promising solution. DP, by carefully perturbing parameter updates with white noise during model training or fine-tuning49, limits the contribution of the data of any individual to the parameter update and, by extension, to the final model. This provably protects the privacy of any data-contributing patient, no matter how unique or atypical their data may be. Our experimental data confirmed that stronger levels of DP protection effectively reduce MIA success for all data-contributing patients. However, we also observed that mitigating MIAs requires stronger levels of DP protection than previously thought3,50. Specifically, our results indicate that fully mitigating MIAs for all data-contributing patients requires implementing DP protection at the patient level rather than at the record level. Recent research has demonstrated that, in practice, AI models can be trained with strong privacy guarantees while incurring minimal degradation in predictive performance compared with a non-private model51,52,53,54. We are thus optimistic that medical AI models protected by DP will have a significant positive impact on health outcomes globally without endangering the privacy of any data-contributing patient.

In summary, we present evidence that MIAs can be highly effective at compromising the privacy of individual data-contributing patients. Given this vulnerability, medical AI models and their deployment contexts should be assessed for the sensitive information that attackers could obtain by successfully inferring training dataset membership. To prevent privacy harm, we recommend that vulnerable models be protected by verifiable risk mitigation strategies and/or strict access controls.

Methods

CheXpert processing and model training details

CheXpert26 is a large chest radiograph dataset from Stanford Hospital comprising 224,313 chest radiographs from 64,539 patients. CheXpert comes with structured labels that were extracted automatically from free-text radiology reports and indicate the presence of 14 common thoracic conditions. The dataset provides demographic data that were obtained from the electronic data storage system of the hospital and includes information on the sex and age of the patient. Race (self-reported) and ethnicity (self-reported) data of the patient were released separately and can be matched by cross-linking patient IDs. Models for this dataset were trained with data augmentation (random horizontal flipping, random pixel shifts and random rotations) to detect the presence of all 14 conditions. We combined official CheXpert test and validation sets to accurately evaluate diagnostic model performance on a large set of expert-consensus-verified labels (total of N = 668 records). On this unseen data, a WRN-28-2 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves a macro-average AUC score of 0.892 across the CheXpert competition classes: cardiomegaly, oedema, consolidation, atelectasis and pleural effusion.

MIMIC-CXR processing and model training details

MIMIC-CXR27 is a large chest radiograph dataset consisting of 377,110 chest radiographs from 65,379 patients at the Beth Israel Deaconess Medical Center. We use MIMIC-CXR-JPG55, a subsequent re-release of the original dataset that contains images in ‘.jpg’ format and structured labels derived from free-text radiology reports. The structured labels indicate the presence of 14 common thoracic conditions and are in an identical format to the CheXpert dataset. The dataset comprises demographic data that were obtained from the electronic data storage system of the hospital and includes patients’ age, sex, race (self-reported), ethnicity (self-reported), marital status, insurance and mortality information. Some of the demographic data were released separately and can be matched by cross-linking patient identifiers. Models for this dataset were trained with data augmentation (random horizontal flipping, random pixel shifts and random rotations) to detect the presence of all 14 conditions. As expert-verified labels are not available for MIMIC-CXR, we evaluate model performance, as we did for CheXpert, on the combined set of records from the official CheXpert validation and test set (N = 668 records). On this unseen data, a WRN-28-2 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves a macro-average AUC score of 0.851 across all classes.

Fitzpatrick 17k processing and model training details

Fitzpatrick 17k32 is a dermatology dataset consisting of 16,523 RGB (red, green and blue) images from an unknown number of patients. Images in the dataset were initially sourced from two open-source dermatology atlases and complemented with Fitzpatrick skin-type labels, which were provided by trained non-experts. Owing to the noise expected in these labels, we grouped together types I/II, III/IV and V/VI, respectively. The dataset contains no demographic information apart from the Fitzpatrick skin-type labels. Models for this dataset were trained with data augmentation (random horizontal flipping and random pixel shifts) to classify images into one of three mutually exclusive classes (non-neoplastic, benign and malignant). Diagnostic performance was evaluated on the expert-verified test set (N = 348 records). On this unseen data, a WRN-28-2 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves an accuracy of 0.746 and a macro-average AUC score of 0.872 across the three classes.

Note that patient identifiers are not available for Fitzpatrick 17k. Thus, to compute patient-level MIA AUC, we assume that each record belongs to a unique patient.

FairVision processing and model training details

FairVision56 is an ophthalmoscopy dataset from the Department of Ophthalmoscopy at Harvard Medical School consisting of 30,000 scanning laser ophthalmoscopy fundus images from 30,000 unique patients. The dataset comprises demographic data that were obtained from the electronic data storage system of the hospital and includes patients’ age, gender, race (self-reported), ethnicity (self-reported), preferred language and marital status. The dataset consists of three patient cohorts, each with 10,000 patients or images and their respective disease labels indicating the presence or severity of age-related macular degeneration (AMD), diabetic retinopathy (DR) and glaucoma. To combine all available data into a single dataset for a seven-class classification problem, we merged the disease labels across the three cohorts under the assumption of mutual exclusivity (no patient from one cohort had a disease from another). The glaucoma label ‘no’, and AMD label ‘normal’ were merged to create the ‘healthy’ label. A basic image quality screening was performed, in which dark, low-contrast images with image-wise mean and s.d. below 1.0 and 3.0, respectively, were discarded. This left 28,542 images from 28,542 patients for model training. Models for this dataset were trained with data augmentation (random horizontal flipping, random pixel shifts) to classify images into one of seven mutually exclusive classes: healthy, non-vision-threatening DR, vision-threatening DR, glaucoma, early AMD, intermediate AMD and late AMD. Diagnostic performance was evaluated on N = 500 randomly chosen records from the official test set; the remaining images were used for model training. On these unseen data, a WRN-28-2 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves a macro-average AUC score of 0.817 across all classes.

EMBED processing and model training details

EMBED38 is a large mammography dataset consisting of 480,323 mammograms from 23,253 patients who underwent breast cancer screening or diagnosis at four hospital sites of Emory University in Atlanta. The dataset comprises demographic data that were obtained from electronic health records and includes patients’ age, race (self-reported), ethnicity (self-reported) and insurance status. We considered only craniocaudal and mediolateral oblique (MLO) views from screening exams and excluded any non-female patients, thus reducing the dataset size to 360,159 mammograms from 19,429 female patients. Following ref. 57, we train models to predict breast density, an important clinical risk factor for breast cancer. Models for this dataset were trained with data augmentation (random horizontal flipping and random pixel shifts) to classify images into one of the four BI-RADS breast density categories. Diagnostic performance was evaluated on a subset of 500 randomly picked patients (N = 11,121 images). On this unseen data, a WRN-28-2 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves a macro-average AUC score of 0.936 across all classes.

PTB-XL processing and model training details

PTB-XL30 is a cardiology dataset comprising 21,799 electrocardiograms from 18,869 patients. The dataset contains 12-lead electrocardiograms sampled at 100 Hz and 10 s long paired with their corresponding diagnostic label (multi-hot encoded), indicating the presence of common cardiac conditions. The dataset comprises demographic data obtained from the electronic data storage system of the hospital, including patients’ age and sex. Models for this dataset were trained without data augmentation to perform multi-label classification for the superclasses: normal, myocardial infarction, conduction disturbance, ST/T-changes and hypertrophy. Diagnostic performance was evaluated on the official test set. On these unseen data, a convolutional residual network58 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves a macro-average AUC score of 0.916 across all classes.

MIMIC-IV-ED processing and model training details

MIMIC-IV-ED29 is a large electronic health record dataset comprising data from 418,007 records from 201,213 patients at the emergency department of the Beth Israel Deaconess Medical Center. The dataset contains demographic data that were obtained from the electronic data storage system of the hospital and includes patients’ age, sex, race (self-reported), ethnicity (self-reported), marital status, insurance and mortality information. We followed the pre-processing steps in ref. 59 and trained models without data augmentation to predict hospitalization outcome (binary classification) based on 64 clinical features (discrete and continuous). Diagnostic performance was evaluated on the official test set in ref. 59. On these unseen data, a tabular residual network60 with around 1.5 million parameters trained on the records from a random 50% patient subset achieves an AUC score of 0.815 across all classes.

General model training details

All investigated models were trained using stochastic gradient descent with momentum, exponential moving parameter average (EMA), weight decay, data augmentation and a learning rate schedule in the form of cosine decay with linear warm-up. Hyperparameter values were determined by a random search to maximize diagnostic performance (macro-average AUC) on unseen data. Specifically, for each dataset, a random search with N = 50 trials over suitable values for learning rate, weight decay, momentum and EMA decay was conducted (Supplementary Table 1). Training and test data were rescaled to have values within [−1, 1]. To prevent overfitting, model checkpointing was used, ensuring that only the model with the best generalization performance was retained after the training process terminated. We estimate that reproducing all of our experimental data would require around 900 h on a single A100 GPU.

Strategies to improve training efficiency

Improving training efficiency was crucial because of the limited computational resources available to us. Thus, we resized all images to a spatial resolution of 64 × 64 pixels unless otherwise stated in the main text. Moreover, to further improve training efficiency, we used compilation caching with Keras61 JAX62 backend and used mixed-precision training with the default Keras policy: ‘mixed_float16’. Supplementary Table 1 shows training times of residual networks with about 1.5 million parameters for each of the investigated datasets using a single A100 GPU. Notably, the training runs required for our analysis are independent of one another, so the process can be easily accelerated by parallelizing them across multiple GPUs, with no need for inter-device communication during training.

Adapting LR-MIAs to multi-label classification

LR-MIAs were designed for multi-class classification and are thus not directly compatible with multi-label classification settings in which multiple classes (diseases) may be present in a single input. We adapt LR-MIAs to the multi-label classification setting by using a softmax activation function on the raw network output (instead of the sigmoid function typically used). Then, to obtain a scalar quantity, we take the sum over the confidence values for all present classes (conditions). Note that our strategy does not change the assumed threat model. An attacker with access to the post-sigmoid activation confidence values can obtain the values necessary for our adapted attack by applying the inverse sigmoid (logit) function. Empirically, we found that our strategy performed better than estimating multivariate Gaussians for the confidence values of all present classes (the more natural adaptation of LR-MIAs to multi-label settings). This is probably because accurately estimating multivariate Gaussians requires more samples than their univariate counterparts.

Estimating record-level LR-MIA success

Quantifying record-level MIA vulnerability requires measuring MIA performance independently for each target record across many target models. To do this with a manageable amount of computational resources, we leverage the fact that LR-MIAs are essentially hypothesis tests and directly estimate the empirical sampling distributions of the confidence of the target model under the null and alternative hypotheses. As these distributions are assumed to take Gaussian form in both LiRA3 and RMIA4, the record-level MIA AUC for a target record x admits a closed-form solution:

$$\hat{\theta }(x)=\varPhi \left(\frac{{\mu }_{x}-{\mu }_{{x}^{{\prime} }}}{\sqrt{{\sigma }_{x}^{2}+{\sigma }_{{x}^{{\prime} }}^{2}}}\right).$$

(1)

Here, Φ denotes the CDF of the standard normal and μx, μx′, \({\sigma }_{x}^{2}\), \({\sigma }_{{x}^{{\prime} }}^{2}\) denote sample means and variances of the logit-transformed confidence values provided by target models partitioned into models trained on x and models not trained on x (denoted by x and x′, respectively). To obtain a scalar quantity from the vector of predicted confidences for all possible classes, we follow the standard protocol3,4 and index the vector at the correct class indicated by the corresponding label. Multi-label confidence values were transformed, as explained in the previous section. Note that, for the Gaussian assumption of LR-MIAs to hold, confidence values need to be logit-transformed (\({\rm{logit}}(p)\,=\,{\rm{ln}}\,\frac{p}{1-p}\)). For models trained with data augmentation, we use N = 16 augmented queries to the target model; confidence values are then averaged across augmentations.

We calculate the standard error of \(\widehat{\theta }\) (the bi-normal AUC) as described in ref. 23:

$$\mathrm{SE}(\hat{\theta }(x))=\sqrt{\frac{\hat{\theta }(x)(1-\hat{\theta }(x))+({N}_{{x}^{{\prime} }}-1)({Q}_{1}-\hat{\theta }(x))+({N}_{x}-1)({Q}_{2}-\hat{\theta }{(x)}^{2})}{{N}_{{x}^{{\prime} }}\times {N}_{x}}},$$

(2)

where \(\widehat{\theta }(x)\) is the estimated AUC score from equation (1), \({Q}_{1}=\frac{\widehat{\theta }(x)}{1-\widehat{\theta }(x)}\) and \({Q}_{2}=\frac{2\widehat{\theta }{(x)}^{2}}{1+\widehat{\theta }(x)}\). Nx and \({N}_{{x}^{{\prime} }}\) refer to the number of target models trained and not trained on the target record, respectively. To ensure that each target model effectively reduces the standard error of the estimated AUC across all records, it is important to train the target models on random data subsets sampled in an unbiased fashion (see ref. 3 for details). This guarantees that Nx and \({N}_{{x}^{{\prime} }}\) are equal for all x.

Note that it is important to perform the subset sampling at the patient- rather than the record level to accurately estimate patient-level MIA vulnerability. This is to ensure that either all or none of a given patients’ records are included in a target model’s training dataset. In initial experiments, we sampled subsets at the record level and found that this led to underestimating MIA success for patients who contributed multiple similar records.

Model scaling and differential privacy experiments

All models in the model scaling experiments were trained with the same hyperparameter values found during the initial hyperparameter tuning procedure described above. This decision was made because of the limited computational resources available to us.

Models for the DP experiments were trained with DP-Adam49,63 using the jax-privacy library64. Concretely, hyperparameters for these models were determined for a fixed privacy budget of ε = 100 using a random search with N = 20 trials over suitable values for the learning rate (see Supplementary Table 2 for results); EMA and weight decay were not used. The clipping threshold for DP-Adam was determined using the procedure described in ref. 65. Note that the diagnostic performance of the models could likely be improved by performing a separate hyperparameter search for each privacy budget and adapting the other engineering techniques proposed in ref. 53. Bounds on the MIA AUC implied by the record-level (ε, δ)-DP guarantee, where calculated from the trade-off curve obtained by converting the privacy profile of the sub-sampled Gaussian mechanism66 to f-DP67.

Attacking open-source chest radiograph models

We follow the standard protocol as described in the original publication4 to conduct RMIA on the CheXpert and MIMIC-CXR models from TorchXrayVision. We use offline RMIA with a single pre-trained reference model (PadChest25, also from TorchXrayVision). Offline approximation parameter a was calculated as described in ref. 4; a = 0.857 and 0.918 for CheXpert and MIMIC-CXR, respectively. We did not use augmented queries for the target model; multi-label confidence values were transformed as described previously.

Notably, the open-source implementation of RMIA relies on a large outer product, which requires prohibitive amounts of memory when conducting the attack in parallel for many target records. Our adapted implementation greatly improves on this by holding only relevant entries of the large outer product matrix in memory, thereby easily scaling to large numbers of target records.

Measuring MIA success with partial target records

We simulate settings in which MIAs are executed using only partial access to target records. Specifically, for MIMIC-IV-ED and PTB-XL, we train target models on random subsets of the full datasets and subsequently obtain confidence scores for these models when queried on partial target records, in which we redacted increasing amounts of information. We then compute record- and patient-level MIA AUC scores from confidence values as described previously. To redact information from target records, we simply set all relevant feature values to 0. For MIMIC-IV-ED, we investigate a setting we term ‘realistic subset’, in which all except the following 20 features are redacted: age, gender_binarized, triage_temperature, triage_heartrate, triage_resprate, triage_o2sat, triage_sbp, triage_dbp, triage_pain, triage_acuity, chiefcom_chest_pain, chiefcom_abdominal_pain, chiefcom_headache, chiefcom_shortness_of_breath, chiefcom_back_pain, chiefcom_cough, chiefcom_nausea_vomiting, chiefcom_fever_chills, chiefcom_syncope and chiefcom_dizziness. These features represent patients’ age, sex, chief complaints and vital signs at triage that we deem realistically obtainable for an adversary. We refer to Extended Data Table 1 from ref. 59 for a full description of the features in MIMIC-IV-ED. For MIMIC-IV-ED, we investigate another setting ‘most predictive’, in which all features except the following 10 features that ref. 59 identified as most predictive for hospitalization are redacted: age, triage_acuity, triage_sbp, triage_heartrate, triage_dbp, triage_temperature, triage_pain, triage_o2sat, triage_resprate and n_hosp_365d. For PTB-XL, we investigate settings in which an attacker has access to only temporal snippets of the 12-lead ECG signal and another setting in which only the signal from lead I is available. As for MIMIC-IV-ED, relevant information was redacted by replacing all signal values with zeros.

Subgroup analysis

Our subgroup analysis compares the composition of the records most vulnerable to MIAs to that of the overall dataset. We measure O, the number of records observed for a given subgroup in the 99th record-level MIA AUC percentile and compare it with E, the expected number of records for this subgroup in the 99th percentile based on the composition of the overall training dataset. To this end, we compute the χ2 test statistic and Pearson residuals, which measure the contribution of a subgroup to the test statistic:

$$\mathrm{Pearson}\,\mathrm{residual}=\frac{O\,-\,E}{\sqrt{E}}.$$

(3)

The relative change between the observed and expected counts in the 99th percentile, as reported in the main body text, is calculated as

$${\varDelta }_{\mathrm{relative}}( \% )=\frac{(O\,-\,E)}{E}\times 100.$$

(4)

Note that we excluded records with incomplete (other/unknown) subgroup information from the subgroup analysis. Small discrepancies between Pearson residuals and data in Supplementary Tables 3–16 are thus to be expected.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

All datasets used in this study are publicly available for research purposes. See GitHub (https://github.com/moritzknolle/leakoscope) for access instructions.

Code availability

Code and instructions to reproduce our results are available at GitHub (https://github.com/moritzknolle/leakoscope) and are permanently archived on Zenodo68 (https://doi.org/10.5281/zenodo.20124226).

References

Fleming, K. A. et al. The Lancet Commission on diagnostics: transforming access to diagnostics. Lancet 398, 1997–2050 (2021).

Article

PubMed

PubMed Central

Google Scholar

Shokri, R., Stronati, M., Song, C. & Shmatikov, V. Membership inference attacks against machine learning models. In Proc. 2017 IEEE Symposium on Security and Privacy (SP) 3–18 (IEEE, 2017).

Carlini, N. et al. Membership inference attacks from first principles. In Proc. 2022 IEEE Symposium on Security and Privacy (SP) 1897–1914 (IEEE, 2022).

Zarifzadeh, S., Liu, P. & Shokri, R. Low-cost high-power membership inference attacks. In Proc. 41st International Conference on Machine Learning 58244–58282 (PMLR, 2024).

Carlini, N. et al. Extracting training data from large language models. In Proc. 30th USENIX Security Symposium 2633–2650 (USENIX, 2021).

Carlini, N. et al. Extracting training data from diffusion models. In Proc. 32nd USENIX Security Symposium 5253–5270 (USENIX, 2023).

Nasr, M. et al. Scalable extraction of training data from aligned, production language models. In Proc. Thirteenth International Conference on Learning Representations (ICLR, 2025).

Tramèr, F., Zhang, F., Juels, A., Reiter, M. K. & Ristenpart, T. Stealing machine learning models via prediction APIs. In Proc. 25th USENIX Security Symposium 601–618 (USENIX, 2016).

Carlini, N. et al. Stealing part of a production language model. In Proc. 41st International Conference on Machine Learning 5680–5705 (ICML, 2024).

Yoo, S.-K. et al. Prediction of checkpoint inhibitor immunotherapy efficacy for cancer using routine blood tests and clinical data. Nat. Med. 31, 869–880 (2025).

Article

CAS

PubMed

PubMed Central

Google Scholar

FDA. Artificial intelligence-enabled medical devices. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices (2024).

Seh, A. H. et al. Healthcare data breaches: insights and implications. Healthcare 8, 133 (2020).

Article

PubMed

PubMed Central

Google Scholar

Albert Haro Abad, S. C. Health Threat Landscape: ENISA Report 2023. Technical report (European Union Agency for Cybersecurity, 2023).

Narayanan, A. & Shmatikov, V. Robust de-anonymization of large sparse datasets. In Proc. 2008 IEEE Symposium on Security and Privacy (sp 2008) 111–125 (IEEE, 2008).

Gadotti, A., Rocher, L., Houssiau, F., Creţu, A.-M. & de Montjoye, Y.-A. Anonymization: the imperfect science of using data while preserving privacy. Sci. Adv. 10, eadn7053 (2024).

Article

PubMed

PubMed Central

Google Scholar

Rocher, L., Hendrickx, J. M. & de Montjoye, Y.-A. A scaling law to model the effectiveness of identification techniques. Nat. Commun. 16, 347 (2025).

Article

ADS

CAS

PubMed

PubMed Central

Google Scholar

Suri, A., Zhang, X., Evans, D. Do parameters reveal more than loss for membership inference? In Proc. 2nd Workshop on High-dimensional Learning Dynamics (HiLD, 2024).

Geiping, J., Bauermeister, H., Dröge, H. & Moeller, M. Inverting gradients - how easy is it to break privacy in federated learning? In Proc. 34th International Conference on Neural Information Processing Systems 16937–16947 (Curran Associates, 2020).

Fowl, L. H., Geiping, J., Czaja, W., Goldblum, M. & Goldstein, T. Robbing the fed: directly obtaining private data in federated learning with modified models. In Proc. International Conference on Learning Representations (ICLR, 2022).

Feng, S. & Tramèr, F. Privacy backdoors: stealing data with corrupted pretrained models. In Proc. 41st International Conference on Machine Learning 13326–13364 (PMLR, 2024).

Warnat-Herresthal, S. et al. Swarm Learning for decentralized and confidential clinical machine learning. Nature 594, 265–270 (2021).

Article

ADS

CAS

PubMed

PubMed Central

Google Scholar

Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 27, 861–874 (2006).

Article

ADS

Google Scholar

Hanley, J. A. & McNeil, B. J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36 (1982).

Article

CAS

PubMed

Google Scholar

Cohen, J. P. et al. TorchXRayVision: a library of chest X-ray datasets and models. In Proc. 5th International Conference on Medical Imaging with Deep Learning 231–249 (PMLR, 2022).

Bustos, A., Pertusa, A., Salinas, J.-M. & de la Iglesia-Vayá, M. PadChest: a large chest x-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020).

Article

PubMed

Google Scholar

Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proc. Thirty-Third AAAI Conference on Artificial Intelligence 590–597 (PKP Publishing, 2019).

Johnson, A. E. W. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data 6, 317 (2019).

Article

PubMed

PubMed Central

Google Scholar

Yeom, S., Giacomelli, I., Fredrikson, M. & Jha, S. Privacy risk in machine learning: analyzing the connection to overfitting. In Proc. 2018 IEEE 31st Computer Security Foundations Symposium (CSF) 268–282 (IEEE, 2018).

Johnson, A. et al. MIMIC-IV-ED (v.1.0) (PhysioNet, 2021).

Wagner, P. et al. PTB-XL, a large publicly available electrocardiography dataset. Sci. Data 7, 154 (2020).

Article

PubMed

PubMed Central

Google Scholar

Kaplan, J. et al. Scaling laws for neural language models. Preprint at https://doi.org/10.48550/arXiv.2001.08361 (2020).

Groh, M. et al. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proc. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 1820–1828 (CVPRW, 2021).

Zagoruyko, S. & Komodakis, N. Wide residual networks. In Proc. British Machine Vision Conference (BMVC) (eds Wilson, R. C. et al.) 87.1–87.12 (BMVA Press, 2016).

Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 45–67 (ICLR, 2021).

Raghu, M., Zhang, C., Kleinberg, J. & Bengio, S. Transfusion: understanding transfer learning for medical imaging. In Proc. 33rd International Conference on Neural Information Processing Systems 3347–3357 (NIPS, 2019).

Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y. & Ghassemi, M. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nat. Med. 27, 2176–2182 (2021).

Article

CAS

PubMed

PubMed Central

Google Scholar

Daneshjou, R. et al. Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci. Adv. 8, eabq6147 (2022).

Article

ADS

PubMed

PubMed Central

Google Scholar

Jeong, J. J. et al. The EMory BrEast imaging Dataset (EMBED): a racially diverse, granular dataset of 3.4 million screening and diagnostic mammographic images. Radiol. Artif. Intell. 5, e220047 (2023).

Article

PubMed

PubMed Central

Google Scholar

Long, Y. et al. A pragmatic approach to membership inferences on machine learning models. In Proc. 2020 IEEE European Symposium on Security and Privacy (EuroS&P) 521–534 (IEEE, 2020).

Aerni, M., Zhang, J. & Tramèr, F. Evaluations of machine learning privacy defenses are misleading. In Proc. 2024 on ACM SIGSAC Conference on Computer and Communications Security 1271–1284 (CCS, 2024).

Kulynych, B., Yaghini, M. & Cherubin, G., Veale, M., Troncoso, C. Disparate vulnerability to membership inference attacks. In Proc. Privacy Enhancing Technologies 460–480 (sciendo, 2022).

Chang, H. & Shokri, R. On the privacy risks of algorithmic fairness. In Proc. 2021 IEEE European Symposium on Security and Privacy (EuroS&P) 292–303 (IEEE, 2021).

Carlini, N. et al. Quantifying memorization across neural language models. In Proc. Eleventh International Conference on Learning Representations (ICLR, 2023).

Feldman, V. Does learning require memorization? a short tale about a long tail. In Proc. 52nd Annual ACM SIGACT Symposium on Theory of Computing 954–959 (STOC, 2020).

Feldman, V. & Zhang, C. What neural networks memorize and why: discovering the long tail via influence estimation. In Proc. 34th International Conference on Neural Information Processing Systems 2881–2891 (NIPS, 2020).

World Health Organization et al. World Report on Social Determinants of Health Equity, 2025 (WHO, 2025).

Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).

Article

ADS

CAS

PubMed

Google Scholar

Dwork, C. & Roth, A. The algorithmic foundations of differential privacy. Foundations Trends Theoret. Comput. Sci. 9, 211–487 (2014).

Article

MathSciNet

Google Scholar

Abadi, M. et al. Deep learning with differential privacy. In Proc. 2016 ACM SIGSAC Conference on Computer and Communications Security 308–318 (CCS, 2016).

Nasr, M., Songi, S., Thakurta, A., Papernot, N. & Carlini, N. Adversary instantiation: lower bounds for differentially private machine learning. In Proc. 2021 IEEE Symposium on Security and Privacy (SP) 866–882 (IEEE, 2021).

Ziller, A. et al. Reconciling privacy and accuracy in AI for medical imaging. Nat. Mach. Intell. 6, 764–774 (2024).

Article

Google Scholar

Berrada, L. et al. Unlocking accuracy and fairness in differentially private image classification. Preprint at https://doi.org/10.48550/arXiv.2308.10888 (2023).

De, S., Berrada, L., Hayes, J., Smith, S. L. & Balle, B. Unlocking high-accuracy differentially private image classification through scale. Preprint at https://doi.org/10.48550/arXiv.2204.13650 (2022).

Mckenna, R. et al. Scaling laws for differentially private language models. In Proc. 42nd International Conference on Machine Learning 43375–43398 (PMLR, 2025).

Johnson, A. E. W. et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. Preprint at https://doi.org/10.48550/arXiv.1901.07042 (2019).

Luo, Y. et al. FairVision: equitable deep learning for eye disease screening via fair identity scaling. Preprint at https://doi.org/10.48550/arXiv.2310.02492 (2023).

Khara, G. et al. Generalisable deep learning method for mammographic density prediction across imaging techniques and self-reported race. Commun. Med. 4, 21 (2024).

Article

CAS

PubMed

PubMed Central

Google Scholar

Wang, Z., Yan, W. & Oates, T. Time series classification from scratch with deep neural networks: a strong baseline. In Proc. 2017 International Joint Conference on Neural Networks (IJCNN) 1578–1585 (IEEE, 2017).

Xie, F. et al. Benchmarking emergency department prediction models with machine learning and public electronic health records. Sci. Data 9, 658 (2022).

Article

PubMed

PubMed Central

Google Scholar

Gorishniy, Y., Rubachev, I., Khrulkov, V. & Babenko, A. Revisiting deep learning models for tabular data. In Proc. 35th International Conference on Neural Information Processing Systems (eds Ranzato, M. et al.) 18932–18943 (NIPS, 2021).

Chollet, F. et al. Keras. https://keras.io (2015).

Bradbury, J. et al. JAX: composable transformations of Python+NumPy programs. GitHub http://github.com/google/jax (2026).

Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations (ICLR, 2015).

Balle, B. et al. JAX-Privacy: algorithms for privacy-preserving machine learning in JAX. GitHub http://github.com/google-deepmind/jax_privacy (2026).

Ponomareva, N. et al. How to DP-fy ML: a practical guide to machine learning with differential privacy. J. Artif. Intell. Res. 77, 1113–1201 (2023).

Article

MathSciNet

Google Scholar

Balle, B., Barthe, G. & Gaboardi, M. Privacy profiles and amplification by subsampling. J. Priv. Confid. https://doi.org/10.29012/jpc.726 (2020).

Dong, J., Roth, A. & Su, W. J. Gaussian differential privacy. J. R. Stat. Soc. Ser. B. 84, 3–37 (2022).

Article

MathSciNet

Google Scholar

moritzknolle. moritzknolle/leakoscope: V1 (v.v1). Zenodo https://doi.org/10.5281/zenodo.20124226 (2026).

Download references

Acknowledgements

We thank A. Ziller, L. Lux and many others from the AIM Lab for their feedback and support. We acknowledge computational resources provided by the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities.

Funding

This work was partially funded by the Konrad Zuse School of Excellence in Reliable AI (relAI), ERC Grant Deep4MI (grant no. 884622) and the German Research Foundation (project no. 532139938). B.G. received support from the Royal Academy of Engineering as part of his Research Chair in Safe Deployment of Medical Imaging AI. Open access funding provided by Technische Universität München.

Author information

Author notes

These authors contributed equally: Ben Glocker, Daniel Rueckert

Authors and Affiliations

Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany

Moritz A. Knolle, Martin J. Menten, Friederike Jungmann, Felix Meissen & Daniel Rueckert

Munich Center for Machine Learning, Munich, Germany

Martin J. Menten, Ben Glocker & Daniel Rueckert

Department of Computing, Imperial College London, London, UK

Martin J. Menten & Daniel Rueckert

Institute for Diagnostic and Interventional Radiology, TUM University Hospital, Technical University of Munich, Munich, Germany

Friederike Jungmann

Hasso Plattner Institute for Digital Engineering, University of Potsdam, Potsdam, Germany

Georgios Kaissis

Authors

Moritz A. Knolle

Martin J. Menten

Friederike Jungmann

Felix Meissen

Ben Glocker

Daniel Rueckert

Georgios Kaissis

Contributions

M.A.K. and G.K. conceptualized the study with input from M.J.M., B.G. and D.R.; M.A.K. created the core technical implementation and conducted the experiments. G.K. and M.A.K. contributed to the differential privacy experiments. F.J. provided clinical inputs to the research. F.M. provided feedback and helped frame the initial draft of the paper. All authors contributed to the drafting of the manuscript. M.A.K., M.J.M., F.J., B.G., D.R. and G.K. contributed to revising the manuscript.

Corresponding author

Correspondence to

Moritz A. Knolle.

Ethics declarations

Competing interests

B.G. is a part-time employee at DeepHealth. G.K. is a part-time employee at Google DeepMind.

Peer review

Peer review information

Nature thanks Sana Tonekaboni and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Additional MIA success data.

a, eSF plots of raw record-level MIA AUC scores computed using N = 200 target models. b, eSF plots of mean patient-level MIA AUC scores computed using N = 200 target models. c–e, ROC analysis of aggregate MIA success (R = 190 reference models, online) for LiRA on a linear scale (c) and RMIA on linear (d) and log-log scale (e). ROC curve plots show mean and standard deviation (vertical average) aggregate attack success for N = 10 target models; on the log-log scale, the standard deviation was omitted to improve visibility. Dashed grey lines in ROC curve plots indicate random-guessing performance, dashed lines in eSF plots indicate 95% Greenwood CI.

Extended Data Fig. 2 Ablation study on the number of target models used to estimate record-level MIA AUC scores.

Panels show box plots of the standard error values for record-level MIA AUC scores in a given dataset when varying the number of target models used for record-level MIA AUC estimation. White box plots show data for all records in a given dataset, red box plots show data for the 99th record-level MIA AUC percentile. In box plots, the centre line indicates the median, box bounds represent 25th and 75th percentiles, whiskers extend to 1.5 times the interquartile range from the box bounds, and triangles indicate largest outlier value.

Extended Data Fig. 3 MIAs remain highly successful with partial target record access.

a, Patient- and record-level MIA AUC scores for MIMIC-IV-ED. We simulate an attacker with access to: all clinical features (F = 64), a realistic subset of features related only to a patient’s current stay (F = 20) and the ten most predictive features for hospitalisation (F = 10). b, Patient- and record-level MIA AUC scores for PTB-XL. We simulate an attacker with access to: the full twelve-lead ECG signal, Lead I only, and a range of temporal snippets, each 1 to 5 seconds long. Dashed lines indicate 95% Greenwood confidence intervals; record-level MIA AUC scores were computed using N = 200 target models for each experimental setting.

Extended Data Fig. 4 Subgroup differences in extreme MIA risk for FairVision, Fitzpatrick17k and PTB-XL.

Panel rows show two-sided χ2-test results and Pearson residuals for subgroup counts in the set of highly vulnerable records (99th record-level MIA AUC percentile) for FairVision (a), Fitzpatrick 17k (b) and PTB-XL (c). Column titles show stratification variables; bars are coloured according to group size in the training dataset; *** and n.s. denote p ≤ 0.001 and not significant, respectively; multiple comparison correction was performed row-wise using the Bonferroni method. From left to right, the FairVision disease abbreviations refer to healthy, non-vision-threatening diabetic retinopathy; vision-threatening diabetic retinopathy; glaucoma; early age-related macular degeneration (AMD); intermediate AMD; and late AMD. From left to right, the Fitzpatrick 17k disease abbreviations refer to non-neoplastic, malignant, benign, inflammatory, malignant epidermal, genodermatoses, benign dermal, benign epidermal, malignant melanoma, benign melanocyte, malignant cutaneous lymphoma, and malignant dermal. Fitzpatrick skin type is a clinical categorisation of human skin colour that describes sensitivity to sun exposure, ranging from sensitive, always burns, never tans (Type I) to sun-insensitive, never burns, deeply pigmented (Type VI). From left to right adjusted p values read. Fairvision: < 1 × 10−100, 0.53, 0.85. Fitzpatrick 17k: < 1 × 10−100, < 1 × 10−100, 0.74. PTB-XL: 0.67, 0.53.

Extended Data Fig. 5 Underrepresented groups experience disproportionally high MIA risk.

Scatter plot of Pearson residuals against group size (relative number of records contributed) for all subgroup comparisons. Across the investigated datasets, group size is weakly/moderately inversely correlated with Pearson residual as indicated by Spearman’s rank correlation coefficient ρ and Pearson’s correlation coefficient r.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Cite this article

Knolle, M.A., Menten, M.J., Jungmann, F. et al. Disparate privacy risks from medical AI.

Nature (2026). https://doi.org/10.1038/s41586-026-10688-0

Download citation

Received: 18 March 2025

Accepted: 21 May 2026

Published: 24 June 2026

Version of record: 24 June 2026

DOI: https://doi.org/10.1038/s41586-026-10688-0