An algorithm can stay exactly the same while its predictions quietly become less reliable. In healthcare, understanding why is a question of clinical safety before it is a question of statistics.
The model has not changed. The patients have.
Imagine a hospital introducing an AI system to predict complications after cardiac surgery. In development the model reaches an AUROC above 0.85, its predicted probabilities look well calibrated, and internal validation suggests it can be trusted.
It goes into clinical use. Two years later the code, the coefficients and the decision thresholds are exactly as they were. Something, though, is off: the predicted risks no longer track the outcomes clinicians are seeing.
How can a model that has not changed by a single coefficient become less reliable? Perhaps the hospital now operates on older, more complex patients. Perhaps perioperative care has improved and mortality has fallen. Perhaps a new electronic health record (EHR) codes comorbidities differently. Or the laboratory has changed an assay, and nobody updated the software that prepares the model’s inputs.
Same algorithm. Different clinical environment.
This is dataset shift. It matters because medical AI is often judged as if validation were a permanent property of the model. It is not. A validation describes performance in one population, one setting and one period of time; change any of the three and the question has to be asked again. I have long argued that static risk models sit uneasily in a specialty that changes as quickly as ours, and dataset shift is, in a sense, that argument restated in statistical terms.
Finlayson and colleagues made a related point in The New England Journal of Medicine: clinicians and multidisciplinary governance teams have complementary roles in recognising dataset shift and responding to it [1]. Nor is the problem confined to deep neural networks. Conventional risk scores, regression models, image classifiers and AI-enabled decision support are all exposed to it.
What changes when a dataset shifts?
A prediction model estimates an outcome, Y, from a set of clinical variables, X. During development it learns from one joint distribution, P development (X,Y). In another hospital, or in the same hospital some years later, the corresponding distribution P deployment (X,Y) may differ.
Does every difference matter? No. Some changes barely touch the predictions. Others undermine discrimination, calibration or clinical utility. Two terms are worth keeping apart: dataset shift is a change in the environment that generates the data; performance drift is a change in how well the model performs. The literature does not always use them consistently, hence the explicit definitions. Data drift is sometimes used, more narrowly, for changes in the input distributions over time.
In practice several mechanisms often operate at once.
When the patients change: covariate shift
In the strict statistical sense, covariate shift means that the distribution of the predictors changes while their relationship with the outcome stays the same: P(X) moves, P(Y|X) does not.
Take a cardiac surgical model developed in patients with a mean age of 62. A few years later the mean age is 71, with more renal dysfunction and more diabetes. The model now meets a different case mix, possibly including combinations of characteristics that were rare in its development data.
That alone does not make the model wrong. If the predictor–outcome relationship holds and the new patients fall within the range the model was built on, individual risk estimates can remain valid. Concern grows when the model has to extrapolate beyond that range, or when the change in case mix arrives together with changes in treatment and in the predictor–outcome relationship itself. In cardiac surgery, in my experience, the two seldom arrive separately.
When the event rate changes: label shift
Label shift is a more restrictive situation: the prevalence of the outcome changes, while the distribution of the predictors within each outcome class, P(X|Y), stays stable. A change in referral patterns can alter prevalence, although real-world referral changes seldom meet the assumptions of pure label shift.
Why should prevalence matter if the test itself is unchanged? Consider an illustrative diagnostic test with 80% sensitivity and 90% specificity, applied to 1,000 patients:
| Measure | Disease prevalence 5% | Disease prevalence 15% |
|---|---|---|
| Patients with the disease | 50 | 150 |
| True positives | 40 | 120 |
| False positives | 95 | 85 |
| Positive predictive value | 29.6% | 58.5% |
Sensitivity and specificity are identical in both columns. What a positive result means is not: the positive predictive value roughly doubles. The figures are plain arithmetic, not estimates from a clinical cohort. Prevalence shifts can also disturb calibration while ranking performance stays much the same.
When clinical relationships change: concept drift
Concept drift usually refers to a change in P(Y|X): patients with the same recorded characteristics no longer carry the same risk.
Cardiac surgery offers an intuitive example. Better myocardial protection, perioperative haemodynamic management, postoperative intensive care and mechanical circulatory support can improve outcomes in patients who used to be considered high risk. A mortality model built under earlier conditions will then tend to overestimate.
The opposite happens when a new complication, pathogen or treatment-related hazard raises risk. In both directions the model may go on ranking patients reasonably well while assigning absolute probabilities that are systematically wrong.
Which direction is worse? Overestimation can steer a patient away from an operation they would have tolerated. Underestimation can make a team complacent. Neither is benign.
When the measurement changes: coding and measurement shift
Sometimes the biology stays put and it is the data entering the model that change.
Moving from a conventional to a high-sensitivity troponin assay can alter both the scale and the interpretation of the values. Switching creatinine reporting from mg/dL to µmol/L without converting units produces a far cruder error, since the numbers jump by a factor of about 88. Would the model notice? It would not. It simply receives a number. Changes in ICD coding, EHR fields, imaging protocols and laboratory interfaces can have similar effects.
Missing data deserve a word of their own. An absent blood test may mean that it was not clinically indicated, that it was unavailable, or that the testing protocol changed. If the model has learned patterns tied to which tests get ordered, a change in clinical behaviour can move its predictions even when the patients themselves are much the same.
When access to treatment changes: selection shift
Clinical populations are not random samples. Referral pathways, admission criteria, local expertise and the availability of treatments all shape them.
The arrival of a transcatheter alternative to conventional surgery is a good example, because it changes which patients end up having an operation. A surgical mortality model can therefore face a new risk distribution even though the hospital itself has not changed at all. This is the form of shift I think about most: TAVI has redrawn who reaches the operating table for aortic valve replacement, and it has done so faster than most risk scores are updated. What does a surgical score learn from a population that, in part, no longer comes to surgery?
Figure 1. Three illustrative mechanisms of dataset shift: a change in case mix, a change in outcome prevalence, and a change in the relationship between predicted risk and observed outcomes. All curves and values are hypothetical; no patient-level data were used.
The practical message is that different kinds of shift call for different investigations, and retraining is not the automatic answer to all of them.
What has actually happened in clinical studies?
Dataset shift is not just a theoretical worry. Published evaluations document both genuine change over time and failures to generalise from one healthcare setting to another. The two are different phenomena, and I will try to keep them apart.
EuroSCORE II: good discrimination does not guarantee accurate risk
Van Dijk and colleagues followed the performance of EuroSCORE II in 103,404 cardiothoracic procedures recorded in the Netherlands Heart Registration between 2013 and 2019 [2]. Across three-month periods, observed mortality ranged from 1.9% to 3.6%, while mean predicted mortality ranged from 3.4% to 4.2%. Observed-to-expected ratios ran from 0.50 to 0.95. Discrimination, meanwhile, stayed good, with c-statistics between 0.82 and 0.89.
Over the same years the case mix moved: the share of elective procedures fell from about 75% to 54%. What the study shows, then, is persistent overprediction against a background of changing case mix. It does not show a progressive loss of discrimination.
For those of us who use the score at the bedside, the consequence is concrete. A figure used for informed consent, treatment selection or comparative audit has to be credible as an absolute probability; ranking patients correctly is not enough. How often, in those years, did a consent discussion rest on a number that was too pessimistic?
Chest radiographs: what did the network really learn?
Zech and colleagues tested pneumonia detection on chest radiographs from three hospital systems [3]. A model trained on pooled data from two of them reached an internal AUROC of 0.931; on the third, external system it fell to 0.815. A separate network identified the hospital system an image came from with near-perfect accuracy. And on the pooled internal test set, a trivial predictor that ignored the images altogether and used only each system’s pneumonia prevalence reached an AUROC of 0.861.
So how much of the 0.931 was pneumonia, and how much was postcode? The study cannot split the two precisely. It does show how institutional characteristics and disease prevalence can hand a network a shortcut: information about where an image was taken rather than features that carry over to a new hospital. This is a problem of transportability and external validation, which is not the same as deterioration over time.
Intensive care: temporal deterioration can have clinical consequences
Using MIMIC-IV, Guo and colleagues trained models for mortality, prolonged stay, sepsis and invasive ventilation and tested them across calendar periods [4]. Sepsis prediction deteriorated most, with a maximum AUROC drop of 0.090. The authors translated this into a scenario of 100 consecutive ICU admissions. A model trained on 2008–2010 and applied to 2017–2019 meant roughly one additional missed case among 11 patients with sepsis, plus seven additional false positives, compared with its performance in the original period.
An AUROC change of 0.09 sounds abstract. One septic patient missed at the moment when treatment could have started does not.
The Epic Sepsis Model: the difference between deployment and validation
Wong and colleagues independently evaluated the Epic Sepsis Model in 38,455 hospitalisations [5]. Its hospitalisation-level AUROC was 0.63. At the alert threshold studied, it missed 67% of sepsis cases while firing in roughly 18% of all hospitalisations.
This was mainly an external validation finding rather than evidence of drift over time. It still makes two points worth holding on to: a widely implemented model cannot be assumed to work in every clinical environment, and alert burden belongs in any assessment of decision support. Would any of us accept a laboratory test that flagged almost one patient in five and still missed two thirds of the cases?
Stable AUROC does not settle the question
Cabanillas Silva and colleagues examined delirium, sepsis and acute kidney injury models in two German hospitals, with data spanning 2009 to 2021 [6]. Year-to-year AUROC differences were not statistically significant, yet calibration shifted for some of the sepsis and delirium models. The authors suggest that changes in discharge coding may be part of the explanation; the AKI labels, defined by laboratory criteria, stayed stable. Decision curve analysis found no significant change in net benefit at the thresholds of interest.
Results like this are easy to overlook, and they deserve the same weight as the alarming ones. A statistical shift is not the same thing as demonstrated clinical harm. Monitoring has to separate a change that can be detected from a change that alters decisions.
Can shift be detected before outcomes are known?
Subasri and colleagues analysed 143,049 hospital encounters in seven hospitals in the Greater Toronto Area, with models predicting in-hospital mortality [7]. They found shifts linked to patient demographics, hospital type, admission source and laboratory assays (troponin, BNP and D-dimer among them), and built a label-agnostic pipeline to flag such changes before outcomes were available. In their experiments, transfer learning and drift-triggered continual learning improved performance under several conditions, including the period spanning the COVID-19 pandemic. One result I did not expect: models trained on community hospitals were harmed when transferred to academic centres, but not the other way round.
The approach is promising. Still, an alarm raised on unlabelled data cannot by itself establish that a model has become unsafe, and retrospective gains do not prove better patient outcomes after real-world implementation.
Why the clinical consequences matter more than the statistical label
The practical question is not only has the distribution changed? It is which decision does the model influence, and what happens to the patient if the prediction is wrong?
A risk estimate used for counselling
Suppose a patient is told that 30-day mortality after an operation is 8%, when the contemporary risk for comparable patients is closer to 4%. The model may rank patients perfectly well, and the overestimate can still change how the patient sees the operation, the course of the consent discussion, or the comparison between surgical and transcatheter options. Anyone who has had that conversation knows how much weight a single number carries.
An alert used to trigger action
A sepsis alert that loses sensitivity may miss patients in whom early assessment would have mattered. One that stays sensitive but grows steadily less specific breeds alert fatigue. The measures that count here are threshold-specific sensitivity, the false alert rate, time to action and the consequences of the interventions that follow. AUROC alone says little about any of them.
A model used to allocate scarce resources
A model predicting the need for intensive care may influence staffing, bed allocation and the level of postoperative monitoring. Changes in admission policy, step-down care or discharge criteria can affect both the inputs and the observed outcome. Is the patient sicker, or has the pathway changed? From the model’s output alone there is often no way to tell.
Errors hidden inside apparently acceptable averages
Good overall performance can hide poor calibration or low sensitivity in the subgroups that matter: older patients, patients with severe renal dysfunction, uncommon procedures, patients arriving through a new referral pathway. Subgroup analyses should be prespecified and read together with their uncertainty intervals. With few events, an apparently dramatic change can be noise.
How much drift is acceptable is therefore a clinical decision informed by statistics, not a statistical threshold chosen in isolation.
When the model changes the outcome it predicts
Clinical AI has a particularly awkward complication: the prediction can change how the patient is managed.
Imagine a model predicting postoperative acute kidney injury (AKI). Clinicians respond to high-risk predictions by reviewing haemodynamics, avoiding potentially nephrotoxic drugs where appropriate and intensifying surveillance. If those measures work, fewer patients develop AKI than otherwise would.
The observed AKI rate among high-risk patients falls. A naive calibration analysis then concludes that the model overpredicts. Part of that gap, however, may be the effect of the care the model itself triggered.
Prediction → clinical response → altered outcome → altered apparent performance.
This is sometimes called intervention-induced shift or performative prediction. It is a causal problem as much as a statistical one: risk under model-guided care is not necessarily the risk that would have been observed without that care. Should a model be recalibrated downwards because it has succeeded? Probably not, and that is exactly where a purely statistical reading can mislead.
For systems that directly influence management, postdeployment monitoring should record whether alerts were seen, which interventions followed and when. If the question is whether the model actually improves outcomes, properly designed prospective or quasi-experimental studies are needed, and sometimes randomisation.
A familiar cardiac surgical example: almost the same AUROC, different risk estimates
Consider a hypothetical cardiac surgical mortality model predicting 30-day death from preoperative information, unchanged between the initial and the recent evaluation:
| Metric | Initial evaluation | Recent evaluation |
|---|---|---|
| AUROC | 0.84 | 0.83 |
| Mean predicted 30-day mortality | 3.2% | 4.6% |
| Observed 30-day mortality | 3.1% | 3.0% |
| Observed / expected events | 0.97 | 0.65 |
All values are illustrative, not published results or actual hospital data.
The model still ranks patients in much the same way. It now predicts about 46 deaths per 1,000 patients where 30 are observed: average predicted mortality has risen while observed mortality has barely moved. That gap should prompt an investigation. It should not prompt immediate retraining.
What would I want to know first? Have procedural indications changed? Have high-risk patients been redirected to transcatheter treatment? Has mortality genuinely improved? Has the registry changed its definitions, or the completeness of follow-up? Could deploying the model have altered management?
Figure 2. Illustrative monitoring example. The AUROC changes little, while the gap between mean predicted and observed mortality widens. The figure reproduces only the hypothetical table above and does not imply a statistically significant temporal trend.
If discrimination remains acceptable and the main problem is inaccurate absolute probabilities, recalibration may be enough. If important predictor–outcome relationships have changed, more extensive updating may be needed. Neither decision should be taken before the data and the clinical context have been checked.
What should a hospital actually monitor?
A safe implementation needs a monitoring plan before the system goes live. A practical framework has four connected layers.
| Layer | What to monitor | Clinical questions |
|---|---|---|
| 1. Inputs and data quality | Case mix, units, assay changes, missingness, out-of-range values, coding and extraction versions | Are these still the data the model was designed to receive? |
| 2. Model outputs | Score distributions, average predicted risk, alert frequency, invalid predictions, patients above action thresholds | Have prediction patterns changed, and why? |
| 3. Outcomes and performance | Event prevalence, AUROC/AUPRC, calibration, O:E ratio, sensitivity/specificity at operational thresholds, subgroup results | Does the model still predict reliably for the current patients? |
| 4. Workflow and impact | Alert acknowledgement, time to intervention, override rates, unnecessary investigations, missed cases, adverse events | Does using the model still help, or is it disrupting care? |
The first two layers can often be monitored before outcomes are known. Distribution plots, the Kolmogorov–Smirnov test, the population stability index, Jensen–Shannon divergence and multivariate distance measures are all available. Two caveats, though. In a large dataset a small P value proves nothing about meaningful harm, and an infrequent but serious change may slip through altogether.
Once outcomes become available, performance has to be reassessed. AUPRC is useful for rare outcomes but depends heavily on event prevalence. Calibration needs particular care, because calibration-in-the-large, the calibration slope, the observed-to-expected ratio and graphical calibration curves each answer a different question [8]. Summary measures such as the integrated calibration index (ICI) and its companions E50 and E90, which describe the average, median and 90th-percentile absolute gap between predicted and observed risk, are convenient for tracking calibration over time [9]. The Brier score summarises probabilistic prediction error, but it is not a pure calibration metric [8].
Decision curve analysis helps to judge the consequences of model-based action at clinically defensible probability thresholds [10]. For a system that triggers a low-risk confirmatory test, false positives may be tolerable; for one that steers patients towards invasive treatment, the trade-off is quite different.
Two further clinical constraints apply:
- Outcomes may be delayed or incomplete. A 30-day mortality model cannot be assessed fairly on cases with 10 days of follow-up. Linkage to death registries, transfers, readmissions and changes in coding or surveillance all deserve explicit checks. Sample size should be driven by the precision required of the performance estimates, especially when events are uncommon [11].
- The prediction target must stay stable. A sepsis algorithm may appear to drift simply because case definitions, clinical coding or testing practices have changed. A true change in patient outcomes has to be told apart from a change in how the outcome is ascertained.
Monitoring should also specify who reviews the alerts and what happens next. Who looks at the plot, and what are they allowed to do about it? A dashboard without an escalation pathway is just a collection of plots.
What to do when drift is detected
A reasonable response follows a sequence, and the order matters.
1. Check the data pipeline before touching the model. Verify units, assay versions, coding systems, feature mapping, missing data, software releases and outcome ascertainment. If a laboratory field has been transformed incorrectly, retraining on the resulting data can only make things worse.
2. Estimate the clinical consequences. Has sensitivity fallen at the threshold actually used for decisions? Are false alerts increasing? Is risk underestimated in a vulnerable group? Is the model causing delays or unnecessary interventions? Depending on the answers, a signal may justify investigation, closer surveillance, a temporary restriction or, for a serious safety concern, suspension of use.
3. Choose the smallest correction that does the job. A stable model with systematic overprediction may only need its calibration intercept updated; where necessary, intercept and slope can both be revised. In simplified form: logit(p_updated) = α + β × logit(p_original), where α shifts the overall level of risk and β controls how widely the original probabilities are spread.
If the underlying relationships have changed more substantially, the coefficients or the whole model may need revision. More complex retraining is not automatically better: an inappropriate update can bring overfitting, new subgroup errors and instability.
4. Validate the modified model before it returns to clinical use. Use independent data, ideally including a later temporal cohort, and check discrimination, calibration, clinically relevant thresholds, fairness across relevant groups and operational consequences. A silent prospective phase, with the model running but its outputs hidden from clinicians, can be useful before activation.
Figure 3. A proposed workflow for investigating possible performance drift in a clinical decision-support model. This is an illustrative governance framework, not a formally validated clinical protocol.
The organisation should keep a record of model versions, training and validation periods, feature definitions, decision thresholds, identified shifts, investigations, approved updates and the people responsible. Clinical staff need a simple way to flag predictions that no longer look plausible, and they need to hear back when their concerns are investigated [1]. Without that feedback, people stop flagging. Why would anyone keep reporting into a void?
The methodological literature increasingly thinks in terms of the whole life cycle. TRIPOD+AI sets reporting standards for prediction-model studies [12]; PROBAST+AI addresses study quality, risk of bias and applicability [13]; DECIDE-AI covers early-stage clinical evaluation, including human factors [14]. They complement one another. None of them replaces ongoing operational oversight.
In the United States, the FDA has issued final guidance on predetermined change control plans for AI-enabled device software functions, describing how specified modifications can be planned and authorised in advance [15]. It was first published in December 2024; the current version is dated August 2025. In Europe, the EU AI Act and the medical device regulations may impose further obligations depending on a system’s intended purpose, classification and the provisions that apply; implementation timelines and scope should be checked against current official guidance [16]. Neither framework makes arbitrary automatic updating clinically acceptable.
Can we design models that are more robust from the start?
Training on several hospitals and calendar periods may expose a model to more realistic variation. Domain-generalisation methods try to learn relationships that transfer across settings; domain adaptation brings in data from the target environment; continual learning updates the model as new information arrives. Approaches informed by causal reasoning look for relationships that depend less on local shortcuts.
None of them is a guarantee. In the MIMIC-IV evaluation by Guo and colleagues, the domain generalisation and adaptation methods produced no more robust models than a simpler model trained on a broader historical period [4]. A 2025 systematic review of 32 healthcare studies likewise found substantial heterogeneity and no detection or correction method that generalised broadly across use cases [17].
Should that surprise us? Perhaps not. A more advanced algorithm is not necessarily a more dependable clinical tool. Better data, careful validation, effective monitoring and clinical accountability may matter more than one more layer of model complexity.
Does the same problem apply to clinical language models?
Yes, although the monitoring tools differ. A language model used to summarise discharge letters, extract diagnoses or answer clinical questions may meet changes in documentation style, terminology, treatment guidelines or the content of its linked knowledge sources. A new EHR template can change the input text while the patients stay exactly the same.
There is a further complication, and it is one reason I prefer to run language models locally whenever I can. A system accessed through an external service can be updated by its provider, sometimes with little notice. A change in clinical output may then reflect a new model version, a modified prompt or retrieval pipeline, or a genuine change in the clinical environment. Those causes have to be disentangled.
For free-text outputs, AUROC and probability calibration may not be the right measures. Monitoring should rely instead on predefined, task-specific assessments: clinically consequential omissions, factual errors, inappropriate recommendations and clinician review of representative cases. The TRIPOD-LLM reporting guideline gives further methodological context for evaluations involving large language models [18]. The problem is related, but it should not be conflated with temporal drift in binary risk prediction.
Conclusion: clinical validity is not permanent
An AI model can stay technically identical while becoming less useful, or less safe, because the patients, the decisions, the measurements or the care pathways around it have changed.
Dataset shift is not automatically a failure, and a statistically detectable difference is not automatically harm. The task is to work out whether the difference compromises a decision that matters to a patient.
No clinician would assume that a laboratory analyser stays accurate indefinitely without quality control. Why should a prediction model be exempt? It deserves the same continued scrutiny, adapted to the particular complexities of clinical practice.
So the question is not only whether an algorithm was valid when it was developed. It is whether it is still valid for the patients we are treating today.
References
- Finlayson SG, Subbaswamy A, Singh K, et al. The clinician and dataset shift in artificial intelligence. N Engl J Med. 2021;385(3):283-286. https://doi.org/10.1056/NEJMc2104626
- van Dijk WB, Leeuwenberg AM, Grobbee DE, et al. Dynamics in cardiac surgery: trends in population characteristics and the performance of the EuroSCORE II over time. Eur J Cardiothorac Surg. 2023;64(3):ezad301. https://doi.org/10.1093/ejcts/ezad301
- Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 2018;15(11):e1002683. https://doi.org/10.1371/journal.pmed.1002683
- Guo LL, Pfohl SR, Fries J, et al. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Sci Rep. 2022;12:2726. https://doi.org/10.1038/s41598-022-06484-1
- Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181(8):1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626
- Cabanillas Silva P, Sun H, Rezk M, et al. Longitudinal model shifts of machine learning-based clinical risk prediction models: evaluation study of multiple use cases across different hospitals. J Med Internet Res. 2024;26:e51409. https://doi.org/10.2196/51409
- Subasri V, Krishnan A, Kore A, et al. Detecting and remediating harmful data shifts for the responsible deployment of clinical AI models. JAMA Netw Open. 2025;8(6):e2513685. https://doi.org/10.1001/jamanetworkopen.2025.13685
- Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17:230. https://doi.org/10.1186/s12916-019-1466-7
- Austin PC, Steyerberg EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Stat Med. 2019;38(21):4051-4065. https://doi.org/10.1002/sim.8281
- Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565-574. https://doi.org/10.1177/0272989X06295361
- Riley RD, Debray TPA, Collins GS, et al. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med. 2021;40(19):4230-4251. https://doi.org/10.1002/sim.9025
- Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://doi.org/10.1136/bmj-2023-078378
- Moons KGM, Damen JAA, Kaul T, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. https://doi.org/10.1136/bmj-2024-082505
- Vasey B, Nagendran M, Campbell B, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med. 2022;28(5):924-933. https://doi.org/10.1038/s41591-022-01772-9
- US Food and Drug Administration. Marketing submission recommendations for a predetermined change control plan for artificial intelligence-enabled device software functions. Final guidance; first issued December 2024, current version dated August 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence
- European Commission. Navigating the AI Act: frequently asked questions. Accessed October 2026. https://digital-strategy.ec.europa.eu/en/faqs/navigating-ai-act
- Silva GFDS, Barcellos Filho FN, Wichmann RM, et al. Strategies for detecting and mitigating dataset shift in machine learning for health predictions: a systematic review. J Biomed Inform. 2025;170:104902. https://doi.org/10.1016/j.jbi.2025.104902
- Gallifant J, Afshar M, Ameen S, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60-69. https://doi.org/10.1038/s41591-024-03425-5