Model Lifecycle & Drift Monitoring
- Model and dataset pairs that showed temporal quality degradation, across 32 datasets from four industries (2022 study)
- 91%
- Model and dataset pairs that showed temporal quality degradation, across 32 datasets from four industries (2022 study)
- Scientific Reports (Nature Portfolio), 2022(opens in a new tab)
- GPT-4 accuracy on the same prime-number task, March versus June 2023: same model name, different behavior
- 84% to 51%
- GPT-4 accuracy on the same prime-number task, March versus June 2023: same model name, different behavior
- Harvard Data Science Review, 2024(opens in a new tab)
- US hospitals using predictive AI that did not know whether their models had post-implementation monitoring (2024)
- 18%
- US hospitals using predictive AI that did not know whether their models had post-implementation monitoring (2024)
- Assistant Secretary for Technology Policy, US Department of Health and Human Services, 2025(opens in a new tab)
The problem
Degradation is the norm, not the exception. Researchers who tested four standard model types on 32 datasets from four industries found temporal quality degradation in 91% of the model and dataset pairs1. Generative models move too, sometimes because the vendor changed them: GPT-4's accuracy at telling prime from composite numbers fell from 84% in March 2023 to 51% in June 2023, which the authors say shows the need for continuous monitoring of LLMs2.
Monitoring is still patchy where the stakes are high. In 2024, 18% of US hospitals using predictive AI did not know whether they conducted post-implementation evaluation or monitoring of those models, and 21% did not know whether models were evaluated for bias3. The same federal brief notes that while most hospitals evaluated models, fewer did so for all or most of them3. Unit tests at launch do not cover this: in a changing world, live monitoring with automated response is what keeps a model reliable4.
Supervisors have updated their expectations. In April 2026, US banking agencies replaced SR 11-7 with SR 26-2, which calls for ongoing monitoring of whether a model performs as expected given changes in products, clients, data relevance or market conditions, with overlays, adjustment or redevelopment when it does not5. SR 26-2 puts generative and agentic AI outside its scope, while still expecting governance for them5. In the EU, providers of high-risk systems must run post-market monitoring6.
What you get
- 01
Model inventory and tiering
An inventory of every model in use, with owner, purpose, materiality tier, data sources, known limitations and validation status, kept current as models are added, changed or retired.
- 02
Monitoring plan per model
The metrics each model is judged on (accuracy against outcomes, calibration, input and output drift, performance by customer or patient group), where ground truth comes from, and thresholds with a pre-agreed action for each.
- 03
Drift and fairness monitoring in production
Automated checks, dashboards and alerts built into your existing data and ML platform, with proxy measures for the period before real outcomes arrive.
- 04
Retraining and sign-off process
Retraining triggers, challenger testing, regression and bias tests, documented approval by someone independent of the developers, and a rollback path if a new version underperforms.
- 05
Audit evidence pack
Monitoring reports, threshold breaches, decisions and sign-offs kept in a form internal audit, model risk and examiners can follow from alert to action.
Where it goes wrong
The failure modes we design against from the first week, and the control for each.
Watching inputs but not outcomes
Input drift alerts are noisy and do not tell you whether decisions got worse. SR 26-2 points to outcomes analysis, such as back-testing against real-world results, as part of ongoing monitoring5. Measure performance against outcomes, and use proxies only while labels lag.
Thresholds with no action attached
Alerts fire, a dashboard turns red, and nobody is sure who decides. Set thresholds at validation, attach a specific response to each (investigate, overlay, restrict use, retrain, retire), and name the person who makes the call.
Retraining without revalidation
Retraining can fix one problem and introduce another, because in ML systems changing anything changes everything4. Run the same regression and bias tests as the original validation, and require independent sign-off before a retrained model replaces the current one.
Assuming a vendor model stays the same
Hosted models are updated without much notice, and behavior can shift sharply within months2. Pin versions where you can, keep a fixed evaluation set, and rerun it on every vendor change and on a schedule.
Standards and rules
SR 26-2, Revised Guidance on Model Risk Management (Federal Reserve, OCC, FDIC)
Primary text on federalreserve.gov(opens in a new tab)Replaced SR 11-7 in April 2026: expects ongoing monitoring, outcomes analysis, effective challenge and a model inventory scaled to materiality, and excludes generative and agentic AI from its scope.
NIST AI RMF, MEASURE 2.4 and MANAGE 4.1
Primary text on nvlpubs.nist.gov(opens in a new tab)The behavior of an AI system must be monitored in production, and post-deployment monitoring plans must cover override, decommissioning, incident response and change management.
EU AI Act, post-market monitoring
Primary text on digital-strategy.ec.europa.eu(opens in a new tab)Providers of high-risk systems must run a post-market monitoring system and report serious incidents, from 2 December 2027 for Annex III uses and 2 August 2028 for AI in regulated products.
How to measure it
- Share of models in the inventory with an owner, a tier, thresholds and a current monitoring report
- Time from threshold breach to a documented decision
- Performance against real outcomes per model and per group, compared with the validation baseline
- Number of models running past their scheduled revalidation date
- Share of model changes promoted with documented independent sign-off
Questions buyers ask
What is model drift and how do you detect it?
Drift is a change between the conditions a model was built for and the ones it now sees. Inputs can shift, or the relationship between inputs and outcomes can change. You detect it by comparing input and output distributions with the training baseline, and above all by measuring performance against real outcomes once they arrive. The second check tells you whether drift actually hurt decisions.
How often should a model be retrained?
There is no single interval. Set a scheduled review based on how material the model is and how fast its environment changes, and add triggers: a threshold breach, a product or policy change, or a new data source. Retrain when evidence says performance has slipped, then revalidate before release. Retraining on a calendar alone can push untested changes into production.
Does SR 26-2 cover generative AI and large language models?
No. SR 26-2 states that generative and agentic AI models are outside its scope and applies to traditional statistical models and non-generative AI. It still expects a bank's risk management and governance to set controls for tools it does not cover. In practice, most banks apply the same discipline to LLM systems: inventory, testing against a fixed evaluation set, ongoing monitoring and documented sign-off.
Sources
Every figure on this page comes from the sources below. We checked each link on .
- 1.Temporal quality degradation in AI models(opens in a new tab)Scientific Reports (Nature Portfolio), July 8, 2022 · nature.com
- 2.How Is ChatGPT's Behavior Changing Over Time?(opens in a new tab)Harvard Data Science Review, March 12, 2024 · hdsr.mitpress.mit.edu
- 3.Hospital Trends in the Use, Evaluation, and Governance of Predictive AI, 2023-2024 (ASTP Data Brief No. 80)(opens in a new tab)Assistant Secretary for Technology Policy, US Department of Health and Human Services, September 2025 · healthit.gov
- 4.Hidden Technical Debt in Machine Learning Systems(opens in a new tab)Advances in Neural Information Processing Systems 28 (NeurIPS 2015), 2015 · proceedings.neurips.cc
- 5.SR 26-2: Revised Guidance on Model Risk Management(opens in a new tab)Board of Governors of the Federal Reserve System, April 17, 2026 · federalreserve.gov
- 6.AI Act(opens in a new tab)European Commission, October 19, 2026 · digital-strategy.ec.europa.eu
- 7.Artificial Intelligence Risk Management Framework (AI RMF 1.0)(opens in a new tab)National Institute of Standards and Technology, January 2023 · nvlpubs.nist.gov