Evaluation-gated model promotion: why your candidate model has to earn traffic
The model shipped eighteen months ago. Nobody has retrained it since, nobody is measuring drift, and the engineer who built it has moved on. This isn’t an edge case — it’s the default state of most production ML, once nobody is explicitly accountable for it.
The fix isn’t “retrain more often.” A model retrained on a schedule and promoted automatically is just a faster way to ship a worse model, if nothing checks it first. The fix is a gate: a candidate model has to beat the champion on recent data before it serves a single request. No gate, no promotion.
What the gate actually checks
A useful evaluation gate isn’t a single accuracy number. It’s a small battery of checks run against a held-out, time-based split of genuinely recent data — never a random split, because random splits leak future information into the past in any system with a time dimension, and quietly flatter every model that gets evaluated that way. The checks that matter:
- Does the candidate beat the champion on the metric that matters for the business — not just the metric that’s easiest to compute.
- Does it beat the champion on the segments that matter, not just in aggregate. A model can improve overall accuracy while getting measurably worse for your highest-value segment, and an aggregate-only gate will wave it through.
- Is calibration stable — a model that’s directionally right but confidently wrong is often worse than one that’s honestly uncertain.
Canary rollout, not a coin flip
A model that passes the gate doesn’t get 100% of traffic on day one. It earns it — canary rollout to a small percentage, watched against the same metrics the gate checked, with automatic rollback if production behaviour doesn’t match the evaluation. The gate tells you the model should be better. The canary tells you it actually is, on live data the offline evaluation couldn’t see.
Why this has to be built before retraining, not after
Teams build the retraining pipeline first because it’s the more interesting engineering problem, and bolt on evaluation gates later once something has already gone wrong in production. We build it the other way round. A retraining loop without a gate is a liability that runs on a schedule — it doesn’t fail loudly, it fails quietly, one slightly-worse model at a time, until someone notices the metric that was supposed to be improving has been declining for two quarters.
The question that tells you where you stand
If you want to know how exposed a production ML system is, there’s one question that cuts through the rest: when was the model last retrained, and what stopped the last candidate that didn’t beat the champion? If the honest answer to the second half is “nothing — we don’t have one,” the gate is the next thing to build, before the next retraining run, not after.