A version number tells you which training run produced a checkpoint. It tells you nothing about whether the model is safe to ship. That gap is where perception teams lose the thread on versioning — and where long-tail regressions slip into production unnoticed.
Walk into most automotive perception teams and versioning looks like a filename convention. detector_v3.pt, then detector_v4.pt, then a -final suffix nobody trusts. The version is tagged to a training run — new data went in, weights came out, increment the counter. Then someone asks the only question that matters at release time: is v4 better or worse than v3, and in which conditions? A bare version number cannot answer that. And so the team re-runs the headline mAP, sees it ticked up two points, and ships. Three weeks later a heavy-rain, low-sun corner case that v3 handled fine starts dropping pedestrians. The version number was never the problem. The problem was what the version number failed to carry.
What machine learning model versioning actually binds
Tags and timestamps are not versioning; linking artifact, lineage, and evidence is. It is the mechanism that makes robustness comparable across releases. A useful version binds a set of weights to the scenario-class evidence it was validated against — the weather, lighting, edge classes, and sensor-mounting variance it actually held accuracy across. Strip that binding away and you have a checkpoint with a name, not a version you can defend.
The distinction is easiest to see through a contrast. Software versioning inherited from Git conditioned a whole generation of engineers to treat a version as a pointer to a code state, reproducible by construction: the same commit compiles to the same binary. Machine learning breaks that assumption in two places. First, the artifact is a function of data, hyperparameters, random seeds, and the training stack — reproduce all of those and you might reproduce the weights. Second, and more importantly, the thing you actually care about is not the weights but the behaviour, and behaviour is only observable through validation. So a perception model version has to point at two things at once: the inputs that produced it (data snapshot, config, code, seed) and the evidence that characterises it (per-scenario-class accuracy against a defined validation distribution).
Tools like DVC, MLflow, and Weights & Biases have made the first half routine. You can pin a data hash, log the training config, and reproduce a run. That is necessary and it is not sufficient. The reproducibility layer answers “how was this built.” It does not answer “is this safe to ship relative to the last thing we shipped,” and in our experience with production perception teams, that second question is where releases actually get stuck.
Why a version number on a checkpoint is not enough to defend a release
Ask yourself: what evidence would you demand before signing off? They are not being asked whether v4 exists, or whether it trained cleanly. They are being asked to sign off that v4 is at least as safe as the last defensible release across the conditions the vehicle will encounter. A single headline metric collapses that question into a scalar and hides exactly the information a reviewer needs.
Aggregate mAP is an average over a validation set. If that set is dominated by clear-daylight, well-lit highway frames — as most are, because those are the cheap frames to collect — then a two-point mAP gain can coexist with a five-point drop on night-time occluded pedestrians. The average moved the right way. The safety case moved the wrong way. This is the mechanism behind most post-release perception surprises we see: the regression was always present in the evidence, but the versioning discipline threw the evidence away and kept only the average. We unpack the trap of trusting a single aggregate figure in what each machine learning model performance metric actually proves, which is the companion to this piece on the metrics side.
The correct framing: a version’s defensibility lives in its scenario-class decomposition, not its headline. The number that matters is the accuracy delta per scenario class between the incoming version and the last shipped release. Positive across every class you care about, you have a clean improvement. Negative on even one safety-relevant class, you have a regression you must either justify or block — regardless of what the aggregate did.
How do you tie a model version to its scenario-class robustness evidence?
Fold validation records directly into the version artifact, not into a separate wiki. A defensible version record contains the training inputs, the validation distribution it was scored against, the per-scenario-class metrics, and the deltas against the prior release. Anything less and the next reviewer is re-deriving the safety case from scratch.
A version record a release reviewer can accept
| Record field | What it must contain | Why the reviewer needs it |
|---|---|---|
| Build provenance | Data snapshot hash, training config, code commit, seed | Answers “how was this built” and makes the run reproducible |
| Validation distribution | The scenario-class taxonomy and frame counts per class it was scored on | Prevents an average from hiding a sparse but safety-critical class |
| Per-class metrics | Accuracy / recall per scenario class, not just aggregate | The behaviour characterisation the release actually rests on |
| Baseline reference | The version ID of the last defensible release | Deltas are meaningless without a fixed comparison point |
| Per-class delta | Signed change per scenario class vs. baseline | The single view that tells a reviewer improve-or-regress |
| Accepted-evidence pointer | Link to the evidence pack the prior reviewer accepted | Lets each release build on the last instead of re-litigating it |
The scenario-class taxonomy in row two is the load-bearing part, and it is upstream of versioning. If you have not defined the classes — the specific weather bands, lighting conditions, edge object types, and sensor-mounting variance you require accuracy across — then there is nothing for the version to bind to. Getting that specification right is its own discipline; we cover it in writing a robustness specification for an automotive perception model. Versioning consumes that specification. It does not replace it.
How do you compare two versions to detect long-tail regressions?
Side-by-side version testing is the justification for the entire framework. Once each version carries per-scenario-class evidence scored against the same validation distribution, comparing two releases is a subtraction, not an investigation. You lay the incoming version’s per-class metrics next to the baseline’s, compute the signed delta per class, and read off exactly where the model improved and where it degraded.
Long-tail regressions are precisely the ones a headline hides, because the long tail is, by definition, sparse in any aggregate. A scenario class that is 0.4% of your validation frames (illustrative) — say, low-sun glare with a partially occluded cyclist — contributes almost nothing to mean accuracy but everything to a specific fatality mode. Scored per class, a drop there is a red flag the reviewer sees immediately. Averaged away, it is invisible until the field. The comparison surface below is the operational form of this.
Per-scenario-class release gate (worked example)
Assume a detector where the last defensible release is v3 and the incoming candidate is v4. The numbers here are illustrative, to show the shape of the comparison, not measured results:
| Scenario class | v3 recall | v4 recall | Delta | Verdict |
|---|---|---|---|---|
| Clear day, highway | 0.94 | 0.96 | +0.02 | Improved |
| Night, urban, lit | 0.88 | 0.90 | +0.02 | Improved |
| Heavy rain, dusk | 0.81 | 0.83 | +0.02 | Improved |
| Low-sun glare, occluded VRU | 0.79 | 0.71 | −0.08 | Regression — block or justify |
| Aggregate mAP | 0.87 | 0.89 | +0.02 | Misleading |
The aggregate says ship. The per-class view says the opposite: v4 traded a broad, cheap gain for a narrow, expensive loss on the exact class where a miss is a vulnerable-road-user collision. This is an observed-pattern we see repeatedly across perception engagements — not a benchmarked rate, but a recurring shape. The versioning discipline is what turns that shape from a post-mortem finding into a pre-release gate.
Versioning against the production driving distribution and sensor-mounting variance
Validation data must represent the deployment environment, or version claims collapse. Two things make this harder than it sounds. The production driving distribution drifts — the vehicle encounters conditions the validation set under-represents, and a version validated against last year’s distribution is defending against last year’s world. Monitoring the live distribution and feeding drift back into the scenario-class taxonomy is what keeps versioning honest; we treat the monitoring side in depth in machine learning monitoring for provenance-preserving compliance.
Sensor-mounting variance is the subtler one. The same model behaves differently across a fleet because the cameras are not mounted identically — small differences in extrinsics change the effective input distribution per vehicle. A version validated on one mounting geometry is not automatically defensible on another. Practically, this means the version record’s validation distribution has to span the mounting variance the fleet actually exhibits, and a version that only ever saw one rig’s geometry carries a quiet caveat the reviewer should know about. The failure modes this produces — the ones that survive clean benchmarks and only appear in the field — are worth understanding directly, and we catalogue them in the perception failure modes that survive benchmarks.
When does a new version justify a release?
Ship only when improvements are proven per scenario class, or when regressions carry written justification. “The mAP went up” is not a justification. “Per-class deltas are non-negative across all safety-relevant classes, and the one flat class was already at ceiling” is. The whole point of binding evidence to the version is that the release decision becomes reviewable by someone who was not in the room for training.
This is also the connection to the broader safety-evidence picture. Versioning does not exist to make model management tidy; it exists so that each release stands on the accepted evidence of the last one, and the audit trail compounds instead of resetting. How perception models meet automotive safety demands, from ASIL evidence down to the validation pack, is the wider frame this sits inside — we develop it in how perception models meet safety evidence, and versioning is the release-comparison layer that makes that evidence portable across releases.
If you are building the monitoring and evidence layer that makes this comparison possible, our work on the Production AI Monitoring Harness and our broader computer vision engineering practice is where the version-bound evidence pack becomes an operational artifact rather than a spreadsheet.
FAQ
What’s worth understanding about machine learning model versioning first?
Every model version must cryptographically link trained weights back to the exact data snapshot, hyperparameters, code commit, and validation metrics that define its provenance. It is the mechanism that makes robustness comparable across releases, not a filename on a checkpoint. Tools like DVC and MLflow handle the reproducibility half; the evidence-binding half is what makes a version defensible.
Why is a version number on a checkpoint not enough to defend a perception model release?
A bare version number tells a reviewer that a checkpoint exists and trained cleanly, but not whether it is at least as safe as the last defensible release across real conditions. A single headline metric collapses that question into a scalar and hides the per-class information a reviewer actually needs — a two-point aggregate gain can coexist with a five-point drop on a safety-critical class. The defensibility lives in the scenario-class decomposition, not the number.
How do you tie a model version to the scenario-class robustness evidence it was validated against?
You make the evidence part of the version record, not a report that lives next to it. The record contains build provenance, the validation distribution and its scenario-class taxonomy, per-class metrics, the baseline version ID, per-class deltas against that baseline, and a pointer to the evidence pack the prior reviewer accepted. The scenario-class taxonomy is upstream of versioning — if the classes are not defined, there is nothing for the version to bind to.
How do you compare two versions to detect long-tail regressions before they reach production?
Once both versions carry per-scenario-class metrics scored against the same validation distribution, comparison is a subtraction: lay the incoming metrics next to the baseline and read the signed delta per class. Long-tail regressions are exactly the ones a headline hides, because a sparse class contributes almost nothing to a mean but everything to a specific failure mode. Scored per class, a drop there is an immediate red flag rather than a field surprise.
What should a version record contain so a release reviewer can accept it?
Build provenance (data hash, config, code commit, seed), the validation distribution with its scenario-class taxonomy and per-class frame counts, per-class accuracy metrics, the baseline release’s version ID, the signed per-class delta against that baseline, and a pointer to the evidence the prior reviewer accepted. The last two turn each release into an increment on an accepted safety case rather than a fresh litigation of a headline score.
How does versioning relate to the production driving distribution and sensor-mounting variance across a fleet?
A version is only as strong as the validation distribution it was scored against, and that distribution has to track the fleet rather than the lab. The production driving distribution drifts over time, so a version validated against last year’s data is defending last year’s world. Sensor-mounting variance means the same model behaves differently across vehicles, so the version record’s validation distribution should span the mounting geometries the fleet actually exhibits.
When does a new version justify a release, and what versioned evidence demonstrates it improved rather than regressed?
A new version justifies a release when its per-class deltas are non-negative across every scenario class the safety case depends on — or when a specific regression is explicitly accepted with a documented rationale. “The mAP went up” is not a justification; “per-class deltas are non-negative across all safety-relevant classes” is. The versioned evidence that demonstrates it is the per-scenario-class delta table against the last defensible release, not the aggregate score.
Versioning is not the interesting engineering problem — it is the boring discipline that keeps the interesting problems from reaching the road. The question worth carrying forward is narrower than “which version is best”: for the one scenario class where a miss is a collision, can you show, release over release, that the number went the right way? If your versioning cannot answer that in one subtraction, it is still a filename convention.
Versioning strategy for safety-critical perception stacks
Perception systems require artifact immutability, pipeline reproducibility, and inference traceability to source. If Machine Learning Model Versioning is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
