LLM Model Versioning: Track, Register, Release, Roll Back

Sketches framing an LLM versioning title

LLM model versioning means giving every deployed combination of weights, prompts, retrieval data, tools and runtime a fixed identity, so you can reproduce it, compare it with its successor and return to it if something breaks. Versioning the weights file alone is not enough. The model name in your config is one of many inputs that decide how the system behaves. This guide covers what to record, how a registry fits in, and how to ship a new version in environments where data cannot leave your perimeter.

TL;DR

Version the whole system as one release manifest: model, adapters, prompts, retrieval index, tools, safety rules and container images.
Store weights in an access-controlled, signed, immutable registry, ideally as OCI artifacts, so existing container tooling does the heavy lifting.
Test every candidate against a baseline set run on the current production version before it sees users.
Release in stages (shadow, small traffic share, full) and decide the rollback path before launch.
In air-gapped networks, add signed release sets, controlled transfers and internal mirrors.

What LLM versioning covers that code versioning does not
What belongs in a release manifest
Naming releases: semantic versioning adapted
The LLM model registry: your source of truth
Versioning in private and air-gapped environments
Testing a new version against the one in production
Rolling out and rolling back
Governance: making versions auditable
Our perspective on versioning sovereign LLMs
Put versioning into a private deployment
FAQ
Sources

What LLM versioning covers that code versioning does not

Traditional software versioning tracks source code. Classic machine-learning versioning adds data and trained artifacts. LLM versioning adds more moving parts. Research on versioning LLM-agent software describes interdependent components, namely weights, prompts, data and orchestration, that change at different speeds and produce non-deterministic output. The same paper points to a problem that hits teams using hosted models: providers can update weights without changing the API version.

That last point is why many regulated organizations run models inside their own infrastructure. When you control the weights, a version number means something. A 2026 academic paper on governing updates in the LLM supply chain argues that provider-side updates can cause behavioral drift with no change at the endpoint, and that a provider’s own evaluation may not guarantee your application still works.

Four differences matter in practice:

Behavior is the product. A prompt edit can change outcomes as much as a new checkpoint. Prompts are release artifacts, not config comments.
Evaluation is open-ended. There is rarely a single accuracy number. You judge text quality, grounding and refusals against test sets you maintain.
Retrieval is part of the model. A re-indexed document library changes answers even when no weights move.
Cost shifts with each version. Token usage, latency and energy per request vary between versions, so they belong in the release record. Our guide to tracking joules per token on premises shows how to measure that.

If you want a hands-on feel for pinning a model on a single machine first, a walkthrough on running Llama locally offline is a reasonable starting point before you scale to a managed release process.

What belongs in a release manifest

The most useful habit in LLM versioning is to stop thinking about a “model version” and start thinking about a release. The research on LLM-agent versioning recommends binding cryptographic hashes and signatures for model checkpoints, adapter weights, prompts, retrieval indices, safety-rule documents, tool and API schemas, and container images in a single manifest. Other practitioners add decoding settings, parsers, validators and correction policies to that list, as in this guide on versioning a complete system configuration.

A practical manifest looks like this:

Component	What to pin	Why it matters
Base model	Checkpoint hash, source, license	Identifies exactly which weights are serving
Adapters	Hash per adapter, training data reference	Fine-tunes change behavior without changing the base
Prompts and templates	Content hash, version tag	Small edits shift tone, format and refusals
Decoding settings	Temperature, top-p, max tokens, stop sequences	Alters output even with identical weights
Retrieval	Index build ID, embedding model, chunking rules	Re-indexing changes what the model can cite
Tools	API schemas and permission grants	Tool contracts break silently when they drift
Safety rules	Policy documents, filter configs	Needed to prove what guardrails were active
Runtime	Container image digest, inference engine version	Engine upgrades can alter numerics and speed
Evaluation evidence	Test set version, results, sign-off	Links the release to the proof it was safe

For reproducing a run, general engineering guidance says to record the configuration, code version, software environment, random seeds and data and save dependency files with the run. Apply the same discipline to production, not only to experiments.

One interaction deserves its own line: caching. If a response cache keeps answers across versions or tenants, a new model can serve stale or wrongly scoped output. Include the release ID in cache keys, and read our notes on LLM caching strategies that prevent tenant data leaks.

Naming releases: semantic versioning adapted

Semantic versioning is a good starting grammar: Major for breaking changes, Minor for new features, Patch for bug fixes. The trick is deciding what “breaking” means for an LLM system. A workable convention:

Major: a new base model, a new embedding model (which forces re-indexing), or a changed tool contract that downstream systems depend on.
Minor: a new adapter, a new capability, a prompt change that alters output format or scope.
Patch: a prompt wording fix, a corrected retrieval document, a runtime security update that your evaluation shows leaves behavior unchanged.

Write the rule down and apply it consistently. The number matters less than the fact that everyone agrees what a Major bump triggers: a full evaluation pass, stakeholder sign-off and a slower rollout.

The LLM model registry: your source of truth

A model registry is the controlled store where approved model artifacts live, with their versions, metadata, lineage and access rules. It answers the questions an auditor or an on-call engineer asks: which weights are in production, who approved them, what were they trained or tuned from, and who can change them.

Models are large binary files, and teams often move them by hand, which creates what the cloud-native community calls operational fragility. A CNCF article on the infrastructure of AI models describes how Harbor manages models as OCI artifacts with immutable tags and digests, fine-grained access control, signature verification, lifecycle policies, replication and audit logs. The same piece notes that treating weights as OCI artifacts lets model delivery reuse container tooling for security scanning, signed provenance, GitOps deployment and Kubernetes-native pulling.

Whatever product you choose, look for these properties:

Immutability. A version tag, once published, never points to different bytes. Pin by digest in deployments.
Signing and verification. Artifacts are signed at publication and verified before they are pulled to a serving node.
Access control. Separate who can publish, who can promote to production and who can read.
Lineage. Each entry links to its parent model, adapter training data reference and evaluation results.
Lifecycle policy. Rules for retiring old versions while keeping any version that is still within your rollback window or audit retention period.
Audit logs. Every push, promotion and pull is recorded.

If you already run Kubernetes for inference, the registry slots into the same delivery pipeline. Our private deployment guide to Kubernetes for LLM inference covers how pulled artifacts become serving pods.

Versioning in private and air-gapped environments

Offline environments change the mechanics, not the principles. You cannot pull a model from a public hub on demand, and that is a feature: every artifact that enters the network can be inspected first. Guidance for enterprise local deployments suggests securing air-gapped workflows with immutable internal registries, signed and scanned release sets, controlled transfers with two-person custody where warranted, and internal mirrors.

A typical flow looks like this:

Stage outside the boundary. Download the candidate weights and dependencies in a connected staging zone and record source and hashes.
Scan and sign. Run malware and supply-chain checks, then sign the release set as a unit.
Transfer under control. Move the set across the boundary on approved media, with two people present for sensitive environments.
Verify on arrival. Check signatures and hashes against the manifest before anything is imported.
Publish to the internal registry. The registry is now the only place serving nodes pull from.
Mirror what you need. Container base images, Python packages and inference engine builds get internal mirrors, so a rebuild never needs the internet.

Signing matters beyond tidiness. The Harbor-focused CNCF article notes that signature verification before distribution is meant to prevent model-poisoning attacks. The OWASP guidance on secure AI model operations goes further: store models in access-controlled registries, sign model binaries, and encrypt weights and datasets at rest.

Testing a new version against the one in production

A new version is a hypothesis until it beats the old one on your own work. Microsoft’s guidance on validating an AI model migration lays out a simple discipline: run a test set of business-critical and high-frequency scenarios against the current production model, then run the same set against the candidate and investigate individual regressions.

To make that useful:

Build the test set from real work. Pull representative questions from your logs (with privacy rules applied), plus the rare, expensive failures that your team remembers.
Freeze and version the test set. If the test set changes, results are not comparable. It is an artifact in your manifest.
Look at individual regressions, not only averages. A candidate that wins on average can still fail the one case your compliance team cares about.
Add adversarial cases. Prompt injection, data-extraction attempts and policy edge cases belong in the suite. See how we approach measurable red teaming in sovereign deployments.
Measure operations too. Latency, memory, throughput and energy per request can make a better model unusable on your hardware.

Treat the evaluation output as part of the release. Store the results next to the manifest, so a year later you can show what was tested and what was accepted.

Rolling out and rolling back

Testing reduces risk but does not remove it, because live traffic is messier than any test set. A staged rollout limits the blast radius. A test plan for model upgrades describes the pattern: shadow the candidate on sampled live traffic without returning its outputs, compare response and performance measures, then roll out by traffic share.

In practice:

Shadow. The candidate receives a copy of sampled requests. Users see only the current version’s answers. You compare outputs, latency and failure rates.
Canary. Route a small share of real traffic, ideally from a low-risk user group, to the candidate.
Ramp. Increase the share in steps, pausing at each one long enough to see rare cases.
Full release. Keep the previous version warm and deployable for your rollback window.

Decide the rollback before launch. That means the previous release manifest is intact, its artifacts are still in the registry, its retrieval index has not been deleted, and someone knows who is authorized to trigger the switch. A rollback that requires rebuilding an index overnight is not a rollback.

Retrieval indices deserve particular care. If the new model needs a new embedding model, the old and new indices must coexist until the old release is retired. The broader process of gating releases through testing, evaluation, verification and validation is laid out in our 90 day LLMOps roadmap for secure sovereign deployment.

Governance: making versions auditable

In finance, defense and other regulated settings, versioning is also evidence. Several practices turn your registry and manifests into an audit trail.

Record decisions. A NIST-hosted document on machine learning best practices recommends a standard stakeholder sign-off document at project checkpoints, giving an auditable record of expectations at each major decision point. Attach that sign-off to the release manifest.

Approval sheet secured inside a release binder

Plan for maintenance. The NIST AI Risk Management Framework notes that AI systems may require frequent maintenance and corrective-maintenance triggers because of data, model or concept drift. Define those triggers in advance: an evaluation score below a threshold, a change in input distribution, a new document corpus. For a concrete picture of how drift is flagged on devices at the edge, see this write-up on edge model drift detection.

Treat foundation models as supply-chain dependencies. That means knowing where weights came from, verifying signatures, and requiring a re-review when a model changes materially. Pair it with formal change management so a Major version bump cannot reach production through a quiet pipeline run.

Restrict and log. Separate duties between those who publish a model, those who approve it and those who deploy it. Keep logs of every promotion.

Our perspective on versioning sovereign LLMs

We see the same pattern in almost every private AI program. The model gets all the attention, and the release process around it gets built last. Then the first upgrade arrives, and nobody can say with certainty what is running, what changed, or how to go back.

Our view is that versioning is an operating discipline, not a tool purchase. A sovereign deployment has an advantage that hosted services cannot offer: every artifact is inside infrastructure you control, so every version can be pinned, signed, tested and reproduced on your schedule, not a provider’s.

That is how we approach engagements as an official webAI systems integrator. We deploy and operate webAI’s local-first platform, including its Personas, Intelligence Delivery Network and webAI Frontline, inside your data center, edge site or air-gapped network. We design the registry, the signing chain, the evaluation gates and the rollback path alongside the deployment, so the first upgrade is routine.

Our team brings more than 20 years in high-consequence and Fortune 100 environments, where “we think it is the same as last month” is never an acceptable answer.

John

Put versioning into a private deployment

If your models run, or will run, inside your own perimeter, versioning should be designed with the deployment, not bolted on later. Forge AI Deployment designs, deploys and operates private and air-gapped AI systems on webAI’s platform, covering architecture, secure deployment, model integration and ongoing operation, with data kept inside infrastructure you control.

Forge AI Deployment

See how secure local and air-gapped deployment works, or read about the webAI platform behind our deployments.

FAQ

What is the difference between model versioning and prompt versioning?

Model versioning tracks the weights and adapters that generate text. Prompt versioning tracks the instructions and templates sent to them. Both change behavior, so a release should record both in the same manifest, with the evaluation results that justify the pair.

Do I need a model registry if I only run one model?

Yes, if the system is in production and anyone needs to audit it. One model still gets updated, fine-tuned and rebuilt over time. A registry with immutable versions and access control gives you a reliable way to return to a known state and to show who changed what.

How is an LLM model registry different from a container registry?

They share a lot. Treating weights as OCI artifacts lets you use the same storage, signing, scanning and access-control patterns as for container images. The extra layer is metadata: lineage, evaluation results and approval status tied to each model version.

How do I version models in an air-gapped network?

Stage and scan artifacts outside the boundary, sign the release set, transfer it under controlled custody, verify on arrival and publish to an internal registry. Serving nodes pull only from that registry, and internal mirrors cover dependencies such as container images and packages.

When should a change trigger a full re-evaluation?

Any Major change: a new base model, a new embedding model, a changed tool contract. Minor changes such as a new adapter or a prompt that alters output format also need the baseline test set. Patch changes should still run a quick regression check to confirm behavior is unchanged.