OCTOBER 11, 2026

Air Gapped Kubernetes: Verify Artifacts Before Sovereign AI Runs

Build an air gapped Kubernetes workflow with Harbor, Zarf, and Hauler; verify artifacts before deployment and keep sovereign AI systems offline.

Air Gapped Kubernetes: Verify Artifacts Before Sovereign AI Runs
Air Gapped Kubernetes: Verify Artifacts Before Sovereign AI Runs

Sketch of verified artifacts framing a title

Air-gapped Kubernetes runs without internet by prebuilding, signing, and transferring all required binaries and container images into an internal registry, then enforcing integrity and admission policy at runtime. The workflow follows a consistent sequence: prepare, transfer, import, verify, run. It relies on package bundlers like Zarf and Hauler, a private registry such as Harbor, and signed SBOMs. This guide covers both the initial bootstrap and the ongoing operations that keep a disconnected cluster patched and auditable.

TL;DR:

  • A true air gap has no physical or logical outside connection, while DDIL permits intermittent links; use a connected mirror, transfer enclave, and isolated high side.
  • Pin images by digest, generate and store SBOMs, then verify each transfer’s checksum, signature, and vulnerability scan in that order.
  • Before kubeadm init, preload control plane and separate CNI images, match operating system packages to node architecture, and point configuration at your internal registry.
  • Set monthly or biweekly patch transfers, expedite critical vulnerabilities, and test every bundle in staging before production; log fixes and arrival dates.
  • Encrypt scheduled etcd snapshots, store a copy on separate physical media, and regularly test full restores with the bootstrap bundles needed for bare metal recovery.

Table of Contents

When Should You Use a True Air Gap Versus DDIL?

Not every disconnected environment needs the same level of isolation. A true air gap means zero physical or logical connection to any outside network, ever. A DDIL environment (disconnected, degraded, intermittent, or limited) has occasional connectivity windows, which changes your patch cadence and tooling choices considerably.

  • True air gap: no physical uplink exists; every artifact enters through removable media or a controlled transfer station.
  • DDIL: intermittent or low-bandwidth links exist, so updates can sync opportunistically rather than through manual transfer.
  • Low-side/high-side split: a connected “low side” mirrors upstream sources while an isolated “high side” runs production workloads.

Stricter isolation raises operational cost and slows patch delivery, a trade-off the Kubernetes documentation frames as a spectrum rather than a binary choice. This guide uses a three-zone model (connected mirror, transfer enclave, high side) that applies to both true air gaps and DDIL setups.

How Should You Structure the Three-Zone Network Architecture?

The three-zone pattern separates concerns cleanly. The connected mirror station pulls and bundles artifacts from public sources. The transfer enclave moves bundles across the gap under controlled procedures. The high-side cluster runs production workloads with no outbound path at all.

  • Connected mirror: hosts your build tools, Zarf/Hauler bundlers, and SBOM generators, with outbound internet access only here.
  • Transfer enclave: a controlled checkpoint, often physical media or a one-way diode, where artifacts are scanned and logged before crossing.
  • High-side cluster: runs Harbor, etcd, and your Git server, all reachable only from inside the isolated network.

Avoid flat L2 networks on the high side. Micro-segment with VLANs, keep baseboard management controllers (BMCs) on a separate management plane from workload traffic, and apply Kubernetes NetworkPolicy objects so a compromised pod cannot reach etcd or the registry directly.

Pro Tip: Place Harbor and your Git server on the same high-side VLAN as the control plane, but firewall them from worker node subnets so a compromised workload cannot reach your source of truth.

How Do You Package and Transfer Artifacts Across the Gap?

Every binary, image, and chart your cluster needs must be inventoried before it ever leaves the connected side. That list typically includes Kubernetes binaries, CNI images, Helm charts, OS packages, a vulnerability database, and generated SBOMs.

  1. Inventory everything, pinning images by digest rather than tag so the bundle is reproducible and auditable.
  2. Bundle with Zarf or Hauler, which package images, charts, and init steps into a single transportable archive and can generate SBOMs automatically, as described in SEI’s work on edge Kubernetes.
  3. Use skopeo, crane, or oras when you need raw OCI archives instead of a full bundler, particularly for one-off image pulls.
  4. Generate SBOMs with Syft and store them alongside the bundle, not as an afterthought.
  5. Sign everything with cosign or Notation before the bundle leaves the connected side.
  6. Log the transfer, recording checksums, operator identity, and timestamp, ideally with a second person verifying the handoff.

Pro Tip: Treat digest pinning as non-negotiable. A tag can point to a different image tomorrow; a digest cannot, which is what makes your bundle verifiable months later.

SEI’s Vessel project addresses a related problem: confirming that a rebuilt image is bit-for-bit identical to what was signed, which matters when your transfer process spans weeks or involves multiple build environments.

How Do You Configure Harbor for Offline Scanning and Policy Enforcement?

Your internal registry has to do double duty: host every artifact your cluster pulls and enforce policy without ever phoning home. Harbor is the common choice because it combines project-level quotas, robot accounts, and built-in vulnerability scanning, a pattern detailed in KubeDojo’s air-gap security module.

  • Use robot accounts with scoped, short-lived credentials for automated imports instead of shared admin logins.
  • Set per-project quotas and retention policies so a runaway CI job cannot exhaust high-side storage.
  • Transfer the Trivy or Aqua vulnerability database manually, and watch for the separate Java DB component, which is easy to miss and leaves Java artifacts unscanned if skipped.
  • Verify DB freshness on import, since a stale database gives false confidence rather than no confidence.
  • Enforce admission policy with Kyverno’s verifyImages or Sigstore’s policy-controller so unsigned images are rejected before they ever schedule.

What Should Your kubeadm Bootstrap Checklist Include?

Bootstrapping with kubeadm on an air-gapped host fails most often because an expected artifact simply isn’t there yet. The official kubeadm air-gap guide lays out the dependency list you need to preload before running kubeadm init.

  1. Preload control plane images (API server, scheduler, controller manager, etcd) into your local registry before init runs.
  2. Install matching OS packages for containerd, kubelet, and kubeadm itself, matched exactly to your target architecture.
  3. Preload CNI images separately. Calico and Flannel are not bundled with kubeadm and are a frequent cause of pods stuck in a pending state.
  4. Point kubeadm’s configuration at your internal registry, not the default registry.k8s.io, or the init process will try to reach the internet and fail.
  5. Consider a mutating webhook approach, as Zarf does, to retag images automatically rather than hand-editing manifests for every component.

Architecture mismatches (pulling amd64 images onto arm64 nodes) and registry misconfiguration account for most init-time failures operators report.

How Do You Manage GitOps, etcd Security, and Backups Day to Day?

Once the cluster is running, GitOps and a hardened etcd keep it maintainable without reopening the gap. A local Git server, Gitea or GitLab, initialized from a signed bundle, gives Flux something to poll internally rather than assuming Docker Hub or GitHub APIs are reachable.

  • Run Flux against internal Git and OCI sources, and disable any image automation path that assumes an external registry metadata API.
  • Secure etcd with mutual TLS and firewall it so only the API server can reach it, per Kubernetes control-plane communication guidance.
  • Encrypt etcd snapshots before they leave the cluster for backup storage, and transfer them under the same two-person discipline as any other artifact.
  • Set a fixed cadence for vulnerability DB and OS patch bundle transfers rather than reacting ad hoc.

Pro Tip: If you run multiple high-side clusters, replicate signed bundles to each one from a single verified source rather than re-bundling per cluster. It cuts verification work and keeps every cluster on the same artifact set.

What Verification Order Protects Your Supply Chain?

Every artifact crossing the gap needs the same sequence applied before it touches a running cluster, and the order matters. Scanning a tampered file for vulnerabilities before confirming its integrity wastes effort on an input you can’t trust anyway, a point the KubeDojo module makes directly.

  1. Checksum first. Confirm the file matches its recorded SHA256 hash.
  2. Signature second. Verify the cosign or Notation signature against your trusted key.
  3. Policy scan third. Run Trivy or Grype against the verified artifact, cross-referencing the SBOM generated on the connected side.

Keep SBOMs stored next to each bundle so exposure questions can be answered with a simple search rather than a full rescan later.

How Does Forge AI Deployment Apply This to Sovereign AI?

Forge AI Deployment designs and operates private, air-gapped infrastructure for sovereign AI systems, acting as an official webAI systems integrator. Our engagements follow the same discipline outlined above: design the zone architecture, assemble signed bundles, move them through a controlled transfer, import into Harbor, and enforce runtime policy before anything serves traffic.

  • We deploy webAI’s platform, including Personas and the Intelligence Delivery Network, entirely inside infrastructure our clients control.
  • We operationalize webAI Frontline, giving frontline teams source-backed answers from large technical document libraries, fully offline, on a single device.
  • We apply the same checksum, signature, and policy-scan sequence to AI model artifacts that this guide applies to container images.

How Do You Monitor and Alert Without Internet Access?

Standard observability stacks assume a path to a SaaS backend, so an air-gapped cluster needs that assumption removed at every layer. Run Prometheus and Grafana locally, scraping metrics from within the high side only, with no remote-write target pointed outside the network.

Log aggregation works the same way: a local Loki or Elasticsearch instance, fed by Fluent Bit or Vector agents running inside the cluster, keeps logs searchable without an external shipper. Store at least enough retention locally to cover your patch and audit cycle, since you cannot simply re-pull historical data from a cloud log service if local storage rotates out.

Alerting has to route internally too. Alertmanager can still page through email or a local chat server if one exists on the high side, but webhook integrations to external paging services won’t work, so plan for a local escalation path, whether that’s a monitored console, an internal ticketing system, or a physically present on-call operator.

Dashboards should surface the state of your own supply chain alongside workload health: registry quota usage, vulnerability DB age, and signature verification failures all belong next to CPU and memory graphs. A cluster can look healthy on every standard metric while running against a six-month-old vulnerability database, and that gap is exactly what tends to go unnoticed without a dedicated panel for it.

Treat your monitoring stack itself as an artifact that crosses the gap under the same discipline as everything else: bundled, signed, and verified on import, not assembled ad hoc on the high side from whatever packages happen to be lying around.

How Do You Manage Patches Without a Live Connection?

Patch management in a disconnected cluster is a scheduling problem as much as a technical one. Without automatic update checks, vulnerabilities can sit unpatched simply because nobody noticed a new CVE existed, so the connected mirror side needs a standing responsibility to watch upstream advisories and package new bundles on a fixed interval.

Build a patch bundle the same way you built your initial artifact set: pin new image digests, regenerate SBOMs, sign the bundle, and route it through the same transfer enclave and verification order (checksum, then signature, then policy scan) as any other artifact. Treat OS package updates, CNI image updates, and the vulnerability database itself as separate bundle types with their own cadence, since a kernel patch and a Trivy DB refresh have very different urgency profiles.

Patch bundle verification steps in order

Stage every patch in a non-production high-side cluster, or a dedicated staging namespace, before rolling it into production. Air-gapped environments often run mission-critical workloads where a failed update is expensive to roll back, so the staging step is not optional overhead, it’s the only chance to catch a regression before it matters.

Keep a patch log separate from your transfer log: which CVEs a given bundle addresses, which components it touched, and when it reached each cluster. When an auditor or incident responder asks whether a specific vulnerability was patched and when, that log is what answers the question, not a reconstruction after the fact.

Finally, assign a fixed cadence rather than reacting purely to severity. A monthly or biweekly patch bundle cycle, with an expedited out-of-cycle path for critical CVEs, keeps the connected mirror side from becoming a bottleneck every time a new advisory appears.

What Backup and Disaster Recovery Methods Fit an Air-Gapped Cluster?

Disaster recovery planning changes shape once you can’t restore from a cloud snapshot or pull a fresh base image on demand. Your backup strategy has to assume the only copies of your data are the ones you deliberately placed on the high side, or physically carried out of it.

etcd snapshots are the core of cluster-state recovery. Take them on a fixed schedule, encrypt them before they leave the cluster, and store at least one copy on separate physical media from the production environment, following the same mTLS and firewall discipline described earlier for live etcd traffic.

Persistent volume data needs its own backup path, typically through a tool like Velero configured against local object storage rather than a cloud bucket. Verify that Velero’s plugins and images were included in your original artifact bundle, since adding backup tooling after the fact means another transfer cycle before it’s usable.

Test restores on a real cadence, not just as a one-time exercise after initial setup. A snapshot that can’t actually rebuild a cluster is worse than no snapshot, because it creates false confidence until the moment a restore is needed.

For multi-cluster deployments, replicate etcd snapshots and PV backups to each high-side site from a single verified source rather than letting each cluster manage its own independent backup chain. That keeps your recovery point objective consistent across sites and avoids a scenario where one cluster’s backups silently drift out of sync with the others.

Document the full restore procedure, including which bundles and binaries are needed to rebuild a cluster from bare metal, not just the etcd restore command. In a true air gap, rebuilding from scratch means you need the original bootstrap bundle on hand, not just the data snapshot.

What Hardware and Physical Security Measures Matter Most?

An air gap protects the network layer, but hardware that enters a high-side facility carries its own risk. Supply-chain compromise at the hardware level, sometimes called a hardware trojan, is a concern specifically because software-level verification can’t detect it: a compromised component can sit below the layer your checksums and signatures ever inspect.

Source hardware from vetted suppliers and inspect it on arrival before it joins the network, rather than assuming a sealed box is a secure one. Where the threat model justifies it, some high-security deployments use hardware attestation or dedicated verification hardware before new nodes are trusted to join the cluster.

Physical security around the facility matters as much as the network boundary. Removable media is usually the only path artifacts take across the gap, which makes the transfer enclave a physical checkpoint as much as a technical one: controlled access, logged handoffs, and ideally a two-person rule for anything crossing in either direction.

Network hardware deserves the same scrutiny as compute. Switches, BMCs, and out-of-band management interfaces are common targets precisely because they’re often overlooked relative to the workloads running on top of them. Keep BMC traffic on an isolated management VLAN, change default credentials before a device ever joins production, and audit firmware versions against known-good baselines where your supplier provides them.

Finally, treat decommissioned hardware with the same care as incoming hardware. Drives and storage media leaving a high-side facility need secure wiping or physical destruction procedures, since a cluster node that once held production secrets doesn’t stop being sensitive just because it’s being retired.

How Do You Validate AI Model Updates Inside the Gap?

AI models introduce a wrinkle on top of standard container and binary patching: a model update isn’t just a new image, it’s a new set of weights whose behavior needs validation, not just a successful checksum.

Apply the same transfer discipline to model artifacts that you apply to container images: pin the exact model version, generate an SBOM or equivalent manifest describing its provenance, sign it, and move it through the transfer enclave under checksum, then signature, then policy-scan order, exactly as described earlier in this guide.

Validation goes a step further than a container image typically requires. Before a new model version reaches production, run it against a fixed evaluation set on the high side and compare outputs to the previous version’s known-good results. A model that passes every integrity check but produces materially different answers on your evaluation set needs a human decision before rollout, not an automatic promotion.

New model outputs compared before release

Keep model updates on their own bundle and release cadence, separate from OS patches and CNI updates, since model validation cycles often run longer than a typical security patch review. Trying to bundle a model update with an unrelated infrastructure patch just to save a transfer cycle tends to slow down whichever of the two actually needs to ship faster.

Store the validation results alongside the signed bundle, the same way you’d store an SBOM: the record of what evaluation a model version passed before reaching the high side is part of its provenance, not a separate artifact that can get lost in a different system.

Practical Perspective: What Actually Reduces Risk Here

Tooling like Zarf and Hauler removes real toil, but it doesn’t replace governance. Reproducible builds and disciplined transfer logs are what make a cluster auditable months later, not the bundler itself. Test the full pipeline in a lab, automate bundle creation, but keep a human two-person check on every transfer, and prioritize SBOMs, signatures, and admission policy over speed.

— John Ezzell, Founder

How Forge Can Deploy and Operate This for You

If you’re weighing the engineering cost of building this pipeline in-house against bringing in a team that has run it before, we design, deploy, and operate exactly this kind of infrastructure for sovereign AI systems. We handle secure local and air-gapped deployment, private AI assistant rollout, and sovereign MLOps and runtime orchestration.

Forge AI Deployment

  • We assemble and sign your artifact bundles following the checksum, signature, policy-scan order outlined above.
  • We import and manage your Harbor registry, robot accounts, and admission policy on your behalf.
  • We operate webAI’s platform, including Personas and the Intelligence Delivery Network, entirely inside the client infrastructure.

If you’re running document-heavy field or technical operations where connectivity is unreliable, pairing this kind of air-gapped cluster with edge model drift detection for offline fleets can catch anomalies before they reach your high side. Visit our solutions page to talk through a deployment or operations engagement.

FAQ

What is air gapping in computer networks?

Air gapping means a system has no physical or logical network connection to the internet or any other unsecured network. Data and software can only enter or leave through controlled, manual transfer, such as removable media passed through a checked transfer point.

Is Netflix using Kubernetes?

Netflix has publicly discussed using container orchestration and cloud-native infrastructure at scale, though its specific internal architecture choices are not something this guide can verify as current fact. For air-gapped deployments, the relevant point is that Kubernetes itself runs the same way regardless of whether the environment is a public cloud or a fully isolated network.

What is ingress in Kubernetes?

Ingress is a Kubernetes resource that manages external access to services inside a cluster, typically handling HTTP and HTTPS routing. In an air-gapped cluster, ingress controllers and their images need to be included in your initial artifact bundle, since they’re not part of the core kubeadm install.

What is a Kubernetes cluster?

A Kubernetes cluster is a set of machines, called nodes, running containerized applications that are managed together through a control plane. In an air-gapped setup, every binary and image that the control plane and nodes need must be preloaded locally rather than pulled from the internet, as described in kubeadm’s air-gap bootstrap guidance.

Does Forge AI Deployment handle air-gapped Kubernetes for AI workloads specifically?

Yes, we design and operate air-gapped Kubernetes infrastructure for sovereign AI deployments, including webAI’s platform, Personas, and Frontline, as an official webAI systems integrator. Pricing and scope for these engagements are available on request through our solutions page.

Sources

← All articles

BEGIN INSIDE THE PERIMETER

Let's talk about your environment.

Start a confidential conversation