NVKE Is a Kubernetes Cluster Factory, and NVIDIA Has Already Proposed Rebuilding Its Assembly Line

Sponsored by

SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | November 15, 2026

The API ships unauthenticated, over plain HTTP. Even after you turn authentication on, NVKE does no authorization: any admitted caller can create or delete any customer's cluster. And four days ago, an architecture decision record proposed retiring the Postgres-backed REST API at the center of the current design and rebuilding NVKE as Cluster API providers.

None of that makes NVKE uninteresting. It makes it a well-engineered preview, built by people who write unusually candid docs, with a durable core (kubeadm, SPIFFE, CAPNICo, per-cluster isolation) and an orchestration layer that's about to be replaced.

The promise: by the end you'll know how a POST /api/clusters becomes a running, isolated Kubernetes cluster on NVIDIA bare metal, which parts of that path are durable and which are slated for replacement, and what NVKE's own benchmarks do and don't show.

The fence: we read the NVKE repository at 06b758af (November 9, 2026; 1,555 commits since August 4; 69 contributors; about 194K lines of non-test Go across the controller, agent, CLI, CAPI provider, management engine and auth modules, and more test code than product code in the controller). That includes the architecture doc, the preview-status page, the outbox processor, the supported-platforms reference, the roadmap, the November 5 CAPI ADR and the published benchmark results. We didn't install NVKE. NVIDIA's docs site wasn't fetched separately; the repo's docs/ tree is its source.

What It Actually Does

NVKE is the management plane an AI-factory operator installs to hand customers their own Kubernetes clusters on NVIDIA bare metal. It isn't a Kubernetes distribution, and it says it isn't a Cluster API flavor. It calls itself "a self-hosting factory for Kubernetes clusters."


Status	Preview, Apache 2.0, "we do not recommend NVKE for production workloads"
Clusters	A throwaway seed cluster creates a management cluster, which creates many workload clusters, each with its own control plane, etcd and node pools
Infrastructure	NICo (NVIDIA Infra Controller, formerly Forge) via the CAPNICo provider; LaunchLayer optional; `nico-sim` for laptops
Kubernetes	1.36.3 pinned; OVN-Kubernetes 2.4.2 (NVIDIA build); IPv4 only
Identity	SPIRE attestation by TPM, join token or DSX IMDS; SPIFFE mTLS between controller and node agent
Lifecycle	Create, delete, scale node pools, replace nodes, control-plane upgrade (one minor at a time), etcd encryption, OIDC, admin-cert renewal
Binaries	controller, agent, nvkectl, updater, management-engine, tenant-api, operator-api, plus test tools
Companions	NVKR (builds signed node images and charts), AICR (optional accelerated runtime), NKTP (telemetry)

Think of a car factory. The management cluster is the factory and each workload cluster is a car it builds and services. Customers drive their car with their own keys (kubectl) and never enter the factory.

The Architecture, Unpacked

What to notice: the API never talks to Temporal. It writes two rows in one transaction, and a separate poller turns committed rows into workflows. The node agent never calls home; the controller dials it.

Four decisions define NVKE today.

One: the transactional outbox. A cluster record and its pending work commit together or not at all, so there's never a cluster without queued work or work without a cluster. The architecture doc calls this "the part most people get wrong when they first read the handlers."

Two: one process, three engines. The controller binary runs a REST API with Postgres, a Temporal worker, and controller-runtime reconcilers. The doc warns that reading only one of them "gives an incomplete picture of what reacts to a change."

Three: hardware-rooted node identity. Nodes attest to SPIRE, by TPM in the intended setup, before receiving join credentials. Workload-cluster nodes get the management cluster's SPIRE trust bundle in their user data.

Four: hard isolation per customer. Every workload cluster is a full Kubernetes control plane with its own etcd, not a namespace or a virtual cluster. That's the strongest tenancy model, and the most expensive.

The Code, Annotated

Snippet one: the outbox processor (excerpt from controller/internal/temporal/outbox_processor.go, our comments marked // SnackOnAI:).

func (p *OutboxProcessor) processBatch(ctx context.Context) error {
	return p.db.RunInTx(ctx, func(ctx context.Context, tx db.Tx) error {
		// List unprocessed entries with FOR UPDATE SKIP LOCKED to avoid duplicate processing
		entries, err := tx.ListUnprocessedWorkflowOutboxEntries(ctx, p.cfg.OutboxBatchSize)  // ← THIS is the trick:
		...                                                       //   replicas can poll in parallel; locked rows are skipped
		for _, entry := range entries {
			// A failure on one entry is logged and does not stop the rest of the batch.
			p.dispatchEntry(ctx, tx, entry)  // SnackOnAI: Temporal is called INSIDE the open transaction,
		}                                    //   so row locks are held for the duration of those RPCs
		return nil
	})
}

func (p *OutboxProcessor) dispatchEntry(ctx context.Context, tx db.Tx, entry db.WorkflowOutbox) {
	ctx = otelx.Restore(ctx, entry.TraceContext)   // SnackOnAI: the trace continues from the original HTTP request
	if err := p.startWorkflow(ctx, entry); err != nil {
		return                                     // SnackOnAI: left unprocessed, so the next poll retries it
	}
	tx.MarkWorkflowOutboxEntryProcessed(ctx, entry.ID)
}

The outbox gives at-least-once delivery. The entry is marked processed only after Temporal accepts the workflow, so a crash between the two causes a redelivery, which snippet two makes harmless.

Snippet two: making redelivery idempotent (same file).

workflowOpts := client.StartWorkflowOptions{
	TaskQueue: p.cfg.TemporalTaskQueue,
	ID:        entry.WorkflowID,                 // ← THIS is the trick: a deterministic workflow ID per entry
}
we, err := p.temporalClient.ExecuteWorkflow(ctx, workflowOpts, entry.WorkflowType, inputMap)
if err != nil {
	// A WorkflowExecutionAlreadyStarted error means a prior delivery of this same
	// outbox entry already reached Temporal successfully. Treat it as success so the
	// entry is marked processed and doesn't loop forever.
	var alreadyStarted *serviceerror.WorkflowExecutionAlreadyStarted
	if errors.As(err, &alreadyStarted) {
		return nil                                // SnackOnAI: duplicate delivery becomes a no-op
	}
	return fmt.Errorf("failed to start workflow: %w", err)
}

Outbox plus deterministic workflow IDs adds up to effectively-once cluster provisioning: the textbook pattern for coordinating a database with a workflow engine, done correctly.

Snippet three: the customer-facing lifecycle (commands as documented in the repo's guides; flags abbreviated).

# Operator: create a customer cluster. NVKE provisions machines through NICo.
nvkectl create cluster --name acme-prod ...        # → POST /api/clusters → outbox → Temporal

# Upgrade the control plane: one minor version at a time, one operation at a time (others get HTTP 409)
nvkectl patch cluster acme-prod --control-plane-kubernetes-version 1.37.2

# Node pools: nvkectl patches only the count; a pool's Kubernetes version needs a raw API call
nvkectl patch nodepool <POOL_ID> --desired-node-count 16
curl -X PUT .../api/nodePools/<POOL_ID> -d '{"kubernetesVersion":"1.37.2"}'   # ← THIS is the gap: old nodes are
                                                                              #   only MARKED NeedsRecycle, never drained

The preview automates creation, scaling and control-plane upgrades, but rolling workers to a new version is half-manual: NVKE marks outdated nodes and you, or a break-fix system like NVSentinel, delete them.

It In Action

Input: an operator creates a workload cluster for customer "acme": 3 control-plane nodes and a 42-worker pool on NICo, Kubernetes 1.36.3, etcd encryption on.

Step one, request. POST /api/clusters. One Postgres transaction writes the clusters row and a workflow_outbox row.

Step two, dispatch. On its jittered polling interval, the OutboxProcessor locks the row (SKIP LOCKED) and starts the provision workflow with a deterministic ID, continuing the request's OpenTelemetry trace.

Step three, machines. The workflow creates CAPI resources. CAPNICo asks NICo for machines, which boot the NVKR node image. Their user data carries the management cluster's SPIRE trust bundle.

Step four, identity and bootstrap. Each nvke-agent attests to SPIRE (TPM). The controller dials each agent over gRPC with SPIFFE mTLS, runs kubeadm init on the first control-plane node and kubeadm join on the rest, then installs OVN-Kubernetes.

Step five, hand-off. The cluster reaches Provisioned. The customer gets the admin kubeconfig, whose client certificate is valid for one year. The controller checks it about every 15 minutes and renews it when fewer than 30 days remain.

Output: what a cluster like this sustains (NVKE's own published results, Kubernetes 1.34.2, 3 control-plane + 2 monitoring + 42 workers of 128 cores and ~1 TB each):

Benchmark	Load	Result
cluster-density	3,780 pods in 378 namespaces + 12,474 objects in 710 s	pod-ready p99 2 s (bar 13 s), etcd WAL fsync 0.99 ms
node-density	4,120 pods, ~105 per node, in 212 s	pod-ready p99 2 s (bar 15 s)
ClusterLoader2 SLO	peak 3,920 pods, 40 QPS	pod startup p99 2.54 s (SLO ≤ 5 s), API p99 ~50 ms, zero 429s
scheduler baseline	100 to 5,000 simulated (KWOK) nodes	algorithm p99 19.1 ms at 5,000; queues up to 727 s when `kube-api-qps` < workload rate

scheduler algorithm-phase p99 vs simulated node count (NVKE results)
   500 nodes  ▏                      ~1 ms
 1,000 nodes  ████                   3.7 ms
 5,000 nodes  ████████████████████   19.1 ms   ← and binding queues if kube-api-qps is set too low

The workload clusters behave like well-tuned upstream Kubernetes. The benchmarks measure the clusters NVKE builds, not the factory: none measures provisioning time, clusters per management plane, or GPU workloads.

Why This Design Works (and what it trades away)

It works because isolation is physical. Separate control planes and etcd mean one customer's runaway controller or etcd compaction can't touch another's. For GPU clouds selling dedicated capacity, that's the right default.

It works because orchestration is durable. Temporal workflows survive controller restarts mid-provision, and the outbox guarantees no request is lost between the API and the workflow engine.

It works because trust starts in hardware. TPM-backed SPIRE attestation means a node proves what it is before it gets join credentials, and controller-to-agent traffic can be mTLS without static secrets.

It trades away cost efficiency. A full control plane per customer (the docs require at least 3 voters for some operations) is pure overhead for small tenants. Hosted control planes appear only as a "potential hosting model" in the new ADR.

It trades away operational simplicity. Postgres, Temporal, CAPI, CAPNICo, SPIRE, OVN-Kubernetes and NICo all need to be operated and upgraded. The management cluster itself can't yet upgrade, scale, back up or restore.

It trades away a stable API. The November 5 ADR proposes replacing the Postgres-backed controller REST API with a CAPI-native NVKE Core, Service Layer and NVKE API, and states that "the legacy API will be retired entirely."

Technical Moats

The NICo integration. CAPNICo, user data carrying SPIRE trust bundles, and node images built and signed by NVKR. A generic CAPI setup doesn't know how to boot NVIDIA's bare-metal stack, and NVKE does. This is NVIDIA's real advantage.
The people. Davanum Srinivas, a long-time Kubernetes maintainer, authored about a third of all commits. Upstream fluency shows in the defaults: no feature gates changed, upstream SLOs as the benchmark bar.
Evidence culture. Benchmarks are defined as data, with honest scope notes ("reading it as 5,000 machines would be wrong"). The preview page lists every known limitation, including FDE images that don't protect data at rest. That candor builds trust with operators.
Not the orchestration code. Outbox, Temporal and reconcilers are standard patterns, and NVIDIA is itself proposing to replace them with CAPI providers.

Contrarian Insights

One: "multi-tenant" today means isolated clusters, not isolated tenants at the API. The preview API listens on port 30080 of every management node, over plain HTTP, with authentication off until you configure JWTs. Even then, "NVKE does no authorization. Every caller that API authentication admits can call every endpoint, including the ones that create and delete workload clusters." Customers get only an admin kubeconfig that "cannot be revoked" until its certificate expires. Per-customer authorization is on the roadmap as "coming soon." Until it ships, NVKE is a tool for an operator's own staff, not an API you expose to tenants.

Two: you'd be adopting an architecture its authors have proposed replacing. The ADR is candid: NVKE "was originally built for internal use around a specific use case," and broader adoption needs "a complete managed Kubernetes product composed of reusable components." The plan is three layers (NVKE Core as CAPI control-plane and bootstrap providers, a Service Layer for curated releases and add-ons, and a new NVKE API) and retiring the current REST API with its outbox and Temporal path. The durable bets are kubeadm, SPIFFE identity, CAPNICo and per-cluster isolation. The orchestration layer described in today's architecture doc is the part most likely to change.

Surprising Takeaway

NVKE's published benchmarks were run on Kubernetes 1.34.2 with CPU-only workers (128 cores, ~1 TB of RAM, no GPU listed), while the release pins Kubernetes 1.36.3 and the node image stages 1.34.2 packages that are explicitly "not supported." The numbers are excellent and honestly scoped, with a 2-second pod-ready p99 and every SIG-Scalability SLO met. But they describe an older version, measure pause pods rather than GPU jobs, and say nothing about the factory itself: how long a cluster takes to provision, or how many clusters one management plane can run.

TL;DR For Engineers

NVKE is a management plane that creates fully isolated Kubernetes clusters (own control plane and etcd) on NVIDIA bare metal through NICo, CAPI, kubeadm and a gRPC node agent with SPIRE/TPM identity.
POST /api/clusters writes a cluster row and an outbox row in one transaction. A SKIP LOCKED poller starts a Temporal workflow with a deterministic ID, so provisioning is effectively once.
Preview reality: plain HTTP, auth off by default, no authorization, irrevocable admin kubeconfigs, no management-plane upgrade or backup, FDE images without real data-at-rest protection.
A November 5 ADR proposes rebuilding on CAPI providers and retiring the current REST API. Build integrations against kubectl and CAPI, not the legacy endpoints.
The benchmarks show healthy upstream-like clusters (2 s pod-ready p99 on 42 workers) but were run on 1.34.2 with CPU nodes and don't measure provisioning or fleet scale.

Explain It Like I'm New

Companies that buy big racks of NVIDIA GPUs often want to rent them out to customers, the way cloud providers do. Customers increasingly expect that capacity to come as Kubernetes, the standard software for running applications across many machines.

Setting up and maintaining Kubernetes for many customers is hard. Each customer needs their own private cluster, machines have to be installed and checked, software has to be upgraded, and security certificates have to be renewed, all without customers interfering with each other.

NVKE is NVIDIA's open-source answer: a control center the GPU owner runs. When the owner asks for a new customer cluster, NVKE reserves physical machines, installs the operating system, proves each machine is genuine using a security chip on the board, and assembles a fresh Kubernetes cluster that belongs only to that customer. Later it can add machines, upgrade the cluster, or replace broken nodes.

It's a factory that builds and services Kubernetes clusters. The customer drives the finished car, and the factory owner keeps it running.

It's still an early preview, with important security pieces unfinished. But it shows where AI infrastructure is heading: GPU owners becoming small cloud providers, using shared open tools instead of building everything themselves.

See It In Action

NVKE repository and local quickstart, NVIDIA on GitHub. Run it without GPUs. The local quickstart builds a management plane and a workload cluster on simulated NICo machines with virtual TPM, on one Linux host (6 CPUs, 12 GiB, 100 GiB). It's the most concrete way to watch the outbox, workflows and agent bootstrap.
NVKE benchmark results, NVIDIA NVKE Scale Team. Read the scale runs. Cluster density, node density, ClusterLoader2 SLOs, scheduler limits and network throughput, each with a reproduction recipe and honest scope notes.
Composable NVKE architecture ADR, NVIDIA on GitHub. Read the proposal. The three-layer CAPI design that would replace today's controller API, with diagrams. Required reading before building on NVKE's API.

We found no conference talk or video walkthrough of NVKE as of November 9.

Community Conversation

NVKE maintainers, Preview status and known limitations. An unusually blunt limitations page: no authorization, plain HTTP, no management-plane upgrade or backup, irrevocable kubeconfigs, and FDE images that "do not protect data at rest."
Trey Thomas (NVIDIA), CAPI architecture ADR, PR #2171. Argues that broader adoption needs reusable CAPI control-plane and bootstrap providers so operators can adopt NVKE "at progressively lower layers," and commits to retiring the legacy API.
NVIDIA NVKE Scale Team, scheduler baseline. A practical lesson for any operator: set the scheduler's kube-api-qps at or above your pod creation rate, or binding queues grow without bound, up to 891 s in their worst cell.
Kubernetes SIG Cluster Lifecycle, Cluster API. The upstream project NVKE is converging on. Its provider contracts are what the ADR adopts for control-plane and bootstrap lifecycle.

Conclusion: Bet on the Clusters, Not the Current API

NVKE gets the hard, durable parts right: per-customer control planes and etcd, TPM-rooted node identity, kubeadm bootstrap, NICo integration, and an outbox-plus-Temporal path that provisions effectively once. Its benchmarks show the clusters it builds behave like well-tuned upstream Kubernetes. As a product it's a preview, and its own docs and ADR say so: no API authorization, no management-plane upgrades, and a proposal to replace the very API most integrations would target. If you run NVIDIA bare metal, evaluate NVKE now with the local quickstart and influence the CAPI redesign. Hold off on exposing it to customers until per-tenant authorization ships, and integrate through kubectl and Cluster API, the interfaces that will survive the rebuild.

References

Large-scale cluster management at Google with Borg, Verma et al. The cluster-management lineage Kubernetes and its management planes descend from.
Borg, Omega, and Kubernetes, Burns et al. Why reconciliation loops and declarative state became the model NVKE's reconcilers and CAPI follow.
Solving the Bottom Turtle: A SPIFFE Way to Establish Trust in Your Infrastructure via Universal Identity, SPIFFE community. The identity model behind NVKE's TPM-attested node trust.
Sagas, Garcia-Molina and Salem. The long-running-transaction pattern that workflow engines like Temporal implement for multi-step provisioning.

NVKE is NVIDIA's open-source preview for building isolated, per-customer Kubernetes clusters on bare metal, combining a transactional outbox, Temporal workflows, CAPI with CAPNICo, and SPIRE/TPM node identity. Its clusters benchmark well, but the API has no authorization yet, and a new ADR proposes replacing the current controller API with CAPI-native layers.

Sponsored Ad If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad—it helps us keep building and delivering value 🚀

How AI-Era Pricing Is Reshaping Finance Operations

Usage-based and hybrid pricing models are changing how B2B companies generate revenue — and creating new headaches for the finance teams behind them.

Tabs co-founder Rebecca Schwartz and PwC Partner Amit Dhir sat down to unpack exactly what that means in practice: how pricing model decisions ripple into revenue recognition, forecasting, and financial ops — and what it takes to scale without piling on manual work.

Watch the on-demand recording to get practical frameworks, real-world examples, and a clear path to operationalizing usage-based revenue — including a forward-looking take on how AI will reshape financial workflows. If your team is navigating pricing complexity heading into the back half of the year, this is worth an hour.

Watch the recording now