Rafay is bringing NVIDIA Kubernetes Engine (NVKE) into its Kubernetes-as-a-Service (K8s) offering

TL;DR: A working tenant cluster about 21 minutes after logging in to a bare VM. Each new cluster after that in about three minutes. The whole lifecycle in five nvkectl verbs.

NVIDIA is building more of the AI factory software stack in the open, and Kubernetes lifecycle is part of it. NVIDIA Kubernetes Engine (NVKE), developed in the dsx-ai-factory project, builds, scales and tears down Kubernetes clusters on GPU machines. We wanted to see what it looks like from a standing start, so we ran the quick start end to end on a single cloud VM and recorded every step.

The rest of this post shows what gets installed and what comes up, and ends with what an operator still needs before selling these clusters to customers.

What's in the box

NVKE is a Kubernetes lifecycle engine built on Cluster API. In this quick start everything runs on one machine: a Kind cluster acts as the management plane, and simulated bare-metal machines stand in for real GPU nodes. Even in this all-in-one form you can see the production architecture. Tenant clusters are declared through a CLI, provisioned by durable workflows, attested over TPM-backed SPIFFE identities, and networked with OVN-Kubernetes and a BGP-advertised control-plane VIP.

The run at a glance

Step 1: Start from a clean VM

Ubuntu 24.04.5 LTS · kernel 7.0.0-1013-oracle · x86_64

The host is an ordinary cloud VM with 16 GB of memory and a 242 GB root disk. After a quick free -g; df -h / sanity check, switch to root and install the base packages: a build toolchain, jq, tmux, iptables-persistent, and the kernel's extra modules package.

sudo -i
ARCH=$(dpkg --print-architecture)
apt-get update
apt-get install -y git make gcc build-essential jq curl netcat-openbsd dpkg-dev tmux \
ca-certificates gnupg iptables-persistent "linux-modules-extra-$(uname -r)" || \
apt-get install -y git make gcc build-essential jq curl netcat-openbsd dpkg-dev tmux ca-certificates gnupg iptables-persistent

Step 2: Install the toolchain

Docker 29.9.3 · Go 1.27.3 · kind 0.32.0 · kubectl 1.37.2 · Helm 4.3.1 · Tilt 0.37.8

Docker comes from the convenience script. Go is unpacked into /usr/local, and the PATH is extended up front to include /root/nvke/cli/bin, where nvkectl will land later. kind, kubectl, Helm and Tilt each install with one line.

curl -fsSL https://get.docker.com | sh

curl -fsSL https://go.dev/dl/go1.27.3.linux-${ARCH}.tar.gz | tar -C /usr/local -xz
echo 'export PATH=/usr/local/go/bin:/root/go/bin:/root/nvke/cli/bin:$PATH' >> /root/.bashrc
source /root/.bashrc

curl -fsSLo /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-${ARCH} && chmod +x /usr/local/bin/kind
curl -fsSLo /usr/local/bin/kubectl "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/${ARCH}/kubectl" && chmod +x /usr/local/bin/kubectl
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4 | bash
curl -fsSL https://raw.githubusercontent.com/tilt-dev/tilt/master/scripts/install.sh | bash

Step 3: Verify versions and prepare the host

Three pieces of host prep keep the rest of the run quiet:

• Firewall. Accept traffic on the Docker and Kind bridges (docker0, br-+) so management and tenant containers can reach each other, then persist the rules.
• inotify limits. Tilt, Kind and many controllers watch files; the defaults run out quickly.
• Port check. Confirm nothing already holds the ports the stack needs: BGP (179), Postgres (5432), Temporal (7233), the NVKE API (8080/8081), Tilt (10350), and the NodePorts 18080, 30440 and 30443.

go version; kind version; kubectl version --client; helm version --short; tilt version; docker version --format '{{.Server.Version}}'

iptables -I INPUT -i br-+ -j ACCEPT
iptables -I INPUT -i docker0 -j ACCEPT
iptables -I FORWARD -i br-+ -j ACCEPT
iptables -I FORWARD -o br-+ -j ACCEPT
netfilter-persistent save

cat > /etc/sysctl.d/99-nvke-inotify.conf <<'EOF'
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 1048576
EOF
sysctl --system | grep inotify

ss -ltn | grep -E ':(179|5432|7233|8080|8081|10350|18080|30440|30443) ' || echo "all free"

Then clone the repo, bring up the host-side pieces, and build the CLI:

cd /root
git clone https://github.com/dsx-ai-factory/nvke.git
cd /root/nvke
make -C controller host-up # expect: /dev/vtpmx ready (or already present)
make -C cli build
nvkectl config view

Step 4: Stage the NICo Cluster API provider

nvkectl v0.3.0-73-gab0154a7 · cluster-api-provider-nico v0.1.0

With nvkectl built (and an empty site list, as expected), pull the CAPNICO provider module with Go and copy it into a writable directory. Tilt deploys its Helm chart from there.

CAPNICO_MODULE_DIR="$(go mod download -json github.com/dsx-ai-factory/cluster-api-provider-nico | jq -r .Dir)"
echo "$CAPNICO_MODULE_DIR"
mkdir /root/nvke-quickstart-capnico
cp -R "$CAPNICO_MODULE_DIR/." /root/nvke-quickstart-capnico/
chmod -R u+w /root/nvke-quickstart-capnico
ls /root/nvke-quickstart-capnico/chart/Chart.yaml

Step 5: Configure Tilt and create the management cluster

tilt_config.json picks which providers the dev control plane runs: Cluster API core, a BGP peer container, and NICo with a container-backed machine simulator. .env.tilt holds the required credentials; the quick start fills them with placeholder values.

cd /root/nvke
cat > controller/tilt_config.json <<'EOF'
{
"providers": ["capi_core", "bgp-containers", "nico"],
"nico_sim_backends": ["container"],
"capnico_source_dir": "../../nvke-quickstart-capnico"
}
EOF
cp controller/.env.tilt.example controller/.env.tilt
sed -i -e 's/^DGXC_TENANT_USER=$/DGXC_TENANT_USER=example-user/' \
-e 's/^\(DGXC_TENANT_PASSWORD\|CNPG_SUPERUSER_PASSWORD\|CNPG_CLOUD_PASSWORD\|CNPG_TEMPORAL_PASSWORD\)=$/\1=example-password/' \
controller/.env.tilt

make -C controller capd-up

Pick the Postgres password before the first tilt up. The comment in .env.tilt warns that the local Postgres chart is installed with CNPG_SUPERUSER_PASSWORD; changing it later means deleting the Postgres PVC. The placeholder values are fine for a local test only.

Step 6: Watch it build everything locally

This is the longest stretch of the run, about five minutes on this VM. capd-up builds from source what a production install would pull from a registry:

• An OVN-Kubernetes image, packaged into nvke-cni-ovn-charts and nvke-cni-ovn-images .debs (v2.4.2) for the node OS image.
• The nvke-agent and nvke-agent-updater binaries (v0.4.0-73-gab0154a7).
• A SPIRE agent bundle with the TPM attestor plugin (spire 1.14.1, spire-tpm-plugin 1.9.0).
• A custom Kind node image the CAPD machines boot from.

Once the images exist, it starts pull-through caches for Docker Hub and registry.k8s.io, so later clusters don't hit upstream registries again, and installs Cluster API core with the CAPD provider and cert-manager v1.21.1.

Step 7: Bring up the control plane with Tilt

cd /root/nvke/controller && tilt up --stream=true

Tilt loads the controller's Tiltfile and starts the BGP peer first: an FRR container on the Kind network (172.18.0.0/16, gateway 172.18.0.1) that kube-vip in each tenant cluster will peer with to advertise its control-plane VIP.

In a second terminal, a watch on Tilt's UI resources that hides anything already healthy gives a shrinking to-do list:

watch -n 10 'tilt get uiresources -o custom-columns=NAME:.metadata.name,RUNTIME:.status.runtimeStatus,UPDATE:.status.updateStatus | grep -vE "ok +ok|not_applicable +ok|ok +not_applicable"'

Over the next six minutes the list drains: CRDs for machine images, cluster releases and node pool releases are applied, SPIRE starts issuing X.509 SVIDs, and finally nvke-controller is built and deployed. When the watch shows an empty table, the stack is ready and the NVKE API is listening on :8080.

Step 8: Point nvkectl at the site and create a cluster

Export a kubeconfig for the management cluster, register it as a site called local, and check providers. NICo showing CONFIGURED with NO-IDENTITY is fine for the simulator; LaunchLayer isn't installed in this setup.

cd /root/nvke
kubectl config view --raw --minify --context kind-controller-capi > local-mgmt.kubeconfig
chmod 600 local-mgmt.kubeconfig
nvkectl config set-site --name local --url http://localhost:8080 --kubeconfig "$PWD/local-mgmt.kubeconfig"
nvkectl get providers # nico row: CONFIGURED (NO-IDENTITY is fine)

nvkectl create cluster -f cli/examples/nico-mock-cluster.yaml
nvkectl get clusters

Behind that one command the controller runs a ProvisionNode workflow per machine: it waits for the CAPI kubeconfig secret, for the SPIRE agent identity on the NICo machine (TPM attestation), runs node preflight, then joins and labels each node.

Step 9: Get into the tenant cluster

About four minutes after the create, pull the tenant kubeconfig and look around.

nvkectl get kubeconfig my-cluster --output-file my-cluster.kubeconfig
chmod 600 my-cluster.kubeconfig
ip route | grep 172.31 # the VIP route via the control-plane node
kubectl --kubeconfig my-cluster.kubeconfig get nodes -o wide
kubectl --kubeconfig my-cluster.kubeconfig get pods -A

Step 10: Scale the worker pool

Day-2 changes go through the same API. Patching the node pool's desired count adds a second worker:

nvkectl patch nodepool worker --cluster my-cluster --patch 'desiredNodeCount: 2'
kubectl --kubeconfig my-cluster.kubeconfig get nodes -w

The runbook also chains a fix-nodes helper after the patch. It wasn't on the PATH in this run (command not found), and scaling completed without it.

Step 11: Add a second cluster

Clusters are plain YAML, so a second one is a rename and a new VIP. diff shows exactly what changes: the cluster name, the control-plane VIP (172.31.2.1 → 172.31.2.2), and the template names.

sed -e 's/my-cluster/cluster2/g' -e 's/172\.31\.2\.1/172.31.2.2/g' cli/examples/nico-mock-cluster.yaml > cluster2.yaml
diff cli/examples/nico-mock-cluster.yaml cluster2.yaml
nvkectl create cluster -f cluster2.yaml
watch -n 15 nvkectl get clusters

Step 12: Tear it all down

cd /root/nvke
nvkectl delete cluster my-cluster
nvkectl delete cluster cluster2 # if you made one
watch -n 15 nvkectl get clusters # wait until empty

From engine to product

NVKE gives an operator a fast, well-built way to create Kubernetes clusters. Selling those clusters to enterprises takes another layer. In this run, every cluster came from one root user on the management host typing nvkectl. An operator’s customers need:

• A self-service console and API under the operator’s own brand
• Tenants, users and roles, so each customer sees only its own clusters
• Quotas and GPU allocation per tenant
• Usage metering sent to the operator’s billing system

Rafay is integrating NVKE, so operators can offer NVKE clusters as a governed, metered service alongside the rest of their GPU cloud.

“NVKE gives AI factory operators an open, NVIDIA-validated Kubernetes engine, and with Rafay, they can turn it into governed, self-service Kubernetes and GPU services, with one way to control access, manage quotas, and meter usage.” — Mohan Atreya, Chief Product Officer, Rafay

Takeaways

A working tenant cluster about 21 minutes after logging in to a bare VM, and most of that is one-time local image building. With images and mirrors warm, a new cluster reaches Provisioned in roughly three minutes and deletes in under two.

The developer surface is small. After setup, the whole lifecycle is five verbs: create cluster, get clusters, get kubeconfig, patch nodepool, delete cluster.

The hard parts aren't stubbed out. TPM-rooted SPIFFE identity, Temporal-driven provisioning with traceable workflows, OVN-Kubernetes, and BGP-advertised control-plane VIPs all run here, against simulated machines instead of real GPU servers.

Things to watch on your own run: make sure the listed ports are free and inotify limits are raised before tilt up, choose the Postgres password up front, and expect a few benign TLS handshake error ... EOF lines from spire-server in the Tilt log while health probes connect.