NVIDIA Dynamo in Practice: Scaling Disaggregated Inference on Rafay Token Factory

October 29, 2026

Inference is becoming the workload that defines AI infrastructure economics. Whether the infrastructure belongs to a neocloud, a sovereign AI cloud, or an enterprise AI platform, the measure is now the same: tokens delivered per GPU, at what latency, and at what cost.

The limits of single-engine serving

The common approach is to run a single open-source inference engine on Kubernetes, with each model replica handling every request start to finish on the same GPUs. This aggregated serving works well for smaller models and moderate traffic. At scale, three limits appear.

The two phases of inference compete for the same GPU. Prefill reads the prompt and builds the key-value (KV) cache, which is compute-heavy. Decode generates tokens one at a time, which is limited by memory bandwidth. On a shared GPU, one long prompt can slow responses for every other user.

Capacity scales in one block. Adding replicas adds prefill and decode capacity together, even when only one is the bottleneck, so GPUs sit underused.

The engine sees one replica, not the fleet. It cannot tell which worker already holds a relevant cache, so shared prompts and conversation history are often recomputed.

Splitting prefill and decode onto separate GPU pools addresses the first two limits. This is disaggregated serving. The hard part is running it reliably across a fleet.

Single-engine serving runs prefill and decode on the same GPUs. With NVIDIA Dynamo, prefill and decode workers each get their own GPU pool, and NIXL moves the KV cache between them.

What NVIDIA Dynamo is

NVIDIA Dynamo, part of NVIDIA DSX AI Factory platform, is an open-source framework for running inference across a distributed GPU fleet. It sits above engines such as vLLM, SGLang, and NVIDIA TensorRT™-LLM and coordinates them into one inference system. Its core capabilities map directly to the limits above:

Disaggregated prefill and decode. Requests go to a prefill worker first, then continue on a decode worker, so each pool scales to its own bottleneck.

KV cache transfer through NIXL. State moves directly between GPUs over the fastest available path, such as NVIDIA NVLink™ or InfiniBand, without blocking work already in progress.

KV-aware routing. Requests go to workers that already hold relevant cache, so shared prompts and conversation history are reused instead of recomputed.

Adjustable worker ratios. The balance of prefill and decode workers can change as the traffic mix changes.

Dynamo also keeps aggregated serving as a first-class mode, so operators can start simple and disaggregate only where the workload justifies it.

In practice: a leading Canadian telco

One of Canada's leading telecommunications providers shows what this looks like in practice. The operator serves open models on its own GPU infrastructure through Rafay Token Factory, and as demand grew, it needed more throughput and steadier latency from the same GPUs.

Its original serving setup ran every request end to end on a single engine, and that approach was not keeping up. Its workloads mix long, prefill-heavy prompts with sustained high-volume generation, and handling both on the same GPUs held performance back. To serve both well, the operator adopted NVIDIA Dynamo for disaggregated serving.

The results have scaled quickly. Disaggregated prefill and decode went live in early August. Within weeks, throughput on the same GPUs rose nearly 90%, beating the operator's sign-off target by more than 125% while keeping time to the first token within target. Rebalancing the prefill-to-decode ratio for latency-sensitive traffic cut p90 time to first token by 81%.

Since the cluster expanded, the deployment has gone from hundreds of millions of tokens per hour at go-live to billions of tokens per hour today, an increase of more than 350%, with throughput per GPU up nearly 90%.

Rafay continues to work directly with NVIDIA's Dynamo team on model support and performance tuning as more of the operator's traffic moves over.

How Rafay Token Factory turns Dynamo into a service

Dynamo coordinates GPUs. Turning a model into a service that tenants can consume and an operator can bill is the job of Rafay Token Factory.

In Token Factory, NVIDIA Dynamo is a selectable engine on any model deployment. Operators pick the backend framework, turn on disaggregated prefill where it helps, and size prefill and decode workers separately, while the commercial layer stays in place:

  • Input and output token pricing set per deployment
  • Rate limits and explicit sharing with specific tenant organizations
  • Self-service access for tenant users through API keys and an integrated playground
  • Token usage reported per tenant, split by input and output

Because Token Factory is inference-engine agnostic, supporting vLLM, NVIDIA NIM™, and NVIDIA Dynamo, operators can adopt new engines as they mature without rebuilding billing, tenancy, and governance. NVIDIA NIM™, part of NVIDIA AI Enterprise, is a set of easy-to-use microservices designed for secure, reliable deployment of high-performance AI model inferencing across clouds, data centers, and workstations.

Why it matters, whatever you run

Token economics reward throughput per GPU, and throughput now depends on how well the serving layer coordinates GPUs, memory, and network. The pattern in this deployment applies well beyond telcos.

To see the configuration in detail, read Get Started with Rafay Token Factory and NVIDIA Dynamo, or book a demo to explore Token Factory on your own infrastructure.

Share this post

Want a deeper dive in the Rafay Platform?

Book time with an expert.

Book a demo

You might be also be interested in...

Self-Service Bare Metal Is Where a GPU Cloud Starts, Not Where It Ends

Read Now

Product

Rafay is bringing NVIDIA Kubernetes Engine (NVKE) into its Kubernetes-as-a-Service (K8s) offering

A working tenant cluster about 21 minutes after logging in to a bare VM. Each new cluster after that in about three minutes. The whole lifecycle in five nvkectl verbs.

Read Now

Product

NVIDIA AI Cloud-Ready at Rack Scale: Rafay on GB200 NVL72 with NVIDIA Infra Controller

The rack is becoming the unit of AI compute. Rafay is extending its NVIDIA AI Cloud-Ready validation from single systems to full GB200 NVL72 racks with NVIDIA Infra Controller.

Read Now