Inference is becoming the workload that defines AI infrastructure economics. Whether the infrastructure belongs to a neocloud, a sovereign AI cloud, or an enterprise AI platform, the measure is now the same: tokens delivered per GPU, at what latency, and at what cost.
The common approach is to run a single open-source inference engine on Kubernetes, with each model replica handling every request start to finish on the same GPUs. This aggregated serving works well for smaller models and moderate traffic. At scale, three limits appear.
The two phases of inference compete for the same GPU. Prefill reads the prompt and builds the key-value (KV) cache, which is compute-heavy. Decode generates tokens one at a time, which is limited by memory bandwidth. On a shared GPU, one long prompt can slow responses for every other user.
Capacity scales in one block. Adding replicas adds prefill and decode capacity together, even when only one is the bottleneck, so GPUs sit underused.
The engine sees one replica, not the fleet. It cannot tell which worker already holds a relevant cache, so shared prompts and conversation history are often recomputed.
Splitting prefill and decode onto separate GPU pools addresses the first two limits. This is disaggregated serving. The hard part is running it reliably across a fleet.
NVIDIA Dynamo, part of NVIDIA DSX AI Factory platform, is an open-source framework for running inference across a distributed GPU fleet. It sits above engines such as vLLM, SGLang, and NVIDIA TensorRT™-LLM and coordinates them into one inference system. Its core capabilities map directly to the limits above:
Disaggregated prefill and decode. Requests go to a prefill worker first, then continue on a decode worker, so each pool scales to its own bottleneck.
KV cache transfer through NIXL. State moves directly between GPUs over the fastest available path, such as NVIDIA NVLink™ or InfiniBand, without blocking work already in progress.
KV-aware routing. Requests go to workers that already hold relevant cache, so shared prompts and conversation history are reused instead of recomputed.
Adjustable worker ratios. The balance of prefill and decode workers can change as the traffic mix changes.
Dynamo also keeps aggregated serving as a first-class mode, so operators can start simple and disaggregate only where the workload justifies it.
One of Canada's leading telecommunications providers shows what this looks like in practice. The operator serves open models on its own GPU infrastructure through Rafay Token Factory, and as demand grew, it needed more throughput and steadier latency from the same GPUs.
Its original serving setup ran every request end to end on a single engine, and that approach was not keeping up. Its workloads mix long, prefill-heavy prompts with sustained high-volume generation, and handling both on the same GPUs held performance back. To serve both well, the operator adopted NVIDIA Dynamo for disaggregated serving.
The results have scaled quickly. Disaggregated prefill and decode went live in early August. Within weeks, throughput on the same GPUs rose nearly 90%, beating the operator's sign-off target by more than 125% while keeping time to the first token within target. Rebalancing the prefill-to-decode ratio for latency-sensitive traffic cut p90 time to first token by 81%.
Since the cluster expanded, the deployment has gone from hundreds of millions of tokens per hour at go-live to billions of tokens per hour today, an increase of more than 350%, with throughput per GPU up nearly 90%.
Rafay continues to work directly with NVIDIA's Dynamo team on model support and performance tuning as more of the operator's traffic moves over.
Dynamo coordinates GPUs. Turning a model into a service that tenants can consume and an operator can bill is the job of Rafay Token Factory.
In Token Factory, NVIDIA Dynamo is a selectable engine on any model deployment. Operators pick the backend framework, turn on disaggregated prefill where it helps, and size prefill and decode workers separately, while the commercial layer stays in place:
Because Token Factory is inference-engine agnostic, supporting vLLM, NVIDIA NIM™, and NVIDIA Dynamo, operators can adopt new engines as they mature without rebuilding billing, tenancy, and governance. NVIDIA NIM™, part of NVIDIA AI Enterprise, is a set of easy-to-use microservices designed for secure, reliable deployment of high-performance AI model inferencing across clouds, data centers, and workstations.
Token economics reward throughput per GPU, and throughput now depends on how well the serving layer coordinates GPUs, memory, and network. The pattern in this deployment applies well beyond telcos.
To see the configuration in detail, read Get Started with Rafay Token Factory and NVIDIA Dynamo, or book a demo to explore Token Factory on your own infrastructure.
From workload scheduling and inference to GPU fleet health, the Rafay Platform integrates with NVIDIA DSX OS across the layers operators run every day, and turns the reference stack into services operators can sell.
Read Now
Hosted AI agents raise two security questions: what the agent can do, and what the platform sees of its data. NVIDIA OpenShell, Protopia AI, and Rafay together let NeoClouds and sovereign AI providers sell governed Agent-as-a-Service and protected Tokens-as-a-Service on shared GPUs.
Read Now
.png)
Rafay has joined Lenovo AI Innovators. Here's why the AI control plane matters for hybrid AI, with context from Lenovo's Noam Rosen at our Barcelona summit.
Read Now
