10 Numbers I Can't Stop Thinking About After the Rafay AI Infrastructure Summit
I came back from two days in Barcelona with a notebook that was almost entirely numbers.
The summit had plenty of discussion about GPUs, networking, power, inference, and AI factories, but the numbers were what stayed with me, and Warren Barkley, VP of Product for DSX at NVIDIA, explained why in one answer. Asked what he would tell someone who thinks buying the GPUs means the hard part is done, he estimated that the hardware gets you only about 5% of the way there. The other 95% is turning it into infrastructure that can actually serve customers: software, people, operations, reliability, and services. Nearly every number I wrote down was a measurement of that 95%.
Andrew Leece of Sharon AI put the pace of that build-out in perspective. His first deal was for fewer than 500 GPUs, and less than a year later the order pipeline was measured in tens of thousands.
“The first deal that we did was for less than 500 GPUs… As we stand here today… we have 72,000 GPUs in our order pipeline.”
Growth at that speed changes what the infrastructure has to be. Processes that hold up for several hundred GPUs can collapse once demand moves into the tens of thousands, so automation, standardized architectures, multi-tenancy, and repeatable operations have to be considered early.
Here are 10 numbers I haven't stopped thinking about since.
1. 128 to 384 to 768 to 1,500 to 15,000 GPUs
This was the scaling path Karan Kirpalani, Chief Product Officer at Neysa, put on screen. It wasn't a projection of what a neocloud might someday do. It was how Neysa actually scaled its own infrastructure: 128 GPUs, then 384, 768, 1,500, and ultimately 15,000.
Karan's explanation mattered as much as the number. Each tranche of GPUs had a plan and a high-utilization commitment before the next tranche arrived. Neysa also designed its architecture to support different workload and pricing models without rebuilding the infrastructure each time.
The lesson: scale is something you earn through utilization, not simply something you purchase.
2. $270,000 per month for one idle rack
Simon Dumbleton, European CTO at World Wide Technology, put a dollar figure on something that often gets discussed as an engineering problem: delays.
WWT estimated that an idle GB300 NVL72 rack costs about $270,000 in forgone revenue every month. Scale that to 1,000 deployed-but-idle GPUs and the estimate rises to $3.8 million per month.
The model assumes $7.50 per GPU-hour and 70% utilization, using market rates as of August 2026, so it is an economic model rather than a universal benchmark. Still, every week between hardware arrival and the first productive GPU-hour has a cost.
Time-to-revenue is becoming an infrastructure KPI.
3. 1,000 GPUs installed. About 340 actually doing arithmetic.
This might have been my favorite visualization of the event.
Izhar Sharon, SVP of AI Customer Advocacy at DDN, showed why “GPU utilization” can be a dangerously imprecise term. His illustrative example starts with 1,000 GPUs installed, 900 sold, 855 allocated to running jobs, and roughly 340 with cores actually busy.
DDN is careful about the caveat: the example is illustrative, and the ~340 figure uses a published median from another GPU fleet. It isn't a claim about every operator.
In a published six-month trace covering 4,704 A100 GPUs and 1.09 million jobs, monitoring could report 97–99% utilization while median core activity was closer to 40%.
That's the difference between GPUs being allocated and GPUs doing useful work. Economically, that difference is enormous.
4. 4x more performance from the same hardware in about three months
Laura Morselli, Senior AI Solutions Architect at NVIDIA, showed how quickly software can change the economics of hardware you've already purchased.
On a specific GB200 NVL72 DeepSeek-R1 benchmark, NVIDIA showed roughly a 4x performance improvement between October 2025 and January 2026. Same generation of hardware, roughly three months apart.
NVIDIA's slide described the effect provocatively: it was like getting three additional GB200s, through software improvements alone, for the one you had already purchased.
That doesn't mean every workload suddenly gets 4x faster. But AI infrastructure capacity is no longer fixed on the day you buy the hardware.
Software can create capacity too.
5. 410 billion tokens a month, and an 8x swing in the bill
Neysa presented a high-volume AI workload processing more than 410 billion tokens per month.
What caught my attention was how the cost of that workload changed as it moved down the stack toward infrastructure optimized specifically for it. The comparison ranged from $77,250 to $9,000, roughly an 8x difference in cost across the delivery options shown.
At the final optimized step, Neysa also reported P99 latency falling from 14 seconds to under 2 seconds while reaching roughly 2,500 tokens per second. The engineering work included vLLM, tensor parallelism, KV-cache tuning, and time-to-first-token optimization.
This is where the phrase “AI factory” starts to make economic sense.
The question becomes: how much useful AI work can I produce, at what latency, and at what cost?
6. ~900 tokens to ~7,500 to ~40,000 to ~900,000 to millions
Noam Rosen, Director of Enterprise AI for Europe & META at Lenovo, showed one of the best illustrations of why agentic AI changes infrastructure demand.
His progression looked like this: simple chat, ~900 tokens; retrieval, ~7,500; agentic support, ~40,000; coding agent, ~900,000; deep autonomy, millions of tokens.
The user might still see a relatively simple outcome on screen. Behind it, the system may be reading, retrieving, reasoning, acting, checking the result, revising, and repeating.
Noam summarized the infrastructure impact as 10–100x more inference per task, along with higher concurrency and longer-running workloads.
That's why a declining cost per token doesn't necessarily mean declining infrastructure demand. If each task consumes orders of magnitude more tokens, cheaper inference may simply unlock much more inference.
7. 5,000 GPUs could represent $250M+ in additional annual value
David Wood of Accenture made a broader point about neocloud economics: the competitive advantage is shifting from who owns the most compute to who can generate the most revenue and margin from every GPU.
One slide made that argument concrete. Accenture showed how pooling, oversubscription, and optimization can reduce wasted capacity, improve average utilization, sell more logical demand on the same physical infrastructure, increase throughput, and lower cost per token. The example on the slide: 5,000 GPUs could represent $250M+ in additional value annually.
The exact outcome will depend on pricing, workload mix, utilization, and the services layered on top. The point is the direction of the economics: software changes what the same physical GPU fleet can earn.
That is why moving from bare metal to managed infrastructure, inference and token factories, AI platforms, and ultimately enterprise outcomes is a monetization strategy as much as a product strategy.
The infrastructure may be the same. What you can sell on top of it is not.
8. $300B+ waiting, and as much as 5–10 years for power
Warren Barkley showed a slide that put the power constraint in perspective: $300B+ in equipment backlogs tied to AI infrastructure projects waiting on power and supply-chain capacity, 200 GW in the U.S. grid interconnection queue, and, in some cases, a 5–10 year standard utility-upgrade window before new AI-factory power can come online.
For the last few years, the AI infrastructure conversation has centered on GPU scarcity. The binding constraint is increasingly power.
And if new megawatts take years to obtain, extracting more useful compute from the megawatts you already have becomes far more valuable.
9. Only 50% of provisioned power utilized, and potentially 40% more GPUs per MW
The next NVIDIA number took that idea one step further. Warren's presentation said that in the environments NVIDIA described, only about 50% of provisioned power is actually used.
NVIDIA's MaxLPS approach aims to recover that stranded power through dynamic allocation, with the slide describing the potential for up to 40% more GPUs per megawatt.
A second slide put numbers on it. Under static allocation, a 540 kW power budget consumed only 370 kW, leaving 170 kW stranded. With dynamic allocation, the same 540 kW budget supported 470 kW of actual consumption and one additional rack.
That changes capacity planning: what matters is less the number of megawatts you have than how much compute you can safely extract from each one.
10. 300,000 users could mean 35–40 trillion tokens a month, and 20–30 MW
Farul Mohd Ghazali of Aras Integrasi connected sovereign AI demand all the way back to the physical infrastructure required to support it.
He started with actual August usage data from public-sector users. Developers running multiple agents were consuming 2–6 billion tokens per user per month. Office power users were consuming 200–500 million.
Then he modeled what broader adoption might look like. A planning scenario covering 300,000 users, half of a 600,000-person addressable market, produced an estimate of approximately 35–40 trillion tokens per month, which Farul translated into roughly 20–30 MW of infrastructure. He explicitly labeled this a planning scenario based on production consumption ranges, not a measured future outcome.
Users → agents → tokens → GPUs → megawatts.
AI demand ultimately becomes an infrastructure problem.
What all 10 numbers told me
I went to Barcelona thinking about AI infrastructure mostly in terms of capacity. I left thinking about yield.
How quickly can a GPU become productive? How much arithmetic is it actually doing? How many tokens can it generate? How much useful work can software extract from the same hardware? How many GPUs can fit inside the same power envelope? And, ultimately, how much revenue can that infrastructure produce?
The first phase of the AI infrastructure boom was about getting GPUs. Then came the competition for power and capital.
The next phase looks increasingly like a race to turn those scarce resources into the greatest possible amount of useful AI work, and then turn that work into services customers will pay for.
Metal → tokens → services → revenue.
That's the shift I heard again and again in Barcelona.
