A joint post by Rafay and Clockwork.io
At Scale, AI training jobs fail every few hours, and each failure means restarting from a checkpoint and burning GPU-hours someone must keep paying for. Rafay and Clockwork.io have partnered so platform teams can make fault tolerance a standard part of every GPU cluster they run: Clockwork's AI Fault Tolerance and AI Fabric Observability products are now a turnkey deployment option on the Rafay Platform, turned on once through a Rafay blueprint and enforced across the fleet. Engineering teams from both companies built and tested the integration, packaging Clockwork as Rafay add-ons and cluster blueprints. If you already use Rafay to roll out Kubernetes-based GPU clusters, including Slurm-as-a-Service on Kubernetes, you can now deploy Clockwork as part of the same workflow, with no separate installation project.
Large AI training and inference jobs run thousands of GPUs in lock-step. A single GPU error, link flap or node failure can crash an entire job. Recovery usually means reprovisioning nodes, restarting from the last checkpoint and recomputing the lost work. Worse, many faults are gray failures: they pass health checks and dashboards stay green while GPU-hours are wasted.
A Meta study of its own research clusters found that the mean time to failure for 1,024-GPU jobs is 7.9 hours, and projects 1.8 hours for 16,384-GPU jobs.
That cost lands on both platform teams and AI developers. For platform teams delivering GPU infrastructure, those failures turn into lost tenant capacity, broken SLAs and long troubleshooting sessions. For tenants running on those clusters, the same failures mean wasted GPU-hours they already paid for, lost productivity and slower time to market.
Clockwork AI Fault Tolerance keeps jobs alive when the infrastructure underneath them fails. LinkPass handles the network: when a link flaps or a NIC fails, it detects the failure as it occurs and shifts traffic to other healthy NICs. It is delivered as a standard NCCL net plugin.
TorchPass handles the compute: when a GPU dies or a node fails, it captures the affected worker's state, live-migrates it to a replacement over RDMA and resumes from the exact iteration with no lost compute. TorchPass also takes multi-node platform snapshots of a distributed job in under 20 seconds, with zero model code changes. Together they mean GPU errors, link flaps and node failures no longer take down AI workloads.
When a developer or data scientist reports a slow job, the first question is always the same. Is it the fabric?
FleetLens, part of Clockwork AI Fabric Observability, detects slow or failing jobs and quickly rules the fabric in or out. When the fabric is to blame, it pinpoints the exact link, switch or NIC, and surfaces problems that conventional monitoring misses, including micro-congestion, topology-specific degradation and gray failures.
Clockwork keeps jobs running; Rafay makes that protection standard across the entire fleet. Rafay cluster blueprints are how platform teams define, version and enforce the add-ons and configuration every cluster in a fleet should carry. Clockwork's components are packaged as Rafay add-ons from Clockwork's Helm charts and bundled into a blueprint. When an operator applies that blueprint to a cluster, or updates a cluster to a new blueprint version, the Rafay Platform deploys the Clockwork components, tracks their status alongside every other add-on, and enforces the configuration across the fleet with drift detection and version pinning. Because blueprints are versioned, a Clockwork upgrade rolls out as a new blueprint version across the fleet, with no per-cluster work. Platform teams turn it on once, and every tenant cluster that carries the blueprint gets it. For GPU cloud providers, that turns a fault-tolerant tier into something you can offer every tenant, not a custom engineering project.
In joint testing, the teams deployed the Clockwork solutions to a GPU cluster through a blueprint, then ran a training job with simulated NIC failures. The job kept running through both failover episodes, and the Clockwork dashboard in Grafana showed the failover timeline and the idle GPU-hours prevented.
Figure 1. A “Kubernetes with Clockwork Cluster Resilience" Blueprint containing Clockwork’s Linkpass Plugin and Linkpass Monitor, deployed onto a cluster.
Figure 2. Two all-reduce jobs are started at the same time, one with and one without LinkPass. Only the job with LinkPass survives a NIC failure.
At scale, the next failure is usually hours away. With Rafay, fault tolerance becomes part of the blueprint every cluster in the fleet carries, and Clockwork keeps the jobs on those clusters running through failures.
Clockwork technology is already proven at scale:
The same software now deploys on your clusters through Rafay, so you can offer a fault-tolerant tier to customers.
Ready to enable fault tolerance on your AI clusters?
Request a Rafay + Clockwork.io demo
Already a Rafay customer? Talk to your account team to turn it on.
Want the technical details first? Explore Rafay cluster blueprints and Clockwork AI Fault Tolerance.
Learn how NeoClouds can use NVIDIA Cluster Readiness Engine to certify GPU Kubernetes clusters before tenant handoff and validate AI workload readiness.
Read Now
The largest infrastructure build-out of our era is underway, and it is sorting the operators who simply own compute from the ones building lasting businesses on top of it. Rafay's AI Infrastructure Leadership Summit brings those operators into one room for two days on the journey from GPU to token: where margin actually comes from, how the best move from selling GPU hours to selling outcomes, and what it takes to build an AI business that lasts.
Read Now
Compare Nebius Token Factory vs Rafay to decide whether to buy managed AI inference or build a token-metered inference business on your GPU fleet.
Read Now
