Principal Software Engineer, AI Inference Platform - U.S.
About Rafay
Rafay builds the platform layer that turns GPU infrastructure into consumable, metered AI services for cloud providers, telcos, and enterprises. Our customers use Rafay to stand up inference and GPU compute as a product of their own: multi-tenant, metered, and in production for their customers.
The market moved here fast. Organizations that bought GPU capacity are now under pressure to turn it into revenue, and doing that takes metering, multi-tenancy, and inference that performs well enough to sell. That gap is the business Rafay is in, and it is where the engineering in this role sits.
The role
As a Principal Engineer you are a technical leader and a hands-on architect. You will set direction for some of the most critical services on the platform and multiply the impact of the engineering teams around you. We expect you to be in the code and setting the standard at the same time.
What you'll own
- Architecture of core platform services. The multi-tenant, distributed control plane that the rest of the product is built on.
- The control plane for GPU workloads. Scheduling, placement, and lifecycle across heterogeneous GPU fleets and multiple clouds. Placement decisions here are the difference between a viable unit economic and an unviable one.
- The serving path. Inference systems that hold their latency and throughput targets under multi-tenant load, while keeping expensive hardware busy.
- Multi-tenancy that holds up commercially. Isolation, quotas, fairness, and noisy-neighbor behavior, in an environment where the tenants are our customers' paying customers.
- Technical direction across teams. Architectural vision for major platform areas, and the alignment work that makes it stick.
- The engineering bar. Design and code review, standards, and mentoring senior and junior engineers alike.
- Ongoing R&D. This stack moves monthly. Working out what is real and what is hype is part of the job.
- Production reality. Partnering with solutions and support teams on the stability of live systems, and treating a customer-found issue as a first-class engineering input.
What we're looking for
- 12+ years building and delivering large-scale production systems
- Deep distributed systems fundamentals. Consistency, availability, scalability, and a real instinct for how these systems fail. This is the core of the role
- Expert-level Go and/or Python
- Experience designing and running microservices
- Strong understanding of multi-tenant isolation: per-tenant quotas, noisy-neighbor mitigation, and data, workload, and network isolation boundaries
- Production experience with containers and Kubernetes
- Working familiarity with GPU infrastructure and modern LLM serving. Enough to reason about the cost, latency, and throughput tradeoffs, even if you have not specialized in it
- Strong troubleshooting and debugging instincts, and a track record of shipping fixes for customer-found issues
- Demonstrated ability to influence technical direction and build consensus across teams
Strong pluses
- Depth in LLM inference performance: continuous batching, KV cache management, prefill vs. decode behavior, quantization, tensor and pipeline parallelism. If this is your specialty, we want to talk to you
- Hands-on GPU scheduling and sharing: NVIDIA GPU Operator, MIG, time-slicing, multi-tenant GPU allocation
- Familiarity with high-performance GPU interconnects and topology-aware placement
- Experience operating a platform as a product for external customers, not only internal ones
Why join Rafay
Rafay is a named NVIDIA AI Grid control-plane partner, and the platform runs on top of the leading inference engines, including vLLM, NVIDIA NIM, and NVIDIA Dynamo, for customers building AI businesses on it.
We work in a collaborative environment that rewards creative thinking, and we hold high expectations of each other and of the systems we build. On top of that we offer a competitive salary, robust benefits, and attractive stock options.
