Post-Kubernetes Infrastructure for GenAI Workloads

Last week Modal published Scaling to 1 million concurrent sandboxes in seconds. The headline numbers: a million concurrent sandboxes, all created in under a minute, sub-500ms median start times, scheduling latency in the tens of milliseconds. The numbers are impressive. The architecture notes underneath them are the interesting part.

What they actually did

Modal rebuilt their sandbox platform around a single design index: launch speed. Their stated rule is that everything taking O(sandboxes) or O(nodes) load must be horizontally scalable by default, the creation path should be as simple as possible, and everything else is secondary.

Follow that rule honestly and you end up somewhere radical. They removed central coordination entirely: a horizontally-scaled fleet of scheduling servers doing in-memory load balancing, proposing placements that workers accept or reject based on their own local resource state. Workers are their own source of truth and publish state asynchronously to streams. Nothing durable gets written on the creation path at all. Global consistency is traded away, on purpose, and they say so plainly.

What I respect most about the post is that it isn't Kubernetes bashing. It's a precise deep dive into the assumptions Kubernetes is built on. Scheduling that is O(nodes × pods) and serialised by default. A central store (etcd) that takes O(containers) writes across pod lifecycles plus O(nodes) heartbeats, and is not natively shardable within a keyspace. None of these are bugs. They are the price of a strongly consistent view of the cluster - a price that made sense for the workloads Kubernetes grew up with: long-lived services, deployments that change a few times a day, reconciliation loops that can take their time because the workload lives for weeks.

The signal

GenAI workloads invert those assumptions. Agent sandboxes, eval runs, code-execution bursts - massive fan-out of ephemeral, short-lived compute where creation latency is the product. A container that lives for forty seconds cannot amortise a scheduling path designed for a service that lives for forty days.

To my knowledge, this is the first credible signal that Kubernetes is not adapting fast enough to what GenAI infrastructure needs. Modal didn't extend it. They didn't write an operator, or a custom scheduler, or lean on any of the extension points the ecosystem usually reaches for. They looked at the assumptions, concluded the assumptions were the problem, and walked around the whole thing.

If that pattern holds, my speculation is that the eventual winner for this class of workload looks less like Kubernetes and more like an HPC system with high levels of asynchronous orchestration - a mutation of the batch-scheduler lineage and the Kubernetes lineage. Throughput-first scheduling, weakly consistent by default, horizontally scalable in every path, comfortable with rejection and retry where Kubernetes insists on global agreement.

The caveat

It is very early to call any of this. And it's worth being honest about what shaped Modal's constraints: sandboxes are security permission boundaries. Their unit of compute is hard isolation around untrusted code. When that's your product, you index on creation speed and blast radius, and you can afford eventual consistency because sandboxes share almost nothing with each other. Not everyone will index towards those constraints. If your workload is trusted, long-lived, or genuinely needs a consistent global view, the Kubernetes price is still buying you something real.

Where we sit

In Sympozium we think about this as a coming decoupling of coordination from execution. Multi-agent workflows need, in places, the opposite of perfect isolation: overlapping security boundaries, shared memory, agents that can sense each other's state. That coordination plane wants strong semantics, governance, and provenance - the things Kubernetes-shaped systems are actually good at. The execution plane underneath it wants exactly what Modal built: isolation boundaries that appear in milliseconds and disappear without ceremony. Those are different layers with different physics, and I don't think one system wins both.

Which is the real takeaway. The stack for GenAI infrastructure is not settled - not the execution layer, not the coordination layer, not the seam between them. Modal just showed one very sharp answer to one layer of it. Right now it is anyone's game to win, and anyone's game to define.