How AI workloads are reshaping Kubernetes autoscaling

Kubernetes has quietly run much of the world’s containerized software on a division of labor between two control loops that rarely had to talk to each other. The scheduler places each pod on a node that already exists; the autoscaler adds nodes when pods have nowhere to fit. For a decade that separation was a sound design, because compute was largely interchangeable and either loop could work without knowing much about the other’s decisions. AI workloads, and the accelerators they run on, changed the terms; the clean split still serves most work well, and the pace of AI is what makes its edges worth revisiting.

It surfaces in a situation anyone running GPUs will recognize. A pod that should schedule simply will not, even though the cluster is not truly out of capacity, and the delay can leave costly accelerator capacity unused. The instinct is to assume the region has run dry, and sometimes it has. In other cases, the capacity is available and the pod still stays pending, because the node the autoscaler is willing to add is not one the scheduler is willing to use. The autoscaler answers the question it was built for, whether to add a node, when the real question is whether that node is even the right shape for the work. The rest of this piece examines how the community has closed part of it, how it plans to close the rest, and how the autoscaler is likely to grow into a more scheduler-aware partner.

The hard problems, and how the community is answering them

Several of the challenges accelerators bring share a thread. Each is something both loops now have to understand the same way. Not all are new or unique to AI; some were long familiar to high-performance and batch computing, and accelerators made them everyone’s problem. Three show how far the shared answers have come.

Reservations, capacity committed before the work arrives

This one is the most familiar. Because supply is tight and idle accelerators are expensive, teams may reserve capacity ahead of demand. When a scale-up draws from a reservation, the autoscaler must consider not only whether to grow but also whether the resulting node uses capacity the team has already committed to and is paying for. Reservations are not an autoscaling invention. They are a long-standing capacity practice that an autoscaler may need to account for, because a node can be launched against committed capacity rather than relying on unreserved supply.

Topology, the interconnect between devices, not just their count

Topology is about how devices are wired to one another, not simply how far apart they sit. An accelerator leans on the parts that feed it, including the CPU and memory in its local NUMA domain, the network interface that shares its PCIe path, and the high-bandwidth links to its peer accelerators; it runs best when those pieces share a locality so traffic does not cross a slower hop. That locality is a hierarchy, from devices on the same bus inside a server, to servers in a rack joined by a fast accelerator fabric, to racks linked across a data-center network, each tier a step slower than the one beneath it. The largest models span many servers at once, so “keep these close” reaches well beyond a single node. Historically, Kubernetes commonly represented accelerator resources as counts and lacked a portable, structured way to express many device relationships; a node advertised a number, a pod asked for some, and the scheduler matched the two integers. Dynamic Resource Allocation supplies what was missing. Instead of a bare count, a workload files a structured claim describing what it needs, including a relationship like an accelerator together with the network interface that feeds it. The scheduler reads those claims directly, and DRA-aware autoscalers are starting to use the same information when they simulate capacity, though support still varies by implementation.

The cost of a blind boundary. A DRA-aware autoscaler can account for a topology claim when selecting capacity; without that support it may provision an incompatible node.

Sharing, putting a whole device to work, not part of it

A modern accelerator is powerful enough that handing an entire one to a small job wastes most of it, so a single device can be partitioned into isolated instances that run concurrently. Some runtimes also time-slice devices, but that is a separate execution mechanism and not what consumable capacity models. Partitionable devices let one device be offered as smaller slices on demand, and consumable capacity goes finer, letting several workloads each draw a measured portion, so a request can be satisfied by a fraction of a device rather than requiring an entire device. The scheduler tracks those allocations directly, and DRA-aware autoscalers are starting to use the same model when they weigh a prospective node, though support still varies. Finer slicing can leave leftover fragments, but sensible minimum and step sizes help keep that smaller than the all-or-nothing waste it replaced.

Efficiency moved inside the device. Partitioning splits one device into isolated fixed instances; consumable capacity lets several claims draw measured shares at once.

Topology and sharing matter most for what they share. Together, DRA and its extensions give the scheduler and the autoscaler the same language for hardware, so the two are far less likely to disagree about what a viable node even looks like. That shared vocabulary is real progress, and it is also where the next problem begins.

Why a shared vocabulary is not enough

A common language lets the two loops agree on what a node is; it does not make them act as one, and some problems need exactly that. The clearest is gang scheduling, an all-or-nothing case that is not new but has grown far more acute under AI and tight capacity. A large job needs every pod to be scheduled together or none at all, and this is where shared facts alone fall short.

This part is being addressed, but only on one side. Through workload-aware scheduling, whose Workload and PodGroup APIs are beta upstream and remain disabled by default, the scheduler is learning to treat a group as one all-or-nothing unit, holding its pods until enough can be placed together so a half-placed job no longer strands hardware. The autoscaler was not part of that change. It still reasons pod by pod, so it can add a node or two for a few pending members of a gang and waste that capacity while the job never starts.

A companion mechanism, the provisioning request, is the current bridge. It lets a workload ask the autoscaler for a whole group’s capacity at once rather than pod by pod. With a configured admission check, Kueue reserves quota, creates the request, waits for capacity to be reported ready, and then admits the workload. Atomic scale-up is best effort and depends on provider support. It remains a bolt-on across a seam rather than a shared understanding, and it points at the deeper pattern. A capable autoscaler already reasons a little like a scheduler, predicting where the scheduler would place pods to decide whether a new node would help. The clearest sign is duplicated effort already baked in. The autoscaler simulates the scheduler’s placement to decide what to launch, and once the node exists the scheduler makes its own placement decision, repeating much of the same reasoning.

What the community is building next

This is why the two loops, which still do different jobs, increasingly have to be designed together, and that convergence is becoming an active design discussion. The workload-aware scheduling effort has signaled that its scope is meant to reach beyond the scheduler toward autoscaling over time, and a public proposal in the SIG-Autoscaling community now sketches integration APIs that let the two speak directly. One would let the scheduler see capacity that could be provisioned rather than only the nodes already present, one would express placement preferences across it, and one would request it explicitly. The deeper motivation is process. If autoscaling is a first-class concern while a feature is designed, provisioning gets considered while a new placement primitive is taking shape rather than only after it ships, so the seam becomes something the community weighs up front rather than something autoscalers scramble to close after each release.

What to take away

The mental model worth keeping is one hardware model, two consumers. A scarce accelerator is only as useful as the agreement between the loop that schedules it and the loop that provisions it, and every place only one of them understands a capability is a place they can disagree and may leave costly hardware idle. So when a pod stays pending, do not assume the capacity is gone; check whether the scheduler and the autoscaler agree on what a usable node looks like. Inference is what makes this pressing rather than academic, since bursty serving and agentic workloads have no patience for a cold node. These interfaces are being drawn in the open right now, in SIG-Autoscaling and SIG-Scheduling rather than in any one vendor’s product, so anyone running accelerators has both a stake in the outcome and a chance to shape it before the design sets.