Skip to content

Autoscaling

Three controllers can each resize a running workload on their own initiative, and none of them talk to each other directly. Knowing which one changes what — and where two of them can quietly fight over the same number — is what turns "just add capacity" from a guess into a diagnosis.

Scope: scheduling picked where a Pod lands given today's Nodes and replica count. This page covers the three controllers that change those numbers over time — more replicas, right-sized requests, or more Nodes — and how they interact (and occasionally collide).

TL;DR — in 30 seconds:

  • Three separate controllers, none talking to each other directly: HPA changes replica count (reacts to metrics), VPA changes a container's requests/limits (reacts to usage history), Cluster Autoscaler changes Node count (reacts to Pending Pods / underused Nodes).
  • HPA needs a resource request set on the target — CPU utilization is measured as a percentage of the request, so an unset request makes the percentage meaningless.
  • HPA and VPA rarely mix on the same metric (they can chase each other's changes) — the safe pairing is VPA on memory while HPA drives replica count off CPU or a custom metric.

1. The three autoscalers, at a glance

Autoscaler Changes Reacts to Typical use
HPA (Horizontal Pod Autoscaler) replica count of a Deployment/StatefulSet metrics (CPU, memory, or a custom/external metric) stateless services under variable load
VPA (Vertical Pod Autoscaler) a container's requests/limits historical usage right-sizing workloads whose per-instance footprint is hard to guess up front
Cluster Autoscaler the number of Nodes unschedulable ("Pending") Pods, and underused Nodes making sure the cluster has enough capacity for what HPA/VPA ask for — and shrinking it back down when idle

The mental model: HPA and VPA operate on Pods, deciding how many there should be and how big each one is. The Cluster Autoscaler operates one layer down, on Nodes, making sure there's somewhere for those Pods to actually run.

2. Horizontal Pod Autoscaler (HPA)

The HPA controller runs a loop: read a target metric, compare it to a target value, adjust replicas.

  • The classic case is CPU utilization: target 50% average CPU across the Deployment's Pods. If real usage runs hotter, HPA scales replicas up; if it runs cooler, it scales back down.
  • HPA can also track memory, or a custom metric (something your app exports, e.g. queue depth or requests-per-second) via the metrics pipeline, and an external metric (something outside the cluster, e.g. a cloud queue's backlog).
  • Scaling isn't instant on either edge: HPA re-checks on an interval and applies stabilization — scale-up reacts quickly, but scale-down is deliberately slower, so a brief dip in load doesn't thrash replica count up and down.

Note. HPA requires the target Deployment's Pods to declare a resource request for whatever metric it targets — CPU utilization is measured as a percentage of the request, so an unset request makes the percentage meaningless. (Go deeper: the upstream Horizontal Pod Autoscaling walkthrough — optional, not required to defend this.)

3. Vertical Pod Autoscaler (VPA)

Where HPA asks "how many copies," VPA asks "how big should one copy be." It watches a workload's actual CPU/memory usage over time and recommends — or, in Auto mode, applies — new requests/limits.

Three modes, increasing in how much VPA does on its own:

  • Off — only produces recommendations; nothing is applied automatically. Useful for sizing a workload before committing to fixed requests.
  • Initial — sets requests only when a Pod is created; existing running Pods are left alone.
  • Auto — updates requests on running workloads too, which today means evicting and recreating the Pod with the new values (there is no in-place resize in the common case), so it causes a brief disruption.

Warning — HPA and VPA rarely mix on the same metric. If VPA is resizing a container's CPU request while HPA is also watching CPU utilization of that same request, they can chase each other's changes. The common safe pairing is VPA on memory (which HPA usually isn't scaling on) while HPA drives replica count off CPU or a custom metric. (Go deeper: the upstream VPA project — optional, not required to defend this.)

4. Cluster Autoscaler

HPA can decide a Deployment needs five more replicas, but if no Node has room, those new Pods sit Pending — a scheduling filtering failure. The Cluster Autoscaler watches for exactly that signal and reacts by adding capacity:

  • Scale up — a Pod is unschedulable because no Node satisfies its requested resources (or its affinity/ taint rules) → the Cluster Autoscaler provisions a new Node (in EKS, typically by growing a managed node group / Auto Scaling Group) and the Pod schedules onto it once it joins.
  • Scale down — a Node is significantly underutilized and everything on it could safely reschedule elsewhere → the Cluster Autoscaler cordons and drains it, then removes it. A Node running something that can't safely move (a Pod outside a controller, one that ignores PodDisruptionBudget) is left alone. (Go deeper: the upstream Cluster Autoscaler project — optional, not required to defend this.)
flowchart TB
  hpa["HPA<br/>watches metrics"] -->|"raises replicas"| deploy["Deployment"]
  vpa["VPA<br/>watches usage history"] -->|"resizes requests"| deploy
  deploy -->|"more/bigger Pods"| sched["Scheduler"]
  sched -->|"fits"| node["Existing Nodes"]
  sched -->|"no Node fits"| pending["Pod: Pending"]
  pending -->|"triggers"| ca["Cluster Autoscaler"]
  ca -->|"adds"| newnode["New Node joins"]
  newnode --> sched

5. How they interact

Read top to bottom: HPA and VPA push demand up by asking for more or bigger Pods; the scheduler tries to place that demand on today's Nodes; the Cluster Autoscaler is the layer that expands the pool of Nodes when that demand can't be placed, and shrinks it back when demand falls. None of the three talks to the others directly — they're each closing their own observe-vs-desired gap, and they compose through the Pending signal and real utilization, not through direct coordination.

6. How this connects

  • Pending is the same state the observability/debugging page teaches you to read with get→describe — a Pod stuck Pending because of insufficient requested capacity is the exact signal that triggers Cluster Autoscaler scale-up.
  • Autoscaling only has Nodes to add because something already provisions the node group underneath — the AWS module's EC2 Auto Scaling / EKS managed node groups.
  • QoS classes (the scheduling & resources page) still apply during scale-down: the Cluster Autoscaler won't remove a Node if doing so would violate a PodDisruptionBudget, regardless of QoS.

Common gotchas

  • Running HPA and VPA on the same metric (e.g. both watching CPU) for the same workload. Fix: they can chase each other's changes — pair VPA on memory (which HPA usually isn't scaling on) with HPA driving replicas off CPU or a custom metric instead.
  • Enabling HPA on CPU utilization without setting a resource request. Fix: CPU utilization is a percentage of the request — with no request set, the percentage is meaningless and HPA has nothing sensible to scale against.
  • Expecting VPA Auto mode to resize a running Pod without disruption. Fix: today that generally means evicting and recreating the Pod with new values, not an in-place resize — expect a brief disruption, and consider Initial mode if that's not acceptable.
  • Assuming new replicas from HPA will always find somewhere to run. Fix: if no Node has room, the new Pods just sit Pending — that's a scheduling filtering failure, and it's the exact signal the Cluster Autoscaler needs to add a Node; HPA scaling up isn't the same as capacity actually existing.
Check yourself — HPA scales a Deployment from 3 to 10 replicas, but the new Pods stay Pending. What's happening, and which controller reacts to it?

HPA only changes the desired replica count — it doesn't guarantee the cluster has room. If no existing Node has enough free requested capacity, the new Pods are unschedulable (Pending), which is the exact signal the Cluster Autoscaler watches for to provision a new Node.

Check yourself — why is running VPA in Auto mode on CPU while HPA also scales on CPU utilization a bad pairing?

VPA would be resizing the container's CPU request while HPA is computing utilization as a percentage of that same request — each one's change moves the number the other is reacting to, so they can chase each other. The safer split is VPA on a metric HPA isn't scaling on (typically memory) while HPA drives replica count off CPU or a custom metric.

You can defend this when you can say what each of HPA, VPA, and the Cluster Autoscaler actually changes (replicas, requests, Nodes), explain why HPA needs a resource request set to compute CPU utilization, why VPA Auto mode causes disruption, and what signal triggers the Cluster Autoscaler to add a Node.