Scalability Patterns¶

Availability and resiliency patterns keep a system serving traffic through failure. Scalability patterns are about handling load — more requests, more work — without the system falling over or a burst of demand overwhelming a downstream dependency.

TL;DR — in 30 seconds:

Autoscaling adds or removes instances automatically based on real demand (CPU, request rate, queue depth), instead of running a fixed, hand-sized fleet that's either over-provisioned or falls over under load.
Queue-based load leveling puts a message queue between a producer and a consumer, so a burst of work is absorbed by the queue and drained at a rate the consumer can actually handle, instead of hitting it directly.
Cache-aside has the application check a cache first and only fall through to the (slower) primary data store on a miss, populating the cache for next time — cutting repeated load on the data store.
All three answer the same question — "what happens when demand outpaces what's currently provisioned" — at different points in the system.

1. Autoscaling: match capacity to real demand¶

flowchart LR
    M["Metric<br/>(CPU, request rate, queue depth)"] -->|"crosses threshold"| AS["Autoscaler"]
    AS -->|"adds instances"| Pool["Instance pool"]
    Pool -->|"metric drops"| AS
    AS -->|"removes instances"| Pool

Scaling out (more instances) is the common case for stateless services in the cloud; scaling up (a bigger instance) has a ceiling and usually means downtime to resize.
Autoscaling needs a real signal to react to — CPU utilization, request rate, or (for queue-based work, see below) queue depth — and it works hand-in-hand with the load balancer (page 1), which starts routing traffic to instances as soon as the autoscaler brings them up and stops as soon as it removes them.
Scaling out takes real time (a new instance has to start and pass its readiness check) — autoscaling smooths sustained demand changes; it doesn't absorb an instantaneous spike the way a queue can (below).

2. Queue-based load leveling: absorb bursts, drain at a steady rate¶

flowchart LR
    P["Producer<br/>(bursty traffic)"] -->|"enqueues work"| Q["Queue"]
    Q -->|"drains at a<br/>steady, sustainable rate"| C["Consumer"]

Instead of the producer calling the consumer directly (and overwhelming it during a spike), work is dropped onto a queue; the consumer pulls from the queue at whatever rate it can actually sustain.
This decouples the producer's rate of arrival from the consumer's rate of processing — a burst queues up and drains over time rather than causing dropped requests or a crashed consumer.
Queue depth is itself a useful signal for autoscaling the consumer: a growing queue means "add more consumers," an empty queue means "scale back down."

3. Cache-aside: reduce repeated load on the primary store¶

sequenceDiagram
    participant App
    participant Cache
    participant Store as Primary data store
    App->>Cache: read key
    alt cache hit
        Cache-->>App: value
    else cache miss
        Cache-->>App: (miss)
        App->>Store: read key
        Store-->>App: value
        App->>Cache: write key = value
    end

The application, not the data store, owns the caching logic: check the cache first; on a miss, read the primary store and populate the cache with that result for next time.
This is the most common caching pattern because it only caches what's actually requested (rather than pre-loading everything), and a cache outage just means every read falls through to the primary store — slower, but not broken.
The tradeoff is staleness: the cache can serve an out-of-date value until it expires or is invalidated, so cache-aside fits data that tolerates being briefly stale better than data that must always be perfectly current.

Common gotchas

Autoscaling on the wrong signal. Fix: CPU alone can miss an I/O-bound bottleneck — pick a metric that actually reflects the thing about to fail (queue depth, request latency, request rate).
Treating a queue as a substitute for fixing a slow consumer. Fix: a queue absorbs bursts: if the consumer's average processing rate is permanently below the average arrival rate, the queue just grows without bound instead of draining — that's a capacity problem, not a burst.
No cache invalidation strategy. Fix: cache-aside without a plan for expiring or updating stale entries can serve wrong data indefinitely — decide a TTL or an explicit invalidation trigger up front.

Check yourself — a service gets a sudden 10x spike in incoming work for 30 seconds, and the downstream consumer can only sustainably handle 2x normal load. Autoscaling the consumer takes two minutes to bring up new instances. What pattern actually absorbs those first 30 seconds, and why?

Queue-based load leveling. The spike is shorter than the time autoscaling takes to react, so new instances can't help in time — but a queue in front of the consumer absorbs the burst by holding the extra work and letting the consumer drain it at a rate it can sustain, without dropping requests.

Done when: you can explain what signal drives autoscaling, why a queue absorbs a burst that autoscaling alone can't react to in time, and what cache-aside trades away (staleness) for its reduced load on the primary store.