Autoscaling Kubernetes Workloads

Kubernetes autoscaling adjusts application or infrastructure capacity as workload demand changes. Instead of permanently provisioning enough resources for peak traffic, a cluster can add application replicas, increase pod resource allocations, or expand the underlying node pool when additional capacity is required.

Autoscaling is not simply a cost optimization. In production systems, it is part of capacity management and reliability engineering. Correct scaling can absorb traffic spikes and queue growth, while poorly configured scaling can create oscillation, resource contention, dependency overload, and delayed recovery during incidents.

The key design problem is choosing a signal that represents real workload pressure and ensuring the complete system can support the capacity created by the scaling decision.

Table of Contents

• Kubernetes Autoscaling Model
  • Horizontal, Vertical, and Node Scaling
  • Resource Requests and Scaling
• Horizontal Pod Autoscaler
  • CPU and Memory Scaling
  • Custom and External Metrics
• Vertical Pod Autoscaling
  • Resource Right-Sizing
  • Vertical Scaling Trade-Offs
• Node Autoscaling
  • Pending Pods and Cluster Capacity
  • Scale-Down Behavior
• Designing Autoscaling Signals
  • Request-Driven Workloads
  • Queue-Driven Workloads
• Production Design Example
  • Architecture
  • Scaling and Failure Flow
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Kubernetes Autoscaling Model

Kubernetes scaling operates at several layers. Application replicas can change, resource allocations can change, and cluster compute capacity can change.

These mechanisms solve related but different problems. Production architectures frequently combine them because adding pods does not help when the cluster has nowhere to schedule those pods.

Horizontal, Vertical, and Node Scaling

Three major scaling approaches are commonly used:

Horizontal scaling is usually the primary mechanism for stateless application services because requests can be distributed across independent replicas.

Vertical scaling is useful when individual processes need more resources or when resource requests are consistently inaccurate.

Node scaling operates below both. It ensures enough infrastructure exists to satisfy pod scheduling requirements.

Traffic increases
       |
       v
Horizontal Pod Autoscaler
       |
       v
Desired replicas: 6 -> 18
       |
       v
Scheduler
       |
       +---- Capacity available ----> Schedule pods
       |
       +---- Capacity unavailable --> Pods Pending
                                        |
                                        v
                                  Node Autoscaler
                                        |
                                        v
                                   Add Nodes
                                        |
                                        v
                                  Schedule Pods

This creates a scaling chain rather than one instantaneous operation.

Resource Requests and Scaling

Resource requests are central to Kubernetes capacity management. The scheduler uses them when deciding whether a pod fits on a node.

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    memory: "1Gi"

If 20 replicas each request 500 millicores, Kubernetes needs approximately 10 CPU cores of schedulable requested capacity for those pods, independent of their instantaneous CPU usage.

Requests that are too high waste schedulable capacity. Requests that are too low allow excessive workload density and can make the cluster appear to have more practical capacity than it really does.

CPU-based horizontal autoscaling also commonly evaluates utilization relative to CPU requests. Resource requests therefore influence both placement and scaling behavior.

For example, a pod using 400m CPU against a 500m request is at 80% utilization. The same workload using 400m against a 1000m request appears to be at 40%.

Incorrect requests can consequently produce incorrect autoscaling decisions even when raw application demand is unchanged.

Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler adjusts the replica count of scalable workloads such as Deployments according to observed metrics.

The objective is not to keep utilization perfectly constant. The objective is to maintain enough replicas for the workload while avoiding continuous scaling reactions to small metric fluctuations.

CPU and Memory Scaling

A common configuration scales a Deployment according to average CPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shipment-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shipment-api

  minReplicas: 4
  maxReplicas: 30

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

Conceptually, if current utilization is substantially above the target, the autoscaler increases desired replicas. If utilization remains below the target, replicas can eventually be reduced.

CPU works well when CPU consumption correlates strongly with application demand. This is common for compute-heavy request processing, encoding, parsing, transformations, and some API workloads.

CPU is less useful when application saturation occurs somewhere else first. An API may have low CPU while every worker waits on a slow database connection pool.

Memory is even more workload-dependent. Memory usage often does not fall immediately when traffic falls because runtimes retain heaps, caches, buffers, or allocated pages. Scaling directly from memory can therefore produce unexpected behavior.

Custom and External Metrics

Production scaling signals should represent the actual bottleneck or backlog whenever possible.

Useful metrics can include:

• requests per second
• active requests
• queue depth
• messages per consumer
• consumer lag
• concurrent jobs
• application-specific work backlog

Consider a worker system consuming tracking events. CPU utilization might remain at 35% even while the queue grows rapidly because workers spend much of their time waiting on carrier APIs.

Queue backlog is a better scaling signal:

Queue Depth
    |
    v
Scaling Metric
    |
    v
Worker Replicas
    |
    +-- Worker 1
    +-- Worker 2
    +-- Worker 3
    +-- ...

A useful target might be expressed as messages per worker. If 20,000 messages are waiting and the desired target is 1,000 messages per worker, approximately 20 workers are required before considering additional limits and scaling behavior.

The advantage is that scaling now reacts to actual unfinished work rather than an indirect resource signal.

The disadvantage is increased monitoring and metric-pipeline complexity. Custom metrics must be timely, reliable, and semantically correct because metric failures can become capacity-management failures.

Vertical Pod Autoscaling

Vertical scaling changes the CPU or memory resources assigned to individual pods rather than changing the number of replicas.

This is useful when resource requirements are difficult to estimate manually or when a workload cannot efficiently distribute work across many replicas.

Resource Right-Sizing

Suppose a service requests 2 GiB of memory but normally uses 450 MiB. Across 100 replicas:

Requested:
100 x 2 GiB = 200 GiB

Typical usage:
100 x 450 MiB ~= 44 GiB

The scheduler reserves capacity based on requests, so severe over-requesting can require substantially more nodes than actual workload usage suggests.

At the opposite extreme, requesting 256 MiB for a process that routinely requires 1 GiB makes scheduling decisions overly optimistic.

Vertical recommendations based on historical resource behavior can help identify more realistic allocations.

Advantages:

• reduces persistent over-requesting
• improves cluster utilization
• helps identify under-provisioned workloads
• reduces manual resource tuning

Disadvantages:

• resource changes may require replacing pods
• historical usage may not represent future peaks
• recommendations can be distorted by unusual traffic periods
• vertical changes do not solve every throughput bottleneck

Vertical Scaling Trade-Offs

Vertical and horizontal scaling can interact. If horizontal scaling uses CPU utilization relative to requests while another mechanism continuously changes those requests, the scaling system becomes harder to reason about.

For stateless APIs, a common production approach is to use historical resource observations for right-sizing while allowing horizontal scaling to handle demand changes.

For example:

Resource analysis
      |
      v
Set realistic requests
      |
      v
Horizontal autoscaling
handles traffic variation

This keeps responsibilities clearer: resource sizing defines what one replica needs, while horizontal scaling defines how many replicas the workload needs.

Node Autoscaling

Horizontal pod scaling is constrained by cluster capacity. When the scheduler cannot place newly requested pods, additional compute capacity may need to be provisioned.

Node autoscaling connects Kubernetes scheduling demand with the infrastructure layer.

Pending Pods and Cluster Capacity

Suppose an API scales from 10 to 25 replicas, but the existing nodes have capacity for only 16:

Desired replicas: 25

Running: 16
Pending: 9

Existing cluster:
No schedulable capacity
        |
        v
Node scaling
        |
        v
Additional compute
        |
        v
Pending pods scheduled

This creates an important production delay. The HPA may decide to scale within seconds, but adding infrastructure can take much longer because machines need to be provisioned, initialized, registered, and made ready for workloads.

Applications experiencing sudden traffic spikes therefore cannot assume node autoscaling will provide capacity immediately.

Strategies for reducing this delay include maintaining spare cluster capacity, setting realistic minimum node counts, using appropriately sized node pools, and scaling from predictive or leading indicators where justified.

High-availability capacity planning is covered more deeply here: Designing Highly Available Kubernetes Applications.

Scale-Down Behavior

Scaling down infrastructure is more complicated than deleting an apparently underutilized node.

A node can host:

• application pods
• system workloads
• pods with strict topology requirements
• stateful workloads
• pods protected by disruption budgets

Before removing a node, movable workloads generally need somewhere else to run. Aggressive scale-down can therefore cause unnecessary pod movement and interfere with availability constraints.

Scale-down should normally be more conservative than scale-up. Capacity shortages can cause immediate user-facing failures, while retaining an extra node temporarily usually creates only additional infrastructure cost.

This asymmetry is important: fast scale-up and slower scale-down often produce more stable systems than reacting equally quickly in both directions.

Designing Autoscaling Signals

The quality of an autoscaling system depends heavily on its input signal. The best metric is usually one that predicts or directly represents the resource whose saturation would harm service objectives.

A metric can be easy to collect while still being architecturally wrong.

Request-Driven Workloads

For synchronous APIs, useful scaling signals depend on what limits throughput.

Imagine an API where one endpoint performs a cached lookup in 5 ms while another performs a complex report in 800 ms. Scaling purely on request count treats both requests as equal even though their resource costs differ dramatically.

CPU may be a better aggregate signal in that workload, or application metrics may need to distinguish expensive operations.

Autoscaling should therefore be tested under representative traffic mixes, not only synthetic requests to one endpoint.

Queue-Driven Workloads

Asynchronous consumers often have a clearer scaling model because backlog represents unfinished work.

Suppose each worker can process approximately 100 messages per second and the target is to clear bursts within 60 seconds.

A backlog of 120,000 messages represents roughly:

120,000 messages
---------------- = 1,200 worker-seconds
100 msg/s

With 20 workers, idealized processing time is approximately:

1,200 worker-seconds
-------------------- = 60 seconds
20 workers

This provides a more meaningful capacity model than simply targeting 70% CPU.

However, adding workers increases downstream concurrency. If 100 workers simultaneously call a carrier API limited to 50 requests per second, scaling the workers can increase throttling without increasing throughput.

Autoscaling cannot create capacity in a bottleneck that exists outside the scaled workload.

Production Design Example

Consider a logistics platform that receives shipment requests through an HTTP API and processes tracking events asynchronously. Traffic varies significantly during the day, and carrier updates can arrive in large bursts.

The architecture uses different scaling strategies because the API and worker workloads have different demand models.

Architecture

                         Clients
                            |
                            v
                     Load Balancer
                            |
                            v
                         Ingress
                            |
                            v
                     Shipment Service
                            |
                            v
                    Shipment API Pods
                    HPA: CPU + demand
                            |
               +------------+------------+
               |                         |
               v                         v
           Database                  Event Queue
                                         |
                                         v
                                  Tracking Workers
                                  Scale by backlog
                                         |
                                         v
                                    Carrier APIs


      Pod demand exceeds node capacity
                    |
                    v
              Node Autoscaling
                    |
                    v
              Additional Nodes

The Shipment API starts with four replicas and can scale to 30. Four replicas provide baseline redundancy and avoid relying on autoscaling for normal high availability.

Tracking workers scale according to queue backlog because unfinished events are the most useful representation of workload pressure.

Scaling and Failure Flow

Suppose normal API traffic is 1,500 requests per second with six replicas. A customer integration suddenly increases traffic to 5,000 requests per second.

1. Per-pod CPU and request concurrency increase.
2. The HPA observes sustained demand above its target.
3. Desired replicas increase from 6 to 15.
4. The scheduler places seven new pods on existing nodes.
5. Two pods remain Pending because cluster capacity is exhausted.
6. Node autoscaling detects unschedulable demand.
7. Additional nodes are provisioned.
8. The remaining pods are scheduled and initialized.
9. Readiness succeeds and Services begin routing traffic to them.
10. Per-pod load falls toward the target operating range.

This process contains several delays. Metrics need to reflect the traffic increase, the autoscaler needs to react, pods need to start, nodes may need to be provisioned, and new replicas need to become ready.

Autoscaling therefore cannot replace sufficient baseline capacity for abrupt spikes.

Now suppose tracking-event backlog rises from 5,000 to 200,000 messages. Worker replicas scale from 5 to 40.

If every worker opens ten database connections, potential worker connections increase from 50 to 400. If every worker can issue five concurrent carrier requests, external concurrency can increase from 25 to 200.

The worker autoscaler therefore needs a maximum replica count derived from end-to-end system capacity, not only queue size.

A useful safety model is:

Maximum Worker Replicas =
minimum of:

- cluster compute capacity
- database connection capacity
- carrier API concurrency capacity
- queue throughput capacity
- internal network capacity
- operational safety limit

During scale-down, the system should avoid immediately removing most replicas when a temporary metric drop occurs. A stabilization period allows the workload to demonstrate that demand has actually fallen.

An HPA can define explicit scaling behavior:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 60

  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 25
        periodSeconds: 60

This configuration allows comparatively aggressive growth while making scale-down more conservative.

The exact values should come from startup time, traffic behavior, dependency capacity, and observed production metrics rather than being copied across workloads.

Production monitoring should include:

• desired vs current replicas
• autoscaling metric values
• time spent at maximum replicas
• Pending and unschedulable pods
• pod startup and readiness duration
• node provisioning duration
• CPU and memory utilization
• request rate and concurrency
• request latency and error rate
• queue depth and oldest-message age
• database connection utilization
• downstream throttling and timeout rates

Alerts should distinguish between normal scaling and scaling that cannot restore healthy operation. For example, maximum replicas combined with growing latency or queue backlog indicates a capacity problem that autoscaling can no longer solve.

Common Mistakes

Autoscaling problems usually come from incorrect signals, unrealistic capacity assumptions, or treating scaling as instantaneous. The number of replicas is only one variable in an end-to-end production system.

Production Checklist

Production autoscaling should be based on measured workload behavior, realistic startup times, and explicit capacity limits across the complete dependency chain.

• Choose a meaningful scaling signal. Prefer metrics that correlate with the workload's actual saturation or backlog.
• Set realistic resource requests. Validate CPU and memory requests against observed production behavior.
• Maintain baseline replicas. Do not depend on reactive scaling for basic application availability.
• Define maximum replicas from system capacity. Include database, cache, queue, external API, and network constraints.
• Measure scale-up latency. Include metric collection, autoscaler reaction, scheduling, node provisioning, startup, and readiness.
• Monitor Pending pods. Detect when application scaling is blocked by cluster capacity.
• Maintain infrastructure headroom. Keep enough capacity for normal spikes and failure recovery where required.
• Use controlled scale-down. Avoid removing capacity immediately after short metric drops.
• Test peak traffic mixes. Include expensive and inexpensive requests rather than one synthetic endpoint.
• Scale queue consumers from backlog. Use queue depth, lag, or processing-time objectives when they better represent demand.
• Protect downstream dependencies. Bound concurrency, connection pools, and request rates as replicas increase.
• Monitor autoscaler limits. Alert when workloads remain at maximum replicas while saturation continues increasing.
• Track scaling events. Correlate replica changes with application and infrastructure metrics.
• Test node-scale delays. Validate how long additional compute capacity takes to become usable.
• Load-test scaling behavior. Verify that scaling actually restores service objectives instead of moving the bottleneck elsewhere.

Conclusion

Kubernetes autoscaling provides several layers of capacity control. Horizontal scaling changes replica counts, vertical scaling adjusts pod resources, and node scaling expands or contracts cluster compute capacity. Production systems often need more than one of these mechanisms.

The difficult part is not enabling an autoscaler. It is selecting signals that represent real demand, understanding how long additional capacity takes to become useful, and ensuring downstream systems can tolerate the concurrency created by additional replicas.

Key Takeaway: autoscale from the bottleneck that matters, not simply the metric that is easiest to collect. Treat pod scaling, node capacity, startup latency, dependency limits, and service-level performance as one capacity-management system.

More Articles to Read

• Kubernetes Explained: Pods, Nodes, and Clusters
• Deployments, ReplicaSets, and StatefulSets
• Services, Ingress, and Networking
• Designing Highly Available Kubernetes Applications
• Managing Configuration and Secrets
• Kubernetes Best Practices for Production