Kubernetes Best Practices for Production
Kubernetes provides primitives for scheduling, service discovery, health checks, scaling, configuration, and workload recovery, but production reliability depends on how those primitives are combined. A cluster can be technically healthy while applications suffer from poor scheduling, weak failure isolation, incorrect probes, aggressive autoscaling, unsafe rollouts, or overloaded dependencies.
Production Kubernetes design should focus on predictable failure behavior. Workloads need realistic resource requests, multiple replicas, controlled deployments, topology-aware placement, bounded dependency calls, secure configuration, and enough spare capacity to recover when infrastructure disappears.
The most useful best practices are therefore not individual YAML settings. They are architectural rules that make application behavior easier to understand during deployments, scaling events, node failures, dependency outages, and traffic spikes.
Table of Contents
- Design Workloads for Failure
- Manage Resources and Capacity
- Build Safe Application Lifecycles
- Design Networking and Dependencies Carefully
- Control Autoscaling, Configuration, and Security
- Production Design Example
- Common Mistakes
- Production Checklist
- Conclusion
- More Articles to Read
Design Workloads for Failure
Kubernetes is built around replacement rather than repair. Pods are disposable, nodes can disappear, deployments replace workload instances, and autoscaling changes replica counts continuously.
Applications should therefore assume that individual runtime instances are temporary. Reliability comes from preserving service capacity when some of those instances disappear.
Run Multiple Replicas
A single-replica production API remains a single point of failure even when Kubernetes can recreate it automatically.
Single Replica
Client
|
v
Service
|
v
Pod A
Pod A fails
|
v
No serving capacity
until replacement becomes ready
With multiple replicas:
Service
|
+--------+--------+
| | |
v v v
Pod A Pod B Pod C
Pod B fails
|
v
Pod A and Pod C continue serving
The minimum replica count should be based on availability requirements and surviving throughput, not simply an arbitrary number such as two or three.
If four replicas normally run near 80% CPU, losing one replica leaves insufficient capacity even though three replicas remain technically available.
Spread Across Failure Domains
Replica count has limited value when replicas share the same failure domain. Five replicas on one node can disappear together.
Critical workloads should be distributed across nodes and, where required, availability zones:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: shipment-api
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: shipment-api
Strict scheduling requirements improve isolation but can also create Pending pods when eligible infrastructure is unavailable.
Topology design should therefore balance failure isolation against schedulability.
High-availability workload distribution is covered in greater depth here: Designing Highly Available Kubernetes Applications.
Manage Resources and Capacity
Kubernetes scheduling depends heavily on declared resource requests. Production capacity planning becomes unreliable when workloads request resources that have little relationship to actual runtime needs.
Resource configuration should reflect measured workload behavior while still leaving enough safety margin for bursts, garbage collection, temporary imbalance, and failure recovery.
Set Realistic Resource Requests
Resource requests influence where pods can run:
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
memory: "1Gi"
If a workload actually needs around 900 MiB but requests 256 MiB, the scheduler may place too many pods on the same node. Under load, memory pressure can then cause instability or eviction.
If the workload needs 300 MiB but requests 2 GiB, the cluster reserves far more capacity than necessary and requires more nodes.
| Resource Problem | Production Effect | Better Approach |
|---|---|---|
| Requests too low | Nodes become overloaded | Measure real sustained and peak usage |
| Requests too high | Cluster capacity is wasted | Right-size from historical workload data |
| No requests | Scheduling becomes less predictable | Declare explicit baseline requirements |
| Limits too aggressive | Applications may be throttled or terminated unnecessarily | Set limits according to workload behavior |
CPU and memory should be reviewed after traffic changes, runtime upgrades, major feature releases, and architectural changes.
Preserve Recovery Headroom
A cluster running at almost full requested capacity has limited ability to recover from node loss.
Normal operation:
Node A: 90%
Node B: 88%
Node C: 92%
Node D: 89%
Node A fails
Replacement pods need capacity
|
v
Nodes B-D cannot fit them
|
v
Pods remain Pending
Recovery headroom should match the failure model. If the architecture needs to survive one node loss, surviving nodes or node-autoscaling infrastructure must be able to restore required workload capacity.
The same applies to rolling deployments. During a rollout, old and new replicas can overlap temporarily, increasing resource demand.
Efficient utilization is useful, but maximum utilization and maximum resilience are conflicting goals. Production clusters need intentional margin.
Build Safe Application Lifecycles
Kubernetes continuously starts and stops application processes. Safe lifecycle behavior determines whether those transitions are invisible to clients or cause dropped requests and unstable rollouts.
Applications should distinguish startup, readiness, runtime health, and termination as separate states.
Use Health Checks Correctly
Readiness controls whether a pod should receive traffic. Liveness determines whether Kubernetes should restart a stuck process. Startup probes can protect slow-starting applications from premature liveness failures.
startupProbe:
httpGet:
path: /health/startup
port: 8000
periodSeconds: 5
failureThreshold: 30
readinessProbe:
httpGet:
path: /health/ready
port: 8000
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet:
path: /health/live
port: 8000
periodSeconds: 10
failureThreshold: 3
A common mistake is making every probe perform deep checks against databases, caches, queues, and external APIs.
If a database outage causes liveness to fail, every application replica can restart simultaneously. The underlying database problem remains while Kubernetes creates additional instability.
Use readiness to represent whether the application can currently serve meaningful traffic and keep liveness focused on whether the local process can recover without restart.
More detailed health-check design is available here: Health Checks, Readiness, and Liveness Probes.
Deploy and Terminate Gracefully
Production deployments should replace replicas gradually while preserving enough ready capacity.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
New pods should become ready only after initialization is complete. Old pods should stop receiving new traffic before their processes exit.
Applications should handle termination signals and finish active work within a bounded grace period:
terminationGracePeriodSeconds: 30
Graceful termination is particularly important for:
- long-running HTTP requests
- gRPC streams
- WebSockets
- queue consumers
- database transactions
- batch jobs
Deployment behavior is explained in more depth here: Deployments, ReplicaSets, and StatefulSets.
Design Networking and Dependencies Carefully
Kubernetes provides stable Services and workload networking, but distributed-system failure modes still apply. A request can cross ingress, services, application pods, caches, databases, message brokers, and external APIs before completing.
Each synchronous dependency adds latency and another possible failure point.
Keep Traffic Paths Simple
Internal service-to-service calls should normally use internal Services rather than unnecessarily leaving and re-entering the cluster through external ingress infrastructure.
Preferred internal path:
Service A
|
v
Service B
|
v
Pod
Unnecessary path:
Service A
|
v
External / Ingress Layer
|
v
Service B
The second path adds proxying, TLS work, latency, additional capacity dependencies, and another shared failure domain.
Ingress should generally handle controlled external HTTP traffic, while Services provide stable internal discovery.
For more detail, see: Services, Ingress, and Networking.
Prevent Cascading Failures
Kubernetes can keep application pods alive while the service becomes unusable because every pod waits on the same slow dependency.
Consider 40 API pods calling an overloaded inventory service:
Clients
|
v
40 API Pods
|
| many slow synchronous calls
v
Inventory Service
|
v
Overloaded Database
Without bounded waiting time, requests accumulate and consume sockets, workers, memory, database connections, and concurrency limits.
Production applications should use patterns such as:
- timeouts for remote calls
- bounded retries only where retrying is safe
- exponential backoff
- circuit breakers
- bulkheads
- load shedding
- graceful degradation
Retries deserve special attention. If 10,000 failing requests are each retried three times, one dependency outage can create up to 30,000 additional attempts.
Resilience patterns are covered here: Timeouts, Retries, and Exponential Backoff.
Control Autoscaling, Configuration, and Security
Autoscaling and configuration changes both alter runtime behavior without necessarily changing application code. They should be treated as production control systems, not convenience features.
Security has the same operational requirement: access should be explicit, minimal, auditable, and designed so that one compromised workload does not automatically expose unrelated systems.
Scale from Real Bottlenecks
CPU autoscaling works well when CPU correlates with workload demand, but not every system is CPU-bound.
Queue consumers often scale better from backlog:
Queue Depth
|
v
Autoscaling Metric
|
v
Worker Replicas
Request-driven services may scale from CPU, concurrency, request rate, or another workload-specific signal.
Scaling must also account for downstream limits. Increasing workers from 10 to 100 can overload a database or third-party API even if the Kubernetes cluster has enough compute.
Maximum replica counts should therefore reflect the minimum capacity of the complete dependency chain.
Autoscaling design is covered in more detail here: Autoscaling Kubernetes Workloads.
Treat Configuration as a Deployment
Configuration can break production as easily as application code. Incorrect hostnames, timeouts, feature flags, credentials, or certificates can affect every replica.
Configuration changes should therefore be:
- versioned
- reviewed
- validated
- rolled out gradually where practical
- observable
- reversible
Use ConfigMaps for non-sensitive runtime configuration and secret-oriented workflows for sensitive values.
Secrets should not be treated as protected simply because they are stored in Kubernetes Secret objects. Production security still requires access controls, storage protection, rotation, auditing, and careful distribution.
Prefer workload identity and short-lived credentials to long-lived static cloud keys where the surrounding platform supports them.
For more detail, see: Managing Configuration and Secrets.
Production Design Example
Consider a Kubernetes-based logistics platform serving public shipment APIs, internal pricing services, tracking workers, PostgreSQL, Redis, and a message broker. Public traffic is highly variable, and the platform must tolerate one worker-node failure without interrupting core APIs.
The architecture applies availability, capacity, rollout, networking, scaling, and configuration practices together rather than optimizing each Kubernetes resource independently.
Architecture
Internet
|
v
Load Balancer
|
v
Ingress Layer
|
v
Shipment Service
|
+---------------+---------------+
| | |
v v v
Zone A Zone B Zone C
API Pods API Pods API Pods
| | |
+---------------+---------------+
|
+------------+------------+
| |
v v
Pricing Service Queue
| |
v v
Pricing Pods Worker Pods
| |
+------------+------------+
|
+-----------+-----------+
| | |
v v v
Database Redis Carrier APIs
The API has six baseline replicas distributed across three zones. Resource requests are based on load testing and production observations rather than defaults.
The cluster maintains enough free capacity to tolerate one node failure without waiting for node autoscaling before basic API availability is restored.
Ingress runs with multiple replicas across failure domains. Internal calls use ClusterIP Services and do not route through the public ingress layer.
Tracking workers scale from queue backlog instead of CPU because backlog directly represents unfinished work.
Failure and Operational Flow
Suppose one worker node disappears during a traffic peak.
- Several application pods disappear with the node.
- Services stop routing new traffic to unavailable endpoints.
- Remaining API replicas continue processing requests.
- Controllers request replacement replicas.
- The scheduler uses available cluster headroom where possible.
- Additional infrastructure is requested if remaining capacity is insufficient.
- Replacement pods initialize and pass readiness checks.
- Traffic gradually returns to normal distribution.
This recovery works only because the architecture previously addressed several independent concerns.
Replica distribution: not all API instances were placed on the failed node.
Resource capacity: surviving nodes had enough space for replacement workloads.
Readiness: replacement pods received traffic only after initialization.
Dependency capacity: the database could handle traffic redistribution.
Networking: Services removed failed endpoints without clients depending on individual pod IPs.
Observability: alerts distinguish node loss from application-level errors.
Now suppose a deployment begins during the same period. The rolling strategy creates temporary surge replicas.
If cluster headroom was sized only for normal steady-state capacity, recovery pods plus surge pods could exceed schedulable resources. Deployment settings, failure recovery, and capacity planning therefore need to be analyzed together.
A safer operational policy may pause non-essential deployments when infrastructure capacity is degraded.
Production monitoring should include:
- desired, ready, and unavailable replicas
- pod distribution by node and zone
- Pending and unschedulable pods
- node CPU, memory, and allocatable capacity
- pod CPU and memory usage versus requests
- pod restart rate
- readiness and startup duration
- request rate, latency, and errors
- ingress saturation
- service endpoint counts
- dependency timeout and retry rates
- database connection utilization
- queue depth and consumer lag
- autoscaler desired replicas
- configuration and deployment versions
Operational dashboards should connect infrastructure signals with user-facing behavior. A node failure matters because of its effect on request latency, errors, backlog, and available capacity, not merely because a Kubernetes Node object changed state.
Common Mistakes
Most Kubernetes production failures are not caused by a missing feature. They result from incorrect assumptions about scheduling, health, capacity, dependency behavior, or how independent control loops interact.
| Mistake | Production Impact | Better Approach |
|---|---|---|
| Running one replica of a critical API | Pod replacement still creates an availability gap. | Maintain multiple replicas with sufficient surviving capacity. |
| Ignoring replica placement | One node or zone failure removes many replicas simultaneously. | Spread workloads across relevant failure domains. |
| Leaving resource requests inaccurate | Scheduling, capacity planning, and autoscaling become unreliable. | Continuously right-size from measured workload behavior. |
| Running clusters without recovery headroom | Replacement pods remain Pending after node loss. | Reserve capacity according to the expected failure model. |
| Using deep dependency checks for liveness | Shared dependency failures trigger mass restarts. | Keep liveness focused on local process health. |
| Ignoring graceful shutdown | Deployments and scaling terminate active requests or jobs. | Drain traffic and finish work within a bounded grace period. |
| Using CPU autoscaling for every workload | Scaling reacts poorly to queue, I/O, or dependency bottlenecks. | Scale from metrics that represent actual workload pressure. |
| Allowing unlimited retries | Dependency incidents become retry storms. | Use bounded retries, backoff, and retry budgets. |
| Editing production configuration manually | Changes become difficult to audit and roll back. | Version and deploy configuration through controlled workflows. |
| Monitoring Kubernetes objects without service metrics | Clusters appear healthy while users experience failures. | Correlate infrastructure state with latency, errors, throughput, and backlog. |
Production Checklist
A production Kubernetes environment should make expected failures survivable, deployments predictable, scaling bounded, and operational behavior observable.
- Run multiple replicas. Critical services should retain capacity after individual pod failures.
- Spread workloads across failure domains. Use node and zone topology intentionally.
- Set realistic resource requests. Base CPU and memory values on actual workload measurements.
- Maintain recovery headroom. Ensure critical replacement pods can schedule after expected infrastructure loss.
- Use readiness correctly. Route traffic only to pods capable of serving useful requests.
- Keep liveness focused. Avoid restarting workloads because shared dependencies are temporarily unhealthy.
- Handle graceful termination. Drain requests and jobs before pod shutdown.
- Use controlled rolling deployments. Preserve required capacity during releases.
- Keep internal traffic internal. Use Services rather than unnecessary public ingress paths.
- Bound remote calls. Configure timeouts and controlled retry policies.
- Protect downstream systems. Limit concurrency, connections, and autoscaling according to dependency capacity.
- Scale from meaningful signals. Use CPU, concurrency, queue backlog, or application metrics according to the workload.
- Version configuration. Make application and configuration combinations reproducible.
- Protect and rotate secrets. Prefer least privilege and short-lived credentials where possible.
- Monitor scheduling failures. Alert on Pending and unschedulable critical pods.
- Observe service-level behavior. Measure latency, errors, throughput, saturation, and backlog.
- Track topology. Know where critical replicas actually run.
- Test failure scenarios. Exercise pod loss, node loss, dependency failure, and degraded capacity.
- Measure recovery time. Track detection, rescheduling, startup, readiness, and full service restoration.
- Review capacity after major changes. Recalculate assumptions when traffic, dependencies, runtime behavior, or architecture changes.
Conclusion
Production Kubernetes reliability comes from combining several disciplines: failure-aware workload placement, realistic resource management, safe lifecycle handling, simple networking, dependency resilience, controlled autoscaling, secure configuration, and strong observability.
No individual Kubernetes object creates a production-grade system. A Deployment cannot compensate for insufficient cluster capacity, an HPA cannot fix an overloaded database, and multiple replicas do not provide availability when they share the same failure domain.
Key Takeaway: design Kubernetes as a collection of interacting control loops operating around real application constraints. Production systems become reliable when failures remain isolated, enough capacity survives them, recovery is predictable, and every important transition can be observed and tested.
Comments (0)