Managing Configuration and Secrets

Kubernetes applications need configuration that changes independently from application code: database endpoints, feature flags, queue names, service URLs, credentials, API tokens, certificates, and environment-specific settings. Embedding these values directly into container images couples deployment artifacts to environments and makes configuration changes unnecessarily expensive.

Kubernetes provides ConfigMaps for non-sensitive configuration and Secrets for sensitive values. Both can be exposed to containers through environment variables or mounted files, allowing the same container image to run with different runtime configuration.

The difficult production problem is not creating these objects. It is managing ownership, validation, security, rotation, rollout behavior, synchronization, auditing, and failure recovery as configuration changes across many workloads and environments.

Table of Contents

• Configuration in Kubernetes
  • Configuration as Runtime Input
  • ConfigMaps vs Secrets
• Using ConfigMaps
  • Environment Variables
  • Mounted Configuration Files
• Managing Secrets
  • Secret Consumption
  • External Secret Management
• Configuration Updates and Rollouts
  • Configuration Change Behavior
  • Immutable and Versioned Configuration
• Secret Rotation and Failure Handling
  • Safe Secret Rotation
  • Configuration Failure Modes
• Production Design Example
  • Architecture
  • Deployment and Rotation Flow
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Configuration in Kubernetes

A container image should ideally represent application code and its runtime dependencies, while environment-specific configuration is supplied separately. This allows one tested image to move through development, staging, and production without being rebuilt for every environment.

Configuration becomes part of the deployment contract. A valid image with invalid configuration can be just as unavailable as a broken image, so configuration needs versioning, validation, rollout controls, and observability.

Configuration as Runtime Input

Consider a shipment API that requires several environment-specific values:

Application Image
shipment-api:7.3.1
        |
        +--------------------+
        |                    |
        v                    v
  Configuration           Secrets
        |                    |
        v                    v
LOG_LEVEL=info       DATABASE_PASSWORD
QUEUE_NAME=events    CARRIER_API_TOKEN
CACHE_HOST=redis     TLS_PRIVATE_KEY

The same image can run in multiple environments while configuration changes independently:

This separation reduces environment-specific builds and makes configuration changes easier to review independently from application releases.

ConfigMaps vs Secrets

ConfigMaps and Secrets expose similar application-consumption patterns, but they represent different categories of information.

A ConfigMap might contain:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shipment-api-config
data:
  LOG_LEVEL: "info"
  CACHE_HOST: "redis.default.svc.cluster.local"
  QUEUE_NAME: "shipment-events"
  CARRIER_TIMEOUT_SECONDS: "5"

A Secret might contain credentials:

apiVersion: v1
kind: Secret
metadata:
  name: shipment-api-secrets
type: Opaque
stringData:
  DATABASE_PASSWORD: "example-value"
  CARRIER_API_TOKEN: "example-token"

The example values illustrate structure only. Production credentials should not normally be stored directly in deployment manifests committed to source control.

Also, Secret values being encoded for API representation should not be confused with encryption. Encoding does not provide confidentiality. Protection depends on access controls, storage encryption, secret-management architecture, and operational practices.

Using ConfigMaps

Applications can consume ConfigMaps in several ways. Two common approaches are environment variables and mounted files.

The correct choice depends on how the application reads configuration and whether runtime updates need to become visible without replacing the process.

Environment Variables

A Deployment can load values from a ConfigMap:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shipment-api
spec:
  replicas: 4
  selector:
    matchLabels:
      app: shipment-api
  template:
    metadata:
      labels:
        app: shipment-api
    spec:
      containers:
        - name: api
          image: registry.example.com/shipment-api:7.3.1
          envFrom:
            - configMapRef:
                name: shipment-api-config

Environment variables are simple and widely supported. Applications can validate them during startup and fail quickly when required values are missing.

Advantages:

• simple application integration
• easy startup validation
• works with most frameworks and runtimes
• configuration is fixed for the lifetime of the process

Disadvantages:

• changes do not automatically modify the environment of an already running process
• large structured configuration becomes awkward
• careless debugging or process inspection can expose values

Environment variables work particularly well for small, scalar configuration that should change only through controlled application rollouts.

Mounted Configuration Files

ConfigMaps can also be mounted as files:

volumes:
  - name: app-config
    configMap:
      name: shipment-api-config

containers:
  - name: api
    image: registry.example.com/shipment-api:7.3.1
    volumeMounts:
      - name: app-config
        mountPath: /etc/shipment-api
        readOnly: true

Configuration keys become files under the mount location. This works well for structured configuration, certificates, templates, and applications designed to reload configuration files.

Mounted configuration can change independently from the process, but the application still needs explicit reload behavior if changes should take effect without restart.

An application that reads the file only during startup behaves effectively like an environment-variable configuration even if Kubernetes later updates the mounted content.

Dynamic configuration therefore requires two mechanisms:

1. the updated configuration must reach the container
2. the application must safely detect and apply the change

Hot reload can reduce restart requirements, but it also introduces runtime state transitions that need testing. For many services, controlled replacement of pods is easier to reason about than dynamic reconfiguration.

Managing Secrets

Secrets require stronger controls because accidental disclosure can grant access to databases, cloud resources, external APIs, encryption keys, or internal systems.

The objective is to minimize where plaintext secrets exist, who can retrieve them, how long credentials remain valid, and how much damage one compromised credential can cause.

Secret Consumption

A Kubernetes Secret can be exposed as an environment variable:

env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: shipment-api-secrets
        key: DATABASE_PASSWORD

Or mounted as files:

volumes:
  - name: application-secrets
    secret:
      secretName: shipment-api-secrets

containers:
  - name: api
    image: registry.example.com/shipment-api:7.3.1
    volumeMounts:
      - name: application-secrets
        mountPath: /var/run/secrets/application
        readOnly: true

Mounted files are useful for TLS certificates, private keys, and applications that support credential reload.

Environment variables remain convenient for applications expecting credentials at startup, but they tend to encourage credentials to remain unchanged for the process lifetime.

Neither consumption model eliminates the need for authorization. Workloads and operators should receive only the secrets required for their responsibilities.

External Secret Management

For larger production environments, Kubernetes often should not become the original source where humans manually maintain long-lived credentials.

A dedicated secret-management system can provide capabilities such as:

• centralized credential storage
• encryption and key management
• access auditing
• automatic rotation
• short-lived credentials
• integration with workload identities
• central revocation

A common architecture is:

Secret Management System
          |
          | authenticated workload
          | or synchronization
          v
Kubernetes / Application Runtime
          |
          v
Application Pod
          |
          v
Database / External API

One model synchronizes external secrets into Kubernetes Secrets. This provides compatibility with normal Kubernetes workloads but creates another stored copy of the credential.

Another model allows workloads to retrieve or mount secrets through an external integration. This can reduce persistent duplication and support more dynamic credential lifecycles, but it increases runtime dependencies and operational complexity.

Where supported, workload identity and short-lived credentials can be preferable to distributing static cloud credentials. Instead of storing a long-lived access key, the workload proves its identity and obtains narrowly scoped temporary authorization.

Secrets-management architecture is covered more deeply here: Secrets Management in Cloud Applications.

Configuration Updates and Rollouts

Configuration changes are production deployments even when no application image changes. A single incorrect timeout, hostname, feature flag, or credential can make every replica fail simultaneously.

Configuration therefore needs the same engineering discipline as code: review, validation, controlled rollout, observability, and rollback.

Configuration Change Behavior

Suppose a Deployment loads ConfigMap values as environment variables:

ConfigMap v1
    |
    v
Pod A  Pod B  Pod C
    |
ConfigMap changed to v2
    |
    +---- Existing pod environments remain unchanged
    |
    +---- Newly created pods use updated configuration

This can temporarily create two configuration versions within the same Deployment if some pods are replaced and others continue running.

That inconsistency can be dangerous. For example, half the replicas might call an old downstream endpoint while the other half call a new one.

A common deployment pattern is to make the pod template change whenever configuration changes. The Deployment then performs a controlled rolling replacement.

One implementation is to include a configuration hash in pod-template annotations:

template:
  metadata:
    annotations:
      configuration-checksum: "sha256:8b95..."
  spec:
    containers:
      - name: api
        image: registry.example.com/shipment-api:7.3.1

When configuration changes, the checksum changes. Because the pod template is different, the Deployment creates a new ReplicaSet and rolls out new pods.

This connects configuration versioning with normal Kubernetes deployment behavior. More about rolling updates and ReplicaSets can be found here: Deployments, ReplicaSets, and StatefulSets.

Immutable and Versioned Configuration

Another approach is creating a new configuration object for each release:

shipment-api-config-v17
shipment-api-config-v18
shipment-api-config-v19

The Deployment explicitly references one version:

envFrom:
  - configMapRef:
      name: shipment-api-config-v19

This creates a clear relationship between running pods and configuration versions.

Advantages:

• configuration history is explicit
• rollback is easier to understand
• existing configuration is not silently mutated
• different ReplicaSets cannot accidentally reinterpret the same mutable object

Disadvantages:

• old objects require cleanup
• deployment automation becomes more important
• version references must remain synchronized across manifests

Immutable configuration is especially useful when predictable deployments are more important than runtime configuration mutation.

Secret Rotation and Failure Handling

Long-lived secrets eventually need replacement because of security policy, credential expiration, certificate renewal, personnel changes, suspected compromise, or routine key rotation.

Rotation becomes difficult when producers and consumers must switch simultaneously. Production designs should avoid creating a moment where neither the old nor the new credential works.

Safe Secret Rotation

Consider rotating a database password used by 30 API pods.

A dangerous sequence is:

1. Database password changed
2. Existing application pods still use old password
3. All new database connections fail
4. Kubernetes secret updated later
5. Pods restarted

This creates an avoidable outage between steps one and five.

When the target system supports overlapping credentials, a safer model is:

Old Credential Valid
        |
        v
Create New Credential
        |
        v
Old + New Valid
        |
        v
Distribute New Credential
        |
        v
Roll / Reload Applications
        |
        v
Verify New Credential Usage
        |
        v
Revoke Old Credential

This is a dual-validity rotation window. It separates credential distribution from revocation and allows rollback while both versions remain valid.

When overlapping credentials are impossible, rotation requires tighter coordination and may need application-specific reconnection behavior.

Short-lived credentials change the model further. Instead of periodically distributing a permanent secret, workloads continuously obtain temporary credentials using their runtime identity. Compromise duration is then bounded by credential lifetime and authorization policy.

Configuration Failure Modes

Configuration failures can propagate faster than application-code failures because one shared object may affect many replicas.

Common failure scenarios include:

Applications should validate required configuration as early as possible. For example, a service can verify types, ranges, URL formats, and required combinations during startup rather than failing only when a specific request reaches an invalid code path.

However, configuration validation should not automatically require every remote dependency to be reachable during startup. Otherwise a temporary dependency outage can prevent all replacement pods from becoming available.

The distinction is important: validate configuration correctness locally; evaluate dependency availability through appropriate runtime health and resilience mechanisms.

Production Design Example

Consider a logistics platform running shipment APIs and carrier-integration workers. The platform connects to PostgreSQL, Redis, a message broker, and several external carrier APIs.

Application configuration changes frequently, while database credentials, carrier tokens, and TLS material require stronger security and rotation controls.

Architecture

                  Deployment Pipeline
                         |
             +-----------+-----------+
             |                       |
             v                       v
      Versioned ConfigMap       Deployment
             |                       |
             +-----------+-----------+
                         |
                         v
                    Application Pods
                         |
              +----------+----------+
              |                     |
              v                     v
       Non-Sensitive Config     Secret Provider
                                     |
                                     v
                           Secret Management System
                                     |
                    +----------------+----------------+
                    |                |                |
                    v                v                v
                 Database        Carrier API      TLS Keys

The application image contains no production credentials. Non-sensitive configuration such as log level, queue names, timeouts, and feature switches is maintained separately from secrets.

Configuration is versioned with the deployment. A release references an explicit configuration version, making the running combination observable:

Application:
shipment-api:7.3.1

Configuration:
shipment-api-config-v19

Secret generation:
database/shipments/42
carrier/fedex/17

This is operationally valuable during incidents because the important question is not only "which image is running?" but "which image, configuration, and credential generation are running together?"

Deployment and Rotation Flow

Suppose the carrier request timeout needs to change from five seconds to three seconds.

1. A new configuration version is created.
2. Schema and policy validation run before deployment.
3. The Deployment template references the new configuration.
4. A rolling update starts.
5. New pods load and validate the configuration.
6. Readiness succeeds before they receive traffic.
7. Error rate, carrier timeouts, and latency are monitored.
8. The rollout continues while metrics remain healthy.
9. The previous configuration version remains available for rollback.

This treats configuration as a release artifact rather than mutable cluster state edited manually during production operation.

Now consider rotating a carrier API credential.

1. A new credential is created in the secret-management system.
2. The old credential remains valid temporarily if the provider supports overlap.
3. The new credential becomes available to application workloads.
4. Pods reload or are gradually replaced.
5. Authentication success is monitored.
6. The new credential is verified across all replicas.
7. The old credential is revoked.

If the new credential is invalid, rollout can stop while existing replicas continue using the old credential during the overlap period.

Monitoring should include:

• configuration version running on each workload
• configuration validation failures
• secret retrieval failures
• authentication and authorization failures
• credential expiration time
• certificate expiration time
• configuration rollout status
• pods using obsolete configuration
• secret rotation completion
• unexpected configuration changes

Auditability is equally important. Production systems should make it possible to determine who changed configuration, which version was deployed, when it changed, which workloads consumed it, and whether the change succeeded.

Common Mistakes

Configuration and secrets are often treated as secondary deployment details, but failures in this layer can affect every replica simultaneously. Production designs should minimize uncontrolled mutation and make changes explicit, validated, and reversible.

Production Checklist

Configuration management should make application behavior reproducible while secret management should minimize credential exposure and make rotation routine rather than exceptional.

• Separate code from configuration. Use the same application image across environments where practical.
• Classify configuration. Keep sensitive values in Secret-oriented workflows rather than ConfigMaps.
• Keep production secrets out of source control. Store references or encrypted deployment material instead of plaintext credentials.
• Apply least privilege. Allow workloads to access only the credentials they actually require.
• Validate configuration before rollout. Check required keys, types, ranges, formats, and incompatible combinations.
• Version important configuration. Make the relationship between application and configuration releases observable.
• Control configuration rollouts. Treat behavior-changing configuration as a production deployment.
• Preserve rollback capability. Keep the previous known-good configuration available until the rollout is verified.
• Design secret rotation before incidents. Document how credentials can be replaced without widespread downtime.
• Prefer short-lived credentials. Use workload identity where the surrounding platform supports it.
• Monitor credential expiration. Alert before tokens and certificates become invalid.
• Audit sensitive access. Track administrative and workload access to secret-management systems.
• Avoid leaking secrets. Redact credentials from logs, errors, metrics, traces, and diagnostic endpoints.
• Test reload behavior. Verify whether configuration and secret updates require pod replacement or can be safely applied dynamically.
• Test failure recovery. Practice invalid configuration rollback, expired credentials, denied secret access, and failed rotation.

Conclusion

ConfigMaps and Secrets separate runtime settings from application images, but the Kubernetes objects themselves are only the beginning of a production configuration strategy. Versioning, rollout behavior, authorization, validation, rotation, auditing, and recovery determine whether configuration remains manageable at scale.

For non-sensitive settings, favor explicit and reproducible configuration releases. For sensitive values, minimize plaintext copies, restrict access, support rotation, and prefer short-lived identity-based credentials where the surrounding infrastructure allows it.

Key Takeaway: treat configuration and secrets as production dependencies with their own lifecycle. Every important value should have clear ownership, controlled distribution, validation, observable versioning, safe rotation, and a tested rollback path.

More Articles to Read

• Kubernetes Explained: Pods, Nodes, and Clusters
• Deployments, ReplicaSets, and StatefulSets
• Services, Ingress, and Networking
• Designing Highly Available Kubernetes Applications
• Autoscaling Kubernetes Workloads
• Kubernetes Best Practices for Production