What Is Canary Deployment?

Canary deployment is a software release strategy in which a new application version is introduced to a small portion of production traffic before being rolled out to everyone. Instead of replacing all running instances simultaneously, teams expose the new version to a limited audience, monitor its behavior, and gradually increase traffic if the release performs as expected. The goal is to detect problems early while limiting their impact.

Canary deployments are commonly used in microservices, Kubernetes, cloud-native applications, and high-availability systems where releases must happen without significant downtime. A successful implementation requires more than routing 5% of requests to a new version: teams also need reliable health metrics, traffic controls, automated rollback, and a plan for database compatibility and long-running sessions.

Table of Contents

• How Canary Deployment Works
• Canary vs. Other Deployment Strategies
• Traffic Routing and Progressive Rollouts
• Monitoring, Release Gates, and Rollback
• Production Design Example: AWS ECS
• Ready-to-Use Example: Kubernetes
• Common Mistakes
• Production Checklist
• Frequently Asked Questions
• Conclusion

How Canary Deployment Works

Consider an API service running version v1 across 20 production instances. A new version, v2, introduces a different database query and a new caching mechanism. Deploying v2 to every instance simultaneously could affect all users if either change introduces a performance regression.

With a canary deployment, the platform keeps v1 serving most requests while introducing v2 to a smaller traffic segment. The team observes the new version under real workloads before deciding whether to continue.

Typical Canary Rollout Lifecycle

A canary release progresses through several controlled stages.

1. Prepare: Build and test the new application version.
2. Deploy: Start a small number of new-version instances alongside the stable version.
3. Expose: Route a small portion of production traffic to the canary.
4. Observe: Compare error rates, latency, resource usage, and business metrics.
5. Expand: Increase the traffic percentage if the new version meets release criteria.
6. Promote: Route all eligible traffic to the new version.
7. Clean up: Retire the previous version after the rollback window closes.

A rollout might progress through 1%, 5%, 20%, 50%, and 100%. These values are examples rather than mandatory stages. The appropriate progression depends on request volume, service criticality, and the confidence needed to detect regressions.

What a 5% Canary Actually Means

If a service receives 10,000 requests per minute and a load balancer routes 5% of eligible requests to the canary, the expected canary volume is approximately 500 requests per minute.

That does not necessarily mean 5% of users. One highly active user may generate hundreds of requests, and routing decisions may be made per request, connection, or session.

For stateful applications, per-request routing can cause a single user to alternate between versions. This may create inconsistent behavior if the versions use incompatible session formats or APIs.

Some systems therefore use user-based or session-based canaries, where selected users consistently receive the new version.

Canary vs. Other Deployment Strategies

Canary deployment is one of several techniques for releasing software with limited disruption. It is often compared with rolling and blue-green deployments, but these approaches solve different parts of the release problem.

Canary vs. Blue-Green Deployment

Blue-green deployment typically maintains two complete environments: one serving production traffic and another containing the new version. Traffic switches from the old environment to the new one when the release is ready.

Canary deployment focuses on progressive exposure. Both versions serve production traffic simultaneously, with the percentage changing over time.

The two strategies can be combined. For example, a blue-green infrastructure setup can shift traffic gradually between target groups instead of switching everything at once.

For a deeper explanation, see What Is Blue-Green Deployment?.

Canary vs. Rolling Deployment

A rolling deployment gradually replaces application instances. However, the percentage of new instances is not always equivalent to the percentage of requests reaching them.

For example, one newly deployed instance may receive disproportionate traffic because of connection reuse, sticky sessions, or uneven load balancing.

A canary deployment deliberately controls traffic exposure and evaluates the new version before promotion. A rolling deployment can incorporate canary-style analysis, but ordinary rolling replacement does not provide that automatically.

Traffic Routing and Progressive Rollouts

Traffic routing is the central mechanism of a canary deployment. A routing layer determines which requests reach the stable version and which reach the canary.

Depending on the platform, routing can happen at an application load balancer, ingress controller, service mesh, API gateway, or application-level feature router.

Weighted Traffic Routing

Weighted routing assigns a relative share of traffic to each version.

For example, the following NGINX configuration uses a weighted upstream split:

split_clients "${remote_addr}${request_uri}" $backend {
    5%      canary;
    *       stable;
}

upstream stable {
    server app-v1:8000;
}

upstream canary {
    server app-v2:8000;
}

server {
    listen 80;

    location / {
        proxy_pass http://$backend;
    }
}

This is an illustrative NGINX configuration. The split is deterministic for the configured input, so it is not a random independent choice for every request. It also does not guarantee that exactly 5% of requests will reach the canary.

Using the client IP and request URI in the hash can distribute different paths from the same user across versions. For session consistency, use a stable user or session identifier instead.

For load-balancing fundamentals, see What Is a Load Balancer?.

Targeted Canary Releases

Not every canary needs percentage-based routing. Teams may first expose a release to internal employees, selected customers, or a specific geographic region.

Targeted canaries are useful when a change affects a particular user segment or requires human validation before broader exposure.

However, a canary restricted to employees may not reproduce production behavior. Internal users may generate less traffic, use different workflows, or lack the data patterns that trigger a regression.

A common approach combines targeted testing with progressively increasing exposure to representative production traffic.

Traffic Distribution Pitfalls

Request percentages can be misleading when traffic is uneven. A 1% canary receiving only a handful of requests per minute may not provide enough evidence to detect a rare failure.

Connection-level routing also matters. With HTTP/2 or long-lived WebSocket connections, traffic distribution may be influenced by persistent connections rather than individual requests.

Canary analysis should measure the actual number and type of requests reaching each version, not only the configured traffic weight.

Monitoring, Release Gates, and Rollback

A canary is valuable only if the team can detect regressions before expanding traffic. Routing 5% of requests to a new version without comparing outcomes is simply a partial deployment.

Release gates define the conditions that must be satisfied before promotion. They should be based on service behavior and user impact rather than deployment health alone.

Metrics That Matter

Useful canary metrics include:

• Error rate: HTTP 5xx responses, failed RPCs, and application exceptions
• Latency: P50, P95, and P99 response times
• Saturation: CPU, memory, connection-pool usage, and queue depth
• Dependencies: Database errors, cache failures, and downstream service timeouts
• Business outcomes: Checkout completion, successful payments, or completed jobs

Compare stable and canary versions over the same time window and, where possible, equivalent request types. A canary serving more expensive endpoints may appear slower even when the new code is not responsible.

Metrics should include a deployment-version label so dashboards and alerts can distinguish stable from canary traffic.

Automated Release Gates

A basic release gate can reject promotion when the canary's error rate exceeds an absolute threshold or significantly exceeds the stable version's error rate.

The following Python function illustrates a simple gate using request counts and error counts:

def can_promote(
    stable_requests: int,
    stable_errors: int,
    canary_requests: int,
    canary_errors: int,
    min_samples: int = 1000,
    max_error_rate: float = 0.02,
    max_regression: float = 0.01,
) -> bool:
    if stable_requests < min_samples:
        return False

    if canary_requests < min_samples:
        return False

    stable_rate = stable_errors / stable_requests
    canary_rate = canary_errors / canary_requests

    return (
        canary_rate <= max_error_rate
        and canary_rate - stable_rate <= max_regression
    )

This example requires at least 1,000 requests per version, a canary error rate no greater than 2%, and an absolute regression of no more than one percentage point.

It is a simplified policy, not a statistically rigorous canary analysis system. Production release gates should account for sample size, baseline variability, traffic mix, seasonality, and the consequences of false positives or false negatives.

For observability design, see Metrics That Actually Matter.

Rollback Strategy

A rollback typically routes new requests back to the stable version and prevents additional exposure to the canary.

However, changing traffic weights does not automatically terminate existing requests, streaming sessions, or WebSocket connections. These may remain attached to canary instances until they complete or disconnect.

Rollback also becomes more complicated when the new version modifies database schemas, writes incompatible data, or changes shared cache formats.

Deployment rollback should therefore be designed together with data compatibility and connection-draining policies.

Production Design Example: AWS ECS

Consider a FastAPI application deployed on Amazon ECS with AWS Fargate. The service receives 3,000 requests per second and must support releases without interrupting customer requests.

The platform uses an Application Load Balancer (ALB) with separate target groups for stable and canary tasks. Both versions share the same database and external dependencies.

ECS Canary Architecture

The deployment consists of:

• Stable target group: ECS tasks running the current production version
• Canary target group: ECS tasks running the candidate version
• ALB listener: Routes requests between target groups using configured weights
• CloudWatch: Collects target-group metrics, application metrics, and alarms
• Deployment controller: Advances traffic weights or restores the stable route

The deployment starts with a small canary task set and gradually increases its traffic allocation after successful health checks.

A target group must have enough healthy capacity for the traffic it receives. Sending 20% of production requests to one underprovisioned canary task can create overload unrelated to the code change.

ALB Weighted Target Groups

Application Load Balancers support weighted forwarding between target groups. The following AWS CLI command illustrates a listener action that sends approximately 95% of requests to stable and 5% to canary.

aws elbv2 modify-listener \
  --listener-arn "$LISTENER_ARN" \
  --default-actions '[
    {
      "Type": "forward",
      "ForwardConfig": {
        "TargetGroups": [
          {
            "TargetGroupArn": "'"$STABLE_TG_ARN"'",
            "Weight": 95
          },
          {
            "TargetGroupArn": "'"$CANARY_TG_ARN"'",
            "Weight": 5
          }
        ]
      }
    }
  ]'

The listener ARN and target group ARNs must refer to existing resources in the same compatible ALB configuration. This command modifies the listener's default action; production setups may instead need to update a specific listener rule.

Weighted forwarding controls request distribution but does not automatically provide a complete deployment workflow. ECS task lifecycle management, monitoring, and rollback must also be coordinated.

For AWS-managed blue-green and canary-style ECS deployment workflows, evaluate the current ECS deployment strategy and CodeDeploy capabilities supported by the chosen service configuration.

Deployment Progress and Rollback

A possible production rollout uses these stages:

1. Deploy the candidate version to a separate ECS task set or service.
2. Verify readiness and dependency connectivity without production traffic.
3. Route 5% of requests to the canary.
4. Evaluate error rates, latency, and business metrics for a defined observation window.
5. Increase to 20%, then 50%, then 100% if release gates pass.
6. Retain stable capacity temporarily until rollback risk is acceptable.

If a release gate fails, the controller restores the stable target group to full traffic weight and investigates the candidate version.

Database changes should use backward-compatible migrations. For example, add a new nullable column before deploying code that writes it, and remove old fields only after all previous-version instances have been retired.

For broader traffic-management patterns, see Traffic Routing Strategies for Zero-Downtime Deployments.

Ready-to-Use Example: Kubernetes

The following example demonstrates a canary rollout on Kubernetes using the community ingress-nginx controller. It deploys two versions of an HTTP service, exposes them through separate Services, and controls the canary percentage through an Ingress annotation.

This example requires a running Kubernetes cluster and an installed ingress-nginx controller that supports canary annotations. The configuration is specific to ingress-nginx; it is not portable to every Kubernetes Ingress controller.

Kubernetes Deployment Manifests

Save the following as canary.yaml. It uses two NGINX deployments with different response headers to identify which version handled each request.

apiVersion: v1
kind: ConfigMap
metadata:
  name: stable-nginx
data:
  default.conf: |
    server {
      listen 80;
      location / {
        add_header X-App-Version stable always;
        return 200 "stable\n";
      }
    }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: canary-nginx
data:
  default.conf: |
    server {
      listen 80;
      location / {
        add_header X-App-Version canary always;
        return 200 "canary\n";
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-stable
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
      version: stable
  template:
    metadata:
      labels:
        app: demo
        version: stable
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
      volumes:
        - name: config
          configMap:
            name: stable-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
      version: canary
  template:
    metadata:
      labels:
        app: demo
        version: canary
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/conf.d
      volumes:
        - name: config
          configMap:
            name: canary-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: demo-stable
spec:
  selector:
    app: demo
    version: stable
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: demo-canary
spec:
  selector:
    app: demo
    version: canary
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-stable
spec:
  ingressClassName: nginx
  rules:
    - host: canary.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo-stable
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
spec:
  ingressClassName: nginx
  rules:
    - host: canary.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo-canary
                port:
                  number: 80

The stable and canary Services select different pods using version labels. The Ingress controller uses the canary annotation to direct approximately 5% of matching requests to the canary backend.

The actual distribution may vary with request volume, controller configuration, and routing behavior. The percentage should be measured rather than assumed to be exact.

Deploy and Test the Canary

Apply the manifests:

kubectl apply -f canary.yaml

kubectl rollout status deployment/demo-stable
kubectl rollout status deployment/demo-canary

For a local test, forward the ingress-nginx controller's HTTP port to your machine. The following command assumes the controller is installed in the ingress-nginx namespace with the standard Service name:

kubectl -n ingress-nginx port-forward \
  service/ingress-nginx-controller \
  8080:80

In another terminal, send 200 requests and count the returned versions:

for i in $(seq 1 200); do
  curl -fsS \
    -H "Host: canary.local" \
    http://127.0.0.1:8080/
done | sort | uniq -c

With a 5% canary weight, the expected distribution is approximately 190 stable responses and 10 canary responses, although the observed values will vary.

The test verifies routing behavior, not application correctness. Real release validation should also examine latency, error rates, resource usage, and business metrics.

Increase Traffic and Roll Back

After verifying the canary, increase its traffic weight to 20%:

kubectl annotate ingress demo-canary \
  nginx.ingress.kubernetes.io/canary-weight="20" \
  --overwrite

Run the same request-count test again and compare the results.

To stop sending new requests to the canary, change the weight to zero:

kubectl annotate ingress demo-canary \
  nginx.ingress.kubernetes.io/canary-weight="0" \
  --overwrite

Setting the weight to zero is a routing rollback for new requests. It does not undo database changes, terminate all existing connections, or restore previous application state.

After the rollout, remove the demonstration resources:

kubectl delete -f canary.yaml

Automating Progressive Rollouts

Manually adjusting traffic weights is useful for learning, but production teams generally automate the process.

Tools such as Argo Rollouts and Flagger can integrate traffic management with metrics-based promotion and rollback. They support staged releases and automated analysis, subject to the capabilities of the configured traffic provider.

A production pipeline should wait for sufficient observations at each stage, evaluate the correct version-specific metrics, and stop or roll back when release gates fail.

For Kubernetes deployment fundamentals, see Deployments, ReplicaSets, and StatefulSets.

Common Mistakes

Canary deployments reduce release risk only when traffic exposure, monitoring, and rollback behavior are designed together. Several mistakes can make a canary appear successful while serious problems remain undetected.

Promoting with Insufficient Traffic

A canary that receives only a few dozen requests may not reveal intermittent failures or tail-latency regressions.

Require enough representative traffic and observation time before promotion. Low-traffic services may need longer analysis windows or additional synthetic tests.

Monitoring Only Health Checks

A service can return HTTP 200 from its health endpoint while checkout failures, slow queries, or background processing errors increase.

Monitor real request outcomes and business metrics, not just pod readiness or process availability.

Deploying Incompatible Database Changes

During a canary rollout, old and new application versions run simultaneously. A database migration that immediately removes a field required by the old version can break stable traffic before the canary is promoted.

Use expand-and-contract migrations and keep schema changes backward compatible across the deployment window.

Assuming Configured Weights Equal Actual Exposure

Sticky sessions, persistent connections, geographic traffic patterns, and endpoint-specific routing can distort traffic distribution.

Measure actual requests per version and compare similar workloads.

Treating Traffic Rollback as Complete Recovery

Returning traffic to the stable version may not reverse database writes, external API calls, or messages already published by the canary.

Plan rollback at the application and data levels, including compensating operations when necessary.

Production Checklist

Before enabling a canary release, confirm that the new and stable versions can operate safely together and that the deployment controller has reliable evidence for promotion or rollback.

• Traffic control: Support gradual, measurable routing between stable and canary versions.
• Capacity: Provision enough healthy canary instances for the assigned traffic.
• Metrics: Compare version-specific errors, latency, saturation, and business outcomes.
• Release gates: Require sufficient samples and explicit promotion criteria.
• Rollback: Provide a tested mechanism to restore stable routing quickly.
• Data compatibility: Ensure both versions work with the current database schema and shared data formats.
• Connections: Account for sticky sessions, streaming requests, and connection draining.
• Automation: Stop or reverse promotion when health signals deteriorate.
• Observability: Label logs, metrics, and traces by application version and deployment ID.

Frequently Asked Questions

Canary deployment is conceptually simple, but implementation details affect how much risk it actually removes from a release.

What Percentage of Traffic Should a Canary Receive?

There is no universal starting percentage. Teams often begin with 1% to 5%, but low-traffic services may need a larger share to collect meaningful data.

The appropriate percentage depends on service criticality, request volume, canary capacity, and the cost of a potential failure.

Does Canary Deployment Guarantee Zero Downtime?

No. Canary deployment can reduce downtime risk, but it does not eliminate failures caused by infrastructure problems, incompatible database changes, or incorrect routing.

Zero-downtime releases also require readiness checks, sufficient capacity, graceful shutdown, and backward-compatible changes.

Is Canary Deployment the Same as Feature Flags?

No. Canary deployment controls which application version receives traffic. Feature flags control which functionality is enabled within an application version.

They can be combined. A new version may be deployed to all servers while a feature flag exposes a new feature to only a small user segment.

Does Kubernetes Support Canary Deployment Natively?

Kubernetes Deployments support rolling updates, but Kubernetes does not provide a universal built-in mechanism for precise weighted canary routing across arbitrary traffic providers.

Weighted canary releases commonly use an ingress controller, service mesh, gateway, or progressive-delivery controller.

Can a Canary Deployment Be Rolled Back Automatically?

Yes. Progressive-delivery tools can monitor metrics and automatically restore traffic to the stable version when release criteria fail.

Automatic traffic rollback does not necessarily reverse state changes already made by the canary. Data and external side effects require separate recovery strategies.

Conclusion

Canary deployment reduces software release risk by exposing a new version to a limited portion of production traffic before a full rollout. It gives teams an opportunity to observe real user behavior, identify regressions, and stop a problematic release before it affects the entire system.

The most important part of a canary deployment is not the traffic percentage itself. It is the combination of representative traffic, reliable metrics, explicit promotion criteria, and safe rollback. Without those controls, gradual routing alone provides limited protection.

Key Takeaway: Canary deployment is a progressive release strategy that turns production rollout into a series of measurable decisions. Start with limited exposure, validate the new version against meaningful health and business metrics, and increase traffic only when the evidence supports promotion.

More Articles to Read

These related articles explore deployment strategies, reliability, and production traffic management.

• Health Checks, Readiness, and Liveness Probes
• Kubernetes Best Practices for Production