Designing a Notification System

A notification system delivers messages to users through channels such as mobile push, email, and SMS. Sending one notification is straightforward. Sending millions reliably becomes a distributed-systems problem involving asynchronous processing, fan-out, user preferences, provider rate limits, retries, deduplication, prioritization, and delivery tracking.

The central design principle is to separate notification creation from notification delivery. Product services should not wait for external email, SMS, or push providers. They submit a notification request to a durable processing pipeline, while independently scalable workers handle delivery through each channel.

Table of Contents

• Requirements and Scale Estimation
• API and Data Model
• High-Level Architecture
• Routing, Preferences, and Priority
• Retries, Deduplication, and Delivery Guarantees
• Provider Failures and Backpressure
• Production Design Example
• Common Mistakes
• Interview Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

Requirements and Scale Estimation

Start by narrowing the problem. A reasonable interview scope supports three channels:

• mobile push notifications;
• email;
• SMS.

Functional requirements might include:

• accept notification requests from internal services;
• support one or several delivery channels;
• use reusable message templates;
• respect user notification preferences;
• support different priorities;
• retry transient delivery failures;
• track notification status.

Important non-functional requirements include high availability, durable processing, horizontal scalability, controlled duplicate delivery, and low latency for critical notifications.

Assume:

Daily active users:              50 million
Notifications/user/day:         10
Notifications/day:              500 million
Peak multiplier:                5×
Average internal event:         1 KB

Average notification throughput is:

500,000,000 / 86,400
≈ 5,800 notifications/second

Peak throughput becomes:

5,800 × 5
≈ 29,000 notifications/second

However, one logical notification may produce several physical deliveries:

Order shipped
      |
      +--> Push
      +--> Email
      +--> SMS

1 notification
3 delivery attempts

If the average notification produces 1.5 channel deliveries:

29,000 × 1.5
≈ 43,500 peak delivery operations/second

This distinction between notification throughput and delivery throughput matters because provider-facing workers scale according to deliveries rather than incoming logical events.

For more about deriving these numbers, see Estimating Scale and Capacity Planning.

API and Data Model

Internal services should interact with the notification platform through a small API or event contract.

For example:

POST /notifications

{
  "recipient_id": "user_8172",
  "type": "order_shipped",
  "template_id": "order_shipped_v3",
  "channels": ["push", "email"],
  "priority": "high",
  "data": {
    "order_id": "ORD-82913",
    "tracking_number": "TRK-12882"
  }
}

The API should normally acknowledge that the notification was accepted for processing, not claim that it has already been delivered:

202 Accepted

{
  "notification_id": "ntf_924810",
  "status": "queued"
}

This keeps slow external providers outside the synchronous request path.

A useful logical model separates a notification from its individual channel deliveries:

Notification

notification_id
recipient_id
type
template_id
priority
created_at


Delivery

delivery_id
notification_id
channel
provider
status
attempt_count
next_attempt_at
provider_message_id

One notification can therefore create multiple delivery records:

Notification N123
      |
      +--> Delivery D1 --> Push
      |
      +--> Delivery D2 --> Email
      |
      +--> Delivery D3 --> SMS

This separation makes retries, provider status, channel-specific failures, and delivery analytics easier to represent.

Templates should also remain separate from application code when content changes independently:

Template: order_shipped

Push:
"Order {{order_id}} has shipped"

Email:
"Order {{order_id}} is on its way.
Tracking number: {{tracking_number}}"

SMS:
"Order {{order_id}} shipped. Track: {{tracking_number}}"

High-Level Architecture

The basic design separates producers, notification processing, queues, channel workers, and external providers.

                 Product Services
             /        |         \
            v         v          v
        Orders      Chat       Security
            \         |          /
             \        |         /
              v       v        v
              +----------------+
              | Notification   |
              | API            |
              +----------------+
                       |
                       v
              +----------------+
              | Notification   |
              | Processor      |
              +----------------+
                       |
             +---------+---------+
             |         |         |
             v         v         v
          Push       Email      SMS
          Queue      Queue      Queue
             |         |         |
             v         v         v
          Push       Email      SMS
         Workers    Workers    Workers
           / \         |         |
          v   v        v         v
       APNs  FCM   Email API   SMS API

The Notification API should remain relatively lightweight. Its responsibilities can include validation, authentication, idempotency checks, persistence, and durable publication into the processing pipeline.

The queue is a critical architectural boundary. Producers can continue accepting notifications while providers are temporarily slow, and each delivery channel can scale independently.

Suppose email suddenly requires 100,000 deliveries per second while SMS requires only 2,000:

                 Notification Pipeline
                   /             \
                  v               v
             Email Queue       SMS Queue
             100K/sec           2K/sec
                  |               |
             many workers     fewer workers

Independent queues and consumer groups prevent one channel's workload from dictating another channel's capacity.

This is a practical application of asynchronous processing patterns discussed in Background Workers Explained: Designing Reliable Asynchronous Processing.

Routing, Preferences, and Priority

A production notification platform should not blindly send every requested channel. Delivery may depend on user preferences, notification category, urgency, quiet hours, available contact information, and product policy.

Consider:

Requested channels:
Push + Email + SMS

User preferences:
Push:  enabled
Email: enabled
SMS:   disabled

Result:
Push + Email

A preference service can maintain settings such as:

user_8172

transactional:
  push:  true
  email: true
  sms:   true

marketing:
  push:  true
  email: false
  sms:   false

quiet_hours:
  22:00 - 07:00

Frequently accessed preferences can be cached, but changes such as unsubscribing from marketing messages may require tighter consistency than ordinary profile settings.

Priority Queues

Not every notification has the same latency requirement.

An authentication code should not wait behind millions of promotional messages:

BAD:

Single Queue

Marketing
Marketing
Marketing
Marketing
OTP
Marketing
...

A better design separates workloads:

                 Router
                /      \
               v        v
        Critical Queue  Bulk Queue
             |              |
             v              v
        Dedicated       Bulk Workers
         Workers

Priority classes might be:

Separate queues are often safer than implementing strict priority inside one enormous queue because capacity can be reserved explicitly for critical traffic.

Priority should not mean unlimited traffic. A buggy producer generating millions of "critical" events can still overload the system. Rate limits and producer quotas should protect the platform.

Retries, Deduplication, and Delivery Guarantees

External delivery calls fail regularly for transient reasons:

• provider timeout;
• temporary provider outage;
• connection failure;
• rate limiting;
• server-side 5xx errors.

Retrying immediately can make an outage worse.

Provider overloaded
      |
      v
Requests fail
      |
      v
Immediate retries
      |
      v
More provider load
      |
      +------> more failures

Retries should normally use exponential backoff with jitter:

import random


def retry_delay(attempt: int) -> float:
    base = min(2 ** attempt, 60)
    jitter = random.uniform(0, base * 0.25)

    return base + jitter

Not every failure should be retried. Invalid email addresses, invalid phone numbers, revoked device tokens, and permanently rejected messages should normally stop retrying.

Transient failures can follow a sequence such as:

Attempt 1
   |
   | temporary failure
   v
wait 2 sec

Attempt 2
   |
   | temporary failure
   v
wait 4 sec

Attempt 3
   |
   | temporary failure
   v
wait 8 sec

...

Maximum attempts reached
   |
   v
Dead-Letter Queue

For a deeper treatment of retry timing, see Timeouts, Retries, and Exponential Backoff.

Duplicate Delivery

Reliable asynchronous systems often use at-least-once processing. A worker can successfully send a notification and crash before acknowledging the queue message:

Worker receives message
       |
       v
Provider accepts SMS
       |
       v
Worker crashes
before ACK
       |
       v
Queue redelivers message

Sending again can produce a duplicate SMS.

An idempotency key can identify the logical delivery:

notification_id + channel + recipient

N123 + SMS + user_8172

The system records processing state so repeated queue deliveries do not intentionally create additional sends.

However, exactly-once delivery through an external provider is difficult. Consider:

Worker ------ send ------> Provider
Worker <----- timeout ---- Provider

Unknown:
Did provider accept the message?

Retrying may duplicate the message. Not retrying may lose it.

If the provider supports idempotency keys, the same key should be reused across retries. Otherwise, the system must choose the appropriate trade-off according to notification importance.

Delivery guarantees should therefore be described precisely rather than simply claiming "exactly once."

Provider Failures and Backpressure

Queues protect product services from short provider failures, but queues do not create infinite capacity.

Suppose the email provider becomes unavailable while 20,000 email notifications arrive every second.

After 30 minutes:

20,000 × 30 × 60
= 36 million queued emails

When the provider recovers, normal traffic continues arriving.

If workers process exactly 20,000 emails per second:

Incoming:   20,000/sec
Processed:  20,000/sec

Backlog reduction:
0/sec

The queue never catches up.

If recovery capacity is 30,000 per second:

Incoming:           20,000/sec
Processed:          30,000/sec
Backlog reduction:  10,000/sec

36,000,000 / 10,000
= 3,600 seconds
= 60 minutes

Capacity planning must therefore include backlog recovery throughput, not just steady-state throughput.

Provider Rate Limits

Delivery providers frequently enforce quotas. Worker capacity cannot simply scale beyond those limits.

Workers capable of:
50,000 SMS/sec

Provider allows:
10,000 SMS/sec

Effective throughput:
10,000 SMS/sec

A rate limiter should prevent workers from continuously generating rejected requests.

When providers return explicit throttling information, retry behavior should respect it rather than applying generic retry timing.

Provider Failover

Critical channels may use more than one provider:

                SMS Worker
                   |
            +------+------+
            |             |
            v             v
       Provider A     Provider B
        primary       secondary

Failover can improve availability but creates additional complexity:

• provider-specific APIs;
• different rate limits;
• different delivery receipts;
• cost differences;
• duplicate risk during uncertain failures.

A circuit breaker can temporarily stop sending to a provider experiencing sustained failures instead of allowing every worker to repeatedly discover the same outage.

For the broader resilience pattern, see Circuit Breaker vs Bulkhead vs Load Shedding.

Production Design Example

Consider a notification platform serving transactional and marketing traffic through push, email, and SMS.

                         Product Services
                      /       |        \
                     v        v         v
                  Orders    Auth     Marketing
                      \       |        /
                       \      |       /
                        v     v      v
                    +------------------+
                    | Notification API |
                    +------------------+
                             |
                             v
                    +------------------+
                    | Notification DB  |
                    +------------------+
                             |
                             v
                    +------------------+
                    | Routing Service  |
                    +------------------+
                      /       |       \
                     /        |        \
                    v         v         v
              Preferences  Templates  Rate Limits
                     \        |        /
                      \       |       /
                       +------+------+
                              |
              +---------------+---------------+
              |               |               |
              v               v               v
          Push Queues      Email Queues      SMS Queues
         /          \      /          \      /         \
     Critical      Bulk  Critical    Bulk  Critical   Bulk
        |           |       |          |       |        |
        +-----+-----+       +----+-----+       +---+----+
              |                  |                 |
              v                  v                 v
         Push Workers       Email Workers      SMS Workers
          /       \               |             /       \
         v         v              v            v         v
       APNs       FCM        Email Provider   SMS A     SMS B

                              |
                   Provider callbacks/webhooks
                              |
                              v
                     +------------------+
                     | Delivery Tracker |
                     +------------------+

The ingestion flow is intentionally short:

1. Producer submits notification
2. Validate request
3. Check idempotency
4. Persist notification
5. Publish durable processing event
6. Return accepted status

The routing stage then determines actual deliveries:

def build_deliveries(notification, preferences):
    deliveries = []

    for channel in notification.channels:
        if not preferences.allows(
            notification.type,
            channel,
        ):
            continue

        deliveries.append({
            "notification_id": notification.id,
            "recipient_id": notification.recipient_id,
            "channel": channel,
            "priority": notification.priority,
        })

    return deliveries

Each delivery is routed to the appropriate queue. Channel workers can therefore scale independently and implement provider-specific behavior.

A worker might conceptually process a delivery like this:

def process_delivery(delivery):
    if delivery_store.is_completed(delivery.id):
        return

    provider = provider_router.select(delivery.channel)

    try:
        result = provider.send(
            recipient=delivery.recipient,
            payload=delivery.payload,
            idempotency_key=delivery.id,
        )

        delivery_store.mark_sent(
            delivery.id,
            provider_message_id=result.message_id,
        )

    except TemporaryProviderError:
        retry_queue.schedule(
            delivery,
            delay=retry_delay(delivery.attempt),
        )

    except PermanentProviderError as exc:
        delivery_store.mark_failed(
            delivery.id,
            reason=str(exc),
        )

Provider callbacks update later states:

QUEUED
   |
   v
SENDING
   |
   v
SENT
   |
   +--------> DELIVERED
   |
   +--------> BOUNCED
   |
   +--------> FAILED

Not every channel provides the same delivery guarantees. A provider accepting a message does not necessarily mean the end user's device displayed it. Status terminology should reflect what can actually be observed.

The system should monitor both infrastructure and business-level metrics:

Oldest message age is particularly important. A queue containing one million messages may be healthy if workers process them quickly, while a queue containing only 10,000 messages may represent a serious outage if the oldest critical notification has waited 20 minutes.

Common Mistakes

Interview Checklist

• Define channels: establish whether push, email, SMS, and in-app delivery are required.
• Estimate delivery throughput: account for one notification producing several channel deliveries.
• Keep producers asynchronous: do not make product services wait for external providers.
• Separate notification and delivery records: model channel-specific status independently.
• Use durable queues: absorb traffic spikes and temporary provider outages.
• Separate channel workloads: allow push, email, and SMS workers to scale independently.
• Prioritize critical traffic: prevent OTP and security messages from waiting behind bulk campaigns.
• Respect user preferences: evaluate opt-outs, categories, channels, and quiet hours.
• Use idempotency: suppress duplicates caused by producer and consumer retries.
• Classify failures: retry transient failures but stop retrying permanent failures.
• Use backoff and jitter: avoid synchronized retry storms.
• Handle dead letters: isolate messages that repeatedly fail processing.
• Respect provider limits: rate-limit workers according to external delivery capacity.
• Plan provider outages: estimate queue growth and backlog recovery time.
• Track end-to-end latency: monitor how long notifications actually spend in the pipeline.

Conclusion

A scalable notification system is fundamentally an asynchronous delivery pipeline. Product services create notification intent, while queues, routing logic, channel-specific workers, and external providers perform the actual delivery independently.

The difficult problems appear around the happy path: priority inversion, duplicate processing, provider outages, rate limits, retry storms, user preferences, ambiguous delivery outcomes, and large backlogs. Separating channels and priorities allows each workload to scale and fail independently while keeping critical notifications responsive.

Key Takeaway

Design a notification system around durable asynchronous delivery rather than direct provider calls. Separate logical notifications from channel deliveries, isolate priority and channel workloads, respect preferences and provider limits, use idempotency with bounded retries, and capacity-plan for both normal traffic and backlog recovery. Reliability depends as much on controlling failure and overload as on successfully sending messages.

More Articles to Read

• System Design Interviews Explained: What Companies Actually Expect
• Estimating Scale and Capacity Planning
• Designing a URL Shortener
• Designing a Chat Application
• Designing a File Storage Service
• Common System Design Interview Mistakes