Designing a Chat Application

A chat application looks simple from the client perspective: a user sends a message and another user receives it. At scale, that interaction requires persistent connections, message routing, durable storage, ordering, offline delivery, synchronization across multiple devices, and infrastructure capable of maintaining millions of concurrent connections.

The central design challenge is separating two different workloads: real-time message delivery and durable message storage. Connected users expect messages within milliseconds, while disconnected users still expect complete conversation history when they return. A production architecture must support both without making one depend unnecessarily on the other.

Table of Contents

• Requirements and Scale Estimation
• API and Data Model
• Persistent Connections and WebSockets
• Designing the Message Flow
• Message Ordering and Delivery Guarantees
• Offline Users and Multi-Device Synchronization
• Production Design Example
• Common Mistakes
• Interview Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

Requirements and Scale Estimation

A chat system can contain dozens of features, so the interview scope should be narrowed early. A useful design supports:

• one-to-one conversations;
• group conversations;
• real-time message delivery;
• persistent message history;
• offline users;
• multiple devices per user;
• delivery and read status.

Large attachments, voice calls, end-to-end encryption, search, and massive public channels can be treated as separate extensions unless explicitly required.

Important non-functional requirements include low delivery latency, high availability, durable messages, horizontal scalability, and predictable ordering within a conversation.

Assume:

Daily active users:           50 million
Concurrent users:             10 million
Messages/user/day:            40
Messages/day:                  2 billion
Average stored message:       1 KB
Peak multiplier:              4×

Average message creation throughput is:

2,000,000,000 / 86,400
≈ 23,000 messages/second

Peak throughput becomes approximately:

23,000 × 4
≈ 92,000 messages/second

Raw message storage grows by roughly:

2 billion × 1 KB
≈ 2 TB/day

≈ 730 TB/year

Replication, indexes, metadata, backups, and storage-engine overhead increase the physical requirement significantly.

The other important number is not requests per second but concurrent connections:

10 million online users
≈ 10 million persistent connections

If one connection server safely handles 50,000 active connections:

10,000,000 / 50,000
= 200 connection servers

With operational headroom, deployments, and failure capacity, the actual fleet would be larger.

This illustrates why chat systems require both throughput estimation and connection-capacity estimation. The broader methodology is covered in Estimating Scale and Capacity Planning.

API and Data Model

A chat system usually combines HTTP APIs with a persistent real-time protocol.

HTTP can handle operations such as loading conversation history:

GET /conversations/{conversation_id}/messages?before={message_id}&limit=50

Message sending and real-time delivery can use WebSockets:

{
  "type": "message.send",
  "conversation_id": "conv_781",
  "client_message_id": "client_9182",
  "body": "Hello"
}

The server can acknowledge the accepted message:

{
  "type": "message.accepted",
  "client_message_id": "client_9182",
  "message_id": "msg_481927",
  "sequence": 1842
}

The client-generated identifier is useful for idempotency. If a connection fails before the acknowledgement arrives, the client can safely retry the same logical message.

A simplified data model might include:

Conversation

conversation_id
type
created_at


ConversationMember

conversation_id
user_id
joined_at
last_read_sequence


Message

conversation_id
sequence
message_id
sender_id
body
created_at

The primary access pattern is usually:

conversation_id
      +
message sequence/time range
      |
      v
conversation history

That suggests partitioning and indexing messages around the conversation rather than around the sender.

For example, conceptually:

SELECT *
FROM messages
WHERE conversation_id = ?
  AND sequence < ?
ORDER BY sequence DESC
LIMIT 50;

Cursor-based pagination is preferable to large offsets because old conversations can contain millions of messages.

Persistent Connections and WebSockets

Polling is inefficient for real-time chat. A client repeatedly asking whether anything changed generates traffic even when no messages exist:

Client --> Any messages? --> Server
Client --> Any messages? --> Server
Client --> Any messages? --> Server
Client --> Any messages? --> Server

A WebSocket establishes a long-lived bidirectional connection:

Client <======================> Connection Server
           persistent
           WebSocket

The server can immediately push new messages when they become available.

At large scale, clients connect to many connection servers:

                       Load Balancer
                    /       |        \
                   v        v         v
              Gateway 1 Gateway 2 Gateway 3
               /  |       /  |       /  |
              v   v      v   v      v   v
            users       users       users

The load balancer distributes new connections, but once established, each WebSocket remains associated with a particular gateway until it disconnects.

This creates a routing problem. Suppose:

Alice --> Gateway 2
Bob   --> Gateway 9

When Alice sends Bob a message, the system must discover that Bob is currently connected to Gateway 9.

A distributed connection registry can maintain mappings such as:

user_123
  device_phone  --> gateway_9 / connection_829
  device_web    --> gateway_4 / connection_192

user_456
  device_phone  --> gateway_2 / connection_716

Connection state is ephemeral. It changes whenever clients connect, disconnect, switch networks, or reconnect to another server.

The registry therefore needs fast reads and writes but does not need to be treated like permanent conversation history.

Designing the Message Flow

A simple one-to-one message flow can be represented as:

Alice
  |
  v
Gateway A
  |
  v
Message Service
  |
  +------> Message Store
  |
  +------> Message Bus
                |
                v
            Router
                |
                v
           Gateway B
                |
                v
               Bob

The important design decision is where a message becomes durable.

A safe flow is:

1. Receive message
2. Validate membership
3. Assign message ID / sequence
4. Persist message durably
5. Publish delivery event
6. Acknowledge accepted message
7. Route to connected recipients

A message should not be considered safely accepted merely because one application server has it in memory.

The message bus separates durable ingestion from real-time fan-out:

                 Message Service
                       |
                       v
                  Message Bus
                 /     |      \
                v      v       v
            Router  Search   Analytics

Real-time delivery, indexing, analytics, notifications, and other downstream consumers can process the same logical event independently.

This avoids adding every secondary feature to the synchronous message-send path.

One-to-One Fan-Out

For direct messages, the router resolves the recipient's active connections:

def route_message(message):
    connections = connection_registry.get(
        message.recipient_id
    )

    for connection in connections:
        gateway_bus.publish(
            gateway_id=connection.gateway_id,
            event=message,
        )

If no active connection exists, the message remains safely stored and can be synchronized when the recipient reconnects.

Group Chat Fan-Out

Group conversations introduce another scaling dimension.

For a small group:

                   Message
                      |
            +---------+---------+
            |         |         |
            v         v         v
         User B    User C    User D

Fan-out on write is reasonable because the recipient count is small.

For a group with hundreds of thousands of members, immediately generating individual delivery work for every member can create enormous write amplification.

1 message
   |
   v
500,000 members
   |
   v
500,000 delivery operations

Large groups may require a different model: store the message once in a conversation log and let clients consume from their position rather than materializing a separate copy for every user.

The right strategy therefore depends on group size and delivery requirements.

Message Ordering and Delivery Guarantees

Global ordering across the entire chat platform is unnecessary and extremely expensive. What users normally care about is ordering within a conversation.

Suppose two messages arrive concurrently:

Alice: "A"
Bob:   "B"

Different servers may observe them in different orders unless the system establishes a canonical conversation sequence.

One approach assigns monotonically increasing sequence numbers per conversation:

Conversation 781

1840  Hello
1841  How are things?
1842  Good
1843  Great

Clients can then sort and synchronize messages according to this sequence.

The partitioning strategy can help preserve ordering:

partition = hash(conversation_id)

Messages for the same conversation are routed through the same logical partition:

Conversation A ----\
Conversation A -----+--> Partition 3
Conversation A ----/

Conversation B ----\
Conversation B -----+--> Partition 7

This provides an ordering boundary without imposing global serialization on unrelated conversations.

At-Least-Once Processing

Distributed message pipelines commonly use at-least-once delivery.

Consider:

Message Service
      |
      v
Message Bus
      |
      v
Router
      |
      v
Gateway
      |
      v
Client receives message
      |
Gateway crashes before ACK

The event may be processed again.

Clients and services should therefore tolerate duplicate delivery using the stable message_id:

def receive_message(message):
    if local_store.contains(message.id):
        return

    local_store.save(message)
    render(message)

The broader difference between delivery guarantees is covered in At-Most-Once vs At-Least-Once vs Exactly-Once Message Delivery.

Client Retries

Client retries create a similar duplicate problem.

Client sends M1
      |
      v
Server stores M1
      |
      X acknowledgment lost

Client retries M1

Without an idempotency identifier, the server may create two messages.

The client should generate a stable identifier:

client_message_id = 550e8400-e29b-41d4-a716-446655440000

The server stores the relationship:

sender_id + client_message_id
              |
              v
          message_id

Repeated submissions can then return the existing message rather than creating another one.

Offline Users and Multi-Device Synchronization

Real-time delivery is only an optimization over durable message history. If a user is offline, no real-time connection exists:

Alice sends message
       |
       v
Message stored durably
       |
       v
Bob offline
       |
       X no WebSocket delivery

When Bob reconnects, the client synchronizes from its last known position:

Client state:

conversation_781
last_sequence = 1837

Request:

messages after sequence 1837

The server returns:

1838
1839
1840
1841
...

This mechanism is more reliable than trying to keep undelivered messages only in an ephemeral gateway queue.

Multiple Devices

A user may be connected from a phone, laptop, and browser simultaneously:

                    User
               /      |      \
              v       v       v
           Phone    Laptop   Browser
             |         |        |
             v         v        v
         Gateway A Gateway C Gateway F

The connection registry should therefore map a user to multiple active connections, not one server.

Incoming messages can be routed to all active devices. Read state then needs synchronization as well.

Suppose a message is read on the phone:

Phone
  |
  | read through sequence 1842
  v
Read State Service
  |
  +------> Laptop
  |
  +------> Browser
  |
  +------> sender receipt

Storing the latest read sequence is more efficient than storing a separate boolean read record for every message in many designs:

user_123
conversation_781
last_read_sequence = 1842

Every message with a sequence less than or equal to 1842 is implicitly read.

Push Notifications for Offline Users

Offline recipients may need mobile push notifications:

New chat message
      |
      v
Recipient offline?
      |
     yes
      |
      v
Notification System
      |
      v
APNs / FCM

Push delivery should remain outside the core message persistence path. A temporary push-provider outage should not prevent messages from being stored.

The notification pipeline is covered separately in Designing a Notification System.

Production Design Example

A production-oriented chat architecture can combine WebSocket gateways, stateless message services, durable storage, partitioned messaging, connection discovery, and asynchronous downstream processing.

                           Clients
                              |
                              v
                      +----------------+
                      | Load Balancer  |
                      +----------------+
                              |
                  +-----------+-----------+
                  |           |           |
                  v           v           v
              Gateway A   Gateway B   Gateway C
                  |           |           |
                  +-----------+-----------+
                              |
                              v
                    +-------------------+
                    | Message Service   |
                    +-------------------+
                       /             \
                      v               v
              Message Store       Message Bus
                                      |
                               partitioned by
                              conversation_id
                                      |
                                      v
                              Message Router
                               /     |      \
                              v      v       v
                        Gateway A Gateway B Gateway C
                              |
                              v
                           Clients

Connection Registry <------ Gateways

Message Bus
    |
    +------> Notification Service
    |
    +------> Search Indexer
    |
    +------> Analytics

Suppose Alice sends a message to Bob.

Step 1: Alice sends through the existing WebSocket.

Alice --> Gateway A

Step 2: Gateway A forwards the command to the Message Service.

The Message Service validates that Alice belongs to the conversation and checks the client idempotency key.

Step 3: The service assigns the canonical message identity and sequence.

message_id:       msg_900821
conversation_id:  conv_781
sequence:         1842

Step 4: The message becomes durable.

Message Store

partition:
hash(conv_781)

record:
conv_781 / 1842 / msg_900821

Step 5: A delivery event enters the message bus.

partition key = conversation_id

This keeps events from the same conversation on the same logical ordering path.

Step 6: The router resolves Bob's connections.

Connection Registry

user_bob:
  phone --> Gateway B
  web   --> Gateway C

Step 7: The event is delivered to both gateways.

               Message Router
                 /        \
                v          v
          Gateway B    Gateway C
              |            |
              v            v
          Bob phone     Bob web

If Gateway C disappears during delivery, Bob can still recover the message from durable history after reconnecting.

This illustrates a useful principle:

WebSocket delivery
     =
fast path

Message Store
     =
source of durable truth

Real-time delivery improves latency. It should not be the only mechanism guaranteeing that messages eventually appear.

The system should monitor both infrastructure and user-visible behavior:

Percentile latency matters more than averages. A 30 ms average can hide a subset of users waiting several seconds because one gateway, partition, or region is overloaded.

Common Mistakes

Interview Checklist

• Define chat scope: clarify direct messages, groups, history, receipts, and multi-device requirements.
• Estimate concurrent users: persistent connections can dominate gateway capacity.
• Estimate message throughput: calculate average and peak message creation rates.
• Estimate storage: message history can grow into hundreds of terabytes or petabytes.
• Use persistent connections: support low-latency bidirectional communication for active clients.
• Separate connection state from durable state: gateways are ephemeral; message history is not.
• Maintain a connection registry: route messages to the gateways currently serving recipients.
• Persist before relying on real-time delivery: gateway failure should not lose accepted messages.
• Define ordering scope: conversation-level ordering is usually more practical than global ordering.
• Use idempotency: handle client retries and duplicate asynchronous processing.
• Support offline synchronization: recover from a durable conversation position after reconnect.
• Support multiple devices: maintain several active connections and synchronize read state.
• Choose group fan-out carefully: avoid enormous write amplification for very large groups.
• Keep secondary processing asynchronous: notifications, analytics, and search should not block message persistence.
• Monitor user-visible latency: track delivery percentiles, bus lag, reconnects, and synchronization delay.

Conclusion

A scalable chat system combines ephemeral real-time infrastructure with durable conversation storage. WebSocket gateways maintain millions of active connections, while message services, partitioned storage, and asynchronous routing ensure that accepted messages survive gateway failures and disconnected recipients.

The most important design boundaries are conversation-level ordering rather than global ordering, durable storage rather than gateway memory, and synchronization rather than assuming every real-time delivery succeeds. Multi-device users, large groups, retries, and offline clients then become extensions of the same model.

Key Takeaway

Real-time delivery should be fast but disposable; message history should be durable and recoverable. Use persistent connections for active users, store accepted messages independently of connection servers, partition ordering by conversation, make retries idempotent, maintain a distributed connection registry, and allow every client to recover missed messages from a durable synchronization position.

More Articles to Read

• System Design Interviews Explained: What Companies Actually Expect
• Estimating Scale and Capacity Planning
• Designing a URL Shortener
• Designing a Notification System
• Designing a File Storage Service
• Common System Design Interview Mistakes