System Design Interview: How Would You Prevent a Payment from Being Processed Twice?

A payment request times out. The customer sees an error and presses Pay again. The first request may have failed, or it may have successfully charged the card while only the response was lost.

This is a classic system design interview problem because simply saying "use an idempotency key" is not enough. A production design must handle concurrent requests, retries, database failures, payment-provider timeouts, duplicate queue messages, duplicate webhooks, and the uncomfortable state where the system does not know whether money moved.

Table of Contents

• Why Duplicate Payments Happen
• Define the Correctness Requirement
• Use an Idempotency Key
  • Idempotency Key Lifecycle
  • Reject the Same Key with Different Payment Data
• Make Deduplication Atomic
  • Why Check Then Insert Is Broken
  • Use a Database Unique Constraint
• Model Payment as a State Machine
• The Hardest Case: Payment Provider Timeout
  • Treat Unknown as a Real State
  • Extend Idempotency to the Payment Provider
• Handle Asynchronous Processing Safely
  • Avoid the Database-Queue Dual-Write Problem
  • Make Consumers Idempotent
• Webhooks and Reconciliation
• What Does Not Solve the Problem
• Production Design
• How to Answer This in a System Design Interview
• Conclusion
  • More Articles to Read

Why Duplicate Payments Happen

The obvious source of duplicate payments is a customer pressing Pay twice, but UI behavior is only a small part of the problem.

Duplicates naturally appear in distributed systems because communication is unreliable. Consider this sequence:

1. The client sends POST /payments.
2. The payment service sends the charge to a payment provider.
3. The provider successfully charges the card.
4. The response is lost because of a network timeout.
5. The client receives no confirmation.
6. The client retries the payment.

From the client's perspective, retrying is reasonable. From the server's perspective, processing the retry as a new payment can charge the customer twice.

Duplicates can also originate from load-balancer retries, service retries, worker crashes, queue redelivery, webhook redelivery, or an operator replaying failed jobs.

This is why retries must be designed together with idempotency. A retry is safe only when repeating the operation cannot repeat the financial side effect. Retry behavior and retry storms are covered in more detail in Timeouts, Retries, and Exponential Backoff.

Define the Correctness Requirement

An interview answer often starts with "exactly-once payment processing." That is a useful business requirement, but it needs a more precise technical interpretation.

Networks can duplicate requests, queues can redeliver messages, services can crash after performing an operation but before recording completion, and clients can retry after timeouts.

A more practical goal is:

Multiple attempts to execute the same logical payment must produce at most one financial charge, while retries eventually return the result of that same payment.

The system therefore expects duplicate delivery and makes the effect idempotent.

Conceptually:

At-least-once attempts → Idempotent processing → One logical payment effect

This distinction matters in interviews. Trying to eliminate every duplicate request is unrealistic. Designing the system so duplicate requests are harmless is much stronger.

Use an Idempotency Key

An idempotency key identifies one logical payment attempt. The client generates the key before submitting the payment and sends the same key with every retry of that operation.

POST /payments
Idempotency-Key: 86de81ae-95d9-47ef-8cb3-dcc12fa81144

The server persists the key with the payment record. If the same request arrives again, the service finds the existing operation instead of creating another charge.

The key should represent the logical operation, not the individual HTTP request. Generating a new key for every retry defeats the entire mechanism.

Idempotency Key Lifecycle

Consider a customer purchasing order order_7821.

The first request sends:

{
    "idempotency_key": "pay-5b7935d0",
    "order_id": "order_7821",
    "amount": 12999,
    "currency": "USD"
}

The service creates payment payment_991 and associates it with that key.

If the response disappears and the client retries with pay-5b7935d0, the server returns the existing payment instead of initiating another charge.

If the original operation already succeeded, the saved success response can be returned. If it is still processing, the server can return the current state rather than executing the payment again.

Reject the Same Key with Different Payment Data

Idempotency keys can themselves be misused. A client bug might accidentally reuse one key for two different payments.

For example, the original request might contain:

{
    "order_id": "order_7821",
    "amount": 12999,
    "currency": "USD"
}

A later request could reuse the same key with an amount of 49999.

Returning the old result would hide a serious application bug. Processing the new request would violate idempotency.

A practical design stores a hash of immutable request fields:

import hashlib
import json


def payment_request_hash(
    order_id: str,
    amount: int,
    currency: str,
) -> str:
    payload = json.dumps(
        {
            "order_id": order_id,
            "amount": amount,
            "currency": currency,
        },
        sort_keys=True,
        separators=(",", ":"),
    )

    return hashlib.sha256(payload.encode()).hexdigest()

A retry with the same key but a different hash should be rejected as a conflicting request.

Make Deduplication Atomic

Idempotency keys identify duplicates, but identification alone does not prevent concurrent execution. The database must atomically decide which request owns the payment attempt.

Why Check Then Insert Is Broken

A naive implementation might do this:

payment = find_by_idempotency_key(key)

if payment is None:
    payment = create_payment(key)
    charge_card(payment)

This looks correct until two identical requests reach two application instances simultaneously.

Both can execute find_by_idempotency_key() before either creates the record. Both see nothing, both create a payment, and both attempt the charge.

Application-level check-then-insert is not atomic.

Use a Database Unique Constraint

The uniqueness decision should be delegated to a strongly consistent operation such as a relational database unique constraint.

CREATE TABLE payments (
    payment_id UUID PRIMARY KEY,
    merchant_id UUID NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    request_hash VARCHAR(64) NOT NULL,
    order_id UUID NOT NULL,
    amount BIGINT NOT NULL,
    currency CHAR(3) NOT NULL,
    status VARCHAR(32) NOT NULL,
    provider_payment_id VARCHAR(255),
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL,

    UNIQUE (merchant_id, idempotency_key)
);

Now two concurrent requests can attempt the insert, but only one succeeds.

The losing request reads the existing payment and returns its current or final result.

INSERT INTO payments (
    payment_id,
    merchant_id,
    idempotency_key,
    request_hash,
    order_id,
    amount,
    currency,
    status,
    created_at,
    updated_at
)
VALUES (
    :payment_id,
    :merchant_id,
    :idempotency_key,
    :request_hash,
    :order_id,
    :amount,
    :currency,
    'CREATED',
    CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP
)
ON CONFLICT (merchant_id, idempotency_key)
DO NOTHING;

The database is now the concurrency boundary. There is no interval in which two application instances can independently decide that they are both the first request.

This is a broader example of designing safe concurrency using atomic database operations rather than relying on timing assumptions.

Model Payment as a State Machine

Preventing duplicate rows is only part of the solution. A payment is not a single database insert; it is a workflow involving external systems and several possible outcomes.

A simplified state model might contain:

• CREATED — payment intent exists but processing has not started;
• PROCESSING — a provider request has been initiated;
• SUCCEEDED — provider confirmed the payment;
• FAILED — provider definitively rejected the payment;
• UNKNOWN — the provider may have processed the payment, but the result is unavailable.

State transitions should themselves be controlled. For example, two workers should not independently move the same payment from CREATED to PROCESSING.

UPDATE payments
SET status = 'PROCESSING',
    updated_at = CURRENT_TIMESTAMP
WHERE payment_id = :payment_id
  AND status = 'CREATED';

The affected-row count becomes a lightweight compare-and-set operation. Exactly one worker can successfully claim the transition.

Explicit state machines also make recovery easier. A restarted worker can inspect the durable state instead of guessing whether the previous attempt completed.

The Hardest Case: Payment Provider Timeout

The most important interview scenario is not two simultaneous clicks. It is an ambiguous external side effect.

Suppose the payment service sends a charge to the provider. The provider charges the card, but the connection fails before the response reaches the payment service.

The local database still says PROCESSING.

Retrying with a completely new provider request could charge the card again. Marking the payment FAILED would also be incorrect because the first charge may have succeeded.

Treat Unknown as a Real State

A timeout does not mean failure. It means the result is unknown.

The payment can transition to UNKNOWN and enter a recovery path rather than immediately issuing another independent charge.

Recovery can use:

• the provider's payment-status API;
• a provider webhook;
• the provider's transaction identifier;
• a later reconciliation process.

This distinction is fundamental to reliable distributed systems: failure to receive a response is not proof that the remote operation failed. Similar recovery problems are discussed in Failure Recovery in Distributed Systems.

Extend Idempotency to the Payment Provider

Local idempotency prevents the application from intentionally creating two logical payments, but the external provider must also be protected against repeated network requests.

When the provider supports idempotency, the payment service should send a stable provider-side idempotency token derived from the logical payment.

provider_result = provider.charge(
    amount=payment.amount,
    currency=payment.currency,
    idempotency_key=str(payment.payment_id),
)

If the request times out and must be retried, the same provider key is reused.

The protection is therefore end-to-end:

Client key → Payment record → Provider key → One logical charge

Without provider-side idempotency, the service should first query the provider using a stable external reference before deciding whether another charge attempt is safe.

Handle Asynchronous Processing Safely

Payment processing often becomes asynchronous because external providers can be slow or temporarily unavailable. A service may persist the payment intent and enqueue work for background processing.

That introduces another correctness problem: coordinating the database with the message broker.

Avoid the Database-Queue Dual-Write Problem

Consider this sequence:

1. Create the payment in the database.
2. Publish ProcessPayment to a queue.

If the process crashes between these operations, the database contains a payment that will never be processed.

Reversing the order creates the opposite problem: a worker may receive a message for a payment that was never committed.

The transactional outbox solves this by writing the payment and an outgoing event in the same database transaction.

BEGIN;

INSERT INTO payments (
    payment_id,
    merchant_id,
    idempotency_key,
    request_hash,
    order_id,
    amount,
    currency,
    status,
    created_at,
    updated_at
)
VALUES (
    :payment_id,
    :merchant_id,
    :idempotency_key,
    :request_hash,
    :order_id,
    :amount,
    :currency,
    'CREATED',
    CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP
);

INSERT INTO outbox (
    event_id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    :event_id,
    :payment_id,
    'PROCESS_PAYMENT',
    :payload,
    CURRENT_TIMESTAMP
);

COMMIT;

A separate publisher reads the outbox and sends events to the message broker. Publishing can be retried because consumers are designed to tolerate duplicates.

This pattern is useful whenever a service must coordinate local data with events owned by another part of a distributed architecture. Related data-ownership and cross-service consistency problems are covered in Managing Data Across Multiple Services.

Make Consumers Idempotent

Message brokers commonly favor reliable redelivery over pretending that duplicates can never happen. A worker may process a message and crash before acknowledging it, causing the broker to deliver the same message again.

The payment worker therefore needs the same principle as the HTTP API: duplicate delivery must not create a duplicate effect.

The worker can use the payment state and an atomic conditional transition to determine whether processing should proceed.

claimed = repository.transition(
    payment_id=payment_id,
    expected_status="CREATED",
    new_status="PROCESSING",
)

if not claimed:
    return

process_payment(payment_id)

The second delivery finds that the payment is no longer CREATED and does not independently start the same operation.

Webhooks and Reconciliation

Even careful request processing cannot assume that the synchronous provider response is the final source of truth.

Payment providers commonly send webhooks when payment status changes. Webhooks must themselves be treated as duplicate and potentially out-of-order messages.

A provider event ID can be persisted with a unique constraint:

CREATE TABLE processed_provider_events (
    provider_event_id VARCHAR(255) PRIMARY KEY,
    received_at TIMESTAMP NOT NULL
);

Receiving the same webhook twice then produces one logical state transition.

State validation is equally important. A delayed webhook should not incorrectly move a payment backward from a later valid state.

Reconciliation provides another safety layer. A scheduled process compares internal payment records against provider records and identifies discrepancies such as:

• provider says successful while local state remains UNKNOWN;
• local system recorded success but no matching provider transaction exists;
• amount or currency differs;
• unexpected duplicate provider transactions exist;
• payments remain in intermediate states beyond an expected duration.

Real financial systems should assume that some failures escape the synchronous request path. Reconciliation turns silent inconsistencies into detectable operational events.

What Does Not Solve the Problem

Several solutions sound reasonable in an interview but cover only one failure mode.

Another weak approach is detecting duplicates using user_id + amount + timestamp. Two legitimate purchases can have the same amount, while retries may occur outside an arbitrary time window.

The system needs an explicit identity for the logical operation rather than guessing whether two transactions look similar.

Production Design

A robust design uses multiple layers because no single mechanism covers every failure boundary.

1. The client generates an idempotency key for one logical payment attempt.
2. The payment API validates the request and computes a request hash.
3. The database atomically creates a payment using a unique constraint on the merchant and idempotency key.
4. A duplicate request reads and returns the existing payment rather than creating another one.
5. The payment progresses through an explicit state machine.
6. If asynchronous processing is used, a transactional outbox atomically records the payment and processing event.
7. Workers use conditional state transitions so duplicate messages are harmless.
8. The provider request uses a stable provider-side idempotency key.
9. A provider timeout becomes UNKNOWN, not automatically FAILED.
10. Status APIs and idempotent webhooks resolve uncertain outcomes.
11. Reconciliation detects discrepancies that survive the normal request path.

The design should expose operational signals for each layer. Useful metrics include duplicate idempotency-key rate, payments stuck in PROCESSING, payments in UNKNOWN, provider timeout rate, provider latency, webhook processing failures, queue retry count, reconciliation mismatches, and payment completion latency.

Alerts should focus especially on payments remaining in ambiguous states. A rising UNKNOWN count can indicate a provider outage even when the API itself still appears healthy.

Retries, recovery, idempotency, and explicit failure handling are parts of broader production reliability engineering. Additional practices are covered in Reliability Best Practices for Production Systems.

How to Answer This in a System Design Interview

A strong interview answer can begin with the failure scenario rather than jumping directly to technology:

The same logical payment can reach the system multiple times because of double clicks, client retries, network timeouts, queue redelivery, or service crashes. The design should therefore tolerate duplicate attempts instead of assuming exactly-once delivery.

Then build the solution in layers.

1. Give the logical payment an identity. Require a client-generated idempotency key and reuse it for retries.
2. Make the claim atomic. Persist the key with a database unique constraint rather than check-then-insert application logic.
3. Store and replay the result. A retry returns the existing payment instead of executing another charge.
4. Protect concurrent processing. Use explicit states and conditional transitions.
5. Protect the external side effect. Send a stable idempotency key to the payment provider.
6. Handle ambiguity. A timeout means unknown, so resolve it through provider lookup, webhook, or reconciliation before creating another independent charge.
7. Make asynchronous paths idempotent. Expect duplicate messages and webhooks.

This answer demonstrates the important system-design insight: preventing duplicate payments is not a frontend problem or a single database check. It is an end-to-end correctness property across every retry and failure boundary.

Conclusion

Preventing a payment from being processed twice starts with idempotency, but production correctness requires more than an idempotency-key header. The key must be stored durably, claimed atomically, associated with the original request, and propagated through the payment workflow.

The hardest case is an uncertain external result. When a payment provider times out, the system cannot assume failure because money may already have moved. Stable provider identifiers, explicit payment states, webhooks, status lookups, and reconciliation are what make recovery safe.

The central system-design principle is simple: duplicates are inevitable in distributed systems; duplicate financial effects are not. Design every retry boundary so the same logical operation can safely arrive more than once while producing one intended payment.

More Articles to Read

• Timeouts, Retries, and Exponential Backoff
• Failure Recovery in Distributed Systems
• Managing Data Across Multiple Services
• Reliability Best Practices for Production Systems