What Is an Idempotency Key?

5.0 out of 5 from 1 votes
By Girlway — Published on
1 Likes
0 Dislikes
What Is an Idempotency Key?
What Is an Idempotency Key?

An idempotency key is a unique value attached to a request so a server can recognize repeated attempts to perform the same logical operation and avoid executing it more than once.

Idempotency keys are especially important for operations with side effects: creating payments, submitting orders, booking reservations, provisioning resources, or publishing jobs. Networks fail, clients retry, workers restart, and responses get lost. Without idempotency protection, a perfectly reasonable retry can accidentally create a duplicate operation.

Table of Contents

Why Idempotency Keys Exist

Consider a checkout service receiving a request to create a $100 payment.

Client → POST /payments → Payment Service

The service successfully charges the customer and stores the payment. Before the HTTP response reaches the client, the network connection fails.

The client sees only an error:

Client → POST /payments
             |
             +→ Payment created
             |
             X Response lost

The client cannot know whether the request failed before or after the payment was created.

Retrying is reasonable:

Client → POST /payments → Payment Service

But if the server treats the retry as a completely new request, the customer may be charged twice.

The problem is not specific to HTTP. The same ambiguity appears when workers crash, queues redeliver messages, databases time out, or one service loses its response from another service.

An idempotency key gives repeated attempts a shared identity.

Attempt 1 → Idempotency-Key: payment-7f82
Retry 1   → Idempotency-Key: payment-7f82
Retry 2   → Idempotency-Key: payment-7f82

The server can recognize that all three requests represent the same logical operation.

How an Idempotency Key Works

The client generates a unique key before sending an operation:

POST /payments
Idempotency-Key: 8c77c3cb-cc9d-49d5-a83c-283a04be65a2

The request might contain:

{
    "order_id": "ord_82914",
    "amount": 10000,
    "currency": "USD"
}

When the server receives the request, it checks its idempotency store.

If the key does not exist, the server reserves it and performs the operation.

If processing succeeds, the result is associated with the key:

8c77c3cb... → payment pay_91827 → HTTP 201

If the client retries using the same key, the server does not create another payment. It returns the previously recorded result.

First request → Execute operation → Save result
Retry         → Find saved result → Return result

The client therefore receives the same logical outcome even when the physical request is transmitted multiple times.

This is particularly useful with at-least-once delivery, where duplicate attempts are expected. Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once explains why reliable delivery frequently requires consumers to tolerate duplicates.

Idempotency vs Deduplication

Idempotency and deduplication are closely related, but they describe slightly different ideas.

Deduplication identifies repeated messages or requests and prevents unnecessary duplicate processing.

Idempotency guarantees that repeating the same logical operation does not produce additional unintended effects.

For example, receiving the same payment request twice might be detected through its idempotency key. The server deduplicates the second execution so the overall payment operation remains idempotent.

Concept Main Question
Deduplication Has this request or message already been seen?
Idempotency Can this logical operation safely be attempted again?

The implementation often combines both ideas. A unique operation identifier detects duplicates, while application logic ensures that duplicates do not repeat side effects.

Idempotency and Deduplication in Distributed Systems covers the broader architectural patterns behind both concepts.

Generating Idempotency Keys

An idempotency key should uniquely identify one logical operation.

A UUID is a common choice:

import uuid

idempotency_key = str(uuid.uuid4())

The client generates the key once and reuses it for every retry of that operation.

A new business operation must receive a new key:

Create payment A → key_123
Retry payment A  → key_123
Retry payment A  → key_123

Create payment B → key_456

Generating a new key for every HTTP attempt defeats the mechanism:

Attempt 1 → key_123
Retry 1   → key_456
Retry 2   → key_789

From the server's perspective, those are three unrelated operations.

Some systems derive keys from business identifiers:

capture-payment:order_82914

This can work when the business rule guarantees exactly one such operation for that identifier. It becomes dangerous when multiple legitimate operations are possible.

For example, using only customer_id as the key for payments would incorrectly treat every future payment from that customer as a duplicate.

The key's scope must therefore match the scope of the operation.

Storing Idempotency Records

The server needs durable state that associates an idempotency key with processing information or a completed result.

A simple relational schema might look like:

CREATE TABLE idempotency_keys (
    key             VARCHAR(255) PRIMARY KEY,
    request_hash    VARCHAR(64) NOT NULL,
    status          VARCHAR(32) NOT NULL,
    resource_id     VARCHAR(255),
    response_code   INTEGER,
    response_body   JSONB,
    created_at      TIMESTAMP NOT NULL,
    expires_at      TIMESTAMP NOT NULL
);

There are several useful storage strategies.

Store the Result

The simplest replay behavior is to store the original response.

key_123
  status        = completed
  response_code = 201
  response_body = {"payment_id": "pay_91827"}

A retry can return that result without repeating the business operation.

This is useful when response reproduction matters, but large response bodies increase storage requirements.

Store a Resource Reference

Instead of storing the entire response, the idempotency record can reference the resource created by the first request.

key_123 → payment_id=pay_91827

The retry loads the payment and reconstructs the response.

This reduces duplication in the idempotency store and works well when an operation creates a persistent resource.

Store Minimal Metadata

For deterministic operations, the system may need only the key, request fingerprint, state, and timestamps.

This minimizes storage but requires confidence that the response or result can be reconstructed safely.

The right model depends on whether the operation creates a resource, how expensive responses are to store, and how closely retries must reproduce the original response.

Handling Concurrent Requests

A simple "check whether the key exists, then insert it" implementation contains a race condition.

Two identical requests can arrive almost simultaneously:

Request A → Check key → Missing
Request B → Check key → Missing
Request A → Execute
Request B → Execute

Both requests observed the key before either created it.

The idempotency reservation must therefore be atomic.

A relational database can use a unique constraint and atomic insert:

INSERT INTO idempotency_keys (
    key,
    request_hash,
    status,
    created_at,
    expires_at
)
VALUES (
    'key_123',
    'hash...',
    'processing',
    NOW(),
    NOW() + INTERVAL '24 hours'
)
ON CONFLICT DO NOTHING;

Only one concurrent request can successfully create the record because key is the primary key.

The winner performs the operation. Other requests inspect the existing record and determine whether the operation is still processing or has completed.

A simplified Python flow might look like:

def create_payment(request, idempotency_key):
    record = reserve_key(
        key=idempotency_key,
        request_hash=hash_request(request),
    )

    if not record.created:
        return handle_existing_request(record)

    try:
        payment = execute_payment(request)
        complete_key(idempotency_key, payment.id)
        return payment
    except Exception:
        handle_failure(idempotency_key)
        raise

The atomic reservation is critical. Without it, the idempotency table may record duplicates after they have already happened.

Request Payload Validation

A client should not be allowed to reuse the same idempotency key for a different operation.

Consider:

key_123 → amount=$100
key_123 → amount=$500

If the server simply finds key_123 and returns the original result, the second request may receive a response for an operation it did not actually request.

A common solution is to store a fingerprint of the relevant request fields.

import hashlib
import json

def request_hash(payload: dict) -> str:
    canonical = json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
    )

    return hashlib.sha256(
        canonical.encode("utf-8")
    ).hexdigest()

When a retry arrives, the server compares the new fingerprint with the stored one.

if existing.request_hash != request_hash(payload):
    raise IdempotencyKeyConflict()

The API can then reject reuse of the same key with different parameters rather than silently returning an unrelated result.

Only fields defining the logical operation should participate in the fingerprint. Volatile metadata such as request timestamps or tracing headers can otherwise cause legitimate retries to appear different.

Failure Scenarios

The difficult part of idempotency is not the successful path. It is deciding what the idempotency record means when failures occur between multiple state changes.

Response Lost After Success

This is the classic scenario:

1. Reserve idempotency key
2. Create payment
3. Store completed result
4. Send HTTP response
5. Response is lost

The client retries with the same key.

Because the idempotency record is already completed, the server returns the recorded result without executing the payment again.

This is the easy failure case because durable state clearly shows that the operation succeeded.

Process Crashes During Execution

A more difficult failure happens when the server crashes between the business operation and idempotency completion.

1. Reserve key
2. Create payment
3. Process crashes
4. Mark key completed ← never happens

The database might now contain:

key_123 → processing

But the payment may already exist.

Blindly executing the operation again can create a duplicate. Blindly returning success is also incorrect because the operation may have failed before completion.

This ambiguity should be reduced through transaction boundaries whenever possible.

If both the business record and idempotency record live in the same transactional database, they can be committed atomically:

BEGIN;

INSERT INTO payments (...);

UPDATE idempotency_keys
SET status = 'completed',
    resource_id = 'pay_91827'
WHERE key = 'key_123';

COMMIT;

Either both changes commit or neither does.

External side effects are harder. A payment provider, email service, or third-party API cannot normally participate in the local database transaction.

Those dependencies should ideally support their own idempotency mechanism, or the application needs a workflow that can reconcile uncertain outcomes.

This is a general distributed-systems problem rather than something an idempotency table alone can solve.

Idempotency Keys in Message Processing

Idempotency keys are not limited to HTTP APIs. Message consumers often need the same protection because queues commonly provide at-least-once delivery.

Suppose an order event is delivered twice:

order.created / event_9182
order.created / event_9182

The consumer can treat the event ID as an idempotency key.

def consume(event):
    if processed(event.id):
        acknowledge(event)
        return

    process_order(event)
    mark_processed(event.id)
    acknowledge(event)

The implementation still needs correct transaction boundaries. A crash between process_order() and mark_processed() can cause the operation to execute again.

For database-backed side effects, the processed-message record and business changes can often share one transaction.

BEGIN;

INSERT INTO shipments (...);

INSERT INTO processed_messages (message_id)
VALUES ('event_9182');

COMMIT;

A unique constraint on message_id prevents concurrent consumers from successfully committing the same event twice.

This becomes especially important when messages are recovered from a Dead Letter Queue. Replaying a failed message must be assumed to be duplicate-capable.

Dead-Letter Queues, Retries, and Poison Messages explains how failed messages move through retries, DLQs, and replay workflows.

Expiration and Retention

Idempotency records usually do not need to exist forever.

If an API guarantees that keys remain valid for 24 hours, records can expire after that period:

key_123
created_at = 2026-09-17 10:00
expires_at = 2026-09-18 10:00

After expiration, reuse of the key may be treated as a new operation.

The retention window should be longer than the realistic retry window.

If mobile clients can remain offline for several days and retry later, a one-hour retention period is probably too short. If internal workers retry for no more than 30 minutes, retaining every key for years is unnecessary.

Retention also affects storage capacity.

At 10,000 operations per second:

10,000 × 86,400 = 864,000,000 records/day

Keeping full responses for every operation can become expensive quickly.

High-volume systems may use TTL-enabled storage, partition records by expiration time, store compact metadata, or limit idempotency protection to operations where duplicate side effects matter.

Expiration should be explicit in the API contract. Otherwise clients may incorrectly assume that a key remains protected indefinitely.

Production Design Example

Consider an order API that must prevent customers from creating duplicate orders when checkout requests are retried.

The client generates an idempotency key before submission:

POST /orders
Idempotency-Key: checkout-b83270e7

The service calculates a request fingerprint and attempts to reserve the key.

def create_order(payload, key):
    fingerprint = request_hash(payload)

    record = idempotency_repository.reserve(
        key=key,
        request_hash=fingerprint,
    )

    if record.completed:
        return load_order(record.resource_id)

    if record.request_hash != fingerprint:
        raise IdempotencyKeyConflict()

    if not record.acquired:
        raise OperationAlreadyInProgress()

    return create_order_transactionally(
        payload=payload,
        idempotency_key=key,
    )

The order and completed idempotency state are written in one database transaction:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    total,
    status
)
VALUES (
    'ord_91827',
    'customer_42',
    12500,
    'created'
);

UPDATE idempotency_keys
SET status = 'completed',
    resource_id = 'ord_91827'
WHERE key = 'checkout-b83270e7';

COMMIT;

If the response disappears after the transaction commits, the client retries with the same key.

The service finds the completed record and returns ord_91827 rather than creating another order.

Now consider two simultaneous submissions caused by a user double-clicking the checkout button.

Request A ─┐
           ├→ reserve checkout-b83270e7
Request B ─┘

The unique constraint allows only one request to acquire the key. The second request sees that the operation is already processing or completed.

The design therefore protects against both sequential retries and concurrent duplicate requests.

If order creation also needs to publish an event, committing the order and publishing directly to a broker creates another consistency boundary. Transactional Outbox Pattern for Reliable Messaging explains how to reliably connect database changes with asynchronous event publication.

Common Idempotency Key Mistakes

Idempotency mechanisms often look correct during normal testing while still containing race conditions or failure windows.

  • Generating a new key for every retry. The server cannot recognize attempts as the same operation.
  • Using a key that is too broad. Multiple legitimate operations are incorrectly treated as duplicates.
  • Checking and inserting separately. Concurrent requests can both pass the existence check before either reserves the key.
  • Ignoring request differences. The same key can accidentally be reused with a different payload.
  • Marking completion before the business operation commits. Retries may receive a successful result for work that never completed.
  • Marking completion too late. A crash can leave a completed side effect associated with a permanently processing key.
  • Assuming external APIs are transactional. Local rollback cannot undo a side effect already accepted by another system.
  • Keeping keys for too little time. Late retries can execute as new operations after the protection expires.
  • Keeping everything forever. High-throughput services can accumulate enormous idempotency tables unnecessarily.

Idempotency should be designed around the complete failure path, including concurrency, database commits, external calls, process crashes, retries, and expiration.

The key itself does not make an operation idempotent. The state transitions and transaction boundaries around that key do.

Conclusion

An idempotency key gives repeated attempts of the same logical operation a stable identity. The server can use that identity to recognize retries and prevent side effects such as duplicate payments, orders, bookings, or jobs.

A production implementation needs more than storing a UUID. Key reservation must be atomic, reused keys should be validated against the original request, completed results need durable storage, retention must match retry behavior, and ambiguous failures need carefully designed transaction boundaries.

The practical goal is simple: clients and workers should be able to retry uncertain operations without turning normal distributed-system failures into duplicate business actions.

Author

G

Girlway

Online MP
Sep 16 2

Related articles

Comments (0)