Writing

Retry forever, charge once: idempotency in 2026

Why retries charge customers twice, what Stripe, Adyen, Shopify, Resend and the IETF draft do about it in 2026, a Postgres implementation tested under concurrency, why every queue and workflow engine still delivers at least once, and why AI agents make all of it matter more. Every figure is sourced.

Three identical payment requests, one charge

Every API that moves money, sends a message or creates something makes a promise it rarely writes down: if you are not sure your request went through, send it again, and nothing bad will happen. This article explains how that promise is kept, where it breaks, and what changed in 2025 and 2026.

Version française : Réessayer sans fin, ne débiter qu'une fois : l'idempotence en 2026.

Two incidents to keep in mind

On Black Friday 2024, Voyado, a Swedish customer-engagement platform, could not send its customers' SMS campaigns properly. Its SMS database was overloaded and failed to scale, the API returned errors from 10:22 to 10:51 CET, and about 5% of intended recipients received nothing. Others received the same message several times. In Voyado's own words: "Automatic retries designed to make sure all messages are always delivered had led to batches of messages being sent multiple times, somehow bypassing the checks present to prevent such duplicates." The team then had to stop resending the messages that had failed, "to avoid the risk of further duplicates being sent" (Voyado status page, post-mortem of 6 December 2024).

On 27 August 2026, an attacker took two signed withdrawal messages on the ICON blockchain network and submitted them 1,492 times in twenty minutes. 1,490 of those calls succeeded and released 119,866,000 ICX and 531,600 bnUSD from accounts held by the ICON Foundation; the confirmed net loss was about 150.2 ETH and 31,204 USDC. The contract did have a guard against processing a message twice. The post-mortem explains why it failed: "the contract's uniqueness check (the guard meant to stop a message from being processed twice) only validated the high bits of the serial number, while the portion actually covered by the cryptographic signature (the low 256 bits) never changed between calls" (ICON Foundation post-mortem, 30 August 2026).

Both systems had duplicate protection, and both still did the same thing twice. One lost it under load, the other checked the wrong bytes. Keep both in mind: by the end you will know which part of a design would have caught each one.

The problem in one sentence

A timeout does not tell you whether the operation happened. Stripe's documentation says it plainly: treat the outcome of a request that returns a 500 "as indeterminate" (Stripe, low-level error handling). The client knows it sent something and did not get an answer. It does not know which of these three things happened.

Three ways a request goes wrong: the request is lost, the response is lost, or the same request arrives twice at once

  1. The request was lost. Nothing happened on the server. Retrying is correct.
  2. The request was processed and the response was lost. The payment exists. Retrying without protection pays twice.
  3. The same request arrived twice at the same time, because a user double-clicked, a load balancer retried, or a mobile client resent on reconnect. Both copies are in flight before either has finished.

The client cannot tell case 1 from case 2, so it has two choices. It can give up, and some operations that should have happened never will. Or it can retry, and some operations will happen twice. Idempotency is what lets it retry safely: the server recognises the second attempt and answers it with the result of the first.

What HTTP already promises

HTTP defines the word. RFC 9110, the current HTTP semantics standard, says: "A request method is considered 'idempotent' if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request. Of the request methods defined by this specification, PUT, DELETE, and safe request methods are idempotent" (RFC 9110, §9.2.2). The safe methods are GET, HEAD, OPTIONS and TRACE.

POST is not on the list, and neither is PATCH. That is the whole problem: creating a payment, sending an email or placing an order is a POST. The standard also tells clients what to do about it: "A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent."

An idempotency key is that "means to know". It turns a non-idempotent POST into an operation the client may repeat.

The idempotency key

The IETF draft discussed below credits the pattern to PayPal and Stripe. The client generates a unique value, usually a random UUID, sends it with the request, and sends the same value again on every retry of that request. The server stores the key with the result of the first attempt. When the key comes back, it does not run the operation again; it returns the stored result.

Three questions decide whether an implementation works:

  • Is the key compared with the request? If a client reuses a key for a different request, a bug on its side, the server must refuse rather than return an unrelated result.
  • What happens while the first attempt is still running? A duplicate that arrives mid-flight must not start a second execution.
  • How long is the key kept? A retry that arrives after the key has been deleted is a new request.

The IETF draft, revived this month

There is a standard in progress: The Idempotency-Key HTTP Header Field. It reached working group last call in January 2024, went back to the working group in June 2024, and its version 07 expired in April 2026. Version 08 was published on 22 October 2026, with a new co-author and new text for the retry case that the authors mark "still open for discussion". It is still an Internet-Draft, not an RFC.

Its answers to the three questions are the ones most APIs already use:

  • a missing key on an endpoint that requires one: 400;
  • a key reused with a different payload: 422;
  • a retry while the original request is still being processed: 409;
  • an expiry policy that the server "SHOULD define" and "publish in the documentation".

What the large APIs do in 2026

APIWhere the key goesHow long it is keptSame key, different request
Stripe API v1Idempotency-Key header, on POSTAt least 24 hoursError. Otherwise the saved response is replayed, including a 500
Stripe API v2Same header, on POST and DELETE; generated automatically if absent30 daysA failed first attempt is executed again
Adyenidempotency-key header, up to 64 characters, POST7 to 14 daysNot documented; 409 or 422 with code 704 while in flight
Shopify GraphQL@idempotent(key:) directive; mandatory on refundCreate and inventory mutations from API version 2026-0424 hoursIDEMPOTENCY_KEY_PARAMETER_MISMATCH
ResendIdempotency-Key header, up to 256 characters24 hours409 invalid_idempotent_request
PayPalPayPal-Request-Id header, on the APIs that support itVaries by APIReturns the latest status of the previous request
AWS EC2ClientToken parameter, up to 64 ASCII charactersLifetime of the resource plus a marginIdempotentParameterMismatch
MTN MoMoX-Reference-Id header, a UUID v4, on financial POST requestsNot statedRESOURCE_ALREADY_EXIST: duplicated reference

Sources: the documentation pages listed at the end, read on 31 October 2026.

Three changes stand out.

Stripe's API v2 rethought the semantics. In v1, "Retries with the same key return the saved response, including 500 errors", and keys can be removed after 24 hours. In v2, keys live for 30 days, and "If the first request failed (or partially failed), the API re-executes the failed requests and returns the new response." If you send no key, "Stripe automatically generates a UUID for you" (Stripe, API v2 overview). The v2 behaviour is closer to what a caller actually wants: the stored result of a success, and a real second chance after a failure.

Shopify made it mandatory. In December 2025 Shopify added the @idempotent directive to its GraphQL Admin API and announced that it becomes mandatory for refunds and inventory adjustments in version 2026-04. The changelog gives the reason without euphemism: "Duplicate inventory adjustments and duplicate refunds have long been a pain-point for merchants and for Shopify, resulting in inventory inconsistencies and financial losses" (Shopify changelog, 12 December 2025).

Email joined payments. Resend added idempotency keys to its sending API in May 2025, and to batch sends in June 2025. An email sent twice does not cost money, but it costs trust, and Voyado's SMS show that messaging systems retry as hard as payment systems.

Why a key and not a hash of the request

A tempting shortcut is to skip the key and deduplicate on a hash of the request body. AWS explains why it does not do that, with an example from EC2: "if those requests were for launching an Amazon EC2 instance, then our assumption might not hold. It's possible that the caller actually wants two identical EC2 instances" (AWS Builders' Library, Making retries safe with idempotent APIs). Two identical payments of 5,000 to the same merchant can be two coffees. Only the caller knows whether a request is a retry or a new intention, so the caller supplies the key.

The hash still has a job: stored next to the key, it detects a key reused for a different request. AWS does exactly that: "we also store the parameters used to make the initial request along with the client request identifier."

What the server does with the key

On the server, a key has a short life with four states.

The life of an idempotency key: claimed, completed, replayed, or taken back after its lock expires

  • Absent. First time this key is seen. The request claims it and runs.
  • Claimed. The first attempt is running. A duplicate gets 409 and tries again later.
  • Completed. The result is stored. A duplicate with the same request gets that result, marked as a replay; Stripe sets Idempotent-Replayed: true. A duplicate with a different request gets 422.
  • Expired. Removed after the retention window. The next request with this key is treated as new.

Two rules from the people who built this at scale make the states trustworthy.

AWS: "the process that combines recording the idempotent token and all mutating operations related to servicing the request must meet the properties for an atomic, consistent, isolated, and durable (ACID) operation." The key and the effect are written together, or neither is.

Brandur Leach, who worked on Stripe's API, described the harder case in 2017, when the operation calls another system: "once we make our first foreign state mutation, we're committed one way or another" (Implementing Stripe-like Idempotency Keys in Postgres). His answer splits the request into atomic phases, local transactions between external calls, and records a recovery point after each one, so that a retry with the same key can "jump back to the point in the lifecycle just before the last attempt failed."

AWS adds a third rule that is easy to get wrong: do not answer a retry with "already exists". It looks harmless, since nothing was created twice, but "returning ResourceAlreadyExists has a side effect from the client's perspective." The client asked for a payment and was told something went wrong. The retry must receive the same answer the first attempt would have received.

An implementation in Postgres

Here is a complete implementation for a NestJS or Express service on Postgres. I ran it against PostgreSQL 18.6 and 19 RC 1, with ten tests described below.

schema.sql
CREATE TABLE idempotency_keys (
  account_id    bigint      NOT NULL,
  key           text        NOT NULL CHECK (char_length(key) BETWEEN 1 AND 255),
  request_hash  bytea       NOT NULL,
  locked_until  timestamptz NOT NULL DEFAULT now(),
  response_code integer,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, key)
);
 
CREATE TABLE payments (
  id          uuid    PRIMARY KEY DEFAULT uuidv7(),
  account_id  bigint  NOT NULL,
  amount      integer NOT NULL CHECK (amount > 0),
  currency    text    NOT NULL,
  provider_id text
);

The key is scoped to the account, as Stripe, Adyen and the ACP specification all do: two customers who happen to generate the same key must not see each other's results. uuidv7(), new in PostgreSQL 18, gives time-ordered identifiers that index better than random UUIDs.

When the whole effect is in your database

If creating the payment only writes to your own database, one transaction is enough. The claim, the payment and the stored reply commit together or not at all.

payments.ts
const fingerprint = (r: PaymentRequest) =>
  createHash('sha256').update(JSON.stringify([r.amount, r.currency])).digest();
 
// The key, the payment and the stored reply commit together, or not at all.
export async function createPayment(db: Pool, accountId: number, key: string, req: PaymentRequest): Promise<Reply> {
  const hash = fingerprint(req);
  const tx = await db.connect();
  try {
    await tx.query('BEGIN');
    // A second request with the same key waits here until the first one commits or rolls back.
    const claim = await tx.query(
      `INSERT INTO idempotency_keys (account_id, key, request_hash) VALUES ($1, $2, $3)
       ON CONFLICT DO NOTHING`,
      [accountId, key, hash],
    );
    if (claim.rowCount === 0) {
      await tx.query('ROLLBACK');
      return replay(db, accountId, key, hash);
    }
    const { rows: [payment] } = await tx.query(
      `INSERT INTO payments (account_id, amount, currency) VALUES ($1, $2, $3)
       RETURNING id, amount, currency`,
      [accountId, req.amount, req.currency],
    );
    await tx.query(
      `UPDATE idempotency_keys SET response_code = 201, response_body = $3
       WHERE account_id = $1 AND key = $2`,
      [accountId, key, payment],
    );
    await tx.query('COMMIT');
    return { status: 201, body: payment };
  } catch (err) {
    await tx.query('ROLLBACK');
    throw err;
  } finally {
    tx.release();
  }
}
 
async function replay(db: Pool, accountId: number, key: string, hash: Buffer): Promise<Reply> {
  const { rows: [saved] } = await db.query(
    `SELECT request_hash, response_code, response_body FROM idempotency_keys
     WHERE account_id = $1 AND key = $2`,
    [accountId, key],
  );
  if (!saved.request_hash.equals(hash)) return { status: 422, body: { error: 'key_reused_with_another_request' } };
  if (saved.response_code === null) return { status: 409, body: { error: 'request_in_progress' } };
  return { status: saved.response_code, body: saved.response_body, replayed: true };
}

The concurrency is handled by the primary key, not by application code. When two transactions insert the same key, the second one waits on the unique index until the first commits or rolls back. If the first commits, ON CONFLICT DO NOTHING inserts nothing, and the second request reads the stored reply. If the first rolls back, the second inserts the key and does the work itself. PostgreSQL's documentation says the same of the related ON CONFLICT DO UPDATE: one of the two outcomes "is guaranteed, even under high concurrency" (INSERT).

Fifty concurrent requests with the same key produce one payment and fifty identical answers, one of them original and forty-nine marked as replays. The same fifty requests without a key produce fifty payments. A crash between the payment and the commit leaves no key behind, and the retry pays once. Commit the key on its own before the payment instead, let the process die in between, and the key stays claimed with no reply: every retry receives 409, and no payment is ever made.

One detail is worth checking in your own stack: that a COMMIT really committed. When a statement fails inside a transaction and the error is caught, PostgreSQL answers the following COMMIT with a ROLLBACK tag rather than an error, and a client that ignores the tag reports success. I found that bug in the PGlite driver of MikroORM in October 2026; the maintainer fixed it in #8552, reusing my test. In an idempotency handler, it means telling a client "201, done" for a payment that does not exist.

When the effect is somewhere else

Most real payments call a provider: Stripe, Adyen, a mobile money operator. That call cannot sit inside your transaction, so the request runs in three steps.

payments.ts
// 1. Claim the key, or take it back from a request that died holding it.
const claim = await db.query(
  `INSERT INTO idempotency_keys (account_id, key, request_hash, locked_until)
   VALUES ($1, $2, $3, now() + interval '30 seconds')
   ON CONFLICT (account_id, key) DO UPDATE SET locked_until = EXCLUDED.locked_until
     WHERE idempotency_keys.response_code IS NULL
       AND idempotency_keys.locked_until < now()
       AND idempotency_keys.request_hash = EXCLUDED.request_hash`,
  [accountId, key, hash],
);
if (claim.rowCount === 0) return replay(db, accountId, key, hash);
 
// 2. Same key downstream: if we crash after this line, the retry charges nothing new.
const charge = await provider.charge(`${accountId}:${key}`, req);
 
// 3. Record the payment and the reply together, unless a retry that took the key back got there first.
//    In one transaction:
//      UPDATE idempotency_keys SET response_code = 201
//        WHERE account_id = $1 AND key = $2 AND response_code IS NULL
//      -- zero rows: roll back and replay; otherwise insert the payment and store the reply.

Step 1 commits the claim with a lock that expires after thirty seconds. A duplicate that arrives while the lock is held gets 409. A retry that arrives after the lock has expired, because the first attempt crashed, takes the key back, but only if the request matches and no reply was stored.

Step 2 is the rule that makes the whole thing safe: forward the key. If the process dies after the provider charged the card and before step 3, the retry calls the provider again with the same key, and the provider returns the first charge instead of making a second one. Without that, no amount of care in your own database helps, because the money has already left.

Step 3 handles the last race: the first attempt was only slow, not dead, and finishes after a retry has taken its key and completed. The WHERE response_code IS NULL condition makes the late finisher see that someone else has written the reply; it rolls back and replays.

The tests cover each of these: twenty concurrent requests call the provider once; a crash after the charge followed by a retry after the lock expires charges once; a slow first attempt that finishes after a retry produces one payment and returns the retry's answer. Removing the forwarded key makes the crash test charge twice. Removing the response_code IS NULL condition makes the slow-attempt test record two payments.

What PostgreSQL 19 will change

PostgreSQL 19 reached its first release candidate on 15 October 2026 and is not final yet. It adds ON CONFLICT DO SELECT, which "returns the existing row that conflicts with the row proposed for insertion" (PostgreSQL devel docs). The claim and the replay become one statement:

INSERT INTO idempotency_keys (account_id, key, request_hash) VALUES ($1, $2, $3)
ON CONFLICT (account_id, key) DO SELECT
RETURNING old IS NULL AS claimed, request_hash, response_code, response_body;

On 19 RC 1, claimed is true when the key was new and false when the row already existed, in which case the stored hash and reply come back in the same round trip. Until 19 is released, the two-step version above is what to run.

What the libraries give you

If you run on AWS Lambda, Powertools for AWS Lambda has an idempotency utility for TypeScript (version 2.36.0, October 2026) that implements the same states, INPROGRESS, COMPLETE and EXPIRED, in DynamoDB or a Redis-compatible cache. It expires records after an hour by default, stores a hash of the payload to raise IdempotencyValidationError on a mismatch, and records the remaining invocation time so that a record left INPROGRESS by a timed-out function can be taken back (Powertools for AWS Lambda, Idempotency). It is the same design as above, with a different store.

Queues deliver at least once, still

Everything so far is about a client retrying an HTTP request. The same problem exists between your own services, and in 2026 no mainstream message system removes it.

Amazon SQS. Standard queues deliver at least once: "you might get that message copy again when you receive messages. Design your applications to be idempotent." FIFO queues deduplicate, but only for five minutes: "If a message is sent successfully but the acknowledgement is lost and the message is resent with the same MessageDeduplicationId after the deduplication interval, Amazon SQS can't detect duplicate messages" (SendMessage).

Google Cloud Pub/Sub offers exactly-once delivery, with conditions. It applies to pull subscriptions only, within one region: "If your subscriber application is spread across multiple regions, it can lead to duplicate message delivery, even when exactly-once delivery is enabled." It also does not cover the publishing side: "A subscription might receive multiple copies of the same message due to publish side duplicates" (Pub/Sub, exactly-once delivery, updated 26 October 2026).

Apache Kafka has had exactly-once semantics since 2017, and version 4.4.0 shipped on 22 October 2026. The guarantee is real and narrow: it covers reading from Kafka, processing and writing back to Kafka, because the consumer's position is written in the same transaction as the output. For anything outside Kafka, the documentation is explicit: "Exactly-once delivery for other destination systems generally requires cooperation with such systems" (Kafka 4.4 design). The proposal that would let Kafka take part in a two-phase commit with an external database, KIP-939, was accepted, then paused, and its client code was removed from the 4.2 release branch. As of 4.4 it has not shipped.

The transactional outbox is the standard way to publish an event when a database row changes: write the event to an outbox table in the same transaction, and let a separate process send it. It guarantees that the message is sent "if and only if the database transaction commits", and immediately adds the price: "The Message relay might publish a message more than once... As a result, a message consumer must be idempotent" (microservices.io).

Duplicates are not rare edge cases at scale. In 2017 Segment measured that "approximately 0.6% of events that are ingested within a 4-week window are duplicate messages that we've already received", and built a deduplication layer holding about 60 billion keys, 1.5 TB on disk, for a four-week window (Delivering billions of messages exactly once). And duplicates arrive in bulk exactly when you are busiest recovering. During PagerDuty's Kafka outage of 28 August 2025, caused by a bug that created a new Kafka producer for every API request, "nearly 4.2 million extra producers per hour at peak", customers received duplicate webhooks and duplicate chat messages "while the system processed the backlog of stale messages" (PagerDuty Engineering, 4 September 2025).

The consumer side uses the same table as the HTTP side. The message ID plays the role of the key, and recording it commits with the effect.

Durable execution: every step runs at least once

The big infrastructure trend of 2025 and 2026 is durable execution: workflow engines that record each step and resume after a crash. Cloudflare Workflows became generally available on 7 April 2025. AWS launched Lambda durable functions on 2 December 2025, for Node.js and Python. Temporal, Restate, DBOS and Inngest offer the same model.

They differ in many ways and agree on one: a step that crashed before its result was recorded runs again.

  • AWS: "Steps have at-least-once execution semantics by default" (Lambda durable functions, idempotency).
  • Cloudflare: "Because a step might be retried multiple times, your steps should (ideally) be idempotent" (Rules of Workflows).
  • Temporal: "Temporal recommends that Activities be idempotent", and suggests "a combination of the Workflow Run ID and the Activity ID as an idempotency key" (Activity definition).
  • DBOS: "Steps are tried at least once but are never re-executed after they complete."
  • Inngest, about the gap they all share: "A request can succeed before Inngest records the step result."

Durable execution moves the retry loop out of your code. It does not remove the need for the effect at the end of the loop to recognise a repeat. The engine gives you a stable identifier for each step; pass it to the payment provider, the email API or your own table, as the key.

AI agents retry too

The newest callers of your API are AI agents, and they bring retries at every layer.

The SDKs retry by default. The official Anthropic and OpenAI SDKs both retry failed requests twice by default: connection errors, 408, 409, 429 and server errors, and Anthropic's documentation adds that timed-out requests are retried too (Anthropic SDK, OpenAI SDK). That retries the call to the model. The tool the model then decides to call is a separate request, and it is your API.

The tool protocol expects repeats. The Model Context Protocol, the open protocol many agents use to call tools, lets a tool declare idempotentHint: "If true, calling the tool repeatedly with the same arguments will have no additional effect on its environment." The default is false, and the spec warns that annotations are hints that clients "MUST consider" untrusted unless they come from trusted servers. The current revision, dated 28 July 2026, also removed stream resumability: "A broken response stream loses the in-flight request; clients MUST re-issue it as a new request with a new request ID" (MCP changelog, 2026-07-28). A tool call whose stream breaks after the tool ran will be sent again.

Agent frameworks say so in their own docs. The OpenAI Agents SDK, in its human-in-the-loop guide: "Approval permits a tool call to execute; it does not guarantee exactly-once execution. If a tool performs an external action and the process stops before the application stores the updated run state, resuming from the older approved snapshot can execute the tool again. For example, a payment can complete even though the saved snapshot has no record of the payment result" (openai-agents-python). LangGraph: "A task that started but did not finish may run again on that resume, so design side effects to be idempotent." Inngest recommends using the model's tool call ID as the payment provider's idempotency key.

Agentic payments made it mandatory. The Agentic Commerce Protocol, launched by Stripe and OpenAI on 29 September 2025 for purchases inside ChatGPT, shipped with an optional Idempotency-Key header. In the 2026-04-17 version of the specification it is required on every POST, and the checkout RFC adds: "Servers MUST NOT re-execute side effects (e.g., payment capture, inventory reservation) on replay" (ACP specification). Coinbase's x402 protocol added a payment-identifier extension in February 2026 that "enables clients to provide an id that serves as an idempotency key".

I could not find a public, verified incident of an agent paying twice. The risk is documented instead by the people who build the protocols and the frameworks, and they have all converged on the same answer: a stable key per logical action, forwarded to the system that performs it.

Where duplicates come from in 2026: SDK retries, tool-protocol re-issues, queues, workflow steps, all landing on the same handler

Nine ways to get it wrong

1. A new key on every retry. If the client generates the key inside the retry loop, every attempt is a new request. The key belongs to the user's intention: generate it when the user clicks "Pay", store it with the pending order, and reuse it until a definitive answer arrives.

2. A key that does not cover the request. ICON's guard checked part of a serial number while the signature covered another part, so a message could be replayed under a "new" number that was still validly signed. The fix in the post-mortem is the general rule: "the number that gets signed and the number used for the uniqueness check are now guaranteed to be identical". Whatever authorises the action must be what you deduplicate on.

3. The key recorded after the effect. If the payment is made and the process dies before the key is written, the retry finds no key and pays again. Record the key before the effect, in the same transaction when the effect is local, as a claim with a lock when it is not.

4. Errors stored as if they were answers. A 500 stored against a key turns a transient failure into a permanent one: every retry receives the same error. ACP forbids it ("Responses with HTTP 5xx status codes MUST NOT be cached against the idempotency key"), and Stripe's v2 re-executes failed requests. Store definitive answers, including a declined card, and let transient failures retry.

5. A window shorter than your retries. SQS FIFO deduplicates for five minutes, Resend and Shopify keep keys for 24 hours, Stripe v2 for 30 days. A job that retries with exponential backoff for six hours, a mobile client that resends a queued request the next morning, or a workflow resumed after a weekend outage can all outlive the window. Brandur suggested about 72 hours for his own reaper, so that a bug deployed on a Friday can be fixed on Monday.

6. The key not forwarded. Your table can be perfect, and a crash between the provider's charge and your commit still charges twice if the provider received no key. Every external call in the request takes the key, or a key derived from it.

7. Two systems, no shared record. In February 2025 HM Revenue and Customs, the UK tax authority, acknowledged that self-assessment refunds had been paid twice to some taxpayers: "there is a very niche scenario, where our CESA and PAYE systems can both reconcile a tax year resulting in duplicate refunds being issued" (Computer Weekly, 24 February 2025). Each system was idempotent on its own. The business key, one refund per taxpayer per tax year, had no single owner.

8. Humans retry too. In September 2024, an employee at the Taichung post office in Taiwan launched a salary transfer, believed the file had been deleted, and launched it again the next day; two different supervisors approved the two transfers. More than 10,000 employees were paid twice, NT$380 million in total, about US$12 million (Taiwan News, 7 September 2024). A batch file needs a key as much as an API call does: the file's identity, checked before the second run.

9. Charging for every retry. In July 2023 the US Consumer Financial Protection Bureau ordered Bank of America to pay at least $80.4 million in redress and a $60 million penalty, partly because the bank charged a $35 insufficient-funds fee "each time" a merchant re-presented the same returned payment, "despite already having charged consumers an NSF Fee on the initial returned transaction" (CFPB consent order 2023-CFPB-0006). That one was a fee policy, not a bug. It shows the same failure from the business side: a retry treated as a new event.

Back to the two incidents

Voyado shows the trade-off at the heart of the subject. A sender that cannot know whether a message left must choose between losing it and sending it twice; Voyado chose delivery, and its duplicate check did not hold under the overload. The defence is to record each send against a key such as campaign and recipient before handing it to the SMS gateway, and to pass that key along when the gateway accepts one. When it does not, a duplicate check that holds under load is the only protection left. The second lesson is in their last sentence: once duplicates were possible, they could no longer safely resend what had failed. Without idempotency, retrying is unsafe, and so is recovering from the incident.

ICON had a check, and the check was on the wrong bytes. A uniqueness key that does not cover exactly what the signature authorises is not a key; it is a counter the attacker controls. The table in this article stores a hash of the request next to the key for the same reason: the key says "this is a retry", the hash proves it is a retry of this request.

That is the honest version of "exactly once": nothing in the network, the queue or the workflow engine delivers it. Every layer delivers at least once, and the last step, the one that moves money or sends a message, recognises the repeat because it carries a key that was chosen once, by the caller, and travels with the request all the way down.

Sources

Réessayer sans fin, ne débiter qu'une fois : l'idempotence en 2026FR

architecture / postgres / api
31 min read

One database, a thousand customers, zero leaks

architecture / postgres / saas
25 min read

Multi-tenant : une base, mille clients, zéro fuiteFR

architecture / postgres / saas
28 min read