DEV Community

Zero Heartbeat
Zero Heartbeat

Posted on Edited on Originally published at delta1labs.com

Replaying failed webhooks from a dead-letter queue

Your billing reconciler listens for license.revoked on a webhook endpoint. One Tuesday you ship a deploy, the endpoint 502s for ninety seconds while the new pods roll, and in that window three refunds fire license.revoked. If the licensing service used fire-and-forget delivery, those three events are gone — and your "active customers" count is now wrong in a way no retry will ever correct, because nothing remembers the events existed. The outage was ninety seconds; the data drift is permanent.

The fix is not to make your endpoint perfect. It is to make delivery durable: every event is persisted before it is sent, retried on a schedule when it fails, and — only when retries are genuinely exhausted — parked somewhere you can find it and send again. This post is about the producer side of Keyright's outbound webhooks: the delivery state machine, the retry schedule, the dead-letter queue that catches what the schedule cannot, and the replay endpoint that lets you re-drive a parked event without double-counting the ones that already landed.

A delivery is a record, not a fire-and-forget call

The unit Keyright tracks is not the event — it is the delivery: one attempt to push one event to one subscription. When the issuing service commits a state change, it writes the event and, for each subscription matching that event type, a delivery row in pending state. The HTTP POST happens after the commit, driven by a worker reading those rows. This is the ordinary transactional-outbox discipline, and it is what makes the guarantee honest: the event is safe on disk before anyone promises to deliver it, so a crash between "commit" and "send" loses nothing.

Each delivery accumulates a history as the worker works it:

{
  "id": "whd_01JD9Q4ZB2K7M3X8V0",
  "eventId": "evt_01JD9Q4ZB2K7M3X8V0",
  "subscriptionId": "whs_7c1f9a",
  "type": "license.revoked",
  "status": "dead_lettered",
  "attempts": 7,
  "nextAttemptAt": null,
  "attemptHistory": [
    { "at": "2026-11-17T14:02:11Z", "code": 502, "latencyMs": 1200, "error": "Bad Gateway" },
    { "at": "2026-11-17T14:02:26Z", "code": 502, "latencyMs": 1180, "error": "Bad Gateway" },
    { "at": "2026-11-17T14:03:11Z", "code": null, "latencyMs": 30000, "error": "timeout" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The attemptHistory is the part that earns its keep. When a delivery finally fails for good, you do not get "it didn't work" — you get the status code and error of every attempt, which tells you instantly whether the problem was your endpoint (a run of 502s), a signature mismatch on your side (a run of 401s you returned), or a slow handler (timeout). Triage starts from evidence, not a guess.

The retry schedule

A non-2xx response or a connection failure does not fail the delivery — it schedules the next attempt. Keyright uses exponential backoff with jitter, so a fleet of deliveries to the same recovering endpoint do not all retry in lockstep and re-knock it over. The schedule stretches from seconds to hours across a bounded attempt count:




pending

delivering

delivered

retrying

dead_lettered


2xx

timeout / non-2xx

backoff elapsed · attempts left

attempts exhausted

replay — re-enqueue same event id
Attempt 1 immediate · 2 at ~15s · 3 at ~1m · 4 at ~5m · 5 at ~30m · 6 at ~2h · 7 at ~6h, then dead-letter

A ninety-second deploy blip falls entirely inside the first two or three attempts, so the customer-facing effect is nothing — the event lands a few seconds late and you never hear about it. The schedule is deliberately long at the tail (hours, not minutes) so that a longer outage — a bad certificate, a DNS mistake, an endpoint behind a vendor whose weekend maintenance ran over — still has a good chance of succeeding on a later attempt once you fix it. The attempt count is bounded, though. When it runs out, the delivery stops retrying and moves to dead_lettered. Retrying forever would turn one permanently-broken subscription into an unbounded backlog that starves every healthy one.

When attempts are exhausted: the dead-letter queue

dead_lettered is a terminal-but-recoverable state. The delivery is no longer retried automatically — nothing will push it again on its own — but it is not deleted. It sits in the dead-letter queue with its full attempt history, waiting for a human to decide what to do. List the queue through the admin API:

GET /v1/admin/webhooks/deliveries?status=dead_lettered&subscription=whs_7c1f9a HTTP/1.1
Host: license.acme.example
Authorization: Bearer {admin_token}
Enter fullscreen mode Exit fullscreen mode

or from the CLI, which is where most on-call triage actually happens:

$ keyright webhooks dlq list --subscription whs_7c1f9a
whd_01JD9Q4ZB2K7M3X8V0  license.revoked   7 attempts  last 502  2026-11-17T14:03Z
whd_01JD9Q7PA1F2C9K4M0  license.renewed   7 attempts  last 502  2026-11-17T14:03Z
whd_01JD9QB3X8V0M3K7J2  license.revoked   7 attempts  last 502  2026-11-17T14:03Z

3 dead-lettered deliveries · oldest 2026-11-17T14:03Z
Enter fullscreen mode Exit fullscreen mode

Three deliveries, all 502, all timestamped inside your deploy window — the queue has told you exactly what the outage cost you and which events need to be re-sent. The dead-letter queue is doing two jobs at once here: it is a safety net that kept the events, and it is an incident record that scopes the blast radius.

Replaying a parked delivery

Replay re-enqueues a dead-lettered delivery as a fresh pending delivery of the same event — same eventId, same payload bytes, same signature inputs. Your endpoint sees exactly what it would have seen on the first attempt:

# Re-drive one delivery...
$ keyright webhooks replay whd_01JD9Q4ZB2K7M3X8V0
whd_01JD9Q4ZB2K7M3X8V0 → re-enqueued as whd_01JDA2M5R0 (pending)

# ...or every dead-lettered delivery for a subscription, once the endpoint is healthy.
$ keyright webhooks dlq replay --subscription whs_7c1f9a --all
3 deliveries re-enqueued
Enter fullscreen mode Exit fullscreen mode

The same action is an admin endpoint, which is what your own recovery automation calls:

// After a health check confirms the endpoint is back, drain its dead-letter queue.
async Task DrainDeadLetterQueueAsync(string subscriptionId, CancellationToken ct)
{
    var parked = await _admin.ListDeliveriesAsync(
        status: DeliveryStatus.DeadLettered,
        subscription: subscriptionId,
        ct: ct);

    foreach (var delivery in parked)
    {
        // Replay preserves the original eventId, so a consumer that dedupes
        // on eventId treats an already-processed event as a harmless no-op.
        await _admin.ReplayDeliveryAsync(delivery.Id, ct);
    }
}
Enter fullscreen mode Exit fullscreen mode

The reason bulk replay is safe to run without first working out which of the three events your endpoint did eventually process is that the event id does not change on replay. The consumer side is already built to survive at-least-once delivery — it records each eventId it has handled and ignores a repeat — so re-driving the whole queue is idempotent from the receiver's point of view. A delivery that actually landed on attempt 6 (and so never dead-lettered) is not in the queue at all; one that dead-lettered but had been partially processed is recognised by its id and skipped. You do not have to be precise about the boundary. You replay everything that parked and let the consumer's deduplication sort it out.

// The receiving side that makes blind replay safe: dedupe on eventId.
app.MapPost("/webhooks/keyright", async (HttpRequest req, LedgerDb db) =>
{
    var raw = await ReadRawBodyAsync(req);
    if (!WebhookSignature.IsValid(req, raw, _secret))
        return Results.Unauthorized();

    var evt = JsonSerializer.Deserialize<WebhookEvent>(raw)!;

    // First writer wins; a replayed duplicate collides on the unique eventId and is dropped.
    if (!await db.TryRecordProcessedAsync(evt.Id))
        return Results.Ok();      // already handled — acknowledge and stop the retries

    await Reconcile(evt);         // apply the state carried in evt.Data
    return Results.Ok();
});
Enter fullscreen mode Exit fullscreen mode

When the endpoint will be down longer than the schedule

The retry schedule and the dead-letter queue together cover outages up to the retention window. For a planned outage longer than that — a migration, a multi-day endpoint rewrite — do not let events burn their seven attempts against a URL you know is dead and dead-letter in bulk. Pause the subscription instead:

$ keyright webhooks pause whs_7c1f9a --reason "endpoint migration"
subscription whs_7c1f9a paused — events held, not delivered
Enter fullscreen mode Exit fullscreen mode

A paused subscription still records every matching event but holds its deliveries in pending without attempting them, so nothing ages out of the dead-letter retention window while you work. When the new endpoint is live, keyright webhooks resume whs_7c1f9a releases the held deliveries into the normal schedule. Pausing trades immediacy for durability deliberately; it is the right tool when you already know the endpoint is gone.

Caveats worth stating plainly

Replay does not restore ordering. Re-enqueued deliveries race through the schedule independently and can arrive interleaved. If your handler applies relative changes ("subtract one seat"), an out-of-order replay corrupts the count. Carry the resulting state in the payload — the new seat total, the absolute expiresAt — so every event is a snapshot that is safe to apply in any order, and reconcile against the license record when it is not.

A poison event dead-letters on your bug, not an outage. If your handler throws on a specific payload it will fail all seven attempts and park, exactly as a down endpoint would. The attempt history disambiguates — a 500 you returned on every attempt points at your code, not the network — so read it before you replay, or you will just re-park the same event.

Retention is bounded. The dead-letter queue is a recovery buffer, not an event store. Triage it within the window; for longer-term history, consume license.* events into your own audit log as they arrive, which is the system of record the queue was never meant to be.

Takeaway

Durable webhook delivery is three mechanisms working together. The retry schedule absorbs the short outages that are 99% of reality, so you never notice them. The dead-letter queue catches the long tail the schedule cannot beat and turns a lost event into a listed, inspectable row with its full failure history. The replay endpoint re-drives those rows with their original event ids, so a consumer that dedupes on event id can take a blind bulk replay and come out consistent. Build the receiver to dedupe and to apply state snapshots rather than deltas, and recovering from an endpoint outage stops being a data-reconciliation project and becomes one command: list the queue, replay it, done.

Top comments (0)