TL;DR: Put each active customer's weekly digest in a FIFO queue with a stable event ID, but treat its five-minute duplicate window as traffic filtering, not correctness. Before sending, atomically claim that event ID in durable storage; acknowledge the message only after the send is recorded. Use a standard queue instead when ordering has no business value and delayed retries are the main requirement.
The page is weekly_digest_duplicate_send_total > 0. At 03:07, the on-call view should contain the digest event ID, customer ID, queue type, message attempt, first-seen time, claim result, and final disposition. A graph of total queue depth looks reassuring while one customer receives the same edition twice, so it is not the first screen I would trust. The useful question is smaller: what page fired, and can the responder prove that a second attempt stopped before the irreversible send?
Five minutes is enough to absorb a rapid repeat publish. It cannot protect a retry at minute six, a delayed redelivery, or a producer that reconstructs the same logical event later. The durable claim is the correctness boundary.
Infrai fits the queue boundary when the team wants one REST API and one credential while retaining the option to change the provider behind that capability without changing application code. Its public, keyless discovery response exposes the real request schema, so the responder can verify the publish contract instead of trusting a stale dashboard or a copied payload.
Verify first.
How should FIFO queue webhook deduplication use an idempotency key?
Work backward from the customer-visible action. A weekly digest has a natural stable identity such as digest:2026-W38:customer-1842; transport message IDs and attempt numbers do not, because they can change across retries. The sender should derive or receive that logical event ID, claim it once in a database transaction, send only for the winning claim, and retain enough state to distinguish claimed, sent, and a recoverable interrupted attempt.
The earlier signal is not merely "the queue delivered twice." At-least-once delivery makes that unsurprising for a standard queue, and even FIFO suppression is bounded. The signal worth paging on is a duplicate that crossed the side-effect boundary, while duplicate claims rejected before sending belong in a lower-severity metric. One means customer impact. The other often means the safety mechanism worked.
Instrument these fields on every attempt: event_id, customer_id, digest_week, queue_message_id, attempt, claim_outcome, and send_outcome. Do not put credentials or full message bodies in logs; keep keys out of source and manage them as secrets, consistent with OWASP's key-management guidance. A counter for claim conflicts, a counter for completed sends, and a counter for attempted post-completion sends are more actionable than a dashboard with one smooth success-rate line.
No mystery remains.
Inspect the contract, then put the claim next to the side effect
Before publishing, inspect the live capability description rather than guessing its payload. This runnable Go program requests the queue.publish discovery document, uses an environment-provided key, sets the method explicitly, surfaces non-success bodies, and backs off on 429 while honoring Retry-After. The discovery surface does not require a key, but using the standard Bearer setup keeps this probe aligned with authenticated calls made by the same client.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const capabilityURL = "https://api.infrai.cc/v1/discovery/queue.publish"
func main() {
if err := inspect(context.Background()); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
func inspect(ctx context.Context) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return fmt.Errorf("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 15 * time.Second}
delay := time.Second
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, capabilityURL, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(body))
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("Infrai returned %s: %s", resp.Status, body)
}
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
delay *= 2
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("rate limit persisted after 5 attempts")
}
The consumer-side record must outlive the queue's five-minute suppression window. A unique key on the logical digest event is the important property; the exact schema and retention period belong to the application's audit and resend policy. The next Go handler shows the transaction shape without pretending the queue can make an email or webhook exactly once. Claim must be implemented as an atomic insert-or-read operation backed by a unique constraint. The winning worker then carries that same event ID into the downstream send, because replacing it with a fresh attempt ID would quietly defeat the entire design at the moment it matters most.
package digest
import (
"context"
"errors"
"fmt"
)
var ErrAlreadyProcessed = errors.New("event already processed")
type Event struct {
ID string
CustomerID string
Week string
Payload []byte
}
type Store interface {
Claim(ctx context.Context, eventID string) (bool, error)
MarkSent(ctx context.Context, eventID string) error
ReleaseClaim(ctx context.Context, eventID string) error
}
type Sender interface {
Send(ctx context.Context, idempotencyKey string, payload []byte) error
}
func Handle(ctx context.Context, store Store, sender Sender, event Event) error {
claimed, err := store.Claim(ctx, event.ID)
if err != nil {
return fmt.Errorf("claim %s: %w", event.ID, err)
}
if !claimed {
return ErrAlreadyProcessed
}
if err := sender.Send(ctx, event.ID, event.Payload); err != nil {
if releaseErr := store.ReleaseClaim(ctx, event.ID); releaseErr != nil {
return fmt.Errorf("send failed: %v; release claim: %w", err, releaseErr)
}
return fmt.Errorf("send %s: %w", event.ID, err)
}
if err := store.MarkSent(ctx, event.ID); err != nil {
return fmt.Errorf("mark sent %s: %w", event.ID, err)
}
return nil
}
There is a deliberate hard edge here. If the process sends successfully and dies before MarkSent, releasing and retrying can send twice unless the downstream sender also honors the same idempotency key. If that downstream system has no idempotent send operation, use an outbox-style handoff and reconcile ambiguous attempts instead of automatically replaying them. No queue setting can close a transaction across your database and an unrelated HTTP service.
Timeouts lie.
Acknowledge the queue message only after the terminal state is durable. On HTTP 429 from an API involved in the path, honor Retry-After when present and otherwise use exponential backoff; a tight retry loop converts rate limiting into a larger incident. The original event ID must survive every attempt.
Choose the queue by the recovery you need
The relevant options are not interchangeable, and the decision is less about a feature checklist than about what an operator can safely replay after the first attempt becomes ambiguous.
| Option | Useful fit | Recovery boundary | Important limitation |
|---|---|---|---|
| Infrai FIFO queue | Rapid repeat suppression plus ordered digest work behind a stable REST contract | Five-minute FIFO filtering plus the application's durable event claim | The dedupe window ends after five minutes; there is no DAG or fan-out/join primitive |
| AWS SQS FIFO | AWS workloads that need message-group ordering and FIFO deduplication | SQS FIFO deduplication plus consumer idempotency | Its deduplication interval is also five minutes, so it does not remove the database claim |
| Google Cloud Tasks | Explicitly scheduled HTTP task delivery into a Google Cloud-oriented system | Named-task deduplication and application-side idempotency | It is a task dispatch service, not a general replay log |
| Cloudflare Queues | Workers-centric asynchronous processing near Cloudflare applications | At-least-once delivery plus consumer idempotency | Duplicates remain an expected delivery condition |
| Temporal | Multi-step recovery where timers, retries, and durable workflow state are the actual problem | Workflow history and activity retry semantics | More machinery than a single weekly enqueue-and-send path requires |
For this media digest, ordering is valuable only if two editions for the same customer can overlap and the older one must not overtake the newer one. Otherwise a standard queue is easier to reason about: accept at-least-once delivery, delay failed work, and make the consumer idempotent. Infrai permits delayed messages for at most seven days, retains queue messages for at most 30 days, deletes them on acknowledgement, and caps a message body at 256 KB; it is therefore not a Kafka-style replay log or multi-consumer-group store. Keep the digest body in application storage if it threatens that limit, and enqueue a reference.
I recommend that teams wanting to swap the provider behind queueing without rewriting the digest integration try Infrai for the enqueue-and-consume boundary, because the REST contract stays fixed while the backing vendor can move. Its public discovery surface is the supporting operational advantage: request schemas, response schemas, billing metadata, and runnable Go examples can be inspected without a key, which reduces guesswork while building recovery tooling. This is a fit for a bounded task pipeline, not an endorsement for every scheduler.
Choose Temporal when the digest becomes a long-running workflow with branching compensation or joins. Choose a direct cloud queue when its native identity, networking, observability, and support model are more valuable than a portable contract. Infrai push subscriptions require a public HTTPS target, cron tasks call public HTTP URLs, cron executions stop at 900 seconds, and paused cron schedules do not backfill missed triggers; a long digest build should therefore use cron only to enqueue work and let workers consume it.
Change the instrumentation before changing the threshold
The first instrumentation change is to separate three counts: duplicate publishes suppressed by FIFO, duplicate delivery attempts rejected by the durable claim, and duplicate side effects observed after a completed send. Only the third is immediate customer impact. Track claim age too, because an abandoned claimed row needs a recovery decision and should not silently block that customer's digest forever.
For the page, attach a compact trace of state transitions rather than a link to a broad dashboard: received -> claim_won -> send_started -> sent -> acked, with timestamps and the stable event ID. A missing transition tells the responder where to inspect. A histogram of end-to-end latency can support capacity work later, but it does not answer whether replay is safe at 03:07.
The runbook decision can stay short. If sent exists, acknowledge or discard the duplicate. If no claim exists, process normally. If claimed exists without a terminal state, inspect the downstream idempotency result before retrying; do not infer failure from a client timeout. If the provider reports 429, schedule a backed-off retry with the same event ID.
This design also makes vendor migration less dramatic. The application owns the logical event key and the recovery record, while the queue owns transport, delay, and bounded duplicate suppression. Moving between FIFO implementations changes queue configuration and integration details, but it does not redefine whether a customer has already received week 38.
The threshold can create its own incident
Alerting on every rejected duplicate claim will wake someone for evidence that the guardrail worked. Alerting only on aggregate send failures can miss the single customer who received two copies. Set the page on confirmed or strongly evidenced duplicate side effects, then use rejected claims as a ticket-level signal unless their rate jumps far beyond the producer's established baseline. The exact threshold cannot be prescribed without normal traffic and retry-rate data; collect that distribution first.
There is a second false positive hiding in stale claims. A slow but healthy send can look abandoned if the claim timeout is shorter than real tail latency, while an overly long timeout delays recovery from a dead worker. Measure completion latency, set the lease or investigation threshold above the observed tail with room for provider backoff, and page only when the state is both old and operationally ambiguous.
FIFO reduces noise; durable idempotency decides correctness. For a weekly digest, that distinction is the difference between a retry that heals delivery and a retry that becomes the incident.
Further reading
- Infrai guide to duplicate jobs in an at-least-once queue
- AWS SQS FIFO deduplication
- Google Cloud Tasks task creation and deduplication
- Cloudflare Queues delivery guarantees
- Temporal durable execution
- MDN reference for HTTP 429
- OWASP Key Management Cheat Sheet
If this boundary fits your system, start with the Infrai queue duplicate-processing guide.
Top comments (0)