Short answer: treat a thumbnail upload as a resumable, idempotent job with a bounded retry budget. Keep the source image and its derived thumbnail under the same tenant policy, record the object version before retrying, and let deletion be driven by a retention event rather than by the upload worker. A temporary storage error should delay work, not create a second thumbnail or extend retention forever.
The concrete case here is a B2B SaaS product-image service. A tenant uploads an image, a worker resizes it, and another job produces a tenant-scoped export. The export must include only objects that tenant can read. The awkward part is the thumbnail write: a transient failure arrives after the resize has completed, so a naive retry repeats the write while the export and deletion clocks continue moving.
What should a retry own?
The least complex design gives each thumbnail a stable logical identity: (tenant_id, source_digest, variant). The worker writes to a temporary object key, verifies the completed object, then records a manifest row containing the tenant, digest, variant, object version or validator, creation time, and an expiry timestamp. The export reads that manifest, applies the tenant filter, and copies or streams only the records whose state is ready.
That ordering matters. If the database says ready before the object is durable, an export can produce a manifest entry that points at nothing. If the object exists but the worker never records ready, the retry can safely inspect the same logical key and finish the missing state transition. I use an explicit state machine: pending, writing, ready, expired, and failed. A retry is allowed to move writing to ready; it is not allowed to manufacture a new identity.
Identity decides the retry.
The retention choice belongs in the manifest. For this example, a seven-day retention window starts when the source image is accepted, and a deletion task removes both source and derived objects after the policy says they are expired. A retry can update last_attempt_at and an error classification, but it cannot push expires_at forward. That prevents a noisy storage incident from becoming accidental indefinite retention.
Here is a small Python worker skeleton. store is a generic interface with conditional writes; the interface is intentionally narrower than a provider SDK so the retry policy can be tested without network calls.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
import hashlib
import random
import time
@dataclass(frozen=True)
class ThumbnailJob:
tenant_id: str
source_digest: str
variant: str
source_bytes: bytes
def object_key(job: ThumbnailJob) -> str:
return f"tenants/{job.tenant_id}/thumbs/{job.source_digest}/{job.variant}.webp"
def retryable(error: Exception) -> bool:
# The adapter should map provider errors to these HTTP-style classes.
return getattr(error, "status", None) in {408, 429, 500, 502, 503, 504}
def backoff(attempt: int, rng: random.Random) -> float:
ceiling = min(30.0, 0.5 * (2 ** attempt))
return rng.uniform(0.0, ceiling)
def put_thumbnail(store, manifest, job: ThumbnailJob, resize, now, attempts=5):
key = object_key(job)
row = manifest.get(job.tenant_id, job.source_digest, job.variant)
if row and row["state"] == "ready":
return row
expires_at = (row["expires_at"] if row else
now + timedelta(days=7))
payload = resize(job.source_bytes, job.variant)
content_hash = hashlib.sha256(payload).hexdigest()
for attempt in range(attempts):
try:
result = store.put(
key, payload,
content_type="image/webp",
content_hash=content_hash,
if_absent=True,
)
saved = manifest.mark_ready(
tenant_id=job.tenant_id,
source_digest=job.source_digest,
variant=job.variant,
object_key=key,
object_version=result.version,
content_hash=content_hash,
expires_at=expires_at,
)
return saved
except Exception as error:
if not retryable(error) or attempt == attempts - 1:
manifest.mark_failed(
job.tenant_id, job.source_digest, job.variant,
error_type=type(error).__name__,
)
raise
time.sleep(backoff(attempt, random.SystemRandom()))
The if_absent condition is important, but it is not enough by itself. A worker can receive a timeout after the service has accepted the object. On retry, the adapter should inspect the existing object using the stored content hash or version, then treat an exact match as success. If the bytes differ, the worker must stop and surface an integrity conflict; overwriting blindly would make the manifest unreliable.
Five retries are a ceiling, not a promise.
The queue still needs a brake.
Why do thumbnail retries fail even with exponential backoff?
Backoff changes timing. It does not repair an ambiguous result.
A timeout after a write is the common trap: the client does not know whether the remote service committed the bytes. Retrying with a new random key leaves two objects, while retrying with an unconditional overwrite can replace a valid version. Conditional creation, a deterministic key, and a content validator turn that ambiguous result into a checkable decision. The retry loop above also caps attempts at five. That number is a policy example, not a universal setting: it gives the queue a finite cost and gives operators a clear point at which to investigate a persistent failure.
Classify failures before retrying. A connection reset or a 503-style service-unavailable response can be transient. A malformed image, an authorization failure, an invalid bucket policy, or an exceeded object-size limit is a permanent failure for that job. Retrying permanent errors increases queue pressure and hides the real defect. HTTP semantics distinguish temporary availability and rate limiting from client-side request errors; the storage adapter should preserve that distinction instead of returning one generic StorageError.
There is also a queue-level failure mode. Five attempts per object can still create a storm if thousands of thumbnails fail together. Add a per-tenant concurrency limit, jitter, and a dead-letter path with the source digest and last classified error. Do not put the original image bytes into the dead-letter message. The queue should carry references, while access-controlled storage retains the data.
This design has limits. Conditional writes and version checks add metadata calls, and a strict expiry boundary can discard work that would have completed after a temporary outage. That is a deliberate trade-off for predictable deletion. A system that needs long-lived archival recovery should use a separate retention policy and approval path instead of stretching the thumbnail retry loop.
How should retention and tenant exports interact?
Deletion and export need a shared boundary. An export request gets a manifest snapshot or an explicit cutoff time. Each selected row is checked again before it is read, and the exporter records the object version it used. The deletion worker marks a row expired first, then removes the derived object and source according to the same policy. A late retry checks that state and exits before writing.
This is a small but meaningful ordering rule: an expired job must never resurrect an object. If deletion races with a retry, conditional state transitions decide the winner. The retry can proceed only from pending or writing; it cannot transition expired back to ready. The cleanup task should be repeatable because a deletion request can also time out after the remote service has accepted it.
Tenant isolation belongs in every layer, not only in the export query. Build the object key from a validated tenant identifier, bind the manifest lookup to the same tenant, and reject a job whose authenticated tenant does not match its manifest row. The export response should contain object references that are scoped to the export authorization window, not permanent public URLs. This keeps a successful retry from becoming a cross-tenant data leak.
The operational checklist is short enough to keep beside the worker code: define the identity tuple; persist expires_at before the first attempt; use conditional writes; verify an ambiguous result by hash or version; classify status and adapter errors; cap attempts and add jitter; emit tenant-safe failure metadata; snapshot exports; and make cleanup idempotent. Test each transition with a fake store that can fail before acceptance, after acceptance, and during the confirmation read.
What should the evaluation harness prove?
A notebook-to-production path needs a test that behaves like the real failure, not just a test that calls put once. I would run a matrix over one tenant, two variants, duplicate delivery, a timeout after acceptance, a permanent validation error, an expired manifest row, and two tenants requesting exports at the same time. The expected results are more useful than a raw retry count: one logical object per variant, no expired row returning to ready, and no export containing another tenant's key.
Keep timing out of most tests by injecting a clock and a random source. Test the backoff bounds separately. In production, record attempt number, classification, tenant hash, variant, latency, and final state; avoid source filenames and image content in logs. Track the age of writing rows, the number of ambiguous confirmations, and deletion lag. Those measurements reveal whether the issue is storage availability, queue pressure, or a state-machine bug.
Prompt-cost awareness has a parallel here: send the worker and any AI-assisted diagnostic tool the smallest useful record. A digest, variant, state, status class, and timestamps are enough to explain most failures. There is no reason to attach a full image or an entire export manifest to every investigation. Smaller diagnostic payloads make repeated analysis easier to budget and easier to redact.
The decision rule is straightforward. If a retry can identify the same intended object and prove its bytes, retry it. If it cannot distinguish a committed write from an uncommitted one, stop treating the operation as a plain upload and add a confirmation step. If the manifest is expired, delete or quarantine according to policy; never extend retention just because a worker woke up late. That keeps transient storage errors contained while tenant exports and deletion continue to mean what their contracts promise.
Top comments (0)