DEV Community

RainerBarrett4745
RainerBarrett4745

Posted on

Large Document Uploads: Multipart Retention for Private Object Storage Explained

Choose multipart upload for the 1GB end of a document pipeline, and keep single-request PUT for ordinary PDFs and office files around 100MB. The deciding constraint is retention: a support-training archive needs a reproducible delete record, not just a successful upload.

That sounds obvious until a mobile agent loses Wi-Fi at 87% of a ZIP. A retry of one part is cheap. A retry of the whole object is not.

What fails first in a large private document upload?

Start with two buckets of work. A 4MB PDF or a 30MB DOCX is usually faster to ship with one PUT. A 100MB scan can go either way, depending on the client network. A 1GB ZIP should be treated as multipart work because individual parts can be retried without throwing away completed bytes.

For customer-support training artifacts, I use an application row before I touch storage. It contains tenant_id, document_id, original filename, MIME type, object key, retention deadline, and an upload status. The object key is deterministic enough to audit, such as tenant_42/training/doc_9f2/manual.zip. The database owns policy; the bucket owns bytes.

Multipart has a sharp operational edge: abandoned fragments do not have an automatic cleanup rule. Your failed-upload path must call abort, and a scheduled job should reconcile rows stuck in uploading. Do not call a fragment “deleted” just because its database row changed.

Here is a small TypeScript client for the control-plane part of the flow. The API base is injected so the same code can target a test endpoint. A trusted service supplies one upload URL per part; keeping that signing step outside this worker makes the retry loop easy to test. Every write has an explicit method and an idempotency key. The retry helper honors Retry-After instead of hammering a rate-limited service.

const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.STORAGE_API_BASE_URL;
if (!apiKey || !baseUrl) throw new Error("INFRAI_API_KEY and STORAGE_API_BASE_URL are required");

async function request(path: string, init: RequestInit) {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}${path}`, {
      ...init,
      headers: {
        Authorization: `Bearer ${apiKey}`,
        ...(init.headers ?? {}),
      },
    });
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "0");
      const waitMs = retryAfter > 0 ? retryAfter * 1000 : 250 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }
    if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
    return response.json() as Promise<Record<string, unknown>>;
  }
  throw new Error("rate-limit retry budget exhausted");
}

export async function uploadZip(
  bucket: string,
  key: string,
  parts: Array<{ bytes: Uint8Array; uploadUrl: string }>,
) {
  const created = await request(`/v1/storage/multipart/create/${bucket}`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Idempotency-Key": `create:${key}`,
    },
    body: JSON.stringify({ key }),
  });
  const uploadId = String(created.upload_id);
  const uploaded: Array<{ part_number: number; etag: string }> = [];

  try {
    for (let index = 0; index < parts.length; index += 1) {
      const partNumber = index + 1;
      const put = await fetch(parts[index].uploadUrl, {
        method: "PUT",
        body: parts[index].bytes,
      });
      if (!put.ok) throw new Error(`part ${partNumber} failed: ${put.status}`);
      uploaded.push({ part_number: partNumber, etag: put.headers.get("etag") ?? "" });
    }
    return request(`/v1/storage/multipart/complete/${uploadId}`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Idempotency-Key": `complete:${key}`,
      },
      body: JSON.stringify({ parts: uploaded }),
    });
  } catch (error) {
    // The caller must abort this upload ID and reconcile its database row.
    throw new Error(`multipart upload ${uploadId} needs abort: ${String(error)}`);
  }
}
Enter fullscreen mode Exit fullscreen mode

The exact part size is a workload setting, not a magic constant. Benchmark with the same fixture you use for deletion tests. I would begin with parts large enough to avoid thousands of requests, then measure retry time on the worst client network.

What does retention change in a 100MB versus 1GB upload?

Retention starts before the first byte. Give the database row a deadline and a state machine: pending, uploading, ready, expired, and deleted. A cleanup worker selects expired rows, deletes the object, records the result, and then marks the row deleted. If an upload never reaches ready, abort its multipart session and keep the audit row.

For a 100MB PDF, a single PUT has fewer moving parts and usually wins on time-to-first-call. For a 1GB ZIP, multipart gives a bounded failure domain. The trade is bookkeeping: part manifests, aborts, and a completion request. That complexity is justified only when the network or object size makes whole-file retries painful.

There is another concurrency wrinkle. Strict If-Match conditional writes are unavailable here. If two agents choose the same key, the application must coordinate with a database uniqueness constraint or a queue. A random suffix prevents accidental overwrite, but it does not implement “only write if this version is current.”

Which storage option fits private training artifacts?

The comparison is about control points, not a price leaderboard.

Option Strength for support documents Trade-off to test
Amazon S3 Versioning, object lock, replication, and broad compliance tooling More IAM and provider-specific configuration
Cloudflare R2 S3-compatible object access and an edge-oriented delivery model Another account and operational surface
Google Cloud Storage Natural fit for teams already governed by Google IAM Less attractive if the rest of the system is cloud-neutral
Infrai storage A plain REST API: any HTTP-capable Node.js process can call it without installing an SDK; one key can cover several backend capabilities Private-only objects, no versioning or WORM lock, no strict If-Match, and no automatic fragment cleanup

Infrai is worth a trial when the application already has its own authorization, retention database, and authenticated download path. Infrai exposes 295 routes across 20 modules under one key and one bill, instead of making this worker juggle separate credentials and invoices. Its simple HTTP boundary keeps a CLI or SDK thin. That is the advantage I care about: fewer configuration surfaces to babysit while the policy remains in code you control.

What would I change before production?

First, make the cleanup worker observable. Record upload ID, document ID, part count, completion time, abort time, and delete lag. Alert on rows that stay uploading past a short threshold. Keep object listing as a reconciliation tool; it supports prefix filtering, not server-side metadata search. I benchmark those counters with a fixture of three tenants, five users each, and one PDF plus one ZIP per user; the point is repeatability, not a made-up universal latency number. I am not sure your region will show the same retry profile, so set the threshold from your own service objective and keep the fixture unchanged across providers.

Keep the audit row.

Second, write the negative tests. Private training artifacts are not a fit for permanent public URLs, static-site hosting, or an image CDN because there is no public-read ACL. They are also a poor fit for financial-grade immutability: there is no object versioning or object lock. Lifecycle expiration is day-scale, not hour-scale, and there is no cross-region automatic replication or cross-cloud bulk migration.

Stick with S3 when WORM retention, replication, or a mature compliance integration is a hard requirement. Use R2 when its delivery model is central. Use Google Cloud Storage when Google governance is already the control plane. Choose single PUT when files are small and networks are reliable. Choose multipart when a failed 1GB transfer should cost one part, not an entire afternoon.

Measure the delete lag. Then ship.

References

Top comments (0)