Large blob handling
Multipart upload, object storage, metadata stores, signed URLs, and lifecycle policies.
Large files change the shape of a system because the payload is bigger than the request logic. This page gets you ready to keep bytes off the app path while still enforcing safety and ownership.
Read this if your last attempt…
- You wrote "upload to our server; our server writes to S3"
- You don't know what a signed URL is
- You can't explain chunked / resumable uploads
- You haven't thought about content-type spoofing or file-size limits
The concept
The anti-pattern: client POSTs a 500 MB video to your app server; app server streams it to S3. You just paid bandwidth + memory + 10s of latency — and if your app process dies mid-upload, the user starts over.
The right pattern: direct-to-blob with signed URLs.
1. Client asks app for an upload URL (POST /uploads with metadata: size, content-type, filename).
2. App validates (size limits, content-type allowlist, auth), creates a DB row with status=pending, generates a pre-signed S3 PUT URL scoped to that specific object key, and returns it.
3. Client PUTs the file directly to S3 — app server is out of the data path.
4. S3 triggers a Lambda / webhook on upload completion → app marks DB row status=ready + runs virus scan / transcoding / indexing.
App issues the URL; client uploads directly to S3; app never sees the bytes. S3 event triggers post-processing.
Upload strategies.
| Strategy | Good for | Bad for |
|---|---|---|
| POST through app server | Tiny files (<1 MB), tight auth | Anything bigger — bandwidth + memory tax |
| Signed URL PUT | Most files up to ~100 MB | No restart on failure |
| Multipart / resumable | Large files, flaky networks, progress UI | Small files (overhead not worth it) |
| Browser direct with tus.io | Consumer apps with progress UX | Requires tus server |
How interviewers grade this
- Client uploads directly to blob storage via signed URL; app is out of the data path.
- App validates size + content-type BEFORE issuing the URL.
- The signed request binds the declared type and size (exact length on a PUT, a range with a POST policy); the real check is size, magic bytes and a scan after upload.
- Post-processing (scan, transcode, index) is event-driven, not synchronous.
- Large files use multipart / resumable upload.
Variants
Signed URL PUT (simple)
One signed URL; one PUT; done.
The default for files up to ~100 MB. Works with any S3-compatible blob store. Client gets a URL, PUTs, we get an event.
Pros
- +Dead simple
- +Zero bytes through your app
- +Works everywhere
Cons
- −No resume on failure
- −Bad for multi-GB files
Choose this variant when
- Files up to ~100 MB
- Stable networks
S3 multipart / tus resumable
Split into chunks; upload each; atomic complete.
For anything big. 5 MB minimum chunk for S3 multipart (except the last). Failed chunks retry independently. Client state tracks uploaded parts for resume.
Pros
- +Resumable on failure
- +Parallel chunk uploads for speed
- +Progress UI natural
Cons
- −More complex client logic
- −Abandoned uploads need cleanup policy (S3 lifecycle rule)
Choose this variant when
- Files > 100 MB
- Flaky networks (mobile, international)
- User expects a progress bar
Pre-signed POST (HTML form)
Client posts a form directly to S3 with a signed policy.
Pre-signed POST uses a signed policy document, which can enforce things a pre-signed PUT can't: a size range (content-length-range) and conditions on the key prefix and content type. That makes it the better choice when you need S3 itself to reject oversized uploads. It also works from a plain HTML form (no JS).
Pros
- +S3 enforces a size range and key/content-type conditions
- +Works without JS
- +Well-documented
Cons
- −More complex signing (policy doc)
- −Multipart form body instead of a raw PUT
Choose this variant when
- You need a server-enforced max size
- Browser uploads of user files
- JS-free form uploads
Worked example
Scenario: video upload for a course platform. Videos 50 MB – 5 GB.
Flow:
- 1Client POST /uploads with {filename, size, mime}. App checks size ≤ 10 GB, mime in [video/mp4, video/webm, …], auth OK. Inserts DB row
uploads(id, user_id, status=pending, size, mime). - 2App creates a multipart upload in S3 (InitiateMultipartUpload), returns
upload_id+ chunk-signing endpoint. - 3Client splits file into 10 MB chunks. For each chunk, requests a signed UploadPart URL from app; PUTs the chunk. App authorises per-chunk.
- 4Client POSTs /uploads/:id/complete with part ETags. App calls S3 CompleteMultipartUpload.
- 5S3 event → Lambda → enqueue transcoding job (MediaConvert or FFmpeg on containers). Transcode into HLS/DASH renditions (240p, 480p, 720p, 1080p). Write manifest + chunks to CDN-fronted bucket.
- 6App updates row status=ready + CDN URL.
Cleanup: S3 lifecycle rule aborts multipart uploads abandoned > 24h. uploads rows in pending > 7d are deleted by a sweeper job.
Security:
- Signed URLs scoped to one specific object key, one user, 1-hour expiry. The app caps the number of parts it will sign and binds an exact Content-Length to each part URL, so the total can't run past the declared size; after completion it checks the object size anyway and deletes or quarantines anything oversized.
- The signature binds the declared Content-Type (a mismatched header fails the signature check), but that's the client's claim, not the file's real type.
- After CompleteMultipartUpload: HEAD the object to confirm its size, sniff the magic bytes, and virus-scan in the raw bucket before moving it to the clean bucket.
- Transcoding bucket is behind CloudFront with signed URLs for paid content.
Good vs bad answer
Interviewer probe
“How do users upload 500 MB videos to your app?”
Weak answer
"They POST to our /upload endpoint, and we stream it to S3."
Strong answer
"Direct-to-S3 via signed URLs — the app server never sees the bytes. Client POSTs metadata; we validate size + mime + auth; issue an S3 multipart InitiateUpload + per-chunk signed URLs. Client PUTs chunks direct to S3, calls CompleteMultipartUpload. S3 event triggers a Lambda that marks the DB row ready and enqueues transcoding. For files this big, multipart is mandatory — single PUT of 500 MB fails often on mobile. The app is out of the data path; our bandwidth bill is metadata + events only. Security: per-part signing with a capped part count and an exact Content-Length per part, the declared content-type bound to the signature, short expiry, and — because the header is only a claim — size check, magic-byte sniffing and a virus scan in a quarantine bucket before publishing."
Why it wins: Direct-to-blob, multipart, security boundaries, event-driven post-processing, honest about bandwidth cost.
When it comes up
- Users upload images, video, documents, or any large file
- The interviewer says "store and serve user media"
- A flow involves files bigger than a few MB
- Profile pictures, attachments, video uploads, backups
Order of reveal
- 11. Keep the app out of the data path. The app never streams the bytes. The client uploads directly to blob storage via a pre-signed URL.
- 22. Validate before issuing. Before I hand out the URL, the app checks size limits, a content-type allowlist, and auth. S3 enforces the size (exact length on a PUT, a range with a POST policy); the content type is only a claim, so I verify it after upload.
- 33. Multipart for large files. Anything over ~100 MB uses multipart / resumable upload, so a network blip retries one chunk instead of the whole file.
- 44. Event-driven post-processing. The upload-complete event triggers virus scan, transcoding, thumbnailing, and indexing — none of it blocks the request.
- 55. Downloads. Serve via signed GET URLs fronted by a CDN; I only proxy bytes through the app when DRM or per-request watermarking demands it.
Signature phrases
- “The bytes never flow through my app — signed URL, direct to S3.” — The core pattern that separates a scalable design from a toy one.
- “I validate before issuing the URL, let S3 enforce size, and verify the real file type after upload.” — Shows you know what the signature does and doesn't protect.
- “Files over ~100 MB are multipart so a blip retries one chunk.” — Demonstrates you handle real-world flaky uploads.
- “Post-processing is event-driven off the upload-complete event.” — Keeps the write path fast and decoupled.
Likely follow-ups
?“Why not just let the app receive the upload? It is simpler.”Reveal
It is simpler until you count the real costs. Even a streaming proxy that never buffers the whole file holds a connection and a worker for the entire upload — minutes for a slow mobile client — pays the bandwidth twice (in, then out to S3), needs careful backpressure and timeouts for slow clients, and a deploy or crash mid-upload loses the whole transfer. Direct-to-blob takes all of that off your app and is the common industry pattern.
?“How do you stop a user uploading a 10 GB malicious payload?”Reveal
Layers: (1) a content-type allowlist and size limit checked before the URL is issued; (2) size enforced by S3 — exact Content-Length on a signed PUT, content-length-range on a POST policy, or for multipart a capped part count with an exact Content-Length signed into each part URL; (3) after upload, verify the real size and magic bytes and run a virus scan in a quarantine bucket before the object is exposed to anyone else; (4) per-user upload rate limits and an account quota so a free tier cannot be flooded with junk. No single check is enough — S3-side size enforcement plus post-upload verification are the key two.
?“How do you serve private downloads — say paid course videos?”Reveal
Pre-signed GET URLs with a short expiry, fronted by a CDN with signed-URL/signed-cookie support (CloudFront). The client gets a time-boxed URL, the CDN validates the signature and serves from the edge, and the origin bucket stays private. I only stream bytes through the app when I need per-request logic the CDN cannot do — DRM license issuance or per-user watermarking — and even then only the license step, not the media payload.
Common mistakes
Your app pays for the bandwidth, memory for streaming, and any failure restarts the whole upload. Use signed URLs.
A client can upload an executable labelled image/png. Signing the Content-Type header only binds the claim. Enforce size at S3 where you can (exact Content-Length on a PUT, content-length-range on a POST policy), then verify the real size, sniff magic bytes and scan in a quarantine bucket before anyone else can fetch the file.
Each abandoned multipart upload consumes storage and accrues cost. S3 lifecycle rule to abort after N days.
Same pattern, download direction. Use signed GET URLs + CDN; don't stream bytes through your app. Exception: per-request watermarking or DRM that genuinely needs server-side assembly.
Practice drills
Walk me through an upload of a 2 GB video from a mobile client on 4G.Reveal
Client POSTs metadata → app validates → creates S3 multipart upload, returns upload_id. Client splits into (say) 20 chunks of 100 MB. For each chunk, requests signed UploadPart URL, PUTs to S3. If a chunk fails (tunnel, network flip), retries that chunk only. On completion, calls /uploads/:id/complete; app calls S3 CompleteMultipartUpload. S3 fires event; Lambda enqueues HLS transcoding; app marks ready once transcoded. User sees progress through all this, backed by real per-chunk acks.
Interviewer: "can't we just let the app handle uploads? It's simpler."Reveal
"Simpler" until you look at the real costs. Even a well-written streaming proxy (no full buffering) holds a connection and a worker for the whole upload — 20 minutes for a slow mobile client — pays the bandwidth twice (in, then out to S3), needs backpressure and timeout tuning for slow clients, and a deploy or crash mid-upload forces the user to restart. Direct-to-blob removes all of that from your app and is the common industry pattern.
How do you stop users from uploading a 10 GB malicious payload?Reveal
Layered: (1) content-type allowlist and size limit checked before issuing the URL; (2) size enforced by S3 — exact Content-Length signed into a PUT, or content-length-range in a POST policy, or a capped number of signed parts, each with an exact Content-Length, for multipart; (3) after upload, verify the real size and magic bytes and virus-scan in a quarantine bucket before exposing it to other users; (4) rate-limit uploads per user (abuse pattern = flood a free tier with trash); (5) quota per account.
Cheat sheet
- •App never sees the bytes. Direct-to-blob via signed URL.
- •Small (<100 MB): signed PUT. Large: multipart / tus.
- •Validate size + mime + auth BEFORE issuing the URL.
- •Enforce size at S3 (exact length on PUT, a range with a POST policy); treat Content-Type as a claim.
- •Quarantine → verify size + magic bytes → scan → publish.
- •Event-driven post-processing (virus scan, transcode, index).
- •S3 lifecycle cleans up abandoned multipart uploads.
- •Downloads: signed GET + CDN. Don't proxy.
Practice this skill
These problems exercise Large blob handling. Try one now to apply what you just learned.
Read this if