Object Storage

An object store holds objects (files of any size, from a few bytes to 5 TB) organized into buckets (top-level containers). Each object is addressed by a flat key: s3://my-bucket/users/42/avatar.png. There is no hierarchy — slashes in keys are just characters, not directories (though most tools pretend they are for convenience).

Access is via HTTP REST: PUT to upload, GET to download, DELETE to remove, HEAD for metadata. There is no open, seek, read, write, close — you write the whole object at once (or in parts for large files via multipart upload), and you read the whole object (or a byte range). You cannot edit 4 bytes in the middle of a 1 GB object. To change it, you overwrite the whole thing.

This constraint is the source of object storage's superpower: because objects are immutable once written, replication, durability, and caching become tractable at internet scale. S3 famously offers 99.999999999% (11 nines) durability — once you PUT an object successfully, it is essentially never lost.

Durability vs availability — know the difference:

Object storage markets durability (will my data still be there in 10 years?) and availability (can I read it right now?). These are different.

• Durability: S3 standard is 11 nines — over 10 years, you'd expect to lose one object per 10 million stored. Achieved by replicating each object across multiple facilities and checksumming silently in the background. Your data is essentially permanent.
• Availability: S3 standard is 99.99% — about 50 minutes of downtime per year. Much lower than durability, because outages happen at the facility level even though your data survives.

This is why you should never confuse the two: 'my data is 11-nines durable' does not mean 'I can always read it.' For very rare access (backups, compliance archives), use cheaper tiers with lower availability (S3 Glacier: 99.99% availability, minutes-to-hours retrieval latency) — the durability is still 11 nines, you just can't read it instantly.

The other famous distinction is read-after-write consistency. Modern S3 (since late 2020) is strongly consistent: a successful PUT is immediately visible to a subsequent GET. Older S3 was eventually consistent — a read right after a write could return the old version. If you're designing for older storage systems, assume eventual consistency for overwrites and deletes; verify before you rely on read-after-write.

Pre-signed URLs — the key to scalable uploads and downloads:

Naive pattern: the client uploads a file to your API, your API streams it to S3. This works, but your API server pays for every byte — bandwidth, memory, CPU — and becomes a bottleneck for large files.

Better pattern: the client asks your API for a pre-signed URL — a time-limited, signed URL that grants the bearer permission to upload directly to S3 (or download from it) for, say, the next 5 minutes. The client uploads directly to S3, bypassing your API entirely. Your API then gets a notification (S3 event) that the upload completed and records the metadata in the database.

This pattern is huge for scalable systems:

• Your API never sees the file bytes — bandwidth and memory stay low.
• Uploads scale with S3's capacity, not your API's.
• Downloads can be served directly from S3 (or via a CDN in front of S3), again bypassing your API.
• Permissions are scoped per-object and per-time-window — minimal blast radius if a URL leaks.

For public-read content (e.g., profile avatars), put a CDN in front of S3 and serve directly. For private content (e.g., medical records), pre-signed URLs with short TTLs let specific users download specific objects for a limited time.

When NOT to use object storage:

Object storage is wrong when you need:

• Random access within a file — object storage only lets you overwrite the whole object or read byte ranges. You can't update 4 bytes in the middle. Use block storage or a database.
• Transactional updates — no ACID, no locks, no compare-and-swap on the object's bytes. Use a database.
• Low-latency access — every request is an HTTP round trip (tens of ms). For sub-millisecond access, use local disk or a database with caching.
• Querying the data — object storage has no indexes, no SQL, no filtering by content. (Athena and similar tools scan all objects to query them — slow and expensive at scale. For queryable data, use a database or a data warehouse.)
• Streaming with seeking — you can byte-range GET, but you can't efficiently seek within a compressed file. For media that needs adaptive bitrate streaming, use specialized formats (HLS, DASH) that pre-segment the content.
• Strong consistency on deletes and overwrites — verify your specific provider's guarantees. Some legacy systems are still eventually consistent on these operations.

The right question: 'Do I need random access, low latency, or transactions?' If yes, don't use object storage. If you need cheap, durable, write-once-read-many storage of named blobs — object storage is the answer.

Mental model. Think of Object Storage as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Object Storage mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Example: 5M writes/day × 1 KB/row ≈ 5 GB/day of logical data. Add indexes, replication, backups and growth headroom before sizing a real store.

Answer this without notes: When would you choose Object Storage, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.