Design URL Shortener

What are we designing? A service that takes a long URL like https://www.example.com/very/long/path?with=query&params=here and returns a short alias such as https://bit.ly/aB3x9. When anyone visits the short URL, the service redirects them to the original long URL. Alongside the redirect, the service records click analytics (referrer, geolocation, timestamp, user-agent) so marketers can measure campaign performance.

Two functional primitives define the system: a shorten endpoint (write path) and a redirect endpoint (read path). The interesting design tension comes from three facts: reads dwarf writes by roughly 100:1, every short code must be globally unique forever, and the redirect latency budget is brutal because a redirect adds a round trip to the user's actual destination.

Functional requirements.

• A user submits a long URL and receives a short code (e.g. aB3x9) that maps back to the original.
• Visiting https://short.io/<code> returns an HTTP redirect to the long URL.
• Users may optionally pick a custom alias (e.g. short.io/my-summit-2024).
• The service records click events: timestamp, IP, referrer, user-agent.
• Links can be expired or deleted by their owner.

Non-functional requirements.

• Read latency: p99 < 30 ms for redirects (the redirect is on the critical path of the user's browser).
• Availability: 99.99% for reads (a dead short link breaks every SMS / email that embeds it).
• Durability: shortened URLs must keep resolving for years; we cannot lose a mapping.
• Write latency: < 200 ms is fine — shortening is not latency-critical.
• Analytics freshness: clicks visible in dashboards within ~1 minute (eventual).
• Scale: 100M new URLs/month, 10B redirects/day at peak (read-heavy, 100:1).

Non-goals (v1). No A/B redirect variants, no password-protected links, no per-link rate limiting. These can be layered on later without changing the core.

Capacity estimation. Assume 100M new short links/month and a 100:1 read:write ratio.

Writes. 100M / 30 days / 86400 s = ~40 shortens/sec average, peak ~200/sec. Trivial — a single database node could handle this.

Reads. 100M x 100 = 10B redirects/month = ~3,850 redirects/sec average, peak ~20,000/sec (3-5x for marketing bursts, US-east daytime). At 20K QPS a single Redis shard is borderline; we will need a small cache cluster.

Storage. Each mapping is ~500 bytes (short code, long URL, owner, created_at, expires_at, metadata). 100M/month x 500 B = 50 GB/month = ~600 GB/year. Five years = 3 TB. Fits in a sharded PostgreSQL cluster or a DynamoDB-style KV store; no need for Hadoop-class storage.

Bandwidth. Redirect response is ~1 KB (HTTP headers + 302 + cache headers). 20,000 QPS x 1 KB = 20 MB/s = ~160 Mbps peak egress. Add CDN caching for hot links and origin egress drops 10x.

Analytics writes. 10B clicks/day, each event ~200 bytes -> 2 TB/day of click logs. These go to a streaming pipeline (Kafka -> S3/warehouse), not the redirect path.

Short-code keyspace. With 7-character base-62 codes (a-z, A-Z, 0-9 = 62 chars), the keyspace is 62^7 = 3.5 trillion. Plenty for years of growth; 6 chars (62^6 = 56 billion) would also suffice but leaves less collision margin.

APIs.

POST /v1/shorten
  body: { long_url, custom_alias?, expires_at?, owner_id? }
  resp: { short_code, short_url, long_url, created_at }

GET  /<short_code>           -> 301/302 redirect to long_url

DELETE /v1/links/<short_code>   -> mark deleted (returns 410 on redirect)

GET  /v1/links/<short_code>/stats?from=&to=   -> click counts, top referrers

Two design choices worth flagging. First, 301 vs 302: a 301 Moved Permanently is browser-cached, which is great for latency but means we lose the click event (the browser never hits us again). 302 Found is not cached, so every click traverses our service — required for analytics. bit.ly uses 301 with a separate + suffix (e.g. bit.ly/aB3x9+) for the stats page. We will use 302 with Cache-Control: private, max-age=30 so clicks are counted but a single user's repeat clicks within 30s don't double-hit origin.

Second, the redirect endpoint is unauthenticated and on the hot path; it must not touch the database on every request. A Redis cache-aside in front of the mapping table absorbs >95% of reads.

Data model.

Link mappings (sharded SQL, primary store):

links (
  short_code   VARCHAR(7)  PRIMARY KEY,
  long_url     TEXT        NOT NULL,
  owner_id     BIGINT,
  created_at   TIMESTAMP,
  expires_at   TIMESTAMP NULL,
  deleted      BOOLEAN     DEFAULT false
)

Index (owner_id, created_at) for the user dashboard.

ID generator. Two options:

1. MD5(long_url) then base-62 encode the first 7 chars — deterministic, idempotent (same long URL -> same short code), but collision-prone and you cannot choose the length.
2. Counter-based: a global auto-increment (Ticket Server pattern, like Flickr's ticket servers) produces a 64-bit ID, then base-62 encode it. Deterministic, collision-free, sortable by creation time.

We pick the counter-based approach with two independent ticket-server DBs (odd/even IDs) for HA, then base-62 encode. 7 chars of base-62 supports 3.5 trillion URLs.

Click events (analytics path, separate pipeline):

click_events (
  event_id   BIGINT,
  short_code VARCHAR(7),
  ts         TIMESTAMP,
  ip         INET,
  referrer   TEXT,
  ua         TEXT
)

Sharded into Kafka -> S3 / ClickHouse for analytics.

Deep dive: short-code generation and collision handling. The single hardest design decision is how to mint short codes that are unique, short, and unguessable enough to discourage enumeration. Three families of approach:

1. Hash-and-truncate (MD5(long_url)[:7]). Pros: deterministic, idempotent, no coordinator. Cons: collisions grow quadratically (birthday paradox — 62^7 keyspace means ~50% collision probability around 70M codes); also reveals that two users shortened the same URL.

2. Counter + base-62 encode (our choice). A 64-bit monotonic counter from a ticket server is encoded as [0-9a-zA-Z]. Pros: globally unique by construction, lexicographically sortable, no collisions ever, 7 chars cover 3.5T URLs. Cons: requires a coordination service (the ticket server), and codes are enumerable — an attacker can scan bit.ly/aaa, bit.ly/aab, ... and harvest every link. Mitigation: either accept this (most public shorteners do, and add abuse detection) or XOR the counter with a per-service secret to obfuscate ordering.

3. Random 7-char codes with retry on collision. Pick uniformly from 62^7, check the DB, retry if exists. Pros: no coordinator, codes unguessable. Cons: as the keyspace fills, collision retries explode (birthday paradox again); also requires a unique-index check on every write.

Ticket server HA. Flickr's pattern: two MySQL masters, one hands out even IDs, the other odd. Either can fail and the system keeps minting IDs. The ticket server is the one true write-bottleneck — at 200 writes/sec it is nowhere near saturation, but we still want redundancy.

Cache-aside specifics. On a redirect: (1) GET short_code from Redis. (2) On miss, SELECT long_url FROM links WHERE short_code=?; if found, SET into Redis with TTL of 1 hour (so stale-then-deleted links clear themselves). (3) Return 302. We pre-warm the cache for any code that trends; the hot set is small (Pareto: top 1% of links = 80% of traffic).

Anti-abuse. A shortener is a phisher's best friend. We add a Google-Safe-Browsing-style blocklist check on shorten, and on redirect we check a bloom filter of known-bad codes. Reports of malicious links are pushed to the blocklist within seconds.

Bottlenecks and failure modes.

• Hot short code. A celebrity tweets a bit.ly link -> 100K redirects in 60 seconds, all to one Redis key. The shard owning that key becomes a hotspot. Mitigation: detect hot keys (top-K window) and replicate them to a pool of read-only 'shadow' keys (aB3x9__1, aB3x9__2); the redirect service picks one at random. CDN also absorbs most of this if the link owner opted into longer cache TTLs.

• Ticket server failure. If both ticket DBs are down, no new links can be created (but redirects still work). Mitigation: each ticket server pre-allocates a block of 10K IDs into an in-memory counter on the shorten service, so the service can mint IDs for several minutes even with the ticket server gone.

• Redis cache failure. If Redis dies, all reads fall through to the DB. At 20K QPS this will probably crush the DB. Mitigation: Redis cluster with replicas + client-side circuit breaker that, on Redis failure, returns 302 to a stale cached long URL from a local LRU (better to redirect to the right place than to 5xx). If no local cache, return 503 with Retry-After and degrade.

• CDN misconfiguration caching a 404. If we cache 404 Not Found for a code that doesn't exist yet (eventually consistent propagation), a later shorten of that code will return 404 for the CDN TTL. Mitigation: never cache 4xx/5xx at the CDN; only cache 302s.

• Analytics pipeline backpressure. If Kafka backs up, the redirect service must not block. Mitigation: fire-and-forget analytics writes via an async client with a tiny in-memory buffer; drop events rather than stall a redirect.

• Link rot. A long URL 404s years later. We can periodically crawl long URLs in the background and flag dead links.

Scaling strategy and trade-offs.

Read path scaling. The cache hierarchy is CDN -> Redis cluster -> DB. Each layer absorbs roughly 10x of the previous: CDN catches 60% (repeat-clicks by the same user within TTL), Redis catches 35% of what's left, the DB only sees 5%. At 20K QPS this means ~1K QPS hits the DB — comfortable for a 3-shard PostgreSQL cluster.

Write path scaling. The bottleneck is the ticket server. We can shard ticket servers by geography (US-east mints IDs 0..N, EU mints N+1..2N) but then IDs are no longer globally sortable. Alternative: use Snowflake-style IDs (timestamp + worker_id + sequence) — no coordinator, sortable, fits in 64 bits.

Database sharding. Shard links by short_code hash with consistent hashing so adding a shard doesn't rehash everything. Each shard is a primary + 2 replicas. Cross-shard queries (e.g. 'all links owned by user X') require a secondary index shard keyed by owner_id.

Multi-region. Reads must be fast globally. We deploy the redirect service + Redis cache in 5+ regions (us-east, us-west, eu-west, ap-south, ap-northeast). The links DB is replicated asynchronously to read replicas in each region. Writes (shorten) go to a single primary region to keep ID assignment simple; this is fine because writes are not latency-sensitive.

Trade-offs made explicit.

• We chose 302 over 301 — gained accurate click analytics, lost free browser caching (mitigated by Cache-Control: max-age=30).
• We chose counter-based codes over hash-based — gained no collisions and sortability, lost idempotency (the same long URL shortened twice yields two codes) and accepted enumerability.
• We chose Redis cache-aside over read-through — gained simplicity, lost automatic freshness (a deleted link can be served stale for up to Redis TTL). We mitigate by issuing DEL on delete.
• We chose single-region writes — gained simple ID assignment, lost write availability if the primary region fails (acceptable: writes are <1% of traffic and not on the user's critical path).

Mental model. Think of Design URL Shortener as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Design URL Shortener mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Design URL Shortener, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.