Federation

Federation splits a single logical database into multiple physical databases along functional boundaries. Where sharding divides one table's rows across machines, federation divides the schema: a users database, a posts database, a messages database, a billing database. Each is independently scaled, deployed, backed up, and operated.

The motivation isn't usually raw throughput — it's organizational and operational. A single shared database means:

• One team's schema migration blocks everyone's deploy.
• A hot row in messages saturates the entire instance, slowing down login.
• A catastrophic bug in one service corrupts data for everyone.
• Engineering teams queue on a single DBA team to provision capacity.

Federation solves this by giving each functional slice its own database (and, in mature versions, its own service that owns it). The cost is that cross-domain queries — once a JOIN — now require either application-level stitching, denormalized copies, or asynchronous event-based synchronization.

This pattern is sometimes called 'vertical partitioning' in older texts, but that term is overloaded — in SQL circles it can also mean splitting a table's columns across tables on the same machine. 'Federation' is the unambiguous term for splitting by function across machines.

Choosing federation boundaries is the same art as choosing microservice boundaries: align with bounded contexts (domain-driven design) and team ownership. A few rules of thumb:

1. Split by entity that's read/written together — if every 'create post' writes only to posts and tags, those can share a database; if it also writes to a user's post_count, consider whether that count belongs in the posts DB instead.
2. Split by access pattern — billing (transactional, ACID) and analytics (read-heavy, columnar) want different databases anyway.
3. Split by team — Conway's Law says your database shape will mirror your org chart. Embrace it: one team owns one database, end to end.
4. Split by scale curve — if messages grow 10× faster than users, separating them lets messages get their own capacity planning.
5. Don't split by foreign key — that's a sign you need the data together, not apart.

Once split, the application code is responsible for assembling cross-domain views. A user's profile page that shows their posts, message count, and recent purchases must query three services and merge results in memory — or rely on denormalized projections updated by events.

Federation and sharding are orthogonal and often combined:

• Federation splits by function: users DB, posts DB, messages DB. Each slice owns different tables.
• Sharding splits by row within one table: users 1-1M on shard A, 1M-2M on shard B.
• Both: a federated posts DB that's also sharded by user_id for write scaling.

The progression most teams follow:

1. Single database — start here. Simplest. Fine until ~10k QPS or team coordination pain.
2. Read replicas — scale reads without changing the data model. Writes still bottlenecked.
3. Cache layer (Redis/Memcached) — absorb read traffic for hot data. Writes still bottlenecked.
4. Federation — split by function. Solves the coordination problem, lets each piece scale independently. Cross-domain queries now require stitching.
5. Sharding within a federated slice — when one slice's write throughput exceeds a single machine.

A common mistake is jumping to sharding before federation. Federation is usually cheaper, simpler, and addresses the operational pain first. Sharding is complex (rebalancing, cross-shard queries) and should be reserved for when even a federated slice outgrows one machine.

Federation shifts you from ACID transactions within one database to eventual consistency across databases. A user registration flow that used to be INSERT user; INSERT welcome_message; INSERT default_preferences; COMMIT is now three writes to three databases — and there's no atomic commit.

Patterns for handling this:

• Saga pattern — orchestrate multi-database writes as a sequence with compensating actions on failure. The de-facto replacement for 2PC in microservices.
• Outbox pattern — write the data and the 'event to publish' in the same database transaction; a separate process reads the outbox table and publishes events. Guarantees the event is emitted if and only if the data is committed.
• Idempotent retries — every operation must be safely retriable; partial failures must not corrupt state.
• Acceptable inconsistency — for non-critical data (analytics counters, search index updates), eventual consistency is fine.

Two-phase commit (2PC) across federated databases is technically possible but slow, fragile, and almost universally avoided in modern systems. Sagas are the modern answer.

Mental model. Think of Federation as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Federation mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Federation, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

A rough capacity check: required write throughput ≈ peak writes/s × average record size. At 5,000 writes/s and 2 KB average payloads, the raw incoming data stream is about 10 MB/s before indexes, replication and overhead.