Pipes & Filters

Pipes & Filters is one of the oldest architectural patterns — described in the 1996 Pattern-Oriented Software Architecture book and prefigured by Unix shells in the 1970s. The idea is to split a complex transformation into a chain of simple filters connected by pipes.

• Filter: a component that consumes input, performs one transformation, and produces output. Crucially, a filter is self-contained — it doesn't know what produced its input or what consumes its output. Filters should be reusable and composable.
• Pipe: a connector that moves data between filters. Pipes can be in-process queues, OS pipes, message brokers (Kafka, RabbitMQ), or files. The pipe provides buffering, decoupling producers and consumers.

Canonical example (Unix):

cat access.log | grep '500' | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

Each command is a filter; the | is the pipe. Each filter does one thing well; combined, they form a complex transformation.

Modern equivalents:

• ETL: extract from source → transform (clean, enrich, validate) → load into warehouse. Each stage is a filter; queues (Kafka, S3) are pipes.
• Stream processing: Kafka topic → parse → filter → enrich → aggregate → sink. Flink/Spark topologies are pipes-and-filters graphs.
• CI/CD: checkout → build → test → package → deploy. Each job is a filter; artifact storage is the pipe.
• Media processing: decode → resize → watermark → encode. Each filter is a separate process/container.

The pattern's power comes from composability: a filter written for one pipeline can be reused in another, because it has a clean input/output contract and no knowledge of context.

Variants of the pattern:

• Linear pipeline — filters in a straight line, like the Unix example. Simplest form.
• Branching pipeline — one filter's output goes to multiple downstream filters (fan-out). Used for parallel processing or for routing to different sinks.
• Merging pipeline — multiple filters' outputs combine into one downstream filter (fan-in). Used for joining data from multiple sources.
• Cyclic pipeline — output of one filter feeds back to an earlier filter. Used for iterative algorithms (training loops, fixpoint computations).
• Batch vs streaming — pipes can carry batches (one file per stage) or streams (one event at a time). Streaming pipelines (Kafka Streams, Flink) are pipes-and-filters applied to infinite streams.

Key design questions when building a pipeline:

• Where does state live? Stateless filters are easy to parallelize and recover. Stateful filters (aggregations, joins) need checkpointing.
• What's the failure model? Should a failed filter retry, dead-letter, or abort the pipeline? Idempotency matters — reprocessing must be safe.
• What's the back-pressure strategy? If a downstream filter is slow, the upstream filter must slow down (blocking) or buffer (memory/disk risk).
• How is the pipeline observed? Per-stage throughput, error rate, queue depth — without these, you can't find the bottleneck.
• What's the granularity of the filter? Too coarse → monolith. Too fine → overhead per stage exceeds the work.

Strengths:

• Composability — filters are reusable across pipelines. A parse-json filter is useful in dozens of pipelines.
• Testability — each filter has a clear input/output contract; unit test it in isolation.
• Parallelism — independent filters can run in parallel; multiple consumers of a pipe can scale a slow stage.
• Decoupling — filters don't know each other; replacing one doesn't affect others (as long as the contract holds).
• Observability — per-stage metrics reveal the bottleneck.
• Heterogeneous tech — filters can be written in different languages; pipes are language-agnostic (JSON, Protobuf, Avro).

Weaknesses:

• Throughput cap = slowest stage — like any pipeline, the slowest filter sets the rate. Speeding up other stages does nothing.
• Latency overhead — each stage adds serialization, queueing, and processing latency. A 10-stage pipeline has 10× the per-event latency of a monolithic function.
• Error handling complexity — what happens when filter #4 fails? Retry just that stage? Replay from filter #1? Dead-letter? Each model has trade-offs.
• Operational complexity — many moving parts to deploy, monitor, and version.
• Schema evolution — a change to a filter's output schema affects every downstream filter.
• Over-decomposition — pipelines with 50 stages where 5 would do; per-stage overhead exceeds the work.

Mental model. Think of Pipes & Filters as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Pipes & Filters mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Pipes & Filters, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.