How Compound AI Systems Work: Multi-Model Pipelines Explained

For about six months I chased the single best model. Every leaderboard shift, I’d re-read the benchmarks and wonder if I was paying for the wrong subscription. That was the wrong frame, and it cost me time.

Here’s the promise: by the end of this, you’ll see that how compound AI systems work multi-model pipelines isn’t some enterprise architecture you need to build — it’s the industrialized version of a decision you already make by hand every day. I’ll walk through what such a system actually is, how the pieces connect, where I got it wrong, and a plain table that untangles it from mixture-of-experts and AI agents.

The reframe I want you to leave with: you are already the router.

The moment I realized I was routing all along

Watch what you do in a normal work morning.

A long email thread lands. You paste it into ChatGPT and ask for a quick summary, because GPT is fast and cheap for that. Then a genuinely hard problem shows up — a contract clause, a tangled logic bug in a spreadsheet — and you open Claude instead, because it holds a long argument together better. Later you need a number that changed this week, so you switch to Gemini for live web. And when the answer has to come from your files, you use whichever chat can read your uploaded documents.

Nobody taught you a system. You just learned, hit by hit, which paid tool to open for which job. That habit has a name in AI research. It’s called model routing, and it sits at the center of every multi-model pipeline.

The only difference between you and an orchestrated setup is that yours runs in your head, ad hoc, one tab at a time. The system version wires those same choices into software so they happen automatically. Once I saw that, the whole topic stopped feeling like someone else’s infrastructure problem.

What the system actually is

Let me give you the clean definition first, then unpack it.

A compound AI system is an AI application built from several connected components — multiple models, a retriever, tools, and control logic — instead of one giant model answering alone. A router or fixed pipeline decides which piece handles each step, so the system as a whole outperforms any single model call.

That framing comes from a 2024 essay by Berkeley’s AI research lab, The Shift from Models to Compound AI Systems{rel=”noopener”}, which is where the term was coined. Their argument, in plain words: the best AI results in the real world no longer come from one bigger model. They come from systems that combine models with retrieval, tools, and logic.

The Berkeley team backed this with a survey number I keep returning to: roughly 60% of enterprise language-model applications already used some form of retrieval-augmented generation, and about 30% used multi-step chains. Read that as a signal, not a stat to memorize — even inside serious companies, the winning pattern was already “combine components,” not “buy the smartest single model.” I’d been optimizing the exact variable that mattered least.

So the system has, at minimum, three moving parts:

• Components — the specialized pieces. Different models, a search step, a code runner, a database lookup.
• A control flow — the logic that decides the order and the routing. This is the “router” leg.
• An interface — where you (or an app) hand it a request and get an answer back.

Vendors like Databricks{rel=”noopener”} and IBM{rel=”noopener”} describe the same shape, usually to sell an orchestration product. That’s fine — the concept is sound. But their framing assumes you’re building this for a company. You’re not. You’re trying to understand what you’re already using.

The everyday multi-model pipelines you already touch

Here’s the part the enterprise pages skip: you don’t have to build one. You already use several.

RAG is the most common multi-model pipeline on earth. When a chat reads your uploaded PDF and answers from it, that’s retrieval-augmented generation: a retriever pulls the relevant chunk, then the model writes an answer grounded in it. Two components, one pipeline. Retrieval-augmented generation is the reason a model can cite your document instead of hallucinating — and it’s the single most common multi-model pipeline pattern in production.

ChatGPT itself is a stack like this. Ask it to make an image and it calls a separate image model. Ask it to do arithmetic reliably and it runs code in a sandbox. Ask about today’s news and it browses. The chat box is one interface sitting on top of a language model plus an image model plus a code interpreter plus a browsing/retrieval tool. That’s a multi-model pipeline with a router deciding which tool your request needs. OpenAI documents these as built-in tools; the routing is invisible, which is exactly why most people never notice they’re using a compound system.

Your manual GPT-versus-Claude-versus-Gemini habit is the third one — except the router is you. I wrote a whole routing framework for picking which AI to open in an earlier post, before I had the vocabulary for it. That framework is the manual, human-powered version of what the pipeline automates in code.

Once you hold those three examples together, the abstract concept collapses into something concrete: the system is what happens when the routing you do by hand gets written down as logic.

The piece that saves money: model routing and cascades

The most useful leg of the pipeline, for a working professional, is the cost leg.

Not every request needs the smartest, priciest model. A one-line summary doesn’t need frontier reasoning. So a well-built pipeline uses routing (send easy jobs to a cheap model, hard jobs to a strong one) and cascades (try the cheap model first; only escalate if the answer fails a quick check).

Stanford researchers published a paper called FrugalGPT{rel=”noopener”} showing that a cascade — cheap model first, escalate only when needed — could match strong single-model quality at a fraction of the cost on their test tasks. I won’t quote a headline percentage as gospel; the point is the shape of the saving, not a number you’ll reproduce. Route by difficulty, and you stop paying frontier prices for trivial work.

You already do a crude version of this. When you keep a quick reformat in the cheap free tier and save the hard reasoning for your paid Claude Pro tab, that’s a manual cascade. The system version just measures “did this answer pass?” and escalates automatically.

The tools leg matters too. When a model needs to reach outside itself — search your calendar, hit a database, call an API — it uses a tool protocol. I unpacked one open standard for that in how the Model Context Protocol connects models to tools. MCP is, in effect, the plumbing for the “tools” component of these systems: it’s how the model in your pipeline actually reaches your retriever or your files.

And the components are swappable. That’s the quiet superpower. If a cheaper open model gets good enough for the summary step, you swap it in and the rest of the pipeline doesn’t care. I keep a running list of the open-source LLM tools I’d slot into a pipeline exactly because modularity is the point — you upgrade one leg without rebuilding the whole thing.

Compound system vs mixture-of-experts vs AI agent

This is where the vendor pages go muddy, so here’s the plainest version I can draw.

The line that matters is explicit versus implicit workflow — is the path decided in advance, or decided at runtime?

	Compound AI system	Mixture-of-experts (MoE)	AI agent
What it is	Several separate components (models, retriever, tools) wired into a pipeline	One model with expert sub-networks inside it	A model that decides its own next step in a loop
Where the routing lives	In the app’s control flow, between components	Inside the model, invisible to you	In the model’s reasoning, at runtime
Workflow	Explicit — the pipeline path is designed	Explicit — a gating layer picks experts per token	Implicit — the path is chosen live, per task
Everyday example	RAG; ChatGPT with tools	Many frontier models are MoE under the hood	An agent that plans, calls tools, and retries on its own
Who’s in control	The designer (or you, manually)	The model architecture	The model itself

Three quick clarifications, because these get blurred constantly:

Mixture-of-experts is not a multi-model pipeline. MoE is one model that internally routes each token to a subset of expert sub-networks. It’s a way to build a single efficient model — the routing is inside the weights. A compound system routes between separate model calls and components. One is internal architecture; the other is external plumbing.

An AI agent is a compound system with the workflow removed. In a compound pipeline, you (or a developer) decide the path ahead of time: retrieve, then answer. In an agent, the model decides its own path at runtime — plan, call a tool, check, retry. Same Lego pieces, but the agent chooses the order live instead of following a fixed pipeline. I drew that autonomy line in more detail in how to use AI agents at work.

So the spectrum runs from fixed to free: a hard-coded pipeline is the most predictable, an agent is the most autonomous, and a compound system sits in the middle — flexible components, but a workflow you can see and trust.

Where I was wrong (and what it cost me)

Here’s the trial-and-error part, because the mistake shaped everything above.

For months my whole strategy was “own the single smartest model.” I treated the leaderboard like a scoreboard I had to win. Whenever a new top model dropped, I’d second-guess my subscriptions and burn an evening reading benchmark threads that changed nothing about my actual work.

Then I tried to build a small manual pipeline — a two-model workflow to draft, then critique, then rewrite a piece of writing. I ran everything through the strongest, most expensive model for every step, because “best model” was still my instinct. In my testing over a couple of afternoons, the token cost climbed faster than I expected for what was mostly boring reformatting work. The draft step didn’t need a frontier brain. It needed a cheap, fast one, with the expensive model reserved for the critique.

Two things broke. First, cost — I was paying premium rates for trivial steps that a cheap model handled fine. Second, brittleness — when I hand-copied outputs between tabs, I’d lose formatting, drop context, and occasionally paste the wrong version. My “pipeline” was really me being a slow, error-prone router at 7 a.m.

What I changed: I stopped asking “which model is best?” and started asking “which model is sufficient for this step?” The moment I split the job by difficulty — cheap for the draft, strong for the judgment — the cost dropped and the output got more consistent. That’s the entire lesson of compound systems, learned the expensive way. The leverage was never in owning the smartest model. It was in routing.

If I did it again, I’d write the routing rule down before opening a single tab: which step goes to which model, and what triggers an escalation. That one sheet is the difference between a pipeline and a mess.

Where this mental model falls short

I don’t want to oversell the frame. Two honest limits.

First, routing adds its own failure mode. Every time you split a job across components, you add hand-off points — places where context leaks or the wrong step fires. A single strong model is simpler and, for many everyday tasks, plenty. Don’t build a pipeline where one good prompt would do.

Second, “you’re already the router” is a mental model, not a build guide. Recognizing that you route between GPT, Claude, and Gemini won’t automatically make you a systems engineer. It changes how you think about your tools and your costs. Turning it into real automation is a separate, hands-on step — one I’ll actually attempt in the next post rather than hand-wave here.

FAQ

What is a compound AI system in simple terms? It’s an AI app made of several connected parts — multiple models, a search step, and tools — instead of one model answering alone. Logic routes each request to the right piece. RAG and ChatGPT-with-tools are common examples you already use daily.

How is a compound AI system different from a single large language model? A single model answers everything itself from its training. This kind of system surrounds that model with other components — retrieval, tools, a second model — and adds logic to route between them. The system beats any one model call because each step uses the right piece for the job.

Is RAG a compound AI system? Yes. Retrieval-augmented generation combines a retriever (which fetches relevant documents) with a language model (which writes the answer). That’s two components in a pipeline, which is the simplest and most common such setup in production today.

What’s the difference between compound AI systems, mixture-of-experts, and AI agents? Mixture-of-experts is one model routing internally between sub-networks. A compound AI system routes between separate components with a fixed, designed workflow. An AI agent uses the same components but decides its own path at runtime instead of following a set pipeline.

What is model routing, and how does it save money? Model routing sends each request to the cheapest model that can handle it, escalating to a stronger model only when needed. Since most tasks are easy, you stop paying frontier prices for trivial work. Cascades — cheap first, escalate on failure — are the classic cost-saving pattern.

Do I need to be a developer to use a compound AI system? No. You already use several — RAG in document chat, ChatGPT calling image and code tools. Building your own pipeline needs some tooling, but understanding and using these systems takes zero code. You’re already the router; the concept just names the habit.

The reframe: the router was never a tool you install

I spent months looking for the smartest model to buy. The whole time, the important machine was already running — in my own decisions, one tab at a time.

That’s the shift these systems make concrete. The router isn’t software you download or an enterprise stack you assemble. It’s a decision you’re already making by hand, every time you pick GPT for speed, Claude for reasoning, Gemini for the live web. An orchestrated setup just writes that decision down so it stops living in your head at 7 a.m.

So the question isn’t “which model should I own?” It’s “what’s my routing rule, and where does it break?” Answer that on paper first, and the pipeline almost builds itself.

Next in this Framework Deep Dive series: I’ll wire up a real two-model routing pipeline — cheap draft, strong critique — with an actual cost ceiling, and log exactly where it breaks. Less theory, more receipts.

---

seonjae — Korean office worker documenting his transition into AI systems, agents, and vibe coding — without a CS background. Shipping in public.