30–40 min read · Conceptual
Before writing a single line of Chapter 2 code, you need a map. “Use an LLM” is not an architecture — it is a starting point. Production AI engineering means choosing the right system shape for the problem, not grabbing the most impressive-sounding tool.
This lesson surveys the five major LLM system types, gives you one concrete real-world example of each, and reintroduces the Staircase of Complexity as the chapter-wide design lens. Every lesson that follows builds one of these system types. Knowing the map before the details makes the details stick.
What it is: Raw, unstructured documents come in — PDFs, emails, HTML pages, scanned images. The pipeline extracts, classifies, enriches, and outputs structured data.
Real example: A law firm receives thousands of contracts per month. The pipeline reads each one, extracts the parties, key dates, jurisdiction, and governing law clause, classifies the risk level, and writes the structured fields to a database. Lawyers review the high-risk ones; the rest are handled automatically.
Characteristics: typically batch (not real-time), deterministic in structure (same fields every time), heavily parallelisable (each document is independent), and structured-output-heavy. Chapter 2 Project P004 builds this type.
What it is: A chatbot or assistant that maintains a conversation over time, understands context, and helps a user accomplish a goal through dialogue.
Real example: Notion AI’s Q&A feature. A user asks “what was decided in the last product review?” — the assistant retrieves relevant pages from the user’s workspace (RAG, covered in Chapter 4) and answers with citations. Follow-up questions refer to the previous answer.
Characteristics: real-time (low latency matters), stateful (conversation history must be managed), often needs tool access (search, calendar, database queries), and must handle off-topic or ambiguous input gracefully. You built the foundation of this type in Chapter 1.
What it is: Systems that produce new content — text, summaries, translations, ad copy, code — at scale, often with human review in the loop.
Real example: An e-commerce company uses an LLM pipeline to draft product descriptions from structured data (SKU, category, dimensions, materials). A human editor reviews the top 5% most complex products; the rest publish automatically. The pipeline runs nightly and processes 10,000 items.
Characteristics: output quality varies — evaluation is essential; often uses evaluator-optimizer patterns to self-correct (Lesson 14); high throughput with parallelism; temperature matters more here than in extraction tasks.
What it is: An LLM embedded in a backend service as one step in a larger workflow — classifying, routing, enriching, or triaging — without the user ever knowing an LLM is involved.
Real example: A SaaS support platform uses an LLM to classify incoming tickets (billing / technical / account / other), set a priority (low / medium / high / urgent), and extract customer account ID from the body — all before a human sees the ticket. The LLM is one node in a larger orchestration pipeline alongside database lookups and routing rules.
Characteristics: latency budget is tight; must not hallucinate structured fields; structured outputs are non-negotiable; cost per call compounds quickly at volume; error rates must be measured and monitored (Chapter 5).
What it is: Multiple LLM agents working in concert — each specialised for a sub-task — with one coordinating agent orchestrating the overall workflow and routing work between specialists.
Real example: An automated research assistant (this chapter’s Chapter Project, P006). A user asks a business question. A coordinator agent decides whether the answer needs web search, database queries, or financial analysis. It dispatches to specialist agents, collects their outputs, feeds them to an evaluator agent that checks quality and requests clarification if needed, then assembles the final report.
Characteristics: most complex to build and debug; failures compound across agents; requires careful state management, observability (Chapter 5), and stopping conditions. Only use this type when a single agent provably cannot handle the task. The Staircase applies at the system level: reach for multi-agent last.
In Lesson 6 you learned the Staircase as a prompting heuristic: try zero-shot before few-shot, few-shot before chain-of-thought, and so on. In Chapter 2, the same ladder applies at the system level — across the entire architecture, not just a single prompt.
When a new requirement arrives, work up the Staircase from the bottom:
Each step up adds development time, debugging complexity, latency, and cost. The systems that fail in production are usually the ones that jumped to step 5 when step 2 was sufficient.
| If the problem looks like… | Start with… |
|---|---|
| Structured extraction from documents | Document pipeline (type 1) — single LLM call with structured output |
| User dialogue, ongoing task | Conversational assistant (type 2) — chat history + optional tool calling |
| Bulk content at scale | Content generation (type 3) — parallelised pipeline with evaluator |
| Background enrichment / triage | Backend automation (type 4) — LLM as one node, not the whole system |
| Open-ended research or complex multi-step task | Multi-agent (type 5) — only after proving simpler types cannot work |
.txt or .md fileAnswer each question from memory, then click to reveal.
A document pipeline is primarily about reading and transforming documents — it processes files and outputs structured data. Backend automation embeds LLM intelligence inside a running service — the LLM is one node among many (database lookups, routing rules, queues), and the end user often doesn’t know an LLM is involved.
→ System type 4: Backend automationMultiple agents are step 6 — the last resort. Each step up the Staircase adds development time, debugging complexity, latency, and cost. Failures compound across agents. You use multi-agent only when a single agent provably cannot handle the task — usually due to scale, specialisation requirements, or context window limits.
→ The Staircase as a design toolExtraction tasks have a single correct answer — the invoice number is what it is. Temperature should be near zero to ensure consistency. Content generation benefits from variation — creative writing, ad copy, and product descriptions are better when they do not all sound identical. Higher temperature produces the variety that makes bulk-generated content feel less mechanical.
→ System type 3: Content generation