Advanced prompting — tokens, cost, and reliability

Motivation

Writing a prompt that works once in a notebook is not engineering. Engineering is writing a prompt that works reliably at scale — consistently, cheaply, and without crashing when the API returns an error or gets temporarily overloaded.

This lesson covers the four things that separate hobby code from production AI systems: understanding token costs so you can design efficiently, controlling randomness with temperature, handling rate limits without crashing, and knowing the mental model that tells you how much AI a problem actually needs.

Tokens and cost — what you are actually paying for

Every token that passes through the API costs money. There are two types:

• Input tokens — all the text you send (system message + full conversation history + user’s message). Billed at a lower rate.
• Output tokens — all the text the model generates. Billed at a higher rate (2–4× more expensive than input).

At those prices, a single API call costs a fraction of a cent. But at scale — a system processing 100,000 documents per day — the numbers matter. Here is how to think about it:

def estimate_cost(prompt_tokens: int, completion_tokens: int, model: str = "gpt-4o-mini") -> float:
    rates = {
        "gpt-4o-mini": (0.15, 0.60),
        "gpt-4o":      (2.50, 10.00),
    }
    input_rate, output_rate = rates[model]
    cost = (prompt_tokens / 1_000_000 * input_rate) + (completion_tokens / 1_000_000 * output_rate)
    return cost

# After a call:
cost = estimate_cost(response.usage.prompt_tokens, response.usage.completion_tokens)
print(f"This call cost: ${cost:.6f}")

How to reduce token cost

• Trim history — every old message in the conversation costs input tokens. Keep only what the model needs.
• Concise system prompts — a system prompt is resent on every call. A 500-token system prompt on 10,000 calls = 5 million tokens billed.
• Right-size the model — use gpt-4o-mini for everything that does not require deep reasoning. Reserve gpt-4o for tasks that actually need it.
• Limit output length — use the max_tokens parameter to cap output at what you actually need.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    max_tokens=200,    # stop generating after 200 tokens
)

Temperature — controlling randomness

When an LLM generates each token, it does not always pick the single most-likely word. It samples from a probability distribution — sometimes choosing a less likely word to introduce variety. The temperature parameter controls how much sampling randomness there is.

• Temperature 0 — always pick the most likely token. Fully deterministic output. Same prompt always gives the same (or very similar) answer.
• Temperature 0.7 — the default. A balanced amount of variation. Good for most tasks.
• Temperature 1.0+ — high randomness. Creative, unpredictable. Useful for brainstorming; dangerous for factual extraction.

For production systems where consistency matters, default to a low temperature and increase it only if outputs are too repetitive.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    temperature=0.2,   # low: consistent, less creative
)

Rate limits and retry logic

Every OpenAI account has rate limits: a maximum number of requests per minute and a maximum number of tokens per minute. Exceed them and the API returns a RateLimitError (HTTP 429). This is not a bug — it is a normal operating condition for any production system.

The right response to a rate limit error is to wait and retry — with a delay that grows each time the error repeats. This is called exponential backoff: try again after 1 second, then 2 seconds, then 4 seconds, then 8, up to a maximum wait.

import time
from openai import OpenAI, RateLimitError, OpenAIError

def call_with_retry(client: OpenAI, messages: list, model: str = "gpt-4o-mini",
                    max_retries: int = 4) -> str:
    delay = 1.0
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
            )
            return response.choices[0].message.content

        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            print(f"Rate limit hit. Waiting {delay:.0f}s...")
            time.sleep(delay)
            delay *= 2   # double the wait each time

        except OpenAIError:
            raise   # don't retry on other errors

This handles the vast majority of transient rate limit errors transparently. In Chapter 5 you will learn how to instrument these retries with proper observability.

Streaming responses

By default, the API waits until the model has finished generating the entire response, then returns it all at once. For long responses — a 500-word essay, a detailed explanation — this can mean waiting 5–10 seconds before anything appears.

Streaming sends each token back to your code as it is generated, so you can display text incrementally — exactly like ChatGPT does. Set stream=True on the call:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain how photosynthesis works."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

print()  # newline at the end

Each chunk contains a small piece of the response. The delta.content field holds the new text (or None for control chunks). flush=True forces each piece to print immediately rather than buffering.

Use streaming for any interface where the user is waiting for a response — chat UIs, CLI tools, document generators. Skip it for background processing pipelines where the user is not watching.

The Staircase of Complexity

The most important mental model in AI engineering is not a prompting technique — it is a question: how much AI does this problem actually need?

Adding AI adds cost, latency, and unpredictability. A regex that takes 0.001 seconds is always better than an LLM call that takes 2 seconds, when the regex is good enough. The goal is to use as little AI as possible to solve the problem reliably.

The Staircase of Complexity is a framework for finding that minimum. Each step is more powerful than the one above it — and more expensive. Always start at the top and climb only as high as the problem forces you:

You will encounter all of these in this curriculum, in roughly this order. The Staircase tells you where to start (Step 1) and when to climb. Most production problems are solved at steps 1–4. Steps 5–8 exist for specific, justified cases.