Conversations and chat history

Motivation

Every real AI product — a customer support bot, a coding assistant, a document Q&A tool — needs to remember what was said earlier in the conversation. If a user says “summarise this contract” and then asks “who are the parties involved?”, the model must know what “this” and “the parties” refer to. Without that memory, every message is a fresh conversation with an amnesiac.

This lesson explains how chat history actually works under the API — and it is simpler than most developers expect. There is no magic. You are the memory system.

The API is stateless — it has no memory of its own

The single most important thing to understand about the OpenAI API — and every other LLM API — is that it is stateless.

Stateless means that every API call is completely independent. When you call client.chat.completions.create(), the model has no knowledge of any previous calls you made. It does not matter if you called it one second ago or one year ago — from the model’s perspective, each call is the first and only time it has heard from you.

This surprises developers who assume the API works like ChatGPT, where you can refer back to earlier messages. ChatGPT does remember — but ChatGPT is an application built on top of the same API. The application is responsible for storing the conversation history and sending it along with each new request. That is what you are about to do.

The model only knows what you put in the messages list. If you want it to know what was said three turns ago, you must include those three turns in the list.

The messages list is the full conversation

In Lesson 2, the messages list had a single entry: the user’s question. A multi-turn conversation simply has more entries. Each entry is one message in the conversation thread, in chronological order.

There are three possible role values:

• "system" — background instructions for the model. Comes first in the list. Sets tone, persona, rules, and constraints. The user never sees this.
• "user" — a message from the human in the conversation.
• "assistant" — a message from the model. When you include previous model replies in the list, this is the role you use for them.

Here is what a two-turn conversation looks like in code:

messages = [
    {"role": "system",    "content": "You are a helpful assistant."},
    {"role": "user",      "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
    {"role": "user",      "content": "What is its population?"},
]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages
)

print(response.choices[0].message.content)
# → "The population of Paris is approximately 2.1 million in the city..."

The model sees the entire list and understands that “its” in the final message refers to Paris — because Paris is in the conversation history you provided.

The system message

The "system" role message is your lever for controlling the model’s behaviour without the user knowing. You can tell the model what persona to adopt, what domain to focus on, what format to use, and what rules to follow:

{"role": "system", "content": "You are a senior Python engineer. Answer only Python-related questions. When you show code, use type hints. Be concise."}

System messages are not a security boundary — a determined user can often override them with clever prompting — but they are effective at steering normal use. They belong at the very start of the messages list, before any user messages.

Building a conversation loop

To make a stateful chatbot, you maintain a list in memory and append to it after every turn: first with the user’s message, then with the model’s reply. On the next turn, you send the whole list again.

Here is the complete pattern — a REPL (Read–Eval–Print Loop) that holds a conversation until the user types “quit”:

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def chat():
    # The conversation history. System message goes first.
    messages = [
        {"role": "system", "content": "You are a helpful assistant. Be concise."}
    ]

    print("Chat started. Type 'quit' to exit.\n")

    while True:
        user_input = input("You: ").strip()
        if user_input.lower() == "quit":
            break
        if not user_input:
            continue

        # 1. Append the user's message to history.
        messages.append({"role": "user", "content": user_input})

        # 2. Send the full history to the API.
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages
        )

        reply = response.choices[0].message.content

        # 3. Append the model's reply to history.
        messages.append({"role": "assistant", "content": reply})

        print(f"\nAssistant: {reply}\n")

if __name__ == "__main__":
    chat()

The key insight is in the three numbered comments. The history list grows by two entries every turn — one user message and one assistant reply. By the fifth turn, you are sending ten messages to the API. The model reads all of them and replies with full context.

The context window limit

The model cannot read an infinitely long conversation. Every LLM has a context window — the maximum number of tokens it can process at once, counting both the full messages list you send and the reply it generates. For gpt-4o-mini, this is 128,000 tokens — roughly a full-length novel.

For most applications, this is not a practical limit in the short term. A typical back-and-forth conversation of 20 turns uses fewer than 5,000 tokens. But long documents, automated pipelines that run for hours, or conversations where users paste large amounts of text can hit the limit.

When the context window is exceeded, the API returns an error. The production solution is to trim the history — keeping the system message and the most recent N turns, discarding older messages. A simple trim function:

def trim_history(messages: list, max_turns: int = 10) -> list:
    """Keep the system message + the most recent max_turns pairs."""
    system = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]
    # Each turn = 1 user + 1 assistant message = 2 entries
    trimmed = conversation[-(max_turns * 2):]
    return system + trimmed

Call this before each API request: messages = trim_history(messages). This keeps memory usage predictable regardless of conversation length.

Where history actually lives

In the REPL example above, the history lives in a Python list — in memory, for the duration of the script. When the script exits, it is gone.

In a real product, you need to persist conversation history across sessions. The approach depends on the product:

In Chapter 3 you will build a backend with PostgreSQL to store conversations properly. For now, the in-memory list is the right tool.