# Conversation history versus persistent memory **AI Engineering Visualised** · Context · Film, 1:48 · Updated 2026-09-12 A language model keeps nothing between calls. Conversation history is the transcript an application resends each turn; memory is whatever it stores outside the context window and retrieves back in when the window overflows. > A model keeps nothing between calls, so an application resends the conversation and fills a context window. Follow one real conversation as the window overflows, truncation silently drops the turn that set a naming rule, and retrieved memory and pinned conventions bring it back, with every reply recorded. Watch the film: https://understandingdata.com/ai-engineering-visualised/history-versus-memory/ Video file: https://assets.understandingdata.com/viz/history-versus-memory/video/film.mp4?v=1f26fcb60d A language model is stateless. Each API call starts from nothing: the model reads the tokens it is sent, writes a reply, and keeps nothing. A chat that seems to remember is an application that stores the conversation and resends it, so the model's memory is whatever the application decides to put in the context window on the next call. This film makes that choice visible. It opens on the conversation as the API receives it: a list of messages, with the developer's system message first, then the user's message and the assistant's reply. Every call sends that whole list again. The recorded calls send the first role as system; OpenAI's newer models call the same slot developer. The scenario is one real conversation with `google/gemini-3.8-flash` at temperature 0, recorded on 11 September 2026. In turn 1 the user says the API returns every JSON field in snake_case, never camelCase, and the model promises to follow it. Turn 3 adds that money is stored as integer pence. Four unrelated questions follow, about pagination, cursor headers, caching and request ids. Then the user asks for the TypeScript type of an order response, which only comes out right if the model still knows the rules. Each chapter sends that same request with a different window and shows one of five recorded replies, with the tally across all five. On the left of the bench is the transcript shelf: the application's stored conversation history, every turn with its real token count from the o200k_base tokenizer. Next to it is the memory store, a few saved facts from this session and earlier ones, and below it the slot for the current request. The upright column is the context window, 300 tokens to the red limit, and every card in it is as tall as its token count. The model box sees only the column. Each call copies cards into the column from the three compartments; whatever is left behind does not exist for the model. Sent whole, the conversation is 417 tokens and every run uses snake_case. Asked after turn 3, at 192 tokens, the rule is in the window and every run uses it, and four of five name the field total_pence. After turn 7, an application with a 300-token budget does what many chat applications do by default and keeps the newest turns that fit. Turns 1 to 3 drop out without any error or warning, and all five runs switch to camelCase, exactly like a call sent with no history at all. The model did not forget. It was never told. Persistent memory is the usual fix, and the film shows why it is not a guarantee. A memory store is still just text the application selects and resends. Here the application embeds the request, retrieves the two most similar saved facts and puts them in the system prompt. The snake_case fact comes back, and so does snake_case in every run. The pence fact scores 0.06 against a request that never mentions money units, ranks last behind facts about deploy days and the on-call rota, and is not sent, so no run says pence. Pinning the conventions that must always hold, so they ride in every call whatever the similarity, restores both rules to the window: snake_case holds in five of five runs, and the total is named in pence in two. Having the rule in the window makes it available, not obeyed, so check the output too. Pinned text is not free either: it takes the same slice of the budget on every call, which here left room for only turns 5 to 7 of the history. The practical rules follow. Decide what must persist, and pin it into every call rather than hoping retrieval ranks it. Retrieve the rest deliberately, and log what was retrieved and what was left out. Track the token budget per call, because truncation is silent from the model's side. Summarising older turns (compaction) is another option not recorded here: it keeps more history in fewer tokens, but a summary can drop a rule just as quietly as truncation does. The honest caveat is scale. Real context windows run to hundreds of thousands of tokens, so a 300-token budget is shrunk to make the window fill in seven turns; in production the same truncation happens after far longer conversations, or much sooner when an application caps history to save cost and latency. There are five runs per condition, one model and one conversation, and the pence check reads only field names, so a reply that keeps pence in a comment would count as a miss. ## The maths ### The window budget ```latex T_{\text{sys}} + T_{\text{mem}} + \sum_{i \in S} T_i + T_{\text{req}} \;\le\; B, \qquad B = 300 ``` Every call is rebuilt from pieces: the system prompt, any saved facts, the stored turns the application chooses (the set S) and the new request. Each card in the column is as tall as its token count, and the red bar is B. The film shrinks B to 300 so a seven-turn conversation of 417 tokens can overflow it. ### Truncation keeps the newest turns that fit ```latex S \;=\; \{\, n, \dots, 7 \,\}, \qquad n \;=\; \min\Bigl\{ m : T_{\text{fixed}} + \sum_{i=m}^{7} T_i \le B \Bigr\} ``` The common default walks back from the newest turn and stops at the first one that does not fit. With the system prompt and request fixed at 60 tokens, turns 4 to 7 fit in 285 and turns 1 to 3 drop out, including the one that set snake_case. Nothing in the reply says so. ### Retrieval chooses by similarity, not importance ```latex \text{mem} \;=\; \operatorname{top}_k \cos\bigl(e(\text{request}),\, e(f_j)\bigr), \qquad k = 2 ``` Each saved fact f is scored against the request with openai/text-embedding-3-small. The snake_case fact scored 0.19 and was sent; the integer-pence fact scored 0.06, last of five, below facts about deploy days and the on-call rota, so it never reached the window. ### The model conditions only on the window ```latex y \;\sim\; p_\theta(\,\cdot \mid w\,), \qquad w = \text{what this call sends} ``` The reply is a function of the window w and the fixed weights, nothing else. A stored turn that is not in w has exactly the influence of a turn that was never said, which is why the truncated run and the no-history run give the same camelCase reply. ## Transcript ### 0:00 A list of messages Every chat reaches the model as a list of messages: a system message from the developer sets the rules, the user asks, and the assistant replies. The model keeps no memory between calls, so every call sends this whole list, system message first. ### 0:18 Every call starts empty Call one sends just the system message and the first user message, and the model promises snake_case. Then the window empties: the model sees only what the application puts in it. ### 0:30 Resend the history So the application resends the whole conversation on every call, and seven turns in it comes to 417 tokens, against a window we have shrunk to 300 so you can watch it fill. ### 0:44 Still in the window Ask for the order type while the first turn still fits, at 192 tokens, and all five runs answer in snake_case, with the total named in pence in four. ### 0:55 Silent truncation Here is the failure mode: after turn seven the application keeps only the newest turns that fit, the first three drop out silently, and all five runs switch to camelCase. ### 1:08 Retrieved memory Memory is the obvious fix, yet it is only text the application picks to resend: similarity brought snake_case back in all five runs, but ranked the pence fact last, at 0.06, so it never arrived. ### 1:24 Pinned conventions Pinning the conventions that must always hold keeps snake_case in five of five runs, yet only two named the total in pence, so check the output too. ### 1:34 What to keep So pin what must persist, retrieve the rest on purpose, and watch the budget, because truncation is silent: the model never knows what it did not see. ## Related terms - [Stateless](https://understandingdata.com/ai-coding-dictionary/stateless/) - [Stateful](https://understandingdata.com/ai-coding-dictionary/stateful/) - [Context window](https://understandingdata.com/ai-coding-dictionary/context-window/) - [Session](https://understandingdata.com/ai-coding-dictionary/session/) - [Memory system](https://understandingdata.com/ai-coding-dictionary/memory-system/) - [Compaction](https://understandingdata.com/ai-coding-dictionary/compaction/) ## More visualisations - [RAG: from question to evidence to answer](https://understandingdata.com/ai-engineering-visualised/rag-question-to-answer/) - [The anatomy of a good prompt](https://understandingdata.com/ai-engineering-visualised/anatomy-of-a-prompt/) - [What are tokens?](https://understandingdata.com/ai-engineering-visualised/what-are-tokens/) - [Lost in the middle](https://understandingdata.com/ai-engineering-visualised/lost-in-the-middle/) --- Source: https://understandingdata.com/ai-engineering-visualised/history-versus-memory/ From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-12.