Conversation history versus persistent memory

A language model is stateless. Each API call starts from nothing: the model reads the tokens it is sent, writes a reply, and keeps nothing. A chat that seems to remember is an application that stores the conversation and resends it, so the model's memory is whatever the application decides to put in the context window on the next call. This film makes that choice visible.

It opens on the conversation as the API receives it: a list of messages, with the developer's system message first, then the user's message and the assistant's reply. Every call sends that whole list again. The recorded calls send the first role as system; OpenAI's newer models call the same slot developer.

The scenario is one real conversation with `google/gemini-3.8-flash` at temperature 0, recorded on 11 September 2026. In turn 1 the user says the API returns every JSON field in snake_case, never camelCase, and the model promises to follow it. Turn 3 adds that money is stored as integer pence. Four unrelated questions follow, about pagination, cursor headers, caching and request ids. Then the user asks for the TypeScript type of an order response, which only comes out right if the model still knows the rules. Each chapter sends that same request with a different window and shows one of five recorded replies, with the tally across all five.

On the left of the bench is the transcript shelf: the application's stored conversation history, every turn with its real token count from the o200k_base tokenizer. Next to it is the memory store, a few saved facts from this session and earlier ones, and below it the slot for the current request. The upright column is the context window, 300 tokens to the red limit, and every card in it is as tall as its token count. The model box sees only the column. Each call copies cards into the column from the three compartments; whatever is left behind does not exist for the model.

Sent whole, the conversation is 417 tokens and every run uses snake_case. Asked after turn 3, at 192 tokens, the rule is in the window and every run uses it, and four of five name the field total_pence. After turn 7, an application with a 300-token budget does what many chat applications do by default and keeps the newest turns that fit. Turns 1 to 3 drop out without any error or warning, and all five runs switch to camelCase, exactly like a call sent with no history at all. The model did not forget. It was never told.

Persistent memory is the usual fix, and the film shows why it is not a guarantee. A memory store is still just text the application selects and resends. Here the application embeds the request, retrieves the two most similar saved facts and puts them in the system prompt. The snake_case fact comes back, and so does snake_case in every run. The pence fact scores 0.06 against a request that never mentions money units, ranks last behind facts about deploy days and the on-call rota, and is not sent, so no run says pence. Pinning the conventions that must always hold, so they ride in every call whatever the similarity, restores both rules to the window: snake_case holds in five of five runs, and the total is named in pence in two. Having the rule in the window makes it available, not obeyed, so check the output too. Pinned text is not free either: it takes the same slice of the budget on every call, which here left room for only turns 5 to 7 of the history.

The practical rules follow. Decide what must persist, and pin it into every call rather than hoping retrieval ranks it. Retrieve the rest deliberately, and log what was retrieved and what was left out. Track the token budget per call, because truncation is silent from the model's side. Summarising older turns (compaction) is another option not recorded here: it keeps more history in fewer tokens, but a summary can drop a rule just as quietly as truncation does.

The honest caveat is scale. Real context windows run to hundreds of thousands of tokens, so a 300-token budget is shrunk to make the window fill in seven turns; in production the same truncation happens after far longer conversations, or much sooner when an application caps history to save cost and latency. There are five runs per condition, one model and one conversation, and the pence check reads only field names, so a reply that keeps pence in a comment would count as a miss.