There are two phases in a model's life. Training is the one-off, enormously expensive process of learning the parameters. Inference is what happens every time you use the finished model: you pass in some text and it computes the next tokens. Training happens once; inference happens on every single request.
What actually happens on a request
At inference time the model does one thing repeatedly: predict the next token, append it, and predict again. It generates the response one token at a time, each new token conditioned on everything before it. That is why responses stream in, and why a long answer takes longer and costs more than a short one.
Why the distinction is useful
Keeping training and inference separate clears up a lot of confusion:
- The model does not learn from your conversation. Chatting with it is inference, not training. Nothing you say updates its parameters, which is why it starts every new session blank.
- Cost and latency live at inference. When you think about speed or spend for a coding agent, you are thinking about inference, because that is the part that runs on every request.
- Sampling makes it non-deterministic. Inference usually picks among likely next tokens with a bit of randomness, which is why the same prompt can give two different answers.
Related terms
Concept
Model
A model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
Read definition →ConceptTraining
Training is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
Read definition →ConceptToken
A token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.
Read definition →Explore it visually
- Attention as geometryAttention is dot products and a softmax. Drag the query and key vectors with your hands and feel it: alignment wins weight, and vector length is secretly a temperature.
- Attention head fingerprintsHeads are not redundant copies of each other. Step through twelve fingerprints and watch a previous-token head, an induction head and an attention sink behave in visibly different ways.
- Attention variants: MHA, GQA, MQA, MLA and sliding windowsPress play and watch a KV-cache warehouse fill an H100: full attention needs 68.7 GB at Llama 3.1 8B’s full context, grouped-query attention cuts it to 17.2 GB, one shared head to a thirty-second, DeepSeek-V3’s latent attention compresses 71 times, and a sliding window caps it at 537 MB, all computed from the models’ published configs.
- Beam searchPlay beam search over a token tree built to trap greedy decoding. The most probable first word is not on the best sentence, and width two is enough to prove it.
- Continuous batchingThe same twelve requests scheduled twice: static batches strand GPU slots behind their slowest member, continuous admission refills them next step. Watch the score diverge live.
- Diffusion models: from noise to imagePress play and watch pure static become a picture in fifty denoising steps: how training noises real pictures, why the cosine schedule beats the linear one, what the U-Net or DiT actually predicts, how a prompt steers sampling through classifier-free guidance, why too much guidance backfires, and why real systems diffuse a compressed latent, with every number computed from an exact toy model.
- Embedding spaceWords become points in a 3D space where nearness means similar meaning. Watch the clusters form, trace king to its nearest neighbours, see directions carry meaning, and find out what any flat picture of an embedding hides.
- FlashAttentionFlashAttention is not a faster formula, it is a better memory schedule. Walk the tiles, watch the running softmax, and see why never writing the score matrix is worth re-reading the keys.
- Image editing with masks and referencesSend one real photo, a mask, a reference mug and an instruction through OpenAI image models, then diff every result against the original. Under gpt-image-1.5 a tight mask still let 7 to 9 percent of the rest of the photo change. The same edits on gpt-image-2.5 changed under a tenth of a percent, with or without a mask, but a loose mask came back as a solid black block.
- KV cache and pagingServing a language model is a memory problem. See why reserving the declared context length for every request wastes most of your accelerator, and what paging the cache recovers.
- Matrix multiplication on a GPUA GEMM is exactly 2MNK operations whatever runs it. Watch a GPU tile one, then compare three ways of building the hardware for it, CUDA cores, tensor cores and a TPU systolic array, and see why the fast ones are so hard to feed.
- Mixture-of-experts routingHow a model can have a trillion parameters and run like a much smaller one. Route tokens to experts, watch the router collapse onto favourites, and see tokens dropped at capacity.
- Prefill vs decode: time to first token and tokens per secondPress play and watch one GPU serve Llama 3.1 8B: prefill floods the whole prompt through at once and is limited by arithmetic, while decode drips one token at a time, rereads all 16 GB of weights for each, and is limited by memory bandwidth, with real timings measured on a live endpoint.
- Prompt Caching VisualisedChange a prompt and trace exactly which cached blocks survive. Explore shared prefixes, block size, expiry and cache isolation in an interactive 2D and 3D prefix tree.
- Prompt caching: reuse the stable prefixA support bot resends the same 5,897-token prefix on every call. Watch real Claude Sonnet 5 calls read it back from a prompt cache for a tenth of the price, see a question-first prompt reuse nothing, and see one timestamp at the top of the system prompt throw the whole cache away.
- Quantisation and outliersWhy int4 sometimes works and sometimes destroys a model. Watch a handful of outlier channels consume the entire numeric range, and watch per-channel scaling give it back.
- Softmax, temperature and cross-entropyThe softmax function turns raw logits into probabilities in two moves: exponentiate, then divide by the sum. Watch a machine do it with the real numbers printed at every step, then heat it with temperature and read the loss off a gauge.
- Speculative decodingRun a second model and finish sooner. Watch a cheap draft model propose tokens, a large model verify them in a single batched pass, and the speedup curve peak and then fall.
- State space models vs attention: S4, Mamba and hybridsPress play and watch attention’s cache grow by 8 KiB with every token while a Mamba layer keeps one 128 KiB state, then follow the recurrence, the step size Δ, Mamba’s selectivity, the parallel scan and the exact recall a fixed state gives up, all computed from small, labelled toys.
- The transformer block, explodedPull one transformer block apart like a watch movement and follow a token through it: embedding, attention, the residual stream, the MLP, and out to a distribution over the next word. Every part is counted, so you can see where the parameters actually live.
- Token decodingA model does not produce an answer, it produces a distribution over next tokens. Move temperature, top-k and top-p and watch the fan of possible continuations widen or collapse.