Short narrated films and interactive labs on how language models actually work: tokens, attention, the KV cache, RAG, LoRA, RLHF and more. Watch the mechanism move, then read the maths behind it.
31 narrated films and 19 interactive labs, by James Phoenix. Each one pairs a mechanism you can watch with the equations behind it and the terms it connects to.
Tokens, embeddings, softmax and attention: the pieces every language model is built from, and what the numbers inside them look like.
FoundationsPress play and watch ReLU rise, get sanded into GELU, become the SwiGLU valve, then see the outputs normalised, dropped out and added to the residual stream. Every slider still works mid-film.
Film 1:273D
Open Activation functions and normalisation
RepresentationsWords become points in a 3D space where nearness means similar meaning. Watch the clusters form, trace king to its nearest neighbours, see directions carry meaning, and find out what any flat picture of an embedding hides.
Film 1:413D
Open Embedding space
FoundationsThe softmax function turns raw logits into probabilities in two moves: exponentiate, then divide by the sum. Watch a machine do it with the real numbers printed at every step, then heat it with temperature and read the loss off a gauge.
Film 1:383D
Open Softmax, temperature and cross-entropy
InterpretabilityPack five feature directions into two coordinates, measure the resulting cross-talk, and run a sparse decoder that reveals both what it can recover and what compression has erased.
Film 1:392D
Open Superposition and sparse decoding
RepresentationsFollow one query down an HNSW tower as it hops from the roof to its ten nearest neighbours with 63 distance computations instead of 500, see what ef buys in 32 dimensions, then watch real embeddings and BM25 each miss a question the other gets right, and what reciprocal rank fusion does with the two.
Film 2:453D
Open Vector search: HNSW, cosine similarity and BM25
RepresentationsAn embedding is the list of numbers a language model uses in place of a word. Watch a token id open a drawer in the embedding table, see similar words share similar numbers and fall into groups in 3D, measure the angle between two words, and do sums with meaning.
Film 2:063D
Open What are embeddings?
FoundationsA language model never reads your words. A tokeniser cuts the text into pieces called tokens and swaps each one for an id number. Watch a machine do it to real sentences, with the real GPT-2 and GPT-4o splits and ids printed on every tile.
Film 1:343D
Open What are tokens?Attention is dot products and a softmax. Drag the query and key vectors with your hands and feel it: alignment wins weight, and vector length is secretly a temperature.
2D
Open Attention as geometryHeads are not redundant copies of each other. Step through twelve fingerprints and watch a previous-token head, an induction head and an attention sink behave in visibly different ways.
2D
Open Attention head fingerprintsType anything and watch a working BPE tokeniser build it from characters upward, one merge at a time. Shrink the vocabulary and see the same text cost more tokens.
2D
Open Byte pair encodingHow a transformer block is put together, and the alternatives: mixture-of-experts, state space models, diffusion, rotary positions.
ArchitecturePress play and watch pure static become a picture in fifty denoising steps: how training noises real pictures, why the cosine schedule beats the linear one, what the U-Net or DiT actually predicts, how a prompt steers sampling through classifier-free guidance, why too much guidance backfires, and why real systems diffuse a compressed latent, with every number computed from an exact toy model.
Film 2:293D
Open Diffusion models: from noise to image
ArchitecturePress play and watch attention’s cache grow by 8 KiB with every token while a Mamba layer keeps one 128 KiB state, then follow the recurrence, the step size Δ, Mamba’s selectivity, the parallel scan and the exact recall a fixed state gives up, all computed from small, labelled toys.
Film 2:472D
Open State space models vs attention: S4, Mamba and hybrids
ArchitecturePull one transformer block apart like a watch movement and follow a token through it: embedding, attention, the residual stream, the MLP, and out to a distribution over the next word. Every part is counted, so you can see where the parameters actually live.
Film 1:273D
Open The transformer block, explodedHow a model can have a trillion parameters and run like a much smaller one. Route tokens to experts, watch the router collapse onto favourites, and see tokens dropped at capacity.
2D
Open Mixture-of-experts routingRoPE encodes position by rotating vectors, and the rotation cancels. Move a query and a key together and watch the attention score refuse to change.
2D
Open Rotary position embeddingsBackpropagation, optimisers, scaling laws, LoRA and RLHF: how a model learns, and what it costs to change one.
TrainingPress play and watch backpropagation run the error back through one neuron by the chain rule, then through twelve layers, where the sigmoid lets it fade to 18 million times weaker and ReLU, careful initialisation and residual connections keep it alive.
Film 2:213D
Open Backpropagation and vanishing gradients
TrainingFollow a frozen weight matrix and two trainable factors through a forward pass, one real training step, a rank limit, an exact parameter budget and a merge.
Film 1:582D
Open LoRA: learn a correction, keep the base
TrainingPrompting changes the instructions, RAG changes the evidence in the context, and fine-tuning changes the weights. One real support request runs through all three lanes, each fix repairs only its own fault, and the lane that learns from past replies also learns their out-of-date refund rule.
Film 2:063D
Open Prompting, RAG and fine-tuning: what changes where?
TrainingWatch one real GRPO step on a small model: eight answers to one maths question, a judge model standing in for the reward model, advantages computed against the group instead of a value model, and the reward hacking that followed within five steps, even with the KL penalty on.
Film 2:363D
Open RLHF to GRPO: reward models and the KL leashSlide a compute budget across the Chinchilla map and watch the optimal split between parameters and tokens follow the ridge, with the tokens-per-parameter ratio sitting in the teens: close to, not exactly, the famous twenty.
2D
Open Scaling lawsDrop a ball anywhere on a loss surface and watch gradient descent live. Feel a learning rate diverge, fall into a decoy minimum, then add momentum and escape it.
2D3D
Open The loss landscapeSGD, momentum and Adam released together on an ill-conditioned valley. Drag the condition number and watch the exact learning rate at which SGD must explode, while Adam barely notices.
2D
Open The optimiser raceWhat happens per token when a model runs: the KV cache, prefill and decode, batching, speculative decoding, quantisation and the GPU underneath.
SystemsA 7B model does not fit on one 80 GB card for training, and a 70B does not fit on eight. Slice the model four different ways across a rack and watch what each strategy costs in memory, traffic and idle time.
Film 1:213D
Open Distributed training: data, tensor and pipeline parallelism
SystemsA GEMM is exactly 2MNK operations whatever runs it. Watch a GPU tile one, then compare three ways of building the hardware for it, CUDA cores, tensor cores and a TPU systolic array, and see why the fast ones are so hard to feed.
Film 1:443D
Open Matrix multiplication on a GPU
InferencePress play and watch one GPU serve Llama 3.1 8B: prefill floods the whole prompt through at once and is limited by arithmetic, while decode drips one token at a time, rereads all 16 GB of weights for each, and is limited by memory bandwidth, with real timings measured on a live endpoint.
Film 2:343D
Open Prefill vs decode: time to first token and tokens per second
InferenceA support bot resends the same 5,897-token prefix on every call. Watch real Claude Sonnet 5 calls read it back from a prompt cache for a tenth of the price, see a question-first prompt reuse nothing, and see one timestamp at the top of the system prompt throw the whole cache away.
Film 1:443D
Open Prompt caching: reuse the stable prefixPlay beam search over a token tree built to trap greedy decoding. The most probable first word is not on the best sentence, and width two is enough to prove it.
2D
Open Beam searchThe same twelve requests scheduled twice: static batches strand GPU slots behind their slowest member, continuous admission refills them next step. Watch the score diverge live.
2D
Open Continuous batchingFlashAttention is not a faster formula, it is a better memory schedule. Walk the tiles, watch the running softmax, and see why never writing the score matrix is worth re-reading the keys.
2D
Open FlashAttentionServing a language model is a memory problem. See why reserving the declared context length for every request wastes most of your accelerator, and what paging the cache recovers.
2D
Open KV cache and pagingWhy int4 sometimes works and sometimes destroys a model. Watch a handful of outlier channels consume the entire numeric range, and watch per-channel scaling give it back.
2D
Open Quantisation and outliersRun a second model and finish sooner. Watch a cheap draft model propose tokens, a large model verify them in a single batched pass, and the speedup curve peak and then fall.
2D
Open Speculative decodingA model does not produce an answer, it produces a distribution over next tokens. Move temperature, top-k and top-p and watch the fan of possible continuations widen or collapse.
2D
Open Token decodingVector search, HNSW, chunking, hybrid retrieval and GraphRAG, and what a long context does to the answer.
ContextPress play and watch a KV-cache warehouse fill an H100: full attention needs 68.7 GB at Llama 3.1 8B’s full context, grouped-query attention cuts it to 17.2 GB, one shared head to a thirty-second, DeepSeek-V3’s latent attention compresses 71 times, and a sliding window caps it at 537 MB, all computed from the models’ published configs.
Film 2:363D
Open Attention variants: MHA, GQA, MQA, MLA and sliding windows
ContextA model keeps nothing between calls, so an application resends the conversation and fills a context window. Follow one real conversation as the window overflows, truncation silently drops the turn that set a naming rule, and retrieved memory and pinned conventions bring it back, with every reply recorded.
Film 1:483D
Open Conversation history versus persistent memory
ContextFollow one question through retrieval augmented generation: every handbook passage scored by real embedding similarity, the top three stacked into the prompt, and a recorded answer checked claim by claim. Then delete the right page, or add a stale one, and see retrieval and generation fail at different stages.
Film 1:283D
Open RAG: from question to evidence to answerPlay a short animation of one sentence split into readable text chunks. The same words separate in 3D, showing why each cut changes the context a retrieval system can see.
3D
Open Chunking VisualisedSee a Bristol launch question travel through a 3D graph into source passages and a cited answer. Adjust graph hops and passage budget, then press Play.
3D
Open GraphRAG VisualisedA model accepting 128k tokens is not the same as a model using them. Sweep a fact through the context and watch retrieval collapse in the middle.
2D
Open Lost in the middleWhat a prompt is made of, why examples work, how prompts overfit, and how caching makes the stable part nearly free.
PromptingOne outage update goes round a revision loop: a writer model drafts, a critic model scores it against a five-point rubric and returns JSON, your code re-checks what it can count, and the writer revises one target at a time. Every text and verdict is a real model output, including the revision that made things worse.
Film 2:102D
Open Draft, critique, revise, check
PromptingEight support tickets, one router, six prompts: no examples, useful examples, repetitive ones, one contradiction, one more example, and the house rules written down. Every landing is a real model answer recorded ten times, so you can see which examples resolve ambiguity and which only cost tokens.
Film 1:273D
Open Few-shot prompting: examples define the pattern
PromptingSend one real photo, a mask, a reference mug and an instruction through OpenAI image models, then diff every result against the original. Under gpt-image-1.5 a tight mask still let 7 to 9 percent of the rest of the photo change. The same edits on gpt-image-2.5 changed under a tenth of a percent, with or without a mask, but a loose mask came back as a solid black block.
Film 1:373D
Open Image editing with masks and references
PromptingGenerate four shots of one barista with gpt-image-1.5 four ways and let a strict judge check every cut. Prompts written one at a time matched 16 of 42 checks; a shared continuity sheet, the approved first frame as a reference and a state line per shot took it to 54 of 54, because the model carries nothing from one shot to the next.
Film 1:173D
Open Multi-shot prompting and visual continuity
PromptingA prompt optimiser keeps whichever edit raises the score on a handful of examples, which quietly turns that score into a training score. Watch a real loop take eight support tickets from 40 to 100 per cent, then lift a curtain on twenty-four tickets it never saw, where a simpler prompt from two steps in does better.
Film 1:233D
Open Prompt optimisation can overfit
PromptingA prompt has six pieces: task, audience, context, constraints, an example and an output format. Watch one real request assembled plate by plate, with the model’s recorded reply to every version, and see which piece changes the answer most.
Film 1:123D
Open The anatomy of a good promptTool calls, MCP, workflows versus agent loops, parallel work, and the runtime an agent actually executes on.
AgentsRun one real refund email through a fixed workflow of four steps and through an agent loop that picks each next tool from what the last one returned. Recorded runs show the loop costing four times the tokens when the steps are known, earning its cost when information is missing, and running until its budget stops it when a tool keeps failing.
Film 1:472D
Open A fixed workflow versus an agent loop
AgentsSplit one research question across four parallel workers and merge their findings with a lead model, in real recorded runs. Parallel workers finish in a quarter of the time, but the merge becomes the slowest step, overlapping questions buy nothing, one slow branch holds everything up, and without a written merge rule a conflict between two sources quietly disappears.
Film 1:493D
Open Parallel work and the synthesis bottleneck
CodingTrace an executable TypeScript example through a running task, a nested microtask checkpoint and a waiting timer, then compare promise continuations with real task yields in two browser engines.
Film 1:292D
Open The event loop: why a 10 ms timer waits 80 ms
AgentsFollow one customer question through a real tool call: the model writes a request and runs nothing, your code looks the order up, and only when the result goes back does the model answer. Four recorded exchanges show the round trip, a lost result, an error, and a refused cancellation.
Film 1:412D
Open Tool calling is a round tripFollow numbered MCP steps from sender to receiver across the Host Application and MCP Server boundaries. Preview the next operation before advancing the 3D diagram.
3D
Open MCP Architecture VisualisedThese explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.
See how I can help