Attention as geometry
Attention is dot products and a softmax. Drag the query and key vectors with your hands and feel it: alignment wins weight, and vector length is secretly a temperature.
2D
Open Attention as geometryInteractive
AI engineering, visualised.
Interactive explainers for how language models actually behave. Concepts that prose struggles with, built as things you can move, rotate and take apart.
Attention head fingerprints
Heads are not redundant copies of each other. Step through twelve fingerprints and watch a previous-token head, an induction head and an attention sink behave in visibly different ways.
2D
Open Attention head fingerprintsBeam search
Play beam search over a token tree built to trap greedy decoding. The most probable first word is not on the best sentence, and width two is enough to prove it.
2D
Open Beam searchByte pair encoding
Type anything and watch a working BPE tokeniser build it from characters upward, one merge at a time. Shrink the vocabulary and see the same text cost more tokens.
2D
Open Byte pair encodingContinuous batching
The same twelve requests scheduled twice: static batches strand GPU slots behind their slowest member, continuous admission refills them next step. Watch the score diverge live.
2D
Open Continuous batchingEmbedding space
Words become points in space, and the distance between them carries meaning. Explore the clusters in 2D, rotate them in 3D, and watch analogy arithmetic land on the word you expect.
2D3D
Open Embedding spaceFlashAttention
FlashAttention is not a faster formula, it is a better memory schedule. Walk the tiles, watch the running softmax, and see why never writing the score matrix is worth re-reading the keys.
2D
Open FlashAttentionKV cache and paging
Serving a language model is a memory problem. See why reserving the declared context length for every request wastes most of your accelerator, and what paging the cache recovers.
2D
Open KV cache and pagingLoRA and low rank
Rebuild a weight matrix from its top r singular directions and watch structure arrive before noise. The rank slider is the entire budget negotiation behind LoRA.
2D
Open LoRA and low rankLost in the middle
A model accepting 128k tokens is not the same as a model using them. Sweep a fact through the context and watch retrieval collapse in the middle.
2D
Open Lost in the middleMixture-of-experts routing
How a model can have a trillion parameters and run like a much smaller one. Route tokens to experts, watch the router collapse onto favourites, and see tokens dropped at capacity.
2D
Open Mixture-of-experts routingQuantisation and outliers
Why int4 sometimes works and sometimes destroys a model. Watch a handful of outlier channels consume the entire numeric range, and watch per-channel scaling give it back.
2D
Open Quantisation and outliersRotary position embeddings
RoPE encodes position by rotating vectors, and the rotation cancels. Move a query and a key together and watch the attention score refuse to change.
2D
Open Rotary position embeddingsScaling laws
Slide a compute budget across the Chinchilla map and watch the optimal split between parameters and tokens follow the ridge, with the tokens-per-parameter ratio sitting in the teens: close to, not exactly, the famous twenty.
2D
Open Scaling lawsSpeculative decoding
Run a second model and finish sooner. Watch a cheap draft model propose tokens, a large model verify them in a single batched pass, and the speedup curve peak and then fall.
2D
Open Speculative decodingThe loss landscape
Drop a ball anywhere on a loss surface and watch gradient descent live. Feel a learning rate diverge, fall into a decoy minimum, then add momentum and escape it.
2D3D
Open The loss landscapeThe optimiser race
SGD, momentum and Adam released together on an ill-conditioned valley. Drag the condition number and watch the exact learning rate at which SGD must explode, while Adam barely notices.
2D
Open The optimiser raceToken decoding
A model does not produce an answer, it produces a distribution over next tokens. Move temperature, top-k and top-p and watch the fan of possible continuations widen or collapse.
2D
Open Token decodingBuilding with language models?
These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.
See how I can help