Almost everything a language model does is this operation. Attention scores, the value mix, every feed-forward layer, the final projection to the vocabulary: matrix multiplication, or GEMM (general matrix multiply) in the language of the libraries that run it. When people say a model needs a GPU, this is the thing they mean it needs a GPU for. It is worth knowing exactly what the GPU does with it.
The work itself cannot be reduced. Each element of the output is a dot product of a row of A with a column of B, which is K multiplies and K adds, and there are M x N of them. Two times M times N times K, full stop. A 4096-wide square GEMM is 137 billion operations on any hardware ever built. What differs between a slow implementation and a fast one is not the arithmetic. It is how many bytes have to move for each operation, and how many operations can happen at once.
Start with bytes, because that is where naive code dies. If every thread computes one output element and fetches its own row of A and column of B from the GPU's main memory (HBM, the stacked chips beside the die), then each element costs 2K reads for 2K operations: one operation per element read, a fraction of a FLOP per byte. HBM on an H100 moves 3.35 terabytes a second, which sounds enormous and is nowhere near enough. The cores would spend nearly all their time waiting. Set the tile control to 16 and watch what happens instead: a block of threads claims a 16 x 16 tile of C, loads a 16-row slab of A and a 16-column slab of B into shared memory (the fast, small, on-chip scratchpad every core in the block can read), and every loaded element is reused 16 times before it is thrown away. Traffic falls by the tile side. At 128 it is 128 times less than naive. That single idea, tile and reuse, is most of what a CUDA kernel or a Triton kernel is.
The arithmetic intensity stat is the honest score: operations per byte moved. Every compute unit has a ridge, the intensity at which its arithmetic and its memory bus take the same time, and it is simply peak FLOP/s divided by bandwidth. For the fp32 CUDA cores on an H100 the ridge is about 20 FLOP per byte, and a 64-wide tile clears it comfortably. Switch to tensor cores and the ridge jumps to nearly 300. A tensor core takes a whole 4 x 4 x 4 block of multiply-adds in one instruction, sixty-four operations where a CUDA core does one, and the die has hundreds of them. They are so fast that no single level of tiling from HBM can keep them fed. Watch the stat turn red as you switch: the kernel that was compute-bound on CUDA cores is memory-bound on tensor cores with the same tiles.
That is the real engineering problem of a modern GEMM, and it is why the memory hierarchy on the right has four levels rather than two. Real kernels climb it. Registers hold the accumulators for a thread's own outputs. Shared memory holds the current tiles. The 50 MB L2 cache catches the row-block of A that the next tile over is about to ask for, so most of the "HBM traffic" this page counts is actually served from L2 at several times the bandwidth. Kernels also use rectangular tiles like 256 x 128, split K across blocks, and on Hopper multicast one tile to a cluster of blocks. Every one of those tricks is a way to raise intensity past the ridge.
The systolic array is the other answer to the same question, and it is how a TPU does it. Pick that unit and step through the ticks. There is no shared memory and no scheduling at all: a fixed grid of multiply-add cells, A entering from the left and B from the top, each value passed one cell along per tick. Cell (i, j) accumulates C[i][j] and never touches memory mid-computation. The whole product arrives after K + 2(n - 1) ticks. It is rigid, which is its weakness (a shape that does not fit the grid wastes cells) and its strength (no cache misses, no address arithmetic, nearly every transistor doing arithmetic). The 4 x 4 here is a toy. The array in a TPU is 128 x 128 or larger.
Two things the picture simplifies. The traffic counter measures bytes into the streaming multiprocessors with no L2 in the way, which is the pessimistic bound; the L2 reuse described above is why real tensor-core kernels do reach their peak on large shapes. And the walk visits output tiles one at a time, where a real GPU has 132 multiprocessors each working a different tile at once, with several blocks per multiprocessor so one can compute while another waits on a load. The bytes and the FLOPs are exact. The order and the overlap are the simplification.
The maths
- The work is fixed
Every output element is a dot product of length K: K multiplies and K adds. The Work counter is this number and nothing on the page changes it. Tiling, tensor cores and systolic arrays only change what each operation costs in bytes and clocks.
- Bytes moved with square tiles
Each t x t output tile reads a t x K row-block of A and a K x t column-block of B once, then writes itself. With t = 1 this is the naive kernel that re-reads a whole row and column per output. The tile control is t; the "moved from HBM" counter is this expression with ragged edges charged in full.
- Arithmetic intensity and the ridge
Intensity is how much work each byte buys, and it grows with the tile side. The ridge is the intensity at which the compute unit and the memory bus take the same time. Below it the unit waits on memory; above it memory waits on the unit. The stat shows both, and the colour flips when they cross.
- Systolic array latency
Cell (i, j) meets a[i][k] and b[k][j] on tick i + j + k. The far corner sees its last pair on tick K - 1 + 2(n - 1), so the whole product is ready one tick later. The Tick counter in the systolic view runs to exactly this.
Transcript
The narration, chapter by chapter. A timestamp opens the film at that moment.
- 0:00Three set-ups
- Almost every layer of a model is a matrix multiplication whose arithmetic is fixed, so what really separates CUDA cores, tensor cores, and a TPU systolic array is how many bytes each one has to move for every multiply-add.
- 0:15The problem
- Take a 1024 by 1024 product, about a billion multiply-adds: every tile of the floor C needs one row-block of A and one column-block of B, so everything turns on how often those blocks cross the bus.
- 0:31CUDA cores
- Here is the failure mode, plain CUDA cores with no tiling: each output re-reads its row, and its column, from HBM, so eight megabytes of input become eight gigabytes of traffic, and the cores sit idle.
- 0:47Tiling
- Tiling is the fix: each block is loaded into shared memory once and reused thirty-two times, traffic falls to about 270 megabytes, and the same cores climb to around forty percent busy.
- 1:01Tensor cores
- Now, the part most explainers miss: a tensor core does sixty-four multiply-adds per instruction, yet it needs nearly three hundred flops per byte, so the same tiling leaves it starved, idle about ninety-five percent of the time.
- 1:18Systolic array
- A TPU systolic array solves this in hardware: values flow from neighbour to neighbour, so one read from memory feeds a whole row, while the sums build along a diagonal wave.
- 1:31Roofline
- So, check the roofline before you buy compute: below the ridge, bigger tiles and more reuse are what speed you up, and only past it do tensor cores pay off.
Also available as captions (WebVTT), the video file and a Markdown copy of this page.
Related terms
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
More visualisations
Embed this film
Paste this into a post, a course page or a newsletter. The film plays in place, with the same controls as here, and credits the source.