Systems

FlashAttention

FlashAttention is not a faster formula, it is a better memory schedule. Walk the tiles, watch the running softmax, and see why never writing the score matrix is worth re-reading the keys.

Sequence
Block Br 128
Block Bc 128
Run
MEMORY HIERARCHYHBMlarge, ~2 TB/sQ, K, V, O524 KBS (1024x1024)1.0 MB never written by FlashSRAM197 KB, ~20 TB/sQ block 128x64K,V blocks 128x64S tile 128x128SCORE MATRIX, WALKED IN 36 OF 64 TILESkeys / valuesqueriescausally maskedONLINE SOFTMAXrow 64running max m4.527running sum l16.792l across key blocksexact, not approximate
4.7 MB
Standard HBM traffic
1.4 MB
Flash HBM traffic
3.3x
Less memory moved
1.0 MB
Scratch avoided
99 KBfits
SRAM per tile

Query block 1 of 8, key block 1 of 8. The running max and sum carry across the sweep, so the softmax is exact without the 1024 x 1024 matrix ever existing.

Standard attention has a step that nobody designs on purpose. It computes an N by N matrix of scores, writes it to high-bandwidth memory, reads it back to apply softmax, writes the result, and reads it again to multiply by V. At a sequence length of 4096 that matrix is 32 MB per head in bf16, and it crosses the memory bus four times. The arithmetic is trivial. The traffic is not.

FlashAttention removes the matrix. It loads a block of queries and a block of keys and values into on-chip SRAM, computes just that tile of scores there, folds it into a running result, and moves on. The tile never leaves the chip. What used to be a term proportional to N squared becomes a term proportional to N squared times the head dimension divided by the block size, and the constant is large enough to matter more than the asymptotics suggest.

The part that makes this legal is the running softmax. An ordinary softmax needs the maximum of the whole row before it can exponentiate anything, which is precisely what forces you to have the whole row. Instead keep a running maximum and a running sum, and when a later block contains a larger value, rescale what you have accumulated by the exponential of the difference. The result is not an approximation of the full softmax. It is equal to it.

The trade is that keys and values get re-read once per query block. That is a real cost, and it is why block size is a tuning parameter rather than a constant: bigger blocks mean fewer re-reads, until the working set stops fitting in SRAM and the kernel spills, at which point you have lost everything you came for. Push the block sliders past the budget to see that boundary.

One honest limit of the picture: the head dimension here is fixed at 64 and the block sizes are chosen for legibility rather than for a real GPU's shared memory, so the traffic ratio is faithful while the absolute numbers are illustrative.

The maths

Online softmax update
m(k)=max ⁣(m(k1),maxjsj),(k)=em(k1)m(k)(k1)+jesjm(k)m^{(k)} = \max\!\big(m^{(k-1)},\, \max_j s_j\big), \qquad \ell^{(k)} = e^{\,m^{(k-1)} - m^{(k)}}\,\ell^{(k-1)} + \sum_j e^{\,s_j - m^{(k)}}

The running maximum and running sum after block k. When a later block contains a larger score, the accumulated sum is rescaled by the exponential of the difference. This is what makes tiling legal, and the result equals the full-row softmax exactly rather than approximately.

Standard attention HBM traffic
bytes  =  (4Nd+4N22)b\text{bytes} \;=\; \big(4Nd + 4\tfrac{N^2}{2}\big)\,b

Q, K, V and O account for the 4Nd term. The rest is the score matrix crossing the bus four times: written, read for softmax, written again, read for the value product. The causal mask halves it. This term dominates the moment N exceeds the head dimension.

FlashAttention HBM traffic
bytes  =  (2Nd+2d ⁣ ⁣i=1Tr ⁣min((i+1)Br,N))b    N2dBrb\text{bytes} \;=\; \Big(2Nd + 2d\!\!\sum_{i=1}^{T_r}\!\min\big((i{+}1)B_r,\, N\big)\Big) b \;\approx\; \frac{N^2 d}{B_r}\,b

No N by N term. Q and O are touched once; K and V are re-read once per query block, which is the price of never writing the score matrix. Note this is still quadratic in N: the win is the constant, roughly Br divided by d, not the asymptotics.

SRAM working set per tile
M  =  (2Brd  +  2Bcd  +  BrBc  +  2Br)bM \;=\; \big(2 B_r d \;+\; 2 B_c d \;+\; B_r B_c \;+\; 2 B_r\big)\, b

The Q block, the output accumulator, the K and V blocks, the score tile and the running statistics. The output accumulator must stay resident across the whole inner sweep, which is exactly why the query block sits on the outer loop. Exceed the budget and the kernel spills.

Related terms

More visualisations

Building with language models?

These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.

See how I can help