What is a diffusion model? It is a model that makes a picture by removing noise. Nobody can write down a formula for "a picture of a ring", but it is easy to learn a smaller skill: look at a noisy picture and say what the noise is. A diffusion model learns that one skill, then applies it again and again, starting from pure static, until a picture is left. Stable Diffusion, Imagen and the DALL-E models work this way, and OpenAI describes Sora as a diffusion transformer.
Training runs the process forwards. Take a real picture, pick a step between 1 and 1,000, and mix in Gaussian noise by a fixed recipe, the schedule. Under the linear schedule from the original DDPM paper, step 500 keeps only 28 percent of the signal, with the noise at 0.96. The schedule matters more than it looks: the linear one is over 90 percent noise from step 674 onwards, so a third of the training steps are spent on pictures that are nearly all static. The cosine schedule of Nichol and Dhariwal keeps 70 percent of the signal at step 500 and only drowns at step 936, which spends the model's effort where there is something to learn.
The network's job is a regression. Given the noisy picture, the step number and the prompt, predict the noise that was added, and pay the squared error. Two architectures dominate. The U-Net is an hourglass: it shrinks the picture through convolutions, reasons at low resolution, and grows it back, with skip connections carrying detail across; DDPM and Stable Diffusion 1 and 2 use one. The DiT cuts the picture into patches and runs them through a stack of transformer blocks, the same block that powers language models; Stable Diffusion 3 and Sora use transformers. Either way, the function being learned is the same.
Sampling runs the prediction backwards. The film starts from static 20.5 units from any real picture and takes fifty DDIM steps, not a thousand: at each one it predicts the noise, reads off the clean picture that implies, and steps to a lower noise level. It ends 0.84 from the nearest real picture, inside their own spread (a real picture sits about 1.28 from its variant's centre). In the glass cube the same process runs on 600 points in 3D, and they settle into a ring and a spiral.
A prompt steers this through classifier-free guidance: the denoiser is run twice, with and without the prompt, and the sampler steps along the difference, multiplied by the guidance scale. With no prompt, 17 of 40 samples come out as rings; asked for a ring at scale 1, all 40 do. Image tools default to about 7.5, and here that is already too much for the prediction: at 7.5 the guided prediction asks for pixels 6.3 times brighter than white. At 60 the samples crowd together, 18 of 40 on a single ring, two of the six ring variants never appear, and the spread between samples falls by 19 percent. In a trained model the same extrapolation shows up as burnt, over-saturated colour, which is why Imagen introduced dynamic thresholding to clip the predicted picture at each step.
Real systems do not diffuse pixels. A 512 by 512 colour picture is 786,432 numbers, so latent diffusion first compresses it with an autoencoder (Stable Diffusion's keeps 16,384, a factor of 48), runs the whole denoising loop on the compressed code, and decodes once at the end. The film does the same with a linear autoencoder: 20 numbers keep 99 percent of the variation in the 256-pixel pictures, the sampler runs in those 20 dimensions, and the decoder inflates the result back to a ring.
The honest caveats. The pictures are a synthetic toy: 16 by 16 pixels, four shapes in six variants each, so small that the ideal denoiser has a closed form. The film uses that exact denoiser, not a trained network, so it shows what a U-Net or DiT is trying to compute rather than how well one does; real models make errors this one cannot. Because the exact denoiser always pulls the final picture back onto a real one, over-guidance here shows as lost variety and an overshooting prediction rather than as the colour burn you see in real models. The autoencoder is linear, where real ones are deep networks. No real image model was called; every number on screen is computed in your browser by the same code the tests check.
The maths
- The forward process
Any step of the noising has a closed form: a share of the clean picture plus a share of fresh Gaussian noise. The linear schedule raises β from 0.0001 to 0.02 over 1,000 steps; the cosine schedule sets ᾱ from a squared cosine. Drag the step slider and the formula panel shows the two shares for one pixel.
- One DDIM step
Predict the noise, read off the clean picture it implies, then rebuild the picture at a lower noise level with the same noise. Fifty such steps of 20 cover the 1,000-step schedule; the film uses deterministic DDIM, so the same seed always gives the same picture.
- Classifier-free guidance
The denoiser runs twice, once with the prompt and once without, and steps along the difference. s = 0 ignores the prompt, s = 1 follows it, and above 1 it extrapolates past it. Image tools expose s (7.5 is a common default); the paper writes the same thing with w.
- The ideal denoiser
The toy's pictures are a known mixture of 24 narrow Gaussians (four shapes, six variants each), so the best possible denoiser has this closed form: weigh each variant by how likely it is to have produced x_t, and pull its mean slightly towards x_t. A trained U-Net or DiT is an approximation to exactly this function.
- The latent
A linear autoencoder (principal component analysis) keeps the 20 directions with the most variance out of 256. Diffusion runs on the 20 numbers and the decoder maps the result back to pixels. Stable Diffusion's learned autoencoder compresses 786,432 numbers to 16,384, a factor of 48.
Sources and model assumptions
Follow the original mechanism behind this explainer. The interactive examples identify their toy data and simplifying assumptions above.
- Ho, Jain and Abbeel (2020): Denoising Diffusion Probabilistic Models
- Song, Meng and Ermon (2021): Denoising Diffusion Implicit Models
- Nichol and Dhariwal (2021): Improved Denoising Diffusion Probabilistic Models
- Ho and Salimans (2022): Classifier-Free Diffusion Guidance
- Rombach et al. (2022): High-Resolution Image Synthesis with Latent Diffusion Models
- Peebles and Xie (2023): Scalable Diffusion Models with Transformers
- Saharia et al. (2022): Imagen, dynamic thresholding
Related terms
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
- TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.