Training a model is finding a low point on a surface you cannot see. The loss function assigns a number to every possible setting of the parameters, and gradient descent does the only thing available to something standing on a surface in the dark: measure the slope underfoot and take a step downhill.
This landscape is two-dimensional so you can watch, where a real one has billions of dimensions. It is shaped to make the classic behaviours happen on demand. Press anywhere to drop the ball.
The learning rate is the step size, and its failure modes are asymmetric. Too small and the ball crawls, costing you compute. Too large and each step overshoots the narrow valley floor, landing on the far wall, and the ball bounces across the minimum indefinitely instead of settling into it. Here the outer bowl keeps recapturing it; on a real, unbounded loss surface the same overshoot compounds into divergence. Either way the boundary is sharp rather than gradual, which is why learning-rate warmup and schedules exist. Drop the ball between the minima with the rate near its maximum to watch the bounce.
The decoy pit demonstrates the problem momentum solves. Plain gradient descent follows the local slope and nothing else, so a shallow pit on the way to a deep one is a terminal trap: at the bottom the gradient is zero and the update stops. Momentum makes the update a running average of recent gradients instead, so the ball arrives at the decoy carrying speed and rolls straight through, the way a physical ball would.
The honest caveat is dimensionality. In billions of dimensions, true local minima are rarer than this picture suggests; the surface is dominated by saddle points and vast flat regions, and the reason momentum and its descendants like Adam matter there is less about escaping pits and more about making progress along shallow directions while damping oscillation across steep ones. The intuition transfers; the topology does not.
The maths
- The update rule
Eta is the learning-rate slider, beta the momentum slider. At beta = 0 this is plain gradient descent; at 0.9 the velocity is a decaying average of the last ten or so gradients. That accumulated velocity is what can carry the ball through the decoy, though how much you need depends on where you drop it: from the ridge 0.9 is enough, while from directly above the decoy at a small learning rate it takes nearly the top of the slider.
- Why big steps diverge
Near a minimum with curvature lambda, each step multiplies the distance from the optimum by (1 - eta lambda). The moment eta exceeds 2/lambda that factor is below -1, so the error grows and alternates sign. On this landscape the deep pit has the sharpest curvature, which is why the bounce appears there first as you raise the rate.
- What converged means here
The simulation stops when both the slope and the velocity are near zero. The gradient condition alone is not enough with momentum, because the ball can cross a flat spot at speed.
Related terms
- TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.