Quantisation and outliers

Quantisation is usually sold as a straight trade: fewer bits, less memory, slightly worse output. That framing hides the part that decides whether it works, which is what happens when one channel has a far wider range than its neighbours and they are forced to share a scale.

Symmetric quantisation picks a scale from the largest magnitude it has to represent, then divides the range into however many levels the bit width allows. With one scale for the whole tensor, the widest channel sets it. Everything else, which is the overwhelming majority, is left sharing the few levels nearest zero. Drop to int4 with a large outlier present and the entire bulk of the distribution can fall between two representable values, at which point those values are not approximated, they are erased.

That is what the lower panel draws. The red lines are the values the format can actually represent, and the histogram is where the values want to be. Under per-tensor scaling the lines are far apart and the histogram has nowhere to land. Switch to per-channel and the lines close up, because each channel now gets a scale matched to its own range and a narrow channel is no longer punished for sharing a tensor with a wide one. The third setting, mixed precision, does what production kernels do with the worst channels: keep them in a wider format and quantise everything else, which is a different bargain from making the whole tensor share a finer scale.

One clarification, because it is widely garbled. The famous emergent-outlier result is about ACTIVATIONS, not weights: a handful of hidden-state channels develop magnitudes far above the rest, and that is what makes naive int8 inference collapse. Weight distributions are comparatively well behaved, which is precisely why weight-only int4 works at all. The mechanism drawn here, a wide channel starving narrow ones under a shared scale, is the same in both cases, and it is the reason activation-aware methods identify important weights by the size of the activations that multiply them rather than by the size of the weights themselves.

Production schemes diverge from this picture in different directions rather than converging on it. GPTQ compensates for rounding error by adjusting the weights it has not quantised yet. AWQ rescales channels using activation statistics. The int4 files you actually download sit between the two schemes shown here: GGUF k-quants use small groups of 16 to 32 values inside a 256-value super-block, while NF4 uses 64-value blocks and a genuinely non-uniform codebook shaped to a normal distribution. Group-wise scaling is the dominant real practice.

The practical read: a bit width is not a quality setting on its own. Int4 with well-chosen groups can beat int8 with one scale for everything, and the difference is entirely in how the scale is chosen.