Superposition and sparse decoding

In this toy, a feature is represented by a direction in an activation space, and it need not be a single coordinate of that space. Start with two perpendicular feature directions in two coordinates: encoding and projecting back recovers the input exactly. The film then keeps two coordinates while arranging five unit directions around a regular pentagon, so the representation has more feature directions than coordinates.

The input activates only A. Its stored state is [1, 0], but reading that state with the five dot products returns approximately [1, 0.31, -0.81, -0.81, 0.31]. The inactive neighbours B and E respond because they overlap with A. Clipping negative values does not remove those two false positive responses. Every value in the film is computed from the displayed dictionary.

Sparsity changes how often features need the same space simultaneously. It does not make the fixed feature directions more perpendicular. Under the film's independent activation assumption, reducing a feature's activation probability from 0.5 to 0.1 reduces a pair's co-activation probability from 0.25 to 0.01. The displayed expected pair overlap energy is the sum of squared off-diagonal dot products multiplied by that pair probability, not a measured language-model loss.

The decoder solves a real nonnegative optimisation problem for the observed hidden state. Each iteration updates its coefficients to reduce reconstruction error plus a penalty on total activity. With only A active and the default penalty of 0.05, the solution keeps A near 0.95 and suppresses its neighbours. After the film, choose A, A plus B, or all five, change the penalty, and scrub the decoder's iterations to inspect the actual computed states.

The all-five case exposes a limit that a reconstruction metric can hide. Five equal pentagon directions sum to zero, just like no active features. The decoder can reconstruct that zero hidden state perfectly by returning no activity, yet its feature mean squared error against the all-active input is one. A sparse explanation depends on assumptions about the inputs; a good reconstruction alone does not establish that its labels are the true underlying features.

This is an illustrative fixed dictionary, not a trained language model or a claim that real features form a perfect pentagon. The optimisation shown is an iterative sparse decoder; a sparse autoencoder instead learns an encoder and decoder from activations, and evaluates reconstruction, sparsity and feature behaviour. Feature correlations, unequal importance, learned biases and real model behaviour add complications deliberately excluded here. Grokking and the logit lens are separate questions, not consequences demonstrated by this toy.