Image editing with masks and references

An image edit sends a model more than a sentence. OpenAI's image edit endpoint takes the photo to change, an optional mask whose transparent pixels mark where change is allowed, optional reference images, and the instruction. The film stacks those four as glass planes on the left, joins them into the editor in the middle, and prints each recorded result onto the easel on the right, where every changed pixel is painted over it: green where the change was allowed, red where it was not.

Everything in the film is real. A source photo of a desk (a white mug, a closed laptop, a notebook and a succulent) and a reference picture of a blue enamel mug were generated first. Then eight edits were recorded on 11 September 2026 with `gpt-image-1.5` at 1536 by 1024 and quality high: two runs each of a tight mask round the mug, a loose mask over a fifth of the frame, no mask at all, and the tight mask with the reference mug as a second image. The instruction asked for the mug to be replaced and for everything else to stay exactly the same. Each result was diffed against the source in code, and every number on screen is a measured share of changed pixels.

Pick a recorded edit and a run while the film is paused, and switch between the result and its changed pixels. The slate reads two shares: how much of the mug area changed, which is the edit you asked for, and how much of the rest of the photo changed, which is drift. The inset above the easel zooms on one object before and after.

The surprise is how little the mask protects. With the tight mask, 7 to 9 percent of the photo outside it changed visibly, and at a faint threshold of 12 out of 255, half to 70 percent of it moved. The succulent came back with different leaves. The model redraws the whole image and the mask steers where it aims rather than fencing anything off. With no mask at all, the rest changed by 4 and 8 percent, no worse than with the tight mask. The reference mug made the new mug match what was wanted and drifted least, at 5 and 6 percent, but still not zero.

The loose mask is the failure mode. Given room, the model grew the mug and shrank the laptop that sat half inside the mask, with 53 percent of the laptop's pixels changed in run 2. Worse, 52 percent of that collateral change fell inside the mask, so a check that only asks whether change stayed inside the mask passes it. The check that catches it compares the result against what you meant to change, not against the permission you gave.

The practical rule: write down what must stay the same, diff every edit against the original, and when the rest has to be exact, paste the masked region back onto the original in code. That composite is the only row on the board with zero drift, and it costs nothing but a few lines after the call. Feather the seam if the lighting shifted, and diff again.

The honest caveats. A pixel diff is strict: a one-pixel shift of an edge counts as a change, so the visible share mixes real redrawing with small misalignment, and a perceptual metric such as SSIM would score these edits more kindly. The threshold of 40 is a choice, shown next to the faint one so you can see how much depends on it. Two runs per condition show the spread but not a distribution, and a different model, size or quality setting would drift differently. The numbers describe this photo and these edits, not image editing in general.