James Phoenix
Prompting

Image editing with masks and references

A mask tells an image model which pixels it may change and a reference image tells it what to put there. How faithfully a model honours the mask varies by model, and the only way to know is to diff the result against the original.

Send one real photo, a mask, a reference mug and an instruction through OpenAI image models, then diff every result against the original. Under gpt-image-1.5 a tight mask still let 7 to 9 percent of the rest of the photo change. The same edits on gpt-image-2.5 changed under a tenth of a percent, with or without a mask, but a loose mask came back as a solid black block.

By Film published 1:37 watch

Recorded edit
Model
Run
Show
Loading 3D view

Four planesAn image edit stacks four planes, the photo, a mask where change is allowed, a reference and the instruction, to swap one white mug and nothing else.

Narrated with James Phoenix's AI voice.

Tight maskrun 1
Edit
76.6%
Mug area changed
9.3%
Rest of photo changed
Succulent40.2%
Most changed object
gpt-image-1.52026-09-11
Recorded with

Tight mask, 3.5% of the frame: 76.6% of the mug area changed, and so did 9.3% of the photo outside it, most in the succulent (40.2% of its pixels).

An image edit sends a model more than a sentence. OpenAI's image edit endpoint takes the photo to change, an optional mask whose transparent pixels mark where change is allowed, optional reference images, and the instruction. The film stacks those four as glass planes on the left, joins them into the editor in the middle, and prints each recorded result onto the easel on the right, where every changed pixel is painted over it: green where the change was allowed, red where it was not.

Everything in the film is real. A source photo of a desk (a white mug, a closed laptop, a notebook and a succulent) and a reference picture of a blue enamel mug were generated first. Then eight edits were recorded on 11 September 2026 with gpt-image-1.5 at 1536 by 1024 and quality high: two runs each of a tight mask round the mug, a loose mask over a fifth of the frame, no mask at all, and the tight mask with the reference mug as a second image. The instruction asked for the mug to be replaced and for everything else to stay exactly the same. Each result was diffed against the source in code, and every number on screen is a measured share of changed pixels.

The same day, the identical eight edits (same photo, masks, prompts, size and quality, two runs each) went through four more OpenAI models: chatgpt-image-latest, gpt-image-2, and the two variants of gpt-image-2.5 that OpenAI's model list dates to 4 September 2026, flare and sunburst. Google's gemini-3.1-flash-image and gemini-3-pro-image ran the edits that need no mask, since Gemini's API takes none. All of it was measured by the same code.

Pick a model, a recorded edit and a run while the film is paused, and switch between the result and its changed pixels. The slate reads two shares: how much of the mug area changed, which is the edit you asked for, and how much of the rest of the photo changed, which is drift. The inset above the easel zooms on one object before and after.

On gpt-image-1.5 the surprise is how little the mask protects. With the tight mask, 7 to 9 percent of the photo outside it changed visibly, and at a faint threshold of 12 out of 255, half to 70 percent of it moved. The succulent came back with different leaves. The model redraws the whole image and the mask steers where it aims rather than fencing anything off. With no mask at all, the rest changed by 4 and 8 percent, no worse than with the tight mask. The reference mug made the new mug match what was wanted and drifted least, at 5 and 6 percent, but still not zero.

The loose mask is its failure mode. Given room, gpt-image-1.5 grew the mug and shrank the laptop that sat half inside the mask, with 53 percent of the laptop's pixels changed in run 2. Worse, 52 percent of that collateral change fell inside the mask, so a check that only asks whether change stayed inside the mask passes it. The check that catches it compares the result against what you meant to change, not against the permission you gave.

The newer models change the picture. gpt-image-2.5-flare changed 0.04 and 0.07 percent of the photo outside the tight mask, against 9.3 and 6.6 percent on gpt-image-1.5, and at the faint threshold 3.3 and 4.4 percent moved rather than 70.0 and 49.8. With no mask at all it changed 0.01 and 0.02 percent, so the model now does the protecting, not the mask. gpt-image-2.5-sunburst and gpt-image-2 behaved the same way, at 0.38 percent at most in any run without the loose mask. They still redraw every pixel, only within a few levels of the original: in the tight-mask runs at most 1.9 percent of the pixels outside the mug came back bit-identical, so this is not the server pasting the untouched pixels back. gpt-image-2.5 was also quicker and lighter, at 17.9 to 43.2 seconds and 1,372 output tokens a request, against 40.4 to 47.4 seconds and 6,544 to 6,633 tokens on gpt-image-1.5, while gpt-image-2 took 77.9 to 87.2 seconds. chatgpt-image-latest, which the model list dates to December 2025, behaved like gpt-image-1.5, at 7.0 and 7.1 percent outside the tight mask.

The loose mask is where the newer models fail, and they fail differently. In all six loose runs on gpt-image-2 and gpt-image-2.5, the model handed the masked region back solid black, with a mean colour under 1 out of 255, and set the new mug down elsewhere on the desk, outside the mask. One extra run on gpt-image-2.5-flare, with the mask's colour channels carrying the photo instead of black, did the same, so it is the transparent region itself that comes back empty, not the mask's colour. A check against the mask would pass the black block, since it sits inside the mask, and only a look at the result or a diff against the source shows the edit went wrong.

Gemini's models could only run the prompt-only edits. With no mask, or with the reference and no mask, gemini-3.1-flash-image and gemini-3-pro-image changed 0.35 to 0.61 percent of the rest, close to the newer OpenAI models. Gemini returns a 2528 by 1696 JPEG, which was scaled to 1536 by 1024 before diffing. The succulent reads 6.5 to 6.7 percent changed in every Gemini run with no visible difference in its leaves, which looks like re-encoded texture rather than redrawing, so read those numbers as an upper bound.

The practical rule survives the newer models: write down what must stay the same, diff every edit against the original, and when the rest has to be exact, paste the masked region back onto the original in code. That composite is the only row on the board at zero by construction, and it costs nothing but a few lines after the call. Feather the seam if the lighting shifted, and diff again. On gpt-image-2 and 2.5, also keep the mask tight or leave it off: a mask much bigger than the thing being changed invites the black block.

The honest caveats. A pixel diff is strict: a one-pixel shift of an edge counts as a change, so the visible share mixes real redrawing with small misalignment, and a perceptual metric such as SSIM would score these edits more kindly. The threshold of 40 is a choice, shown next to the faint one so you can see how much depends on it. Two runs per condition show the spread but not a distribution, and a different photo, size or quality setting would drift differently. OpenAI's model list does not say how flare and sunburst differ, and they behaved alike here, and aliases such as chatgpt-image-latest can point at a different model tomorrow. The numbers describe this photo and these edits, not image editing in general.

The maths

What counts as a changed pixel
c(x)  =  [ max⁡k∈{r,g,b}∣ Rˉk(x)−Sˉk(x) ∣  ≥  τ ],τ=40c(x) \;=\; \Big[\, \max_{k \in \{r,g,b\}} \big|\, \bar R_k(x) - \bar S_k(x) \,\big| \;\ge\; \tau \,\Big], \qquad \tau = 40

S is the source photo and R the edited result, both 1536 by 1024, and the bars mean each channel difference was averaged over a 3 by 3 box so single-pixel noise does not count. A pixel is changed when any colour channel moved 40 or more out of 255. Choosing Changed pixels under Show paints c(x) over the result.

Where the change fell
mug=1∣M∣∑x∈Mc(x),rest=1∣Mˉ∣∑x∉Mc(x)\text{mug} = \frac{1}{|M|}\sum_{x \in M} c(x), \qquad \text{rest} = \frac{1}{|\bar M|}\sum_{x \notin M} c(x)

M is the mug area, the tight mask’s editable region (3.5 percent of the frame). The slate in front of the easel reads both shares. For the tight-mask edit in the film, mug is 76.6 percent and rest is 9.3 percent on gpt-image-1.5, and 82.5 and 0.04 percent on gpt-image-2.5-flare. The rest is the drift: change nobody asked for.

Why a mask check is not an intent check
ppass  =  ∣{x∉M,  x∈K,  c(x)}∣∣{x∉M,  c(x)}∣p_{\text{pass}} \;=\; \frac{\big|\{x \notin M,\; x \in K,\; c(x)\}\big|}{\big|\{x \notin M,\; c(x)\}\big|}

K is the mask actually sent. When K is the mug area, nothing outside the mug can hide inside it. When K is the loose mask over 22 percent of the frame, 51.7 percent of the collateral change in run 2 falls inside K, so a check against the mask scores it as intended. Pick Loose mask to see it.

Pasting the masked region back
R′(x)  =  K(x) R(x)  +  (1−K(x)) S(x)R'(x) \;=\; K(x)\,R(x) \;+\; \big(1 - K(x)\big)\,S(x)

Keep the model’s pixels only where the mask allows, and the original everywhere else. Outside K, R′ equals S exactly, so the rest-of-photo share is 0.0 percent by construction. Pick Tight + paste back to see the result and its changed pixels.

Transcript

The narration, chapter by chapter. A timestamp opens the film at that moment.

0:00Four planes
An image edit stacks four planes, the photo, a mask where change is allowed, a reference and the instruction, to swap one white mug and nothing else.
0:13One real edit
Trace one real edit: with a tight mask around the mug, OpenAI's gpt-image-1.5 worked for 43 seconds and handed back a blue enamel mug.
0:24Diff it
Diff it against the original: 77 percent of the mug area changed, but so did 9 percent outside the mask, because the model repaints everything.
0:34Loose mask
Widen the mask to 22 percent of the frame, and the mug grows, the laptop shrinks, and a mask check passes half the damage.
0:43No mask
With no mask at all, the rest changed 4 and 8 percent, no more than the tight mask's 7 and 9: a mask steers but locks nothing.
0:54Reference
A reference pins down the new mug, speckles and rim included, and it drifted least, at 5 and 6 percent, but not zero.
1:03Newer model
On OpenAI's newest model, gpt-image-2.5, under a tenth of a percent of the rest changed with the tight mask, and as little with no mask.
1:13Loose, again
Widen the mask, and the region comes back solid black, the mug set down outside it, in all six loose runs on gpt-image-2 and 2.5.
1:24What to keep
So diff every edit, on any model, and when the rest must be exact, paste the masked region back: the only row at zero by construction.

Also available as captions (WebVTT), the video file and a Markdown copy of this page.

Sources and model assumptions

Follow the original mechanism behind this explainer. The interactive examples identify their toy data and simplifying assumptions above.

Related terms

More visualisations

Embed this film

Paste this into a post, a course page or a newsletter. The film plays in place, with the same controls as here, and credits the source.

Building with language models?

These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.

See how I can help