Few-shot prompting: examples define the pattern

The maths

Where a ticket lands

$$\hat{q}(t) \;=\; \arg\max_{q} \sum_{r=1}^{10} \big[\, a_r(t) = q \,\big]$$

Each ticket was sent ten times under each prompt, and a card lands in the queue most of its ten answers chose. The badge on the card, such as 3/10, counts how many of the ten matched the house rules, so a card can land in the right bin and still be shaky.

The two scores on the board

$$\text{tickets} = \sum_{t=1}^{8} \big[\, \hat{q}(t) = q^{*}(t) \,\big], \qquad \text{runs} = \sum_{t=1}^{8} \sum_{r=1}^{10} \big[\, a_r(t) = q^{*}(t) \,\big]$$

The first counts tickets whose majority answer is right, out of 8. The second counts every individual answer, out of 80, so it shows the wobble the majority hides: the repetitive, contradictory and one-more-example prompts all score six of eight by majority, but 60, 59 and 65 of 80 runs.

What examples cost

$$\text{tokens per call} \;=\; |\,\text{instruction}\,| + \sum_{i=1}^{k} |\,\text{example}_i\,| + |\,\text{ticket}\,|$$

Every example rides along with every ticket. The bare prompt cost 44 input tokens a ticket; three near-identical refund examples cost 90, and the rules plus three examples cost 171. A repetitive example is paid for on every call and buys little coverage.

Temperature zero is not a fixed answer

$$T = 0 \;\not\Rightarrow\; a_1(t) = a_2(t) = \dots = a_{10}(t)$$

The recordings used temperature 0, yet several tickets changed answer between runs, such as the seats ticket, which went to account in four runs of ten with no examples. Batching and floating point on the provider side make repeated calls differ, which is why every cell was recorded ten times rather than once.

Transcript

The narration, chapter by chapter. A timestamp opens the film at that moment.

0:00House rules

Routing a support ticket looks easy, but the right queue often hangs on house rules, like single sign-on being a plan feature that belongs to the account team, and no model can guess those.

0:12No examples

With no examples, the model gets five of the eight tickets right, sends single sign-on to feature requests in all ten runs, and flips the seats ticket between billing and account.

0:24Useful examples

Three examples picked from the edges of the rules lift it to seven of eight, and the SAML example beside it fixes single sign-on in all ten runs.

0:34Repetitive examples

Three near-identical refund examples add little: they steady the seats ticket, but single sign-on and the audit log stay wrong in every run, at twice the tokens of no examples.

0:47One contradiction

Here is the failure mode: add one example that calls a double charge after a crash a bug, and that ticket goes to bug in nine runs of ten, even with the three good examples still there.

0:59One more example

The part people miss is that examples teach neighbours, not rules, so an extra example about deleted projects fixes the audit log in only three runs of ten, and knocks the seats ticket down to two.

1:14Write the rule down

So write the rule down and let examples show its edges: the house rules plus the same three examples route all eight tickets correctly in every one of eighty runs.

Also available as captions (WebVTT), the video file and a Markdown copy of this page.

Related terms

• System promptThe system prompt is the standing instruction placed at the very start of the context that sets the model’s role, rules, and tone before the conversation begins. It shapes every reply without being part of the back-and-forth.
• ContextContext is all the text a model can see for a single request: the system prompt, your message, the conversation so far, and any files or tool output the agent has pulled in. It is the only thing the model knows about your specific situation.
• Input tokensInput tokens are the tokens you send in a request: the system prompt, the conversation history, loaded files, and tool definitions. You are billed for them, and they count against the context window.
• Non-determinismNon-determinism is why the same prompt can give you different answers. At inference the model samples among likely next tokens with a controlled amount of randomness, so runs vary.