Reranking: Cheap Recall, Then Expensive Precision

Summary

Reranking splits retrieval into two stages. A cheap vector search over-fetches a shortlist, then a stronger model reorders it. The first pass optimises for recall, the second for precision, and each is good at a different job. One stage cannot be, which is why single-shot vector search keeps handing you plausible-looking junk.

The Problem

A plain embedding search returns whatever sits nearest your query in vector space. Nearness in that space rewards surface similarity, not answerhood. A document that echoes your question back at you can score higher than the document that actually answers it.

I hit this constantly with support-style corpora. Ask “how do I get my money back after buying something?” and the top vector hit is a sentence that restates the emotion of the query rather than the refund policy. It shares more words with the question, so it wins the cosine contest and loses the user.

The Solution

Retrieve wide, then judge narrow.

1. Stage one (recall). Embed the corpus once, embed the query, take the top N by cosine. This is fast and cheap, so you over-fetch on purpose. You are not trying to be right here, only to not miss the answer.
2. Stage two (precision). Hand each shortlisted candidate to a stronger model that reads the query and the document together and scores relevance. A dedicated cross-encoder does this best; an LLM with a 0-to-1 rubric is the zero-infra version. Reorder by that score.

The shape in the runnable example:

// Stage 1: over-fetch by embedding similarity (fast, high recall)
const shortlist = docs
  .map((text, i) => ({ text, vec: cosineSimilarity(q, embeddings[i]) }))
  .sort((a, b) => b.vec - a.vec)
  .slice(0, 3)

// Stage 2: rerank the shortlist with a model that reads query + doc together
for (const cand of shortlist) {
  const { object } = await generateObject({
    model: openai('gpt-5-mini'),
    schema: z.object({ relevance: z.number().min(0).max(1) }),
    prompt: `Query: ${query}\nDocument: ${cand.text}\nScore 0 to 1.`,
  })
}

When I ran it, the distractor (“we know that buying something and then wanting your money back can be frustrating”) won stage one with the highest cosine score of 0.563, then reranked to 0.00. The real refund policy that placed lower on vectors reranked to 1.00. That inversion is the entire value of the pattern.

Why This Works

A bi-encoder (normal embeddings) has to compress each document into a fixed vector before it ever sees your query. It is guessing what you might ask. A cross-encoder scores the query and document in the same forward pass, so it can attend to the specific relationship between them. It is far more expensive per pair, which is exactly why you only run it on a shortlist of ten or twenty, never the whole index.

When To Reach For It

• Your top-k is “mostly right but the ordering is off,” or the true answer is reliably in the top 20 but not the top 3.
• You can afford tens of milliseconds and a few tokens per candidate. If you cannot, cache aggressively or use a hosted reranker.
• Skip it when the corpus is tiny and a single embedding pass already nails position one. Two stages earn their keep at scale, not on ten documents.

Reranking is the cheapest large accuracy win in a RAG stack, because it fixes ordering without touching your index, your chunks, or your embeddings.

Related

• Hybrid Search – the other half of a serious retrieval stage, fused before you rerank
• Contextual Retrieval – fix what the shortlist can even contain
• Adaptive Query Expansion – generate better queries before retrieval
• GraphRAG for Production Agents – when flat retrieval is the wrong shape entirely
• Information Theory for Coding Agents – signal and entropy in what you retrieve

Part of the field guide

This is one of my field notes in Context Engineering, a plain-English guide to deciding what a model sees. The terms behind it are defined in the Context Engineering Dictionary. Runnable version: 11-reranking.mjs in my AI Engineering Examples repo.