Reranking in RAG: Cross-Encoder vs. Bi-Encoder

Reranking is the second retrieval stage in a RAG pipeline: a cross-encoder re-scores the top candidates found by the fast bi-encoder and sorts them by genuine query relevance. According to Anthropic, reranking significantly lowers the retrieval failure rate compared with pure vector retrieval – at the cost of higher latency. Reranking is therefore not an optional nice-to-have, but in production systems the most reliable lever between "semantically similar" and "actually relevant".

• What it does: Reranking re-orders the candidates found by the first retrieval before they reach the LLM prompt.
• Why two stages: The fast bi-encoder provides recall across millions of chunks, while the precise cross-encoder provides precision on the final top-k.
• Concrete effect: Anthropic reports a reduction of the top-20 retrieval failure rate by up to 67 per cent when Contextual Retrieval is combined with reranking.

Why single-stage retrieval is not enough

Classic vector retrieval works with a bi-encoder. It encodes each document chunk into a vector in advance and stores it in an ANN index (usually HNSW). At query time, the query is projected into the same vector space, and similarity is computed via cosine distance. This is extremely fast and scales to millions of entries – but it is an approximation. Query and document never "see" each other jointly; the model only compares two independently produced condensations.

The consequence is a typical anti-pattern: the top-k hits are semantically similar, but not necessarily relevant to the specific question. This is exactly where reranking comes in. It is a post-retrieval stage that applies a more precise – but slower – mechanism to a small set of candidates, rather than to the entire corpus.

Bi-encoder vs. cross-encoder: the central difference

The architectural core of reranking is the switch from the bi-encoder to the cross-encoder.

A bi-encoder processes query and document separately. Both are each encoded into a vector, and relevance only emerges downstream from the distance between these vectors. Because document embeddings can be computed and indexed in advance, the bi-encoder is the only practical choice for the initial retrieval across large corpora.

A cross-encoder processes query and document jointly in a single forward pass. The model sees both texts together and can account for word-by-word interactions, negations, and subtle differences in meaning. The result is a much more precise relevance assessment – however, a computation has to be run for every query-document pair. Across millions of chunks this would be prohibitively expensive; across 50 to 100 pre-filtered candidates it is acceptable.

In between sits the late-interaction approach (ColBERT): it stores token-level embeddings per document and computes relevance via a MaxSim operation between query and document tokens. This offers more accuracy than a classic bi-encoder with better scalability than a full cross-encoder, but costs considerably more memory for the token vectors.

Position in the pipeline and top-k strategy

Reranking is a firmly defined stage in the query path. The canonical arrangement looks like this:

```
[Hybrid Retrieval, top_k = 50-100]
-> [Cross-Encoder Reranker (Cohere Rerank / BGE / Jina)]
-> [top_k = 5-10]
-> [Prompt + source citation]
-> [LLM]
```

The decisive point: the reranker comes only after the cost-effective bi-encoder recall. The initial hybrid retrieval (dense plus BM25, fused via Reciprocal Rank Fusion) deliberately delivers a generous candidate pool of 50 to 100 entries in order to maximise recall. The cross-encoder then reduces this pool to the 5 to 10 truly relevant chunks that reach the prompt.

This narrowing is not just a matter of cost. Too large a top-k passed to the LLM leads to the "lost-in-the-middle" effect: relevant information in the middle of a long context is processed less well, and token costs rise. Reranking plus a tight top-k of 5 to 10 is the standard countermeasure – with the additional recommendation to place the most important chunks at the front of the prompt.

Reranking models at a glance (as of 2026)

For DACH scenarios with sovereignty requirements, the open-source options BGE-Reranker and the Berlin-developed Jina Reranker v2 are relevant, as they can be run on-prem or in EU regions. Anyone who accepts an API and needs multilingual quality including German turns in practice to Cohere Rerank. The LLM-as-judge approach (such as the LLMRanker in Haystack) is the most flexible, but the most expensive per request, and should only be used where reranking quality is critical and the latency budget is generous.

Latency/quality trade-off

Reranking is the most expensive single stage in the online path, because the cross-encoder computes per candidate. In production systems, the total latency from hybrid retrieval plus reranking typically lies at around 100 to 800 milliseconds. This range is directly controllable:

• Size of the input pool: Reranking 100 candidates costs more than 30. Depending on the corpus, a smaller pool is often sufficient.
• Model choice: API rerankers such as Cohere are throughput-optimised; an LLM-as-judge is considerably slower.
• Hardware: Open-source rerankers benefit strongly from GPU inference; on CPU the latency rises considerably.

The trade-off is therefore not "reranking yes or no", but "how many candidates at which latency budget". For most enterprise use cases, the precision gain is worth the additional latency.

Numerical example: precision increase

In the publication "Introducing Contextual Retrieval" (19/09/2024), Anthropic measured concrete figures from its own benchmarks (Code, Fiction, ArXiv papers, Science). To be read as a vendor eval with self-interest, but methodically documented:

• Baseline (pure retrieval): top-20 retrieval failure rate of 5.7 per cent.
• Contextual Embeddings alone: reduction by 35 per cent to 3.7 per cent.
• Contextual Retrieval (embeddings plus BM25): reduction by 49 per cent to 2.9 per cent.
• Contextual Retrieval plus reranking: reduction by 67 per cent to 1.9 per cent.

In concrete terms this means: of the original 57 errors per 1,000 queries, only 19 remain with the full pipeline. Reranking alone – as the step from 2.9 to 1.9 per cent – eliminates around another third of the remaining errors in this setup. For a production RAG system, every avoided retrieval error means one less chunk that tempts the LLM into a wrong or unsubstantiated answer. This is exactly why RAG practice lists "no reranking" as a clear anti-pattern: the top-k are then semantically similar, but not relevant.

For agencies and B2B decision-makers

Anyone setting up a production RAG system in the DACH region should plan reranking as a fixed pipeline stage from the outset – not as downstream tuning. The leverage is large, the implementation effort manageable: an additional API call (Cohere) or a self-hosted open-source reranker (BGE, Jina) between retrieval and LLM. For agencies building knowledge assistants or internal search tools for clients, reranking offers the best ratio of quality gain to effort – and a measurable argument in the pitch, because the precision increase can be demonstrated in evaluation frameworks such as RAGAS (Context Precision). Blck Alpaca designs such two-stage retrieval architectures including EU-compliant model choice and latency budgeting. Have your existing RAG pipeline checked to see whether the right chunks really do arrive at the front.