Reranking in RAG: Cross-Encoder vs. Bi-Encoder

Reranking is the second retrieval stage in a RAG pipeline: a cross-encoder re-scores the top candidates found by the fast bi-encoder and sorts them by genuine query relevance. According to Anthropic, reranking significantly lowers the retrieval failure rate compared with pure vector retrieval, at the cost of higher latency. Reranking is therefore not an optional nice-to-have, but in production systems the most reliable lever between "semantically similar" and "actually relevant".

• What it does: Reranking re-orders the candidates found by the first retrieval before they reach the LLM prompt.
• Why two stages: The fast bi-encoder provides recall across millions of chunks, while the precise cross-encoder provides precision on the final top-k.
• Concrete effect: Anthropic reports a reduction of the top-20 retrieval failure rate by up to 67 percent when Contextual Retrieval is combined with reranking.

Why single-stage retrieval is not enough

Classic vector retrieval works with a bi-encoder. It encodes each document chunk into a vector in advance and stores it in an ANN index (usually HNSW). At query time, the query is projected into the same vector space, and similarity is computed via cosine distance. This is extremely fast and scales to millions of entries, but it is an approximation. Query and document never "see" each other jointly; the model only compares two independently produced condensations.

The consequence is a typical anti-pattern: the top-k hits are semantically similar, but not necessarily relevant to the specific question. This is exactly where reranking comes in. It is a post-retrieval stage that applies a more precise, but slower, mechanism to a small set of candidates, rather than to the entire corpus.

Bi-encoder vs. cross-encoder: the central difference

The architectural core of reranking is the switch from the bi-encoder to the cross-encoder.

A bi-encoder processes query and document separately. Both are each encoded into a vector, and relevance only emerges downstream from the distance between these vectors. Because document embeddings can be computed and indexed in advance, the bi-encoder is the only practical choice for the initial retrieval across large corpora.

A cross-encoder processes query and document jointly in a single forward pass. The model sees both texts together and can account for word-by-word interactions, negations, and subtle differences in meaning. The result is a much more precise relevance assessment; however, a computation has to be run for every query-document pair. Across millions of chunks this would be prohibitively expensive; across 50 to 100 pre-filtered candidates it is acceptable.

In between sits the late-interaction approach (ColBERT): it stores token-level embeddings per document and computes relevance via a MaxSim operation between query and document tokens. This offers more accuracy than a classic bi-encoder with better scalability than a full cross-encoder, but costs considerably more memory for the token vectors.

Position in the pipeline and top-k strategy

Reranking is a firmly defined stage in the query path. The canonical arrangement looks like this:

```
[Hybrid Retrieval, top_k = 50-100]
-> [Cross-Encoder Reranker (Cohere Rerank / BGE / Jina)]
-> [top_k = 5-10]
-> [Prompt + source citation]
-> [LLM]
```

The decisive point: the reranker comes only after the cost-effective bi-encoder recall. The initial hybrid retrieval (dense plus BM25, fused via Reciprocal Rank Fusion) deliberately delivers a generous candidate pool of 50 to 100 entries in order to maximize recall. The cross-encoder then reduces this pool to the 5 to 10 truly relevant chunks that reach the prompt.

This narrowing is not just a matter of cost. Too large a top-k passed to the LLM leads to the "lost-in-the-middle" effect: relevant information in the middle of a long context is processed less well, and token costs rise. Reranking plus a tight top-k of 5 to 10 is the standard countermeasure, with the additional recommendation to place the most important chunks at the front of the prompt.

Reranking models at a glance (as of October 2026)

For DACH scenarios with sovereignty requirements, the BGE-Reranker v2-m3 is the obvious on-prem option: it is licensed under Apache 2.0, multilingual, and may be run commercially in your own data center or in an EU region. The older bge-reranker-large (MIT license) covers only Chinese and English according to its model card. Jina Reranker v2, v3, and v3.5 (released on July 27, 2026) have open weights but are licensed under CC BY-NC 4.0. Commercial on-prem use requires a license from Jina AI; otherwise you are limited to the Jina API, the Elastic Inference Service, or the AWS, Azure, and Google Cloud offerings. Jina AI is based in Berlin but has been part of Elastic since October 2025.

Anyone who accepts an API and needs multilingual quality including German turns in practice to Cohere Rerank. The alternative, Voyage, has been part of MongoDB since February 2025; its current models are rerank-3 and rerank-3-lite. According to the launch post of September 30, 2026, Voyage tested rerank-3 on 95 datasets across 9 domains, 51 of them multilingual datasets from 31 languages. The LLM-as-judge approach (such as the LLMRanker in Haystack) is the most flexible, but the most expensive per request, and should only be used where reranking quality is critical and the latency budget is generous.

Latency/quality trade-off

Reranking is the most expensive stage in the retrieval part of the online path (before LLM generation), because the cross-encoder computes per candidate. In production systems, the total latency from hybrid retrieval plus reranking typically lies at around 100 to 800 milliseconds. This range is directly controllable:

• Size of the input pool: Reranking 100 candidates costs more than 30. Depending on the corpus, a smaller pool is often sufficient.
• Model choice: API rerankers such as Cohere are throughput-optimized; an LLM-as-judge is considerably slower.
• Hardware: Self-hosted rerankers benefit strongly from GPU inference; on CPU the latency rises considerably.

The trade-off is therefore not "reranking yes or no", but "how many candidates at which latency budget". For most enterprise use cases, the precision gain is worth the additional latency.

Numerical example: precision increase

In the publication "Introducing Contextual Retrieval" (September 19, 2024), Anthropic measured concrete figures from its own benchmarks (Code, Fiction, ArXiv papers, Science); the reranker used was Cohere's. To be read as a vendor eval with self-interest, but methodically documented:

• Baseline (pure retrieval): top-20 retrieval failure rate of 5.7 percent.
• Contextual Embeddings alone: reduction by 35 percent to 3.7 percent.
• Contextual Retrieval (embeddings plus BM25): reduction by 49 percent to 2.9 percent.
• Contextual Retrieval plus reranking: reduction by 67 percent to 1.9 percent.

In concrete terms: of 1,000 relevant documents, 57 are initially missing from the top 20; with the full pipeline, only 19. Reranking alone, as the step from 2.9 to 1.9 percent, eliminates around another third of the remaining errors in this setup. For a production RAG system, every avoided retrieval error means one less chunk that tempts the LLM into a wrong or unsubstantiated answer. This is exactly why RAG practice lists "no reranking" as a clear anti-pattern: the top-k are then semantically similar, but not relevant.

For agencies and B2B decision-makers

Anyone setting up a production RAG system in the DACH region should plan reranking as a fixed pipeline stage from the outset, not as downstream tuning. The leverage is large, the implementation effort manageable: an additional API call (Cohere) or a self-hosted reranker under an open license such as BGE v2-m3 between retrieval and LLM.

For agencies building knowledge assistants or internal search tools for clients, reranking offers the best ratio of quality gain to effort, and a measurable argument in the pitch, because the precision increase can be demonstrated in evaluation frameworks such as RAGAS (Context Precision). Blck Alpaca designs such two-stage retrieval architectures including EU-compliant model choice and latency budgeting. Have your existing RAG pipeline checked to see whether the right chunks really do arrive at the front.