How to Build a Reranker for Search and RAG With Open Models (2026 Guide)

Quick answer: Text ranking (reranking) takes a query and a list of candidate passages and scores each pair for relevance, so the best passages move to the top. The shortest path: retrieve 50-100 candidates with BM25 or an embedding model, then rerank them with a cross-encoder such as Qwen3-Reranker, bge-reranker-v2-m3 or mxbai-rerank-v2, loaded through sentence-transformers or FlagEmbedding.

Almost every search box and retrieval-augmented generation (RAG) system has the same weak spot: the first-stage retriever is fast but approximate. It finds roughly the right documents, but the order is often wrong, and the passage that actually answers the question sits at position 14. A reranker fixes that by reading the query and each candidate together and scoring them precisely.

Teams add a reranker to internal knowledge-base search, support bots, e-commerce search, code search and RAG pipelines where only the top 3-5 chunks fit into the LLM prompt. This guide covers what the task is, which open models to use, working code, deployment and evaluation.

What is text ranking?

The Hugging Face text ranking task page defines it as ranking a set of texts by their relevance to a query. The input is one query plus N candidate texts; the output is one relevance score per candidate, which you sort to get a new order. It is normally the second stage of a search stack: a retriever pulls a broad candidate set (for example 100 documents) and the ranking model picks the new top set.

There are two common model designs:

• Cross-encoders. An encoder (BERT, XLM-RoBERTa, ModernBERT) reads [query, passage] as one sequence and a classification head outputs a single score. The cross-encoder/ms-marco family, the BGE rerankers and the Ettin rerankers work this way.
• LLM-based rerankers. A decoder LLM is prompted with the query, the document and an instruction, and the score comes from the probability of answering "yes" versus "no". The Qwen3-Reranker models use this pointwise approach, which is why they accept a task instruction.

Unlike embedding models, a reranker cannot precompute document vectors: every query-passage pair needs a forward pass. That is why rerankers are accurate but too slow to run over a whole corpus, and why you only apply them to a shortlist.

What can you build with a reranker?

• Better RAG answers. Retrieve 50 chunks, rerank, and send only the top 5 to the LLM. Fewer irrelevant chunks means fewer wrong or padded answers.
• Internal knowledge-base and help-centre search on top of an existing Elasticsearch, OpenSearch or Solr index, without re-indexing.
• Product and catalogue search where lexical matches are common but the intent behind the query matters.
• Multilingual search: a multilingual reranker can score a German query against English passages.
• Question routing and FAQ matching: rank a fixed list of answers or support macros for an incoming ticket.
• Search across meeting transcripts, where the same topic is discussed in many meetings and you need the passage that actually contains the decision.

Which models should you use?

These open models cover the main trade-offs between size, languages and licence. Check each model card before production use, because licences and recommended settings change.

• cross-encoder/ms-marco-MiniLM-L6-v2: about 22.7M parameters, English, trained on MS MARCO passage ranking, Apache 2.0. The classic fast baseline that runs comfortably on CPU.
• Ettin rerankers (cross-encoder/ettin-reranker-17m-v1 up to 1b-v1): six English CrossEncoder rerankers built on ModernBERT-style Ettin encoders, Apache 2.0, distilled from mxbai-rerank-large-v2. The Ettin reranker release post publishes the data and training recipe.
• BAAI/bge-reranker-v2-m3: multilingual cross-encoder built on bge-m3 (about 568M parameters), Apache 2.0. A solid default when queries and documents are not only English.
• mixedbread-ai/mxbai-rerank-base-v2 and mxbai-rerank-large-v2: Apache 2.0 rerankers (the large model is about 1.5B parameters) with multilingual, code and long-context support. Mixedbread reports a BEIR average of 57.49 for the large model.
• Qwen/Qwen3-Reranker-0.6B: LLM-based reranker with a 32K context and instruction support. A good balance of quality and cost when you can run a small GPU.
• Qwen/Qwen3-Reranker-4B and Qwen/Qwen3-Reranker-8B: the larger members of the same family, for when accuracy matters more than latency. The Qwen3-Embedding GitHub repo lists 69.76 MTEB-R for the 4B model and 69.02 for the 8B, scored on the top-100 candidates from Qwen3-Embedding-0.6B. The design is described in the Qwen3 Embedding paper.

To compare models, use the reranking and retrieval tabs of the MTEB leaderboard, and browse the community reranker leaderboard thread on the Hugging Face forum, which compares rerankers on nDCG, recall and latency. Leaderboard numbers are a starting point; your own data decides.

How do you build it, step by step?

1. Chunk and index your documents. Split documents into passages of a few hundred tokens and index them with BM25, an embedding model, or both (hybrid search). See our guide to semantic search with embeddings for this stage.
2. Retrieve candidates. For each query, fetch the top 50-100 passages. Recall at this stage caps the final quality: a reranker cannot promote a passage that was never retrieved.
3. Score query-passage pairs. Pass every (query, passage) pair to the reranker in batches. Truncate long passages to the model's maximum length.
4. Sort and cut. Sort by score, keep the top k (3-10 for RAG), and optionally drop anything below a score threshold you calibrated on your own data.
5. Use the results. Show them in the search UI, or insert them into the LLM prompt with their source IDs so answers can cite them.
6. Evaluate and tune. Measure nDCG@10 or MRR on a labelled query set, then tune candidate count, k and model size.

The quickest start is the CrossEncoder class in sentence-transformers. This snippet follows the README example for cross-encoder/ms-marco-MiniLM-L6-v2:

from sentence_transformers import CrossEncoder

# Follows the sentence-transformers README (CrossEncoder.predict / rank)
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

query = "How do I reset my VPN password?"
passages = [
    "To reset your VPN password, open the self-service portal and choose Reset.",
    "The VPN client supports Windows, macOS and Linux.",
    "Passwords for the HR system expire every 90 days.",
]

# Scores for (query, passage) pairs: higher means more relevant
scores = model.predict([(query, p) for p in passages])
print(scores)

# Or let the library sort for you
for hit in model.rank(query, passages, return_documents=True):
    print(round(float(hit["score"]), 3), hit["text"])

For multilingual content, the FlagEmbedding reranker examples load bge-reranker-v2-m3 like this (the FlagEmbedding library is MIT licensed):

from FlagEmbedding import FlagReranker

# Follows the FlagEmbedding reranker example for BAAI/bge-reranker-v2-m3
reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

pairs = [
    ["what is a panda?", "The giant panda is a bear species endemic to China."],
    ["what is a panda?", "Pandas is a Python library for data analysis."],
]
# normalize=True maps raw scores to 0-1 with a sigmoid
scores = reranker.compute_score(pairs, normalize=True)
print(scores)

Mixedbread ships its own package for the v2 rerankers. This follows the mxbai-rerank README:

from mxbai_rerank import MxbaiRerankV2

# Follows the mixedbread-ai/mxbai-rerank README (pip install -U mxbai-rerank)
reranker = MxbaiRerankV2("mixedbread-ai/mxbai-rerank-base-v2")  # or large-v2

query = "Who wrote 'To Kill a Mockingbird'?"
documents = [
    "'To Kill a Mockingbird' is a novel by Harper Lee published in 1960.",
    "The novel 'Moby-Dick' was written by Herman Melville.",
    "Harper Lee was born in 1926 in Monroeville, Alabama.",
]
results = reranker.rank(query=query, documents=documents)
print(results)

The Qwen3-Reranker cards use a different pattern: load the model with AutoModelForCausalLM (transformers 4.51 or later), format each pair with an instruction, query and document template, and read the "yes" and "no" logits from the last token. Copy that code from the model card rather than writing it from memory, because the prompt template has to match exactly.

How do you serve and deploy it?

• CPU. Small cross-encoders such as ms-marco-MiniLM-L6-v2 or the smallest Ettin models can rerank 50 short passages per query on CPU. The ms-marco model repo also ships ONNX and OpenVINO exports for faster CPU inference.
• GPU with Text Embeddings Inference. Hugging Face Text Embeddings Inference (TEI) runs sequence-classification rerankers in a Docker container and exposes a /rerank endpoint that takes a query and a list of texts. This is the simplest way to get a production HTTP service for BGE-style rerankers.
• vLLM for LLM-based rerankers. The Qwen3-Reranker repo includes a vLLM example (vLLM 0.8.5 or later) with prefix caching enabled, which helps because every pair shares the same instruction prefix.
• Half precision and batching. Use fp16 or bf16 on GPU, batch all candidates for a query in one call, and cap passage length. Latency grows with candidates times tokens, so 100 candidates of 512 tokens costs far more than 30 candidates of 256 tokens.
• Hosting. Wrap the model in a small FastAPI service, or deploy TEI on Hugging Face Inference Endpoints. For demos, a Gradio app on a Space is enough.

How do you evaluate it?

Use ranking metrics on a labelled set of queries with known relevant passages:

• nDCG@10: rewards relevant passages near the top and supports graded relevance. It is the headline metric on BEIR and MTEB retrieval.
• MRR@10: how high the first relevant passage appears. Good for question answering where one passage is enough.
• MAP and Recall@k: MAP is the MTEB reranking metric; Recall@k tells you whether the right passage made it into the top k you send to the LLM.
• Latency and cost per query at your real candidate count, measured at p50 and p95.

Public benchmarks include MS MARCO, TREC Deep Learning, the BEIR benchmark for zero-shot generalisation across 18 datasets, and the MTEB reranking and retrieval tasks. For a RAG system, also measure the end result: answer correctness and citation accuracy with and without the reranker.

Pitfalls and tips

• Licences differ. Most models above are Apache 2.0, but some popular rerankers are released for non-commercial use only. Read the licence on the model card before you ship.
• Scores are not probabilities. Raw cross-encoder outputs are logits. Apply a sigmoid (or the library's normalize option) before using a fixed threshold, and calibrate that threshold on your own data.
• Truncation hides answers. If the relevant sentence is beyond the model's max length, the score will be low. Chunk documents sensibly instead of passing whole files.
• Candidate count is a latency knob. Going from 100 to 30 candidates often costs little accuracy and cuts latency sharply. Measure it.
• Instructions matter for Qwen3-Reranker. The model card notes that a task-specific instruction usually improves results, so write one for your domain.
• Fine-tune for narrow domains. Sentence-transformers can train CrossEncoder models on your own query-passage pairs; a small fine-tuned model can beat a larger general one on domain data.
• Privacy. Self-hosting a reranker keeps queries and documents inside your infrastructure, which matters for confidential content.

Where to get help

• The Hugging Face forum, for example the thread on inference toolkits for rerankers.
• Issues and docs on the sentence-transformers GitHub repo, which covers CrossEncoder training and inference.
• The Qwen3-Embedding repo and FlagEmbedding repo for model-specific questions and evaluation code.
• The MTEB leaderboard Space and its discussions for benchmark questions.

Frequently asked questions

What is the difference between a reranker and an embedding model?

An embedding model encodes the query and each document separately into vectors, so document vectors can be precomputed and searched quickly. A reranker reads the query and a document together and outputs one score, which is more accurate but must run for every pair at query time. Most systems use both: embeddings to retrieve, a reranker to order.

How many candidates should I rerank?

A common starting point is 50-100 candidates, then keep the top 3-10. Measure Recall@k of your retriever and the latency of your reranker, and lower the candidate count until quality starts to drop.

Can a reranker run on CPU?

Yes, for small cross-encoders such as ms-marco-MiniLM-L6-v2 or the smaller Ettin models, especially with ONNX exports. Billion-parameter rerankers like Qwen3-Reranker-4B or mxbai-rerank-large-v2 realistically need a GPU for interactive latency.

Which reranker is best for multilingual content?

Good open options include bge-reranker-v2-m3, the mxbai-rerank-v2 models and the Qwen3-Reranker family, all of which advertise multilingual support. Test two or three on a sample of your own queries in each language.

Rerankers are one part of a retrieval stack. For how this plays out in practice, read how we approached RAG across many meetings, which powers Acta's ability to ask questions across all your meetings. Related guide: build extractive question answering.