OpenRouter 实测 RAG 重排序：274 题基准中 precision@1 从 81% 提升到 97%

When a RAG (Retrieval-Augmented Generation) app gives wrong answers even though the right information is in your documents, retrieval is often the cause. Vector search is great at finding chunks on the right topic, but it often falls for near misses, like pulling the bucket limit for the Enterprise plan when the user asked about the Growth plan. Reranking in RAG fixes this by taking your vector search’s top 30 candidates and passing them through a dedicated cross-encoder model that re-scores each chunk directly against the question.

On a 274-question benchmark, standard vector search with google/gemini-embedding-2 landed the exact right chunk at the top 81% of the time. Adding voyageai/rerank-3-lite to rerank the top 30 pushed that to 97% for roughly $0.10 per 1,000 queries. Keep in mind this was tested on a small synthetic dataset, so your real-world gains will depend on your documents, but the workflow below will help you measure your own baseline.

TL;DR
Retrieve about 30 candidates, rerank them, and send the top 3 to the LLM.
voyageai/rerank-3-lite and voyageai/rerank-3 were the most accurate dedicated rerank models and the cheapest of all 11 rerankers, at $0.10 and $0.26 per 1,000 queries.
An LLM reranker (openai/gpt-6-luna) put the right chunk first for 99.6% of questions, but each call took 4.0 seconds against 1.3 for a rerank model.
The cheap voyageai/voyage-4-lite plus voyageai/rerank-3-lite reached 96.7% precision@1, against 81.0% for the more expensive google/gemini-embedding-2 alone.
Set input_type on embedding calls. Leaving it out costs 7 to 12 points of precision@1.
cohere/rerank-v3.5 reads only the first 4,096 tokens (4K) of a document and gives no error.

Why does RAG retrieve the wrong chunks?

Standard vector search struggles because embedding models compress an entire chunk into a single vector before seeing the user’s question. If a 130-word chunk explains bucket limits, the specific plan name is only one or two words out of that entire block. As a result, the vectors for the Starter, Growth, Scale, and Enterprise plans cluster right on top of each other in vector space, and the retriever can easily rank the wrong plan first.

To test retrieval accuracy without relying on subjective scoring, we built a controlled dataset for a fictional storage service named Fernbank. The corpus contains 274 chunks spanning documentation, error codes, SDK guides, changelog entries, and noise, including unanswered forum posts and support tickets designed to trap naive keyword and vector matches. We generated each chunk from a fixed set of facts with anthropic/claude-sonnet-5.5 and checked it with a script. A separate model, openai/gpt-6.1-sol, wrote the questions. Because every question maps to one correct chunk, we can score retrieval without judgment calls.

In the full test set, the question was “How many buckets can I create in a single Fernbank account on the Growth plan?”. google/gemini-embedding-2 ranked the Starter, Enterprise, and Scale plan chunks 1, 2, and 3. An unanswered forum post ranked fourth, and the actual Growth plan answer sat at fifth. If your RAG pipeline only sends its top three candidates to the LLM, the model never even sees the right document.

A reranker fixes this by using a cross-encoder architecture. Rather than evaluating documents in isolation, it processes the user’s question alongside each candidate chunk. Running a cross-encoder across thousands of documents is too slow for real-time queries, so a vector search serves as a fast retriever, narrowing the field to a shortlist before the reranker sorts it.

How to add a reranker to your RAG pipeline

You don’t need to replace your existing embedding search. Keep it as the first retrieval stage, increase the number of candidates it returns, and add a reranking call before passing the final results to your LLM.

Retrieve 30 chunks with embeddings, rerank them, and send the top 3 to the LLM.

The following Python example uses our embeddings and rerank APIs. It retrieves candidate chunks using cosine similarity, reranks them with voyageai/rerank-3-lite, and returns the three highest-ranked results.

You’ll need an OPENROUTER_API_KEY environment variable, along with numpy and requests installed.

Python

import os
 
import numpy as np
import requests
 
API = "https://openrouter.ai/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"}
 
def embed(texts, input_type, model="google/gemini-embedding-2"):
    r = requests.post(f"{API}/embeddings", headers=HEADERS, timeout=120,
                      json={"model": model, "input": texts,
                            "input_type": input_type})
    r.raise_for_status()
    data = sorted(r.json()["data"], key=lambda d: d["index"])
    v = np.array([d["embedding"] for d in data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)
 
def retrieve(question, chunks, chunk_vectors, n=30):
    sims = chunk_vectors @ embed([question], "search_query")[0]
    return [(sims[i], chunks[i]) for i in np.argsort(-sims)[:n]]
 
def rerank(question, candidates, top_n=3, model="voyageai/rerank-3-lite"):
    r = requests.post(f"{API}/rerank", headers=HEADERS, timeout=120,
                      json={"model": model, "query": question,
                            "documents": candidates, "top_n": top_n})
    r.raise_for_status()
    data = r.json()
    ranked = [(res["relevance_score"], candidates[res["index"]])
              for res in data["results"]]
    return ranked, data.get("usage", {}).get("cost")
 
# Your knowledge base, already split into chunks
chunks = [
    "Starter plan limits: an account on the Starter plan can create up to "
    "10 buckets.",
    "Scale plan limits: accounts on the Scale plan can hold up to "
    "2,500 buckets.",
    "Enterprise plan limits: Enterprise accounts can create up to "
    "40,000 buckets.",
    "Growth plan limits: an account on the Growth plan can hold up to "
    "150 buckets.",
    "Q: What's the bucket cap on the Growth plan? Reply: I think it "
    "changed recently, check the docs.",
    "Growth plan object size: a single object on the Growth plan can be "
    "up to 50 GB.",
]
chunk_vectors = embed(chunks, "search_document")
 
question = ("How many buckets can I create in a single account "
            "on the Growth plan?")
retrieved = retrieve(question, chunks, chunk_vectors)
print("Vector search:")
for sim, text in retrieved:
    print(f"  {sim:.3f} {text[:60]}")
 
ranked, cost = rerank(question, [text for _, text in retrieved])
print(f"Reranked (${cost:.7f}):")
for score, text in ranked:
    print(f"  {score:.3f} {text[:60]}")

Output

Vector search:
  0.832 Growth plan limits: an account on the Growth plan can hold u
  0.758 Starter plan limits: an account on the Starter plan can crea
  0.741 Scale plan limits: accounts on the Scale plan can hold up to
  0.737 Enterprise plan limits: Enterprise accounts can create up to
  0.717 Growth plan object size: a single object on the Growth plan
  0.695 Q: What's the bucket cap on the Growth plan? Reply: I think
Reranked ($0.0000038):
  0.922 Growth plan limits: an account on the Growth plan can hold u
  0.793 Q: What's the bucket cap on the Growth plan? Reply: I think
  0.727 Starter plan limits: an account on the Starter plan can crea

With these short, one-sentence chunks, vector search already places the correct Growth plan limit first. The cosine similarity scores range from 0.695 to 0.832, so several candidates are relatively close. After reranking, the correct chunk receives a relevance score of 0.922. The unanswered forum post moves into second place with a score of 0.793, ahead of the Starter plan entry.

The second result illustrates an important limitation that relevance isn’t the same as correctness. The forum post asks the same question, so the reranker considers it relevant even though it doesn’t answer the question. In this simplified example, the reranker doesn’t fix the first result because it was already correct. In the full benchmark, with 130-word chunks, vector search put the wrong chunk first for 19% of questions, including the Growth plan question, and reranking with voyageai/rerank-3-lite fixed most of them.

The practical lesson is to send the LLM a few strong candidates rather than relying on a single result. You should also consider filtering out unanswered discussions and other content that mentions a question without resolving it.

There are two other details worth getting right in the code:

• First, the input_type parameter tells the embedding model whether it’s encoding a search query or a document. Some models treat these inputs differently, so omitting the parameter can hurt retrieval. In our tests, leaving it out reduced precision@1 for Gemini from 81.0% to 74.1%, and for voyageai/voyage-4-lite from 64.6% to 52.9%. See the embeddings API reference, which takes it as a string and documents search_query and search_document as its values.

• Second, top_n controls how many results the reranker returns, not how many it evaluates. The model still scores every candidate you send it, so the number of candidates affects both cost and latency.

How do you know reranking helped?

A reranker can improve result ordering without necessarily improving the answers your users receive. To determine whether it’s worth adding, measure retrieval accuracy before and after the change, then evaluate the final answers.

1. Start with precision@1: the percentage of questions for which the first retrieved chunk contains the correct answer. You don’t need a huge evaluation set to begin. Fifty real questions from your application logs, each labeled with its correct answer chunk, can give you an initial baseline. The guide to building a golden evaluation dataset from production traffic explains how to put one together.

2. Don’t rely on recall@5 alone: It measures whether the correct chunk appears anywhere in the top five, regardless of its position. In our test, Gemini achieved 98.2% recall@5 but only 81.0% precision@1. The right information was usually somewhere in the results, but it wasn’t always at the top.

We also tested the effect on the final answers. We used anthropic/claude-haiku-5.5 to answer all 274 questions using the chunks returned by each retrieval configuration, then compared the responses with the known facts.

Our October 8, 2026 run had 274 questions. Reranking used voyageai/rerank-3-lite.

The results show that reranking helped the LLM answer more questions correctly while using fewer chunks. Most incorrect answers came down to missing information. In nine out of ten cases where the correct chunk wasn’t among the top three, the model responded that the supplied passages didn’t contain the answer. In other words, the application told users it didn’t know something that its own documentation could answer.

Reranking addressed that problem by improving which chunks reached the model. It also let us send three chunks instead of five, reducing the LLM answering cost from $0.22 to $0.16 per 1,000 questions. The rerank call added $0.10, so in our setup reranking cost a net $0.04 more per 1,000 questions.

Whether reranking saves money for you depends on your models, document sizes, and query volume, so measure cost and accuracy across the whole pipeline, from retrieval through the final answer.

Which reranker model is best for RAG?

We compared 11 rerankers using the same 30 candidates retrieved by google/gemini-embedding-2 for each of the 274 questions. That gave us a consistent basis for comparing their accuracy, latency, and cost.

The Voyage rerankers offered the best combination of accuracy and price among the dedicated reranking models we tested.

Our run on October 8, 2026. Costs are the usage cost per response, which matches GET /api/v1/generation. About 0.9 seconds of each median is network overhead from our test machine.

On this dataset, voyageai/rerank-3 ranked the correct chunk first 97.8% of the time, while the Lite version reached 97.1%. That difference amounts to only two questions in our 274-question test, so it’s too small to establish that the more expensive model is consistently better. Start with voyageai/rerank-3-lite and compare it with voyageai/rerank-3 on your own evaluation set. Both use the same /rerank endpoint, so switching models only requires changing the model identifier.

The price differences largely reflect how providers charge, as listed on our rerank model pages. Voyage and Qwen bill by token ($0.02 to $0.20 per million), while Cohere charges $0.001 to $0.0025 per search of up to 100 documents, regardless of document length. Because our 30 candidates contained around 5,000 Voyage tokens, the Voyage models were particularly inexpensive. On these short chunks, Cohere’s rerankers cost roughly 4 to 25 times as much as the Voyage options, depending on the models compared. Sending 101 documents to Cohere also triggered a second search charge in our test.

Reranking made a particularly noticeable difference for questions about specific error codes, such as FB-2411. Embedding search alone put the correct entry first only 25% of the time with voyageai/voyage-4-lite and 53.6% with google/gemini-embedding-2. All 11 rerankers achieved 100% on this subset. Since error-code entries tend to discuss similar errors, their embeddings can look alike. A cross-encoder can compare the exact code in the question with the code in each candidate and distinguish the right entry.

If you’re interested in which models developers are using most, our rerank model rankings provide another useful reference. As of October 20, 2026, voyageai/rerank-2.5 led with 4.68 million requests in the week to October 18, and it scored 96.4% in our test. Usage popularity is worth considering, but it shouldn’t replace testing a model against your own documents. Model performance, prices, and provider latency can change. The reranking model collection lists the available options and their current details.

How many documents should you rerank?

Start by reranking around 30 candidates, then adjust the number based on your evaluation results.

To see how candidate count affects performance, we tested voyageai/rerank-3-lite with different numbers of candidates retrieved by voyageai/voyage-4-lite:

Our run on October 8, 2026, used 274 questions, voyageai/voyage-4-lite as the first stage.

Increasing the candidate count from 10 to 30 improved precision@1 from 94.5% to 96.7%. Increasing it again to 50 brought precision to 97.1%, fixing just one more question while increasing reranking cost by about two-thirds. Going from 50 to 100 candidates didn’t improve precision.

Reranking has limits, as it can promote a relevant chunk only if the first retrieval stage includes it in the candidate set. If vector search misses the answer entirely, no reranker can bring it back.

For a larger knowledge base, use your evaluation set to find the point where retrieving more candidates stops meaningfully improving the percentage of questions for which the correct chunk is available. That’s a better way to choose the candidate count than assuming 30 is optimal for every application.

Can LLMs be used as rerankers?

In our benchmark, LLM-based rerankers were the most accurate, though they took longer to return results. Instead of sending each candidate through a dedicated reranking model, we gave a chat model the question and all 30 numbered chunks, then asked it to return the IDs of the most relevant chunks in order. We used a strict structured output schema to ensure the response followed the expected format.

The two approaches differ in a key way. A dedicated reranker scores each candidate individually against the question. An LLM can read all candidates together and compare them, which helps when several documents contain similar information.

In our test, openai/gpt-6-luna achieved 99.6% precision@1 at $0.70 per 1,000 queries, the low end of an LLM cost range that reached $4.14 for google/gemini-3.8-flash. That’s higher accuracy than the dedicated rerankers, but each call took around four seconds in our setup, compared with about 1.3 seconds for a dedicated model.

That extra time matters in an interactive application, where the user is waiting for an answer. LLM reranking may fit better for batch processing or applications where retrieving the correct information is especially important, and additional latency is acceptable.

Keep in mind that our benchmark used clean, model-generated text. Real knowledge bases have inconsistent formatting, incomplete documents, and other complications, so we can’t assume LLM rerankers will maintain the same edge on messier, real-world data.

Do you still need a reranker with hybrid search or a better embedding model?

Hybrid search and better embedding models can both improve retrieval, but our benchmark suggests that neither was as effective as adding a reranker. Hybrid search combines vector similarity with keyword-based retrieval, often using BM25. We combined the results with reciprocal rank fusion (RRF), which merges the two rankings.

On our test set, hybrid search increased precision@1 for voyageai/voyage-4-lite from 64.6% to 69.3%. With google/gemini-embedding-2, however, precision@1 fell from 81.0% to 78.8%, even though recall@5 improved. A more capable embedding model outperformed the cheaper alternative. Gemini achieved 81.0% precision@1, compared with 64.6% for Voyage 4 Lite. But Gemini’s embedding price was $0.20 per million tokens, versus $0.02 for Voyage 4 Lite. Combining the cheaper embedding model with voyageai/rerank-3-lite reached 96.7% precision@1.

Hybrid search and better embeddings are still useful, because they do different jobs. Hybrid search can help retrieve candidates that vector search misses, and a stronger embedding model can improve the initial ranking. A reranker adds another layer by comparing each candidate directly with the question.

For the best results, evaluate these options separately and together. You can use our embedding model comparison to help choose an embedding model for the first retrieval stage.

What breaks reranking?

Reranking can fail silently when a document exceeds the model’s context limit. For example, cohere/rerank-v3.5 has a 4,096-token context limit, so it may never see the part of a document that contains the answer.

We tested this with two 11,000-token documents that differed only in their final sentence. One contained the answer, but both received the same relevance score: 0.2703. The model had effectively missed the detail that distinguished them. By comparison, voyageai/rerank-3-lite scored the documents 0.7812 and 0.3242, correctly ranking the one with the answer higher. The Cohere 4 models, voyageai/rerank-3, and qwen/qwen3-reranker-8b also ranked the correct document higher. Check your reranker’s context limit and keep chunks comfortably within it.

Another issue is model compatibility, since the /rerank endpoint accepts only rerank models. Sending the chat model openai/gpt-6-luna to it returned 400 Model openai/gpt-6-luna does not exist, the same error a misspelled model identifier gets. Before making a request, check that the model appears in the reranking model collection.

Next steps

• Rerank your top 30 with voyageai/rerank-3-lite, then measure.

• Rerank API reference: every request field, including provider routing.

• RAG with embeddings and rerank: the full pipeline through answer generation.

• How to benchmark LLMs: compare models on your own data.

Frequently asked questions

What is reranking in RAG?

Reranking in RAG is a second retrieval step in which a rerank model scores each chunk returned by vector search against the question and reorders them, so the most relevant few reach the LLM. The rerank model reads the question and chunk together, which lets it catch near-misses such as the right limit for the wrong plan. In our 274-question test, it raised the rate at which the right chunk came first from 81% to 97%.

What are the best reranker models for RAG?

In our October 2026 test, voyageai/rerank-3 and voyageai/rerank-3-lite were the most accurate dedicated rerank models and the cheapest rerankers, putting the right chunk first for 97.8% and 97.1% of 274 questions. LLM rerankers scored higher but took about 4 seconds per call.

How many documents should I rerank?

Rerank about 30 candidates, and for a larger knowledge base, increase the number until the right chunk reliably appears. In our test, going from 30 to 50 raised precision@1 from 96.7% to 97.1% (a one-question gain within noise) and increased rerank cost by two-thirds.

References

• Rerank API reference and embeddings API reference

• Rerank models and rerank rankings

• Structured outputs