RerankIndex: One Abstraction for Many Vector Indexes

Most of the fastest vector search systems nowadays work in two stages. A first stage searches a cheap representation of the collection and returns a few (100-1000 usually) candidates. A second stage rescores those candidates with a more accurate representation and keeps the top k results. Product Quantization¹ or RaBitQ² with full-precision re-ranking, or sparse retrieval³ followed by multivector (ColBERT^4,5) scoring, all follow this pattern.

Each of these combinations is usually implemented as an index, with its own code and its own parameters. We wrote the generic pattern once instead, as a single building block called RerankIndex.

The abstraction

A RerankIndex pairs a first-stage index with a reranking dataset. At query time the first stage returns k_candidates results, the reranking dataset scores each of them against the query, and the top k are returned.

The two stages are independent. They can encode the vectors differently, use different similarity functions, and even hold different kinds of vectors. Two optional tricks make the second stage cheaper: candidate pruning drops candidates whose first-stage score is far below the k-th best, and early exit stops reranking once several candidates in a row fail to enter the top k. A residual mode covers the case where the second stage stores the part of each vector the first stage left out instead of the full vector, and the two scores are summed.

We introduced RerankIndex in our ECIR 2026 paper, Multivector Reranking in the Era of Strong First-Stage Retrievers⁶. There, we showed that a learned sparse first stage followed by multivector reranking is up to 24× faster than state-of-the-art multivector retrieval systems.

RerankIndex is part of Vectorium, our library for storing, compressing and scoring vectors. Since Vectorium 0.9.3, a RerankIndex is an index in its own right, with the same search interface as any other. It can be used anywhere an index is expected, including as the first stage of another RerankIndex.

Two-stage indexes in kANNolo 0.12

kANNolo⁷ is our library of graph and IVF indexes, built on Vectorium. Release 0.12 makes several two-stage combinations available from Python (pip install kannolo).

Dense. DenseRerankHNSW searches an HNSW⁸ graph over compressed codes (PQ¹, RaBitQ² or RaBitQ-ext⁹) and rescores the candidates on a float16 collection, or on 8-bit RaBitQ-ext codes to save memory.

import numpy as np
from kannolo import DenseRerankHNSW
 
data = np.load("embeddings.npy").astype(np.float32)  # shape (n, 768)
query = data[0]
 
index = DenseRerankHNSW.build_from_array(
    data.flatten(), dim=768,
    m=32, ef_construction=200, metric="dotproduct",
    encoder="rabitq", rabitq_query_bits=1,  # 1-bit RaBitQ first stage, float16 rerank
)
 
dists, ids = index.search(query, k=10, k_candidates=100, ef_search=100)

We can change a stage and obtain a new index by just changing an argument:

# 4-bit RaBitQ-ext first stage, reranked on 8-bit codes instead of float16
index = DenseRerankHNSW.build_from_array(
    data.flatten(), dim=768,
    encoder="rabitq-ext", rabitq_total_bits=4, rerank_bits=8,
)
 
# PQ first stage with 32 subspaces
index = DenseRerankHNSW.build_from_array(
    data.flatten(), dim=768,
    encoder="pq", pq_subspaces=32,
)

Sparse. SparseRerankHNSW targets learned sparse vectors obtained by sparse models such as SPLADE³. For each document it keeps the largest entries that together carry a given share of the vector's L1 mass, and builds the graph over them. The remaining entries form a residual: adding the residual score to the first-stage score gives the exact dot product.

import numpy as np
import scipy.sparse as sp
from kannolo import SparseRerankHNSW
 
# One row per document, one column per vocabulary term (ids below 65,536).
X = sp.load_npz("collection.npz").tocsr()
 
index = SparseRerankHNSW.build_from_arrays(
    X.indices.astype(np.int32),   # term ids
    X.data.astype(np.float32),    # term weights
    X.indptr.astype(np.int64),    # where each document starts
    prune_percentage=60.0,        # first stage keeps 60% of each vector's L1 mass
    residual_rerank=True,         # the remaining entries rescore the candidates
    m=32, ef_construction=1000,
)
 
query_components = np.array([12, 845, 2031], dtype=np.int32)
query_values = np.array([0.9, 0.4, 0.2], dtype=np.float32)
 
dists, ids = index.search(
    query_components, query_values,
    k=10, k_candidates=100, ef_search=100,
)

The first stage can also be quantized to 8 bits (first_stage_value_type="fixedu8"), and the second stage can hold the full vectors instead of the residual (residual_rerank=False), which gives an index similar to DenseRerankHNSW. Every index saves both stages to a single file with index.save(path).

How much faster?

We compared the two-stage indexes with kANNolo's plain HNSW on two dense and two learned sparse collections. Throughput is measured on a single thread with k = 10, and accuracy is the fraction of the true top 10 that an index returns. Each point is the fastest configuration of that index reaching that accuracy.

Dense

The dense collections are MS MARCO¹⁰ passages embedded with Qwen3-Embedding-0.6B¹¹ and the DPR¹² Wikipedia passages embedded with jina-embeddings-v5-text-nano¹³.

The dense HNSW stores the vectors in float16. The two-stage indexes search the graph over 1-bit RaBitQ codes (MS MARCO) or 2-bit RaBitQ-ext codes (DPR) and rerank on float16. They are 1.2–1.4× faster on MS MARCO and 1.1–1.25× faster on DPR, with the gap narrowing towards 99% accuracy. They take 7–15% more memory, since they store the compressed codes on top of the float16 vectors.

Sparse

The sparse collections are NQ¹⁴ from BEIR¹⁵, encoded with SPLADE CoCondenser¹⁶, and MS MARCO encoded with SPLADE-v3¹⁷.

Here the first stage keeps 60% (MS MARCO) or 80% (NQ) of each vector's L1 mass. The two-stage index is about 1.5× faster on NQ, and on MS MARCO the gap grows from 1.8× at 90% accuracy to 4.3× at 99%. Memory stays about the same, because the two stages split the nonzero entries between them.

Part of the gain comes from the pruning itself: at 90% both MS MARCO indexes use the same graph parameters, and the first stage scores shorter vectors. The rest comes from build time. We set a fixed time budget for building our indexes, 1.5 hours on MS MARCO. A graph over the pruned vectors builds in less than half the time (2,265 s against 4,838 s with the same parameters). This leaves room in the budget for more construction effort, which the two-stage index uses to be faster at 95% and 99% accuracy. The same settings would take the plain HNSW build to 6,165 and 9,015 seconds, over budget.

What comes next

Since a RerankIndex is itself an index, its two stages can be any index and dataset. A few combinations the library already allows, which we have not benchmarked yet:

• Dense-sparse hybrid pipelines. A sparse first stage rescored with dense embeddings, or a dense first stage rescored with sparse ones, or maybe some rank fusion of the two (requires an extension of the RerankIndex).
• Cascades. A RerankIndex as the first stage of another RerankIndex: 1-bit codes, then 4-bit codes, then float16; or sparse retrieval, then a dense model, then multivector scoring.

If you try kANNolo on your data, tell us how it goes on GitHub.