Reranking - A reranker cannot fix what retrieval missed

The previous post ended with a retriever at 0.464 recall@20, which is hybrid search over 482,105 Amazon products, fusing BM25 with a fine-tuned 110M embedding model. That stage hands 20 candidates to the next one. This post is about re-ordering those 20, so the three products a shopping assistant shows are the right three.

A bge-reranker-base cross-encoder, 278M parameters, fine-tuned for one epoch on the same human judgements, lifts the top three from hit@3 0.621 to 0.659 and nDCG@3 0.408 to 0.451. It cost $0.37 on one Modal A10G.

This is the third of three posts. The first built the benchmark and the baselines, the second did the fine-tune.

Bi-encoder, cross-encoder

Stage 1 is a bi-encoder. It embeds the query and each product separately and compares two vectors. That separation is what makes searching 482k products in milliseconds possible, because every product vector is computed once, offline, long before any query arrives.

It is also what limits it. The product vector was computed without knowing the query, so "without rims", "non marking" and "for 8 month old" all have to survive compression into 768 numbers that must serve every possible query at once.

A cross-encoder feeds the query and one product together through a BERT-style model, with full attention between every query token and every product token, and outputs a single relevance score. It can check the exact constraint, because it sees both sides at the same time. The price is 20 forward passes per query instead of one vector comparison, which is why it only ever sees the short list. Scoring all 8,955 test queries times 20 candidates at 320 tokens took 156 seconds on one A10G. Doing that over 482,105 products per query is not something you can do.

Where gradient-boosted rankers fit

The obvious objection to any of this is that the final ranker is just a gradient-boosted tree. In production e-commerce search, very often it is. The last stage is usually a learning-to-rank model built on gradient-boosted decision trees (GBDT), typically LambdaMART in LightGBM or XGBoost.

Those models do not read text. They take a few dozen to a few hundred features per query-product pair - the BM25 score, the dense score, the cross-encoder score, price, rating, review count, click-through and purchase rates, category match, stock, freshness, personalisation signals. Their job is to combine business and behavioural signals with text relevance, and they are fast, interpretable and cheap to retrain daily.

That settled what to build here. ESCI has text and human relevance labels and nothing else - no prices, no clicks, no sales, no users. A LambdaMART over this dataset could only recombine what the three retrievers already know. So the cross-encoder is the stage worth building, and the GBDT goes at the end as a cheap extension. I have not run it.

What the reranker sees

One test query, its 20 candidates in stage-1 order, and the label a human gave each one. The reranker never sees the labels, only the query and the product text.

Query 226 is "+outside weather station and not digital", with 12 Exact products among 39 judged.

Unjudged means never judged for this query, which counts as not relevant.

Look at the first three slots - Substitute, unjudged, Substitute. The Exact products are at 9, 18 and 19. Every candidate is a weather station, so stage 1 did its job, and almost every one of them is digital, which is the one thing the query ruled out. The ranking is sorted by overall text similarity, and "not digital" is a two-word constraint that a 768-float vector computed before the query existed had no reason to preserve.

So the reranker's job is to move the Exact rows up, on the strength of a constraint that only becomes visible when the query and the product are read together.

Training it

rerank_train.py builds pointwise rows from the training judgements - query, product text, label - with label 1 for Exact and 0 for Substitute, Complement or Irrelevant.

pos = [p for p in g.loc[g["esci_label"] == "E", "product_id"] if p in text]
neg = [(p, l) for p, l in zip(neg["product_id"], neg["esci_label"]) if p in text]
...
rows.extend({"query": q, "passage": text[p], "label": 1.0} for p in pos)
rows.extend({"query": q, "passage": text[p], "label": 0.0} for p, _ in neg)

Up to 3 positives and 5 negatives per query gives 144,806 rows over the 20,039 training queries, trained with binary cross-entropy for one epoch, batch 64, 320 tokens, with pos_weight balancing the two classes.

pos_weight = torch.tensor(info["negatives"] / max(info["positives"], 1))
loss = losses.BinaryCrossEntropyLoss(model, pos_weight=pos_weight)

The negatives are the same human-labelled near misses that produced the whole gain in the previous post, and for a reranker they fit even better. Its candidates in production are near misses by construction, because stage 1 has already filtered out everything obviously wrong.

Results

Four final-list systems over the same 20 candidates. recall@20 is fixed at 0.464 by stage 1 and cannot move here, because reranking reorders rather than retrieves.

With about 9 Exact products per query and only 3 slots, recall@3 is capped near 0.33 for most queries, so it is not the number to read. hit@3, meaning at least one Exact product in the three, and nDCG@3, which grades the order and gives substitutes partial credit, are the ones that matter.

The fine-tune improves every one of them. Against the stage-1 order: hit@3 +0.038, MRR@3 0.520 to 0.567, nDCG@3 0.408 to 0.451. The paired bootstrap over the 8,955 test queries puts it ahead of the stage-1 order in 100% of 2,000 resamples on both headline metrics - hit@3 +0.039 with an interval of +0.030 to +0.047, nDCG@3 +0.043 with +0.037 to +0.049.

Off-the-shelf rerankers helped less than I expected, and the margin over the best one is thin. MiniLM is 23M parameters and trained on MS MARCO, a web question answering dataset, and it gains +0.027 nDCG@3 over the stage-1 order. bge-reranker-base off the shelf manages only +0.013, and sits below a model 12 times smaller than itself.

Against MiniLM I have to be careful about the margin. On nDCG@3 the fine-tuned reranker is +0.016 with an interval of +0.011 to +0.021, ahead in 100% of resamples, which is solid. On hit@3 it is +0.008 with an interval of +0.000 to +0.016, ahead in 98.1%, and the lower bound touches zero. So it is clearly better at ordering the three slots, and only marginally better at getting at least one right answer into them. A 23M off-the-shelf model gets most of the way on the easier metric.

My reading of why the fine-tune is what closes the gap: relevance in product search is attribute matching - "without rims", "not digital", "non marking" - not question answering. A strong leaderboard reranker trained on web questions has no particular reason to transfer, and the measurement says it does not transfer far.

One number not to quote. The in-training dev metric, which reranks each dev query's own judged list, reached nDCG@3 0.737 and MRR@3 0.833 after the epoch. That is a far easier task, about 20 products with roughly 8 Exact among them, rather than 20 retrieved candidates where many Exact products are missing entirely. It is useful for watching training and useless as a result.

The ceiling is stage 1

The fine-tuned reranker reaches 0.659. The gap between that and the 0.825 ceiling splits between a better reranker and better retrieval, and only the second kind of work can raise the ceiling itself.

What it fixes and what it breaks

Per query against the stage-1 order, the fine-tuned reranker fixes hit@3 on 10.8% of queries and breaks it on 6.9%. On nDCG@3 it is better on 35.3% and worse on 25.9%. A positive ratio in both cases, not a free lunch in either.

Both directions are about fine attribute constraints.

The broken cases share a shape, and it points at a weakness of the training setup. Pointwise binary labels throw the grades away - under binary cross-entropy a Substitute is scored exactly like an Irrelevant, both 0. For a query like "compact projector", where the candidate list is 15 Exact products and a pile of near misses, a model trained to say Exact or not has no way to learn that a Substitute belongs above an Irrelevant. And nDCG@3, which gives a Substitute 0.1 and an Irrelevant 0, charges it for that. Those are exactly the queries a graded listwise loss exists for.

The pipeline end to end

Both trained stages learned from the same human judgements, and what each is worth is different. Stage 1's fine-tune is a large, unambiguous gain, +0.040 recall@20 over the same model off the shelf and outside noise by a wide margin. Stage 2's fine-tune is a clear gain over doing nothing, and a thinner one over picking the best off-the-shelf cross-encoder you can find.

Across all three notebooks the project came to about $8.40 at list price, of which $5.60 is the one-time Qwen3-Embedding-8B pass over the catalogue. With the two extra training seeds it is about $8.95.

What is next

Three things I want to try, cheapest first. None of them has been run yet.

A LambdaMART over the scores. LightGBM on CPU, minutes of compute, with features per query-product pair of BM25 score and rank, dense score and rank, cross-encoder score, title length and has-brand. On ESCI it can only combine what the three retrievers already know, but it is the shape production rankers actually have, and it answers the gradient-boosted-tree question with a number instead of an argument. To be explicit, this has not been run. It is a planned step.

Listwise training on the graded labels, so Substitutes are ranked above Irrelevants rather than treated as equally wrong. The flag is already in the CLI, and this is the direct fix for the broken queries above.

Hard candidates from stage 1 for training. Draw the reranker's training rows from the fine-tuned retriever's own top 20 on training queries, rather than from the judged lists only, so the training candidates match the serving candidates.

Data for all three posts is the Shopping Queries Dataset (ESCI), Reddy et al., 2022, arXiv:2206.06588, Apache-2.0, copyright Amazon.com, Inc. or its affiliates.