Product search on Amazon shopping queries
A production-shaped product search pipeline, built and measured end to end on the Amazon Shopping Queries Dataset (ESCI). Retrieval narrows 482,105 products to 20 candidates, a cross-encoder re-orders those 20, and the top 3 are what a shopping assistant would hand to a language model.
The code is on GitHub. Everything heavy ran on Modal, nothing was trained locally, and every number below comes from an executed notebook cell.
What it does
| Stage | System | Judged on | Off the shelf | Fine-tuned |
|---|---|---|---|---|
| 1. Retrieval over 482,105 products | Hybrid BM25 + bge-base, 110M | recall@20 | 0.437 | 0.464 |
| 2. Rerank the 20 candidates | bge-reranker-base, 278M | hit@3 / nDCG@3 | 0.648 / 0.421 | 0.659 / 0.451 |
| 3. Answer | LLM over the 3 products | Not measured here |
The headline
A bge-base-en-v1.5 model, 110 million parameters, fine-tuned for 8 minutes on human-labelled hard negatives scores 0.460 ± 0.002 recall@20 across three seeds, against 0.455 for Qwen3-Embedding-8B off the shelf. It matches or narrowly beats the 8B model on recall@20 and clearly beats it on recall@10, at 1/70 of the parameters and 1/25 of the encode time.
A paired bootstrap over the 8,955 test queries puts it ahead on recall@20 in 98.8% of 2,000 resamples, and on recall@10 in 100%.
What I found
| Which negative beats how many negatives | Adding a random negative bought +0.001 recall@20. Making it a human-labelled near miss bought +0.021 |
| Sparse judgements break negative mining | With 20 of 482,105 products judged per query, the model's own near misses are mostly unjudged right answers, so mined negatives tied with no fine-tuning at all |
| BM25 is a serious opponent | 0.383 recall@20 with no model and no training, beating a 600M embedding model outright |
| Off-the-shelf size buys little | Tripling the parameters bought +0.015. Rank fusion of a 110M model with a lexical index matched it for free |
| A reranker cannot fix recall | 82.5% of queries have a right answer among the 20 candidates, so hit@3 cannot pass 0.825 whatever the reranker does |
How it was built
Three executed notebooks, each ending in a report.json and a row in FINDINGS.md. The test split is the official query split made product-disjoint, which meant removing every product that appears in a test query's judged list from the training judgements. That cost a sixth of the training data and makes the score measure generalisation to new products rather than memorisation.
Total spend: about $8.95 at list price, of which $5.60 is a single pass of the 8B model over the catalogue.
Write-ups
I wrote the project up in three posts:
- Retrieval 1 - Building benchmarks and why keyword search still wins
- Retrieval 2 - Fine-tuning a small embedding model gives surprising results
- Reranking - A reranker cannot fix what retrieval missed
Data is the Shopping Queries Dataset (ESCI), Reddy et al., 2022, arXiv:2206.06588, Apache-2.0, copyright Amazon.com, Inc. or its affiliates.