Retrieval 2 - Fine-tuning a small embedding model gives surprising results

I fine-tuned bge-base-en-v1.5, a 110 million parameter embedding model, on real Amazon shopping queries. One epoch, 8 minutes on a single GPU, $0.38.

It scores 0.460 ± 0.002 recall@20 across three training seeds, against 0.455 for Qwen3-Embedding-8B off the shelf. So it matches or narrowly beats the 8B model on recall@20, and clearly beats it on recall@10, at 1/70 of the parameters and 1/25 of the encode time.

I did not want to claim that on the strength of one run, so there are two checks behind it. The first is a paired bootstrap, which redraws the 8,955 test queries 2,000 times with replacement and scores every system on the same redraw. The fine-tuned model is ahead on recall@20 in 98.8% of resamples - +0.006, with a 95% interval of +0.001 to +0.011, just above zero. On recall@10 it is ahead in 100% of resamples, +0.015, interval +0.011 to +0.020. A narrow but real edge at k = 20, and a comfortable one at k = 10.

This is the second of three posts. The first built the benchmark and the baselines. The third puts a reranker on top.

The bar

From the previous post, on the same frozen 8,955 product-disjoint test queries over the same 482,105-product corpus. The Shopping Queries Dataset (ESCI) labels products Exact, Substitute, Complement or Irrelevant, and recall@20 is the share of a query's Exact products inside the 20 candidates stage 1 hands to the reranker.

0.421 is where the 110M model starts. 0.455 is the best off-the-shelf number. 0.437 is what hybrid fusion gives for free.

The five negative strategies

Full fine-tune with MultipleNegativesRankingLoss. One epoch over 54,880 rows, which is 20,039 product-disjoint training queries times up to 3 Exact products each, at batch 64, learning rate 2e-5, 256 tokens, one A10G per run. 858 steps, about 8 minutes.

The question I wanted answered was which negatives teach the model most. So five runs differ in exactly one thing - which negative is paired with each row.

When a query has no negative of the requested kind, the row falls back to a random product. I report that count, so the comparison is honest about how often it happens.

if negatives == "labelled":
    pool = [p for p in g.loc[g["esci_label"].isin(["S", "C", "I"]), "product_id"] if p in text]
elif negatives == "substitutes":
    pool = [p for p in g.loc[g["esci_label"] == "S", "product_id"] if p in text]
elif negatives == "mined":
    pool = [p for p in (mined or {}).get(int(qid), []) if p in text]
else:
    pool = []
...
for p in exact[:max_pos]:
    row = {"anchor": q, "positive": text[p]}
    if negatives != "none":
        if pool:
            neg = pool[rng.integers(len(pool))]
        else:
            fallback += 1          # reported in the run's report.json
            neg = all_ids[rng.integers(len(all_ids))]

One training row shows the difficulty. For the query "awnmower tires without rims" the Exact product is a MARASTAR 15x6.00-6 front tire assembly for a Craftsman mower. The Substitute pool contains a "2PK 13x5.00-6 Turf Mower Tractor Tire with Gray Rim" - almost every word shared with the positive, separated only by "with rim", which is the one part the customer cares about. BM25 cannot see that. The base model barely can.

Two implementation details mattered more than they look.

loss = losses.MultipleNegativesRankingLoss(model)
targs = SentenceTransformerTrainingArguments(
    per_device_train_batch_size=64,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    lr_scheduler_type="cosine",
    batch_sampler=BatchSamplers.NO_DUPLICATES,   # two rows of one query in a batch would make
                                                 # its other Exact product a false negative
    prompts={"anchor": prompt},                  # bge's query prefix, on the anchor column only
)

MultipleNegativesRankingLoss scores every query in a batch of 64 against all 128 passages in it, and trains a 128-way classification whose right answer is the query's own positive. Every other passage, including the 64 hard negatives, pushes back.

With several Exact products per query, NO_DUPLICATES is not optional. Two rows of the same query in one batch would train the model to push a genuine Exact product away. And bge's prefix goes on queries only, at training and at search time. Encoding passages with it, or queries without it, costs points silently.

Results

The comparison between those five runs is the main finding of the whole project.

The bootstrap puts that far outside noise. Labelled over random is +0.021, interval +0.018 to +0.024, ahead in 100% of resamples. Labelled over pairs-only is +0.022, also 100%.

Substitutes alone, at 0.461, ties with the full S/C/I mix - the difference is −0.000, interval −0.003 to +0.002, labelled ahead in only 38% of resamples. So the Substitutes are doing the work, and the Complements and Irrelevants add little. That is worth noting, because 15.6% of the Substitutes-only rows fell back to a random negative when the query had no Substitute at all.

Mined negatives hurt. The standard recipe when you have no labels is to take the base model's top ranked non-Exact products. That lands at 0.424, which is below pairs-only at 0.439, and against the untrained 0.421 it is a tie - +0.003, interval −0.002 to +0.008, ahead in 88% of resamples. Eight minutes of training for nothing measurable.

Hybrid still adds a little on top, but not separably. Fusing BM25 with the fine-tuned model gives 0.464 recall@20, 0.600 recall@50 and 0.825 hit@20, the best stage-1 numbers in the project. But the gain over the fine-tuned model alone is +0.002, interval −0.002 to +0.006, ahead in 85% of resamples, so it is within noise. Fusion added +0.016 to the base model and nothing measurable to the fine-tuned one, because the dense side now covers what BM25 used to contribute.

One guard rail I would copy. The before row recomputed inside the labelled run reproduced the baseline bge-base numbers exactly, 0.421 / 0.544 / 0.794. Same harness, not a reimplementation.

The left panel caught me out at first. Random and none converge to 0.3 to 0.45, the hard-negative runs sit at 0.9 to 1.0, and the hard runs are the better models. Hard negatives make the 128-way batch classification harder by design, so training loss is not comparable across negative kinds. Only the dev and test metrics rank the runs.

Dev recall@20 ranked them the same way as the test set - labelled 0.784, substitutes 0.785, random 0.775, none 0.775, mined 0.758. Different level, same order.

Where each model wins

Aggregate parity hides disagreement. Per query against Qwen3-8B on recall@20, the fine-tuned model is better on 30.8%, worse on 28.0% and equal on 41.2%. The two are close in the mean and disagree on most of the hard queries.

Against its own base model it is better on 32.8% and worse on 16.6%, and the share of test queries with no Exact product in the top 20 falls from 20.5% to 18.5%. Qwen3-8B is at 18.3% and BM25 at 25.2%.

The three groups make sense when you read them. The gains are constraint-heavy queries the base model fumbled, often going from 0.0 to 1.0 - ".5 inch suction cups without hooks", "10x3 cheesecake boxes without windows", "1/4 plastic ball valve", "50 ton porta power". The losses are single-product queries that flip, usually model numbers or exact titles - "bt-705", "batman dvd", "4 inch non marking caster wheel". Those are what BM25 or a reranker over the union recovers, which is why I kept both. And the queries Qwen3-8B still wins need world knowledge that one epoch on 20k shopping queries will not teach - "2 iron golf club", "alexa bedframe", "an appeal to heaven flag".

Splitting by how many Exact products a query has sharpens it further.

The fine-tuned model wins on queries with 1 to 10 Exact products. Qwen3-8B keeps its edge on 11 or more, and on recall@50 overall, 0.595 against 0.587. If your product surface shows three results, the first three rows are the ones that matter.

How sure I am

Two sources of noise, measured separately. Neither touches the official test split, so the numbers stay comparable with published ESCI results.

The first is which queries ended up in the test set. A single system carries an interval of about ±0.007 and a paired difference about ±0.005.

The second is which random choices the training run made. The seed decides which 3 Exact products each query keeps, which negative each row gets, the batch order and dropout. So I ran two more labelled runs with seeds 43 and 44, same data and settings, scored on the same frozen test set.

Training-side noise is small, a spread of 0.003 in recall@20, which is smaller than the test-side interval. All three seeds are above Qwen3-8B's 0.4551 on recall@20, but the lowest margin is +0.003, and on its own that sits inside the paired interval. On recall@10 every seed is at least +0.013 ahead, well outside it.

The seed-42 run that the tables quote is the best of the three by 0.001 to 0.003, so the number to quote is the mean, 0.460. That is why I say matching or narrowly beating at k = 20 and clearly beating at k = 10, rather than just beats.

Serving, and what it cost

The fine-tuned model is a 440 MB sentence-transformers folder, served exactly like the base model. Encode the catalogue once, about 6 minutes on an A10G, store 768-float vectors, encode each query at request time with the bge prefix.

Against the 8B alternative: 768 floats per product in the vector store rather than 4,096, a model that fits on a CPU box, and a catalogue re-encode in minutes rather than 2.5 hours. That last one comes round again every time, because any fine-tune invalidates every stored product vector. Re-encode cost is what decides whether you can afford to iterate at all.

Plus $0.54 for the two extra seeds, and nothing for the bootstrap, which runs locally on the saved per-query scores. The five-way negatives experiment that this whole post rests on cost $1.57.

Four things I would keep

Stage 2, the cross-encoder reranker over those 20 candidates, is in the next post.