Search Lab Notes

Which is cheaper to compute: cosine distance or dot product? What does it mean to be sensitive to magnitude? Is that good or bad? Sounds like it's bad hence the need for L2 normalization, but I don't have an intuition / mental model as to why it's bad.

Dot product, clearly. For d-dimensional vectors:

• Dot product a·b = Σ aᵢbᵢ — that's d multiplies and d−1 adds. Done.
• Cosine = (a·b) / (‖a‖·‖b‖) — the dot product plus two magnitudes (each is itself a dot-product-and-sqrt), a multiply, and a division.

So cosine is roughly 3× the arithmetic, and worse, it adds a sqrt and a divide — both expensive in the tight inner loop of an ANN scan that runs millions of times.

But here's the trick. If every vector is pre-scaled to length 1 (‖a‖ = ‖b‖ = 1), then the denominator is 1×1 = 1 and cosine collapses into a plain dot product. You pay the normalization cost once, at index time, and then every query-time comparison is the cheap operation. That's the entire reason the pipeline L2-normalizes: you get cosine's behavior at dot-product cost.

What "sensitive to magnitude" means

Decompose the dot product:

a·b = ‖a‖ · ‖b‖ · cos(θ) └── magnitudes ──┘ └ angle ┘

The dot product depends on two things: the angle between the vectors (θ) and their lengths. Cosine keeps only the cos(θ) part — the lengths divide out.

"Sensitive to magnitude" = the score moves when you make a vector longer or shorter, even if its direction doesn't change. Take a vector, double its length, point it the exact same way — its dot-product score with everything doubles, though its meaning is identical.

Why that's bad (the mental model)

The key fact about embeddings: direction encodes meaning; magnitude usually encodes something incidental. Two documents about the same topic point their arrows the same way. The length of the arrow, for a typical sentence encoder, tends to correlate with junk we don't care about — document length, token count, word frequency, how much "stuff" is in the text — not topical relevance.

So picture vectors as arrows from the origin:

• Cosine asks "which way does the arrow point?" → pure topical match.
• Dot product asks "which way, and how long?" → a long arrow pointing slightly off-topic can outscore a short arrow pointing exactly on-topic.

That's the badness: magnitude becomes an unearned thumb on the scale. You wanted to rank by "how semantically similar is this doc to my query" (the angle), but document length leaks in and lets verbose or high-norm documents jump the queue for reasons that have nothing to do with relevance.

The analogy I like: two people give you the same answer, one shouts and one whispers. Dot product rewards the shouter (large magnitude). But loudness isn't correctness. Cosine listens to what they said (direction), ignoring how loud (magnitude). In retrieval you want the answer, not the volume.

The honest nuance

Magnitude sensitivity isn't universally bad — it's bad when magnitude is incidental to what you're measuring, which is the common case for dense sentence encoders. Sometimes magnitude is deliberately meaningful signal you want to keep: learned-sparse models like SPLADE, or systems doing maximum-inner-product search where norm encodes importance/confidence. There you'd use raw dot product on purpose. The rule isn't "cosine good, dot bad" — it's "normalize away magnitude when it's noise; keep it when it's signal." For MiniLM-style embeddings here, it's noise.

Back to this project

The pipeline stores L2-normalized MiniLM vectors, which buys three things at once:

1. Ranking is by angle only — pure semantic match, no length bias.
2. cosine ≡ dot product, so it's the cheapest comparison and it doesn't matter which operator the backend exposes.
3. It's why the dense arm ties across ES and Postgres to four decimals — identical normalized vectors, and cosine/dot are the same thing on them, so both stores' HNSW returns the same neighbors. The "same vectors → stores tie" result from M4 quietly depends on this normalization.

I don't understand these terms very well:

sentence-transformers — the library/method (SBERT, Reimers & Gurevych 2019) that turns a token-level transformer into a sentence embedder: add → pooling + fine-tune in a → contrastive / siamese setup so a single vector per text is cosine-comparable. Behind the embeddings extra.

Pooling — collapsing a transformer's per-token vectors into one sentence vector. all-MiniLM-L6-v2 uses mean pooling (attention-masked average). Off-the-shelf BERT's [CLS] token pools poorly for similarity — pooling choice + training is what makes vectors comparable.

Contrastive learning / in-batch negatives — the training that makes embeddings semantic: pull positive pairs (question↔answer, sentence↔paraphrase) together in vector space and push mismatched pairs apart; other items in the batch serve as negatives. The all-* models trained this way over ~1B pairs. The semantics live in the trained geometry, not the architecture.

These three terms are really one pipeline. Let me build the chain first, then each piece will land, because each term only makes sense as a step toward "one cosine-comparable vector per text."

The problem all three solve

A raw transformer (BERT, MiniLM) doesn't output a sentence vector. You feed it "the cat sat" and it gives you one vector per token — a small bag of ~4 contextualized vectors, not a single embedding. But to do similarity search you need exactly one vector per text so you can compare with cosine/dot and stuff it into an ANN index.

So there are two gaps to close:

1. Many token vectors → one vector. (That's pooling.)
2. Make that one vector's geometry actually mean something — so that "close in vector space" = "close in meaning." A pretrained transformer does not give you this for free. (That's contrastive training.)

sentence-transformers / SBERT is the recipe that does both. Now the pieces:

Pooling — collapsing N token vectors into 1

You have per-token vectors; you need one. Pooling is the collapse. Common choices:

• [CLS] pooling — BERT prepends a special [CLS] token and you just use its output vector as "the sentence." Convenient, but it pools poorly for similarity: [CLS] was trained for BERT's classification objectives, not to sit in a space where cosine means "similar meaning." Out of the box its geometry is wrong for our purpose.
• Mean pooling — average all the token vectors. This is what all-MiniLM-L6-v2 uses, and it's more robust because every token contributes.

"Attention-masked average" is a small but important detail: to run texts in batches, short texts get padded with meaningless [PAD] tokens so every row is the same length. The attention mask marks which tokens are real vs padding. Masked mean pooling averages only the real tokens — you don't want padding dragging the vector toward garbage. So: sum the real token vectors, divide by the count of real tokens.

Key point the glossary is making: pooling choice alone doesn't save you. Mean-pooling a raw BERT still gives mediocre embeddings. Pooling gets you a vector; training is what makes it a good one.

Contrastive learning — where the meaning actually comes from

This is the heart of it. A pretrained transformer's vectors aren't arranged so that geometric closeness = semantic closeness. You have to train the space into that shape, and contrastive learning is how.

The idea: show the model pairs that should be close (positives) and pairs that should be far (negatives), and adjust weights so:

• positives get pulled together (question ↔ its answer, sentence ↔ its paraphrase, query ↔ a relevant passage),
• negatives get pushed apart.

Do this over ~1 billion pairs and the vector space physically reorganizes: paraphrases end up pointing the same direction, unrelated texts point elsewhere. That's why cosine ends up meaning "semantic similarity" — not because of the architecture, but because training forced the geometry into that arrangement. This is the exact origin of the "direction encodes meaning" fact from the cosine discussion: contrastive training is what put the meaning in the direction.

In-batch negatives — the efficiency trick

Where do all those negative pairs come from? You don't collect them explicitly — you steal them from the batch. Concrete example: a batch of 64 (query, correct-passage) pairs.

query₁ ─ passage₁ ← the one true positive for query₁ query₂ ─ passage₂ ... query₆₄ ─ passage₆₄

For query₁, passage₁ is the positive, and the other 63 passages in the batch are almost certainly unrelated to it — so they're used as free negatives. Every example gives you 1 positive + 63 negatives at no extra data-collection cost. The loss says: "make query₁ closest to passage₁, farther from the other 63." Bigger batches → more negatives per step → sharper training signal. (This loss is MultipleNegativesRankingLoss in the library.)

"Siamese" — and the tie to reranking

"Siamese" just means both texts of a pair go through the same encoder (shared weights — twin towers), each producing its own vector independently, and you compare the two vectors afterward. That independence is exactly the bi-encoder property from the reranking conversation: you can encode all 5,000 documents once at index time and only encode the query at search time, then compare cheaply. That's what makes it indexable and fast.

Contrast with the cross-encoder reranker: it does not pool into per-text vectors — it feeds (query, doc) through the transformer together so every query token attends to every doc token, and reads out one score. More accurate, but no reusable vector, so it can't be indexed — hence it only reranks a shortlist. Sentence-transformers gives you the cheap wide first stage; the cross-encoder is the expensive precise second stage.

The whole chain, in one line

▎ Transformer → per-token vectors → pooling (masked mean) → one vector → contrastive training reshapes the space so cosine = meaning → L2-normalize so cosine ≡ dot → cheap, indexable bi-encoder semantic search.

And the sentence to hang onto — "the semantics live in the trained geometry, not the architecture" — means: the transformer + pooling just produce some vector; it's the contrastive training over ~1B pairs that arranges those vectors so nearness means similarity. Swap in the same architecture without that training and your embeddings are near-useless. The model isn't smart because of its shape; it's smart because of how its space was bent during training.

Is it the case that there will be at most 2 arms (one lexical, one semantic), or is their research/discussion to suggest that there could be enough value running multiple lexical and/or semantic arms?

Is it the case that semantic arms will almost always be dense? When would it make sense to use a sparse vector for semantic search?

Two good architecture questions. Short answers: no, not capped at 2, and no, semantic isn't always dense — and the two questions actually meet in the same place (learned-sparse). Let me take them in turn.

Q1 — Can you have more than two arms?

Yes, and it's a real technique, not a hack. RRF is N-way by construction — it just sums 1/(k + rank) over however many ranked lists you hand it; nothing about the math says "two." So "arms" is an open set. Where extra arms earn their keep:

Multiple lexical arms

• Same corpus, different analyzers: a stemmed/stopworded field and an exact/keyword field and an n-gram/shingle field for phrases. Each catches different matches (recall vs precision, morphology vs exactness).
• Field-weighted arms: title-BM25 vs body-BM25 vs anchor/tag-BM25 (though these often collapse into one multi_match rather than separate fused arms).
• Query-expansion arms: the raw query and an expanded one (synonyms, RM3 / pseudo-relevance feedback) fused together.

Multiple semantic arms

• Model ensemble: a general encoder + a domain-tuned encoder (two dense arms that make different mistakes).
• Multi-granularity: the same document encoded at sentence-, passage-, and doc-level, retrieved separately and fused — great when relevant evidence is a needle in a long doc.
• Query variants / HyDE: embed the query and a hypothetical answer, fuse.
• Different representation families: dense + learned-sparse + multi-vector (this is Q2).

The governing principle — and the brake. An arm is only worth adding if it's complementary, not redundant — it has to find relevant docs the others miss. Two near-duplicate BM25 variants add latency, an index to maintain, and a failure mode for ~zero lift. And there's a hard ceiling: arms raise recall (funnel width), but with diminishing returns as recall → 1.0. On SciFact hybrid is already at 0.954 recall — a third arm has almost no room; on a harder dataset there'd be headroom. So in practice 2–3 arms is the common sweet spot (lexical + dense, sometimes + learned-sparse), and you add a fourth only when you can measure the complementary lift on held-out qrels.

One more consequence at scale: parameter-free RRF is forgiving with 2–3 arms, but as arms proliferate the fusion weighting matters more (a weak arm dragging things down — you saw exactly this with Postgres's FTS poisoning its hybrid). Beyond a few arms people often move from RRF to learned fusion (learning-to-rank a small model over the arm scores/ranks) so each arm's contribution is weighted by evidence rather than treated equally. Same discipline as everything else here: each arm is a hypothesis you justify with a number.

Q2 — Is "semantic" always dense? When is sparse worth it?

The crucial unlock: "semantic" and "dense" are different axes. Representation (sparse vs dense) is one axis; whether it captures meaning (lexical vs semantic) is another. Four families:

┌─────────────────────────┬─────────────────────────────────┬───────────────────────────────────────────┐ │ │ Sparse (vocab-sized, mostly │ Dense (384/768-dim, all nonzero) │ │ │ zeros) │ │ ├─────────────────────────┼─────────────────────────────────┼───────────────────────────────────────────┤ │ Lexical (surface terms) │ BM25 — exact tokens, no meaning │ — │ ├─────────────────────────┼─────────────────────────────────┼───────────────────────────────────────────┤ │ Semantic (learned │ Learned-sparse: SPLADE, ELSER │ sentence-transformers, ColBERT │ │ meaning) │ │ (multi-vector) │ └─────────────────────────┴─────────────────────────────────┴───────────────────────────────────────────┘

The bottom-left cell is the answer to your question: learned-sparse retrieval is semantic search that uses sparse vectors. A model like SPLADE or Elastic's ELSER runs a transformer over the text and outputs a sparse, vocabulary-dimensioned vector with learned weights — and crucially it does term expansion: it adds related terms that weren't in the original text (a doc about "myocardial infarction" gets nonzero weight on "heart attack"). So it matches on meaning like dense does, but its representation lives in term space, not a 384-dim blob.

When learned-sparse makes sense over dense:

1. Lexical grounding — rare terms, entities, exact matches. Dense encoders wash out product codes, names, numbers, rare jargon (everything's blended into one vector). Learned-sparse keeps a foot in term space, so it nails exact/rare matches while still expanding semantically. A genuine middle ground between BM25 and dense.
2. Zero-shot / out-of-domain robustness. This is directly relevant here: MiniLM was weak zero-shot on scientific text precisely because a small general dense encoder has no lexical anchor to fall back on. Learned-sparse models have shown strong zero-shot generalization on BEIR because the term grounding transfers across domains. A SPLADE/ELSER arm might beat MiniLM on SciFact for exactly the reason MiniLM struggled.
3. Infrastructure reuse + cost. Sparse vectors ride the existing inverted index (Lucene/Elasticsearch) — no ANN/HNSW, no separate vector store. That's Elastic's whole ELSER pitch: semantic search on the inverted index you already run. Often cheaper to operate and it composes naturally with BM25.
4. Interpretability. A sparse vector is term-weighted — you can see which terms fired and why. Dense is a black box; when relevance goes wrong, sparse is far easier to debug.

When dense still wins: true paraphrase / low lexical overlap, cross-lingual (terms don't line up at all), short texts, multimodal (images must be dense — no vocabulary to be sparse over), and whenever you have a strong in-domain encoder where the geometry is well-shaped.

And this loops back to Q1: the strongest setups frequently fuse all three — BM25 (lexical-sparse) + SPLADE/ELSER (semantic-sparse) + dense — because they're genuinely complementary: exact terms, expanded terms, and paraphrase-level meaning are three different match types. That's a concrete, research-backed case where a third arm is justified.

For this project specifically: a learned-sparse arm (ELSER is already namechecked in the README as a "turnkey relevance" option) is a natural addition to the experiments backlog — and it attacks the weak-dense-arm finding from a different angle than "swap in a better dense encoder." Two distinct hypotheses worth measuring separately:

• Better dense encoder (e5/bge) — strengthen the existing semantic arm.
• Add learned-sparse (SPLADE/ELSER) — a semantic arm that keeps lexical grounding, which is exactly what MiniLM lacked on scientific text.