Vector search
Approximate nearest-neighbour search: HNSW and IVF, filtering, the recall and latency trade-off, and re-embedding without downtime.
Vector search is the building block behind semantic retrieval. This page gets you ready to choose exact or approximate nearest-neighbour search, tune recall against latency, handle filters, and migrate embeddings safely.
Read this if your last attempt…
- Your answer says “put it in a vector database” but cannot explain the index
- You are not sure when cosine, dot product and L2 give the same ranking
- You have seen HNSW, IVF, probe counts or hnsw.ef_search in docs but cannot tune them
- Your filtered search or re-embedding plan can silently lose results
The concept
What vector search answers
Vector search answers k-nearest-neighbour questions over embeddings: given a query vector, return the k stored vectors closest by a distance metric. Concrete stores live on Vector Database; the end-to-end RAG pipeline lives on Retrieval-augmented generation.
Exact kNN compares the query with every vector and is the recall baseline. ANN visits a smaller candidate set through an index, then you measure the misses against exact search.
Vector search approaches: choose by scale, memory, filtering and required recall.
| Approach | Best fit | Knobs | Main trade-off |
|---|---|---|---|
| Exact scan | Small corpora, offline evaluation, correctness baseline. | Parallelism, candidate subset, metric. | Perfect recall but latency grows with corpus size. |
| HNSW | Low-latency serving where high recall matters and memory is available. | m/M, ef_construction, hnsw.ef_search. | More recall usually costs memory, build time and query latency. |
| IVF / IVFFlat | Large corpora where probing selected cells is cheaper than graph memory. | lists/nlist, ivfflat.probes/nprobe, training data. | Fast when probes are low, but misses neighbours in unprobed cells. |
| IVF + PQ | Very large corpora where memory or bandwidth is the limit. | lists/nlist, probes/nprobe, PQ code size, rerank depth. | Compression saves memory but adds quantisation error. |
| Filtered ANN | Tenant, ACL, language or freshness filters that must still return k. | Partitioning, partial indexes, engine filter, scan depth. | Better correctness with more index design and operational complexity. |
How interviewers grade this
- You explain exact search as the recall baseline and ANN as the serving optimisation.
- You choose the distance metric based on the embedding model and normalisation.
- You can describe HNSW as a layered proximity graph and IVF as clustered inverted lists.
- You name the recall-latency knobs: HNSW
m/M,ef_construction,hnsw.ef_search; IVFlists/nlistandivfflat.probes/nprobe. - You handle filters inside retrieval instead of post-filtering a global top k.
- You measure recall@k against exact search before claiming the index is good.
- You migrate embedding models with a new index, dual-writes, backfill and cutover.
Variants
HNSW graph index
A layered proximity graph that trades memory and search breadth for high recall at low latency.
HNSW is usually the first ANN index candidates mention because the story is intuitive: a graph links near neighbours, upper layers make long hops, and lower layers refine. Use it when query latency and recall matter and you can afford memory.
The common tuning path is to raise hnsw.ef_search in pgvector, or efSearch in generic HNSW docs, until recall is acceptable. Then revisit m/M and ef_construction if graph quality or memory footprint is wrong. The trade-off is operational: HNSW can use much more memory than IVF-style indexes, and inserts update the graph.
Pros
- +Strong speed-recall trade-off for many serving workloads.
- +Intuitive knobs and good support across engines.
- +No coarse training step is required before the first insert.
Cons
- −Graph links can make memory the limiting factor.
- −Build and insert cost rise when you ask for higher quality.
- −Filtering still needs deliberate index or engine support.
Choose this variant when
- Semantic search, recommendations or retrieval where recall is important and the index can live in memory.
IVF / IVFFlat
Cluster the space into inverted lists and probe only the closest lists.
IVF trains centroids, assigns vectors to inverted lists and searches only the nearest lists. It is a good fit when you want a compact, trainable index and can reason about the recall lost when a true neighbour’s cell is not probed.
Start with representative training data so the centroids match production vectors, then tune ivfflat.probes in pgvector, or nprobe in FAISS, by measuring recall. A low probe count is fast but brittle; a high one raises recall but scans more of the corpus. If the data distribution drifts, retrain or rebuild the index.
Pros
- +Less graph memory than HNSW.
- +The probe-count knob gives a clear latency-recall curve.
- +Works well for bulk-built, large indexes with stable distributions.
Cons
- −Needs representative training data.
- −Can miss neighbours when the right cell is not probed.
- −Low probes can hurt small or skewed filters.
Choose this variant when
- Large, mostly batch-built corpora where memory matters and you can train and tune the coarse quantizer.
IVF with product quantisation
Compress vectors into short codes so a much larger index can stay in memory.
Product quantisation compresses vectors into short codes. In IVF+PQ, the coarse IVF stage chooses candidate lists, then PQ codes reduce memory and bandwidth while approximate distances are computed.
It is most useful when the dataset is so large that raw float vectors or graph links do not fit the memory budget. The cost is accuracy: PQ adds quantisation error, so you often retrieve a larger candidate set and rerank with the original vectors if you have them.
Pros
- +Cuts memory and bandwidth for very large collections.
- +Combines naturally with IVF candidate generation.
- +Can be followed by reranking for quality.
Cons
- −Quantisation error can lower recall.
- −More training and parameter choices.
- −Debugging relevance is harder than with raw vectors.
Choose this variant when
- The corpus is large enough that raw vectors or graph indexes are too expensive to keep hot.
Filtered ANN
Make metadata predicates part of candidate generation instead of a late clean-up step.
Filtered ANN means the metadata predicate participates in candidate generation rather than being applied only after top k. This matters for tenants, permissions, region, language, freshness and delete state.
If the filter is highly selective, the engine may need a partition, partial index or exact fallback to keep recall high. Many small partitions can improve isolation but increase operational overhead.
Pros
- +Preserves result count under tenant, ACL and language filters.
- +Avoids showing disallowed candidates to rerankers or models.
- +Can improve latency by searching the right subset.
Cons
- −May require partitions, namespaces or engine-specific filter support.
- −Very selective filters can still need exact fallback.
- −Many tiny partitions raise operational overhead.
Choose this variant when
- Enterprise retrieval, multi-tenant search and any system where permission filters are part of correctness.
Dual-index re-embedding migration
Build a new index for the new model, compare it, then cut over traffic.
A re-embedding migration treats embeddings like a derived index rebuild. New writes are embedded with both old and new models, old rows are backfilled into a new index, and a query replay compares quality before cutover.
The old index stays available until rollback is no longer needed. You pay for two embeddings, two indexes and backfill capacity during the migration, but that cost is safer than changing the model in place.
Pros
- +Avoids mixing incompatible vector spaces.
- +Gives rollback while quality is measured.
- +Lets you backfill without stopping writes.
Cons
- −Temporary double write and storage cost.
- −Backfills can compete with serving traffic.
- −Requires query replay and quality gates.
Choose this variant when
- You change embedding model, dimensions, normalisation, chunking or distance metric.
Worked example
Numbers in this section are illustrative.
Scenario: design semantic search for a product-help surface. Numbers in this section are illustrative.
Corpus. Suppose you have 12 million help chunks, each 768-dimensional. Queries return top 20 chunks after tenant, product and language filters. The team wants p95 below 120 ms and high known-answer recall.
Metric and baseline. Choose the metric from the embedding model. If vectors are unit-normalised, cosine, dot product and L2 rank the same neighbours. Build an exact benchmark, for example 5,000 queries across tenants and languages, and use brute force as the truth set for recall@20.
Index choice. Start with HNSW if recall and latency matter more than memory. In pgvector, tune m and hnsw.ef_search; generic docs say M and efSearch. If memory is tight, try IVF: choose pgvector lists or FAISS nlist, then raise ivfflat.probes or nprobe until recall meets the target.
Filters. Tenant and ACL filters are not optional. If you post-filter top 20 across all tenants, a restrictive tenant filter may leave two usable results. Put tenant in the index layout, namespace, partition or engine-level filtered search.
Decision. Run the curve. One illustrative result might be: HNSW at low breadth gives 88% recall@20 with 35 ms p95; higher breadth gives 96% recall@20 with 80 ms p95; IVF with low probes gives 84% recall@20 with 25 ms p95. Pick the smallest setting that meets product quality.
Migration. For a new embedding model, create a second index, dual-write new chunks, backfill old chunks, compare recall and click metrics, then switch query traffic. Do not mix vectors from both models in one index.
Good vs bad answer
Numbers in this section are illustrative.
Interviewer probe
“A support-search corpus has tens of millions of vectors, tenant filters, and a planned new embedding model. How do you design vector search?”
Weak answer
“I would use a vector database with cosine similarity and ask for the top 10. If it is slow, I would add an index and maybe cache the results. For filters I would just filter the returned rows.”
Strong answer
“I would separate four decisions. First, metric: use the embedding model’s recommended metric, and if vectors are unit-normalised then cosine, dot product and L2 have the same ranking. Second, exact versus approximate: keep exact search as the offline baseline, but serve with ANN once the corpus is large.
For ANN, I would start with HNSW when recall and latency matter more than memory. I would tune pgvector m, ef_construction and hnsw.ef_search (generic HNSW: M, efConstruction, efSearch). If memory becomes the limit, I would test IVF or IVFPQ: lists/nlist controls cells, ivfflat.probes/nprobe controls scanned cells, and PQ reduces memory with some accuracy cost.
For filters, I would not post-filter a global top k because restrictive tenant or ACL filters can return too few results. I would use filtered search, partitions or namespaces, then measure recall@k against exact search under realistic filters.
If we change embedding models, I would create a new index, dual-write, backfill, compare retrieval quality, then cut over. I would not mix old-model and new-model vectors because distance only has meaning within one embedding space.”
Why it wins: The strong answer names the metric, the exact baseline, HNSW and IVF knobs, filtered-search failure mode, recall@k measurement and the dual-index migration plan. It treats vector search as a tunable retrieval system rather than a product checkbox.
When it comes up
- The system needs semantic search, similar-item lookup, deduplication or retrieval candidates.
- The interviewer asks what a vector database is doing under the hood.
- A design has tenant, ACL, language or freshness filters on vector search.
- The corpus grows beyond exact search latency, or a new embedding model is planned.
Order of reveal
- 11. Define the query. This is k-nearest-neighbour search over embeddings. Exact search scans every vector; ANN serves faster by trading some recall.
- 22. Pick metric and model contract. Documents and queries use the same embedding model. I choose cosine, dot or L2 based on the model, and normalisation can make their ranking equivalent.
- 33. Choose the ANN family. HNSW is a layered graph tuned by graph degree and search breadth. IVF clusters into lists and tunes how many lists to scan. PQ is the memory lever.
- 44. Put filters in retrieval. Tenant and ACL filters cannot be a final post-filter over a global top k. They need index-level filtering, partitions or namespaces.
- 55. Measure and migrate. I measure recall@k against exact search and p95 latency, and I migrate embedding models with dual indexes and cutover.
Signature phrases
- “Exact search is my oracle; ANN is my serving compromise.” — Separates correctness measurement from production latency.
- “HNSW tunes `m`, `ef_construction` and `hnsw.ef_search`; IVF tunes `lists` and `ivfflat.probes` in pgvector.” — Names the index families at useful interview depth.
- “Post-filtering can under-fill k.” — Catches a common filtered-search bug in one line.
- “A new embedding model means a new vector space and a new index.” — Shows safe migration thinking.
Likely follow-ups
?“How do you handle metadata filters?”Reveal
Name the three placements. Pre-filtering narrows the corpus first, which is accurate but can be slow if it becomes exact search over a large subset. Post-filtering runs ANN first and drops non-matching results, which is fast but can return fewer than k. Filtered ANN applies the predicate inside the index traversal or index layout, which is the usual production answer for tenants and ACLs.
?“How do you know the ANN index is good enough?”Reveal
Run exact search offline for a labelled or representative query set, then compute recall@k for the ANN results. For each query, recall@k is the share of exact top-k ids that appear in the approximate top k. Raise hnsw.ef_search or ivfflat.probes (FAISS nprobe) until recall meets the target, and record the p95 latency cost under realistic filters.
?“Cosine, dot product or L2?”Reveal
Use the metric the embedding model was trained or documented for. If vectors are unit-normalised, cosine similarity, dot product and L2 distance produce the same ranking, so implementation speed and index support can decide. If they are not normalised, cosine ignores magnitude while dot product uses it, so changing metrics can change relevance.
?“How do you change embedding models safely?”Reveal
Create a new index for the new model. Dual-write new and changed documents, backfill old documents, replay queries against both indexes, then cut over traffic gradually. Keep the old index for rollback. Do not mix model versions in one index because distances across embedding spaces are not meaningful.
Code examples
-- Illustrative pgvector knobs for an HNSW experiment.
CREATE INDEX help_chunks_embedding_hnsw
ON help_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
BEGIN;
SET LOCAL hnsw.ef_search = 100;
SELECT chunk_id
FROM help_chunks
WHERE tenant_id = 't_123' AND language = 'en'
ORDER BY embedding <=> $1
LIMIT 20;
COMMIT;def recall_at_k(exact_ids: list[str], ann_ids: list[str], k: int) -> float:
exact_top = set(exact_ids[:k])
ann_top = set(ann_ids[:k])
if not exact_top:
return 1.0
return len(exact_top & ann_top) / len(exact_top)
# Run this over a fixed query set and plot recall against p95 latency.Common mistakes
Approximate indexes can miss true neighbours. Keep exact search as an offline baseline and report recall@k for the settings you serve with.
If the filter is restrictive, most of the global top k can be ineligible. Put tenant, ACL and freshness filters into the vector search or index layout.
A new model defines a new coordinate system. Old and new vectors may have the same dimension but distances between them are not meaningful. Use a new index, backfill and cut over.
Cosine, dot product and L2 can rank the same only when vectors are normalised. If magnitude carries meaning, dot product and cosine are different choices.
Raising search breadth or probe count usually improves recall and costs latency. Without an exact baseline and p95 latency curve, knob changes are anecdotes.
A high-recall HNSW graph can be fast but memory-heavy. IVF and PQ exist because raw vectors and graph edges can dominate cost.
Replacing vectors in the live index gives mixed quality and no clean rollback. Dual-write to a new index, replay queries, then switch traffic.
Practice drills
Numbers in this section are illustrative.
Why keep exact search around if you serve with HNSW?Reveal
Exact search is the measurement oracle. You run it offline on a sample or labelled query set, then compare the ANN top k against it to compute recall@k. Without that baseline, you cannot tell whether a faster setting is still good enough.
What happens if you post-filter top 10 results by tenant?Reveal
You may return fewer than 10 results because many global nearest neighbours can belong to other tenants. The fix is to search within the authorised subset through filtered ANN, partitions, namespaces or an exact fallback for small subsets.
How do HNSW and IVF differ in one sentence?Reveal
HNSW walks a layered neighbour graph toward closer vectors; IVF clusters vectors into inverted lists and scans only the nearest lists for the query. HNSW usually spends memory on graph links, while IVF spends training and probing effort on cells.
Why can’t you mix vectors from two embedding models if they have the same dimension?Reveal
The dimension count only says how many coordinates exist, not what each coordinate means. A new model creates a different vector space, so distance between old-model and new-model vectors is not a valid similarity signal. Re-embed into a new index and cut over.
Deep dives
Recall benchmarking playbook
Recall testing is where vector search becomes engineering instead of folklore. Build a query set that covers the real workload: short questions, long questions, rare product names, different tenants, different languages and recent documents. If the product has judgements or click data, include those, but keep exact nearest-neighbour search as the mechanical baseline for the index itself.
For each query, run exact search over the same candidate universe the serving query should be allowed to see. Then run the ANN index with one parameter setting. Compute recall@k, p50, p95 and result count after filters. Repeat for several search-breadth or probe-count values. The graph you want is recall on one axis and latency on the other.
Do not benchmark only unfiltered queries if production uses tenant, ACL or language filters. A setting can look excellent globally and fail for a small tenant because post-filtering drops most candidates. Keep a separate slice for restrictive filters and zero-result queries.
The benchmark should become a regression suite. Run it when you change the embedding model, chunking, index type, filter layout or reranker. If a new model improves average similarity but loses rare identifiers or policy documents, the recall slices should show it before users do.
Filtered search layouts
Filtered search is hard because the predicate and vector score compete for control of the candidate set. If you filter first, you may end up with a small authorised corpus and exact search can be acceptable. If the authorised corpus is still large, exact pre-filtering becomes slow. If you search first and filter later, you may miss k because the best global neighbours belong to the wrong tenant or language.
A good serving design chooses a layout. For hard isolation, put tenants into separate indexes, namespaces, partitions or partial indexes when the tenant size justifies it. For a shared corpus, use an engine that supports filtered ANN during traversal. For very selective filters, keep an exact fallback or ask the engine to scan further until enough candidates are found.
This is also a security boundary in RAG and enterprise search. If the model or reranker sees private text before a late filter removes it, the answer can still leak information. That is why permissions belong in retrieval, not in final rendering.
Cheat sheet
- •Vector search = k nearest neighbours in one embedding space.
- •Exact scan is the recall oracle; ANN is the serving trade-off.
- •Cosine, dot and L2 rank the same for unit-normalised vectors.
- •HNSW: graph; tune
m/M,ef_constructionandhnsw.ef_search. - •IVF: cells and inverted lists; tune
lists/nlistandivfflat.probes/nprobe. - •PQ compresses vectors when memory is the limit.
- •Post-filtering can return fewer than k; filtered ANN or partitioning fixes it.
- •Measure recall@k against exact search plus p95 latency under real filters.
- •New embedding model means dual-write, backfill, compare and cut over to a new index.
Practice this skill
These problems exercise Vector search. Try one now to apply what you just learned.
Read this if