Add prefetch for vector code

[markwwen]

Summary

This PR adds an optional DistanceComputer::prefetch(idx_t) hook and uses it in HNSW search to prefetch vector codes before distance computation.

The default implementation is a no-op, so existing distance computers are unaffected. NegativeDistanceComputer forwards the call to the wrapped distance computer, and FlatCodesDistanceComputer prefetches the first few cache lines of the target code.

HNSW search now prefetches the next neighbor's vector code while iterating over neighbor candidates. This is intended to reduce memory access latency for flat-code-backed HNSW indexes without changing search results.

On sift-128-euclidean, this improves HNSW search QPS by 10% to 15% with no recall change.

[meta-cla]

Hi @markwwen!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[meta-codesync]

@mnorris11 has imported this pull request. If you are a Meta employee, you can view this in D111459405.

[mnorris11]

Thanks for the contribution, will take a look

cc @alexanderguzhva who added the prefetch parts just in case you are interested

[mnorris11]

I wanted to ensure there is no regression on existing flows. I got a slight regression on GIST when testing just the 3 datasets below. This issued 1 query at a time in search() with kHNSWCodePrefetchAhead = 1. We are working on some improved testing, and can take a look again after it is in place.

  SIFT1M — HNSW32 (d=128, L2) — single build per side

  ┌──────────┬────────────────────┬───────────┬───────────┬────────┬──────────┬──────────┬────────┐
  │ efSearch │ recall (base/with) │ kcyc base │ kcyc with │ Δ kcyc │ QPS base │ QPS with │ Δ QPS  │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 16       │ 0.662 / 0.661      │ 408.8     │ 396.9     │ −2.93% │ 7007     │ 7257     │ +3.58% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 32       │ 0.805 / 0.804      │ 607.8     │ 597.0     │ −1.78% │ 4722     │ 4833     │ +2.35% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 64       │ 0.912 / 0.912      │ 975.8     │ 962.7     │ −1.35% │ 2821     │ 2993     │ +6.07% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 128      │ 0.970 / 0.970      │ 1689.8    │ 1682.9    │ −0.41% │ 1686     │ 1699     │ +0.74% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 256      │ 0.992 / 0.992      │ 3044.3    │ 3046.1    │ +0.06% │ 939      │ 935      │ −0.35% │
  └──────────┴────────────────────┴───────────┴───────────┴────────┴──────────┴──────────┴────────┘

  → Win at low–mid efSearch, neutral at high.

  dbpedia-openai — HNSW32 (d=3072, IP) — single build per side

  ┌──────────┬────────────────────┬───────────┬───────────┬────────┬──────────┬──────────┬────────┐
  │ efSearch │ recall (base/with) │ kcyc base │ kcyc with │ Δ kcyc │ QPS base │ QPS with │ Δ QPS  │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 16       │ 0.841 / 0.840      │ 1349.2    │ 1338.9    │ −0.77% │ 2116     │ 2135     │ +0.92% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 32       │ 0.899 / 0.897      │ 2062.3    │ 2037.1    │ −1.22% │ 1387     │ 1389     │ +0.16% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 64       │ 0.935 / 0.936      │ 3345.3    │ 3302.4    │ −1.28% │ 852      │ 854      │ +0.26% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 128      │ 0.958 / 0.958      │ 5892.3    │ 5737.7    │ −2.62% │ 482      │ 491      │ +1.99% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 256      │ 0.972 / 0.972      │ 10635.9   │ 10359.5   │ −2.60% │ 270      │ 275      │ +2.21% │
  └──────────┴────────────────────┴───────────┴───────────┴────────┴──────────┴──────────┴────────┘

  GIST1M — HNSW32 (d=960, L2) — mean of 4 builds per side

  ┌──────────┬────────────────────┬───────────┬───────────┬────────┬──────────┬──────────┬────────┐
  │ efSearch │ recall (base/with) │ kcyc base │ kcyc with │ Δ kcyc │ QPS base │ QPS with │ Δ QPS  │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 16       │ 0.425 / 0.426      │ 1116.8    │ 1121.7    │ +0.44% │ 2535     │ 2528     │ −0.28% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 32       │ 0.563 / 0.563      │ 1727.9    │ 1757.6    │ +1.72% │ 1637     │ 1620     │ −1.05% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 64       │ 0.703 / 0.704      │ 2891.6    │ 2918.8    │ +0.94% │ 985      │ 979      │ −0.64% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 128      │ 0.824 / 0.825      │ 5072.9    │ 5102.4    │ +0.58% │ 560      │ 559      │ −0.13% │
  ├──────────┼────────────────────┼───────────┼───────────┼────────┼──────────┼──────────┼────────┤
  │ 256      │ 0.911 / 0.910      │ 9199.4    │ 9267.9    │ +0.74% │ 310      │ 309      │ −0.28% │
  └──────────┴────────────────────┴───────────┴───────────┴────────┴──────────┴──────────┴────────┘

  → Small, consistent regression (5/5 efSearch points worse; ~+1% kcycles / ~−0.5% QPS).

[markwwen]

Thanks for running these tests and for checking the existing flows. I reran SIFT1M after rebasing the change onto the current main. I also noticed that your benchmark issued one query per search() call, so I repeated the experiment with the same calling pattern rather than passing all 10,000 queries in one call.

Configuration

• Dataset: SIFT1M, 1M database vectors and 10,000 queries, d=128, L2
• Index: IndexHNSWFlat, M=32, efConstruction=40, bounded queue, k=10
• Base commit: 22220d2c4
• Prefetch commit: 400292b4e
• Compiler: GCC 13.3, Release (-O3 -DNDEBUG), AVX-512
• CPU: Intel Xeon Platinum 8480C (Sapphire Rapids)
• Search: one query per search() call, one search thread pinned to CPU 24
• Prefetch distance: kHNSWCodePrefetchAhead = 1
• NUMA: the process was pinned before loading the index so that index pages were allocated on the same NUMA node as the search thread
• Construction: one index build per side, using 32 threads pinned to NUMA node 0 and the same construction settings
• Measurement: one full warm-up pass per efSearch, followed by three measured passes; the table reports the median
• kcyc: serialized TSC ticks per query measured with rdtsc; QPS uses wall time
• Recall below: top-10 overlap with the exact top-10 ground truth

Results

efSearch	recall (base/with)	kcyc base	kcyc with	$\Delta$ kcyc	QPS base	QPS with	$\Delta$ QPS
16	0.86216 / 0.86242	277.4	239.8	-13.55%	7210	8340	+15.67%
32	0.93731 / 0.93751	460.3	394.9	-14.21%	4345	5065	+16.57%
64	0.97818 / 0.97782	809.8	701.3	-13.40%	2470	2852	+15.47%
128	0.99318 / 0.99285	1467.0	1272.4	-13.26%	1363	1572	+15.29%
256	0.99796 / 0.99790	2681.2	2374.1	-11.45%	746	842	+12.93%

As a control for differences caused by the two parallel index builds, I also loaded the base-built index with both binaries. This made recall identical and still reduced TSC ticks per query by 12.0% to 14.6%, depending on efSearch. I also tested passing all 10,000 queries in one search() call; the improvement was similar, so query call granularity does not explain the difference on this machine.

These results are noticeably different from yours, both in speedup and in absolute recall. That suggests there is still an environment or benchmark configuration difference between the runs. In particular, CPU model, compiler/SIMD target, efConstruction, Faiss revision, and the exact recall calculation could all matter here. Could you share the CPU model and the benchmark command or code used for your table? I can then reproduce that setup more closely and also run the same controlled comparison on GIST.