Fix IndexIVFPQ losing polysemous_ht/do_polysemous_training on I/O round-trip

[Juanpacol]

Summary

IndexIVFPQ never serialized its polysemous_ht and do_polysemous_training fields. The write_index() function wrote only the fields common to the IVFPQ family (by_residual, code_size, the ProductQuantizer, and the inverted lists) under the "IwPQ"/"IwQR" fourcc, so read_index() had no data to restore these two members. They silently reverted to their constructor defaults (do_polysemous_training = false, polysemous_ht = 0) on every load.

A caller who trained polysemous codes and tuned a Hamming threshold via ParameterSpace (nprobe=10,ht=32) would get back an index that searches as if polysemous filtering were never configured, with no error or warning to indicate the parameters were dropped.

Reproduction (Issue #2120)

import faiss
n, d, nlist = 10000, 64, 100
X = faiss.randn((n, d))
index = faiss.index_factory(d, f"IVF{nlist},PQ8")
index.train(X)
index.add(X)

params = faiss.ParameterSpace()
params.set_index_parameters(index, "nprobe=10,ht=32")
faiss.write_index(index, "testing.index")

index2 = faiss.read_index("testing.index")
assert index.polysemous_ht == index2.polysemous_ht  # FAILED: 32 != 0

Solution

Introduced new fourcc tags:

• "IwPh" for IndexIVFPQ (replacing "IwPQ")
• "IwQh" for IndexIVFPQR (replacing "IwQR")

These new tags additionally write do_polysemous_training and polysemous_ht right after the ProductQuantizer, before the inverted lists.

The read_ivfpq() function recognizes the new tags and reads the two extra fields only when present. Existing "IwPQ"/"IwQR"/"IvPQ"/"IvQR" files continue to load exactly as before (fields default to false/0, same as pre-fix behavior), preserving full backward and forward binary compatibility with indexes already persisted to disk.

Testing & Verification

• C++ suite: 284/285 tests pass (1 unrelated SIMD-hardware skip)
• Python suite (tests/test_io.py, tests/test_index.py): 79 passed
• New regression test: TestIORoundTrip.test_index_ivfpq_polysemous_ht() validates that polysemous parameters survive a serialize/deserialize cycle
• Backward compatibility: Legacy-format regression test TestIVFPQRead.test_reader confirms old on-disk indexes deserialize correctly
• Build: Verified against libfaiss built from source (CMake + SWIG, CPU-only, Accelerate BLAS/LAPACK)

Closes #2120

[meta-cla]

Hi @Juanpacol!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!