[yiguolei]

Background

ANN indexes identify vectors by their row IDs in a segment. Empty arrays have no index entry, so omitting them requires preserving those IDs and excluding missing vectors from range-search complements. Distance expressions also run without an ANN index, including when index training is skipped or a query falls back to a scan.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Enhancement

Problem Summary:

Allow empty arrays in ANN-indexed columns through an explicit, default-off BE option while preserving segment row IDs. Unify array-distance semantics: NULL and empty vectors produce NULL distances, whether they come from data rows or query literals. Both indexed and scalar query paths handle missing vectors consistently; most of the additional changes cover tests and nullable expression results.

Details

• Change map: 634 changed production lines and 1297 changed test lines; no generated files or documentation changes. Production changes cover ANN writing/persistence and row IDs, range-search domains, Group Commit validation, and FE/BE nullable distance evaluation. Tests cover storage, expression evaluation, and SQL query paths.
• Before and after: With enable_ann_index_empty_array=false, empty ANN arrays still fail writes. With it enabled, empty rows remain in the data file and retain their positions while nonempty vectors are indexed under their data-file row IDs. Range comparisons exclude empty vectors in both directions. NULL or empty query vectors return NULL distances through scalar evaluation instead of entering FAISS.
• Configs and defaults: New mutable BE config enable_ann_index_empty_array=false, observed during write validation. After persisting empty arrays, keep the flag enabled across restarts, compaction, index rebuilds and WAL replay. Readers do not consult the flag; toggling it does not convert stored data.
• Impact on existing users: Distance NULL semantics apply independently of the write flag. Dense indexes retain their native representation, and dense IVF complement searches avoid collecting all inverted-list IDs. The writer retains its existing treatment of historical NULL coordinates. This PR does not enable ANN indexes on nullable columns.
• Behavior change: Array overloads of l1_distance, l2_distance, inner_product, cosine_distance, cosine_similarity, l2_distance_approximate and inner_product_approximate return nullable FLOAT. Either whole-array NULL or an empty array yields NULL, including [] versus [] and explicit literals. NULL elements in non-NULL arrays and mismatched nonempty dimensions remain errors. Nonempty zero-vector behavior and the MAP overload of inner_product are unchanged.
• Compatibility: Plan a coordinated FE/BE upgrade and restart. This change intentionally does not preserve the old FE non-nullable result layout and does not claim rolling mixed-version query compatibility. Enable empty-array writes only after all FEs and BEs run the new version. After those writes, downgrading to an old BE is unsupported: deserializing a native FAISS file does not make old complement, HNSW range-search, untrained-IVF and rewrite paths compatible.
• Index implementation: IVF stores explicit row IDs natively; HNSW uses IDMap for sparse segments. All-empty segments persist an empty index without training. Sparse complement domains are reconstructed lazily from FAISS IDs, including IDs-only reads for IVF_ON_DISK. HNSW range queries translate selectors and output labels around the child index to retain derived search parameters in the pinned FAISS version. FAISS versions and index filenames are unchanged.
• Query execution: Sparse-segment TopN and NULL/empty query vectors use normal SQL evaluation and sorting to preserve filters, NULL ordering and LIMIT. This patch does not implement the separate optimization that folds a missing query vector to a constant NULL and avoids reading the vector column entirely.

Testing

Passed locally on macOS arm64:

• Repository clang-format v16 fix/check scripts on a scoped snapshot of all 25 changed C++ files.
• bash build-support/check-build-hygiene.sh.
• git diff --cached --check.
• Passed Checkstyle 9.3 (the version pinned by fe/pom.xml) on all 11 changed Java files, using the repository configuration and standalone JDK 21. This is a style check, not a JDK 17 compilation or test run.

Added or extended tests cover:

• Writer opt-in rejection without partial buffer mutation, cross-batch labels, all-empty segments and legacy NULL-coordinate writes.
• HNSW/IVF/IVF_ON_DISK native reload, filtering, both range directions, concurrent row-ID initialization, IO retry, and the dense-IVF fast path.
• Reader-level all-empty result contracts for both metrics and all index types.
• Every array-distance function with nullable/non-nullable inputs, data/literal NULLs and empty arrays on either side, constant columns, NULL elements, dimension errors and FE nullable declarations/constant folding.
• Missing-query extraction and SegmentIterator TopN fallback before accessing ANN readers, with candidate rows preserved.
• Group Commit off/sync/async cases and differential indexed/scalar SQL checks, including missing query vectors, cached results, NULL ordering and LIMIT.

Attempted this revision but blocked before execution:

• run-be-ut.sh --run for the ANN writer/reader/search, Group Commit, distance, extraction and SegmentIterator suites.
• run-fe-ut.sh --run for ArrayDistanceTest, InnerProductTest and ArrayArithmeticTest.
• run-regression-test.sh --run -d ann_index_p0 -s ann_empty_array_queries,ann_delete_group_commit,cast_string_as_array -forceGenOut.
• run-regression-test.sh --run -d query_p0/sql_functions/array_functions -s test_array_distance_functions -forceGenOut.

The scripts stop at the JDK 17 prerequisite; the available standalone JDK is 21 and installed BE third-party dependencies are also absent. Compilation, UT and regression results are not verified. Clang-tidy is pending a successful BE build and compilation database. Changed regression .out files have not been regenerated; existing snapshots must be refreshed through the runner before this PR is ready. Recovery/compaction and actual old/new-binary compatibility scenarios also remain unverified. No expected results were handwritten.

Notes for Reviewers

Source self-review checkpoints:

Checkpoint	Conclusion
Goal, scope and parallel paths	Writer, indexed/scalar distances, query extraction and FE constant folding follow the documented semantics; runtime proof remains pending.
Concurrency and lifecycle	Row-domain initialization holds a mutex and publishes only a completed bitmap. RAII owns list borrows and translated selectors; IDMap owns its child. Sparse first-use IO still runs under the initialization lock.
Config and compatibility	Write opt-in is dynamic; coordinated FE/BE restart and the downgrade restriction are explicit.
Special conditions	All-empty indexes obey the reader pointer contract; missing queries fall back before dimension checks/FAISS; nonempty invalid dimensions remain errors.
Persistence, transactions and protocol	Native FAISS persistence retains row IDs. Group Commit validates before accepting blocks into its background path; transaction protocols and Thrift/Proto are unchanged. Crash/WAL behavior needs runtime verification.
Performance and observability	Dense IVF avoids whole-index label collection. Sparse first-use bitmaps and scalar TopN add cost; existing ANN fallback counters/timers and contextual writer logs cover these paths. No benchmark claim.
Tests and outputs	Tests are added, but have not run; result regeneration and recovery/upgrade tests are outstanding.

Release note

Add default-off enable_ann_index_empty_array support for ANN writes. Array distance and similarity functions return NULL for NULL or empty vectors, including query literals. Missing-query and sparse TopN paths preserve SQL NULL ordering through scalar evaluation. Upgrade and restart all FEs and BEs together; keep the write flag enabled after storing empty arrays, and do not downgrade those data files to an unsupported BE version.

Check List (For Author)

• Behavior changed: Yes (see Details).
• Does this need documentation: Yes; document the write flag, NULL semantics, coordinated upgrade and downgrade restriction in the ANN and distance-function documentation.
• Pre-submission status: Not ready: draft pending compilation, runtime tests, generated regression outputs, recovery and compatibility validation.

AI Disclosure

• Tool: OpenAI Codex (GPT-6).
• Generated: Code, tests and this description.
• Human review: The author specified the final NULL/empty semantics and coordinated-upgrade approach; final implementation review is pending.