perf: use compact pruning for large string IN lists

[sunchao]

Why are the changes needed?

Which issue does this PR close?

Related to #8668 and #8609; follows #24074. This PR is stacked on #24525, which must merge first.

Rationale for this change

Queries often select a sparse set of string identifiers. Parquet min/max statistics can make these queries much cheaper by ruling out row groups or pages that cannot contain any requested identifier. For example, this query asks for 21 IDs, spaced ten apart:

SET datafusion.execution.parquet.max_in_list_size = 1024;

SELECT *
FROM events
WHERE customer_id IN (
  'id000', 'id010', 'id020', 'id030', 'id040', 'id050', 'id060',
  'id070', 'id080', 'id090', 'id100', 'id110', 'id120', 'id130',
  'id140', 'id150', 'id160', 'id170', 'id180', 'id190', 'id200'
);

A row group whose values fall between id003 and id007 cannot contain a match. The default pruning limit is 20, so this list is not eligible for the IN min/max rewrite unless the caller raises the limit. #24074 made that limit configurable. With a raised limit, DataFusion can already reject this row group, but it does so by constructing a growing expression tree resembling:

(min <= id000 AND id000 <= max)
OR (min <= id010 AND id010 <= max)
OR ...

For hundreds or thousands of identifiers, building and evaluating that tree can become expensive in its own right. Replacing the list with one enclosing range, [id000, id200], would be cheaper, but would lose the gaps: that broad range overlaps [id003, id007] even though none of the requested IDs is present there.

The aim is to keep the useful pruning precision of the existing per-value checks while making large lists cheaper to represent and evaluate. In the included local microbenchmark, evaluating 1,024 values against 4,096 intervals falls from 68.8 ms to 0.198 ms. This measures pruning work only, not end-to-end query speedup.

What changes were proposed in this PR?

What changes are included in this PR?

Eligible large string lists are stored as a sorted, deduplicated set of values inside the pruning predicate, rather than expanded into one comparison branch per value. For each inclusive statistics interval, DataFusion finds the first requested value at or after the interval's minimum, then checks whether that value is also at or before its maximum.

In the example, the first requested ID at or after id003 is id010. Since id010 > id007, the row group can be skipped. An interval such as [id019, id021] must be kept because it could contain id020. This takes a binary search per interval after sorting the values once per constructed predicate, and the expression tree no longer grows with the number of IDs.

The result remains a conservative pruning decision. An overlapping interval means only that a match is possible; the original IN expression still performs exact row filtering. The original literal information also remains available to other pruning mechanisms, including Bloom filters. Missing or inverted bounds cannot prove that data is safe to skip.

The existing limit continues to control eligibility. Its default remains 20, setting it to zero disables the IN min/max rewrite, and lists beyond the configured cap remain ineligible. Only eligible positive, non-null literal string lists larger than 20 take the compact path. NOT IN, NULL-containing lists, and unsupported expressions keep their existing handling. Page-index pruning now receives the same configured cap as row-group pruning, so raising the limit can benefit both.

The dependency on #24525 matters for correctness: an interval search is only meaningful when the stored bounds use the same comparison order as the query. That companion PR handles legacy or unrecognized Parquet byte-array ordering. It must land before the newly enabled large-list page-pruning path here.

Are there any user-facing changes?

Users who raise datafusion.execution.parquet.max_in_list_size get cheaper min/max pruning for eligible large string lists, and page pruning now honors that setting. The configuration default, exact query results, and existing public APIs are unchanged. This is a focused optimization for literal string lists, not a general rewrite of every large IN expression.

How was this PR tested?

Are these changes tested?

The pruning-crate suite passed 93 tests. The standalone Parquet regressions use lists of 20, 21, 256, and 1,024 values and check both exact query results and scan/pruning metrics. They cover gaps inside the list's enclosing range, row-group pruning, page-only pruning, and the default and zero-cap controls.

Two correctness regressions exercise the less obvious interactions. A direct physical-source test uses NOT IN (..., NULL), row-filter pushdown, and LIMIT 1, so logical optimizer folding cannot hide an incorrect decision to bypass filtering. A real-file test combines this PR with #24525 and verifies that compact page pruning cannot lose a matching row when the footer's ordering is missing or unknown. Against unchanged Apache f1f0449a, the positive row-group/page tests fail as expected; the NOT IN (..., NULL) control passes.

The benchmark compares the actual raised-cap IN path on Apache f1f0449a and this patch, using separate build directories and checking that both return the same nontrivial pruning results. Local results on an Apple M5 Max (18 CPUs, 128 GiB), Rust 1.97.0, release-nonlto, 20 samples:

Values	Predicate construction, main → PR	Evaluate 4,096 intervals, main → PR
20	31.1 → 27.4 µs	177.6 → 165.7 µs
21	30.5 → 3.52 µs	183.8 → 98.1 µs
256	356.6 → 21.4 µs	5.53 → 0.144 ms
1,024	1.466 → 0.081 ms	68.8 → 0.198 ms

A balanced explicit OR tree is included as another comparison: at 1,024 values it takes 8.03 ms to evaluate the same intervals. These measurements isolate pruning overhead; no end-to-end workload improvement is claimed.

Formatting, all-targets/all-features Clippy with warnings denied, and ./dev/rust_lint.sh passed on the combined stack. The extended workspace run passed 10,674 Rust tests, with eight ignored, and all 503 SQL-logic files.

Validation commands

cargo test --locked --profile ci -p datafusion-pruning

cargo test --locked --profile ci -p datafusion \
  --test parquet_integration string_in_list_pruning

cargo bench --locked --profile release-nonlto -p datafusion-pruning \
  --bench string_in_list_pruning -- \
  --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot

RUST_BACKTRACE=1 cargo test --locked --profile ci \
  --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \
  --workspace --lib --tests --bins \
  --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.51064% with 54 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.32%. Comparing base (f1f0449) to head (b20b0e7).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/pruning/src/string_in_list.rs	59.72%	25 Missing and 4 partials ⚠️
datafusion/pruning/src/pruning_predicate.rs	90.55%	3 Missing and 19 partials ⚠️
...afusion/datasource-parquet/src/row_group_filter.rs	96.10%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24526      +/-   ##
==========================================
+ Coverage   81.27%   81.32%   +0.04%     
==========================================
  Files        1116     1118       +2     
  Lines      395017   396357    +1340     
  Branches   395017   396357    +1340     
==========================================
+ Hits       321055   322324    +1269     
- Misses      55166    55188      +22     
- Partials    18796    18845      +49     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.