feat[iceberg]: table statistics suite by sandugood · Pull Request #687 · datafusion-contrib/datafusion-distributed

sandugood · 2026-09-02T16:37:10Z

Follow-up to #676 (closes #681)

What was changed/added:

Statistics are being computed once at the very beginning of the scan and not recomputed
Everything is being computed from ManifestList -> ManifestFile -> Manifest (from where we get DataFile information). Now we are getting datafile level stats directly by mapping field-ids to actual columns in .parquet files.
Casting from iceberg's Datum

sandugood · 2026-09-02T16:44:37Z

By the way, on the current main of iceberg-rust(rev = 1ecd497) there is a nice API of ManifestReader, which would strip the table.file_io() part
However I think that staying with 0.10.0 is also alright. Wdyt?

gabotechs · 2026-09-03T10:53:44Z

🤔 I'd say let's stick to 0.10 until iceberg-rust releases a new version to crates.io that uses datafusion@55. Once that's on crates.io we can start thinking about consuming all those new APIs.

It does look promising though.

gabotechs · 2026-09-03T18:54:46Z

+                    let cs = &mut col_stats[i];
+
+                    if let Some(&n) = df.null_value_counts().get(&id) {
+                        cs.null_count = cs.null_count.add(&Precision::Exact(n as usize));
+                    }
+                    if let Some(scalar) = df.lower_bounds().get(&id).and_then(datum_to_scalar) {
+                        cs.min_value = cs.min_value.min(&Precision::Inexact(scalar));
+                    }
+                    if let Some(scalar) = df.upper_bounds().get(&id).and_then(datum_to_scalar) {
+                        cs.max_value = cs.max_value.max(&Precision::Inexact(scalar));
+                    }
+                }


We are missing the two most important pieces of stats here:

byte_size

distinct_values

Is there any chance we can get them as well?

Not that I know of particularly, following the Iceberg spec: https://iceberg.apache.org/spec/#field-level-metrics-and-statistics

I see there do is a df.column_sizes() method available, but found nothing for the distinct values... that's a shame, that's actually pretty important for estimating aggregations, filter selectivity, etc...

Yeah, without reading the underlying .parquet footers I think we can't get that info, sadly

https://iceberg.apache.org/puffin-spec/#apache-datasketches-theta-v1-blob-type
We actually can get distinct count metadata from the puffin spec. I know internally at datadog we write these. So sometimes it's available depending on the system.

{ "statistics": [ { "snapshot-id": 123, "statistics-path": ".../stats.puffin", "blob-metadata": [ { "type": "apache-datasketches-theta-v1", "snapshot-id": 123, "fields": [4], "properties": { "ndv": "975" } } ] } ] }

gabotechs

Looks good! thanks @sandugood, really nice work.

+1 on my side, let's address @alexanderbianchi's and we are good to pull this in

sandugood · 2026-09-04T08:55:28Z

Added the part of getting NDV from the BlobMetadata's properties

alexanderbianchi

NDV changes look good, we probably need a consistent story for testing across snapshots + creating 1 off metadata to query, doesn't have to be this PR IMO.

…olumnStats and the way they are collected

alexanderbianchi · 2026-09-05T23:27:20Z

Rebased with conflicts from my own PR and merged

Tests can supply complete Iceberg table metadata, explicit table options, and in-memory fixture files through `IcebergTestHarness::builder()`. - `with_table_metadata` accepts native Iceberg metadata without reconstructing its IDs, schemas, or snapshot history. - `with_table_option` replaces duplicated registration SQL in the selected-snapshot and invalid-snapshot-ID tests. - `with_file(uri, bytes)` overrides a fixture file, including manifests and manifest lists. Explicit files take precedence over generated metadata; unmodified paths fall back to the checked-in taxi fixture. An empty-table registration test verifies raw metadata override behavior. `build()` now constructs the session and registers the table directly; there is no intermediate `create()` helper. The harness remains usable without distributed integration features. Shared fixture helpers expose only the original taxi metadata and a native metadata builder starting before the first snapshot. Snapshot recipes stay test-local. The duplicated missing-summary JSON fixture is removed. Rebased onto `iceberg-0.10` at `6507bf8`, including merged #687. The table-options cleanup and file-override support are separate commits. Test-case consolidation is intentionally kept in an independent PR. Validation on #700 at `bb3872e`: - `cargo test -p datafusion-distributed-iceberg --locked` — 90 tests passed, including the doctest - `cargo clippy -p datafusion-distributed-iceberg --all-targets --locked -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check`

Extends `roundtrips_data_source_plan`, the existing codec test from #684, with two explicit storage properties supplied through #700's harness builder. Checks their decoded values, including a quote-containing value that exercises SQL literal escaping, while retaining the existing schema, partitioning, fetch, feed, property-map, and statistics assertions. The diff against #700 remains 15 changed lines in `iceberg/src/codec.rs`, with no new harness methods or test functions. Stacked on #700, now rebased onto `iceberg-0.10` at `6507bf8` after #687 merged. Only #700 is a prerequisite. Validation: - `cargo test -p datafusion-distributed-iceberg --locked` — 90 tests passed, including the codec roundtrip test and doctest - `cargo clippy -p datafusion-distributed-iceberg --tests --locked -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check`

## Less test code, stronger statistics coverage Based directly on `iceberg-0.10` at `f9340ef`, including merged #715 and #716. “Before” refers to that base. Only `iceberg/tests/statistics.rs` changes, with a net reduction of 24 lines. | Test / edge case | Before | After | |---|---|---| | Row count and full-scan byte size | Separate enabled/disabled test bodies | Ordinary named tests share assertions and retain both modes | | Full-schema column statistics | Vector length only, in another pair of tests | Complete expected vector: populated metrics and unknown columns | | Reordered projection / nonconsecutive selected field IDs | Projected vector length and row count | Exact association of metrics with reordered columns using IDs 4, 1, and 5 | | Null-count aggregation across files | **None** | Exact sum of 5 | | Column-size aggregation across files | **None** | Inexact sums of 400 and 600 | | Min/max bounds and scalar types | **None** | Int32 and Int64 bounds checked with their precision; opposing extrema order exercises both updating and retaining bounds | | One file missing a column's null count | **None** | Aggregate stays `Absent`, rather than treating missing as zero | | Entirely missing column metrics | Only an all-empty-metrics fixture | Unknown column alongside columns with known metrics | | Column statistics disabled | Fixture had no metrics even when enabled | Populated fixture must return unknown column statistics when disabled | | Explicit current-snapshot selection | Separate duplicate row-total test | Fixture selects the snapshot explicitly in every matrix case | | Missing snapshot summary totals | Present | Retained | | Filter and projection/sort propagation | Present | Retained | | `COUNT(*)` skips the scan | Duplicated enabled/disabled bodies and snapshots | Both named cases retained, sharing assertions | | Explain formatting | Two identical snapshots plus weak label checks | One diagnostic snapshot; computed values verified by the matrix | ## Structure - Four ordinary named Tokio tests cover full scan / reordered projection × column statistics enabled / disabled. Two more named tests cover `COUNT(*)` scan elimination. No `test_case` macros or dependency. - Session setup uses the merged harness's `configure_session(...)`; tests never access its private context. - Native Iceberg writers create a manifest with two synthetic data-file entries and a matching manifest list. #700's `with_file` supplies the bytes to the harness. - Assertions observe query-output statistics, removing the recursive search/downcast helper and also checking projection propagation. - `insta::allow_duplicates!` is needed only for the shared, identical `COUNT(*)` inline snapshots. - The fixture is planning-only: synthetic Parquet paths are not opened. Multi-manifest merging, delete files, NDV, and verification against actual Parquet contents are not claimed here. - The genuinely historical-snapshot regression remains in #702; this PR consolidates the current-snapshot-only check from #687. ## Validation - `cargo test -p datafusion-distributed-iceberg --test statistics --locked` — all 10 cases passed - `cargo test -p datafusion-distributed-iceberg --locked` — 95 tests passed, including the doctest - `cargo test -p datafusion-distributed-iceberg --features integration --locked` — 96 tests passed, including the doctest - `cargo clippy -p datafusion-distributed-iceberg --all-targets --locked -- -D warnings` — passed, also with `--all-features` - `cargo fmt --all -- --check` - `git diff --check` - Mutation check: replacing computed column statistics with unknown values fails both enabled cases; the other eight tests pass. Mutation reverted. - Bounds mutation check: keeping the first minimum and taking the last maximum fails both enabled cases; the other eight tests pass. Mutation reverted.

sandugood changed the title ~~Changed the way we compute datafile level statistics~~ feat: table statistics suite Sep 2, 2026

sandugood changed the title ~~feat: table statistics suite~~ feat[iceberg]: table statistics suite Sep 2, 2026

gabotechs reviewed Sep 3, 2026

View reviewed changes

Comment thread iceberg/src/data_source.rs Outdated

Comment thread iceberg/src/data_source.rs Outdated

Comment thread iceberg/src/data_source.rs Outdated

sandugood marked this pull request as draft September 3, 2026 16:43

sandugood marked this pull request as ready for review September 3, 2026 17:55

gabotechs reviewed Sep 3, 2026

View reviewed changes

alexanderbianchi mentioned this pull request Sep 3, 2026

test: reproduce selected snapshot statistics bug alexanderbianchi/datafusion-distributed#1

Closed

alexanderbianchi reviewed Sep 3, 2026

View reviewed changes

Comment thread iceberg/src/data_source.rs Outdated

gabotechs approved these changes Sep 4, 2026

View reviewed changes

alexanderbianchi approved these changes Sep 5, 2026

View reviewed changes

This was referenced Sep 5, 2026

refactor: support historical Iceberg snapshot fixtures alexanderbianchi/datafusion-distributed#2

Closed

refactor: construct historical Iceberg fixtures in tests #702

Merged

feat: serialize Iceberg data source plans #684

Merged

alexanderbianchi force-pushed the feat/iceberg-column-stats-refactor branch from dbc3cc0 to ed5ed18 Compare September 5, 2026 22:50

sandugood added 10 commits September 5, 2026 18:51

Changed the way we compute datafile level statistics

84b2b05

Revert the .lock file to main

2472bff

revert of lock file

b846ef2

Moved column stats to separate optional field, added concurrent IO

fa8674a

Get rid of log prints

35ad080

Revisioned the part with the partition_statistics logic

3c5c316

Fixed column statistics collecting in case of a projection

770df2f

fixed formatting

ab04233

Created separate helpers to reduce boilerplate, revisioned optional C…

81e75df

…olumnStats and the way they are collected

Added ndv for col stats + snapshot resolution for time-travel queries

c7d5d3b

alexanderbianchi force-pushed the feat/iceberg-column-stats-refactor branch from ed5ed18 to c7d5d3b Compare September 5, 2026 22:56

alexanderbianchi merged commit 6507bf8 into datafusion-contrib:iceberg-0.10 Sep 5, 2026
32 checks passed

This was referenced Sep 6, 2026

refactor: build Iceberg test metadata at runtime #700

Merged

refactor: make Iceberg codec fixture options explicit #703

Merged

refactor: consolidate and strengthen Iceberg statistics tests #705

Merged

Uh oh!

Conversation

sandugood commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sandugood commented Sep 2, 2026

Uh oh!

gabotechs commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabotechs Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

sandugood Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

gabotechs Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

sandugood Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

alexanderbianchi Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gabotechs left a comment

Choose a reason for hiding this comment

Uh oh!

sandugood commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexanderbianchi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexanderbianchi commented Sep 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sandugood commented Sep 2, 2026 •

edited

Loading

alexanderbianchi Sep 3, 2026 •

edited

Loading

sandugood commented Sep 4, 2026 •

edited

Loading