Repository navigation
Conversation
The inclusive metrics evaluator only reads a data file's record count and its value, null and NaN counts and bounds, yet it requires a whole DataFile. Introduce a crate-internal FileMetrics view that borrows those statistics, evaluate through it, and keep the DataFile entry point as a thin conversion. Statistics held outside a DataFile, such as whole-file statistics carried with a scan task, can then be evaluated without rebuilding one. A record count known to be zero still excludes the file unless empty files are included; an unknown record count does not by itself exclude it, while the available bounds and counts may still prune it. Behaviour for DataFile evaluation is unchanged.
21 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
InclusiveMetricsEvaluatoronly reads a data file's record count and its value, null and NaN counts and lower/upper bounds, but it requires a wholeDataFile.FileMetrics<'a>view that borrows those statistics, withFrom<&DataFile>.InclusiveMetricsEvaluator::eval_metrics(filter, FileMetrics, include_empty_files);eval(filter, &DataFile, include_empty_files)now converts and delegates.include_empty_filesis set. An unknown record count does not by itself exclude the file; the available bounds and counts may still prune it.No behaviour change for existing callers and no public API change (everything new is
pub(crate)). This lets follow-up work evaluate whole-file statistics carried with a scan task without reconstructing aDataFile.Are these changes tested?
Yes:
eval.FileMetricsbuilt from independently constructed maps (noDataFile) prunes and keeps files from bounds and null counts with both a known and an unknown record count, keeps files whose column has no statistics, and skips a known empty file unless empty files are included.AI Disclosure
This change was developed with assistance from Claude Code and reviewed by me. Tests and CI results come from this branch.