Support compound product names with underscores and dict inputs (derived from PR #238) - #272
Draft
grey-nearing wants to merge 28 commits into
Draft
Support compound product names with underscores and dict inputs (derived from PR #238)#272grey-nearing wants to merge 28 commits into
grey-nearing wants to merge 28 commits into
Conversation
- Add comprehensive unit tests for all modules across googlehydrology (evaluation metrics, losses, models, dataset registry, Caravan loading, CLI runners, memory, LSTM initialization, and logging utilities). - Fix UnboundLocalError in samplingutils.py during error message formatting. - Fix chunking on object-dtype attribute columns in datasetzoo/caravan.py. - Clean up invalid TiedFrequencyMSERegularization reference in training/__init__.py. - Fix mock assertion sequence in test_multimet.py to match scaler save-before-scale ordering. - Fix CMAL deterministic quantile clipping assertion in test_uncertainty.py. - Add environments/environment_cpu.yml and update GitHub Actions CI workflow to Python 3.12 with PyTorch CPU and coverage reporting. - Configure pyproject.toml with test markers and warning filters.
…Guide - Add 'main' branch to pull_request and push triggers in .github/workflows/pytest-ci.yml. - Standardize import grouping into distinct standard library, third-party, and first-party blocks across test files. - Enforce 80-character maximum line length across all test suites, docstrings, comments, and fixtures. - Verify all 204 unit tests pass cleanly.
…te flags, and vectorize math - Break up long implied embedding assignments and error messages in samplingutils.py. - Wrap condition and error formatting within 80 characters in test_uncertainty.py. - Remove unused imports across conftest.py, test_caravan.py, test_logging_utils.py, test_losses.py, test_mfdata_loader.py, test_models.py, and test_samplingutils.py. - Restore original root logger handlers upon test exit in test_logging_utils.py. - Use absl flagsaver in test_mfdata_loader.py to prevent global flag state leakage. - Vectorize CMAL quantile monotonicity assertions in test_cmal_math.py. - Assert total dispatcher call count in test_cli.py.
- Add 'permissions: read-all' block to pytest-ci.yml. - Pin actions/checkout to immutable commit hash 11bd71901bbe5b1630ceea73d27597364c9af683 (# v4.2.2). - Pin conda-incubator/setup-miniconda to immutable commit hash d2e6a045a86077fb6cad6f5adf368e9076ddaa8d (# v3.1.0).
…set with deterministic seeds
…ing and path resolution
… clear cached dataset file handles
… datasets in unit tests
…failure on Windows
…is absent on Windows and set compile: False in test config
…earing-unify-input-file-formats
…and update logging
…ict inputs (PR #238) - Add PRODUCT_ALIASES and prefix matching in _product_name_from_feature to handle compound product names with underscores (e.g., CHIRPS_GEFS, ERA5_LAND). - Support dictionary-formatted inputs in _get_products_and_bands_from_features. - Add explicit missing variable validation when loading dynamic product Zarr stores. - Add unit tests in test/test_multimet.py for product name resolution and normalization.
…zedModule - Merge latest origin/main containing PR #263 and PR #264. - Fix state_dict loading in BaseTester._load_weights and BaseTrainer to load into getattr(self.model, '_orig_mod', self.model), enabling seamless weight loading across all combinations of compiled and uncompiled models. - Add parametrized unit tests covering all compile state_dict loading permutations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attribution & Background
This PR incorporates and modernizes the core contribution from PR #238 (originally authored by @joshsturtevant).
As the original PR remained unmerged and inactive following review discussions, this branch ports the necessary product-parsing and validation improvements to maintain full compatibility with the codebase's current Zarr architecture.
Summary of Changes
1. Compound Product Name Normalization & Prefix Matching
CHIRPS_GEFS,ERA5_LAND) previously failed to resolve from flat feature strings (e.g.chirps_gefs_precipitation) because product names were parsed by splitting on the first underscore (feature.split('_')[0]).PRODUCT_ALIASES,_normalize_product_key(), and_product_name_from_feature()ingooglehydrology/datasetzoo/multimet.pyto match normalized prefix aliases against canonical product names.2. Dual Input Configuration Format Support
hindcast_inputsandforecast_inputseither as:list[str]), ordict[str, list[str]])._get_products_and_bands_from_features().3. Explicit Missing Variable Validation & Logging
product_ds.data_vars, raising a descriptiveValueErrorlisting available variables if missing.Key Differences from Original PR #238
timeseries.zarr/dynamics.zarr) are detected._load_forecast_as_csv/basin_fallback_cache) since CSV dynamics loading has been deprecated and removed.KNOWN_GCS_PRODUCTSlists and secondary loops in favor of a single-pass alias dictionary.test/test_multimet.py(test_product_name_parsing_and_normalization) verifying alias resolution, underscore parsing, and dict inputs.Verification
pytest test/test_multimet.py -v: 13/13 passedpytest test/test_scaler.py test/test_convert.py test/test_caravan.py: 47/47 passed