fre.cmor: improved and comprehensive omission tracking

[Copilot]

Describe your changes

When fre cmor run processes multiple variables and some fail, failures were logged individually but never summarized. The existing code even had a placeholder comment # log an omitted variable here....

Changes:

• cmor_mixer.py / cmorize_all_variables_in_dir: Collect failed variables in an omissions list during the processing loop. After the loop, emit a structured summary via fre_logger.warning that includes the variable names, failure reason, and the expected input file paths for each omitted variable:

  --- OMISSION LOG: 2 variables could not be processed ---
    OMITTED local_var=foo / target_var=foo, reason: <exception message>
      file: /path/to/indir/component.19930101-19930201.foo.nc
  --- END OMISSION LOG ---
  

• fre/cmor/tests/test_cmor_mixer_omission_tracking.py: Unit tests for omission tracking covering single failure, multiple failures, all-success (no log emitted), and mixed success/failure cases — including assertions that expected file paths appear in the omission log output.

• fre/tests/test_fre_cmor_cli.py / test_cli_fre_cmor_run_with_logfile: CLI integration test that runs fre -vv -l LOGFILE cmor run --run_one ... against existing test data and verifies the resulting log file contains log lines from both fre.py (the CLI entry point) and fre.cmor.cmor_mixer (the CMOR processing module).

• fre/tests/test_fre_cmor_cli.py / test_cli_fre_cmor_run_with_logfile_omission_case: CLI integration test that uses a varlist where sos->sos succeeds and sosV2->tob fails (tob is a valid CMIP6_Omon variable but the sosV2 file contains sos data, not tob). Runs without --run_one so both variables are attempted, then checks the log file for the OMISSION LOG summary, the OMITTED local_var=sosV2 / target_var=tob entry, the expected .sosV2.nc file path, and cmor_mixer.py module log lines.

Issue ticket number and link (if applicable)

Checklist before requesting a review

• I ran my code
• I tried to make my code readable
• I tried to comment my code
• I wrote a new test, if applicable
• I wrote new instructions/documentation, if applicable
• I ran pytest and inspected it's output
• I ran pylint and attempted to implement some of it's feedback
• No print statements; all user-facing info uses logging module

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 84.31%. Comparing base (836559e) to head (552e25b).

Files with missing lines	Patch %	Lines
fre/cmor/cmor_mixer.py	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #833      +/-   ##
==========================================
+ Coverage   84.12%   84.31%   +0.19%     
==========================================
  Files          71       71              
  Lines        4944     4954      +10     
==========================================
+ Hits         4159     4177      +18     
+ Misses        785      777       -8     

Flag	Coverage Δ
unittests	84.31% <90.90%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
fre/cmor/cmor_mixer.py	91.58% <90.90%> (+1.72%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 836559e...552e25b. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ilaflott]

@copilot we want filepaths with the omission logging

[Copilot]

@copilot we want filepaths with the omission logging

Done in 9fc5417. Each omission entry in the log now lists the expected file paths for that variable:

--- OMISSION LOG: 1 variable could not be processed ---
  OMITTED local_var=sos / target_var=sos, reason: ...
    file: /path/to/indir/component.19930101-19930201.sos.nc
--- END OMISSION LOG ---

The paths are constructed from indir, name_of_set, each datetime in iso_datetime_range_arr, and local_var.

[ilaflott]

@copilot write a single unit test for a CLI call in fre/tests/test_fre_cmor_cli.py using the -l flag to pipe log output to a file. check the file for the output, make sure output from the CMOR module is in there too.

[Copilot]

@copilot write a single unit test for a CLI call in fre/tests/test_fre_cmor_cli.py using the -l flag to pipe log output to a file. check the file for the output, make sure output from the CMOR m...

Done in c462371. Added test_cli_fre_cmor_run_with_logfile to fre/tests/test_fre_cmor_cli.py. It runs fre -vv -l TEST_CMOR_RUN.log cmor run --run_one ... against the existing test data, then reads the log file and asserts:

1. 'fre_file_handler added to base_fre_logger' is present — confirms the fre.py entry point wrote to the file
2. 'cmor_mixer.py' is present — confirms the fre.cmor.cmor_mixer module's log lines were captured

[ilaflott]

Done in c462371. Added test_cli_fre_cmor_run_with_logfile to fre/tests/test_fre_cmor_cli.py. It runs fre -vv -l TEST_CMOR_RUN.log cmor run --run_one ... against the existing test data, then reads the log file and asserts:

1. 'fre_file_handler added to base_fre_logger' is present — confirms the fre.py entry point wrote to the file
2. 'cmor_mixer.py' is present — confirms the fre.cmor.cmor_mixer module's log lines were captured

@copilot add another unit test, very similar, call it case2 or omissionlistcase, and make sure the example only processes one file successfully, skips over the others, and logs the omission in the log file, and the contents of the log file are checked for this

[Copilot]

Done in c462371. Added test_cli_fre_cmor_run_with_logfile to fre/tests/test_fre_cmor_cli.py. It runs `fr...

Added in cd06f44. The new test test_cli_fre_cmor_run_with_logfile_omission_case creates a temporary varlist with {"sos": "sos", "sosV2": "tob"} — sos processes successfully while sosV2->tob fails (the sosV2 file contains sos data, not tob). It runs without --run_one so both variables are attempted, then checks the log file for:

• OMISSION LOG summary header
• OMITTED local_var=sosV2 / target_var=tob entry
• .sosV2.nc file path in the omission output
• cmor_mixer.py module log lines

[ilaflott]

ilaflott/FREmorizer#75 solved this for fremorizer, closing