Skip to content

Modify ConMon to support RRFS - #230

Draft
EdwardSafford-NOAA wants to merge 6 commits into
NOAA-EMC:developfrom
EdwardSafford-NOAA:feature/cm-rrfs-229
Draft

EdwardSafford-NOAA wants to merge 6 commits into
NOAA-EMC:developfrom
EdwardSafford-NOAA:feature/cm-rrfs-229

Conversation

@EdwardSafford-NOAA

Copy link
Copy Markdown
Collaborator

This PR makes the necessary changes to the ConMon to support RRFS. Significant changes include:

  • Drop support for the NAM and some of the time/date calculations that the NAM required. In the NAM the cnvstats for hrs 18z-23z were found in the next day's subdirectory. RRFS puts all the cnvstat files for a given day in the same subdirectory, which simplifies matters.
  • Modify the GrADS plot scripts for time and vertical plots for regional plots to limit them to a single area (termed global within the code). Most of the subdomains simply don't apply to regional models (e.g. S. America, Europe, etc.).
  • Add a new utility, find_last_cycle_rgnl.sh for both RadMon and ConMon use with regional sources.

Additionally there are some maintenance changes included:

  • Modify the GrADS plot scripts for time and vertical plots for global sources to reduce the number of plotted areas when the CONMON_RESTRICT_PLOT_AREAS switch is set to on. This is in an effort to reduce space usage.
  • Eliminate plotting plotstas_time_bias2.gs and plotstas_time_bias2_ps.gs scripts. The resulting files have not been available on the website for a long time (5+ years at least), and they have not been missed. Not running them saves ~2GB in space. The plot files remain in the repo so they can be used in the future if ever desired.
  • In RadMon the $NUM_CYCLES used in time series plots has been amended to plot 20 days worth of regional data by default. The global default value remains unchanged.
  • In RadMon the rm_img_files.pl, a utility script which removes old image files, has been tweaked to reduce the number of days worth of retained images as a space saving measure.

Testing has been done with both RRFS and GFS data on wcoss2.

Closes #229

Have IG working (minus horiz plots) for rrfs.  Save
work in progress.  Still needs testing & cleanup.
Reduce number of plot areas, restrict regional sources to plotting just
'global', meaning all.  Improve file cleanup for ConMon and RadMon.
File cleanup.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incorrect cycle intervals skip regional samples and prevent global plots, while extraction failures can be reported as success and cycle discovery can miss available data.

5 open issues
What changed in this PR

Adds RRFS regional image-generation support to ConMon, addressing issue #229, while reducing plot output and image retention to save storage.

Changes:

  • Replaces NAM-specific extraction logic with RRFS paths and adds regional cycle discovery, plotting, and transfer handling.
  • Limits regional time/vertical plots to one area, reduces optional global areas, and stops generating unused bias2 plots.
  • Sets RadMon’s regional time-series default to 480 cycles and reduces summary-image retention.
File Description
ush/​find_last_cycle_rgnl.sh Adds regional latest-cycle discovery.
src/​Radiance_Monitor/​parm/​RadMon_user_settings Defaults regional time series to 480 cycles.
src/​Radiance_Monitor/​image_gen/​ush/​RadMon_IG_rgn.sh Retains six regional summary-image cycles.
src/​Radiance_Monitor/​image_gen/​ush/​RadMon_IG_glb.sh Retains six global summary-image cycles.
src/​Conventional_Monitor/​parm/​ConMon_config Removes the shared cycle-count default.
src/​Conventional_Monitor/​image_gen/​ush/​Transfer.sh Adds regional transfer destinations and area selection.
src/​Conventional_Monitor/​image_gen/​ush/​transfer_imgs.sh Removes diagnostic logging.
src/​Conventional_Monitor/​image_gen/​ush/​plot_vert.sh Adds regional data paths and interval handling.
src/​Conventional_Monitor/​image_gen/​ush/​plot_time.sh Adds regional paths and drops bias2 plotting.
src/​Conventional_Monitor/​image_gen/​ush/​plot_time_ps.sh Adds regional pressure-data handling and drops bias2 plotting.
src/​Conventional_Monitor/​image_gen/​ush/​plot_hist.sh Locates regional histogram data.
src/​Conventional_Monitor/​image_gen/​ush/​mk_time_vert.sh Adjusts whitespace only.
src/​Conventional_Monitor/​image_gen/​ush/​mk_horz_hist.sh Restricts horizontal-map jobs to global sources.
src/​Conventional_Monitor/​image_gen/​ush/​ConMon_IG.sh Coordinates regional cycles, work paths, and cleanup.
src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_vert_count.gs Limits regional count profiles and optional global areas.
src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_vert_bias.gs Limits regional bias profiles and optional global areas.
src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_time_count.gs Limits time-series count plot areas.
src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_time_count_ps.gs Limits surface-pressure count plot areas.
src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_time_bias.gs Limits time-series bias plot areas.
src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_time_bias_ps.gs Limits surface-pressure bias plot areas.
src/​Conventional_Monitor/​image_gen/​gscripts/​plot_hist.gs Adds histogram plotting for observation statistics.
src/​Conventional_Monitor/​data_extract/​ush/​ConMon_DE_rgn.sh Replaces NAM timing and filename logic with RRFS paths.

Review effort: Balanced

Comment thread ush/find_last_cycle_rgnl.sh Resolved
Fix copilot identified issues.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cycle-discovery bugs prevent plotting available data, while unclosed area files cause incorrect labels on regional plots.

1 open issue
2 previously missed issues

In code that hasn't changed since last review

Medium severity Open area.txt handle prevents reading region labels

src/​Conventional_Monitor/​image_gen/​gscripts/​plotstas_time_count.gs:36

This read leaves area.txt open at EOF, so the region loop's later read of the recreated pathname fails and gives the first plotted region an unknown area label. Close the file after reading $AREA so the loop can reopen it for the region description. Apply the same fix in plotstas_time_bias.gs:34, plotstas_time_count_ps.gs:35, and plotstas_time_bias_ps.gs:34; for the surface-pressure scripts, this affects every regional plot.

Medium severity Newest empty cycle directory hides earlier populated data

ush/​find_last_cycle_rgnl.sh:81

Selecting only the newest directory hides earlier available cycles when that day has no statistics yet, such as during an in-progress or failed first extraction. ConMon_IG.sh then receives an empty timestamp and cannot plot the available data. Skip directories without matching cycle files and select the newest populated one.

5 resolved since last review

Review effort: Balanced

Comment thread ush/find_last_cycle_rgnl.sh Resolved
Final copilot fix.
More small corrections.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Modify ConMon to support regional sources

2 participants