Add developer workflow with test coverage calculation

[AlysonStahl-NOAA]

Fixes #1353
Fixes #1354

[clyden-noaa]

@WenMeng-NOAA @BenjaminBlake-NOAA @AlysonStahl-NOAA
I have completed testing on the RDHPCs.
All tests were successful and finished within timing thresholds
There were no baseline changes, as expected

Please let me know if any additional testing will be required once the build error is resolved on WCOSS2.

[JesseMeng-NOAA]

@AlysonStahl-NOAA Hi Alyson, have you had a chance to look into this? As we discussed, this project does not involve IFI and GTG. Thanks.

@AlysonStahl-NOAA Wen and I discovered an issue with compiling the UPP code with IFI + GTG on WCOSS2 using your branch. Could you take a look? Here are the errors:

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: warning: libjpeg.so.62, needed by /apps/spack/jasper/2.0.25/intel/19.1.3.304/sjib74krrorkyczqpqah4tvewmlnqdx4/lib64/libjasper.so, may conflict with libjpeg.so.9
[ 97%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.x.dir/INITPOST.F.o
[ 98%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.x.dir/INITPOST_MPAS.F.o
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/gcc/x86_64-suse-linux/7/../../../../lib64/crt1.o: in function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:104:(.text+0x20): undefined reference to `main'
make[2]: *** [unit_tests/CMakeFiles/test_dewpoint.dir/build.make:111: unit_tests/test_dewpoint] Error 1
make[1]: *** [CMakeFiles/Makefile2:1093: unit_tests/CMakeFiles/test_dewpoint.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 98%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.x.dir/getIVariableN.f.o
[ 99%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.x.dir/getVariable.f.o
[ 99%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.x.dir/INITPOST_NEMS.f.o
[100%] Linking Fortran executable upp.x
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: warning: libjpeg.so.62, needed by /apps/spack/jasper/2.0.25/intel/19.1.3.304/sjib74krrorkyczqpqah4tvewmlnqdx4/lib64/libjasper.so, may conflict with libjpeg.so.9
[100%] Built target upp.x
make: *** [Makefile:146: all] Error  

[AlysonStahl-NOAA]

Apologies for the delay, I have been looking at this. I have been getting that same warning about the libjpeg and libjasper conflict, but the test program builds fine for me. I don't have access to GTG or IFI, so I can't tell if those options in particular are what's causing the issue.

I did go back to some of my changes and found that I introduced redundant CMake flags, which I since removed. But I can't see how they would cause any issues.

It's not clear to me yet if that libjpeg/libjasper conflict is related to this problem, even if it did manage to build for me. I've been trying to figure out the source of the conflict.

[BenjaminBlake-NOAA]

@AlysonStahl-NOAA Thanks for looking into it! I think the libjpeg/libjasper conflict error can be ignored because I also get those messages when compiling the UPP code, and it still builds successfully.

I just tried building the UPP executable (including GTG and IFI) with your latest changes, but it still has trouble. Here is my working directory on Cactus if that would be helpful for debugging:
/lfs/h2/emc/vpppg/noscrub/Benjamin.Blake/UPP-alyson/tests
Logfile: compile.20251126.log

Within the tests/build directory you can find the files that are referenced inside the compile.20251126.log file.

[BenjaminBlake-NOAA]

@AlysonStahl-NOAA Hope you had a nice Thanksgiving!

It looks like this line inside build/unit_tests/CMakeFiles/test_dewpoint.dir/build.make is where the error occurs:
cd /lfs/h2/emc/vpppg/noscrub/Benjamin.Blake/UPP-alyson/tests/build/unit_tests && $(CMAKE_COMMAND) -E cmake_link_script CMakeFiles/test_dewpoint.dir/link.txt --verbose=$(VERBOSE)

I compared the build.make from my failed build to a build.make file from a successful build, and I noticed the following 4 lines are only included in the build.make of the failed build:

unit_tests/test_dewpoint: sorc/libIFI.fd/src/libIFI/libIFI.a
unit_tests/test_dewpoint: sorc/libIFI.fd/src/libtdrp/libtdrp.a
unit_tests/test_dewpoint: /apps/ops/prod/libs/intel/19.1.3.304/ip/4.0.0/lib64/libip_4.a
unit_tests/test_dewpoint: /apps/ops/prod/libs/intel/19.1.3.304/sp/2.3.3/lib/libsp_4.a

I wonder if the paths to the libIFI.fd directory should be the full paths instead of the relative paths listed above, or if they should be removed entirely from build,make. What do you think?

[AlysonStahl-NOAA]

@BenjaminBlake-NOAA You as well! Apologies for my slowness in responding here. I just returned from time off.

I don't know if I can recommend completely removing them because I don't entirely understand yet how libIFI is used by UPP or if moving the paths will break anything. I do think that the full path is generally preferred.

It's a little more difficult for me to debug without direct access to GTG/IFI. I see that your build uses "INTERNAL_IFI" and I can see the IFI library under sorc in your UPP-alyson directory. Would I be able to copy this and test it out?

Also, is this build the only one that uses internal IFI? I'm wondering if this is a path issue.

[AlysonStahl-NOAA]

I should also note, that it is possible to build without the unit tests using BUILD_TESTING=OFF. As far as I know, they will mostly be run in the GitHub workflows. Although if there is an actual issue exposed here we probably don't want to just get around them by turning unit tests off.

[BenjaminBlake-NOAA]

@AlysonStahl-NOAA No worries! Yes I think you should be able to copy the code from my directory and use it for testing/debugging. I think UPP is the only build that uses internal IFI since it is related to aviation products (in-flight icing).

If the issue becomes troublesome we could alternatively set BUILD_TESTING=OFF when building with the IFI code, that's also true.

[AlysonStahl-NOAA]

Oh, by internal IFI I just mean the Cmake build option INTERNAL_IFI. I noticed that there seems to be a separate option to load it as an existing external module. I wanted to see if maybe the issue was arising with INTERNAL_IFI specifically.

I copied libIFI over and tried building UPP again and ended up getting the same error. One thing I did notice is that, with IFI enabled, it tries to link the test program as a CXX executable rather than a Fortran one. Using IFI seems to add additional compilation flags for the executable.

[AlysonStahl-NOAA]

I added a small change. This fixed it on my end while building with your copy of libIFI. Let me know if that fixes it for you.

[BenjaminBlake-NOAA]

Ahh okay, understood about INTERNAL_IFI.

Your change to set the linker language to Fortran also fixed the error for me. So I think we are good to continue with processing your PR. Thanks for your help!

[BenjaminBlake-NOAA]

@clyden-noaa Now that the compilation error related to IFI has been fixed, let's continue processing this PR. When you get a chance, could you re-run the UPP RTs on the R&D machines?

Thanks!

[clyden-noaa]

@BenjaminBlake-NOAA @WenMeng-NOAA @AlysonStahl-NOAA
Maintenance has started on URSA preventing RTs to be run. I will finish testing on URSA once the maintenance is complete.

[clyden-noaa]

@WenMeng-NOAA @BenjaminBlake-NOAA @AlysonStahl-NOAA
All of my testing is complete.
All of the RTs were successful and finished within accepted timing thresholds across all RDHPCs.
There were no baseline changes, as expected.

[BenjaminBlake-NOAA]

@clyden-noaa Thanks for testing - this PR is ready to be merged.