OpenACC port of halo exchanges by abishekg7 · Pull Request #1355 · MPAS-Dev/MPAS-Model

abishekg7 · 2025-08-07T23:55:17Z

This PR enables execution of halo exchanges on GPUs via OpenACC directives, when the MPAS atmosphere core has been built with an appropriate GPU-aware MPI library. This PR builds on the OpenACC memory consolidation introduced in #1315.

Module mpas_halo is modified in the following ways to enable GPU-aware halo exchanges:

In the call to mpas_halo_exch_group_complete, OpenACC directives copy to device all the relevant fields and metadata that are required for the packing and unpacking loops later.
OpenACC directives are introduced around the packing and unpacking loops to perform the field to/from send/recv buffer operations on the device. The attach clauses introduced to the parallel constructs ensures that the device pointers are attached to the device targets at the start of the parallel region and detached at the end of the region.
The actual MPI_Isend and MPI_Irecv operations use GPU-aware MPI, by wrapping these calls within

!$acc host_data use_device(pointer_to_buffer)

!$acc end host_data

In addition, this PR introduces a new namelist option under the development group, config_gpu_aware_mpi. When set to true, it switches on GPU-direct halo exchanges. The default setting is false, in which halo exchanges occur on the host, which necessitates host<->device data transfers around the halo exchanges as necessary.

Note: This feature is not available to use when config_halo_exch_method is set to 'mpas_dmpar'

src/framework/mpas_halo.F

mgduda · 2026-03-17T20:54:36Z

src/core_atmosphere/Registry.xml

                     units="-"
                     description="Method to use for exchanging halos"
                     possible_values="`mpas_dmpar', `mpas_halo'"/>
+               <nml_option name="config_gpu_aware_mpi" type="logical" default_value="false"


For consistency and readability, let's consider adding a blank line above this. (I do see that the IAU namelist record doesn't use a blank line between options, but we can address that in a separate clean-up PR at a later time.)

mgduda · 2026-03-17T20:55:27Z

src/core_atmosphere/Registry.xml

                     possible_values="`mpas_dmpar', `mpas_halo'"/>
+               <nml_option name="config_gpu_aware_mpi" type="logical" default_value="false"
+                     units="-"
+                     description="Whether to use GPU-aware MPI for halo exchanges"


In the description, we could also mention that this only works when config_halo_exch_method is set to mpas_halo.

mgduda · 2026-03-17T21:25:24Z

src/framework/mpas_dmpar.F

         iErr = MPAS_DMPAR_NOERR
      end if

+      useGPUAwareMPI = .false.


Is this local variable actually needed?

mgduda · 2026-03-17T21:29:46Z

src/core_atmosphere/mpas_atm_halos.F

         call mpas_log_write('')

+         if (config_gpu_aware_mpi) then
+            call mpas_log_write('GPU-aware MPI is not presently supported with config_halo_exch_method = mpas_dmpar',MPAS_LOG_CRIT)


This seems like it's redundant given the check that exists in the mpas_dmpar_exch_group_full_halo_exch routine.

I will retain this check and remove the redundant one in mpas_dmpar_exch_group_full_halo_exch so that it will terminate earlier.

Actually, I think the check in mpas_dmpar_exch_group_full_halo_exch is preferable, since it would catch uses of the routine with withGPUAwareMPI = .true. in cores other than the atmosphere core.

mgduda · 2026-03-17T21:32:16Z

src/framework/mpas_halo.F

        integer, dimension(:,:,:), CONTIGUOUS pointer :: recvListSrc, recvListDst
        integer, dimension(:), CONTIGUOUS pointer :: unpackOffsets

-


I'd suggest leaving this blank line in place, as this change isn't necessary to the objective of this PR and this line wouldn't otherwise be modified.

mgduda · 2026-03-17T21:34:15Z

src/framework/mpas_halo.F

+            return
+        end if
+
+        call mpas_timer_start('full_halo_exch')


In the past, my experience has been that timing for halo exchanges may not be especially valuable. Load imbalance can show up as time apparently spent in halo communications, when it's really just that one MPI task had to finish some extra work before participating in the halo exchange. What do you think about removing these timers?

That's true. I'm fine with removing it for now and revisiting it as needed.

mgduda · 2026-03-17T21:38:34Z

src/framework/mpas_halo.F

 #endif
        rank = group % fields(1) % compactHaloInfo(8)

+        !$acc data present(group % recvBuf(:), group % sendBuf(:)) if(useGPUAwareMPI)


If the parallel directives in this routine all use default(present), is this directive necessary?

Doesn't appear to be required, and will remove it.

This commit enables execution of halo exchanges on GPUs via OpenACC directives, if the MPAS atmosphere core has been built with an appropriate GPU-aware MPI distribution. Module mpas_halo is modified in the following ways to enable GPU-aware halo exchanges: - In the call to mpas_halo_exch_group_complete, OpenACC directives copy to device all the relevant fields and metadata that are required for the packing and unpacking loops later. - OpenACC directives are introduced around the packing and unpacking loops to perform the field to/from send/recv buffer operations on the device. The attach clauses introduced to the parallel constructs ensures that the device pointers are attached to the device targets at the start of the parallel region and detached at the end of the region. - The actual MPI_Isend and MPI_Irecv operations use GPU-aware MPI, by wrapping these calls within !$acc host_data constructs. Note: This commit introduces temporary host-device data movements in the atm_core_init routine around the two calls to exchange_halo_group. This is required just for this commit as all halo-exchanges occur on the device and fields not yet present on the device must be copied over to it before the halo exchanges and back to host after it. These copies will be removed in subsequent commits.

Introducing a new namelist option under development, config_gpu_aware_mpi, which will control whether the MPAS runs on GPUs will use GPU-aware MPI or perform a device<->host update of variables around the call to a purely CPU-based halo exchange. Note: This feature is not available to use when config_halo_exch_method is set to 'mpas_dmpar'

abishekg7 force-pushed the framework/acc_halo_exch branch 2 times, most recently from 56ed028 to a8fda92 Compare August 15, 2025 01:07

abishekg7 mentioned this pull request Aug 19, 2025

Refactoring duplicate definitions of halo_exchange_routine #1359

Merged

mgduda reviewed Sep 24, 2025

View reviewed changes

src/framework/mpas_halo.F Outdated Show resolved Hide resolved

abishekg7 force-pushed the framework/acc_halo_exch branch from d0c1431 to 1e08917 Compare October 13, 2025 22:50

abishekg7 force-pushed the framework/acc_halo_exch branch from 1e08917 to eff87cf Compare February 27, 2026 06:04

abishekg7 marked this pull request as ready for review February 27, 2026 06:06

abishekg7 force-pushed the framework/acc_halo_exch branch from eff87cf to 3d2c334 Compare March 4, 2026 20:02

abishekg7 force-pushed the framework/acc_halo_exch branch from 3d2c334 to 5486740 Compare March 13, 2026 20:53

mgduda self-requested a review March 14, 2026 01:04

abishekg7 changed the title ~~[Draft] OpenACC port of halo exchanges~~ OpenACC port of halo exchanges Mar 17, 2026

abishekg7 force-pushed the framework/acc_halo_exch branch from 5486740 to 6ce8e33 Compare March 17, 2026 17:29

mgduda requested changes Mar 17, 2026

View reviewed changes

G. Dylan Dickerson and others added 5 commits March 17, 2026 23:54

Some cleanup to address review comments

0f88b47

Removing acc data and present clause for send and recv buf

461c33e

Removing duplicate check for abort case

2ae3105

abishekg7 force-pushed the framework/acc_halo_exch branch from 6ce8e33 to 2ae3105 Compare March 18, 2026 06:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenACC port of halo exchanges #1355

OpenACC port of halo exchanges #1355
abishekg7 wants to merge 5 commits intoMPAS-Dev:developfrom
abishekg7:framework/acc_halo_exch

abishekg7 commented Aug 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

mgduda Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

abishekg7 Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

abishekg7 Mar 17, 2026

Uh oh!

mgduda Mar 17, 2026

Uh oh!

abishekg7 Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		integer, dimension(:,:,:), CONTIGUOUS pointer :: recvListSrc, recvListDst
		integer, dimension(:), CONTIGUOUS pointer :: unpackOffsets

Conversation

abishekg7 commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abishekg7 commented Aug 7, 2025 •

edited

Loading