Give Runtime Policy Information to Reducers by MrBurmark · Pull Request #2061 · llnl/RAJA

MrBurmark · 2026-07-16T22:32:23Z

Summary

Add Policy enum to constructor/reset of reducers/multi-reducers to allow them to optimize their resource usage based on the backend they will actually be used with. Constructing/resetting a reducer/multi-reducer to a different policy than the reducer/multi-reducer is subsequently used with is undefined behavior. This is based on the idea from #2059 that was aimed at fixing #1756.

//before
RAJA::ReduceSum<RAJA::cuda_reduce, double> sum(0.0);
RAJA::forall<RAJA::seq_exec>(RAJA::Range(0, N) [=](int i) { sum += i; });

// after
RAJA::ReduceSum<RAJA::cuda_reduce, double> sum(RAJA::Policy::sequential, 0.0);
RAJA::forall<RAJA::seq_exec>(RAJA::Range(0, N) [=](int i) { sum += i; });

// alternatively
RAJA::ReduceSum<RAJA::cuda_reduce, double> sum(RAJA::policy_of<RAJA::seq_exec>::value, 0.0);
RAJA::forall<RAJA::seq_exec>(RAJA::Range(0, N) [=](int i) { sum += i; });

TODO:

Support seq, omp, cuda, hip Reduction objects
Support omp_target, sycl Reduction objects
Support sequential exec policies with omp_target and sycl reduction objects
Test reduction correctness with other supported backends (cuda/hip supports seq, openmp, and cuda/hip)
Document these changes

Breaking change

Some reducers like sequential and openmp when not given a default value would default construct their value, so min reductions would start with a 0 instead of the max of that type. This was changed so reducers now consistently use the identity value instead, fixing min and max reductions when constructed without explicit initial values. Note that we consider using the default constructor and not supplying an initial value to be deprecated.

This PR is a refactoring, bugfix
It does the following (modify list as needed):
- Modifies/refactors reducer to take a Policy enum arg

MrBurmark · 2026-07-18T07:03:17Z

I re-enabled some sycl and openmp target tests, we'll see if we have to disable them again.

rhornung67 · 2026-07-20T15:30:07Z

Unfortunately, we don't have an OpenMP target CI check at the moment. @rchen20 is working on build scripts for that. Then we can re-enable the CI check.

MrBurmark · 2026-07-20T19:25:31Z

I discovered why the forall openmp target reduction tests were not created, they do not get correct answers. I disabled them again.

rhornung67 · 2026-07-20T19:57:39Z

I discovered why the forall openmp target reduction tests were not created, they do not get correct answers. I disabled them again.

Was this all omp target reduction tests, or just the ones that used the "old-style" reductions (reduction object captured by value)?

artv3 · 2026-07-28T19:07:35Z

+The multi-reduction policy specifies how the multi-reduction is done and must
+support the execution policy. For example, ``RAJA::seq_multi_reduce`` does a
+sequential multi-reduction and can only be used with sequential execution
+policies. The ``RAJA::cuda_multi_reduce_atomic`` policy uses atomics and


The detail that cuda_reducers also supports, omp, and sequential can take a double look when first reading it. Should we just call it auto_*?

Well its not exactly auto if you have to tell it which Policy to support otherwise it allocates cuda memory.

artv3 · 2026-07-28T19:09:08Z

+
+If the same multi-reducer object will later be used with CUDA or HIP loops,
+call ``reset(RAJA::Policy::cuda, ...)`` or ``reset(RAJA::Policy::hip, ...)``
+before that use. Use ``RAJA::Policy::undefined`` only when the object


Suggested change

before that use. Use ``RAJA::Policy::undefined`` only when the object

before that use. Use ``RAJA::Policy::auto`` only when the object

what about calling it auto?

I tend to associate undefined behavior as something that I generally don't want

That was just keeping the existing naming. We may have places where we want the undefined vs the "auto" meaning, we'll have to think about that. Thinking about the name all_supported would be more specific than auto though auto is shorter and less awkward.

could all_compat work?

I also like: support_all

or inspired by raja, the forall policy

I went with all_supported before I saw these suggestions.

artv3 · 2026-07-28T19:19:27Z

  sycl
 };

+constexpr const char* get_policy_name(Policy p)


This could also be a std::string_view if you choose

and mark the function noexcept

The unfortunate thing about string_view is that you can't assume that its null-terminated. I think some things are better left const char* for that reason, but it depends on how its used.

artv3

looks good to me!

Only have a non-defaulted constructor and reset. This affects the base, seq, and omp combiners.

This allows you to pass a RAJA::Policy enum that indicates which type of policies the reducer will be used with. For example cuda reducers may be used with cuda, openmp, and sequential loop policies. If you pass in RAJA::Policy::sequential then the reducer only supports sequential policy types and can avoid allocating resources required for openmp or cuda. Note that passing in Policy::undefined is an explicit way to get the current behavior where the reducer may be used with any of the policies it supports. Note that using a reducer with different loop policy types than the Policy type passing in results in undefined behavior.

Identify which objects are user facing and which are not.

Note that we've only done this for seq, omp, cuda, and hip. This also makes the tests more ugly by having to use API_TAG when constructing and resetting reducer objects.

The current support is not correct and the old reducer is deprecated with sycl so I'm trying to make it work correctly with the default sycl resource at least as it did previously.

instead of openmp policy header

It was missing the defaulted identity arg

Using get_queue by ref was an issue as Sycl::get_default returns by value.

Add Policy enum that specifies that all the policies that the reducer should be able to be used with all of the policies that it supports.

from policy_supported

bechols97 · 2026-07-30T20:02:49Z

+           concepts::enable_if_t<
+               type_traits::is_range<Container>,
+               concepts::negate<std::is_convertible<Container, size_t>>,
+               concepts::negate<std::is_base_of<BaseMultiReduce, Container>>>* =


(minor) Do we want this using C++20 concepts instead of the old enable_if_t style?

We are moving in that direction. @johnbowen42 did a lot of transformation already, but I suspect that there is more to do. We can continue to push on that after the release. IMO, this should not hold up the release.

rhornung67 · 2026-07-30T20:09:16Z

FYI, tuo is down for maintenance. We have the same CI checks on tioga and tuo. tioga checks are green. @MrBurmark to maintain momentum, we can merge this while tuo is down. It's your call.

MrBurmark · 2026-07-30T20:38:43Z

FYI, tuo is down for maintenance. We have the same CI checks on tioga and tuo. tioga checks are green. @MrBurmark to maintain momentum, we can merge this while tuo is down. It's your call.

@rhornung67 Go ahead and merge it.

rhornung67 · 2026-07-30T20:42:43Z

@MrBurmark you don't want the honors?

MrBurmark force-pushed the feature/burmark1/reduce_platform branch from e72308f to f1a93e2 Compare July 17, 2026 21:45

MrBurmark changed the title ~~Give Runtime Policy Informatio to Reducers~~ Give Runtime Policy Information to Reducers Jul 17, 2026

MrBurmark force-pushed the feature/burmark1/reduce_platform branch from f1a93e2 to 134e35d Compare July 17, 2026 21:48

MrBurmark marked this pull request as ready for review July 17, 2026 21:49

MrBurmark requested review from a team, adayton1, artv3, rchen20 and rhornung67 July 20, 2026 17:59

MrBurmark force-pushed the feature/burmark1/reduce_platform branch 2 times, most recently from 64db237 to 4ad1093 Compare July 27, 2026 18:31

artv3 reviewed Jul 28, 2026

View reviewed changes

artv3 approved these changes Jul 29, 2026

View reviewed changes

MrBurmark added 13 commits July 29, 2026 09:34

Add get_name func for policy enums

06b2567

Simplify combiners

227e1e3

Only have a non-defaulted constructor and reset. This affects the base, seq, and omp combiners.

cuda/hip reduce unification pass

77dcd3d

Add Policy to cuda and hip reducers

062337b

Add comments to reducer objects.

d153a75

Identify which objects are user facing and which are not.

Add tests for new runtime reductions

f1d464b

Note that we've only done this for seq, omp, cuda, and hip. This also makes the tests more ugly by having to use API_TAG when constructing and resetting reducer objects.

Update openmp target and sycl reducers

2936e92

make style

927f5f2

Add omp, sycl forone, update ompt forone

e785d7f

Add reducer support to cuda/hip forone

6b627e8

Add more tests using forone

7277fe6

Refactor reduce-exec test matrix

50be753

MrBurmark and others added 20 commits July 29, 2026 09:37

Fix allocation/deallocation of m_sync_list in reset

478ac46

Add some lifetime/ownership comments in multireduce cuda/hip

b7983bc

Use policy_supported_or_throw in reduce.

26fd364

Use policy_supported in reducers more

7bf4f94

Simplify sycl reducer support to just the default resource

20c0c96

The current support is not correct and the old reducer is deprecated with sycl so I'm trying to make it work correctly with the default sycl resource at least as it did previously.

inlcude openmp target policy header

a3ea0b6

instead of openmp policy header

Fix reset visibility in Combinable

3acf69c

fix style

6252d87

Fix sycl forone

4eb75c7

Fix TargetReduce constructor

dea988c

It was missing the defaulted identity arg

Fix use after free issue

bb8d434

Using get_queue by ref was an issue as Sycl::get_default returns by value.

fix style

2909c37

Add string include

6b043e1

Use all_supported intead of undefined

bf5c823

Add Policy enum that specifies that all the policies that the reducer should be able to be used with all of the policies that it supports.

Check policies in openmp multi-reducers

4e09783

Avoid checking policy twice in openmp/sycl

b26d3ba

Use policy_supported more directly in ompt,sycl

80a06df

rename policy_matches

3cc7f50

from policy_supported

make style

61ba0b8

Bump allocation time on corona

74b2618

MrBurmark force-pushed the feature/burmark1/reduce_platform branch from 3f4f90d to 74b2618 Compare July 29, 2026 16:38

rhornung67 approved these changes Jul 29, 2026

View reviewed changes

Merge branch 'develop' into feature/burmark1/reduce_platform

75cb65e

bechols97 reviewed Jul 30, 2026

View reviewed changes

bechols97 approved these changes Jul 30, 2026

View reviewed changes

rhornung67 merged commit 81fe792 into develop Jul 30, 2026
18 of 20 checks passed

rhornung67 deleted the feature/burmark1/reduce_platform branch July 30, 2026 20:52

	before that use. Use ``RAJA::Policy::undefined`` only when the object
	before that use. Use ``RAJA::Policy::auto`` only when the object

Uh oh!

Conversation

MrBurmark commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking change

Uh oh!

MrBurmark commented Jul 18, 2026

Uh oh!

rhornung67 commented Jul 20, 2026

Uh oh!

MrBurmark commented Jul 20, 2026

Uh oh!

rhornung67 commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

artv3 Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

artv3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhornung67 commented Jul 30, 2026

Uh oh!

MrBurmark commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhornung67 commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MrBurmark commented Jul 16, 2026 •

edited

Loading

artv3 Jul 28, 2026 •

edited

Loading

MrBurmark commented Jul 30, 2026 •

edited

Loading