Fixes CUDA generic atomic operation by fixing atomic load

[adayton1]

Summary

• This PR is a bugfix from @bechols97
• It does the following:
  • Uses cuda::atomic_ref::load for atomicLoad instead of atomicOr with a value of 0
  • Uses cuda::atomic_ref::store for atomicStore instead of atomicExchange
  • The old implementation of atomicLoad in CUDA was giving incorrect results in Feature: Device messages #1832
  • Performance should be equivalent or better using atomic_ref, but even if not, correctness matters most
  • The RAJA_ENABLE_CUDA_ATOMIC_REF was never getting defined as intended, but now that we require CUDA 12, cuda::atomic_ref should always be available. The cuda/atomic header gives a very clear error message if an architecture older than sm_60 is used.

[smithsg84]

Is there a reason for changing just the implementation for atomicGeneric and not using the refactored atomic_ref implementation in cuda_atomicCAS_loop?

[adayton1]

Is there a reason for changing just the implementation for atomicGeneric and not using the refactored atomic_ref implementation in cuda_atomicCAS_loop?

cuda_atomicCAS_loop has been around for much longer than atomicGeneric and is used by other operations such as atomicMin and atomicMax without any noticeable problems. This new implementation has not been thoroughly tested or benchmarked, so I'm hesitant to make changes to cuda_atomicCAS_loop directly.

[artv3]

Regarding old implementation of atomicGeneric in CUDA is giving incorrect results is that a CUDA related bug? or why is that happening?

[artv3]

LGTM, just to double check -- we have tests for this right?

[adayton1]

LGTM, just to double check -- we have tests for this right?

Yes, there are tests.

[adayton1]

Regarding old implementation of atomicGeneric in CUDA is giving incorrect results is that a CUDA related bug? or why is that happening?

I don't understand the cause yet. Though the old implementation was using atomicOr(acc, 0) to do the load, so maybe there was something weird happening with that?

[artv3]

Regarding old implementation of atomicGeneric in CUDA is giving incorrect results is that a CUDA related bug? or why is that happening?

I don't understand the cause yet. Though the old implementation was using atomicOr(acc, 0) to do the load, so maybe there was something weird happening with that?

Maybe good to capture in an issue if we don't have one already

[adayton1]

Regarding old implementation of atomicGeneric in CUDA is giving incorrect results is that a CUDA related bug? or why is that happening?

I don't understand the cause yet. Though the old implementation was using atomicOr(acc, 0) to do the load, so maybe there was something weird happening with that?

@bechols97 did some more testing and it appears that using cuda::atomic_ref::load in place of the atomicOr for the load fixed the issues. I'm guessing atomic_ref will be faster than the atomicOr, but even if not, it seems like for correctness we should be using atomic_ref for the load.

[adayton1]

Using cuda::atomic_ref instead of this fixed the bug @bechols97 was seeing.

[MrBurmark]

Interesting, I'm pretty sure I added this and it was passing our tests before.

[adayton1]

Yeah, I'm pretty sure we have testing for it, though not tests that use the result in other atomic operations.