Add SRU Exception for NVIDIA CUDA

[antlassagne]

This PR will require approval from an SRU team member

I will also update the [TBD] sections if an SRU team member approves.

Description

Add an SRU exception for the CUDA packages that we are newly delivering in 26.04. It's sets of 37+ prebuilt packages whose patches we will want to backport.

[basak]

Hi!

This isn't a full review, but some quick feedback for faster progress. Broadly this seems OK to me, with some things to clarify please:

1. This should explicitly apply to only the multiverse archive component.
2. SRUs into Ubuntu must not change existing behaviour. Like we did with the Intel graphics stack previously, we should have a public statement from upstream as to what "stable" means for them, ensure that this statement is compatible with our policy, and then document that with links to upstream's statement.
3. SRUs into Ubuntu must not regress hardware support, regardless of upstream's view on support lifetimes (Ubuntu has its own). Again, like we did with the Intel graphics stack previously, we should have a public statement from upstream as to what "stable" means for them, ensure that this statement is compatible with our policy, and then document that with links to upstream's statement.

[basak]

Please could you expand on "follow the same schedule"? SRUs do not happen according to SLAs, so may lag.

[basak]

And further, if we find a future SRU to not comply with our policies or user expectations, an SRU may become blocked, at which point it would fall behind "schedule" by definition.

[antlassagne]

Thank you for your message. I can add a little more context about the schedule. The timeline is vague and I wrongly used the term 'schedule' for the versions: we committed to delivering 13-1, 13-2, 13-3, etc.

The good news is that minor versions are new source packages. Only the patches are subject to SRUs. CUDA patchs are minor, a lag is expected, especially if more discussions are required for a given patch set.

[julian-klode]

FWIW: SRU policy also applies to new source packages, and it roughly halves the size of reviewers to those that also have dual power as archive admins.

[antlassagne]

Yes I was not aware of that at the time I was writing this comment. Now I understand that all these new packages will require a SRU process to land in a stable release (which makes sense). I made changes to this PR to reflect that, the most important fact being that 2 binary metas are common to all these version, and therefore will be superseeded by a SRU'd new package (explanation here)

[antlassagne]

Thank you for your review @basak

1. Indeed, I'll mention it in the doc

2. Can you link me to such a commitment from intel? I was not able to find the right link looking in the Intel-Graphics-Update exception.
   I think NVIDIA makes such commitments already:

• Compatibility across minor release, including binary compatiblity, which ensure that it's also the case with patchs.
• Backward and forward compatibility with the GPU driver
  Does it seem good enough to you?

3. The above links are very much a commitment to that point, too.

A little example might be helpful.

Example:

13.1.1 release note can be found here. 13.2 is a new source package, it's only 13.1.1 that owuld have been subject to SRU (referred as Update 1).
If we watch libcusparse closely, we can read

3.4.1. cuSPARSE: Release 13.1 Update 1

New Features

    Added a new cusparseSpMVOp_bufferSize API that returns the size of the workspace buffer required for SpMVOp computations. Users provide this buffer when creating cusparseSpMVOpDescr_t, removing internal memory allocations.

    Improved SpMVOp performance on B200. [CUSPARSE-2931] [CUSPARSE-2932] [CUSPARSE-2933]

Resolved Issues

    Fixed an accuracy issue in mixed-precision CSR/COO SpMM computations. [CUSPARSE-2349]

    Fixed an issue in CSR SpMM computations when the input dense matrix has a high number of columns. [CUSPARSE-2301]

I assume this is OK in terms of SRU policy for a closed-source software? Anyway, if one day a given SRU does not comply, we can expect discussions. We can write down here that this exception is to SRU patches and bug fixes, not to blindly back-port anything that comes up under the CUDA name.

[MitchellAugustin]

+1 from a DGX (Server) team perspective. Only one minor suggestion (to specify that we'll test on both server and workstation hardware each release).

[MitchellAugustin]

must pass on a range of hardware

nit: let's say "must pass on a range of hardware, including at least one workstation and one server GPU"

[antlassagne]

Thank you, indeed that's better, applied

[antlassagne]

Per discussions with the SRU team, it seems that the best way would be to actually submit a few SRUs, the SRU team will then have some more concrete understanding of the matter, and then we will more properly iterate on the matter.

[basak]

Compatibility across minor release, including binary compatiblity, which ensure that it's also the case with patchs.

Backward and forward compatibility with the GPU driver

Does it seem good enough to you?

The SRU team is currently looking at https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-470/+bug/2155202

This does not seem like an upstream that we can trust to not break support for existing hardware. If the answer after a regression is reported is "Won't Fix", then I don't see that we should accept further requests in this class. I suggest that you address that bug urgently and immediately if you wish to keep this request credible.

[basak]

This does not seem like an upstream that we can trust to not break support for existing hardware.

Apologies, that comment was perhaps misdirected, if upstream made it clear they were dropping support. However, the issue still stands that if you do not have the capacity to avoid regressions due to hardware enablements in SRUs, then perhaps you should not be adding more.

[antlassagne]

Hello Robie and thank you for the feedback.

This 470 driver update story is very unfortunate indeed, but it's very different, too. Of course it would have been so much simpler (and better) if NVIDIA have kept delivering security patches for the driver. I also believe that our management of this issue and especially our communication about it could have been better, but it's opinionated and I understand that the kernel team had no solution that would keep everyone happy there.

Now, to this issue in particular, it's quite different for a number of reasons.

• SRUing patches, eg 13.1.1 to be updated to 13.1.2. This should never, ever, be dropping hardware support. If that were to be the case, I suggest that:
  • this fact should be discovered before the SRU is even submitted, by studying release notes for example
  • We would work with upstream to fix that before any SRU. Even them would most probably consider it a bug.
  • In the case were they would refuse, I suppose we could hold the SRU? But I don't think there is the need to resolve every theoretical issue in here; the SRU team would be the ones taking the decision, and PE would be responsible to provide all the necessary information that is needed.
• SRUing new minor releases, eg 13.2.x when only 13.1.x was in the stable release before
  • Most users would not see a difference even if a minor was to break hardware support, given that the recommended installation described in NVIDIA's tutorials is minor-version tighted (e.g. apt install cuda-toolkit-13-1)
  • Some users, the one installing the broader metas (e.g. cuda-toolkit-13 or cuda-toolkit) would receive an update and potentially break the support indeed.
    • As of now, the compatibility seems tight to driver version, so if NVIDIA was to be trusted about this we could consider that cuda-toolkit-MINOR would also never break hardware support. But it's a big if that we don't have control over.
    • The cuda-toolkit meta allows major version upgrades, so it WILL deprecate hardware. I believe that users installing this particular meta would expect such behavior, given that the minor-tight meta exists. But the SRU team knows better about such a case, so I would like opinions.

[basak]

The cuda-toolkit meta allows major version upgrades, so it WILL deprecate hardware. I believe that users installing this particular meta would expect such behavior, given that the minor-tight meta exists.

That tends not to work in practice. I think a general rule is that upgrade of an existing package via the updates pocket must not break any user and users expect this in practice. We'll need to find another way.