Add a command to re-run only pipelines that had system failures

[alalazo]

Lately, the failure rate of pipelines due to system failures started increasing. To give some concrete example from today, here is a pipeline which should be ✔️ but failed because of system failures:

• https://gitlab.spack.io/spack/spack/-/pipelines/912417

On the same day we run two pipelines on develop:

• https://gitlab.spack.io/spack/spack/-/pipelines/912139
• https://gitlab.spack.io/spack/spack/-/pipelines/911863

They are both ❌ due to system failures.

Whenever something like that happens on a PR, the two solutions users have are:

• Close and reopen the PR, to trigger the creation of a new merge commit, and a new set of pipelines
• Comment @spackbot run pipeline, to re-run all the pipelines

If the failure rate is high enough, there is a fair chance the procedure needs to be repeated a few times to get to a ✔️ CI mark. This has the effect of multiplying by a factor the resources we need to run pipelines, in particular for "generate" jobs, which are always re-run.

I guess the best solution from user's perspective would be having a low failure rate but, in absence of that, I wonder if we could add a new command¹:

@spackbot re-run failed pipelines

that re-runs only pipelines that failed due to system errors. This should:

• Reduce the chance of a possible new failure
• Reduce the resources we need to get a given CI run ✔️

Footnotes

1. Naming is tentative, any better choice is welcome ↩

[RobertMaaskant]

I agree this would be useful. Running into the same issue when submitting changes to packages with a lot of dependents. Especially the ml stacks seem affected.

For example py-wheel and py-setuptools:

• https://gitlab.spack.io/spack/spack/-/pipelines/1008735/failures
• https://gitlab.spack.io/spack/spack/-/pipelines/998235/failures

I don't have a strong opinion on the naming, but would suggest something that adds increasing detail. Say @spackbot <subject> <action> <options>. For example: @spackbot pipelines re-run --failed or @spackbot re-run pipelines --failed. That way if it needs to be extended in the future, similar functionality remains grouped together: @spackbot pipelines re-run --<something>.