Drain the runner manager gracefully on rollout and eviction

[alekc]

The runner manager pod gets no terminationGracePeriodSeconds, so it inherits the Kubernetes default of 30 seconds, and it gets no preStop hook. Anything that rolls the deployment kills the manager 30 seconds later, and every job it was tracking dies with it.

That is not a rare event. The operator rolls the deployment on a config change and on a runner image bump, and node drains and spot evictions do the same thing from below. A CI job that takes longer than 30 seconds, which is to say all of them, is lost rather than finished.

Thirty seconds is also not the real fix on its own, because of how the runner handles signals. gitlab-runner drains gracefully on SIGQUIT: it stops asking for new jobs and waits for the running ones. Kubernetes sends SIGTERM. Upstream hit this in the official chart and added a preStop hook that sends SIGQUIT and waits, alongside a configurable grace period (gitlab-org/charts/gitlab-runner!150), which is the pattern to copy.

So the change is two things, and one without the other does not work:

• a configurable terminationGracePeriodSeconds on the manager pod, defaulting to something more useful than 30s;
• a preStop hook that triggers the graceful path so the grace period is actually spent draining rather than idling before a kill.

Design questions:

• Default. Long enough to drain a typical job, short enough that a rollout is not indefinitely blocked. Upstream's chart default and the job timeouts people actually run are the inputs here.
• Whether the hook belongs in the operator unconditionally, or behind a field. Draining is what almost everyone wants, but it does make a rollout slow, and a Runner whose jobs are all seconds long gains nothing.
• Interaction with the finalizer. A delete already waits on deregistration from GitLab; worth checking the two waits compose rather than fight.
• Note that executor_config.terminationGracePeriodSeconds is a different, inert field (see docs(api): document that terminationGracePeriodSeconds does nothing #72) and is not related. This one is about the manager pod, not the build pod.