RunnerReconciler builds the runner Deployment with a readiness probe that is a bare TCP dial of the metrics port (internal/validate/validate.go:84-96, TCPSocket on 9090, with liveness doing the same). That goes green as soon as the listener binds, which says nothing about whether the runner can reach GitLab, authenticate, or start a job. A runner with a bad token or no network path to the instance reports Ready and sits there failing quietly.
gitlab-runner v19.2.0 added something better. commands/multi.go registers /health/ready, served by serveHealthReady, which returns 200 once bootVerifyReady is set and 503 with boot-verify pending before that. The gate is a startup canary: a synthetic job the runner runs against its own executor before declaring itself ready.
Why it is not a drop-in
Three things have to be true together, which is why this is one issue and not "switch the probe".
The endpoint is meaningless unless the canary is enabled. runBootVerify iterates the configured runners, keeps only those where GetBootVerify() returns non-nil with Enabled set, and returns early when none match; bootVerifyReady.Store(true) then runs regardless. The operator never writes experimental.boot_verify and config.Config is a fixed struct with no raw-toml escape hatch, so today /health/ready answers 200 the moment the mux binds. It carries exactly as much information as the TCP check. Exposing the config key is the substance here; the probe change is the payoff.
A failing canary is a CrashLoop, not a stuck-unready pod. On failure the run loop stores the error for a non-zero exit, drains the executor providers, and cancels the run. Under a Deployment that surfaces as CrashLoopBackOff. That is arguably the right behaviour for a runner that cannot run jobs, but it is a different failure mode from the one an operator user expects from a readiness probe, and it needs saying in the field's CRD description.
spec.image is user-overridable, so the probe cannot change unconditionally. At v19.1.0 the mux registers only /debug/jobs/list and /debug/process/state, with no catch-all, so Go's ServeMux 404s /health/ready. Anyone pinning a pre-v19.2.0 runner would get a permanently failing httpGet probe and a pod that never reaches Ready. The probe therefore has to be conditional: opt-in alongside the config field, or gated on a parsed image version, rather than flipped for everyone.
Upstream also notes the canary needs the runner's default image to be configured for the docker and kubernetes executors, so exposing the field means deciding what the operator supplies there.
Suggested shape
Expose experimental.boot_verify (enabled, timeout, acquire_min_backoff, acquire_max_backoff) as a CRD field, and switch the readiness probe to httpGet /health/ready only for runners that enable it. That keeps the default behaviour byte-identical for everyone else and confines the version constraint to users who opted in.
Found while reviewing the v19.2.2 release for #61.
RunnerReconcilerbuilds the runner Deployment with a readiness probe that is a bare TCP dial of the metrics port (internal/validate/validate.go:84-96,TCPSocketon 9090, with liveness doing the same). That goes green as soon as the listener binds, which says nothing about whether the runner can reach GitLab, authenticate, or start a job. A runner with a bad token or no network path to the instance reports Ready and sits there failing quietly.gitlab-runner v19.2.0 added something better.
commands/multi.goregisters/health/ready, served byserveHealthReady, which returns 200 oncebootVerifyReadyis set and 503 withboot-verify pendingbefore that. The gate is a startup canary: a synthetic job the runner runs against its own executor before declaring itself ready.Why it is not a drop-in
Three things have to be true together, which is why this is one issue and not "switch the probe".
The endpoint is meaningless unless the canary is enabled.
runBootVerifyiterates the configured runners, keeps only those whereGetBootVerify()returns non-nil withEnabledset, and returns early when none match;bootVerifyReady.Store(true)then runs regardless. The operator never writesexperimental.boot_verifyandconfig.Configis a fixed struct with no raw-toml escape hatch, so today/health/readyanswers 200 the moment the mux binds. It carries exactly as much information as the TCP check. Exposing the config key is the substance here; the probe change is the payoff.A failing canary is a CrashLoop, not a stuck-unready pod. On failure the run loop stores the error for a non-zero exit, drains the executor providers, and cancels the run. Under a Deployment that surfaces as
CrashLoopBackOff. That is arguably the right behaviour for a runner that cannot run jobs, but it is a different failure mode from the one an operator user expects from a readiness probe, and it needs saying in the field's CRD description.spec.imageis user-overridable, so the probe cannot change unconditionally. At v19.1.0 the mux registers only/debug/jobs/listand/debug/process/state, with no catch-all, so Go'sServeMux404s/health/ready. Anyone pinning a pre-v19.2.0 runner would get a permanently failinghttpGetprobe and a pod that never reaches Ready. The probe therefore has to be conditional: opt-in alongside the config field, or gated on a parsed image version, rather than flipped for everyone.Upstream also notes the canary needs the runner's default image to be configured for the docker and kubernetes executors, so exposing the field means deciding what the operator supplies there.
Suggested shape
Expose
experimental.boot_verify(enabled,timeout,acquire_min_backoff,acquire_max_backoff) as a CRD field, and switch the readiness probe tohttpGet /health/readyonly for runners that enable it. That keeps the default behaviour byte-identical for everyone else and confines the version constraint to users who opted in.Found while reviewing the v19.2.2 release for #61.