Drupal chart: Let cron pods carry annotations and suspend be configured - #541
Open
TomiMikola wants to merge 1 commit into
Open
TomiMikola wants to merge 1 commit into
TomiMikola wants to merge 1 commit into
Conversation
Long cron jobs on an autoscaled node pool get evicted mid-run when the cluster autoscaler scales a node down. The Kubernetes switch for that is the pod annotation cluster-autoscaler.kubernetes.io/safe-to-evict, but the chart rendered labels only on the cron pod template, so no project could set it. Add php.cron.<name>.annotations and php.cronJobDefaults.annotations. Both are rendered on the cron job pod template, with the per-cron map winning on a key clash, the same precedence as resources and nodeSelector. Nothing is rendered when neither is set. Read spec.suspend from php.cron.<name>.suspend (default false) instead of hard-coding it. A kubectl patch on the CronJob was undone by every deploy; now a project can park a cron in silta.yml. Add activeDeadlineSeconds to values.schema.json for both the per-cron entry and cronJobDefaults. The template already read it, but the schema rejected it, so helm upgrade failed with "Additional property activeDeadlineSeconds is not allowed". It belongs next to safe-to-evict, which pins a node for as long as the pod lives. Bump the chart to 1.38.0. Existing releases render the same manifests apart from the chart label. helm unittest: 213 passed, 7 new.
ArtisKrumins
requested changes
Sep 14, 2026
ArtisKrumins
left a comment
Contributor
There was a problem hiding this comment.
I would have activeDeadlineSeconds fix but rest of PR does not really resolve main issue.
And if crons are running that long they can be scheduled on regular nodes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Three small, opt-in additions to
drupal/templates/drupal-cron.yaml, with values docs, schema entries and unit tests:php.cron.<name>.annotations(map of strings): rendered on the cron job pod template (spec.jobTemplate.spec.template.metadata.annotations).php.cronJobDefaults.annotations(map of strings): the same, for every cron job. A per-cron map wins on a key clash (Helmmerge, same precedence asresourcesandnodeSelectortoday).php.cron.<name>.suspend(bool, defaultfalse): replaces the hard-codedsuspend: false.One schema fix rides along:
activeDeadlineSecondswas already read by the template (per cron and incronJobDefaults) but was missing fromvalues.schema.json, sohelm template/helm upgraderejected it with "Additional property activeDeadlineSeconds is not allowed". It is now allowed as an integer.Nothing is rendered unless a project sets a value. Existing releases render byte-identical manifests.
Why
Batch crons on an autoscaled node pool get evicted mid-run. In one project, four reindex Jobs on a spot cron pool failed in one night after 3–6× their normal duration, all closing inside one hour (06:00–06:52 UTC). In that window there was no spot preemption (
gcloud compute operations list, none on any instance), no node upgrade (last one on the pool the day before) and no auto-repair. The pool's nodes were 126 and 36 minutes old at the read. Cluster-autoscaler scale-down is the only actor left, and the failure shape matches an eviction of arestartPolicy: NeverJob pod exactly: SIGTERM then SIGKILL after 30 s, no PHP shutdown, every pod on the node ends in the same minute, one retry from scratch, thenBackoffLimitExceeded.cluster-autoscaler.kubernetes.io/safe-to-evict: "false"is the autoscaler's own switch for "do not drain the node under this pod". Job pods are movable by default because a Job counts as a controller. That is right for a web replica and wrong for a batch run that cannot resume. Today the chart gives no way to set it: the pod template renders labels only, and there is noextraObjectsfor a PodDisruptionBudget either.This is a Silta-wide exposure, not a project one: every cron longer than the autoscaler's unneeded-node timer can be killed this way, and the workarounds are worse (a hand-applied PDB is invisible in any repo and dies with the namespace; a node-pool move only relocates the exposure; sharding every long pipeline is engineering around the platform).
suspendis the second half of the same operational story: today akubectl patch … suspend=trueis undone by the next deploy, because the template hard-codessuspend: false. Making it a value lets a project park a misbehaving cron insilta.ymluntil it is fixed.Pairing note for users (added to values.yaml comments)
safe-to-evict: "false"pins a node for as long as the pod lives, so a cron that carries it should also carryactiveDeadlineSeconds, so a hung job cannot hold a node for days. That is why the schema fix foractiveDeadlineSecondsis in this PR.Example
What it does not do
safe-to-evictonly speaks to the autoscaler; GCE reclaiming a spot VM still kills the pod.frontendchart, whoseservices-cron.yamlandbackup-cron.yamlalso hard-codesuspend: falseand render no pod annotations. Same change could follow there.suspendtocronJobDefaults.Verification
helm unittest ./drupal: 213 passed (206 before; 7 new cases: no annotations by default, per-cron annotations, defaults applied to every job, per-cron wins over defaults on a key clash, not suspended by default, suspend=true, activeDeadlineSeconds precedence).helm lint ./drupal --values drupal/test.values.yaml: clean.helm templatewith--set-stringannotations and--set php.cron.drupal.suspend=truerenders the annotations block on the pod template andsuspend: true; without those values the output is unchanged from 1.37.0.