Skip to content

Drupal chart: Let cron pods carry annotations and suspend be configured - #541

Open
TomiMikola wants to merge 1 commit into
masterfrom
feature/allow-suspend-conf
Open

TomiMikola wants to merge 1 commit into
masterfrom
feature/allow-suspend-conf

Conversation

@TomiMikola

Copy link
Copy Markdown

What changes

Three small, opt-in additions to drupal/templates/drupal-cron.yaml, with values docs, schema entries and unit tests:

  • php.cron.<name>.annotations (map of strings): rendered on the cron job pod template (spec.jobTemplate.spec.template.metadata.annotations).
  • php.cronJobDefaults.annotations (map of strings): the same, for every cron job. A per-cron map wins on a key clash (Helm merge, same precedence as resources and nodeSelector today).
  • php.cron.<name>.suspend (bool, default false): replaces the hard-coded suspend: false.

One schema fix rides along: activeDeadlineSeconds was already read by the template (per cron and in cronJobDefaults) but was missing from values.schema.json, so helm template/helm upgrade rejected it with "Additional property activeDeadlineSeconds is not allowed". It is now allowed as an integer.

Nothing is rendered unless a project sets a value. Existing releases render byte-identical manifests.

Why

Batch crons on an autoscaled node pool get evicted mid-run. In one project, four reindex Jobs on a spot cron pool failed in one night after 3–6× their normal duration, all closing inside one hour (06:00–06:52 UTC). In that window there was no spot preemption (gcloud compute operations list, none on any instance), no node upgrade (last one on the pool the day before) and no auto-repair. The pool's nodes were 126 and 36 minutes old at the read. Cluster-autoscaler scale-down is the only actor left, and the failure shape matches an eviction of a restartPolicy: Never Job pod exactly: SIGTERM then SIGKILL after 30 s, no PHP shutdown, every pod on the node ends in the same minute, one retry from scratch, then BackoffLimitExceeded.

cluster-autoscaler.kubernetes.io/safe-to-evict: "false" is the autoscaler's own switch for "do not drain the node under this pod". Job pods are movable by default because a Job counts as a controller. That is right for a web replica and wrong for a batch run that cannot resume. Today the chart gives no way to set it: the pod template renders labels only, and there is no extraObjects for a PodDisruptionBudget either.

This is a Silta-wide exposure, not a project one: every cron longer than the autoscaler's unneeded-node timer can be killed this way, and the workarounds are worse (a hand-applied PDB is invisible in any repo and dies with the namespace; a node-pool move only relocates the exposure; sharding every long pipeline is engineering around the platform).

suspend is the second half of the same operational story: today a kubectl patch … suspend=true is undone by the next deploy, because the template hard-codes suspend: false. Making it a value lets a project park a misbehaving cron in silta.yml until it is fixed.

Pairing note for users (added to values.yaml comments)

safe-to-evict: "false" pins a node for as long as the pod lives, so a cron that carries it should also carry activeDeadlineSeconds, so a hung job cannot hold a node for days. That is why the schema fix for activeDeadlineSeconds is in this PR.

Example

php:
  cronJobDefaults:
    annotations:
      cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
  cron:
    long_reindex:
      schedule: "0 22 * * *"
      command: drush my:reindex
      activeDeadlineSeconds: 14400
    broken_for_now:
      schedule: "0 3 * * *"
      command: drush other:thing
      suspend: true

What it does not do

  • It does not stop spot preemption. safe-to-evict only speaks to the autoscaler; GCE reclaiming a spot VM still kills the pod.
  • It does not touch the frontend chart, whose services-cron.yaml and backup-cron.yaml also hard-code suspend: false and render no pod annotations. Same change could follow there.
  • It does not add suspend to cronJobDefaults.

Verification

  • helm unittest ./drupal: 213 passed (206 before; 7 new cases: no annotations by default, per-cron annotations, defaults applied to every job, per-cron wins over defaults on a key clash, not suspended by default, suspend=true, activeDeadlineSeconds precedence).
  • helm lint ./drupal --values drupal/test.values.yaml: clean.
  • helm template with --set-string annotations and --set php.cron.drupal.suspend=true renders the annotations block on the pod template and suspend: true; without those values the output is unchanged from 1.37.0.

Long cron jobs on an autoscaled node pool get evicted mid-run when the
cluster autoscaler scales a node down. The Kubernetes switch for that is
the pod annotation cluster-autoscaler.kubernetes.io/safe-to-evict, but
the chart rendered labels only on the cron pod template, so no project
could set it.

Add php.cron.<name>.annotations and php.cronJobDefaults.annotations.
Both are rendered on the cron job pod template, with the per-cron map
winning on a key clash, the same precedence as resources and
nodeSelector. Nothing is rendered when neither is set.

Read spec.suspend from php.cron.<name>.suspend (default false) instead
of hard-coding it. A kubectl patch on the CronJob was undone by every
deploy; now a project can park a cron in silta.yml.

Add activeDeadlineSeconds to values.schema.json for both the per-cron
entry and cronJobDefaults. The template already read it, but the schema
rejected it, so helm upgrade failed with "Additional property
activeDeadlineSeconds is not allowed". It belongs next to safe-to-evict,
which pins a node for as long as the pod lives.

Bump the chart to 1.38.0. Existing releases render the same manifests
apart from the chart label. helm unittest: 213 passed, 7 new.

@ArtisKrumins ArtisKrumins left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have activeDeadlineSeconds fix but rest of PR does not really resolve main issue.
And if crons are running that long they can be scheduled on regular nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants