fix(connectivity): the agent-manager migrate Job races its CiliumNetworkPolicy (Helm applies the CRD kind after the Job) and a Failed Job stalls the connectivity upgrade despite disableWaitForJobs

[teemow]

Problem

On an installation whose Cilium runs policyEnforcementMode: always, the connectivity release's cut-over Job agent-platform-connectivity-agent-manager-migrate-<hash> (#346) fails before it can do anything, and its failure fails the whole connectivity upgrade — which the design explicitly wanted to avoid ("deliberately not a Helm hook: a migrate failure must never fail the upgrade that renders the platform Harness", components.<name>.disableWaitForJobs).

Seen 2026-09-11 on the first installation of the 4.x line, connectivity 4.7.0 (and the 4.5.x hash next to it):

$ kubectl -n kagent logs job/agent-platform-connectivity-agent-manager-migrate-d7d62d20
Error: kagent API v2 is not served (no agenttemplates resource in group kagent.dev: discover API groups:
Get "https://10.96.0.1:443/api?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout); run migrate after the kagent upgrade — nothing was changed

three times (backoffLimit: 2), then

Helm upgrade failed for release agent-platform/agent-platform-connectivity with chart agent-platform-connectivity@4.7.0+c439a9c72d9c:
failed early due to stalled resources: [Job/kagent/agent-platform-connectivity-agent-manager-migrate-596983f4 status: 'Failed']

Two defects:

1. The Job races its own network policy. Its egress (DNS, the apiserver, api.github.com, the chart registry) is the CiliumNetworkPolicy agent-platform-connectivity-agent-manager-migrate of the same release. Helm applies manifests in kind order and a CRD-backed kind (CiliumNetworkPolicy) comes after Job, so the pod starts, is default-denied, times out against the apiserver and exhausts its retries before the policy exists. (The kubernetes flavour's NetworkPolicy sorts before Job and does not have this problem.)
2. A Failed Job stalls the upgrade regardless of disableWaitForJobs. helm-controller's readiness check (kstatus) reads a Job with status.failed as Stalled and fails the upgrade early; disableWaitForJobs only stops waiting for completion. The migrate Job therefore does gate the upgrade, and with it every release that dependsOn connectivity (substrate, kagent, the managers, backstage) — the cut-over stalls on the object that was meant to be harmless.

Worked around on the installation with agentManager.migration.enabled: false (it has no v1alpha2 objects). An installation with agents to migrate cannot do that.

Proposed solution

• Make the Job wait for its policy instead of racing it: an init container that polls until the pod can reach the apiserver (a kubectl get --raw /version loop with a bound), or render the policy as a hook object of an earlier weight (pre-install,pre-upgrade, weight -5, like the hooks identity) so it exists before any manifest Job starts. The init-container wait is flavour-independent and also covers a slow policy propagation.
• Keep a failed migration from stalling the release: helm.sh/resource-policy: keep does not change kstatus; either exclude the Job from the release's health assessment (helm-controller spec.install/upgrade.disableWait is too broad) or run the migration as a CronJob with suspend: true + a one-shot trigger, or as a Job created by the connectivity chart's post-upgrade hook with hook-delete-policy: hook-succeeded and helm.sh/hook-weight such that its failure is reported (hook failed) but the release itself is already deployed. Whichever shape: a Failed migration must leave the connectivity release Ready and be visible in its own status/report.
• backoffLimit 2 with a 32 s apiserver timeout is under two minutes; a real cut-over deserves a longer window.

Acceptance criteria

• On a cluster with Cilium policyEnforcementMode: always, the migrate Job reaches the apiserver on its first attempt (asserted by a lab run or an ATS variant with a default-deny policy).
• A migrate Job that ends Failed leaves the connectivity HelmRelease Ready=True; the failure is surfaced (Job status, the agent-manager-migrate-report ConfigMap, a log line) — asserted by a test that forces a failure.
• make verify-migration covers the new ordering/shape; CHANGELOG and UPGRADE.md updated.