Define adjacent-release HA update qualification

deployment-files/ha/README.md

@@ -262,3 +262,6 @@ A passive update rechecks the local role, active peer, and control path just
 before stopping Fleet. This crash-only profile accepts the small role-change
 window before process exit; if the peer fails in that interval, durable update
 recovery restarts the local application.
+After the baseline passes, use the
+[update qualification procedure](UPDATE_QUALIFICATION.md) for each adjacent
+release pair supported by the HA application updater.

deployment-files/ha/UPDATE_QUALIFICATION.md

@@ -0,0 +1,219 @@
+# Proto Fleet HA update qualification
+
+Use this report to qualify one adjacent update, `N` to `N+1`, on the supported
+three-host HA profile. Redact addresses, credentials, certificates, device
+names, and customer data from committed evidence.
+
+Run the full procedure and keep a separate result table for every architecture
+enabled by the release. The initial profile requires both amd64 and arm64 to
+pass before promotion.
+
+Publish the final `N+1` tag and assets at the fixed official release origin as
+a GitHub prerelease. HA does not offer UI-triggered updates, so only an explicit
+local operator command can select this candidate. After every gate passes,
+promote that same release without changing its tag or assets. A failed report
+leaves the release marked prerelease and unsupported.
+
+## Test identity
+
+| Field | Value |
+| --- | --- |
+| Source release and commit (`N`) | Pending |
+| Target release and commit (`N+1`) | Pending |
+| Source and target artifacts and SHA-256 digests | Pending |
+| Architecture and operating system | Pending |
+| Started and completed | Pending |
+
+## Procedure
+
+Set `SOURCE_RELEASE` and `TARGET_RELEASE` to the recorded canonical release tags
+and run Fleet commands as
+`sudo /opt/proto-fleet/deployment/ha/fleet-ha update "$TARGET_RELEASE"`.
+On both application hosts, set
+`PROTO_FLEET_HA_QUALIFICATION_TARGET=$TARGET_RELEASE` in the root-owned
+`/etc/proto-fleet/updater.env`, then restart `proto-fleet-updater.service`.
+This permits only that exact prerelease during qualification. Remove the setting
+and restart the updater after the run, whether the report passes or fails.
+Every clean reinstall recreates `updater.env`, so restore this setting and
+restart the updater before each later update attempt. Before triggering an
+update, require both the exact setting below and an active service on each host:
+
+```bash
+sudo grep -Fx "PROTO_FLEET_HA_QUALIFICATION_TARGET=$TARGET_RELEASE" /etc/proto-fleet/updater.env
+sudo systemctl is-active --quiet proto-fleet-updater.service
+```
+
+1. Verify the stable `N` and prerelease `N+1` artifacts and their checksums. Confirm
+   the target was built from a reviewed
+   `qualified-update-from.txt` containing exactly `SOURCE_RELEASE`, and that
+   its manifest-covered `ha_update_from` field matches. An empty file makes the
+   target clean-install-only. Confirm
+   every `N+1` migration is expand-only and remains usable by `N`; reject
+   drops, renames, narrowed types, and newly required values. Apply the
+   migrations to a representative production-size copy of an `N` database and
+   run `N`'s affected database and API integration tests against it. Require
+   completed baseline reports from [QUALIFICATION.md](QUALIFICATION.md) for the
+   exact `N` and `N+1` artifacts, then install `N` for this procedure and load
+   a sanitized production-scale `N` dataset. Record table sizes and row counts.
+   The passive-first update below must run its actual migrations against this
+   dataset while the sustained reads, writes, and commands in step 3 continue;
+   fail on a migration lock or probe gap beyond the stated bounds.
+2. Record the etcd container ID and start time on all three hosts, the Patroni
+   container ID and start time on both database hosts, and
+   `pg_postmaster_start_time()` on both PostgreSQL members.
+3. Before running the update command on the passive application host, start
+   the external append-only recorder, gap rejection, and uncertainty-inclusive
+   interval rules from [QUALIFICATION.md](QUALIFICATION.md). Confirm initial
+   snapshots and gap-free streams, then continuously record direct active
+   health on both hosts with monotonic timestamps at 100 ms or faster. In
+   parallel, stream interface address events from both hosts and fail on any
+   active or VIP overlap. Also start
+   authenticated database-backed reads and uniquely identified idempotent
+   commands through the VIP at least once per second. Give each request a
+   two-second deadline, record command submission and terminal-result times,
+   and require each background command to reach SUCCESS exactly once within 60
+   seconds. FAILED is permitted only for the deliberately interrupted
+   PROCESSING commands in steps 4 and 7, with the expected restart interruption
+   reason.
+   Before the planned handoff, fail if successful reads or command
+   submissions are more than three seconds apart. Continue these probes through
+   step 4. Verify the active host continues serving `N` throughout the migration
+   and mixed-version window.
+4. Keep the recorder and probes from step 3 running. Before the handoff, hold
+   one `N` command in PROCESSING and one
+   successor in PENDING. Then run the completion command. Probe
+   `/api-proxy/health/active`, `/api-proxy/health`, and an authenticated
+   database-backed request at least once per second. On the same monotonic
+   clock, measure from the last successful pre-handoff VIP probe until all
+   three VIP probes succeed and `/api-proxy/health` reports
+   `X-Proto-Fleet-Version: N+1`; require less than 15 seconds. Verify the former
+   active rejoins as passive. After takeover, require the PROCESSING row to
+   fail once without replay, the PENDING row to dispatch on
+   `N+1`, and both accepted IDs to reach exactly one terminal result within 60
+   seconds. Using a client loaded from `N` before handoff,
+   perform a persisted read and a uniquely identified idempotent command against
+   the `N+1` VIP. Before reinstalling for step 5, confirm the infrastructure
+   identities from step 2 are unchanged.
+5. Reinstall the clean `N` baseline and update the passive host to `N+1`. On the
+   active host, create the root-owned
+   `/var/lib/proto-fleet-updater/qualification-pause-after-ha-stop` file and
+   arrange to remove it on every exit. Start `--complete`, wait until updater
+   status enters the activating phase, and prove the old Fleet containers have
+   stopped while the peer remains passive. The barrier now holds the updater
+   immediately before its takeover wait. Before the old lease can expire, stop
+   the updated peer's Fleet application and confirm it never serves the VIP,
+   then remove the barrier. Require the updater to enter the takeover wait,
+   remain there for its full timeout, and restart release `N` automatically.
+   Within 60 seconds, require the old host to serve the VIP, a persisted read,
+   and a successful command, with no manual recovery command remaining. Restart
+   the `N+1` peer as passive and require control readiness with version mismatch
+   as the only failover-readiness degradation. Step 6 reinstalls a common
+   baseline before applying the full-readiness gate.
+6. Reinstall the clean `N` baseline and repeat steps 2 and 3 for a separate
+   mixed-version failover run. Update the passive host to `N+1`, then terminate
+   the active `N` Fleet process instead of running `--complete`. Require the
+   `N+1` peer to take over within 15 seconds with no possible active or VIP
+   overlap,
+   recover interrupted command state, and serve a successful command. Restart
+   the `N` host as passive and update it with the ordinary passive command.
+   Require both hosts to run `N+1`, full failover readiness, and unchanged
+   infrastructure identities from this run's step 2.
+7. Using a controllable test device on the current active host, stall one command
+   after its exact queue row reaches PROCESSING so its successor remains PENDING.
+   SIGSTOP that active `fleet-api`, let its lease expire, and wait for the passive
+   peer to take over. Queue the old plugin result while its process is stopped,
+   then SIGCONT it. Verify the original PROCESSING row is FAILED with the restart
+   interruption reason and is never dispatched again, the old process exits, its
+   stale database transition is rejected, exactly one terminal result remains,
+   the PENDING successor dispatches, and later work for that device succeeds. Reuse the
+   direct active-health and interface-address streams from step 3 through
+   convergence and fail on any active or VIP overlap. Force a second failover
+   in the other direction with the same ownership streams. For both moves,
+   require the former active to reject an active-only request, measure usable
+   service from the last successful pre-failover probe on the same monotonic
+   clock, require less than 15 seconds, and submit a successful command.
+8. Confirm the etcd and Patroni container IDs and start times, and both
+   PostgreSQL postmaster start times, are unchanged from step 2. The application
+   update must not replace or restart these services.
+9. Confirm both updater binaries report `N+1` and their services and local
+   status sockets are healthy. Reboot the two application/database hosts one
+   at a time, restoring full readiness between reboots. Repeat the updater
+   checks and verify both hosts retain Fleet data and can serve a command.
+10. Run these four cases separately, reinstalling the clean `N` baseline and
+    restoring control and failover readiness before each one:
+
+    - For the pre-stop case, first run the ordinary passive update on the peer
+      and require it healthy and passive on `N+1`. On the active `N` host,
+      create the root-owned
+      `/var/lib/proto-fleet-updater/qualification-pause-before-ha-stop` file,
+      run `fleet-ha update "$TARGET_RELEASE" --complete`, and wait for the
+      activating phase while `N` still serves the VIP. SIGKILL the updater,
+      remove the barrier, and require the old active to keep serving while the
+      updater restarts cleanly.
+    - For the two-rename case, create the root-owned
+      `/var/lib/proto-fleet-updater/qualification-pause-between-deployment-renames`
+      file on the passive `N` host and run the ordinary passive update.
+      Power-cycle only after updater status is activating,
+      `/opt/proto-fleet/deployment` is absent, and
+      `/opt/proto-fleet/deployment.previous` exists. Remove the barrier after
+      reboot and require startup to restore `N`.
+    - For the forward-activation case, run the ordinary passive update on the
+      passive `N` host. Power-cycle after the activation marker clears but
+      while updater status is still activating and the target application is
+      not healthy. Require restart recovery to retain `N+1`, finish startup and
+      migrations, and clear pending recovery.
+    - For the self-update case, run the ordinary passive update on the passive
+      `N` host. During self-replacement, obtain the updater service's PID and
+      require its `/proc/<pid>/cmdline` to contain exactly
+      `--self-update-handoff=/usr/local/libexec/proto-fleet/proto-fleet-updater`
+      while the production updater socket is not accepting status requests,
+      then SIGKILL that process. The handoff marker alone is insufficient
+      evidence because it exists before the replacement process starts.
+
+    Repeat a case if its required evidence clears before the fault lands.
+    After each restart, require one valid deployment directory, no pending
+    recovery marker, a healthy updater socket, control readiness, and either
+    the intact old release or fully verified target release according to the
+    recorded recovery state. While hosts differ, require version mismatch to be
+    the only failover-readiness degradation. Then converge both hosts to the
+    expected retained version, by reinstalling `N` after old-release recovery
+    or running `--complete` on the remaining active `N` host after forward
+    recovery, and require full failover readiness. When recovery retains the
+    target release, require the installed updater's `--version` to report the
+    same target version;
+    startup repair must not leave the previous updater paired with the new
+    application.
+
+## Results
+
+| Gate | Required result | Duration | Result | Evidence |
+| --- | --- | --- | --- | --- |
+| Baseline qualification | Exact `N` and `N+1` artifacts pass the clean-install HA report | N/A | Pending | Pending |
+| Migration compatibility | `N` integration tests pass against an `N` database migrated by `N+1` | N/A | Pending | Pending |
+| Live production-scale migration | The real passive updater migrates the recorded sanitized dataset without exceeding traffic or lock bounds | N/A | Pending | Pending |
+| Passive-first update | Passive runs `N+1`; active continues serving `N` | N/A | Pending | Pending |
+| Mixed-version operation | Continuous persisted reads and commands succeed throughout migration while hosts run `N` and `N+1` | N/A | Pending | Pending |
+| Active completion | Active and database-backed probes serve `N+1`; former active rejoins passive; cached `N` client works against `N+1` | `<15s` | Pending | Pending |
+| Takeover timeout recovery | Updated peer is unavailable; old `N` restarts automatically and serves reads and commands | `<60s` | Pending | Pending |
+| Handoff fencing | High-frequency direct health and interface event streams never show active or VIP overlap | N/A | Pending | Pending |
+| Mixed-version forced takeover | `N+1` takes over safely while its peer is `N`; the old host then completes through the ordinary passive update | `<15s` | Pending | Pending |
+| Post-update stale completion | Interrupted row is FAILED without replay; resumed old transition is rejected; PENDING successor and later work succeed | N/A | Pending | Pending |
+| Failover to peer | No active or VIP overlap; old active rejects; VIP serves `N+1`; command succeeds | `<15s` | Pending | Pending |
+| Failover back | No active or VIP overlap; old active rejects; VIP serves `N+1`; command succeeds | `<15s` | Pending | Pending |
+| Infrastructure preserved | etcd and Patroni identity plus PostgreSQL start times are unchanged | N/A | Pending | Pending |
+| Updater refresh | Both updater binaries, services, and local status sockets run `N+1` | N/A | Pending | Pending |
+| Updater interruption recovery | Pre-activation, post-swap, and self-update faults recover automatically | N/A | Pending | Pending |
+| Reboot recovery | Both hosts return on `N+1` with persisted data and working failover | Pending | Pending | Pending |
+| Promotion drift guard | Source is still latest stable; both tags, commits, and complete asset digests still match this report | N/A | Pending | Pending |
+
+## Verdict
+
+**Pending.** Mark the adjacent update supported only when every gate passes on
+both amd64 and arm64 and attach the redacted evidence to each report.
+Immediately before promotion, re-fetch both releases and re-download every
+source and target asset into separate clean directories. Abort unless
+`SOURCE_RELEASE` is still the latest stable release, the target is still a
+prerelease, both tags resolve to their recorded commits, and both complete
+asset name and SHA-256 digest sets exactly match the report. Then promote it with
+`gh release edit "$TARGET_RELEASE" --prerelease=false --latest`. Do not rebuild,
+retag, replace assets, or promote after any drift.