Update passive HA application hosts

[ankitgoswami]

Reviewable diff: +942/-218 across 15 files (excludes generated, test, and story files).

Summary

Adds application-only updates for a passive HA host while preserving the standalone updater path. Operators choose whether a stable release or release candidate is compatible with their deployment; Fleet verifies the artifact and runtime conditions but does not impose adjacent-release or source-version policy. The update replaces only Fleet API and client services, leaves the HA substrate running, and retains enough recovery state to restart the selected application after an interrupted update.

Stack: #890 -> #891. This diff is relative to main; the guided installer from #888 and promotion-observation guard from #914 are merged prerequisites. This PR owns passive-host update and restart recovery. #891 completes the old active host through bounded failover.

How it works

Installation fixes the updater in HA mode. fleet-ha update VERSION requires the local node to be healthy and passive, then sends an idempotent request over the updater's local Unix socket.

The updater downloads the requested official release bundle and checksum, validates the archive manifest, stages and fsyncs the deployment, and rechecks passivity immediately before stopping Fleet. It loads only the packaged Fleet API and client images. The installed HA infrastructure Compose file remains pinned and is copied into the staged deployment.

After the atomic deployment swap, the updater starts the requested application release and verifies that the host is current, healthy, and passive. It then removes dangling images left by the application replacement. Startup repair reconciles interrupted directory swaps and updater handoffs. If the application was left stopped, the updater retains an app-start recovery command and systemd keeps retrying recovery until the application can run.

The updater requires every target to be newer than the installed release. It does not infer whether a skipped-version update is compatible, so operators must still check release notes and migration requirements before choosing a newer stable or RC release.

sequenceDiagram
  participant O as Operator
  participant H as fleet-ha
  participant U as Host updater
  participant F as Fleet application
  O->>H: update VERSION
  H->>H: Require healthy passive role
  H->>U: Send local idempotent request
  U->>U: Download, verify, stage, and fsync
  U->>H: Recheck passive and stop Fleet only
  U->>U: Atomically swap deployment
  U->>F: Start VERSION and verify passive
  U->>U: Prune replaced application images

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
release workflow and deployment metadata	Publish ordinary version, build, and commit metadata without a source-release allowlist	Check that artifact integrity remains independent of compatibility policy
updater manager, server, and API client	Add fixed HA deployment mode, local idempotent requests, staging, swap, and durable restart recovery	Check standalone remains the default and update state survives interruption
server/internal/ha/deployment/update.go	Validate the passive role, manage Fleet-only stop/start/readiness, and prune dangling images after readiness	Check role enforcement and the infrastructure service boundary
server/cmd/fleet-ha/	Add update VERSION and make cancellation clear that an accepted update may continue	Check the local operator flow and actionable errors
domain update service	Disable UI-triggered updates in HA mode	Check the intentional local-operator boundary
installer, systemd, and Compose	Pin HA infrastructure, clean partial updater starts, and retry interrupted application recovery	Check reboot ordering and substrate preservation

Key technical decisions & trade-offs

• The updater enforces monotonic versions in both standalone and HA modes. Operators still own compatibility for skipped-version updates.
• Release checksums and the deployment manifest protect artifact integrity; compatibility policy is deliberately outside the updater.
• The command updates only Fleet API and client services. It never restarts etcd, Patroni, PostgreSQL, or keepalived.
• Passivity is checked at admission, immediately before stop, and after restart. Crash-only takeover handles the remaining check-to-stop race.
• Pre-start failures keep the current application running. Post-migration failures retain forward-recovery state because an older binary may no longer match the schema.
• Dangling-image cleanup is best effort and happens only after the replacement application is proven ready.
• Existing standalone update behavior is unchanged.

Testing & validation

• Tests cover passive enforcement, HA downgrade rejection, staging and swap, failed preflight, interrupted-layout and updater-handoff repair, Unix-socket idempotency, disabled HA UI updates, and infrastructure preservation.
• Targeted updater, HA deployment, CLI, and updater API tests pass.
• Hermit server lint reports zero issues.
• The HA deployment profile and git diff --check pass.
• Active completion remains in Complete HA updates through bounded failover #891. Real systemd retry and Docker image-prune behavior remain qualification work.

Post-Deploy Monitoring & Validation

• Owner: the operator performing the update.
• Window: from update start until both hosts report the expected application version and failover readiness is restored.
• Healthy signals: fleet-ha status reports one active and one passive host, both Fleet endpoints report the expected version, and etcd, Patroni, and PostgreSQL remain continuously running.
• Failure signals: the passive host does not return healthy, the updater reports pending recovery, or any HA infrastructure container restarts.
• Mitigation: stop the update sequence, inspect the local updater log and fleet-ha status, and follow the emitted recovery command before updating the active host.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (877cb51d03be9522574a919a0ef993d1eb262b2c...18a04f2e67836f0abef02cbba6dfe0facf4e24f2, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: NONE

Findings

No concrete security, correctness, or reliability findings were identified in the changed hunks.

Notes

Reviewed the authoritative .git/codex-review.diff and relevant surrounding control flow. Focused tests could not run because the review environment is read-only and prevented Go cache and temporary-file creation.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[Copilot]

Pull request overview

This PR extends the existing host-updater flow to support HA passive-host application-only updates while keeping the standalone updater behavior intact. It introduces a shared Unix-socket updater client, adds an HA deployment mode to the updater manager, and adds a fleet-ha update VERSION operator workflow that updates only fleet-api and fleet-client on a passive HA node.

Changes:

• Add server/internal/updaterapi HTTP-over-unix-socket client and reuse it from the domain updates executor and fleet-ha.
• Add DeploymentMode (standalone vs ha) to the updater manager to run HA-specific preflight/stop/start commands and support rollback to the previous app on failed start.
• Add HA application update helpers (PrepareApplicationUpdate, StopApplication, StartApplication) plus wiring in installer/docs and HA compose to disable UI-triggered updates.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/internal/updaterapi/types.go	Updates package docstring for the shared local updater protocol.
server/internal/updaterapi/client.go	Adds reusable Unix-socket HTTP client with protocol/transport error types.
server/internal/updater/manager.go	Adds HA deployment mode, HA preflight/activation paths, and HA rollback behavior.
server/internal/updater/manager_test.go	Adds HA-mode coverage (passive-only, rollback, preflight failure, interrupted recovery).
server/internal/ha/deployment/update.go	Adds HA application-only update operations (prepare/stop/start + passive verification).
server/internal/ha/deployment/install.go	Installs and configures the host-updater for HA database nodes; exposes ReleaseRoot and compose arg helpers.
server/internal/ha/deployment/install_test.go	Extends golden/witness install assertions for updater installation and enablement.
server/internal/domain/updates/service.go	Maps executor errors to the new updaterapi error types.
server/internal/domain/updates/service_test.go	Updates tests to use updaterapi error types.
server/internal/domain/updates/executor.go	Replaces bespoke unix executor client with updaterapi.Client.
server/internal/domain/updates/executor_test.go	Updates executor tests for updaterapi.Client and error types.
server/cmd/fleet-updater/main.go	Adds --deployment-mode flag/env wiring into updater manager config.
server/cmd/fleet-ha/main.go	Adds fleet-ha update VERSION and internal HA lifecycle subcommands used by the updater.
server/cmd/fleet-ha/main_test.go	Adds tests ensuring passive enforcement and terminal success reporting for update.
deployment-files/ha/tests/test-profile.sh	Adds contract assertions disabling updates in HA and preventing updater socket mounts into services.
deployment-files/ha/README.md	Documents passive-host update workflow (fleet-ha update VERSION).
deployment-files/ha/fleet-compose.yaml	Sets UPDATES_ENABLED=false for HA services (explicit operator-only updates).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf2fe7e12e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5db3353f74

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c363fa3d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5dc52fcc5f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ccca78ce1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 66cee057f2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".