Guide HA installation on supported apt-based hosts

[ankitgoswami]

Reviewable diff: +2415/-255 across 22 files (excludes generated, test, and story files).

Summary

Adds a guided fleet-ha install flow for dedicated Debian-based HA hosts. Operators no longer create node.env, arrange secrets, or set file permissions by hand: ha-a prepares protected, role-specific host bundles, and each host installs with one local command. Supported targets are Debian 12/13, Ubuntu 22.04/24.04, 64-bit Raspberry Pi OS based on Debian 12/13, and compatible apt/systemd derivatives on amd64 or arm64 with 4096-byte pages.

Stack: #888 -> #914 -> #890 -> #891. This PR targets main and provides the installed paths, services, and local operator workflow used by the HA promotion and application-update PRs above it. Merged #887 supplies local HA status and readiness checks. Host replacement, unattended installation, resume, rollback, SSH orchestration, and exact-artifact qualification remain outside this PR.

How it works

The operator transfers the release archive and official checksum to ha-a, verifies the archive before extraction, and runs fleet-ha install through ssh -t. The wizard prompts for the ha-b, ha-c, and VIP addresses, derives ha-a's LAN address and interface, validates the packaged runtime and dedicated host, shows the complete topology and planned changes, and requires the operator to type INSTALL before mutation.

The wizard creates three architecture-independent JSON host bundles and the public service CA certificate. Each bundle contains only the topology, release identity, and credentials required by its role; only ha-a receives the one-time etcd root password. Before mutation, peer installation validates the protected bundle's size, exact fields, role, local host address, release identity, and certificates. The temporary CA private key is discarded because certificate rotation and host replacement are not supported by this initial installer.

flowchart TD
  O["Operator verifies release and runs fleet-ha install on ha-a"] --> T["Enter peer IPs and VIP"]
  T --> V["Validate runtime, topology, network, and dedicated host"]
  V --> C["Operator types INSTALL"]
  C --> B["Generate three role-specific JSON bundles"]
  B --> A["Install ha-a from its bundle"]
  B --> PB["Copy ha-b bundle over SSH"]
  B --> PC["Copy ha-c bundle over SSH"]
  PB --> IB["Run guided local install on ha-b"]
  PC --> IC["Run guided local install on ha-c"]

Loading

Each host follows stable phases for validation, package setup, configuration, image preparation, service startup, peer waiting, and final readiness. Docker is installed from its official Debian or Ubuntu apt repository when absent. An unused compatible Docker Engine and Compose v2 installation can be reused, as can inactive and unconfigured keepalived. Existing containers, orphaned container-runtime data without a working Docker installation, custom Docker configuration, configured or active keepalived, previous Proto Fleet state, occupied HA ports, and a claimed VIP fail before the release is copied or runtime services are installed.

Release packaging builds architecture-specific Fleet API and client images once, stores them in the checksum-covered images/fleet.tar.gz, and pins Compose to the release version. The installer loads and verifies those exact tags; application startup uses --no-build --pull never, so target hosts do not rebuild Fleet from mutable registry inputs.

The installer places the release under /opt/proto-fleet, root-owned configuration under /etc/proto-fleet/ha, and data under /var/lib/proto-fleet/ha. Systemd persists firewall and startup ordering: networking and firewall precede Docker and etcd, Patroni starts on database hosts, Fleet starts after its dependencies, and keepalived advertises the VIP only when the local active endpoint is healthy.

sequenceDiagram
  participant O as Operator
  participant I as fleet-ha installer
  participant S as systemd
  participant E as etcd cluster
  O->>I: Confirm INSTALL
  I->>I: Install validation prerequisites and check host firewall
  I->>I: Install runtime packages and protected configuration
  I->>S: Enable and start proto-fleet-ha.service
  S->>E: Start local peer listener, then wait for quorum
  E-->>S: Local member joins a responding quorum
  S->>S: Start Patroni and Fleet on database hosts
  I->>I: Run final local readiness check

Loading

ha-a enables etcd authentication only after quorum. On restart, existing authentication must accept the generated root, Patroni, and Fleet credentials and expose the expected roles and DCS permissions; mismatches fail closed. Once the local etcd peer listener opens within one minute, the service remains enabled and waits without a fixed deadline while missing peers converge. Final startup still requires the local member to join a responding etcd quorum. Fatal local setup or service-start failures stop the incomplete local service.

Areas of the code involved

Area / package / file	What changed	Why it matters for review
server/cmd/fleet-ha/main.go	Defines the Kong-based CLI and guided install [HOST_BUNDLE] command	Review the operator surface and TTY boundary
server/internal/ha/deployment/guided_install.go	Implements prompts, JSON bundle generation and validation, role resolution, and bundle consumption	Review credential separation and pre-mutation confirmation
.github/workflows/proto-fleet-artifact-build.yml	Builds architecture-specific Fleet images and embeds them in the checksummed release bundle	Review the release integrity boundary and image tags
server/internal/ha/deployment/install.go and preflight.go	Detect apt/systemd platforms, validate runtime and L2 requirements, load packaged images, and install protected files	Review platform selection, image verification, conflict rejection, and privilege boundaries
server/internal/ha/deployment/start.go and bootstrap.go	Start role-specific services, bound local etcd startup, wait for quorum, and bootstrap or verify etcd authentication	Review startup ordering and fail-closed authentication
deployment-files/ha/	Adds persistent firewall/systemd ordering and the guided two-command-per-host runbook	Review reboot persistence and the documented operator flow
*_test.go and deployment-files/ha/tests/	Covers the guided flow, bundle boundary, supported platforms, conflicts, convergence, and profile contracts	Tests; review with the associated behavior

Key technical decisions & trade-offs

• Host bundles are JSON documents rather than archives, structurally avoiding path traversal and symlink handling.
• The official release checksum covers prebuilt Fleet and database image archives; target hosts load release-tagged images instead of rebuilding them.
• Bundle transfer relies on SSH integrity plus protected-file, exact-field, release, host, and certificate validation; redundant per-bundle checksum sidecars are omitted.
• Only role-scoped bundles and the public CA leave the generation workspace. The private CA key is discarded instead of creating an unsupported recovery workflow.
• Service certificates use a ten-year lifetime; certificate rotation remains deferred for this fixed deployment.
• Existing package caches, images, volumes, and networks are allowed when a working, container-free Docker installation is present. Orphaned Docker or containerd state is rejected rather than trusted.
• The HA Fleet container drops inherited NET_ADMIN and retains only NET_RAW for nmap discovery.
• Database peers and the VIP must be directly connected on the configured interface because VRRP/ARP cannot move the VIP across routed topology.
• One apt/systemd path covers the documented targets. RPM systems, non-systemd hosts, 32-bit ARM, and automatic base-release overrides remain out of scope.
• Installation is local and one-shot. There is no SSH controller, enrollment service, resume engine, or rollback state machine.

Testing & validation

• go test ./cmd/fleet-ha ./internal/ha/deployment -count=1
• ./deployment-files/ha/tests/test-profile.sh
• Hermit-backed just _lint-server
• git diff --check
• Guided transcript, role-scoped bundle contents, protected modes, exact-field validation, release/host binding, and total-size bounds.
• Debian, Ubuntu, Raspberry Pi OS, Ubuntu-derivative, and generic apt-derivative detection, including unsupported architecture, missing prerequisites, and 16K-page rejection.
• Dedicated-host reuse, orphaned runtime and firewall conflict cases, minimal container capabilities, role-specific startup, etcd authentication bootstrap and restart, and slow or absent peer convergence.
• Real-host installation and failover remain part of exact-artifact qualification.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (55dfa2292fec28b0f87e8f4d5eb5ef29432bb817...3591848c1388eef4a6490740b866975f29227959, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: MEDIUM

The HA installer has three material reliability and credential-lifecycle risks.

Findings

[MEDIUM] Secret identities are no longer cryptographically validated

• Category: Reliability
• Location: server/internal/ha/deployment/preflight.go:293
• Description: validateSecrets now returns after checking only file presence, ownership, permissions, and password shape. The changed code removes certificate-chain, SAN/EKU, certificate/private-key, and JWT public/private-key verification.
• Impact: A corrupted, mixed, or incorrectly prepared host bundle passes preflight and is installed. etcd, PostgreSQL, Patroni, or nginx can then fail after the installer has mutated the dedicated host, potentially requiring a reimage.
• Recommendation: Restore validation that every certificate chains to the supplied service CA, carries the expected node or VIP identity and EKU, and matches its private key; also verify the JWT key pair before installation begins.

[MEDIUM] Automatic authentication bootstrap cannot recover from partial commits

• Category: Reliability
• Location: server/internal/ha/deployment/bootstrap.go:158
• Description: EnsureRole and EnsureUser are direct RoleAdd and UserAdd calls. If an operation commits but its response is lost, or a later bootstrap step fails, the next systemd-driven attempt aborts on the already-existing role or user before authentication is enabled.
• Impact: A transient failure can permanently wedge initial cluster installation with partially replicated authentication state, requiring manual etcd cleanup or wiping multiple nodes.
• Recommendation: Make bootstrap retry-safe by reconciling existing users, roles, permissions, and passwords against the exact intended policy while rejecting extra privileges. Add fault-and-retry tests after every bootstrap step.

[MEDIUM] Leaf certificates now remain valid for ten years without rotation

• Category: Auth
• Location: server/internal/ha/deployment/secrets.go:254
• Description: All service certificates now inherit the CA's ten-year validity instead of the previous 825-day lifetime. The PR also stops retaining the CA private key, while certificate rotation remains explicitly deferred.
• Impact: A stolen host private key can be used to impersonate its etcd, PostgreSQL, Patroni, or Fleet endpoint for up to ten years, with no supported revocation or renewal path short of replacing the cluster trust root.
• Recommendation: Keep leaf lifetimes substantially shorter and provide a protected offline CA/rotation procedure before discarding renewal capability.

Notes

No changed protobuf wire definitions or mining-pool/wallet configuration were found. git diff --check passed; Go tests could not run because the read-only environment prevented creation of the module cache.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c2d93b7ee

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[ankitgoswami]

🤖 The two current Codex Security Review findings describe accepted reliability tradeoffs for the first guided-installer release:

• Partial etcd authentication bootstrap: the finding is accurate that a failure after creating only part of the auth policy is not automatically recoverable. Making this retry-safe would require reconciliation of users, roles, grants, permissions, and generated passwords. That is effectively installer resume/repair machinery, which this PR deliberately does not support. The install runs on a dedicated new host before the HA control plane is considered ready, and the documented recovery for an interrupted partial install is to reimage the dedicated hosts and rerun the guided installation. We are keeping that bounded operational recovery instead of adding a second bootstrap state machine.

• Certificate/key relationship validation: host bundles are generated by this installer, decoded through a strict typed JSON schema, checked for the exact role-specific secret set, matched to the local node IP and release, protected by ownership/mode checks, and transferred over SSH. Full X.509 chain, SAN/EKU, key-pair, and JWT-pair validation would detect a corrupted generated bundle earlier, but it would duplicate validation already performed when etcd, Patroni, PostgreSQL, nginx, and Fleet load those identities. It also would not establish bundle provenance against deliberate substitution. For this dedicated-host flow, a malformed bundle failing during startup has the same documented remediation: reimage and rerun.

These are not being treated as false positives. We are accepting the narrower failure behavior to keep this PR focused on the golden installation path and avoid roughly a few hundred lines of recovery and cryptographic prevalidation code. If qualification shows either failure mode occurring in normal operation, that evidence would justify expanding the installer.

[ankitgoswami]

🤖 > HA firewall service cannot restart while its table exists

Fixed in c5740ec. The service now ensures the profile-owned table exists, then validates and applies one nftables transaction that deletes and recreates it. On restart, the existing filtering rules remain active until nft atomically commits the replacement. The Go preflight path uses the same transaction shape.

I kept validation focused rather than adding a privileged CI harness: the deployment profile and targeted Go tests pass, and the replacement transaction was applied twice successfully against nftables v1.1.3 inside an isolated network namespace on the qualification hardware.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5740ec42c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3591848c13

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".