Tolerate transient HA promotion observations

[ankitgoswami]

Reviewable diff: +82/-35 across 2 files (excludes generated, test, and story files).

Summary

Keeps Fleet available when Patroni and the connected PostgreSQL server briefly report different timelines during promotion. Fleet now treats that mismatch as unavailable ownership evidence and retries within the last valid proof's existing deadline; convergence avoids an unnecessary process restart and VIP withdrawal, while persistent disagreement still causes crash-only demotion.

Stack: #888 -> #914 -> #890 -> #891. This diff is relative to #888, which supplies the HA observer, ownership lease, crash-only runtime, and keepalived health contract. #890 and #891 remain responsible for passive and active application updates. Guided-installer behavior, update behavior, keepalived policy, topology, and qualification evidence are out of scope here.

How it works

The observer still requires one consistent ownership sample: the Patroni primary and connected PostgreSQL server must agree on address, port, recovery state, and timeline before the lease store can acquire or renew ownership. A mismatch therefore never writes or renews the Fleet ownership lease. Before returning the mismatch, the observer repeats its linearizable DCS read, requires the cluster, leader, writer generation, lease, and member endpoints to remain unchanged, and verifies that the unchanged leader lease is still live.

Before activation, the existing coordinator loop leaves the node passive and retries after its normal interval. After activation, only a timeline mismatch keeps the active lifetime alive. The coordinator marks the current observation unavailable but preserves the prior ownership, proof deadline, and watchdog timer unchanged. A matching observation before that deadline renews ownership and restores current status. If disagreement persists, the unchanged watchdog cancels the active lifetime, closes the command gate, aborts the runtime, and lets keepalived withdraw the VIP.

flowchart TD
  P["Patroni reports primary identity"] --> O["Fleet ownership observer"]
  D["Connected PostgreSQL reports server identity"] --> O
  O --> M{"Address, port, recovery, and timeline match?"}
  M -->|"Yes"| R["Acquire or renew ownership proof"]
  M -->|"No timeline match"| U["Observation unavailable; no lease write"]
  U --> C{"Prior proof still valid?"}
  C -->|"Yes"| T["Retry on existing coordinator cadence"]
  C -->|"No"| A["Crash-only runtime abort"]
  T --> O
  A --> V["keepalived withdraws VIP"]

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
server/internal/ha/observer.go	Closes the DCS observation and validates the unchanged leader lease before returning a timeline mismatch, without running the lease action or creating a fresh proof	Review the unchanged-DCS and live-lease checks plus the partial-observation contract
server/internal/ha/observer_test.go	Covers partial mismatch evidence, DCS writer changes, and expired leader-lease precedence	Tests; review the fail-closed observation boundaries
server/internal/ha/coordinator.go	Retains active lifetime only when the mismatched sample still names the held DCS cluster and writer generation	Review the writer-generation guard and unchanged expiry behavior
server/internal/ha/coordinator_test.go	Proves a mismatched sample from a new writer term still demotes immediately	Tests; review the writer-generation guard
server/internal/ha/runtime_test.go	Uses fake time to cover passive convergence, active convergence without proof extension, and persistent expiry	Tests; review process survival, unchanged watchdog, and crash-only abort assertions

Key technical decisions & trade-offs

• Only ErrTimelineMismatch bracketed by unchanged DCS snapshots for the held writer gets bounded convergence handling; a changed cluster, leader, writer generation, lease, member endpoint, wrong PostgreSQL identity, or DCS failure remains fail closed.
• The last valid proof survives only until its existing lease/DCS deadline; the failed observation does not update ownership, freshness timestamps, or the watchdog.
• Retries use the existing coordinator cadence. There is no promotion sleep, retry counter, new timer, state machine, or configuration.
• The observer and SQL identity checks remain strict. The coordinator owns the narrow decision to retain an already-valid active proof.
• keepalived remains unchanged and continues to follow Fleet's active endpoint.

Testing & validation

• Focused observer, coordinator, runtime, endpoint, and fleetd tests pass.
• Focused tests prove an expired leader lease takes precedence over mismatch evidence; a passive process survives one mismatch and activates only after a matching sample; an active process survives same-writer convergence before expiry; a writer change during observation demotes immediately; persistent mismatch aborts at the unchanged proof deadline; and no mismatched sample renews the lease.
• HA deployment profile checks pass.
• Hermit-backed just _lint-server reports zero issues.
• git diff --check passes.
• Exact-head Raspberry Pi qualification passed with v0.2.10-ha-promotion.2 at 25b6dcac28e5750a49dc3106cb51fe82b331c6aa: Patroni promoted at 21:38:58.325Z, PostgreSQL accepted connections at 21:38:58.398Z, Fleet opened active jobs at 21:39:00.196Z, and keepalived entered MASTER at 21:39:04Z.
• The one-second VIP probe last succeeded at 21:38:54Z, first failed at 21:38:55Z, and returned stable HTTPS beginning at 21:39:04Z (9-second interruption, below the 15-second target). The promoted Fleet container kept the same ID and start time with restart count zero; no timeline-mismatch abort occurred. The old active rejoined as a synchronous standby with about 6 ms replay lag, and both hosts returned to failover_ready: true.

Post-Deploy Monitoring & Validation

• Search Fleet logs for Patroni and PostgreSQL timelines do not match, active lifetime ended, and HA Fleet runtime aborted around promotion.
• Confirm the promoted peer reaches a matching timeline before the existing ownership deadline and that no lease renewal is recorded for a mismatched sample.
• Watch one-second HTTPS probes through the VIP and keepalived state transitions during the three-node reboot test.
• Healthy result: no Fleet container restart, one VIP move, stable HTTPS within 15 seconds, old primary rejoins passive, and both database hosts return to failover_ready: true.
• Failure trigger: persistent mismatch outlives the existing proof deadline, Fleet fails to abort, or VIP recovery exceeds 15 seconds. Stop qualification and retain the previous release while the timeline is analyzed.
• Validation owner and window: completed by the HA release qualifier on the three-node Raspberry Pi cluster on 2026-08-12.

---

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (269d493a7b05e2f08e1fb7c56cbc1e048d07586c...d677dcf040986e8f78007f08b2382b852064aea4, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: NONE

Findings

No concrete security, correctness, or reliability issues were found in the reviewed diff.

Notes

The HA changes retain the existing lease/proof deadline during transient timeline mismatches, reject writer-generation changes, and validate the DCS leader lease before tolerating disagreement. Formatting and diff checks passed. Targeted Go tests could not run because the read-only environment prevented creation of the required module cache/toolchain.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a166eb6c19

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97691a8ec1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Copilot]

Pull request overview

This PR adjusts HA ownership handling so a transient Patroni/PostgreSQL timeline mismatch during promotion does not immediately demote an already-active Fleet runtime; instead, the coordinator marks the observation as unavailable and continues retrying within the existing proof/lease deadline. It also adds/extends tests to cover pre-activation retry, active convergence, and persistent mismatch expiry behavior.

Changes:

• Return the observed writer identity alongside ErrTimelineMismatch so callers can decide whether to tolerate the mismatch.
• Teach the HA coordinator to treat ErrTimelineMismatch as a non-fatal “observation unavailable” state while active (bounded by the existing lease/proof deadline).
• Add runtime/coordinator/observer tests covering transient and persistent timeline-mismatch scenarios.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
server/internal/ha/runtime_test.go	Adds synctest-driven runtime tests for transient/persistent timeline mismatch behavior.
server/internal/ha/observer.go	Returns an observation along with ErrTimelineMismatch so downstream code can validate writer identity.
server/internal/ha/observer_test.go	Extends mismatch test to assert returned observation fields on error.
server/internal/ha/coordinator.go	Special-cases ErrTimelineMismatch during active renewals to mark observations unavailable without deactivating.
server/internal/ha/coordinator_test.go	Adds coordinator tests for retaining active lifetime across mismatch and rejecting mismatch from a new writer; adjusts test observer behavior to return observations on error.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3219a0eaa4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".