diff --git a/docs/invariants.md b/docs/invariants.md new file mode 100644 index 000000000..ac187134a --- /dev/null +++ b/docs/invariants.md @@ -0,0 +1,639 @@ +# SchemaBot invariants + + + +## Table of Contents + +- [Merge gate (MG)](#merge-gate-mg) + - [MG-1 — Uncertainty is never converted into a passing check](#mg-1-uncertainty-is-never-converted-into-a-passing-check) + - [MG-2 — Absence never passes](#mg-2-absence-never-passes) + - [MG-3 — The rollup is conservative](#mg-3-the-rollup-is-conservative) + - [MG-4 — A check belongs to exactly one commit](#mg-4-a-check-belongs-to-exactly-one-commit) + - [MG-5 — Apply-owned check rows are released only by their owner](#mg-5-apply-owned-check-rows-are-released-only-by-their-owner) + - [MG-6 — Started applies keep blocking after the change disappears](#mg-6-started-applies-keep-blocking-after-the-change-disappears) + - [MG-7 — A completed rollback never shows green](#mg-7-a-completed-rollback-never-shows-green) + - [MG-8 — Check trust is by App identity, never by name](#mg-8-check-trust-is-by-app-identity-never-by-name) + - [MG-9 — GitHub being down never invents state](#mg-9-github-being-down-never-invents-state) + - [MG-10 — A blocked aggregate converges without an external event](#mg-10-a-blocked-aggregate-converges-without-an-external-event) +- [Apply state machine (ST)](#apply-state-machine-st) + - [ST-1 — A finished apply stays finished](#st-1-a-finished-apply-stays-finished) + - [ST-2 — Recovery from permanent failure is a fresh plan and apply](#st-2-recovery-from-permanent-failure-is-a-fresh-plan-and-apply) + - [ST-3 — Apply state flows upward, never downward](#st-3-apply-state-flows-upward-never-downward) + - [ST-4 — Stored task state never moves backward](#st-4-stored-task-state-never-moves-backward) + - [ST-5 — Unknown engine states are visible and blocking](#st-5-unknown-engine-states-are-visible-and-blocking) + - [ST-6 — Engine truth is scoped](#st-6-engine-truth-is-scoped) + - [ST-7 — Stop checkpoints conservatively](#st-7-stop-checkpoints-conservatively) + - [ST-8 — Every state has a recovery story](#st-8-every-state-has-a-recovery-story) + - [ST-9 — The retry budget is bounded and never re-runs finished work](#st-9-the-retry-budget-is-bounded-and-never-re-runs-finished-work) + - [ST-10 — Rollouts respect order and fail closed on policy](#st-10-rollouts-respect-order-and-fail-closed-on-policy) +- [Ownership and leases (OW)](#ownership-and-leases-ow) + - [OW-1 — Every drive runs under a claim](#ow-1-every-drive-runs-under-a-claim) + - [OW-2 — Lease-scoped writes must hold the current token](#ow-2-lease-scoped-writes-must-hold-the-current-token) + - [OW-3 — A driver yields no later than a peer may reclaim](#ow-3-a-driver-yields-no-later-than-a-peer-may-reclaim) + - [OW-4 — Lease loss is proven, never inferred](#ow-4-lease-loss-is-proven-never-inferred) + - [OW-5 — One active apply per target](#ow-5-one-active-apply-per-target) + - [OW-6 — There is one way to claim work](#ow-6-there-is-one-way-to-claim-work) + - [OW-7 — Ownership uncertainty fails closed](#ow-7-ownership-uncertainty-fails-closed) +- [Control operations (CO)](#control-operations-co) + - [CO-1 — Durable before acknowledged](#co-1-durable-before-acknowledged) + - [CO-2 — Every command resolves; none wedges](#co-2-every-command-resolves-none-wedges) + - [CO-3 — Engine terminal truth outranks a queued command](#co-3-engine-terminal-truth-outranks-a-queued-command) + - [CO-4 — Stop wins](#co-4-stop-wins) + - [CO-5 — The revert phase owns the outcome](#co-5-the-revert-phase-owns-the-outcome) + - [CO-6 — Commands act only where they have an effect](#co-6-commands-act-only-where-they-have-an-effect) + - [CO-7 — ID namespaces are never conflated](#co-7-id-namespaces-are-never-conflated) + - [CO-8 — Stop terminality is engine truth, told truthfully](#co-8-stop-terminality-is-engine-truth-told-truthfully) + - [CO-9 — Control operations are stateless](#co-9-control-operations-are-stateless) +- [Recovery (RC)](#recovery-rc) + - [RC-1 — Nothing is orphaned](#rc-1-nothing-is-orphaned) + - [RC-2 — Settled applies stop changing](#rc-2-settled-applies-stop-changing) + - [RC-3 — Loading nothing is not owning nothing](#rc-3-loading-nothing-is-not-owning-nothing) + - [RC-4 — Self-healing needs proof](#rc-4-self-healing-needs-proof) + - [RC-5 — Terminal notifications fire exactly once](#rc-5-terminal-notifications-fire-exactly-once) +- [Review integrity and data safety (RV)](#review-integrity-and-data-safety-rv) + - [RV-1 — What gets applied is what was reviewed](#rv-1-what-gets-applied-is-what-was-reviewed) + - [RV-2 — Stale plans never apply](#rv-2-stale-plans-never-apply) + - [RV-3 — Consent is explicit, specific, and re-checked](#rv-3-consent-is-explicit-specific-and-re-checked) + - [RV-4 — Engine refusals are known at plan time and gate the apply](#rv-4-engine-refusals-are-known-at-plan-time-and-gate-the-apply) + - [RV-5 — A dropped table stays recoverable](#rv-5-a-dropped-table-stays-recoverable) + - [RV-6 — Every statement parses, or it is an error](#rv-6-every-statement-parses-or-it-is-an-error) + - [RV-7 — Rollback needs the originals](#rv-7-rollback-needs-the-originals) + - [RV-8 — The plan sees the whole schema, or nothing](#rv-8-the-plan-sees-the-whole-schema-or-nothing) +- [Routing and authorization (AZ)](#routing-and-authorization-az) + - [AZ-1 — The server decides where a change runs](#az-1-the-server-decides-where-a-change-runs) + - [AZ-2 — Authorization fails closed at every tier](#az-2-authorization-fails-closed-at-every-tier) + - [AZ-3 — Identity comes from a verified lane](#az-3-identity-comes-from-a-verified-lane) + - [AZ-4 — Applying takes an authorized actor](#az-4-applying-takes-an-authorized-actor) + - [AZ-5 — Commands never guess](#az-5-commands-never-guess) +- [Availability and blast radius (AV)](#availability-and-blast-radius-av) + - [AV-1 — Schema changes run without GitHub](#av-1-schema-changes-run-without-github) + - [AV-2 — Outages degrade, never destroy](#av-2-outages-degrade-never-destroy) + - [AV-3 — Liveness is process-only](#av-3-liveness-is-process-only) + - [AV-4 — A transient failure never fails the work](#av-4-a-transient-failure-never-fails-the-work) + - [AV-5 — Panics are contained and permanent](#av-5-panics-are-contained-and-permanent) + - [AV-6 — An old binary never destroys newer state](#av-6-an-old-binary-never-destroys-newer-state) + - [AV-7 — An acknowledged delivery is a promise](#av-7-an-acknowledged-delivery-is-a-promise) + - [AV-8 — Untrusted text never reaches operator surfaces raw](#av-8-untrusted-text-never-reaches-operator-surfaces-raw) +- [Structural enforcement](#structural-enforcement) +- [Engineering rules live in AGENTS.md](#engineering-rules-live-in-agentsmd) + + + +The canonical registry of **runtime safety invariants** — the MUST-statements that make +SchemaBot safe to put in front of tier-0 databases. [architecture.md](architecture.md) explains +how the system is built and [apply-lifecycle.md](apply-lifecycle.md) explains how an apply lives +and dies; this document states what must never be false at runtime, why each rule matters, and +where it is enforced. Every invariant here is enforced in shipped code — this is a registry of +existing behavior, not a wishlist. + +Three principles generate most of what follows: + +1. **Fail closed on uncertainty.** Storage uncertainty, GitHub API uncertainty, ownership + ambiguity, or in-flight ambiguity is never converted into a passing check, a completed apply, + or a released lock. +2. **Started applies remain authoritative.** Once schema work has started against a database, + no cleanup, newer commit, PR close, or crash may declare the target reconciled — only apply + completion, rollback completion, or explicit operator action. +3. **A finished apply stays finished.** Terminal states are immutable facts; recovery from + permanent failure is always a fresh plan and a fresh apply. + +Each invariant carries an ID (`MG-*` merge gate, `ST-*` state machine, `OW-*` ownership and +leases, `CO-*` control operations, `RC-*` recovery, `RV-*` review integrity, `AZ-*` routing and +authorization, `AV-*` availability). When you change code near one of these, the invariant is +the review bar: a PR that weakens one must say so explicitly and update this document. + +## Merge gate (MG) + +The GitHub Check Run gate is the tier-0 safety feature: it is what stands between a schema PR +and a merge that contradicts the live database. Full semantics: [check-runs.md](check-runs.md). + +### MG-1 — Uncertainty is never converted into a passing check + +Config-discovery failure, stored-check-state read failure, head-SHA ambiguity, or an in-flight +apply must surface as a blocking or absent check, never a passing one. *Breaks if violated:* a +PR merges while its schema state is unknown. *Enforced:* every check-state write and aggregate +publish path in `pkg/webhook`. + +### MG-2 — Absence never passes + +The aggregate check is created on every PR head commit. A PR that touches no managed schema +files still gets one — an explicit passing "no managed schema changes" — because GitHub treats +a missing required check as not passing. The same logic applies in reverse: when SchemaBot +expects a check, environment, or config and cannot find it, that reads as blocking, never as +not-applicable. A gate must not be removable by the very change it guards. *Enforced:* +aggregate creation on head-commit events; required-check and environment-ordering gates in +`pkg/webhook`. + +### MG-3 — The rollup is conservative + +The aggregate check reports the worst result among its per-database checks: one `in_progress` +blocks, one `failure` fails, one `action_required` demands action, and the aggregate passes +only when every database passes. A conclusion SchemaBot does not recognize blocks. *Enforced:* +the aggregate rollup in `pkg/webhook`. + +### MG-4 — A check belongs to exactly one commit + +Work from an older head SHA never satisfies branch protection for a newer head, and a plan +taken while an apply is mutating the database never becomes the merge-gating source of truth. +*Enforced:* per-SHA check rows; stale-webhook and mid-apply plan guards in `pkg/webhook`. + +### MG-5 — Apply-owned check rows are released only by their owner + +A plan result never overwrites an `in_progress` check row that carries an `apply_id`, and +apply/rollback completion updates the row only if it still holds that `apply_id` and no newer +apply exists for the same PR, environment, and database. *Breaks if violated:* a concurrent +plan or a stale driver flips a gate that live work owns. *Enforced:* `apply_id` ownership +markers on `checks` rows with conditional completion writes; ownership misses are counted. + +### MG-6 — Started applies keep blocking after the change disappears + +A later commit that removes the schema change must not make the aggregate pass by cleanup +alone: the check stays blocked (`schema_removed_after_apply_started`) until the apply settles +and an operator reconciles the target. Closing and reopening the PR does not wash this state +away. *Enforced:* stale-cleanup guards in `pkg/webhook`; close/reopen handlers release nothing +they cannot read. + +### MG-7 — A completed rollback never shows green + +Rollback completion moves the check to `action_required` without ever passing through +`success` — the rollback reverted the PR's change, so the live database no longer matches the +PR. Only a fresh plan finding no diff may turn the gate green. *Enforced:* rollback-aware +terminal check writes. + +### MG-8 — Check trust is by App identity, never by name + +Only Check Runs created by this deployment's GitHub App (or explicitly trusted App slugs) +satisfy any gate; a check with the right name from an unknown app fails the gate closed. A +prior environment's passing check only proves the ordering was respected — it never authorizes +execution. The later deployment still plans, lints, and decides for itself. *Enforced:* +App-identity filters on every check read; cross-deployment trust model in +[check-runs.md](check-runs.md). + +### MG-9 — GitHub being down never invents state + +When GitHub is unreachable, SchemaBot publishes a failing aggregate if it can still verify the +head commit — and publishes nothing at all if it cannot. It never invents a check to fill the +gap. `schemabot unlock` and CLI success never satisfy branch protection by themselves. +*Enforced:* GitHub-unavailable branches in the check publish path; unlock and CLI flows do not +write passing checks. + +### MG-10 — A blocked aggregate converges without an external event + +Aggregates blocked on unresolved or stale state are re-evaluated by SchemaBot itself, not left +waiting for a lucky comment or push — except blocks that no retry can lift (untrusted App, +misconfigured check name), which block immediately and permanently. *Enforced:* stale-check +reconciliation and aggregate re-evaluation in `pkg/webhook`. + +## Apply state machine (ST) + +Canonical model: [apply-lifecycle.md](apply-lifecycle.md) and +[pkg/state/README.md](../pkg/state/README.md). + +### ST-1 — A finished apply stays finished + +Storage refuses any write that would move a terminal apply (`completed`, `failed`, `cancelled`, +`reverted`, `stopped`) back to an active state — not a retry, not an API write, not a crashed +driver replaying stale progress. The only sanctioned re-entry is claiming a `stopped` apply via +`start`. *Enforced:* the terminal guard in the storage apply update path +(`pkg/storage/internal/sqlstore`). + +### ST-2 — Recovery from permanent failure is a fresh plan and apply + +There is no revival path from `failed`. Plans are diffs, so a fresh plan+apply never re-runs +already-landed work; a revival path would re-run stored DDL against a database that may have +drifted. *Enforced:* absence — storage exposes no failed-to-active transition (ST-1). + +### ST-3 — Apply state flows upward, never downward + +A driver writes only the operation row it has claimed; the parent apply's state is derived from +its operations' tasks using a fixed priority order, never written directly by a drive. +*Breaks if violated:* one deployment's driver overwrites the verdict of another's. *Enforced:* +`DeriveApplyState()` (`pkg/state`) and the rollout projection in `pkg/tern`; lease-scoped +writes cannot touch the parent row. + +### ST-4 — Stored task state never moves backward + +A stale engine poll can never rewind stored task state: terminal stored tasks stay terminal, +control-owned states (`stopped`, `failed_retryable`) are never overwritten by active engine +polls, and active states advance strictly forward through the lifecycle. *Enforced:* +forward-only reconciliation in the progress poller (`pkg/tern`). + +### ST-5 — Unknown engine states are visible and blocking + +A raw engine state SchemaBot does not recognize normalizes to `running` — unfamiliar in-flight +work stays visible and keeps blocking, and engine-specific strings never leak into SchemaBot +state or UI. *Enforced:* `NormalizeTaskStatus` default branch (`pkg/state`). + +### ST-6 — Engine truth is scoped + +Engine-reported terminal states are trusted; an engine reporting "no active schema change" is +not proof of completion and marks the task failed rather than done. `stopped` and `cancelled` +are SchemaBot-owned states an engine can never report. *Enforced:* the engine trust model in +the progress sync (`pkg/tern`, [pkg/state/README.md](../pkg/state/README.md)). + +### ST-7 — Stop checkpoints conservatively + +On stop, every non-terminal task goes `stopped` regardless of engine sub-state — a task is +never promoted to `completed` on partial engine progress, and the stop snapshot is taken only +after the engine's own stop returns. *Enforced:* stop handling in `pkg/tern`. + +### ST-8 — Every state has a recovery story + +Every apply state is registered with metadata (label, terminal classification), and every +non-terminal state an apply can sit in is either re-claimable once its driver's heartbeat goes +stale or covered by its own named recovery path. A state added without a recovery path is a CI +failure, not a production orphan. *Enforced:* the state metadata registry and a +reflection-driven completeness test over it (`pkg/state`, `pkg/storage/internal/sqlstore`). + +### ST-9 — The retry budget is bounded and never re-runs finished work + +`failed_retryable` is active, not terminal: recovery re-drives it automatically, but only +`failed_retryable` tasks reset to `pending` — completed tasks are never re-run — and the apply +settles to permanent `failed` when the attempt budget is spent or the recovery window closes. +*Enforced:* retry preparation and expiry in the operator and storage layers; budget semantics +in [apply-lifecycle.md](apply-lifecycle.md). + +### ST-10 — Rollouts respect order and fail closed on policy + +Multi-deployment rollouts claim operations in `deployment_order`, and a failed earlier +deployment blocks later ones unless the config says otherwise. Once an operator releases a +paused rollout it stays released — there is no path back to paused. And an *unrecognized* +`on_failure` value behaves like `halt`, never like `continue`. *Enforced:* the ordered-claim +gate in `FindNextApplyOperation` and the rollout state derivation (`pkg/state`). + +## Ownership and leases (OW) + +### OW-1 — Every drive runs under a claim + +Before a pod may do any work on an apply, it must claim that work in storage. The claim is one +atomic transaction (`FOR UPDATE SKIP LOCKED`) that stamps the row with the claimer's lease — +owner, a fresh token, a heartbeat — so two pods can never pick up the same work. This holds +for every drive including the very first: the pod that accepted the request gets no shortcut +and claims the apply like any other driver. The wake signal sent after enqueueing only makes a +driver check for claimable work sooner; it never hands work to a pod directly. *Enforced:* the +operator claim path (`pkg/api`, `pkg/storage/internal/sqlstore`). + +### OW-2 — Lease-scoped writes must hold the current token + +Apply and task updates, heartbeats, apply logs, and recovered observer side effects all carry +the lease token; when another driver has claimed the row and rotated the token, the displaced +driver's writes fail with an ownership error and it exits rather than posting stale comments or +overwriting newer state. *Enforced:* a token check on every lease-scoped storage write. + +### OW-3 — A driver yields no later than a peer may reclaim + +The heartbeat loop and the claim query read the same staleness window, so a driver that cannot +heartbeat stops driving at or before the moment a peer could consider the row abandoned and +claim it. The same schema change can never be driven by two pods at once. *Enforced:* one +shared staleness constant used by both the heartbeat loop and the claim query. + +### OW-4 — Lease loss is proven, never inferred + +A driver self-fences only after a successful read proves the token changed; a connection error +or transient storage failure is retried, never misread as displacement. *Breaks if violated:* a +storage blip aborts a healthy multi-hour copy. *Enforced:* the split-brain fences in the drive +loop ([storage-outage-behavior.md](storage-outage-behavior.md)). + +### OW-5 — One active apply per target + +At most one active apply exists per (database, database type, environment), enforced whenever +an apply is created or moved back into an active state, under a per-target named lock — and the +reservation covers the apply's whole deployment set until the parent settles. This invariant +does not depend on a user-facing database lock being held: direct API callers and `--no-lock` +flows are equally bound. *Enforced:* the exclusivity check in the storage apply create/activate +path. + +### OW-6 — There is one way to claim work + +Exactly one claim query and one lease-rotation path exist — no side doors. Eligibility rules +live inside the claim query itself, not in checks that run after it: once a destructive +transition (say, `waiting_for_cutover → cutting_over`) has been claimed, the work is already +in motion, and discovering only then that it should not have been claimed is too late. +*Enforced:* the single operation-claim query; tests pin its state arms against divergence. + +### OW-7 — Ownership uncertainty fails closed + +A driver that cannot prove it owns work — or is handed work outside the databases its engine +client can reach — refuses it, leaving the row untouched for a pod that can drive it. A claim +that cannot proceed releases its lease immediately rather than holding the row idle until the +staleness window expires. And a pod never trusts its own in-memory engine state to answer +questions about work it does not own. *Enforced:* scope checks and token-guarded claim release +in the drive path. + +## Control operations (CO) + +Full doctrine: [grpc-control-edge-cases.md](grpc-control-edge-cases.md) and the Control +Operations section of [AGENTS.md](../AGENTS.md). + +### CO-1 — Durable before acknowledged + +An accepted control request (stop, start, cutover, cancel, volume, revert, skip-revert) is +recorded in `apply_control_requests` before SchemaBot reports acceptance; drivers consume it +from storage. A pod restart, lease handover, or webhook redelivery never loses an acknowledged +command; in-memory flags are per-drive optimizations, never the source of truth. *Enforced:* +the durable request path in `pkg/api` / `pkg/webhook`. + +### CO-2 — Every command resolves; none wedges + +A control request resolves to an explicit durable outcome — applied, superseded, or failed. A +command may delay a drive but never wedge it: a failed request requires fresh operator intent +rather than being retried forever, and polling windows are bounded with visible timeout +failures. *Breaks if violated:* an apply loops on a doomed command while holding its database +lock. *Enforced:* request completion and bounded-retry rules in the drive loop. + +### CO-3 — Engine terminal truth outranks a queued command + +When the engine's own record shows the change already settled, the drive adopts that outcome +instead of fighting it — a cancel against a deploy that already completed records the apply as +completed rather than re-sending the cancel forever. Only engines whose backend holds the +authoritative record of the change (PlanetScale, where the deploy request lives server-side) +are consulted this way; for all others the question fails closed. Only settled outcomes are +adopted — a remote state still in motion never is. *Enforced:* terminal-truth preflights on +the control paths in `pkg/tern`. + +### CO-4 — Stop wins + +Stop is the highest-priority intent: once accepted, no actor may knowingly advance the apply +toward deploy, cutover, or completion until the stop is processed — forward-progress commands +are rejected while a stop pends, and contradictory intents are rejected at acceptance rather +than resolved by drive ordering. *Enforced:* pending-stop checks in pollers and the cutover +paths; conflict rejection at request intake. + +### CO-5 — The revert phase owns the outcome + +Once an apply is `reverting` or `skipping_revert`, stop, cancel, and cutover are refused for +the whole phase — storage must never settle on a state that contradicts what the engine is +still doing to the database underneath. *Enforced:* revert-phase gates in the control paths +(`pkg/tern`). + +### CO-6 — Commands act only where they have an effect + +A control operation is rejected up front when nothing can service it — a volume change on an +apply that is finished or has no row copy running, a second cutover while one is already in +flight — instead of being queued for a consumer that will never come. And its effect is scoped +to the one change it targets: an incident-time tuning or one operation's completion never +bleeds onto sibling operations or future applies. *Enforced:* queue-time eligibility gates; +operation-scoped request rows. + +### CO-7 — ID namespaces are never conflated + +Remote data-plane RPCs identify work by the operation's `external_id`; every user-facing +surface (logs, comments, checks, CLI, HTTP) uses the string `apply_identifier`. Sending the +user-facing ID to a remote engine is a routing bug, and the internal numeric row ID appears on +no surface at all. *Enforced:* the remote apply ID invariant +([grpc-control-edge-cases.md](grpc-control-edge-cases.md)); ID resolution at the API boundary. + +### CO-8 — Stop terminality is engine truth, told truthfully + +A Spirit stop is a resumable pause; a PlanetScale stop cancels the deploy request permanently. +The distinction is derived from the engine, not inferred at render time, and every operator +surface says which one happened. *Enforced:* the stop-terminality policy helper and its +renderings. + +### CO-9 — Control operations are stateless + +Any instance can serve any control operation, because controls act by writing shared durable +state — storage rows, or marker tables on the target database itself — never by reaching into +a specific process's in-memory engine. A cutover request landing on a pod that is not driving +the apply still works. *Enforced:* design rule in [architecture.md](architecture.md); the +cutover sentinel table on the target database. + +## Recovery (RC) + +### RC-1 — Nothing is orphaned + +A crash between any two related writes is reconcilable afterward, in both directions. When an +operation reaches a terminal state but its parent apply was never updated, the parent's state +is re-derived from its operations. When the parent has settled but an operation row never +started, a reaper marks that row with the parent's outcome — though only once the parent is +settled and quiet, and never while the parent is `stopped` or `failed_retryable` (those may +still resume). *Enforced:* bidirectional reconciliation in the operator and the +stranded-operation reaper. + +### RC-2 — Settled applies stop changing + +No claim, reaper, or recovery path touches an apply whose verdict is recorded; an apply that +finished last week does not have its rows change today. *Enforced:* settled-state exclusions in +every claim and sweep query (and ST-1 backstops all of them). + +### RC-3 — Loading nothing is not owning nothing + +A loader returning zero rows because of an error is never conflated with an apply that +genuinely owns zero rows: the first blocks and surfaces, only the second may complete as a +no-op. *Breaks if violated:* completion reports success for DDL that never ran. *Enforced:* +separated error/empty handling on recovery load paths. + +### RC-4 — Self-healing needs proof + +Automatic cleanup acts only where the record provably carries no engine work (a `pending` task +has no checkpoint by construction); anything uncertain keeps blocking for an operator. +*Enforced:* narrow eligibility conditions on every self-heal path. + +### RC-5 — Terminal notifications fire exactly once + +The terminal summary (PR comment, check completion) is authorized by an atomic claim that +permanently hands the job to exactly one driver. Unlike a lease, this claim cannot expire and +be re-granted — so no later driver can post the summary a second time, and no handover can +drop it. *Enforced:* atomic observer/notification claims in the drive and recovery paths. + +## Review integrity and data safety (RV) + +### RV-1 — What gets applied is what was reviewed + +A deployment applying a plan reviewed elsewhere independently re-derives the change set and +compares immediately before each per-task engine apply; drift — including DDL it cannot parse — +fails closed. Recomputed deltas are never applied silently. *Enforced:* pre-apply change-set +verification in the multi-deployment apply path. + +### RV-2 — Stale plans never apply + +An apply is rejected when the plan's base is no longer the branch's live base — freshness is +mandatory, is checked at apply time against the current base, and cannot be waived by +`--allow-unsafe`. Plan comments are suppressed as duplicates only when SchemaBot can prove it +is not preserving a stale plan. *Enforced:* the plan-freshness gate on both apply entry points. + +### RV-3 — Consent is explicit, specific, and re-checked + +Unsafe changes (error-severity lint findings: drops, column narrowing) block without +`--allow-unsafe`. Changes an operator cannot undo mid-flight (direct execution's +write-blocking DDL, with no cutover and no revert) require the operator to confirm the +specific consequences disclosed to them — and the re-plan that runs just before execution +re-checks that verdict, so a plan that changed after the confirmation stops rather than +running something the operator never saw. *Enforced:* lint gates and the apply-confirm flow. + +### RV-4 — Engine refusals are known at plan time and gate the apply + +Whether the engine will refuse a statement — or route it to direct execution, a MySQL/Spirit +execution mode — is recorded on the plan using the engine's own checks, not a reimplementation +of them, and an apply on a refused plan is rejected before any lock is taken. For direct +execution's table-size bound, a table whose size cannot be measured is blocked, and a row +estimate is trusted only in the blocking direction — an estimate alone never approves. +*Enforced:* plan-time execution verdicts; apply-time verdict gates; the direct-execution size +bound ([direct-execution.md](direct-execution.md)). + +### RV-5 — A dropped table stays recoverable + +`DROP TABLE` never destroys data at execution time; the table is held for a recovery window +instead. On MySQL, the Spirit engine quarantines the drop as an atomic rename into +`_pending_drops` with a UTC timestamp prefix — tables without a valid timestamp prefix are +never auto-dropped (age unknown), and cleanup passes are serialized per target under an +advisory lock. On Vitess, the drop rides Vitess's native table lifecycle, which holds dropped +tables before purging them. *Enforced:* the pending-drops lifecycle +([pending-drops.md](pending-drops.md)) on Spirit; the table lifecycle native to Vitess online +DDL on PlanetScale. + +### RV-6 — Every statement parses, or it is an error + +All SQL SchemaBot processes must parse with the TiDB parser — no string-splitting fallback, no +silently skipped statements — and a classifier refuses ambiguity (a compound statement never +classifies as its first verb) rather than letting a destructive statement ride past +classification-based gating. *Enforced:* the `pkg/ddl` / Spirit `statement` boundary. + +### RV-7 — Rollback needs the originals + +Rollback requires the original schema files captured at apply time; if artifact capture failed, +rollback is refused rather than reconstructed. *Enforced:* rollback preconditions on stored +artifacts. + +### RV-8 — The plan sees the whole schema, or nothing + +Schema discovery fails closed on any partial view: a GitHub directory listing at the API's +entry cap, a truncated git tree, or config discovery that cannot prove exhaustiveness aborts +the plan — a missing file must never read as a deleted table and produce a spurious +`DROP TABLE` proposal. Symlinked namespaces resolving outside the repository root (or to +themselves) are rejected. *Enforced:* truncation and symlink guards on every schema-fetch path +in `pkg/github`. + +## Routing and authorization (AZ) + +### AZ-1 — The server decides where a change runs + +Targets and deployments resolve from server-owned config; the resolved route is stored on the +plan and reused by the apply, and request decoding rejects client-supplied routing fields. A +schema source (repo + path) is authorized against server config at plan time and re-checked +before apply; missing config fails closed. *Enforced:* server-side routing and source policy in +`pkg/api`. + +### AZ-2 — Authorization fails closed at every tier + +An API route not classified as a read is treated as a write; planning counts as a write (it +stages a change); a target that cannot be resolved never authorizes; a configured grant that +could never match any request is a startup error rather than silent dead config; and a new +mutating endpoint cannot ship without a test proving it denies unauthorized callers. +*Enforced:* route classification with a structural sweep test over the route table. + +### AZ-3 — Identity comes from a verified lane + +Forwarded identity headers are trusted only from explicitly listed proxies; a gateway-verified +service caller never reaches the write tier by self-asserting user headers; and every write and +control action is attributed to the verified caller — a client-supplied caller string never +overrides. *Enforced:* the trust-anchor config (refuses to start without one) and +identity-precedence rules in the auth layer. + +### AZ-4 — Applying takes an authorized actor + +A PR apply requires an actor authorized for the target (configured operators, admin teams, +repo admins, or CODEOWNERS, per config), evaluated per database — and the change's author +cannot satisfy their own review requirement. *Enforced:* the review gate and actor +authorization in `pkg/webhook`. + +### AZ-5 — Commands never guess + +An unscoped PR command resolves to exactly one unambiguous database or is rejected with +guidance — never an arbitrary pick — and a malformed command is rejected rather than +"helpfully" corrected into something executable (especially one carrying `--allow-unsafe`). +Every command receives a response; silence only ever means another instance owns the reply. +*Enforced:* command discovery and the unowned-command policy in `pkg/webhook`. + +## Availability and blast radius (AV) + +Full model: [storage-outage-behavior.md](storage-outage-behavior.md). + +### AV-1 — Schema changes run without GitHub + +SchemaBot works during GitHub outages: CLI and API applies work when GitHub is down, remote +data-plane instances never need a GitHub App, and observer side effects (PR comments, progress +edits) are best-effort — logged on failure, never blocking or failing the change. GitHub is an +observer of execution, never a dependency of it. *Enforced:* the comment-observer pattern in +`pkg/webhook`; data-plane deployments carry no GitHub credentials. + +### AV-2 — Outages degrade, never destroy + +A storage outage degrades the control plane only: in-flight schema changes keep running, +because the safety mechanisms that protect the target database (throttling, lock timeouts, +checksums) live in the engine, not in the drive loop — and uncertainty is never converted into +a verdict. A terminal outcome that cannot be persisted leaves the row non-terminal and +claimable, to be re-derived on the next claim rather than fabricated. *Enforced:* the +per-failure-class handling in the drive loops +([storage-outage-behavior.md](storage-outage-behavior.md)). + +### AV-3 — Liveness is process-only + +Storage reachability belongs on readiness, never liveness — restarting a pod cannot fix an +unreachable database and only aborts in-flight local drives on top of the outage. Boot retries +storage inside the startup budget, re-resolving credentials each attempt. *Enforced:* +`/livez` vs `/health` probe wiring. + +### AV-4 — A transient failure never fails the work + +A storage or transport blip is not a schema-change failure. An error writing a progress update +is logged and the drive continues; an error reading something safety-gating ends this drive +attempt and leaves the row claimable for another; and repeated errors observing remote +progress mark the apply `failed_retryable` — they never trigger a remote stop, because an +observation outage only proves the control plane cannot see, not that the change is unhealthy. +*Enforced:* failure-class handling in the drive loop and the remote progress error limit. + +### AV-5 — Panics are contained and permanent + +A panic is contained to the unit of work that caused it, converted into a permanent (not +retryable) failure — retrying would feed the poisoned input straight back — and made loud with +a log, a durable apply-log entry, and a metric. One bad row must never crash-loop the fleet. +*Enforced:* recover boundaries around drives and webhook work units. + +### AV-6 — An old binary never destroys newer state + +During a rolling deploy, a binary that encounters state written by a newer version skips it and +continues — the one sanctioned exception to fail-closed, because crash-looping every old pod is +worse. It may never delete, rewrite, or terminalize what it does not understand. *Enforced:* +version-aware skip guards on shared-state consumers. + +### AV-7 — An acknowledged delivery is a promise + +A webhook delivery acknowledged with 200 is durable work: it is recorded in the event inbox +before acknowledgment, deduplicated by delivery GUID so a redelivery's side effects run at +most once, drained on shutdown so a rolling deploy loses nothing, and — when a row proves +unprocessable — terminated visibly rather than retried forever or silently dropped. +*Enforced:* the durable webhook inbox — durable dispatch in `pkg/webhook`, backed by the +`webhook_events` table. + +### AV-8 — Untrusted text never reaches operator surfaces raw + +Raw error strings (dial failures, DSN fragments, hostnames) are never rendered into PR or +check markdown — surfaces show a fixed sanitized line while the raw error goes to server logs +with triage identifiers — and inbound payloads are size-bounded before decode. *Enforced:* +sanitized rendering helpers in `pkg/webhook`; request body limits. + +## Structural enforcement + +The strongest invariants are enforced by structure, so regressions fail CI instead of review: + +- **State completeness** — a reflection-driven test walks every registered apply state and + fails on any state that is neither terminal, claimable, nor explicitly allowlisted with its + own recovery arm (ST-8). +- **Route-table sweep** — every mutating endpoint requires a denial fixture proving its + handler-level authorization check (AZ-2). +- **Engine conformance** — engines run a shared conformance kit; an engine that cannot exhibit + a case documents an explicit skip, never a silent opt-out (CO-3, CO-8). +- **Claim parity** — the states the claim query accepts are pinned by tests, so a state added + to one copy of a safety-critical condition cannot silently diverge from another (OW-6). + +New invariants should aspire to this tier: when adding one, prefer a completeness test over the +relevant registry to a hand-maintained list. + +## Engineering rules live in AGENTS.md + +The process-level rules that keep this codebase reviewable — never silently fail; error early, +never swallow; no silent branch cases; wrap errors with identifiers; logs answer the triage +question; name compound predicates; one owner closes a handle; tests prove documented +behavior — belong in [AGENTS.md](../AGENTS.md), not here. This document holds invariants about +the **running system**; that one holds invariants about **how we write and review the code**.