From 7cd0a00c85a67213139bc6860d597e612f5a2e26 Mon Sep 17 00:00:00 2001 From: Armand Parajon Date: Fri, 7 Aug 2026 16:15:52 -0400 Subject: [PATCH] docs: add invariants.md, the runtime safety invariant registry Distills the safety properties that make SchemaBot safe for tier-0 databases into a single registry: merge gate, apply state machine, ownership and leases, control operations, recovery, review integrity, routing and authorization, and availability. Each invariant states the rule, what breaks if it is violated, and where it is enforced. Co-Authored-By: Claude Fable 5 --- docs/invariants.md | 639 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 639 insertions(+) create mode 100644 docs/invariants.md diff --git a/docs/invariants.md b/docs/invariants.md new file mode 100644 index 000000000..ac187134a --- /dev/null +++ b/docs/invariants.md @@ -0,0 +1,639 @@ +# SchemaBot invariants + + + +## Table of Contents + +- [Merge gate (MG)](#merge-gate-mg) + - [MG-1 — Uncertainty is never converted into a passing check](#mg-1-uncertainty-is-never-converted-into-a-passing-check) + - [MG-2 — Absence never passes](#mg-2-absence-never-passes) + - [MG-3 — The rollup is conservative](#mg-3-the-rollup-is-conservative) + - [MG-4 — A check belongs to exactly one commit](#mg-4-a-check-belongs-to-exactly-one-commit) + - [MG-5 — Apply-owned check rows are released only by their owner](#mg-5-apply-owned-check-rows-are-released-only-by-their-owner) + - [MG-6 — Started applies keep blocking after the change disappears](#mg-6-started-applies-keep-blocking-after-the-change-disappears) + - [MG-7 — A completed rollback never shows green](#mg-7-a-completed-rollback-never-shows-green) + - [MG-8 — Check trust is by App identity, never by name](#mg-8-check-trust-is-by-app-identity-never-by-name) + - [MG-9 — GitHub being down never invents state](#mg-9-github-being-down-never-invents-state) + - [MG-10 — A blocked aggregate converges without an external event](#mg-10-a-blocked-aggregate-converges-without-an-external-event) +- [Apply state machine (ST)](#apply-state-machine-st) + - [ST-1 — A finished apply stays finished](#st-1-a-finished-apply-stays-finished) + - [ST-2 — Recovery from permanent failure is a fresh plan and apply](#st-2-recovery-from-permanent-failure-is-a-fresh-plan-and-apply) + - [ST-3 — Apply state flows upward, never downward](#st-3-apply-state-flows-upward-never-downward) + - [ST-4 — Stored task state never moves backward](#st-4-stored-task-state-never-moves-backward) + - [ST-5 — Unknown engine states are visible and blocking](#st-5-unknown-engine-states-are-visible-and-blocking) + - [ST-6 — Engine truth is scoped](#st-6-engine-truth-is-scoped) + - [ST-7 — Stop checkpoints conservatively](#st-7-stop-checkpoints-conservatively) + - [ST-8 — Every state has a recovery story](#st-8-every-state-has-a-recovery-story) + - [ST-9 — The retry budget is bounded and never re-runs finished work](#st-9-the-retry-budget-is-bounded-and-never-re-runs-finished-work) + - [ST-10 — Rollouts respect order and fail closed on policy](#st-10-rollouts-respect-order-and-fail-closed-on-policy) +- [Ownership and leases (OW)](#ownership-and-leases-ow) + - [OW-1 — Every drive runs under a claim](#ow-1-every-drive-runs-under-a-claim) + - [OW-2 — Lease-scoped writes must hold the current token](#ow-2-lease-scoped-writes-must-hold-the-current-token) + - [OW-3 — A driver yields no later than a peer may reclaim](#ow-3-a-driver-yields-no-later-than-a-peer-may-reclaim) + - [OW-4 — Lease loss is proven, never inferred](#ow-4-lease-loss-is-proven-never-inferred) + - [OW-5 — One active apply per target](#ow-5-one-active-apply-per-target) + - [OW-6 — There is one way to claim work](#ow-6-there-is-one-way-to-claim-work) + - [OW-7 — Ownership uncertainty fails closed](#ow-7-ownership-uncertainty-fails-closed) +- [Control operations (CO)](#control-operations-co) + - [CO-1 — Durable before acknowledged](#co-1-durable-before-acknowledged) + - [CO-2 — Every command resolves; none wedges](#co-2-every-command-resolves-none-wedges) + - [CO-3 — Engine terminal truth outranks a queued command](#co-3-engine-terminal-truth-outranks-a-queued-command) + - [CO-4 — Stop wins](#co-4-stop-wins) + - [CO-5 — The revert phase owns the outcome](#co-5-the-revert-phase-owns-the-outcome) + - [CO-6 — Commands act only where they have an effect](#co-6-commands-act-only-where-they-have-an-effect) + - [CO-7 — ID namespaces are never conflated](#co-7-id-namespaces-are-never-conflated) + - [CO-8 — Stop terminality is engine truth, told truthfully](#co-8-stop-terminality-is-engine-truth-told-truthfully) + - [CO-9 — Control operations are stateless](#co-9-control-operations-are-stateless) +- [Recovery (RC)](#recovery-rc) + - [RC-1 — Nothing is orphaned](#rc-1-nothing-is-orphaned) + - [RC-2 — Settled applies stop changing](#rc-2-settled-applies-stop-changing) + - [RC-3 — Loading nothing is not owning nothing](#rc-3-loading-nothing-is-not-owning-nothing) + - [RC-4 — Self-healing needs proof](#rc-4-self-healing-needs-proof) + - [RC-5 — Terminal notifications fire exactly once](#rc-5-terminal-notifications-fire-exactly-once) +- [Review integrity and data safety (RV)](#review-integrity-and-data-safety-rv) + - [RV-1 — What gets applied is what was reviewed](#rv-1-what-gets-applied-is-what-was-reviewed) + - [RV-2 — Stale plans never apply](#rv-2-stale-plans-never-apply) + - [RV-3 — Consent is explicit, specific, and re-checked](#rv-3-consent-is-explicit-specific-and-re-checked) + - [RV-4 — Engine refusals are known at plan time and gate the apply](#rv-4-engine-refusals-are-known-at-plan-time-and-gate-the-apply) + - [RV-5 — A dropped table stays recoverable](#rv-5-a-dropped-table-stays-recoverable) + - [RV-6 — Every statement parses, or it is an error](#rv-6-every-statement-parses-or-it-is-an-error) + - [RV-7 — Rollback needs the originals](#rv-7-rollback-needs-the-originals) + - [RV-8 — The plan sees the whole schema, or nothing](#rv-8-the-plan-sees-the-whole-schema-or-nothing) +- [Routing and authorization (AZ)](#routing-and-authorization-az) + - [AZ-1 — The server decides where a change runs](#az-1-the-server-decides-where-a-change-runs) + - [AZ-2 — Authorization fails closed at every tier](#az-2-authorization-fails-closed-at-every-tier) + - [AZ-3 — Identity comes from a verified lane](#az-3-identity-comes-from-a-verified-lane) + - [AZ-4 — Applying takes an authorized actor](#az-4-applying-takes-an-authorized-actor) + - [AZ-5 — Commands never guess](#az-5-commands-never-guess) +- [Availability and blast radius (AV)](#availability-and-blast-radius-av) + - [AV-1 — Schema changes run without GitHub](#av-1-schema-changes-run-without-github) + - [AV-2 — Outages degrade, never destroy](#av-2-outages-degrade-never-destroy) + - [AV-3 — Liveness is process-only](#av-3-liveness-is-process-only) + - [AV-4 — A transient failure never fails the work](#av-4-a-transient-failure-never-fails-the-work) + - [AV-5 — Panics are contained and permanent](#av-5-panics-are-contained-and-permanent) + - [AV-6 — An old binary never destroys newer state](#av-6-an-old-binary-never-destroys-newer-state) + - [AV-7 — An acknowledged delivery is a promise](#av-7-an-acknowledged-delivery-is-a-promise) + - [AV-8 — Untrusted text never reaches operator surfaces raw](#av-8-untrusted-text-never-reaches-operator-surfaces-raw) +- [Structural enforcement](#structural-enforcement) +- [Engineering rules live in AGENTS.md](#engineering-rules-live-in-agentsmd) + + + +The canonical registry of **runtime safety invariants** — the MUST-statements that make +SchemaBot safe to put in front of tier-0 databases. [architecture.md](architecture.md) explains +how the system is built and [apply-lifecycle.md](apply-lifecycle.md) explains how an apply lives +and dies; this document states what must never be false at runtime, why each rule matters, and +where it is enforced. Every invariant here is enforced in shipped code — this is a registry of +existing behavior, not a wishlist. + +Three principles generate most of what follows: + +1. **Fail closed on uncertainty.** Storage uncertainty, GitHub API uncertainty, ownership + ambiguity, or in-flight ambiguity is never converted into a passing check, a completed apply, + or a released lock. +2. **Started applies remain authoritative.** Once schema work has started against a database, + no cleanup, newer commit, PR close, or crash may declare the target reconciled — only apply + completion, rollback completion, or explicit operator action. +3. **A finished apply stays finished.** Terminal states are immutable facts; recovery from + permanent failure is always a fresh plan and a fresh apply. + +Each invariant carries an ID (`MG-*` merge gate, `ST-*` state machine, `OW-*` ownership and +leases, `CO-*` control operations, `RC-*` recovery, `RV-*` review integrity, `AZ-*` routing and +authorization, `AV-*` availability). When you change code near one of these, the invariant is +the review bar: a PR that weakens one must say so explicitly and update this document. + +## Merge gate (MG) + +The GitHub Check Run gate is the tier-0 safety feature: it is what stands between a schema PR +and a merge that contradicts the live database. Full semantics: [check-runs.md](check-runs.md). + +### MG-1 — Uncertainty is never converted into a passing check + +Config-discovery failure, stored-check-state read failure, head-SHA ambiguity, or an in-flight +apply must surface as a blocking or absent check, never a passing one. *Breaks if violated:* a +PR merges while its schema state is unknown. *Enforced:* every check-state write and aggregate +publish path in `pkg/webhook`. + +### MG-2 — Absence never passes + +The aggregate check is created on every PR head commit. A PR that touches no managed schema +files still gets one — an explicit passing "no managed schema changes" — because GitHub treats +a missing required check as not passing. The same logic applies in reverse: when SchemaBot +expects a check, environment, or config and cannot find it, that reads as blocking, never as +not-applicable. A gate must not be removable by the very change it guards. *Enforced:* +aggregate creation on head-commit events; required-check and environment-ordering gates in +`pkg/webhook`. + +### MG-3 — The rollup is conservative + +The aggregate check reports the worst result among its per-database checks: one `in_progress` +blocks, one `failure` fails, one `action_required` demands action, and the aggregate passes +only when every database passes. A conclusion SchemaBot does not recognize blocks. *Enforced:* +the aggregate rollup in `pkg/webhook`. + +### MG-4 — A check belongs to exactly one commit + +Work from an older head SHA never satisfies branch protection for a newer head, and a plan +taken while an apply is mutating the database never becomes the merge-gating source of truth. +*Enforced:* per-SHA check rows; stale-webhook and mid-apply plan guards in `pkg/webhook`. + +### MG-5 — Apply-owned check rows are released only by their owner + +A plan result never overwrites an `in_progress` check row that carries an `apply_id`, and +apply/rollback completion updates the row only if it still holds that `apply_id` and no newer +apply exists for the same PR, environment, and database. *Breaks if violated:* a concurrent +plan or a stale driver flips a gate that live work owns. *Enforced:* `apply_id` ownership +markers on `checks` rows with conditional completion writes; ownership misses are counted. + +### MG-6 — Started applies keep blocking after the change disappears + +A later commit that removes the schema change must not make the aggregate pass by cleanup +alone: the check stays blocked (`schema_removed_after_apply_started`) until the apply settles +and an operator reconciles the target. Closing and reopening the PR does not wash this state +away. *Enforced:* stale-cleanup guards in `pkg/webhook`; close/reopen handlers release nothing +they cannot read. + +### MG-7 — A completed rollback never shows green + +Rollback completion moves the check to `action_required` without ever passing through +`success` — the rollback reverted the PR's change, so the live database no longer matches the +PR. Only a fresh plan finding no diff may turn the gate green. *Enforced:* rollback-aware +terminal check writes. + +### MG-8 — Check trust is by App identity, never by name + +Only Check Runs created by this deployment's GitHub App (or explicitly trusted App slugs) +satisfy any gate; a check with the right name from an unknown app fails the gate closed. A +prior environment's passing check only proves the ordering was respected — it never authorizes +execution. The later deployment still plans, lints, and decides for itself. *Enforced:* +App-identity filters on every check read; cross-deployment trust model in +[check-runs.md](check-runs.md). + +### MG-9 — GitHub being down never invents state + +When GitHub is unreachable, SchemaBot publishes a failing aggregate if it can still verify the +head commit — and publishes nothing at all if it cannot. It never invents a check to fill the +gap. `schemabot unlock` and CLI success never satisfy branch protection by themselves. +*Enforced:* GitHub-unavailable branches in the check publish path; unlock and CLI flows do not +write passing checks. + +### MG-10 — A blocked aggregate converges without an external event + +Aggregates blocked on unresolved or stale state are re-evaluated by SchemaBot itself, not left +waiting for a lucky comment or push — except blocks that no retry can lift (untrusted App, +misconfigured check name), which block immediately and permanently. *Enforced:* stale-check +reconciliation and aggregate re-evaluation in `pkg/webhook`. + +## Apply state machine (ST) + +Canonical model: [apply-lifecycle.md](apply-lifecycle.md) and +[pkg/state/README.md](../pkg/state/README.md). + +### ST-1 — A finished apply stays finished + +Storage refuses any write that would move a terminal apply (`completed`, `failed`, `cancelled`, +`reverted`, `stopped`) back to an active state — not a retry, not an API write, not a crashed +driver replaying stale progress. The only sanctioned re-entry is claiming a `stopped` apply via +`start`. *Enforced:* the terminal guard in the storage apply update path +(`pkg/storage/internal/sqlstore`). + +### ST-2 — Recovery from permanent failure is a fresh plan and apply + +There is no revival path from `failed`. Plans are diffs, so a fresh plan+apply never re-runs +already-landed work; a revival path would re-run stored DDL against a database that may have +drifted. *Enforced:* absence — storage exposes no failed-to-active transition (ST-1). + +### ST-3 — Apply state flows upward, never downward + +A driver writes only the operation row it has claimed; the parent apply's state is derived from +its operations' tasks using a fixed priority order, never written directly by a drive. +*Breaks if violated:* one deployment's driver overwrites the verdict of another's. *Enforced:* +`DeriveApplyState()` (`pkg/state`) and the rollout projection in `pkg/tern`; lease-scoped +writes cannot touch the parent row. + +### ST-4 — Stored task state never moves backward + +A stale engine poll can never rewind stored task state: terminal stored tasks stay terminal, +control-owned states (`stopped`, `failed_retryable`) are never overwritten by active engine +polls, and active states advance strictly forward through the lifecycle. *Enforced:* +forward-only reconciliation in the progress poller (`pkg/tern`). + +### ST-5 — Unknown engine states are visible and blocking + +A raw engine state SchemaBot does not recognize normalizes to `running` — unfamiliar in-flight +work stays visible and keeps blocking, and engine-specific strings never leak into SchemaBot +state or UI. *Enforced:* `NormalizeTaskStatus` default branch (`pkg/state`). + +### ST-6 — Engine truth is scoped + +Engine-reported terminal states are trusted; an engine reporting "no active schema change" is +not proof of completion and marks the task failed rather than done. `stopped` and `cancelled` +are SchemaBot-owned states an engine can never report. *Enforced:* the engine trust model in +the progress sync (`pkg/tern`, [pkg/state/README.md](../pkg/state/README.md)). + +### ST-7 — Stop checkpoints conservatively + +On stop, every non-terminal task goes `stopped` regardless of engine sub-state — a task is +never promoted to `completed` on partial engine progress, and the stop snapshot is taken only +after the engine's own stop returns. *Enforced:* stop handling in `pkg/tern`. + +### ST-8 — Every state has a recovery story + +Every apply state is registered with metadata (label, terminal classification), and every +non-terminal state an apply can sit in is either re-claimable once its driver's heartbeat goes +stale or covered by its own named recovery path. A state added without a recovery path is a CI +failure, not a production orphan. *Enforced:* the state metadata registry and a +reflection-driven completeness test over it (`pkg/state`, `pkg/storage/internal/sqlstore`). + +### ST-9 — The retry budget is bounded and never re-runs finished work + +`failed_retryable` is active, not terminal: recovery re-drives it automatically, but only +`failed_retryable` tasks reset to `pending` — completed tasks are never re-run — and the apply +settles to permanent `failed` when the attempt budget is spent or the recovery window closes. +*Enforced:* retry preparation and expiry in the operator and storage layers; budget semantics +in [apply-lifecycle.md](apply-lifecycle.md). + +### ST-10 — Rollouts respect order and fail closed on policy + +Multi-deployment rollouts claim operations in `deployment_order`, and a failed earlier +deployment blocks later ones unless the config says otherwise. Once an operator releases a +paused rollout it stays released — there is no path back to paused. And an *unrecognized* +`on_failure` value behaves like `halt`, never like `continue`. *Enforced:* the ordered-claim +gate in `FindNextApplyOperation` and the rollout state derivation (`pkg/state`). + +## Ownership and leases (OW) + +### OW-1 — Every drive runs under a claim + +Before a pod may do any work on an apply, it must claim that work in storage. The claim is one +atomic transaction (`FOR UPDATE SKIP LOCKED`) that stamps the row with the claimer's lease — +owner, a fresh token, a heartbeat — so two pods can never pick up the same work. This holds +for every drive including the very first: the pod that accepted the request gets no shortcut +and claims the apply like any other driver. The wake signal sent after enqueueing only makes a +driver check for claimable work sooner; it never hands work to a pod directly. *Enforced:* the +operator claim path (`pkg/api`, `pkg/storage/internal/sqlstore`). + +### OW-2 — Lease-scoped writes must hold the current token + +Apply and task updates, heartbeats, apply logs, and recovered observer side effects all carry +the lease token; when another driver has claimed the row and rotated the token, the displaced +driver's writes fail with an ownership error and it exits rather than posting stale comments or +overwriting newer state. *Enforced:* a token check on every lease-scoped storage write. + +### OW-3 — A driver yields no later than a peer may reclaim + +The heartbeat loop and the claim query read the same staleness window, so a driver that cannot +heartbeat stops driving at or before the moment a peer could consider the row abandoned and +claim it. The same schema change can never be driven by two pods at once. *Enforced:* one +shared staleness constant used by both the heartbeat loop and the claim query. + +### OW-4 — Lease loss is proven, never inferred + +A driver self-fences only after a successful read proves the token changed; a connection error +or transient storage failure is retried, never misread as displacement. *Breaks if violated:* a +storage blip aborts a healthy multi-hour copy. *Enforced:* the split-brain fences in the drive +loop ([storage-outage-behavior.md](storage-outage-behavior.md)). + +### OW-5 — One active apply per target + +At most one active apply exists per (database, database type, environment), enforced whenever +an apply is created or moved back into an active state, under a per-target named lock — and the +reservation covers the apply's whole deployment set until the parent settles. This invariant +does not depend on a user-facing database lock being held: direct API callers and `--no-lock` +flows are equally bound. *Enforced:* the exclusivity check in the storage apply create/activate +path. + +### OW-6 — There is one way to claim work + +Exactly one claim query and one lease-rotation path exist — no side doors. Eligibility rules +live inside the claim query itself, not in checks that run after it: once a destructive +transition (say, `waiting_for_cutover → cutting_over`) has been claimed, the work is already +in motion, and discovering only then that it should not have been claimed is too late. +*Enforced:* the single operation-claim query; tests pin its state arms against divergence. + +### OW-7 — Ownership uncertainty fails closed + +A driver that cannot prove it owns work — or is handed work outside the databases its engine +client can reach — refuses it, leaving the row untouched for a pod that can drive it. A claim +that cannot proceed releases its lease immediately rather than holding the row idle until the +staleness window expires. And a pod never trusts its own in-memory engine state to answer +questions about work it does not own. *Enforced:* scope checks and token-guarded claim release +in the drive path. + +## Control operations (CO) + +Full doctrine: [grpc-control-edge-cases.md](grpc-control-edge-cases.md) and the Control +Operations section of [AGENTS.md](../AGENTS.md). + +### CO-1 — Durable before acknowledged + +An accepted control request (stop, start, cutover, cancel, volume, revert, skip-revert) is +recorded in `apply_control_requests` before SchemaBot reports acceptance; drivers consume it +from storage. A pod restart, lease handover, or webhook redelivery never loses an acknowledged +command; in-memory flags are per-drive optimizations, never the source of truth. *Enforced:* +the durable request path in `pkg/api` / `pkg/webhook`. + +### CO-2 — Every command resolves; none wedges + +A control request resolves to an explicit durable outcome — applied, superseded, or failed. A +command may delay a drive but never wedge it: a failed request requires fresh operator intent +rather than being retried forever, and polling windows are bounded with visible timeout +failures. *Breaks if violated:* an apply loops on a doomed command while holding its database +lock. *Enforced:* request completion and bounded-retry rules in the drive loop. + +### CO-3 — Engine terminal truth outranks a queued command + +When the engine's own record shows the change already settled, the drive adopts that outcome +instead of fighting it — a cancel against a deploy that already completed records the apply as +completed rather than re-sending the cancel forever. Only engines whose backend holds the +authoritative record of the change (PlanetScale, where the deploy request lives server-side) +are consulted this way; for all others the question fails closed. Only settled outcomes are +adopted — a remote state still in motion never is. *Enforced:* terminal-truth preflights on +the control paths in `pkg/tern`. + +### CO-4 — Stop wins + +Stop is the highest-priority intent: once accepted, no actor may knowingly advance the apply +toward deploy, cutover, or completion until the stop is processed — forward-progress commands +are rejected while a stop pends, and contradictory intents are rejected at acceptance rather +than resolved by drive ordering. *Enforced:* pending-stop checks in pollers and the cutover +paths; conflict rejection at request intake. + +### CO-5 — The revert phase owns the outcome + +Once an apply is `reverting` or `skipping_revert`, stop, cancel, and cutover are refused for +the whole phase — storage must never settle on a state that contradicts what the engine is +still doing to the database underneath. *Enforced:* revert-phase gates in the control paths +(`pkg/tern`). + +### CO-6 — Commands act only where they have an effect + +A control operation is rejected up front when nothing can service it — a volume change on an +apply that is finished or has no row copy running, a second cutover while one is already in +flight — instead of being queued for a consumer that will never come. And its effect is scoped +to the one change it targets: an incident-time tuning or one operation's completion never +bleeds onto sibling operations or future applies. *Enforced:* queue-time eligibility gates; +operation-scoped request rows. + +### CO-7 — ID namespaces are never conflated + +Remote data-plane RPCs identify work by the operation's `external_id`; every user-facing +surface (logs, comments, checks, CLI, HTTP) uses the string `apply_identifier`. Sending the +user-facing ID to a remote engine is a routing bug, and the internal numeric row ID appears on +no surface at all. *Enforced:* the remote apply ID invariant +([grpc-control-edge-cases.md](grpc-control-edge-cases.md)); ID resolution at the API boundary. + +### CO-8 — Stop terminality is engine truth, told truthfully + +A Spirit stop is a resumable pause; a PlanetScale stop cancels the deploy request permanently. +The distinction is derived from the engine, not inferred at render time, and every operator +surface says which one happened. *Enforced:* the stop-terminality policy helper and its +renderings. + +### CO-9 — Control operations are stateless + +Any instance can serve any control operation, because controls act by writing shared durable +state — storage rows, or marker tables on the target database itself — never by reaching into +a specific process's in-memory engine. A cutover request landing on a pod that is not driving +the apply still works. *Enforced:* design rule in [architecture.md](architecture.md); the +cutover sentinel table on the target database. + +## Recovery (RC) + +### RC-1 — Nothing is orphaned + +A crash between any two related writes is reconcilable afterward, in both directions. When an +operation reaches a terminal state but its parent apply was never updated, the parent's state +is re-derived from its operations. When the parent has settled but an operation row never +started, a reaper marks that row with the parent's outcome — though only once the parent is +settled and quiet, and never while the parent is `stopped` or `failed_retryable` (those may +still resume). *Enforced:* bidirectional reconciliation in the operator and the +stranded-operation reaper. + +### RC-2 — Settled applies stop changing + +No claim, reaper, or recovery path touches an apply whose verdict is recorded; an apply that +finished last week does not have its rows change today. *Enforced:* settled-state exclusions in +every claim and sweep query (and ST-1 backstops all of them). + +### RC-3 — Loading nothing is not owning nothing + +A loader returning zero rows because of an error is never conflated with an apply that +genuinely owns zero rows: the first blocks and surfaces, only the second may complete as a +no-op. *Breaks if violated:* completion reports success for DDL that never ran. *Enforced:* +separated error/empty handling on recovery load paths. + +### RC-4 — Self-healing needs proof + +Automatic cleanup acts only where the record provably carries no engine work (a `pending` task +has no checkpoint by construction); anything uncertain keeps blocking for an operator. +*Enforced:* narrow eligibility conditions on every self-heal path. + +### RC-5 — Terminal notifications fire exactly once + +The terminal summary (PR comment, check completion) is authorized by an atomic claim that +permanently hands the job to exactly one driver. Unlike a lease, this claim cannot expire and +be re-granted — so no later driver can post the summary a second time, and no handover can +drop it. *Enforced:* atomic observer/notification claims in the drive and recovery paths. + +## Review integrity and data safety (RV) + +### RV-1 — What gets applied is what was reviewed + +A deployment applying a plan reviewed elsewhere independently re-derives the change set and +compares immediately before each per-task engine apply; drift — including DDL it cannot parse — +fails closed. Recomputed deltas are never applied silently. *Enforced:* pre-apply change-set +verification in the multi-deployment apply path. + +### RV-2 — Stale plans never apply + +An apply is rejected when the plan's base is no longer the branch's live base — freshness is +mandatory, is checked at apply time against the current base, and cannot be waived by +`--allow-unsafe`. Plan comments are suppressed as duplicates only when SchemaBot can prove it +is not preserving a stale plan. *Enforced:* the plan-freshness gate on both apply entry points. + +### RV-3 — Consent is explicit, specific, and re-checked + +Unsafe changes (error-severity lint findings: drops, column narrowing) block without +`--allow-unsafe`. Changes an operator cannot undo mid-flight (direct execution's +write-blocking DDL, with no cutover and no revert) require the operator to confirm the +specific consequences disclosed to them — and the re-plan that runs just before execution +re-checks that verdict, so a plan that changed after the confirmation stops rather than +running something the operator never saw. *Enforced:* lint gates and the apply-confirm flow. + +### RV-4 — Engine refusals are known at plan time and gate the apply + +Whether the engine will refuse a statement — or route it to direct execution, a MySQL/Spirit +execution mode — is recorded on the plan using the engine's own checks, not a reimplementation +of them, and an apply on a refused plan is rejected before any lock is taken. For direct +execution's table-size bound, a table whose size cannot be measured is blocked, and a row +estimate is trusted only in the blocking direction — an estimate alone never approves. +*Enforced:* plan-time execution verdicts; apply-time verdict gates; the direct-execution size +bound ([direct-execution.md](direct-execution.md)). + +### RV-5 — A dropped table stays recoverable + +`DROP TABLE` never destroys data at execution time; the table is held for a recovery window +instead. On MySQL, the Spirit engine quarantines the drop as an atomic rename into +`_pending_drops` with a UTC timestamp prefix — tables without a valid timestamp prefix are +never auto-dropped (age unknown), and cleanup passes are serialized per target under an +advisory lock. On Vitess, the drop rides Vitess's native table lifecycle, which holds dropped +tables before purging them. *Enforced:* the pending-drops lifecycle +([pending-drops.md](pending-drops.md)) on Spirit; the table lifecycle native to Vitess online +DDL on PlanetScale. + +### RV-6 — Every statement parses, or it is an error + +All SQL SchemaBot processes must parse with the TiDB parser — no string-splitting fallback, no +silently skipped statements — and a classifier refuses ambiguity (a compound statement never +classifies as its first verb) rather than letting a destructive statement ride past +classification-based gating. *Enforced:* the `pkg/ddl` / Spirit `statement` boundary. + +### RV-7 — Rollback needs the originals + +Rollback requires the original schema files captured at apply time; if artifact capture failed, +rollback is refused rather than reconstructed. *Enforced:* rollback preconditions on stored +artifacts. + +### RV-8 — The plan sees the whole schema, or nothing + +Schema discovery fails closed on any partial view: a GitHub directory listing at the API's +entry cap, a truncated git tree, or config discovery that cannot prove exhaustiveness aborts +the plan — a missing file must never read as a deleted table and produce a spurious +`DROP TABLE` proposal. Symlinked namespaces resolving outside the repository root (or to +themselves) are rejected. *Enforced:* truncation and symlink guards on every schema-fetch path +in `pkg/github`. + +## Routing and authorization (AZ) + +### AZ-1 — The server decides where a change runs + +Targets and deployments resolve from server-owned config; the resolved route is stored on the +plan and reused by the apply, and request decoding rejects client-supplied routing fields. A +schema source (repo + path) is authorized against server config at plan time and re-checked +before apply; missing config fails closed. *Enforced:* server-side routing and source policy in +`pkg/api`. + +### AZ-2 — Authorization fails closed at every tier + +An API route not classified as a read is treated as a write; planning counts as a write (it +stages a change); a target that cannot be resolved never authorizes; a configured grant that +could never match any request is a startup error rather than silent dead config; and a new +mutating endpoint cannot ship without a test proving it denies unauthorized callers. +*Enforced:* route classification with a structural sweep test over the route table. + +### AZ-3 — Identity comes from a verified lane + +Forwarded identity headers are trusted only from explicitly listed proxies; a gateway-verified +service caller never reaches the write tier by self-asserting user headers; and every write and +control action is attributed to the verified caller — a client-supplied caller string never +overrides. *Enforced:* the trust-anchor config (refuses to start without one) and +identity-precedence rules in the auth layer. + +### AZ-4 — Applying takes an authorized actor + +A PR apply requires an actor authorized for the target (configured operators, admin teams, +repo admins, or CODEOWNERS, per config), evaluated per database — and the change's author +cannot satisfy their own review requirement. *Enforced:* the review gate and actor +authorization in `pkg/webhook`. + +### AZ-5 — Commands never guess + +An unscoped PR command resolves to exactly one unambiguous database or is rejected with +guidance — never an arbitrary pick — and a malformed command is rejected rather than +"helpfully" corrected into something executable (especially one carrying `--allow-unsafe`). +Every command receives a response; silence only ever means another instance owns the reply. +*Enforced:* command discovery and the unowned-command policy in `pkg/webhook`. + +## Availability and blast radius (AV) + +Full model: [storage-outage-behavior.md](storage-outage-behavior.md). + +### AV-1 — Schema changes run without GitHub + +SchemaBot works during GitHub outages: CLI and API applies work when GitHub is down, remote +data-plane instances never need a GitHub App, and observer side effects (PR comments, progress +edits) are best-effort — logged on failure, never blocking or failing the change. GitHub is an +observer of execution, never a dependency of it. *Enforced:* the comment-observer pattern in +`pkg/webhook`; data-plane deployments carry no GitHub credentials. + +### AV-2 — Outages degrade, never destroy + +A storage outage degrades the control plane only: in-flight schema changes keep running, +because the safety mechanisms that protect the target database (throttling, lock timeouts, +checksums) live in the engine, not in the drive loop — and uncertainty is never converted into +a verdict. A terminal outcome that cannot be persisted leaves the row non-terminal and +claimable, to be re-derived on the next claim rather than fabricated. *Enforced:* the +per-failure-class handling in the drive loops +([storage-outage-behavior.md](storage-outage-behavior.md)). + +### AV-3 — Liveness is process-only + +Storage reachability belongs on readiness, never liveness — restarting a pod cannot fix an +unreachable database and only aborts in-flight local drives on top of the outage. Boot retries +storage inside the startup budget, re-resolving credentials each attempt. *Enforced:* +`/livez` vs `/health` probe wiring. + +### AV-4 — A transient failure never fails the work + +A storage or transport blip is not a schema-change failure. An error writing a progress update +is logged and the drive continues; an error reading something safety-gating ends this drive +attempt and leaves the row claimable for another; and repeated errors observing remote +progress mark the apply `failed_retryable` — they never trigger a remote stop, because an +observation outage only proves the control plane cannot see, not that the change is unhealthy. +*Enforced:* failure-class handling in the drive loop and the remote progress error limit. + +### AV-5 — Panics are contained and permanent + +A panic is contained to the unit of work that caused it, converted into a permanent (not +retryable) failure — retrying would feed the poisoned input straight back — and made loud with +a log, a durable apply-log entry, and a metric. One bad row must never crash-loop the fleet. +*Enforced:* recover boundaries around drives and webhook work units. + +### AV-6 — An old binary never destroys newer state + +During a rolling deploy, a binary that encounters state written by a newer version skips it and +continues — the one sanctioned exception to fail-closed, because crash-looping every old pod is +worse. It may never delete, rewrite, or terminalize what it does not understand. *Enforced:* +version-aware skip guards on shared-state consumers. + +### AV-7 — An acknowledged delivery is a promise + +A webhook delivery acknowledged with 200 is durable work: it is recorded in the event inbox +before acknowledgment, deduplicated by delivery GUID so a redelivery's side effects run at +most once, drained on shutdown so a rolling deploy loses nothing, and — when a row proves +unprocessable — terminated visibly rather than retried forever or silently dropped. +*Enforced:* the durable webhook inbox — durable dispatch in `pkg/webhook`, backed by the +`webhook_events` table. + +### AV-8 — Untrusted text never reaches operator surfaces raw + +Raw error strings (dial failures, DSN fragments, hostnames) are never rendered into PR or +check markdown — surfaces show a fixed sanitized line while the raw error goes to server logs +with triage identifiers — and inbound payloads are size-bounded before decode. *Enforced:* +sanitized rendering helpers in `pkg/webhook`; request body limits. + +## Structural enforcement + +The strongest invariants are enforced by structure, so regressions fail CI instead of review: + +- **State completeness** — a reflection-driven test walks every registered apply state and + fails on any state that is neither terminal, claimable, nor explicitly allowlisted with its + own recovery arm (ST-8). +- **Route-table sweep** — every mutating endpoint requires a denial fixture proving its + handler-level authorization check (AZ-2). +- **Engine conformance** — engines run a shared conformance kit; an engine that cannot exhibit + a case documents an explicit skip, never a silent opt-out (CO-3, CO-8). +- **Claim parity** — the states the claim query accepts are pinned by tests, so a state added + to one copy of a safety-critical condition cannot silently diverge from another (OW-6). + +New invariants should aspire to this tier: when adding one, prefer a completeness test over the +relevant registry to a hand-maintained list. + +## Engineering rules live in AGENTS.md + +The process-level rules that keep this codebase reviewable — never silently fail; error early, +never swallow; no silent branch cases; wrap errors with identifiers; logs answer the triage +question; name compound predicates; one owner closes a handle; tests prove documented +behavior — belong in [AGENTS.md](../AGENTS.md), not here. This document holds invariants about +the **running system**; that one holds invariants about **how we write and review the code**.